Efficient scalable coding concept

Abstract

To further improve scalable coding concepts for scalable video coding.SOLUTION: Parallel decoding of inter-dependent layers of a multi-layer video data stream 40 is improved by introducing a long-term syntax element structure 642 which, when assuming a certain value, guarantees to a video decoder that during a predetermined time period greater than short-term syntax element's 602 time intervals, the pictures of the dependent layer are subdivided so that borders between the spatial segments of pictures 15 of a layer 1 overlay every border of the spatial segments of a layer 0. A syntax element is introduced which informs a decoder that interpolation along a base layer's partitions is modified so as not to mix-up pixels / pels of neighboring partitions of a base layer picture. A long-term syntax element structure is introduced which allows the decoder to determine an inter-layer offset for a predetermined time period. Different codecs / standards are allowed to be used for different layers.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present application relates to scalable coding concepts, such as scalable video coding. [Background technology]

[0002] The concept of scalable coding is well known to those skilled in the art. In video coding, for example, H.264 includes an SVC extension (Scalable Video Coding) that enables a base-layer coded video data stream to be accompanied by additional enhancement layer data to improve the playback quality of the base-layer quality video under different conditions, such as spatial resolution, signal-to-noise ratio (SNR), etc. The recently finalized HEVC standard is also extended by an SVC profile. HEVC differs from its predecessor, H.264, in many aspects, such as parallel decoding / encoding and suitability for low-delay transmission. As for parallel encoding / decoding, HEVC supports WPP (wavefront parallelism) encoding / decoding as well as the tile parallelism concept. According to the WPP concept, individual images are divided into substreams in a row-wise manner. The coding order within each substream is directed from left to right. The substreams have a defined decoding order between them, proceeding from the top substream to the bottom substream. Entropy coding of the substreams is performed using probability adaptation. The probability initialization is performed for each substream independently, or based on a pre-adapted state of the probabilities used in entropy coding the immediately preceding substream, from the left end of the preceding substream, such as the end of the second CTB (coding tree block), to a specific position. Spatial prediction does not need to be restricted; that is, spatial prediction can cross the boundary between immediately succeeding substreams. In this way, such substreams can be coded / decoded in parallel with the currently coding / decoding position forming the traveling wavefront, in a gradient manner proceeding from left to right, from the bottom left to the top right. According to the tile concept, the image is divided into tiles, and spatial prediction across tile boundaries is prohibited to allow parallel processing of the coding / decoding of these tiles. Only in-loop filtering across tile boundaries is allowed.To support low-latency processing, the slice concept has been extended: slices can also be switched to initialize new entropy probabilities using the saved entropy probabilities while processing the previous substream, i.e., the substream preceding the substream where the current slice begins, and to use the entropy probabilities that have been continuously updated up to the end of the previous slice. This approach makes the WPP and tile concepts more suitable for low-latency processing.

[0003] Nevertheless, it would be more advantageous to have concepts at hand that further improve the scalable coding concept. Summary of the Invention [Problem to be solved by the invention]

[0004] It is therefore an object of the present invention to provide a concept that further improves the scalable coding concept. [Means for solving the problem]

[0005] This object is achieved by the subject matter of the pending independent claims.

[0006] A first aspect of the present application relates to scalable video coding in conjunction with parallel processing concepts. Parallel processing concepts such as WPP and tile division enable parallel decoding of images of a video in spatial segments, where the image is subdivided, for example, into substreams, tiles, or slices. Like spatial intra-image prediction, inter-layer prediction limits the degree of parallelism when decoding layers that are dependent on each other via inter-layer prediction. More precisely, spatial intra-layer prediction limits the degree of parallelism when decoding images of a single layer, but the problem is solved in a different way. For example, when using tiles as spatial segments, spatial intra-layer prediction is restricted to not cross tile boundaries. In the case of WPP substreams, the parallelism is performed in a staggered manner, resulting in appropriate tilting of the wavefront. In the case of inter-layer prediction, decoding of a dependent layer becomes dependent on the co-located portion of a reference layer. Therefore, once the co-located portion of the reference layer has already been processed / decoded, decoding of the spatial segment of the dependent layer can begin as soon as possible. The area of ​​the "co-located portion" is expanded when enabling "motion compensation," such as in the case of inter-layer prediction in the case of different views as different layers, or for upsampling from a lower layer to a higher layer. That is, it is feasible for a video decoder to support scalable decoding using inter-layer prediction and parallel decoding, in which the short-term syntax elements define a subdivision of the images of these inter-dependent layers into spatial segments, and the degree of parallelism in parallel processing of the inter-dependent layers is derived from the short-term syntax elements for these inter-dependent layers. These short-term syntax elements define how to divide the images of these inter-dependent layers into their spatial segments. However, doing so consistently is cumbersome and computationally complex. Beyond that, when doing so, the video decoder cannot properly schedule decoding threads running in parallel to decode the multi-layer video data stream.Thus, according to a first aspect of the present invention, parallel decoding of interdependent layers of a multi-layer video data stream is improved by introducing a long-term syntax element structure that, assuming certain values, ensures that a video decoder subdivides pictures of dependent layers such that boundaries between spatial segments of pictures of a second layer overlap with any boundaries of spatial segments of a first layer for a predetermined period greater than the time interval of the short-term syntax element. According to this approach, the video decoder can rely on the fact that the multi-layer video data stream was properly encoded so that the subdivision of pictures of interdependent layers into spatial segments does not inadvertently reduce the degree of parallelism possible between these interdependent layers. Rather, the decoder can schedule the distribution of spatial segments onto the parallel processing threads of the video decoder in advance by utilizing the constraint that boundaries of spatial segments of different layers overlap each other in the indicated manner for a predetermined period. However, the long-term syntax element structure makes it possible to turn off this guarantee, thereby making it possible in other application scenarios or for high-end video decoders to perform parallel processing scheduling on a short-term basis depending only on short-term syntax elements, i.e., without utilizing any guarantee regarding the relative positions between spatial segment boundaries of interdependent layers. Long-term syntax elements can also be used for opportunistic decoding decisions.

[0007] In connection with the parallel processing of interdependent layers, another aspect of the present application relates to scalable coding in which images of interdependent layers are dependent on inter-layer prediction using upsampling from a base layer to an enhancement layer. In particular, this aspect relates to interpolation used to perform upsampling from a base layer to an enhancement layer. Typically, this type of interpolation causes adjacent partitions of a base layer image to be dependent on each other. That is, the interpolation causes the interpolation results at the outer circumference of an upsampled portion of a base layer reference image to depend not only on pixels / pels of adjacent partitions but also on pixels / pels between the co-located portions of the base layer image. In other words, the region of the base layer image serving as a reference for inter-layer prediction for the co-located portion to be predicted in the enhancement layer image "smears out" and expands. This type of interdependence caused by the interpolation of inter-layer prediction inadvertently adversely affects the degree of parallelism achievable in the parallel processing of interdependent layers. According to this second aspect of the present application, a syntax element is introduced that informs a decoder that interpolation along a base layer partition is modified so as not to cause confusion of pixels / pels of adjacent partitions of the base layer image, partitions of the base layer image and the enhancement layer image or its upsampled version depending on, for example, spatial segments of both the base and enhancement layers. By introducing this kind of syntax element, the encoder can switch between two modes: when the interpolation is constrained to keep parts of the base layer image self-contained, i.e., when the constraint is on, the maximally achievable degree of parallelism of the parallel decoding of interdependent layers increases with slightly decreasing interpolation quality along the partition edges of the base layer image, and when the interpolation is not constrained, the degree of parallelism, however, decreases with increasing interpolation quality at the partition edges.

[0008] A third aspect of the present application relates to scalable video coding with parallel decoding of interdependent layers and seeks to reduce the decoder's burden of performing parallel processing scheduling, i.e., by introducing a long-term syntax element structure that requires the decoder to distribute spatial segments onto parallel processing threads so that the decoder can determine the interlayer offset—or interlayer delay—for a predetermined period longer than a time interval in which short-term syntax elements indicate the size and position of spatial segments of images of interdependent layers and the spatial sampling resolution of these images. By introducing a long-term syntax element that signals the interlayer offset, a video encoder can switch between two modes: according to the first mode, the encoder guarantees the decoder a certain interlayer offset corresponding to a certain degree of parallelism during decoding of the interdependent layers, and then correspondingly sets the short-term syntax element so that within a predetermined period, the actual interlayer offset is equal to or even lower than the guaranteed one. According to other modes, no such guarantees are provided to the decoder, and therefore the encoder may freely set the short-term syntax elements to meet other criteria, e.g., optimally adapting the short-term syntax elements to the video content for a predetermined period, etc. According to this count over the entire predetermined period, the inter-layer offset thus explicitly indicated in the data stream may be the count of base layer spatial segments that must at least be decoded before the first spatial segment of the temporally co-aligned enhancement layer image starts, when this count does not lead to any conflicts with decoding the first spatial segment of the enhancement layer image for this predetermined period.

[0009] A fourth aspect of the present application relates to scalable video coding and signaling of the layers to which various NAL units of a multi-layer video data stream belong, as well as the positions of these layers within the scalability space as well as the meaning of the scalability dimensions that measure the scalability space. To reduce the tasks to be performed by intermediate network entities, including those forwarding the multi-layer video data stream, this type of information should be easily accessible and easily manageable for these intermediate network entities. The fourth aspect of the present application is based on the inventor's discovery, according to which typical application scenarios justify the expense of a type indicator field that changes the way layer indicator fields within NAL unit headers associated with various layers are to be interpreted: if the type indicator field has a first state, mapping information in the general information NAL unit maps the possible values ​​of the layer indicator field to operation points, and the NAL units of the layers are associated with the operation points using the respective layer indicator fields and mapping information. In this way, the mapping between layers and scalability arrangements can be variably adapted, allowing a large number of scalability spaces to be realized, but with the drawback of increased management overhead. If the type indicator field has the second state, the layer indicator field is divided into multiple parts, and the operational point to which each NAL unit is associated is established using the values ​​of these parts as the coordinates of a vector in the scalability space. With this approach, the mapping between layers and scalability arrangements advantageously allows a small number of scalability spaces to be realized, while reducing management overhead for network entities. The layer indicator field can be the same in both cases regardless of the application scenario, but the way in which the layer indicator field passes the NAL units of a layer through the scalability space can be compatible with this application and its specifications.The conformance effect more than compensates for the need for the additional expense of the type indicator field.

[0010] A fifth aspect of the present application relates to multi-layer video coding, i.e., scalable video coding, which allows different codecs / standards to be used for different layers. The possibility that different codecs / standards can be used for successive layers allows for the late extension of existing video environments to accommodate a multi-layer video data stream that is subsequently enhanced by further enhancement layers, and to use new and potentially better codecs / standards to achieve this. A network sink that cannot understand the codecs / standards of some enhancement layers is still able to process lower layers, and the multi-codec decoder is supplied by a transport layer decoder that identifies, for each NAL unit of the multi-layer video data stream, which codec it is associated with, and therefore hands over the NAL units of the multi-layer video data stream to the multi-standard multi-layer decoder.

[0011] A sixth aspect of the present application relates to multi-layer video coding in which both a base layer image and an enhancement layer image are subdivided into arrays of blocks. In this case, an inter-layer offset can be efficiently indicated by inserting a syntax element structure into a multi-layer video data stream, which indicates the inter-layer offset for parallel decoding of base and enhancement layer images in units of base layer blocks. That is, the sixth aspect of the present application is based on the discovery that explicit transmission of a syntax element structure indicating the inter-layer offset between a base layer and an enhancement layer in units of base layer blocks merely increases the data to be transmitted in only a minor way, but the computational complexity for a decoder is significantly reduced compared to the case in which the decoder would otherwise derive the inter-layer offset for parallel decoding of base and enhancement layer images based on other syntax elements representing, for example, block sizes of base and enhancement layer blocks, sample resolutions of base and enhancement layer images, etc. The sixth aspect is closely related to the third aspect when implementing a syntax element structure as a long-term syntax element structure such that the inter-layer offset is indicated to the decoder as a guarantee that the inter-layer offset is valid for a predetermined period of time that is greater than the time interval that short-term syntax elements in the multi-layer video data stream indicate those hints that would otherwise be necessary to determine the inter-layer offset by incorporating these syntax elements in a relatively complex manner.

[0012] Of course, all of the above aspects may be combined in sets, triplets, quadlets, or all of the above.

[0013] Preferred embodiments of the present application are described in detail below with reference to the drawings. [Brief explanation of the drawings]

[0014] [Figure 1]FIG. 1 shows a video encoder that serves as an illustrative example for implementing any of the multi-layer encoders outlined further with respect to the following figures. [Figure 2] FIG. 2 shows a schematic block diagram of a video decoder compatible with the video encoder of FIG. [Figure 3] FIG. 3 shows a block diagram of an image subdivided into sub-streams for WPP processing. [Figure 4] FIG. 4 shows a schematic diagram of a video decoder according to an embodiment in which inter-layer ordering of spatial segments of base and enhancement layers is utilized to ease the decoding process. [Figure 5] FIG. 5 shows a structural diagram of an image subdivided into coding blocks and tiles, with tiles containing integer multiples of coding blocks and a decoding order defined among the coding blocks after subdivision of the image into tiles, respectively. [Figure 6] FIG. 6 is a diagram showing an example of syntax for implementing the embodiment of FIG. [Figure 7] FIG. 7 shows a pair of structural diagrams of a base layer and an enhancement layer image that are differently subdivided into tiles. [Figure 8] FIG. 8 illustrates another exemplary syntax that can be applied in conjunction with the embodiment of FIG. [Figure 9] FIG. 9 shows a block diagram of an image, its subdivision into tiles and application of interpolation filters to perform upsampling for inter-layer prediction. [Figure 10] FIG. 10 shows a schematic block diagram of a multi-layer decoder configured to respond to a syntax element in a multi-layer data stream to switch on or off upsampling interpolation separation. [Figure 11]FIG. 11 illustrates a diagram of a pair of base layer and enhancement layer images using inter-layer prediction from the base layer to the enhancement layer using upsampling to pass from the base layer sample resolution to the increased enhancement layer sample resolution. [Figure 12] FIG. 12 shows a block diagram illustrating the switchable upsampling interpolation separation according to FIG. [Figure 13] FIG. 13 shows a block diagram of an overlay of base layer and enhancement layer images, both subdivided into WPP sub-streams. [Figure 14] FIG. 14 illustrates an exemplary syntax for implementing the embodiment of FIG. [Figure 15a] FIG. 15a shows a diagram of a spatially aligned overlay of base and enhancement layer images, both of which are differently subdivided into spatial segments. [Figure 15b] FIG. 15b shows a structural diagram of the overlay of the base and enhancement layer images of FIG. 15a, but illustrates another possibility for selecting the partition on which the upsampling interpolation separation is performed. [Figure 16] FIG. 16 shows a schematic block diagram of a video decoder according to an embodiment in which the video decoder responds to a long-term syntax element structure within a multi-layer video data stream to derive or not derive guarantees regarding inter-layer offsets between base and enhancement layer decoding. [Figure 17a] FIG. 17a shows a block diagram of a pair of base layer and enhancement layer images, both subdivided into tiles, to illustrate the inter-layer offset signaling conveyed by the long-term syntax element structure of FIG. 16 according to an embodiment. [Figure 17b]FIG. 17b shows a structural diagram of a pair of base and enhancement layer images, both subdivided into sub-streams for WPP processing, to illustrate an example for implementing the long-term syntax element structure of FIG. 16 according to an embodiment. [Figure 17c] FIG. 17c shows a block diagram of a pair of base and enhancement layer images, both subdivided into slices, to illustrate an embodiment for realizing the long-term syntax element structure of FIG. 16 according to a further embodiment. [Figure 18] FIG. 18 shows a block diagram of an image subdivided into sub-streams for WPP processing, additionally showing the wavefront resulting from parallel decoding / encoding of an image using WPP according to an embodiment. [Figure 19] FIG. 19 is a diagram illustrating a table illustrating the relationship between the minimum inter-layer decoding offset and block size and the sampling resolution ratio between the base and enhancement layers according to an embodiment of the present application. [Figure 20] FIG. 20 is a diagram illustrating an exemplary syntax for implementing the long-term syntax element structure signaling according to FIG. [Figure 21] FIG. 21 is a diagram illustrating another embodiment for syntax for implementing the embodiment of FIG. [Figure 22] FIG. 22 is a diagram illustrating the syntax of a NAL unit header according to an embodiment for an HEVC-like structure. [Figure 23] FIG. 23 illustrates a block diagram of a network entity in accordance with an embodiment in which scalable coding is mitigated by allowing switching between various layer indicator field interpretations. [Figure 24] FIG. 24 shows a block diagram illustrating a method for switching in response to a type indicator field. [Figure 25] FIG. 25 shows an illustrative block diagram illustrating switchable layer indicator field interpretation in accordance with an embodiment in more detail. [Figure 26]FIG. 26 is a diagram illustrating an illustrative syntax for implementing the switchable layer indicator field interpretation according to FIG. [Figure 27] FIG. 27 is a block diagram illustrating the layer indicator field switching for the syntax of FIG. [Figure 28] FIG. 28 shows a block diagram of a transport stream decoder configured to simply discard enhancement layer NAL units of codecs other than the base layer codec. [Figure 29] FIG. 29 shows a block diagram of a transport stream decoder interfacing with a single-standard multi-layer decoder, illustrating the behavior of the transport stream decoder according to an embodiment. [Figure 30] FIG. 30 illustrates a transport stream decoder interfacing with a multi-standard multi-layer decoder and the behavior of the transport stream decoder according to an embodiment. [Figure 31] FIG. 31 is a diagram illustrating another embodiment for syntax for implementing switchable layer indicator field interpretation according to a further embodiment. [Figure 32] FIG. 32 is a diagram illustrating an image of a given layer subdivided into blocks showing a further subdivision of the image into spatial segments. [Figure 33] FIG. 33 shows a diagram of an image of an arbitrary layer subdivided into blocks and tiles. [Figure 34] FIG. 34 shows a block diagram of an image subdivided into blocks and substreams. [Figure 35] FIG. 35 illustrates a schematic block diagram of a video decoder according to an embodiment in which the video decoder is configured to use a syntax element structure in a data stream to derive inter-layer offsets between block-based inter-predictively processed images having a raster scan order defined therebetween. [Figure 36]FIG. 36 is a block diagram illustrating an operable mode of operation of the video decoder of FIG. 34 with respect to syntax element structures within a data stream according to an embodiment. [Figure 37] Figure 37 shows a block diagram illustrating the operational modes of the video decoder of Figure 35 in connection with a further embodiment whereby inter-layer offset signaling is switchable between different distinct signaling types, i.e., signaling in units of different types. [Figure 38] FIG. 38 shows a block diagram illustrating a mode of operation of the video decoder of FIG. 35 according to a further embodiment in which inter-layer offsets are successively examined during parallel decoding of the base enhancement layer picture. [Figure 39] FIG. 39 shows the relationship between the rank of a particular block of an image according to the raster scan decoding order on the one hand and its row and column index according to an embodiment on the other hand. [Figure 40] FIG. 40 is a diagram illustrating various embodiments for relatively regular subdivision of base and enhancement layer images into blocks, and the results resulting from these various embodiments. [Figure 41] FIG. 41 is a diagram illustrating an example syntax for implementing any of the examples of FIGS. [Figure 42] FIG. 42 shows an example of another syntax as an alternative to the one in FIG. [Figure 43] FIG. 43 is a diagram illustrating an example of syntax in which the embodiments of FIGS. 16 and 35 may be indicated in other portions of a multi-layer data stream. [Figure 44] FIG. 44 shows a schematic block diagram of a transport layer decoder interfacing with a multi-layer multi-standard decoder according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] First, as an overview, an example is given for an encoder / decoder structure, which fits any of the concepts presented thereafter.

[0016] FIG. 1 shows the general structure of an encoder according to an embodiment. The encoder 10 may be implemented in a multi-threaded manner or not, i.e., simply to operate in a single thread. That is, the encoder 10 may be implemented using, for example, multiple CPU cores. In other words, the encoder 10 may support parallel processing, but need not do so. The resulting bitstream can also be generated / decoded by a single-threaded encoder / decoder. The encoding concept of the present application, however, allows a parallel processing encoder to efficiently apply parallel processing without compromising compression efficiency. With regard to parallel processing capabilities, a similar statement applies to the decoder described below with reference to FIG. 2.

[0017] Encoder 10 is a video encoder, although in general, encoder 10 may also be an image encoder. Image 12 of video 14 is shown input to encoder 10 at input 16. Image 12 represents a particular scene, i.e., image content. However, encoder 10 also receives at its input 16 other images 15 associated with the same moment along with images 12 and 15 belonging to various layers. Simply for ease of explanation, image 15 is shown as belonging to layer 1, while image 12 is shown as belonging to layer zero. FIG. 1 indicates that layer 1 may include a higher spatial resolution, i.e., a larger number of picture samples, relative to layer zero, representing the same scene. However, this is so merely for ease of explanation; images 15 of layer 1 may have the same spatial resolution but differ, for example, in the view direction relative to layer zero, i.e., images 12 and 15 may have been captured from different viewpoints.

[0018] Encoder 10 is a hybrid-breed encoder: images 12 and 15 are predicted by predictor 18, and the prediction residual 20 obtained by residual determiner 22 is subjected to a transform, e.g., a spectral decomposition such as DCT, and quantization in transform / quantization module 24. The transformed and quantized prediction residual 26 thus obtained is subjected to entropy coding in entropy encoder 28, e.g., arithmetic coding using context adaptation or variable-length coding. A reconstructable version of the residual is available to the decoder: a dequantized and retransformed residual signal 30 is recovered by retransform / requantization module 31 and recombined with the prediction signal 32 of predictor 18 by combiner 33, thereby resulting in a reconstruction 34 of images 12 and 15, respectively. However, encoder 10 operates on a block basis. Therefore, the reconstructed signal 34 is subject to discontinuities at block boundaries, and accordingly, a filter 36 can be applied to the reconstructed signal 34 to generate reference pictures 38 for images 12 and 15, respectively, on which basis the predictor 18 predicts subsequently coded images of the various layers. As indicated by the dotted lines in Figure 1, the predictor 18 may, however, directly use the reconstructed signal 34 without the filter 36 or intermediate version in other prediction modes, such as spatial prediction mode.

[0019] The predictor 18 may select among various prediction modes to predict a particular block of the image 12. Such a block 39 of the image 12 is shown as an example in FIG. 1. A temporal prediction mode may exist in which the block 39, which is representative of any block of the image 12 into which the image 12 is partitioned, is predicted based on a previously coded image of the same layer, e.g., image 12′. A spatial prediction mode may also exist, according to which the block 39 is predicted based on a previously coded portion of the same image 12, a neighboring block 39. A block 41 of the image 15 is also shown as an example in FIG. 1 because it is representative of any of the other blocks into which the image 15 is partitioned. For the block 41, the predictor 18 may support the just-mentioned prediction modes, i.e., temporal and spatial prediction modes. In addition, the predictor 18 may provide an inter-layer prediction mode in which the block 41 is predicted based on a corresponding portion of the image 12 of a lower layer. "Corresponding" in "corresponding portion" means spatial correspondence, i.e., the portion within image 12 indicates the same part of the scene as block 41 to be predicted in image 15.

[0020] The prediction of the predictor 18 may of course not be limited to image samples: the prediction may apply to any coding parameters, i.e. prediction mode, motion vectors in temporal prediction, disparity vectors in multiview prediction, etc. Simply the residual may then be coded in the bitstream 40.

[0021] A specific syntax is used to edit the quantized residual data 26, i.e., transform coefficient levels and other residual data, and the coding parameters, including, for example, prediction modes and prediction parameters for individual blocks 39 and 41 of images 12 and 15 as defined by, for example, predictor 18 and syntax elements, are subjected to entropy coding by entropy encoder 28. The thus obtained data stream 40 as output by entropy encoder 28 forms the bitstream 40 output by encoder 10.

[0022] FIG. 2 shows a decoder that conforms to the encoder of FIG. 1, i.e., that can decode bitstream 40. The decoder of FIG. 2 is generally designated by reference numeral 50 and includes an entropy decoder, a retransform / inverse quantization module 54, a combiner 56, a filter 58, and a predictor 60. Entropy decoder 42 receives the bitstream and performs entropy decoding to recover residual data 62 and coding parameters 64. Retransform / inverse quantization module 54 inverse quantizes and retransforms residual data 62 and forwards the resulting residual signal to combiner 56. Combiner 56 also receives prediction signal 66 from predictor 60, which in turn forms prediction signal 66 using coding parameters 64 based on a reconstructed signal 68 determined at combiner 56 by combining prediction signal 66 and residual signal 65. The prediction reflects the prediction finally selected by the predictor 18, i.e. the same prediction modes are available and these modes are selected for the individual blocks of the pictures 12 and 15 and proceed according to the prediction parameters. As already explained above with reference to Figure 1, the predictor 60 may instead or in addition use a filtered version or some intermediate version of the reconstructed signal 68. The pictures of the various layers to be finally reconstructed and output at the output 70 of the decoder 50 may likewise determine the unfiltered version of the combined signal 68 or a slightly filtered version thereof.

[0023] According to the tile concept, images 12 and 15 are subdivided into tiles 80 and 82, respectively, and prediction of blocks 39 and 41 within at least these tiles 80 and 82, respectively, is restricted to using only data associated with the same tile of the same image 12, 15 as the basis for spatial prediction. This means that spatial prediction of block 39 is restricted to using previously coded portions of the same tile, but the temporal prediction mode is not restricted to relying on information from previously coded images, such as image 12′. Similarly, the spatial prediction mode of block 41 is restricted to using previously coded data only from the same tile, but the temporal and interlayer prediction modes are not restricted. The subdivision of images 15 and 12, respectively, into six tiles is selected solely for convenience of explanation. The respective subdivision into tiles may be selected and indicated within bitstream 40 for images 12′, 12′, and 15, 15′, respectively, individually. The number of tiles per image 12 and 15 may be 1, 2, 3, 4, 6, etc., respectively, and the division of the tiles may be limited to the normal division of tiles into rows and columns. For completeness, it should be noted that the method of encoding tiles separately may not be limited to intra-prediction or spatial prediction, but may include any prediction of coding parameters across tile boundaries, and may be limited to context selection in entropy coding relying only on data of the same tile. In this way, the decoder may perform the just-mentioned operations in parallel, i.e., tile by tile.

[0024] The encoder and decoder of Figures 1 and 2 may alternatively or additionally be able to use the WPP concept, see Figure 3. The WPP substream 100 also represents a spatial division of the images 12, 15 into WPP substreams. In contrast to tiles and slices, the WPP substream does not impose restrictions on prediction and context selection beyond the WPP substream 100. The WPP substream 100 extends, for example, in the row direction beyond the LCUs (Maximum Coding Units) 101, i.e., the largest possible row of blocks that the predictive coding mode can individually transmit in the bitstream, and only one compromise is made with respect to entropy coding to allow parallel processing. Specifically, an order 102 is defined among the WPP substreams 100, which is illustratively conducted from top to bottom, and for each WPP substream 100, except for the first WPP substream in the order 102, the probability estimates of the symbol alphabet, i.e., the entropy probabilities, are not completely reset but are instead taken from or set equal to the probabilities obtained after entropy encoding / decoding the immediately preceding WPP substream up to the second LCU, as indicated by line 104, and the LCU order, or decoder order of the substreams, starts for each WPP substream on the same side, such as the left side of images 12 and 15, respectively, and is conducted to the other side in the LCU row direction, as indicated by arrow 106. Thus, by following the same coding delay between the sequences of WPP substreams of the same images 12 and 15, these WPP substreams 100 can be decoded / encoded in parallel, so that the parallel, i.e. simultaneously, coded / decoded portions of each image 12 and 15 simultaneously form a kind of wavefront 108 moving across the image in a tilted manner from left to right.

[0025] It is briefly noted that the orders 102 and 104 also define a raster scan order between LCUs, proceeding row by row from the top to bottom, from the upper left to the lower right LCU of LCU 101. WPP substreams may each correspond to one LCU row. For brevity, tiles may be referred to, and the latter may be constrained to be aligned to an LCU boundary. Where a boundary between two slices within a substream is concerned, the substream may be fragmented into one or more slices without collapsing at an LCU boundary. However, when transitioning from one slice of a substream to the next, entropy probability is employed in that case. In the case of tiles, the entire tile may be aggregated into one slice, or, where a boundary between two slices within a tile is concerned, one tile may again be fragmented into one or more slices without collapsing at an LCU boundary. In the case of tiles, the order between LCUs changes to first traverse a tile in raster scan order before proceeding to the next tile in tile order.

[0026] As described above, image 12 can be divided into tiles or WPP substreams, and similarly, image 15 can also be divided into tiles or WPP substreams. Theoretically, a WPP substream division / concept can be selected for one of images 12 and 15, while a tile division / concept is selected for the other of the two. Alternatively, a restriction can be imposed on the bitstream by requiring that the concept type, i.e., tile or WPP substream, be the same between layers. Another example for spatial segments includes slices. Slices are used to divide the bitstream 40 for transmission purposes. Slices are packed into NAL units, which are the smallest entities for transmission. Each slice is independently codable / decodable. That is, any prediction across slice boundaries, as well as context selection, is prohibited. These are just three examples for spatial segments: slices, tiles, and WPP substreams. In addition, all three parallelization concepts—tiles, WPP substreams, and slices—can be used in combination, i.e., image 12 or image 15 can be divided into tiles, and each tile can be divided into multiple WPP substreams. Slices can also be used to divide the bitstream into multiple NAL units, for example (but without limitation) at tile or WPP boundaries. If images 12, 15 are divided using tiles or WPP substreams, and additionally using slices, and if the slice division deviates from other WPP / tile divisions, then a spatial segment is defined as the smallest independently decodable section of image 12, 15. Alternatively, restrictions can be imposed on the bitstream if a combination of concepts can be used within an image (12 or 15) and / or boundaries between different used concepts must be aligned.

[0027] Before discussing the above presented concepts of the present application, referring again to Figures 1 and 2, it should be noted that the block structures of the encoder and decoder in Figures 1 and 2 are merely for illustrative purposes and the structures may differ.

[0028] According to a first aspect, which may be referred to as "tile boundary alignment," a long-term syntax element structure is used to indicate that, over a predetermined period of time, e.g., spanning a series of images, the second layer images 15 are subdivided so that the boundaries 84 between the spatial segments 82 of the second layer images overlap with every boundary 86 of the spatial segments 80 of the first layer. The decoder still periodically determines the actual subdivision of the first and second layer images 12, 15 into spatial segments 80 and 82 based on the short-term syntax elements of the multi-layer video data stream 40 over a predetermined period of time, e.g., at a time interval shorter than the unit of individual images, i.e., the image pitch interval, but knowledge of the alignment is already useful in scheduling parallel processing workload allocation. For example, the solid line 84 in FIG. 1 represents an embodiment in which the tile boundaries 84 are perfectly spatially aligned with the tile boundaries 86 of layer 0. The just-mentioned guarantee, however, allows the tile division of layer 1 to be finer than that of layer 0, such that the tile division of layer 1 further includes additional tile boundaries that do not spatially overlap with any tile boundaries 86 of layer 0. In any case, knowledge of the tile registration between layer 1 and layer 0 assists the decoder in allocating available workload or processing power among spatial segments that are simultaneously processed in parallel. Without the long-term syntax element structure, the decoder would have to perform workload allocation at smaller time intervals, i.e., per image, thereby wasting computational power to perform the workload allocation. Another aspect is "opportunistic decoding": a decoder with multiple CPU cores can use knowledge of layer parallelism to decide whether or not to decode layers of greater complexity, i.e., higher spatial resolution or a larger number of layers. Bitstreams that exceed the capabilities of a single core may be decodable by utilizing all cores of the same decoder. This information is particularly useful when the profile and level indicators do not include this kind of indication of minimum parallelism.

[0029] For a clearer understanding of the just-outlined aspects of the present application, reference is made to Figure 4, which illustrates a video decoder 600 that may be implemented as may be described with respect to Figure 2. That is, the decoder 600 is configured to decode a multi-layer video data stream in which a scene is encoded in a hierarchy of layers using inter-layer prediction from a first layer 0 to a second layer 1, as already described above. The video decoder supports parallel decoding of the multi-layer video data stream in spatial segments into which the images of the layers are subdivided, the spatial segments being, for example, tiles, WPP sub-streams, etc. In other words, the video decoder may decode the multi-layer video data stream in parallel, and in doing so, the video decoder 600 operates on images 12 and 15 of layers 0 and 1 in units of spatial segments.

[0030] For example, as outlined above, the spatial segments may be tiles, and the video decoder 600 may be configured to decode images 12 and 15 of layers 0 and 1 using intra-picture spatial prediction, with the video decoder 600 suspending intra-picture spatial prediction for each tile at its tile boundary. The division of images 12 and 15 into tiles may be signaled in the data stream 40 by short-term syntax elements, e.g., individually, i.e., for each pair of images 12 and 15 belonging to a time period 604, for each time period 604 to which each image 12 and 15 pertains, on a short-term basis, e.g., in units of time intervals. As noted above, the subdivision of images 12 and 15 into tiles may be limited to rectangular, regular subdivisions only, i.e., rows and columns of tiles. The short-term syntax elements 602 thus set the number of rows and columns of tile subdivision for each image 12 and each image 15 of both layers. When decoding inbound multi-layer video data stream 40, video decoder 600 is configured to apply spatial prediction and potentially temporal prediction. Optionally, video decoder 600 entropy decodes each of the tiles separately. If probability adaptation is used during decoding of each tile, video decoder 600 initializes the entropy probability for each tile separately so that the tiles can be entropy decoded in parallel. In addition to spatial prediction and optionally temporal prediction, video decoder 600 supports inter-layer prediction as far as decoding tiles of layer 1 image 15 is concerned. As mentioned above, inter-layer prediction may relate to different parameters included in decoding layer 1: inter-layer prediction may predict the prediction residual of layer 1, for example, it may predict transform coefficients, the prediction mode used in decoding layer 1, the prediction parameters used in decoding layer 1, samples of the enhancement layer 1 image, etc.Inter-layer prediction predicts portions within a tile of image 15 of layer 1 based on already decoded portions of image 12 of layer 0 - either directly (centrally) co-located or slightly spatially deviated from the directly co-located position, e.g., via disparity vector prediction parameters controlling inter-layer prediction, in the case of layers 0 and 1 for various views of the same scene.

[0031] In response to the long-term syntax element structure of data stream 40, video decoder 600 separately processes predetermined period 608 after long-term syntax element structure 606, as indicated by reference numeral 606 in FIG. 4 . The predetermined period 608 then includes several time intervals, i.e., multiple time frames 604, in which short-term syntax elements 602 individually indicate the subdivision of the image into tiles. It should be noted that 608 may relate to a range (=period) of SPS, and a change in SPS will in any case result in a significant reinitialization. As long as long-term characteristics are mentioned therein, the just-mentioned remarks are also valid for all embodiments relating to other aspects. In particular, if long-term syntax element structure 606 assumes the original values ​​of the first set of possible values, video decoder 600 interprets this situation as a guarantee that image 15 of layer 1 will be subdivided during the predetermined period so that the boundaries between tiles of image 15 overlap each boundary of tiles of image 12 of layer 0. Although the video decoder 600 still then examines the short-term syntax elements 602 to determine the subdivision of images 12 and 15 into tiles for the time interval 602 within the predetermined period 608, the video decoder 600 may rely on the fact that the boundaries of the base layer tiles of image 12 are completely covered by the boundaries of the enhancement layer tiles of image 15, i.e., the tile subdivision of image 15 locally corresponds to or represents a spatial refinement of the subdivision of image 12 into tiles by comparing each time-aligned pair of images 12 and 15. As mentioned above, the video decoder 600 may utilize this type of signaling, i.e., the long-term syntax element structure 606, to consider the values ​​underlying the first set of possible values ​​by scheduling parallel processing of the tiles of images 12 and 15 within the predetermined period 608 in parallel, i.e., by parallel decoding of each time-aligned pair of tiles of images 12 and 15.For example, in the case of a long-term syntax element structure assuming values ​​that form the first set of possible values, video decoder 600 may know the following: for a particular image 12 of layer 0, the first tile in tile order among the tiles of image 12 locally matches a respective tile of the time-aligned enhancement layer image 15, or completely locally covers the first tile of the time-aligned enhancement layer image 15 in tile order among the tiles of enhancement layer image 15. Thus, at least in the case of inter-layer prediction without disparity / motion compensation, as soon as decoding of the first tile of the time-aligned base layer image 12 finishes, video decoder 600 may start decoding the first tile of the enhancement layer image 15, because the just-mentioned guarantee informs video decoder 600 that the co-located portion of base layer image 12 needed for inter-layer prediction is available for the entire first tile of enhancement layer image 15. The inter-layer offset, or degree of parallelism between the base layer picture 12 and the enhancement layer picture 15, may thus be recognized / determined by the video decoder 600 to be equal to one tile of the base layer picture 12. The offset may be increased slightly in the case of inter-layer prediction involving disparity vectors with a non-zero vertical component and / or disparity vectors with a horizontal component that shifts the corresponding portion within the base layer picture towards the right, where the tile order within the tile may proceed in row-wise raster scan order from the top-left corner of the picture 12, 15 towards its bottom-right corner.

[0032] However, if the long-term syntax element structure assumes a second set of possible values, which is different from the first set of possible values, then video decoder 600 does not make any guarantees, but potentially plans and schedules short-term syntax element 602, parallel decoding of tiles of pictures 12 and 15, at least some parallel decoding of time-aligned pairs of pictures 12 and 15, tiles of the base and enhancement layers, on a short-term basis. In that case, video decoder 600 determines the minimum inter-layer offset or inter-layer spatial processing offset for parallel decoding between layers 0 and 1 on a short-term basis, which is, however, a tedious process. For at least a subset of the sets of possible values ​​of the short-term syntax element, there are boundaries between spatial segments of the image of the second layer that do not overlap any boundaries of spatial segments of the first layer. However, there is a further subset of the sets of possible values ​​for the short-term syntax element at boundaries between spatial segments of the image of the second layer that overlap each boundary of spatial segments of the first layer. The latter subset is only used in the case of long-term syntax elements that indicate tile boundary alignment between the base and enhancement layers.

[0033] Additionally or alternatively, video decoder 600 may use or take advantage of the fact that the long-term syntax element structure considers the value as the source of a first set of possible values ​​to perform the test, i.e., refrain from performing this test in the case of long-term syntax element structure 606 considering the value as the source of a second set of possible values, and attempt to perform Layer 1 decoding anyway. In that case, particularly for battery-powered devices, valuable computational power is saved in cases where success or success in decoding enhancement Layer 1 in time, i.e., in real time, is theoretical. It is worth mentioning that the suppression may be selected based on a level indicator described below with respect to the fourth aspect.

[0034] Although FIG. 4 has been described above using tiles as exemplary spatial segments, it is clear that video decoder 600 may utilize the long-term syntax element structure and guarantees implied thereby in connection with other spatial segments, such as substreams or slices. In the former case, video decoder 600 decodes the spatial segments of the images of the first layer 12 in parallel and supports intra-picture spatial prediction across spatial segment boundaries of the images of the first layer, and decodes images 12 and 15 of the layer using intra-picture spatial prediction, subject to a decoding delay during the decoding of these spatial segments, i.e., substreams. As noted above, the substreams may correspond to the horizontal stripes of each image, i.e., vertically subdivide each image. In decoding each substream, video decoder 600 may use a decoding order that generally proceeds from left to right, and the decoding order defined within the image substreams may proceed from top to bottom. In this way, using a typical spatial prediction concept in which spatial prediction is performed from the already decoded portion adjacent to the top edge of the current image and the already decoded portion to the left, with a specific decoding delay between immediately succeeding substreams, allows for parallel decoding of substreams. For example, the decoding delay may be measured in units of LCUs. This may be done for image 12 of layer 0 as well as image 15 of layer 1. Thus, parallelism in decoding a video data stream may include both parallelism within images 12 and 15 individually, but also parallel decoding of substreams belonging to images 12 and 15 of different layers in one time frame 604. As for any entropy decoding of substreams, this may include adaptation of entropy probabilities during the decoding of each substream. The first substream in the substream order of each image 12 or 15 may be subject to individual initialization of entropy probabilities independent of the other substreams.Any subsequent substream may be subjected to entropy probability initialization depending on the adapted entropy probability while decoding the previous substream up to a certain distance from the left of each preceding substream, for example after decoding two LCUs of the previous substream, in the middle of the previous substream in the decoding order of the same image by adopting the adapted entropy probability.

[0035] Even in the case of wavefront parallel processing substreams, the video decoder 600 may utilize the long-term syntax element structure 606: if a guarantee is signaled by this syntax element structure 606, the video decoder 600 can rely on the fact that all boundaries between consecutive / adjacent substreams of the base layer picture 12 within a predetermined time period 608 are overlaid by the respective boundaries between adjacent / adjacent substreams of the time-aligned enhancement layer picture 15. That is, the base layer substreams locally coincide with the respective enhancement layer substreams of the time-aligned enhancement layer picture 15, or the same exactly correspond to two or more substreams of the time-aligned enhancement layer picture. Thus, if the guarantee applies, the decoder 600 knows that decoding of the first substream of the time-aligned enhancement layer picture 15 can begin as soon as decoding of the first substream of the base layer picture 12 finishes.

[0036] As noted above, unlike tile subdivision, the short-term syntax elements 602 may be selected to define the location of the substreams in pictures 12 and 15 relative to the subdivision of these pictures into several coding blocks, such as LCUs. A substream may therefore be a collection of one or more rows of such coding blocks. As in the case of tile subdivision, a time interval 604 may exist such that the short-term syntax elements 602 indicate the subdivision of pictures 12 and 15 into substreams on a per picture basis, i.e., per image frame 604 basis. If no guarantee is provided by the long-term syntax element structure 606, the video decoder 600 may nevertheless attempt to decode substreams of different layers of a common time frame in parallel; however, to do so, the video decoder 600 needs to examine the short-term syntax elements 602.

[0037] As is the case with using tiles as spatial segments, video decoder 600 may make speculative attempts to decode enhancement layer 1 that depend on the values ​​assumed by long-term syntax element structure 606.

[0038] As far as the video encoder side is concerned, it should be noted that each video encoder, which may be realized as shown in FIG. 1, may decide whether to insert and set a long-term syntax element structure 606 into data stream 40 to provide a guarantee to decoder 600. If allowed, the encoder restricts the possibilities for setting short-term syntax elements 602 for a corresponding predetermined period 608 to a setting that complies with the boundary alignment guarantee. Otherwise, the encoder is free to set short-term syntax elements 602 as it likes during period 608. When using tiles as spatial segments, the encoder follows the constraints that spatial prediction does not cross tile boundaries and that any entropy coding of tiles of images 12 and 15 is performed in a self-contained manner for each tile. For example, entropy probabilities are newly initialized for each tile independent of other tiles. In the case of substreams, the entropy probability initialization of the substreams is performed de novo, i.e. independently of any other substreams, for any first substream of each image 12, 15, by taking the entropy probabilities to be adapted up to the midpoint of the immediately preceding substream, as far as any second and subsequent substreams are concerned. Spatial prediction is performed without any restrictions regarding crossing substream boundaries.

[0039] The constellation concept may be introduced into currently envisioned extensions of the HEVC standard in the manner described with respect to Figure 4, i.e., as described below. To that extent, the explanation given immediately below should also be interpreted as a basis for possible implementation details regarding the explanation given above with respect to Figure 4.

[0040] HEVC allows for dividing the CTB of a coded base layer picture by a grid of vertical and horizontal boundaries into rectangular regions called tiles that can be processed independently except for in-loop filtering, which can be turned off at the tile boundaries to make them completely independent.

[0041] To reduce tile boundary artifacts, in-loop filters, when properly configured, may cross tile boundaries, while analytical and predictive dependencies are broken at tile boundaries that closely resemble image boundaries. Thus, processing of individual tiles is entirely, or very largely, dependent on the filter configuration and independent of other tiles within the image. A restriction is introduced: all CTBs of a tile must belong to the same slice, or all CTBs of a slice must belong to the same tile. As seen in Figure 1, tiling forces the CTB scanning order to be considered tile order, i.e., passing through all CTBs belonging to the first tile (e.g., the upper-left tile) before continuing with the CTB belonging to the second tile (e.g., the upper-right tile). The tile structure is defined through the number and size of CTBs in each tile's rows and columns, which form a grid within the image. This structure can vary on a frame-by-frame basis or can be constant throughout the entire coded video sequence.

[0042] Figure 5 shows a typical division of a CTB into nine tiles within an image. The thick black lines represent tile boundaries, and the numbering represents the scan order of the CTB, which in turn represents the tile order.

[0043] An enhancement layer tile of an HEVC extension can be decoded as soon as all tiles covering the corresponding image area in the base layer bitstream have been decoded.

[0044] The following sections describe improvements to the constraints, signaling and decoding process that allow easier access to base layer information using the concepts of FIG.

[0045] The simplest case for tile-level parallelization is when the tile boundaries of the base and enhancement layers are aligned. For SNR scalability, this means that the boundaries are at exactly the same position. For spatial scalability, this means that for every two enhancement layer pixels that belong to the same tile, the corresponding base layer pixel also belongs to the same tile, and for every two base layer pixels that belong to the same tile, the corresponding enhancement layer pixel also belongs to the same tile.

[0046] HEVC features short-term signaling, corresponding to 602 in Figure 4, to indicate tile size and structure within an image on a picture-by-picture basis by using the picture parameter set given below by column_width_minus1[i] and row_height_minus1[i] from [1]. Figure 6 shows an exemplary syntax.

[0047] For example, HEVC features signal restrictions that ensure a specific configuration of an HEVC coded video sequence, for example, to indicate a fixed tile structure in a single-layer HEVC coded video sequence (compare tiles_fixed_structure_flag in the VUI syntax provided below). Further restriction of tiles in a scalable coded video sequence is beneficial for decoder initialization and operation. It is not necessary to perform full alignment to allow the decoder to begin decoding the enhancement layer image area associated with a base layer tile after that base layer tile is finished. In particular, in spatial scalability, it may be useful to allow more tiles in the enhancement layer than in the base layer. For example, in a factor of two spatial scalability, four times as many pixels are included in the enhancement layer image area compared to the corresponding base layer image area. Thus, it may be useful to allow four tiles in the enhancement layer per base layer tile. See FIG. 7, which shows an example for aligned tile boundaries with spatial scalability. All vertical boundaries are aligned in the base and enhancement layers. An additional tile (horizontal boundary) in the enhancement layer is used to allow parallelization by using the same number of pixels per enhancement layer tile as in the base layer tile division.

[0048] In this way, we define the tile border array in such a way that only each base layer boundary has a corresponding boundary in the enhancement layer, but not vice versa. Precisely, this means that for every two enhancement layer pixels that belong to the same tile, the corresponding base layer pixel also belongs to the same tile.

[0049] Signaling 606 helps initialize a parallel decoder environment that would otherwise have to gather information by analyzing multiple parameter sets. Additionally, in the form of bitstream constraints, the concepts of Figure 4, for example, ensure that the constraints described above apply to the complete coded video sequence.

[0050] If the tile boundaries of the base layer are a subset of the tile boundaries of the enhancement layer, a possible embodiment allows for pared-down signaling of the base layer tile boundaries in the enhancement layer.

[0051] Information about tile ordering can be explicitly indicated in the bitstream for easy access by the decoder.

[0052] As given in FIG. 8, in a particular embodiment, the signaling may be achieved using a flag in the VUI parameters of the enhancement layer SPS:

[0053] tiles_fixed_structure_flag equal to 1 indicates that each picture parameter set activated in the coded video sequence, when present, has the same values ​​of the syntax elements num_tile_columns_minus1, num_tile_rows_minus1, uniform_spacing_flag, column_width_minus1[i], row_height_minus1[i], and loop_filter_across_tiles_enabled_flag, and tiles_fixed_structure_flag equal to 0 indicates that the tiles syntax elements in different picture parameter sets may or may not have the same values. When the tiles_fixed_structure_flag syntax element is not present, it is inferred to be equal to 0.

[0054] Note that signaling tiles_fixed_structure_flag equal to 1 is a guarantee to the decoder that each picture in the coded video sequence has the same number of tiles distributed in the same way, which may be useful for workload distribution in case of multi-threaded decoding.

[0055] tile_boundaries_aligned_flag corresponds to structure 606 in Figure 4. Ditto equal to 1 indicates that all corresponding tile boundaries in the base layer image have corresponding tile boundaries in the given enhancement layer. tile_boundaries_aligned_flag equal to 0 indicates that no restrictions exist regarding the tile configuration between the corresponding base layer and the given enhancement layer.

[0056] Note that for n, depending on the value of the long-term syntax element structure, the long-term syntax element structure may guarantee that, during a given period, e.g., an image sequence, the minimum number of spatial segments 82 into which the second layer image 15 is subdivided exceeds the minimum number of spatial segments 80 into which the first layer image 12 is subdivided n times, or that each spatial segment of an image 12 consists of exactly n spatial segments of time-aligned images 15, where n depends on the long-term syntax element structure. In the case of Figure 7, n would be equal to 3. A decoder may still periodically determine the actual subdivision of the first and second layer images 12, 15 into spatial segments 80 and 82 based on the short-term syntax elements of the multi-layer video data stream 40 at time intervals shorter than the given period, but the guarantee may also be utilized by the decoder to perform workload distribution more efficiently. Another aspect is "opportunistic decoding": a decoder with multiple CPU cores may use the guarantees as hints regarding layer parallelism and on top of that to decide to decode layers of greater complexity, i.e., in higher spatial resolution or a larger number of layers. Bitstreams that exceed the capabilities of a single core may be decodable by utilizing all cores of the same decoder. This information is particularly useful when the profile and level indicators do not include this kind of indication regarding minimum parallelism.

[0057] The second aspect described and presented below relates to a concept that may be called "constrained inter-layer upsampling": a syntax element in the bitstream (compare independent_tile_upsampling_idc as an example) is used to operate the upsampling filter 36 in the case of spatially scalable multi-layer video. When upsampling filtering is performed in layer 0 across a spatial segment boundary 86, the delay to be accommodated in the parallel encoding / decoding of spatial segment 82 of layer 1 relative to the encoding / decoding of spatial segment 80 of layer 0 increases as the upsampling filtering combines and thus represents interdependent information of adjacent spatial segments of layer 0, serving as prediction references 38 used in the inter-layer prediction of block 41 of layer 1. See, for example, FIG. 9. Images 12 and 15 are shown in an overlapping manner, with both images sized and registered to each other according to spatial correspondence, i.e., portions showing the same parts of the scene overlap each other. Images 12 and 15 are shown as an example to be divided into 6 and 12 spatial segments, such as tiles, respectively. Filter kernel 200 is illustratively shown as interlayer predicting any block within a tile of image 15 and moving across the top-left tile of image 12 to obtain an upsampled version of it that serves as a basis for spatially overlaying the top-left tile. In some intermediate instances, such as 202, kernel 200 overlaps with adjacent tiles of image 12. The central sample value of kernel 200 at position 202 of the upsampled version depends on both the sample in the top-left tile of image 12 and the sample to the right of the tile of image 12. When an upsampled version of image 12 serves as the basis for interlayer prediction, the interlayer offset increases when processing segments of layers in parallel. This restriction helps increase the amount of parallelism across different layers and thus reduce the overall coding delay. Naturally, syntax elements can also be long-term syntax elements that apply to a series of images.The constraint can be achieved in one of the following ways: for example, filling the overlapping portion of kernel 200 at overlapping location 202 with the central tendency of the sample values ​​within the non-dashed portion of kernel 200, extrapolating the non-dashed portion using a linear or other function to the dashed one, etc.

[0058] To illustrate the latter aspect even more clearly, reference is made to FIG. 10 , which illustrates a decoder 610 receiving a spatial scalable bitstream 40 in which images are coded in different spatial layers, corresponding to image 12 of layer 0 and image 15 of layer 1. For at least one of these spatial layers, the decoder 610 is configured to decode the same in spatial segments. Consistent with the above description, these spatial segments may be tiles, substreams, or slices. Similarly, the decoder 610 may be configured to perform parallel decoding of such spatial segments of image 12 or image 15. That is, the base layer image 12 may be subdivided into spatial segments such as tiles and / or substreams and / or slices, and / or the enhancement layer image 15 may be subdivided into tiles and / or substreams and / or slices. For details regarding parallel decoding, reference is made to the above description of FIG. 4, which can be readily applied to the decoder 610 of FIG. 10. That is, when decoding the base layer image 12, if the base layers 12 and 15 are, for example, part of layered video, the decoder 610 uses spatial prediction and optionally temporal prediction. In the case of tiles, spatial prediction is restricted to not crossing tile boundaries, and the same applies to entropy decoding. And, if entropy decoding is used, it is performed completely separately for tiles. In addition, while inter-layer prediction is supported, the same applies to the enhancement layer image 15. As already mentioned above, inter-layer prediction not only relates to prediction parameters of the enhancement layer being predicted based on corresponding prediction parameters that were used when decoding the base layer, but also to predictions derived from reconstructed samples of the base layer image in portions co-located with respect to the portion of the enhancement layer image 15 currently predicted using inter-layer prediction.However, since bitstream 40 may be a spatially scalable bitstream, any co-located portion of base layer picture 12 intended to form the basis for inter-layer prediction of a currently processed portion of enhancement layer picture 15 may be upsampled by decoder 610 to account for the higher spatial resolution of picture 15 relative to picture 12. See, for example, FIG. 11. The currently predicted portion of enhancement layer picture 15 is indicated in FIG. 11 using reference numeral 612. The co-located portion of base layer picture 12 is indicated using reference numeral 614. Due to the higher spatial resolution of enhancement layer picture 15, the number of sample locations within portion 612 (indicated by a dot) is shown to be greater than the number of samples within portion 614 (also indicated by a dot). Thus, decoder 610 uses interpolation to upsample the already reconstructed version of portion 614 of base layer picture 12. In doing so, decoder 610 of FIG. 10 responds to syntax element 616 in spatially scalable bitstream 40.

[0059] In particular, the aforementioned responsiveness will be explained in more detail with reference to FIG. 12. FIG. 12 shows a portion 614 in the base layer picture 12, i.e., a reference portion in its non-upsampled version, and its corresponding upsampled version, indicated with reference numeral 618. As just mentioned, the version 618, which is subsequently used for inter-layer prediction by, for example, copying the respective upsampled sample values ​​of the portion 618 to the portion 612 of the enhancement layer 15, is obtained from the base layer picture 12 by interpolation 620. However, the interpolation 620 depends on the just-mentioned syntax element 616. The manner in which the interpolation 620 varies depending on the syntax element 616 relates to the division of the base layer picture 12 and the area along the boundary between adjacent portions of its upsampled version, respectively. In particular, the division depends on the above-mentioned spatial segments into which at least one of the pictures 12 and 15 is subdivided. The division is indicated in FIG. 11 within the base layer picture 12 using a dotted line 622. The partition 622 may correspond, for example, to a conjunction or disjunction combination of the spatial overlay of the boundaries of the spatial segments of the pictures 12 and 15, as outlined in more detail below, or may spatially coincide with the partition defined by the spatial segments of the enhancement layer picture 15. In either case, depending on the syntax element 616, the decoder 610 performs the interpolation 620 without regard to or taking into account the partition 622. When taking the partition into account, the decoder 610 performs the interpolation 620 such that all samples in the upsampled part 618 are derived from, dependent on, or affected by only samples of one of the parts of the partition 622, while being independent of other parts of the partition 622. For example, if the partition 622 is a local conjunction or disjunction combination of the boundaries of the spatial segments of the pictures 12 and 15, this means that all samples in the interpolated part 618 originate solely from one partition of the base layer picture 12.However, if syntax element 616 instructs decoder 610 to be insensitive to division 622 , it may occur that various samples within interpolated portion 618 arise from adjacent divisions of division 622 .

[0060] 612 illustrates the case where, for example, filter kernel 200 is used to perform interpolation 620, and kernel 624 covers the boundary between two adjacent divisions of division 622 to obtain the interpolated samples circled in FIG. 12 . In this case, decoder 610 responds to syntax element 616, illustrated using hashing in FIG. 12 , by filling filter kernel 624 normally, i.e., by applying filter kernel 624 entirely to samples contained in each of base layer images 12, or by filling each of the fragments of filter kernel 624 that extend into adjacent divisions that do not include portions 614 and 618, using an alternative rule according to which each fragment is filled independently of the underlying sample of base layer image 12. For example, the fragmented portion of filter kernel 624 is filled by some average reference or some extrapolation of the sample values ​​of the non-fragmented portion of filter kernel 624. In another variation, sample values ​​of base layer image 12 covered by the fragmented portion are filled using a predetermined value, such as 0. Typically, decoder 610 may treat the portion containing portion 614 and the division boundary separating its adjacent divisions as the outer edge of image 12 itself, and may, for example, use the same substitution rules in interpolation 620 as are used in performing upsampling / interpolation near or outside image 12.

[0061] According to one embodiment of the present application, the partitions 622 are selected to match the subdivision of the base layer image into spatial segments regardless of any subdivision into spatial segments of the enhancement layer image 15. Then, the inter-layer offset between decoding the base layer image 12 and the enhancement layer image 15 by the decoder 610 can be reduced because a portion, e.g., portion 614 of the base layer image 12, does not need the decoder 610 to have an adjacent partition / spatial segment to be decoded before performing inter-layer prediction of portion 612 of the enhancement layer image 15.

[0062] Alternatively, the decoder 610 may be configured to determine the partitions 622 to locally match the subdivision of the image 15 into spatial segments. In another alternative, the decoder 610 may be configured to select the partitions 622 to be spatially matched, i.e., to consist solely of the boundaries of the spatial segments of the images 12 and 15, to correspond to the intersection of the boundaries of the images 12 and 15 - in other words, the boundaries of the image 15 subdivided into spatial segments form corresponding boundaries between the divisions of the partitions 622 that correspond to the respective boundaries of the base layer image 12 spatially subdivided into spatial segments.

[0063] That is, the syntax element 616 can also allow the decoder 610 to distinguish various ways of selecting the division 622 as well as whether to ignore the division 622 in the interpolation 620, as outlined in more detail below. See, for example, FIG. 9 , where tiles are used as an example of spatial segments. If the syntax element 616 indicates to the decoder 610 to perform the interpolation 620 separately for the division 622, the decoder 610 can, for example, use the boundaries of the base layer image 12 as the boundaries of the division 622, since they correspond to the finer subdivision of the enhancement layer image 15 into tiles. Thus, to start decoding the second tile of the top tile row of the enhancement layer image 15, the decoder 610 does not need to wait for the completion of decoding the second tile in the top row of the base layer image 12, because the "interpolation separation" prohibits any mixing of the reconstructed samples of the first two tiles in the top row of the base layer image 12. 9 to determine the division 622, and the decoder 610 may start decoding the top-left tile of the enhancement layer image 15 even earlier, i.e., as soon as it has completed decoding the co-located sub-portion of each of the first tiles of the base layer image 12. In this regard, it should be noted that even when decoding tiles, the decoder 610 may use some decoding order, which may include, for example, some raster scan order, proceeding from the top-left corner of each tile to the bottom-right corner of each tile in a row-wise manner.

[0064] That is, according to the second aspect, the encoder forming bitstream 40 may select between two modes by means of syntax element 616: when syntax element 616 is set—and inserted into bitstream 40—to configure decoder 610 to be insensitive to partition 622, better inter-layer prediction may be achieved due to better interpolation, but the obtainable degree of parallelism in parallel decoded pictures 12 and 15 is reduced, i.e., the minimum inter-layer offset to follow is increased. In the other mode, syntax element 616 causes decoder 610 to take partition 622 into account when performing interpolation 620 for inter-layer prediction purposes, and inter-layer prediction quality therefore decreases due to the increased degree of parallelism and the reduced minimum inter-layer decoding offset in parallel decoded pictures 12 and 15, respectively.

[0065] Although the description of the second aspect of the present application has focused primarily on the tile subdivision or tile parallel processing concepts, it should be clear that using syntax element 616 to control interpolation 620 is advantageous when using WPP substreams. See, for example, FIG. 13 , which illustrates a case in which a base layer image 12 is illustratively subdivided into two substreams, where co-located portions of an enhancement layer image 15 are each subdivided into two substreams. When applying interpolation separation in response to syntax element 616, the decoder 610 may begin decoding the first, i.e., top, substream of the enhancement layer image 15 as soon as it has decoded the first substream of the base layer image 12 that is far enough away to cover the respective left portion of the first enhancement layer substream of image 15. And this is true even for those portions of the first enhancement layer substream of picture 15 that border the second substream of picture 15, since the interpolation separation represents any inter-layer prediction independent of any reconstructed portions of the base layer substream of picture 12 that spatially overlay the second enhancement layer substream.

[0066] Before providing below detailed examples implementing the switchable constrained inter-layer upsampling outlined above, it should be noted that, for example, syntax element 616 may indicate or switch constraints for a time frame in the case of pictures 12 and 15, which are a time-aligned pair of pictures of a video, just as is the case in FIG. 4. Furthermore, it should also be noted that a decoder according to an embodiment of the present application may be consistent with the descriptions and functionality provided above with respect to both FIG. 4 and FIG. 10. In this regard, it should be noted that the descriptions provided above with respect to FIG. 4 regarding the signaling of the short-term syntax elements and the location and arrangement of the spatial segments of pictures 12 and 15 are considered to apply equally to the embodiments described with respect to FIG. 10 through 13. Finally, it should be noted that if the decoder of FIG. 10 is an image decoder decoding images of layers 0 and 1, the second aspect would also be advantageous. The time element is optional.

[0067] Achieving constrained inter-layer upsampling in HEVC can be done as follows: In spatial scalability, an upsampled base layer picture is used to predict an enhancement layer picture. In this method, a prediction value is calculated for each pixel location in the enhancement layer using multiple pixel values ​​of the corresponding base layer picture area (typically in the horizontal and vertical directions). If pixels from a different base layer tile are used, the enhancement layer tile cannot be decoded solely from the base layer tile information covering the same picture area as the enhancement layer tile. Signaling constrained inter-layer upsampling as a bitstream restriction ensures to the decoder that the spatial partitioning indicated in all parameter sets of the coded video sequence obeys the restriction, thus simplifying the initialization and operation of parallel multi-layer decoders.

[0068] The concept of Figure 10 can be implemented as a mechanism that disallows the use of neighboring pixel information for upsampling that is not included in the base layer tile that corresponds to the enhancement layer tile. Whether the decoder is allowed to use pixels outside the corresponding image area of ​​the base layer at all enhancement layer tile boundaries is indicated in the bitstream.

[0069] Alternatively, it is indicated in the bitstream whether the decoder is allowed to use pixels outside the corresponding image area of ​​the base layer at all enhancement layer tile boundaries only for enhancement layer tile boundaries that correspond to the base layer tile boundary.

[0070] In a particular embodiment, upsampling of the base layer is performed at tile boundaries because neighboring pixels lie on image boundaries where they are unavailable.

[0071] In a particular embodiment, the signaling may be achieved using a flag in the picture parameter set of the enhancement layer, as given in FIG.

[0072] independent_tile_upsampling_idc corresponds to syntax element 612. independent_tile_upsampling_idc not equal to 0 restricts the upsampling filter to not cross tile boundaries. If independent_tile_upsampling_idc equals 2, base layer samples located outside the image area corresponding to the enhancement layer tile should not be used for upsampling. If independent_tile_upsampling_idc equals 1, the restriction applies only to enhancement layer tile boundaries that are aligned to base layer tile boundaries. independent_tile_upsampling_idc equal to 0 does not imply this restriction.

[0073] Finally, Figure 15a shows, by way of example, two tiled images 12 and 15 that are covered in a spatially corresponding manner to illustrate an example of syntax element 612 of Figure 14: independent_tile_upsampling_idc equal to 2 restricts the upsampling filter to not cross any enhancement layer tile boundaries. See dashed simple dotted line 400. If independent_tile_upsampling_idc equal to 1, the restriction applies only to enhancement layer tile boundaries that are aligned with base layer tile boundaries. See dash-dotted line 402. independent_tile_upsampling_idc equal to 0 does not imply this restriction.

[0074] Alternatively, in mode independent_tile_upsampling_idc=2, or further modes like independent_tile_upsampling_idc=3, the upsampling filter is constrained to not cross any tile boundaries, neither those of the base layer nor those of the enhancement layer, see line 404 in Figure 15b.

[0075] That is, depending on the mode, at boundary 400, 402 or 404, the upsampling filter will be treated as described above with respect to FIG.

[0076] Before starting with the next aspect of the present application, it will be briefly described that the above-mentioned interpolation 620 may be performed, for example, in the predictor 60 of FIG. 2 to obtain an inter-layer prediction result. Since the encoder performs the same prediction on the encoding side, the interpolation 620 may also be performed on the encoder side, for example, in the predictor 18, depending on the setting of the syntax element 616. On the encoding side, the decision on how to set the syntax element may be made depending on, for example, the application scenario. For example, if low delay is of higher importance, the syntax element 612 may be set to limit inter-layer upsampling, and in other application scenarios, having good prediction and increasing the compression ratio may be more important, so it may be desirable to set the syntax element 612 to not limit inter-layer upsampling.

[0077] The just-mentioned minimum encoding delay—or offset—between the encoding of spatial segments of successive layers is also the subject of the next aspect, which may be termed a “layer decoding delay indication.” A decoder may determine this type of minimum decoding delay or offset between the encoding of spatial segments of image 15 associated with spatial segments of image 12 based on short-term syntax elements. However, according to the next concept, a long-term syntax element structure is used to indicate this inter-layer delay or offset at a predetermined time in advance. This also assists the decoder in performing workload distribution within the scope of parallel decoding of bitstream 40. So that a measurement for “delay” or “offset” spatial segments can be used, i.e., the offset may be expressed in units of spatial segments (tiles, slices, or CTB rows for WPP).

[0078] To describe the latter embodiment in more detail, reference is made to Figure 16, which largely corresponds to Figure 4. Accordingly, where possible, the same reference numerals are used in Figure 16, and with respect to elements related by these common elements, the explanations given above with respect to Figure 4 apply equally to Figure 16. It should also be noted that the video decoder 640 shown in Figure 16 may incorporate the functionality described in Figure 4 with respect to long-term syntax elements 606, in addition to the functionality described below. Currently, the present embodiment of Figure 16 also uses a long-term syntax element structure, i.e., 642, which is also inserted into the bitstream 40 to refer to or relate to the predetermined period 608. 4 in the same manner as syntax element structure 642, only the latter functionality, outlined further below, is important for decoder 640 of FIG. 16, while the functionality regarding syntax element structure 606 and its presence in bitstream 40 is optional for video decoder 640. However, the explanations set forth above with respect to video decoder 600 apply equally to video decoder 640. That is, video decoder 640 may decode a multi-layer video data stream 40 in which a scene is coded in a hierarchy of layers using inter-layer prediction, from a first layer, layer 0, to a second layer, layer 1. Video decoder 40 supports parallel decoding of a multi-layer video data stream in spatial segments, in which images of a layer are divided by sequentially traversing the spatial segments in a temporally overlapping manner with an inter-layer offset between the traversal of the spatial segments of an image of a first layer relative to the traversal of the spatial segments of an image of a second layer. As a minor note, it is stated that a spatial segment can be a tile, a substream, or a slice, but even a mixture of the just mentioned segment units is feasible. In combining the slice concept with the tile and / or substream concepts, the definition of what a "spatial segment" actually is can be different.

[0079] In any case, with respect to pictures 12 and 15 of a common time frame 604, the video decoder 640 of Figure 16 may decode spatial segments of picture 12 on the one hand and spatial segments of picture 15 on the other hand in parallel, i.e., in a temporally overlapping manner. Naturally, to achieve this goal, the video decoder 640 needs to obey some minimum decoding offset as with inter-layer prediction between both layers, and the currently decoded part of enhancement layer 1 in the scope of picture 15 must belong to an already decoded part of picture 12 that is temporally aligned with layer 0.

[0080] In the case of FIG. 16, the long-term syntax element structure 642 is used by the video decoder 640 to determine the inter-layer offsets in advance of the predetermined period 608 .

[0081] 16, the inter-layer offset is a scalar measure for the "distance" of the first spatial segment of image 12 on the one hand and the temporally aligned image 15 on the other hand. The "distance" is preferably measured spatially. Furthermore, to be meaningful, the inter-layer offset, which may be determined based on the long-term syntax element structure 642, applies to the entire decoding process of the first spatial segment of image 12. That is, if the first "inter-layer offset" spatial segment of base layer image 12 has been previously decoded, all necessary reference portions in image 12 for inter-layer prediction may be used to decode the entire first spatial segment of image 15.

[0082] As mentioned above, the "currently decoded portion" within the image 15 traverses the image 15 in a certain manner, i.e., in the tile order described above when using tile parallelism, and in the form of a tilted wavefront when using the WPP concept using sub-streams. The same applies to the spatial segments of the base layer image 12. The interlayer offset determines the portion of the traverse of the image 12 that has already been processed before the first spatial segment of the image 15 can be the first to be decoded.

[0083] To describe this in more detail, reference is made to Figures 17a and 17b. Figure 17a describes the inter-layer offset determinable from the long-term syntax element structure 642 in more detail with respect to the tile concept, while Figure 17b describes the inter-layer offset determined based on the long-term syntax element structure 642 in more detail with respect to WPP. Later, in connection with Figure 17c, it will be shown that the inter-layer offset signaling concept using the long-term syntax element structure 642 is not limited to the use of the tile and / or WPP concepts. Rather, it represents the interpretation of the inter-layer offset based on the long-term syntax element 642, which is possible by a simple subdivision of the image into slices that are decodable in a self-contained manner by definition (i.e., entropy decoding and spatial intra-picture prediction are performed entirely "for a slice" or independently from adjacent slices).

[0084] FIG. 17a illustrates two time-aligned images 12 and 15, both of which are subdivided into tiles. As is clear from the explanation of the tile concept presented above, there is typically no fixed order within the tiles of image 12 or 15 to be decoded. Rather, the tiles can be decoded in any order. Nevertheless, in connection with the example of FIG. 16, the tile order is defined at least with respect to the tiles of base layer image 12. The tile order 644 is defined to proceed in the raster scan order row direction from the top-left tile to the bottom-right tile of the regular array of tiles. According to the example of FIG. 17a, the inter-layer offset indicated by the long-term syntax element structure 642 indicates the number of tiles that must already be decoded in the tile order 644 of base layer image 12 in order for decoder 640 to begin decoding the first tile of enhancement layer image 15. To determine the "first tile" among the tiles of the enhancement layer image 15, the first tile of the enhancement layer image 15 may be fixedly defined as the top-left tile of the enhancement layer image 15. Starting from this first tile of the enhancement layer image 15, the video decoder 640 may adapt the tile order for traversing the enhancement layer tiles of image 15 depending on the tile subdivision of image 12. In the case of FIG. 17a, for example, the subdivision of image 12 into tiles consists of two rows and three columns of tiles, but image 15 is subdivided into four rows of tiles with two columns of tiles. In that case, it may be advantageous for the decoder 640 to select a tile order for traversing the enhancement layer tiles to first traverse the tiles to the left of the first two rows, then traverse the ones to the right of the first two rows, and then repeat this traversal for the lower row of tiles of the enhancement layer image 15, as indicated by arrow 646. According to another embodiment that applies to all aspects described herein, however, the tile decoding order among the enhancement layer tiles of image 15 is fixed and independent of the subdivision of the base layer image into tiles.In the case of simply using the signaled inter-layer coding offset as a trigger to begin / start decoding the enhancement layer picture, no reordering is necessary anyway. Using dotted lines, Figure 17a shows the position of picture 12 that locally corresponds to the position of the first tile of enhancement layer picture 15. As becomes clear from Figure 17a, in the illustrative case of Figure 17a, the inter-layer offset determined by long-term syntax element structure 642 is "2" because the first two tiles of picture 12 had to be decoded before decoder 640 could start decoding the first tile of enhancement layer picture 15. In just that case, the co-located portion needed for inter-layer prediction is available in base layer picture 12.

[0085] That is, in the case of FIG. 17a, the video decoder 640 determines from the long-term syntax element structure 642 that the inter-layer offset between the traversal 644 of the base layer tiles of image 12 in relation to decoding the first tile of the enhancement layer tiles of image 15 is two base layer tiles: decoding of the first two base layer tiles in tile order 644 must wait before the video decoder 640 can begin decoding the first tile of the enhancement layer tiles of image 15 in tile order 646.

[0086] Figure 17b relates to the typical case in which both time-aligned pictures 12 and 15 are subdivided into substreams, i.e., two substreams in the case of Figure 12 and four substreams in the case of Figure 15. For example, the substreams may correspond to the above-mentioned regular subdivision of pictures 12 and 15 in the rows and columns of coding blocks, i.e., each substream corresponds to one row of such coding blocks. In any case, for WPP processing, as described above, there is a decoding order defined among the substreams of picture 12, and the substreams of picture 15 proceed from top to bottom in decoding orders 648 and 650, respectively. As in Figure 17a, decoder 640 is configured to determine from long-term syntax element structure 642 the number of proceeding substreams that should already be decoded before starting to decode the first substream of picture 15. In this case, the long-term syntax element structure 642 indicates the inter-layer offset to be 1, since fully decoding the first sub-stream of the base layer picture 12 is sufficient to provide the necessary basis for any inter-layer prediction of the first sub-stream of the enhancement layer picture 15.

[0087] Figure 17c shows a pair of time-aligned pictures 12 and 15, both subdivided into slices. A slice order or decoding order is also defined between the slices of picture 12 and between the slices of picture 15, respectively, and both orders 652 and 654 proceed from top to bottom. In the exemplary case of Figure 17c, the boundaries between slices within one picture 12 and the other picture 15 locally correspond to each other. Thus, depending on the "blurring" introduced by inter-layer prediction from base layer picture 12 to enhancement layer picture 15, long-term syntax element structure 642 indicates an inter-layer offset that should be equal to 1 or 2. In particular, due to stretching of the corresponding co-located reference portion of image 12 relative to the inter-layer predicted portion of image 15, e.g., due to disparity compensation vectors or upsampling interpolation filter kernels as described above with respect to FIG. 9, for example, the first two slices of image 12 in slice order 652 may already have to be decoded before decoder 640 can start decoding the first slice of enhancement layer image 15. However, this type of blurring option of inter-layer prediction is deselected or not applied, because, for example, if the sampling resolutions between images 12 and 15 are equal to each other and images 12 and 15 both relate to the same view such that no disparity compensation occurs, the long-term syntax element structure will be set equal to 1 by encoder 640 so that decoder 640 can start decoding the first slice of enhancement layer image 15 as soon as the first slice of base layer image 12 is completely decoded.

[0088] Thus, by informing the decoder based on the long-term syntax element structure about the inter-layer offset, i.e., that applies to the entire predetermined period 608 and that relates to the number of spatial segments of the base layer picture 12 that should already be decoded before starting the first spatial segment of the time-aligned picture 15, the description of Figures 16 through 17c makes clear that the use of the long-term syntax element structure 642 helps support the decoder when the encoder plans to optionally parallel decode also the time-aligned pictures 12 and 15. It should be noted that the video decoder 640 can determine the inter-layer offset indicated by the long-term syntax element 642 even based on inspection / evaluation of the short-term syntax element 602, and that further syntax elements regarding potential options are related to inter-layer prediction, with these options switching on or off the just-outlined blurring in inter-layer prediction from the base layer to the enhancement layer. However, because the video decoder 640 is provided by the long-term syntax element structure 642, it must examine a multitude of syntax elements to derive the same information, and the video decoder 640 can only derive the same information on a short-term basis in advance rather than simply over the long predetermined period 608.

[0089] In the following, similar to aspects 1 and 2, a method that can bring the delayed display aspect to HEVC is described below.

[0090] Firstly, with reference to Figure 18, how WPP is currently implemented in HEVC will be described, i.e. this description will also form the basis for any implementation of the WPP process of any of the above embodiments.

[0091] In the base layer, wavefront parallelism allows for parallel processing of coding tree block (CTB) rows. Prediction dependencies are not broken across CTB rows. For entropy coding, as seen in Figure 18, WPP shifts the CABAC dependency on the upper-left CTB in each upper CTB row. Entropy coding of a CTB in the next row can begin once entropy decoding of the corresponding upper-right CTB is completed.

[0092] In the enhancement layer, decoding of a CTB can start as soon as the CTB containing the corresponding image area is fully decoded and available.

[0093] The decoding delay or offset indication concept of FIG. 16 can end up in signaling that facilitates decoder initialization and operation for layered bitstreams that use tiles, WPPs, or slices for parallelization.

[0094] When spatial scalability is used, decoding of an enhancement layer CTB can start only if the base layer CTB covering the corresponding image area is available. When parallelizing a WPP with a layered bitstream with spatial scalability, the layers can differ in terms of image size; for example, image size scaling factors of 1.5 and 2 between layers are stated in the published request for proposals for scalable extensions to HEVC [1], and further, maximum CTB sizes, such as 16, 32, and 64 image samples, are supported in the Main Profile of HEVC. For good scalability, the image size scaling factor is usually constant, but the maximum CTB size can still differ between layers.

[0095] The ratio between the maximum CTB size and the layer's picture size scaling factor affects the layer decoding delay, i.e., the CTB row offset before the first CTB row of the enhancement layer can be decoded with respect to the decoding of the base layer CTB row. Figure 19 reports the ratio of CTBs covering the corresponding image area in both layers with typical parameter values ​​for the picture size scaling factor and CTB size with respect to the CTB row and the introduced layer decoding delay.

[0096] For good scalability between layers, the picture size scaling factor between layers is usually equal to 1, while the maximum CTB size in each layer may still be different and affect the layer decoding delay.

[0097] The syntax element structure 642 provides a decoder hint to the bitstream that indicates the layer decoding delay for dependent spatial or quality enhancement layers when WPP processing is parallelized across spatial enhancement layers.

[0098] An embodiment uses a picture size scaling factor and a maximum CTB size scaling factor between each layer to determine the signaled layer decoding delay.

[0099] Depending on the type of scalability between the independent base and dependent enhancement layer bitstreams, the factors that affect the layer decoding delay are different.

[0100] In multiview scalability, layers represent camera views, and an inter-layer prediction mechanism is used to perform prediction between camera views from different angles. The prediction uses a motion compensation mechanism to compensate for different camera positions within the camera setup. In this case, the layer decoding delay is further bounded by the maximum or actual motion vector in the vertical direction compared to the spatial or fine scalability case.

[0101] The syntax element structure 642 describes a decoder hint in the bitstream that indicates the layer decoding delay for dependent camera views when WPP processing is parallelized across multiple camera views.

[0102] An embodiment uses the vertical picture size scaling factor, maximum CTB size scaling factor, and maximum motion vector length between each layer to determine the signaled layer decoding delay.

[0103] The examples show the layer decoding delay for spatial, quality and multiview scalability for spatial segments, i.e., rows of the CTB, when WPP is used in the VUI syntax of an SPS associated with an enhancement layer or in editing within the VPS extension syntax.

[0104] Tile and Slice Parallel processing with other partitioning techniques, such as tiles or slices, can benefit from hints within the bitstream that indicate the decoding delay based on the partitioning of the image into spatial segments, i.e., tiles or slices. Information in the base layer (e.g., reconstructed image data) may be necessary for the enhancement layer decoding process.

[0105] The syntax element structure 642 describes decoder hints in the bitstream that indicate the layer decoding delay for tiles and / or slices.

[0106] A possible embodiment of the present invention uses spatial segments as units to represent the layer processing delay introduced depending on the type of parallelization technique used in the coded video sequence.

[0107] The syntax in FIG. 20 provides an exemplary embodiment for the representation of min_spatial_segments_delay (an embodiment for syntax element structure 642) in the VUI parameters of the parallelization tool WPP, enhancement layer SPS for tiles and slices.

[0108] min_spatial_segment_delay describes the decoding delay of the current layer induced by the coding dependency with respect to the corresponding base layer in terms of spatial segments.

[0109] Depending on the value of min_spatial_segment_delay, the following applies:

[0110] If min_spatial_segment_delay is equal to 0, no restriction on the minimum delay between layer decoding is indicated.

[0111] Otherwise (min_spatial_segment_delay is not equal to 0), it is a requirement for bitstream conformance that exactly one of the following conditions be true: For each picture parameter set that is activated in the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0 (i.e., neither tiles nor WPP are used in the video sequence), and when the first min_spatial_segment_delay slices of the base layer in bitstream order are fully decoded, all base layer resources for the decoding process of the first slice of the current layer in bitstream order are available. For each picture parameter set that is activated in the coded video sequence, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0 (i.e., the tile is used in the video sequence), and when the first min_spatial_segment_delay tiles covering the same image area are completely decoded, all base layer resources for the decoding process of the first tile of the current layer in bitstream order are available. For each picture parameter set that is activated in the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 1 (i.e., WPP is used in the coded video sequence), and when the first min_spatial_segment_delay CTB row of the base layer is completed, all base layer resources for the decoding process of the first CTB row of the current layer are available.

[0112] Another exemplary embodiment is the representation of min_spatial_segments_delay in the VPS extension syntax, as reported in [4] and extended as shown in FIG.

[0113] min_spatial_segment_delay describes the decoding delay of layer[i] with respect to the spatial segment, induced by coding dependencies with respect to the corresponding base layer.

[0114] Depending on the value of min_spatial_segment_delay, the following applies: If min_spatial_segment_delay is equal to 0, no restriction on the minimum delay between decoding of layers is indicated.

[0115] Otherwise (min_spatial_segment_delay is not equal to 0), it is a requirement for bitstream conformance that exactly one of the following conditions be true: In each picture parameter set that is activated during the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0 (i.e., neither tiles nor WPP are used in the video sequence), and when the first min_spatial_segment_delay slices of the base layer in bitstream order are fully decoded, all base layer resources for the decoding process of the first slice of the current layer in bitstream order are available. For each picture parameter set that is activated during a coded video sequence, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0 (i.e., the tile is used in the video sequence), and when the first min_spatial_segment_delay tiles covering the same image area are completely decoded, all base layer resources for the decoding process of the first tile of the current layer in bitstream order may be utilized. In each picture parameter set that is activated in the coded video sequence, tiles_enabled_flag is equal to 0, entropy_coding_sync_enabled_flag is equal to 1 (i.e., WPP is used in the coded video sequence), and when the first min_spatial_segment_delay CTB row of the base layer is completed, all base layer resources for the decoding process of the first CTB row in the current layer are available.

[0116] To enable parallel processing concepts, such as tile and / or WPP concepts, various prediction modes supported by the encoder and decoder, as well as restrictions imposed on prediction modes, as well as context derivation for entropy encoding / decoding, were described above. It was also mentioned above that the encoder and decoder may operate on a block basis. For example, the prediction modes described above are selected on a block basis, i.e., at a finer granularity than the image itself. Before continuing to describe other aspects of the present application, the relationship between slices, tiles, WPP substreams, and the just-mentioned blocks will be explained.

[0117] FIG. 32 illustrates a layer 0 image, such as layer 12, or a layer 1 image, such as image 15. The image is periodically subdivided into an array of blocks 90. These blocks 90 are often referred to as largest coding blocks (LCBs), largest coding units (LCUs), coding tree blocks (CTBs), etc. The subdivision of the image into blocks 90 may form a kind of base or coarsest granularity at which the above-described prediction and residual coding is performed, and this coarsest granularity, i.e., the dimensions of the blocks 90, may be indicated and set by the encoder for layers 0 and 1, respectively. For example, a multi-tree subdivision, such as a quadtree, may be used and indicated within the data stream to subdivide each block 90 into a prediction block, a residual block, and / or a coding block, respectively. In particular, the coding blocks may be leaf blocks of a recursive multi-tree subdividing the blocks 90, and some prediction-related decisions may be indicated at the granularity of the coding block, e.g., the prediction mode. Then, the prediction block at the granularity at which prediction parameters such as motion vectors in the case of temporal inter prediction and disparity vectors in the case of inter layer prediction are coded, and the residual block at the granularity at which prediction residuals are coded are leaf blocks of the recursive multi-tree subdivided into coding blocks.

[0118] A raster scan encoding / decoding order 92 may be defined between the blocks 90. The encoding / decoding order 92 limits the possibility of adjacent portions for spatial prediction purposes: only the portion of the image according to the encoding / decoding order 92 precedes the current portion, e.g., block 90 or a smaller number of blocks. The syntax element currently to be predicted is associated with the portion used for spatial prediction within the current image. Within each layer, the encoding / decoding order 92 traverses all blocks 90 of the image, then continues to traverse the blocks of the next image in the respective layer in an image encoding / decoding order that does not necessarily follow the temporal reproduction order of the images. Within individual blocks 90, the encoding / decoding order 92 is refined during scanning between smaller blocks, e.g., coding blocks.

[0119] With respect to the blocks 90 and smaller blocks just outlined, each image is further subdivided into one or more slices along the just-mentioned encoding / decoding order 92. Thus, slices 94a and 94b, shown by way of example in Figure 32, cover each image contiguously. The boundary or interface 96 between successive slices 94a and 94b within an image may or may not be aligned with the boundary of an adjacent block 90. ​​More precisely, and as shown on the right side of Figure 32, successive slices 94a and 94b within an image may abut each other at the boundary of a smaller block, such as a coding block, i.e., a leaf block of one subdivision of block 90.

[0120] Slices 94a and 94b of a picture may form the smallest unit in which portions of a data stream in which a picture is coded can be packetized into packets, i.e., NAL units. Further possible properties of slices, i.e., restrictions on slices regarding prediction across slice boundaries and determination of entropy context, have been described above, for example. Slices with such restrictions may be called "normal" slices. As will be outlined in more detail below, in addition to normal slices, "dependent slices" may exist as well.

[0121] When a tile division concept is used for an image, the encoding / decoding order 92 defined within the array of blocks 90 can vary. This is shown by way of example in FIG. 33, which illustrates an image divided into four tiles 82a-82d. As shown in FIG. 33, the tiles themselves are defined as regular subdivisions of the image in units of blocks 90. That is, each tile 82a-82d is composed of an array of n×m blocks 90, with n set individually for each row of the tile and m set individually for each column of the tile. After the encoding / decoding order 92, the blocks 90 in the first tile are scanned first in raster scan order before proceeding to the next tile 82b, etc., where tiles 82a-82d are themselves scanned in raster scan order.

[0122] According to the WPP stream division concept, an image is subdivided into WPP substreams 98a-98d in units of one or more rows of blocks 90 along the encoding / decoding order 92. Each WPP substream may cover one complete row of blocks 90, for example, as shown in Figure 34.

[0123] The tile and WPP substream concepts may, however, be mixed, in which case each WPP substream covers, for example, one row of blocks 90 within each tile.

[0124] Even the slice division of an image can be shared with the tile division and / or WPP substream division. For each tile, one or more slices, the image can be subdivided to form exactly one complete tile, one or more complete tiles, or even just a subportion of one tile along the encoding / decoding order 92. The slices can be used to form WPP substreams 98a-98d. For this purpose, slices forming the smallest units for packetization can be composed of regular slices on the one hand and dependent slices on the other hand: while regular slices impose the above-mentioned restrictions on prediction and entropy context derivation, dependent slices do not impose such restrictions. A dependent slice starting at an image boundary where the encoding / decoding order 92 indicates a substantial deviation from the row direction can adopt the entropy context resulting from the entropy decoding block 90 in the row immediately preceding block 90, and a dependent slice starting elsewhere can adopt the entropy coding context resulting from entropy encoding / decoding the previous slice up to that edge. By this measure, each WPP substream 98a-98d may consist of one or more dependent slices.

[0125] That is, the encoding / decoding order 92 defined among the blocks 90 proceeds linearly from the first side of each image, illustratively the left side here, to the opposite side, illustratively the right side here, and then downward / bottom to the next row of blocks 90. The available, i.e., already encoded / decoded, portions of the current image are therefore located primarily to the left and primarily at the top of the currently encoded / decoded portion, e.g., the current block 90. ​​Due to prediction confusion and entropy context derivation across tile boundaries, tiles of one image can be processed in parallel. The encoding / decoding of tiles of one image can even begin in parallel. The limitation on crossing tile boundaries arises from the aforementioned in-loop filtering. Next, the encoding / decoding of the WPP substreams is performed sequentially in a staggered manner, from top to bottom. The intra-picture delay between successive WPP substreams is measured in blocks 90, two blocks 90.

[0126] However, it is preferable to parallelize even the encoding / decoding of images 12 and 15, i.e., different layer instances. Clearly, the encoding / decoding of image 15 of the dependent layer must be delayed relative to the encoding / decoding of the base layer to ensure that a "spatially consistent" portion of the base layer is already available. These considerations are even valid when no parallelization of encoding / decoding is used within any of images 12 and 15 individually. Even when using one slice to cover the entire images 12 and 15, the encoding / decoding of images 12 and 15 can be parallelized by not using tile and WPP substream processing, respectively. The signaling described next, i.e., aspect 6, can represent even this type of decoding / encoding delay between layers, regardless of where or whether tile or WPP processing is used for any of the layer images.

[0127] As is evident from the above discussion regarding the minimum encoding delay during the encoding of successive layers, it is also evident that a decoder may determine the minimum decoding delay based on short-term syntax elements. However, when using long-term syntax elements as described above to indicate this inter-layer time delay further in advance for a given period of time, the decoder may use the guarantees provided to plan ahead and more easily perform workload distribution within the context of parallel decoding of bitstream 40.

[0128] The next described aspect of the present application, i.e., the sixth aspect of the present application, is somehow related to aspect 3 in explicitly signaling any inter-layer offset. However, with respect to the sixth aspect of the present application, the syntax element structure that explicitly signals the inter-layer offset does not need to be indicated on a long-term basis in association with the short-term syntax element from which the inter-layer offset can otherwise be derived. Rather, the sixth aspect of the present application utilizes another discovery: as will become clear when describing Figures 17a to 17c, if both the base and enhancement layer images are subdivided into blocks with a raster scan decoding order defined therebetween, then the inter-layer offset between the base and enhancement layers can be efficiently indicated explicitly by measuring the inter-layer offset in units of base layer blocks. As with the aspects described further below, in units in which the inter-layer offset is indicated explicitly, the base layer blocks are not limited to spatial segments. Rather, other coding blocks can be used for this purpose. Therefore, when describing an embodiment with respect to the sixth aspect of the present application with respect to Figure 34, the reference numerals used with respect to Figures 16 to 17c are mainly shared, and the explanations given above with respect to the latter figures apply equally to the embodiments described further below in order to avoid unnecessary repetition. Beyond this, reference is made to the explanations of Figures 32 and 33, as these figures form the basis for a possible coexistence between coding blocks on the one hand and spatial segments on the other.

[0129] Thus, Figure 35 illustrates a video decoder 720 configured to receive a multi-layer video data stream 40 in which a scene is encoded in a hierarchy of layers using inter-layer prediction from a portion of a first layer to a common location in a second layer. Similar to the previous figures, Figure 35 illustrates layer 0 juxtaposed to layer 1 as a representative example. Two time-aligned images 12 and 15 of both layers are illustrated in Figure 35 as an example. Image 12 of base layer 0 is subdivided into a first array of blocks 722, and image 12 of enhancement layer 1 is subdivided into a second array of blocks 724. Raster scan decoding order 726 is defined in block 722, and similarly, raster scan decoding order 728 is defined in block 724.

[0130] Video data stream 40 comprises a syntax element structure 730 indicating the inter-layer offset for pictures 12 and 15 decoded in parallel by successively traversing first and second blocks 722 and 724 measured in units of base layer blocks 722 in a time-overlapping manner with an inter-layer offset between traversals of the sequential first and second blocks 722 and 724. Video decoder 720 is configured to respond to this syntax element structure 730. In particular, the video decoder determines the inter-layer offset from the latter.

[0131] 16 , syntax element structure 730 may indicate the inter-layer offset as a guarantee for successful parallel decoding of time-aligned pictures 12 and 15 for a predetermined period longer than the short-term time interval indicating the subdivision of any syntax element signal pictures 12 and 15 into blocks 722 and 724, respectively. This is not mandatory, however. Rather, explicit signaling of the inter-layer offset as syntax element structure 730 may be implemented to a different extent in data stream 40, e.g., for each time-aligned pair of base and enhancement pictures 12 and 15, e.g., at the same interval as signaling regarding the size and subdivision into blocks 722 and 724.

[0132] In further agreement with the above embodiment, the decoder 720 may use the inter-layer offset specified in the syntax element structure 730 as a magnitude for the offset at which to begin traversing a second block 724 relative to the traversal of a first block 722 of pictures 12 and 15 to decode in parallel, respectively. In other words, the video decoder 720 may be configured to derive the count value from the syntax element structure 730, allowing it to start decoding the order of blocks 724 of the enhancement layer picture 15 along the decoding order 728 only after counting the already decoded blocks 722 that has reached the minimum count specified by the syntax element structure 730, and counting the already decoded blocks 722 of the base layer picture 12 along the raster scan decoding order 726. Thus, the video decoder 720 does not need to examine any very complex and distributed portions of the video data stream 40. 7. This allows video decoder 720 to calculate the actual minimum inter-layer offset between starting decoding block 722 on the one hand and block 724 on the other hand.

[0133] Interestingly, however, according to the embodiment of Figure 35, blocks 722 and 724 do not necessarily represent spatial segments dedicated to any particular parallel processing. Rather, blocks 722 and 724 may be ordinary coding blocks, in which the content of images 12 and 15, respectively, is coded into video data stream 40. For example, blocks 722 and 724 may be tree root blocks into which images 12 and 15 are regularly subdivided, i.e., into rows and columns. The tree root blocks are then further independently subdivided in a recursive multi-tree manner, as described above with respect to Figure 32. For example, inter-layer prediction may be indicated in video data stream 40, with the resulting leaf blocks of the tree root blocks representing the subdivision of images 12 and 15 into coding blocks, whether spatial, temporal, or inter-layer prediction, as selected with respect to Figure 15.

[0134] To explain the embodiment of Figure 35 in more detail, reference is made to Figure 36. As shown in Figure 36, by starting counting from the first block 722 of image 12 in raster scan decoding order 726, video decoder 720 may use counter 732 to count the number of already decoded blocks 722 of base layer image 12. Comparator 734 of decoder 720 compares this steadily increasing count output by counter 732 with an explicitly indicated value of syntax element structure 730 as obtained from video data stream 40. In the event that the count satisfies a predetermined relationship with the value indicated by syntax element structure 730, for example, as soon as the count of counter 732 reaches or equals the value indicated by syntax element structure 730, the comparator activates or causes the start of decoding of enhancement layer image 15, i.e., causes decoding of the first block 724 of enhancement layer image 15 in raster scan decoding order 728 to begin.

[0135] As will be described in more detail below, for an explicit embodiment for the syntax for syntax element structure 730, it is possible to unify the syntax element structure of FIG. 35 with the syntax element structure and its object of FIG. 16. This means that syntax element structure 730 may have a set of possible values, i.e., a set of probable values. As mentioned by way of example with respect to FIG. 16, an implicit inter-layer offset origin value of the probable values ​​may cause video decoder 720 to ignore the value of syntax element structure 730 and not perform parallel decoding of pictures 12 and 15, or to determine any inter-layer offset based on short-term syntax element 602. In the case of syntax element structure 730 assuming an origin value of a second set of possible values, this causes video decoder 720 to already perform the operations outlined, for example, with respect to FIG. 36, according to which the value of syntax element structure 730 explicitly indicates the inter-layer offset in units of base layer block 722. However, there may be other subsets of possible values ​​of syntax element structure 730, which, when assumed by syntax element structure 730, cause video decoder 720 to act as described above with respect to Figure 16: determine the inter-layer offset between decoding of base layer picture 12 and enhancement layer picture 15 based on the value of syntax element structure 730 by interpreting the latter as measuring the inter-layer offset in units of spatial segments, which may, but may not necessarily, consist of integer numbers of blocks 722 and 724, respectively.

[0136] The just-mentioned possibility of combining the embodiment of Figure 35 with that of Figure 16 is illustrated with respect to Figure 37. As shown in Figure 37, a video decoder may examine syntax element structure 730 to determine whether it has a value in first subset 736, second subset 738, or third subset 740 from which a set of possible values ​​742 results. Depending on the results of the examination or check, decoder 720 also does not derive the guarantees underlying syntax element structure 730 and does not derive explicit signaling regarding inter-layer offset from syntax element structure 730, i.e., performs derivation of inter-layer offset from syntax element structure 730 in units of spatial segments or blocks. In the case of the second subset 738, no derivation / guarantee occurs, derivation of the interlayer offset in units of spatial segments occurs in the case of subset 736, and derivation of the interlayer offset in units of blocks occurs in the case of syntax element 730 assuming the underlying values ​​of the third subset 740. In a specific syntax embodiment outlined further below, the syntax element structure includes two flags, namely, ctb_delay_enabled_flag and min_spatial_segment_delay, where ctp_delay_enabled_flag=0 and min_spatial_segment_delay≠0 corresponds to the case of subset 736, min_spatial_segment_delay=0 corresponds to the second subset 738, and ctp_delay_enabled_flag=1 and min_spatial_segment_delay≠0 corresponds to the third subset 740.

[0137] Finally, with reference to Figure 38, it is shown that the decoder 720 can be configured to interpret any inter-layer offset indicated by the syntax element structure 730. When followed, it is not an inter-layer offset relative to the start of decoding of the first block or spatial segment of the enhancement layer picture 15, but a consecutive inter-layer offset, resulting in consistent parallel decoding of pictures 12 and 15, respectively. As shown in Figure 38, the counter 732 still counts the number of already decoded blocks 722 of the base layer picture 12, where an additional counter 744 similarly counts the already decoded blocks 724 of the enhancement layer picture 15 along the decoding order 728, where a subtractor 746 forms the difference of both counts, i.e., s and t-1, i.e., calculates s-t+1. This difference value is compared by a comparator 734 with the inter-layer offset value derived from the syntax element structure 730. Then, as soon as both values, i.e., the derived inter-layer offset and the difference value of the count, have a predetermined relationship, for example, the difference value is equal to or greater than the derived inter-layer offset, it starts decoding block t in the decoding order 728 among the enhancement layer blocks 724. This approach introduces a continuous search between the causes of decoding between the blocks 722 of the base layer picture 12 on the one hand and the blocks 724 of the enhancement layer picture 15 on the other hand.

[0138] It is clear that sequential search can also be applied to spatial segments according to Figure 38. More generally, the description of Figures 38 and 36 can be readily transferred to spatial segments as well, and this description also applies to the embodiment of Figure 16 in which syntax element structure 642 can serve as the associated syntax element structure shown at 730 in Figures 36 and 38. In other words, at least when using tiles as spatial segments, there is also a raster scan decoding order defined therebetween, so that the concepts of Figures 36 and 38 described with respect to coding blocks can be easily transferred to traversing tiles and their decoding.

[0139] To briefly summarize the sixth aspect and embodiments described thereto, a syntax element structure 730 may be inserted into a bitstream by a video encoder to provide explicit hints to a decoder for controlling parallel decoding of base and enhancement layer images relative to each other. The inter-layer offset explicitly indicated by the syntax element structure may be activated or deactivated. If activated, the indication may be in units of blocks such as CTBs or via more precise signaling, in the indicated one of such units of blocks and units of spatial segments. Since a raster scan order is used between the base layer blocks, on the one hand, and the enhancement layer blocks, on the other hand, both proceed row-wise, e.g., from the top to the bottom, row by row, from the upper left corner to the lower right corner of each image 12 / 15, the explicitly indicated inter-layer offsets are interpreted merely as a "trigger" for undertaking / starting the decoding of the first block of the enhancement layer image 15, or as a continuous "safe distance" between the currently decoded block of the base layer image 12 on the one hand and the currently decoded block of the enhancement layer image 15 on the other hand, i.e., for determining the trigger for decoding each block of the enhancement layer image 15. The explanations given for the sixth aspect may be transferred to the explanations and examples of at least the third aspect, as far as the explanation of tiles with respect to spatial segments, the interpretation and checking of compliance with the signaled inter-layer offsets are concerned, and the examples of Figures 36 and 38 may be used. That is, the traversal of decoded tiles within the base and enhancement layer images along the raster scan tile decoding order can be controlled in a manner related to the illustrations of FIGS.

[0140] In this way, measurements for the "delay" spatial segment can be used, i.e. The delay may be expressed in units of spatial segments (tiles, slices, or CTB rows for WPP), or the delay / offset may be measured in units of blocks 90.

[0141] The High Efficiency Video Coding (HEVC) standard may be extended to meet the sixth aspect as follows: In particular, parallel decoding of individual layers (or views) is possible if reference data is available. The minimum delay (particularly the layer decoding delay) between the decoding of a base layer coding treeblock (CTB) and the decoding of the corresponding dependent enhancement layer CTB is determined through the guarantee of parallelization tools as tiles, wavefronts or slices or motion compensation vectors may be applied (e.g., in stereoscopic or multi-view video coding).

[0142] FIG. 20 shows the layer decoding delay indication realized by the enhancement layer sequence parameter set syntax and the semantics realized by the syntax element min_spatial_segment_delay.

[0143] min_spatial_segment_delay described the decoding delay of the current layer induced by the coding dependency with respect to the corresponding base layer in terms of spatial segments.

[0144] A mechanism in the HEVC high-level syntax that allows for arbitrary representation of layer decoding delay between an independent base layer and dependent enhancement layer regardless of the underlying parallelization technique based on the number of vertical and horizontal CTBs used can be obtained as follows.

[0145] A flag or index (eg, ctb_delay_enabled_flag) may be used to indicate that the layer decoding delay (as indicated in the second syntax element) is expressed as a specific CTB address in the coded image.

[0146] The CTB addresses in raster scan order clearly define the horizontal and vertical positions within the image used to represent the delay as can be seen from FIG. 39 and according to the following.

[0147] CTB coordinates = (CTB address % PicWidthlnCTBs, CTB address / PicWidthlnCTBs)

[0148] PicWidthInCTBs describes the width of the image in CTB units.

[0149] Figure 39 shows that a CTB address within the image, say 7, defines a horizontal CTB column and a vertical CTB row, say the tuple (2,1).

[0150] If the flag is enabled, when decoding a CTB in the currently dependent layer, the values ​​of other syntax elements (compare min_spatial_segment_delay) should be interpreted as offsets to the CTB address of the co-located CTB in the relative base layer image.

[0151] As shown in FIG. 40 and detailed during the following description, the co-located CTBs can be calculated according to the size of the CTBs in the two respective layers and the width of the images in the two respective layers.

[0152] Figure 40 includes three examples from left to right showing various setups of CTB size and image size in the two respective base and enhancement layers, regardless of image scaling. The bold boxes in the base layer images mark the image area at the size of the enhancement layer CTB and the image area located in the same position in the respective base layer CTB placement.

[0153] An example of this kind of arbitrary CTB based on layer decoding delay indication is given in the enhancement layer sequence parameter set syntax and semantics by the syntax element min_spatial_segment_delay in FIG.

[0154] ctb_based_delay_enabled_flag equal to 1 indicates the delay indicating that min_spatial_segment_delay is given in units of CTBs. ctb_based_delay_enabled_flag indicates that min_spatial_segment_delay is not given in units of CTBs. min_spatial_segment_delay describes the decoding delay of the current layer induced by coding dependencies with respect to the corresponding base layer with respect to spatial segments. Depending on the value of min_spatial_segment_delay, the following applies: If min_spatial_segment_delay is equal to 0, no restriction on the minimum delay between decoding of layers is indicated. Otherwise (if min_spatial_segment_delay is not equal to 0) and ctb_based_delay_enabled_flag is equal to 1, it is a requirement for bitstream conformance that the following condition is true: CtbSizeY A , PicWidthInCtbsY A and ctbAddrRsA is the Ctb Address of the Ctb in base layer A in raster scan order, and CtbSizeY is the PicWidthInCtbsY of base layer A. B , PicWidthInCtbsY B and ctbAddrRs B is the CtbSizeY and PicWidthInCtbsY of dependent layer / view B and the Ctb Address of dependent layer B's Ctb in raster scan order, and CtbScalingFactor BA , CtbRow BA (ctbAddrRsB) and CtbCol BA (ctbAddrRs) is defined as follows: CtbScalingFactor BA =(PicWidthInCtbsY A / PicWidthInCtbsY B ) CtbRow BA (ctbAddrRs)= Ceil((Floor(ctbAddrRs / PicWidthInCtbsY B )+1)*CtbScalingFactor BA )-1 CtbCol BA (ctbAddrRs)= Ceil(((ctbAddrRs%PicWidthInCtbsY B )+1)*CtbScalingFactor BA )-1 ctbAddrRs of current enhancement layer / view B B When decoding a CTB with PicWidthInCtbs YA *CtbRow BA (ctbAddrRs B )+CtbCol BA (ctbAddrRs B ctbAddrRs equal to min_spatial_segment_delay A When the base layer CTB with ≠ 0 is fully decoded, all necessary base layer resources will be available. Otherwise (min_spatial_segment_delay is not equal to 0 and ctb_based_delay_enabled is equal to 0), it is a requirement for bitstream matching that exactly one of the following conditions be true: In each picture parameter set that is activated in the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0 (i.e., neither tiles nor WPP are used in the video sequence), and when the first min_spatial_segment_delay slices of the base layer in bitstream order are fully decoded, all base layer resources for the decoding process of the first slice of the current layer in bitstream order are available. For each picture parameter set that is activated in the coded video sequence, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0 (i.e., the tile is used in the video sequence), and when the first min_spatial_segment_delay tiles covering the same image area are completely decoded, all base layer resources for the decoding process of the first tile of the current layer in bitstream order are available. In each picture parameter set that is activated in the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 1 (i.e., WPP is used in the coded video sequence), and when the first min_spatial_segment_delay CTB row of the base layer is completed, all base layer resources for the decoding process of the first CTB row in the current layer may be available.

[0155] Alternatively, the inter-layer offset can be indicated as a kind of worst-case delay for the ctb_based_delay_enabled_flag, instead of the initial delay for the first slice / tile / CTB row as in the previous embodiment. The worst-case delay provides a guarantee that all necessary respective base layer resources are available in the process of decoding the spatial segments of the dependent images when the co-located spatial segments and the signaled offsets are fully decoded.

[0156] An example for the syntax is shown in FIG.

[0157] min_spatial_segment_delay describes the decoding delay of the current layer induced by coding dependencies with respect to the corresponding base layer with respect to the spatial segment. Depending on the value of min_spatial_segment_delay, the following applies: If min_spatial_segment_delay is equal to 0, no restriction on the minimum delay between decoding of layers is indicated. Otherwise (min_spatial_segment_delay is not equal to 0), it is a requirement for bitstream matching that exactly one of the following conditions be true: In each picture parameter set that is activated in the coded video sequence, tiles_enabled_flag is equal to 0, and entropy_coding_sync_enabled_flag is equal to 0 (i.e., neither tiles nor WPP are used in the video sequence), and all base layer resources for the decoding process of any slice segment A of the current layer in bitstream order may be utilized when the first slice segment C that follows (min_spatial_segment_delay-1) slices in bitstream order in the base layer after the last slice segment B in bitstream order that includes at least a portion of the same image area as slice A in the current layer is fully decoded. For each picture parameter set that is activated in the coded video sequence, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0 (i.e., the tile is used in the video sequence), and when the first tile C that follows the (min_spatial_segment_delay-1) tiles in bitstream order after the last tile B in bitstream order that contains at least a portion of the same image area as tile A has been fully decoded, all base layer resources for the decoding process of any tile A of the current layer in bitstream order may be utilized. For each picture parameter set that is activated in the coded video sequence, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 1 (i.e., WPP is used in the coded video sequence), and all base layer resources for the decoding process of any CTB row A in the current layer are available when the first CTB row C that follows the (min_spatial_segment_delay-1) CTB row after the last CTB row B of the base layer that covers at least a portion of the same image area as a CTB row A of the enhancement layer in bitstream order is fully decoded.

[0158] Picture or sub-coded video sequences based on the signaling of min_spatial_segment_delay are also possible. The scope of the SEI message for the associated NAL unit is smaller than the coded video sequence in the time domain as given in Figure 20 and is also defined by the position of the SEI message in the bitstream or by its index. One example is given in Figure 43, Layer_decoding_delay_SEI.

[0159] The semantics for the above examples may vary to reflect the range of SEI messages and their syntax elements.

[0160] The above explicit embodiment may be slightly modified. In the above embodiment, the syntax element structure includes ctb_based_delay_enabled_flag with min_spatial_segment_delay and min_spatial_segment_delay measuring either in a one-dimensional or scalar manner—along the spatial segment / CTB decoding order—and the inter-layer coding offset in units of spatial segments or CTBs depends on ctb_based_delay_enabled_flag. However, since the number of CTBs in a base layer image is usually larger than the number of spatial segments, e.g., tiles or sub-streams of a base layer image, in a slightly different embodiment, when ctb_based_delay_enabled_flag indicates a CTB-based inter-layer offset indication, this inter-layer offset is not determined solely based on min_spatial_segment_delay; rather, the latter syntax element is then interpreted as indicating the position of the CTb of the base layer image in the horizontal dimension, in which complete decoding can be used as a trigger by the decoder to start decoding of the enhancement layer image. Naturally, min_spatial_segment_delay can instead be interpreted as indicating the CTB of the base layer image along the vertical dimension. Depending on the ctb_based_delay_enabled_flag, i.e., if it indicates a CTB-based display, further syntax elements are transmitted in the data stream indicating, in the other dimension, the position of the CTB of the base layer image acting as the just-mentioned trigger.

[0161] That is, the following syntax fragments may be used for signaling, i.e. as syntax element structures:

[0162] The indexes i and j may indicate the layer ids of the base layer and enhancement layer.

[0163] JPEG2025179048000002.jpg47164

[0164] The semantics of the above syntax elements can be described as follows:

[0165] min_spatial_segment_offset_plus1[i][j] indicates that the spatial domain, as specified below, is not used for inter-layer prediction for decoding of any image of the i-th layer by itself or together with min_horizontal_ctu_offset_plus1[i][j] in each image of the j-th direct reference layer of the i-th layer. The value of min_spatial_segment_offset_plus1[i][j] ranges from 0 to refPicWidthInCtbsY[i][j]*refPicHeightInCtbsY[i][j] inclusive. When not present, the value of min_spatial_segment_offset_plus1[i][j] is inferred to be equal to 0. ctu_based_offset_enabled_flag[i][j] equal to 1 specifies that, in units of CTUs, min_spatial_segment_offset_plus1[i][j] and min_horizontal_ctu_offset_plus1[i][j] together indicate that, in each image of the jth direct reference layer of the ith layer, the spatial domain is not used for inter-layer prediction for decoding of any image of the ith layer. ctu_based_offset_enabled_flag[i][j] equal to 0 specifies that, in units of slice segments, tiles, or CTU rows, min_spatial_segment_offset_plus1[i] alone indicates that, in each image of the jth direct reference layer of the ith layer, the spatial domain is not used for inter-layer prediction for decoding of any image of the ith layer. When not present, the value of ctu_based_offset_enabled_flag[i] is inferred to be equal to 0. When ctu_based_offset_enabled_flag[i][j] is equal to 1, min_horizontal_ctu_offset_plus1[i][j] indicates that the spatial domain, together with min_spatial_segment_offset_plus1[i][j], in each image of the jth direct reference layer of the ith layer, is not used for inter-layer prediction for decoding of any image of the ith layer, as specified below. The value of min_horizontal_ctu_offset_plus1[i][j] ranges from 0 to refPicWidthInCtbsY[i][j] inclusively.

[0166] When ctu_based_offset_enabled_flag[i][j] is equal to 1, the variable minHorizontalCtbOffset[i][j] is derived as follows: minHorizontalCtbOffset[i][j]=(min_horizontal_ctu_offset_plus1[i][j]>0)?

[0167] (min_horizontal_ctu_offset_plus1[i][j]-1):(refPicWidthInCtbsY[i][j]-1) Variable curPicWidthInSamples L [i],curPicHeightInSamples L [i],curCtbLog2SizeY[i],curPicWidthInCtbsY[i] and curPicHeightInCtbsY[i] are the PicWidthInSamples of the ith layer. L , PicHeightInSamples L , CtbLog2SizeY, PicWidthInCtbsY and PicHeightInCtbsY, respectively. Variable refPicWidthInSamples L [i][j],refPicHeightInSamples L [i][j],refCtbLog2SizeY[i][j],refPicWidthInCtbsY[i][j] and refPicHeightInCtbsY[i][j] are the PicWidthInSamples of the jth direct reference layer of the ith layer. L , PicHeightInSamplesL, CtbLog2SizeY, PicWidthInCtbsY and PicHeightInCtbsY, respectively. The variables curScaledRefLayerLeftOffset[i][j], curScaledRefLayerTopOffset[i][j], curScaledRefLayerRightOffset[i][j] and curScaledRefLayerBottomOffset[i][j] are set equal to scaled_ref_layer_left_offset[j]<<1, scaled_ref_layer_top_offset[j]<<1, scaled_ref_layer_right_offset[j]<<1, scaled_ref_layer_bottom_offset[j]<<1, respectively, for the jth direct reference layer of the ith layer. In the jth direct reference layer image of the ith layer, the variable colCtbAddr[i][j], which means the raster scan address of the CTU located in the jth direct reference layer image of the ith layer, has a raster scan address equal to ctbAddr in the ith layer image, and is derived as follows: The variables (xP, yP) specifying the location of the top-left luminance sample of the CTU with a raster scan address equal to ctbAddr relative to the top-left luminance sample in the image of the ith layer are derived as follows: xP=(ctbAddr%curPicWidthInCtbsY[i])< <curCtbLog2SizeY yP=(ctbAddr / curPicWidthInCtbsY[i])< <curCtbLog2SizeY The variables scaleFactorX[i][j] and scaleFactorY[i][j] are derived as follows: curScaledRefLayerPicWidthInSamples L [i][j]=curPicWidthInSamples L [i]-curScaledRefLayerLeftOffset[i][j]-curScaledRefLayerRightOffset[i][j] curScaledRefLayerPicHeightInSamplesL[i][j]=curPicHeightInSamplesL[i]-curScaledRefLayerTopOffset[i][j]-curScaledRefLayerBottomOffset[i][j] scaleFactorX[i][j]=((refPicWidthInSamples L [i][j]<<16)+(curScaledRefLayerPicWidthInSamples L [i][j]>>1)) / curScaledRefLayerPicWidthInSamples L [i][j] scaleFactorY[i][j]=((refPicHeightInSamples L [i][j]<<16)+(curScaledRefLayerPicHeightInSamples L >>1)) / curScaledRefLayerPicHeightInSamples L [i][j]

[0168] The variables (xCol[i][j], yCol xCol[i][j]) specifying the image's assigned luminance sample position in the jth direct reference layer of the luminance sample position (xP, yP) of the ith layer are derived as follows: xCol[i][j]=Clip3(0,(refPicWidthInSamples L [i][j]-1),((xP-curScaledRefLayerLeftOffset[i][j])*scaleFactorX[i][j]+(1<<15))>>16)) yCol[i][j]=Clip3(0,(refPicHeightInSamples L [i][j]-1),((yP-curScaledRefLayerTopOffset[i][j])*scaleFactorY[i][j]+(1<<15))>>16)) The variable colCtbAddr[i][j] is derived as follows: xColCtb[i][j]=xCol[i][j]>>refCtbLog2SizeY[i][j] yColCtb[i][j]=yCol[i][j]>>refCtbLog2SizeY[i][j] colCtbAddr[i][j]=xColCtb[i][j]+(yColCtb[i][j]*refPicWidthInCtbsY[i][j]) When min_spatial_segment_offset_plus1[i][j] is greater than 0, it is a bitstream matching requirement that the following apply: When -ctu_based_offset_enabled_flag[i][j] is equal to 0, exactly one of the following applies: For each PPS associated with the jth direct reference layer image of the ith layer, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0, and the following applies: Let slice segment A be any slice segment in the image of the i-th layer, and let ctbAddr be the raster scan address of the last CTU in slice segment A. Let slice segment B be a slice segment that belongs to the same access unit as slice segment A, belongs to the j-th direct reference layer of the i-th layer, and contains a CTU with raster scan address colCtbAddr[i][j]. Let slice segment C be a slice segment that exists in the same image as slice segment B, follows slice segment B in decoding order, and exists between slice segment B and min_spatial_segment_offset_plus1[i]-1 slice segments in decoding order. When slice segment C exists, the syntax elements of slice segment A are constrained so that the samples or syntax element values ​​of slice segment C or any slice segments of the same image that follow C in decoding order are not used for inter-layer prediction in the decoding process of any samples within slice segment A. For each PPS referenced by an image in the jth direct reference layer of the ith layer, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 1, and the following applies: Let tile A be any tile in any image picA of the i-th layer, and let ctbAddr be the raster scan address of the last CTU in tile A. Let tile B be a tile in image picB that belongs to the same access unit as picA, belongs to the j-th direct reference layer of the i-th layer, and contains a CTU with raster scan address colCtbAddr[i][j]. Let tile C be a tile that is also present in picB and follows tile B in decoding order, and there are min_spatial_segment_offset_plus1[i]-1 tiles between tile B and that tile in decoding order. When slice segment C is present, the syntax elements of tile A are constrained so that the sample or syntax element values ​​in tile C or any tile of the same image following C in decoding order are not used for inter-layer prediction in the decoding process of any sample within tile A. For each PPS referenced by an image in the jth direct reference layer of the ith layer, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 1, and the following applies: Let CTU row A be any CTU row in any image picA of the i-th layer, and let ctbAddr be the raster scan address of the last CTU in CTU row A. Let CTU row B be a CTU row that belongs to the same access unit as picA, exists in image picB of the j-th direct reference layer of the i-th layer, and contains a CTU with raster scan address colCtbAddr[i][j]. Let CTU row C be a CTU row that also exists in picB and follows CTU row B in decoding order, and there are min_spatial_segment_offset_plus1[i]-1 CTU rows in decoding order between CTU row B and that CTU row. When CTU row C exists, the syntax elements of CTU row A are constrained so that neither samples nor syntax element values ​​in CTU row C, nor rows of the same image following C, are used for inter-layer prediction in the decoding process of any samples within CTU row A. - Otherwise (ctu_based_offset_enabled_flag[i][j] is equal to 1), the following applies: The variable refCtbAddr[i][j] is derived as follows: xOffset[i][j]=((xColCtb[i][j]+minHorizontalCtbOffset[i][j])>(refPicWidthInCtbsY[i][j]-1))? (refPicWidthInCtbsY[i][j]-1―xColCtb[i][j]): (minHorizontalCtbOffset[i][j]) yOffset[i][j]=(min_spatial_segment_offset_plus1[i][j]-1)*refPicWidthInCtbsY[i][j] refCtbAddr[i][j]=colCtbAddr[i][j]+xOffset[i][j]+yOffset[i][j] Let CTUA be any CTU in any picture picA of the i-th layer, and ctbAddr be the raster scan address of CTUA. Let CTUB be a CTU existing in a picture that belongs to the same access unit as picA, belongs to the j-th direct reference layer of the i-th layer, and has a raster scan address greater than refCtbAddr[i][j]. When CTUB exists, the syntax elements of CTUA are constrained so that the syntax element values ​​in sample or CTU row B are not used for inter-layer prediction in the decoding process of any sample within CTUA.

[0169] That is, to summarize the example just provided, the CTB-based representation of the inter-layer offset can be made in a switchable manner using a flag that switches between this CTB-based representation on the one hand and a spatial segment-based representation of the inter-layer offset on the other hand. And in doing so, the CTB-based alternative may use a syntax element that is transmitted unconditionally, i.e., regardless of whether the inter-layer offset indication is CTB-based or spatial segment-based. That is, the syntax element indicates the inter-layer offset in terms of spatial elements when the spatial segment direction indication is indicated as one component, e.g., the horizontal or vertical component, of the "trigger CTB" location in the base layer image. Depending on the syntax element that switches between CTB-based and spatial segment based on the inter-layer offset indication, further syntax elements may or may not be transmitted from it. In particular, if a CTB-based indication is indicated, it is transmitted. In that case, the latter syntax element indicates the missing dimension of the "trigger CTB" location. A decoder may thus identify the "trigger CTB" among the regular arrangement of CTBs of the base layer image in rows and columns using the two syntax elements, and as soon as this CTB is decoded, the decoder may begin decoding the enhancement layer image. Nevertheless, any inter-layer offset indication can be completely turned off by using one of the indication possible states of the first syntax element, i.e., min_spatial_segment_delay. In the case of CTB-based inter-layer offset indication, with the default decoding order defined between CTBs, the decoder may still transmit the horizontal and vertical components of the trigger CTB's position to the number of CTBs of the base layer picture that should be fully decoded before starting decoding of the first CTB of the enhancement layer picture, so that the decoder may use the embodiment of Figure 36 to control according to the CTB-based indication of inter-layer offset.

[0170]

[0010] In the following, further aspects of the present invention are described in more detail. A fourth aspect concerns the problem that all participating network entities receiving a bitstream from an encoder are preferably enabled to easily distinguish between the various layers conveyed in a multi-layer data stream. For example, an intermediate network entity may be interested in excluding certain information layers from further transmission, e.g., layers relating to sample resolutions above some resolution threshold. The following description provides an overview of the current state of the envisioned extensions to HEVC.

[0171] The Video Parameter Set (VPS) of HEVC [1] provides high-level access to the coded bitstream and contains information that is essential for processing bitstreams at intermediate or end devices. The upcoming scalable and multiview extensions of HEVC will further benefit from the VPS extension, which provides syntax designed for scalable bitstreams. One of the main tasks of the VPS extension is to provide a unified way to interpret nuh_reserved_zero_6bits in NAL unit headers. nuh_reserved_zero_6bits, renamed as layer_id, is intended to serve as a general layer identifier for scalable video coding scenarios. The layer_id syntax element in the NAL unit header is given in [1] and is given in the following table, along with the NAL unit header as shown in Figure 22.

[0172] Two general approaches were considered during the design process [5]: first, to map the value of a single identifier in the header of a NAL unit to potentially multiple scalable identifiers indicated in the VPS extension; and second, to assign individual bits (or blocks of bits) of a single identifier in the NAL header to specific scalable identifiers indicated in the VPS extension.

[0173] As reported in [4], the current VPS extension syntax design uses the mapping method but already includes all the syntax elements required for both methods, namely, two syntax elements indicating the scalability type (compare scalability_map) and the number of layers per scalability dimension (dimension_id_len_minus1).

[0174] If the encoder chooses to sparsely assign non-contiguous layer_id values, the mapping method introduces additional syntax elements into the VPS extension syntax, namely the effective value of the scalable identifier coded as u(v) and optionally layer_id.

[0175] In many highly likely scalable scenarios, e.g., two or three spatial layers, two or three views, etc., it is not necessary to utilize all 63 values ​​of the 6-bit layer identifier in the NAL unit header. For these scalable scenarios, the method of assigning individual bits of the layer identifier in the NAL unit header to specific scalable dimensions has two advantages when compared to the mapping-based method: · The interpretation of the layer identifier value in the NAL unit header in terms of the scalability dimension identifier does not require indirection or lookup. The VSP extension syntax elements required for the mapping method do not need to be transmitted, which constitutes a significant part of the VSP extension bits for scalability signaling. Intermediate devices do not need to store mapping tables for each video bitstream that passes through them.

[0176] According to the concept of the fourth aspect described below, there may be hints in the high-level syntax of HEVC that indicate which mapping or partitioning method to use.

[0177] According to one embodiment, depending on the hint, the mapping parts associated with the syntax elements (compare vps_nuh_layer_id_present_flag, layer_id_in_nuh[i] and dimension_id[i][j]) are transmitted or ignored, and the syntax elements for the scalability type (compare scalability_mask) and the number of layers per scalability (compare cp.dimension_id_len_minus1) are indicated and must be interpreted according to the hint as information about partitioning or as a mapping of scalable identifiers in the NAL unit header.

[0178] An embodiment corresponding to or utilizing the concepts of the fourth aspect of the present invention is illustrated with reference to FIG. 23. FIG. 23 illustrates a network entity that may be a video decoder, such as any one of those already described above, or may be an intermediate network entity occurring between an encoder and a decoder. The network entity is generally indicated with reference numeral 680. It is for processing a multi-layer video data stream 682, such as any one of data streams 40 described above. If network entity 680 is a video decoder, the processing includes decoding multi-layer video data stream 682. In the case of an intermediate network entity, the processing may include, for example, forwarding the video data stream.

[0179] A scene is encoded into a multi-layer video data stream in layers, such that in each layer, this scene is encoded at a different operational point in a scalability space measured by a scalability dimension, where the multi-layer video data stream includes first NAL units, each associated with one of the layers, and second NAL units embedded within the first NAL unit to indicate general information about the multi-layer video data stream. In other words, the first NAL unit 684 may carry one or more slices of a video image, along with an "image" corresponding to any of the layers of the multi-layer video data stream 682. In the above example, for simplicity of explanation, only two layers, layer 0 and layer 1, are described. Naturally, the number of layers may be greater than two, and even the type of information that a layer contributes to any of the previous layers may differ from layer to layer. In addition to the first NAL unit 684, NAL unit 686 indicates that it is incorporated into NAL unit 684, but their transmission may be performed via a separate channel compared to the first NAL unit 684. The second NAL unit indicates general information about the multi-layer video data stream in a manner that will be described in more detail below.

[0180] To describe the association between the first NAL unit 684, on the one hand, and the set of layers of the data stream 682, on the other hand, reference is made in more detail to Figure 24. Figure 24 shows the first NAL unit 684 as representative for all first NAL units 684, and includes, within a header 688, a layer indicator field 690. Besides the header 688, the NAL unit 684 includes payload data 692 related to slice data, as described above with respect to other figures, i.e., data related to video content coded using inter-layer prediction. Figure 24 also shows a set of layers, i.e., 694. In particular, the set of layers 694 shown in Figure 24 refers to all of the possible layers that can be represented and distinguished from each other via the layer indicator field 690 of the NAL unit 684. That is, the association function between the set 694, on the one hand, and the possible values ​​of the layer indicator field 690 is considered to be bijective. In Figure 24, the individual layers of the set 694 are illustratively shown using small circles. and each with a certain number is inscribed therein. Note that although these inscribed numbers suggest an order to be defined among the layers of set 694, the following description makes clear that the way in which the layers in set 694 are arranged or sorted is not derived solely on the basis of layer indicator field 690. Rather, this requires network entity 680 to examine type indicator field 696 in distributed second NAL unit 686. However, this is discussed below.

[0181] In other words, up to now, each element of set 694 in FIG. 24 merely represents one of the possible states of layer indicator field 690 in NAL unit 684. The layers of set 694 are identifiable by layer indicator field 690, but the semantic meaning and order between these layers is unclear to network entity 680 without the additional information provided by second NAL unit 686. Nevertheless, in practice, the layers of set 694 form tree nodes with branches between the trees corresponding to specific scalability dimensions or axes. One of the layers is, for example, the base layer, and corresponds to the root of the tree. Each branch connecting two layers, i.e., two nodes of the tree, describes how a particular layer contributes to the other layer, i.e., what kind of information is additionally provided using inter-layer prediction. The type of information corresponds to the scalability dimension and can be, for example, spatial resolution increase, SNR increase, etc. For simplicity, Figure 24 shows a two-dimensional scalability space 698 passed by two scalability dimensions 700 and 702, corresponding, for example, to spatial scalability and SNR scalability. A typical tree of layers and their extension by space 698 is shown in Figure 24. Figure 24 also shows that not all of the possible layers of set 694 can be present in data stream 682. In the typical case of Figure 24, for example, only five layers are actually used in data stream 682.

[0182] FIG. 22 shows, for example, that the layer indicator field can have 6 bits. 624 using reference numeral 704. An "operational point" refers to the position of at least one layer actually present in set 694 within scalability space 698. For example, the origin of scalability space 698 may relate to the base layer or the tree root, while each branch along either of axes 700 and 702 of scalability space 698 may have a fixed length of 1. A vector indicating an operational point of scalability space 698 may thus have coordinates with integer values.

[0183] To briefly summarize the discussion so far, the multi-layer video data stream 682 provides information about video content or scenes in multiple layers. The layers are arranged in a tree with each layer connected to the tree via a branch. Starting with a base layer forming the tree root, each subsequent layer contributes to a reconstructable version of the video content information with respect to one specific type of information that can be interpreted as a scalability dimension. Thus, each layer is either the tree root layer or is connected to the latter via a specific path of branches, and NAL units 684 belonging to layers present along this path are necessary to reconstruct the video content at each layer. Of course, it is preferable if the mapping 704 is performed so that any "contributing" layer proceeding at the end of each branch relative to the tree root has a layer indicator field 690 value that is higher than the value of the layer indicator field of the layer at the proximal end of the respective branch.

[0184] Figure 25 illustrates the meaning of the type indicator field 696 in more detail. Figure 25 illustrates the layer indicator field 690 as a field of fixed bit length. In any case, the length of the field 690 is independent of the value of the type indicator field 696. However, if the type indicator field has a first state, the layer indicator field 690 is treated as a whole, i.e., all of its n bits are treated as common to distinguish its possible values. When the type indicator field 696 assumes the first state, an n-bit integer is preferably derived from the layer indicator field 690 by the network entity 680. In the case of the type indicator field 696 assuming the first state, the network entity 680 performs a mapping 704 of the possible values ​​of the m-bit field 690 onto operational positions using mapping information conveyed within the second NAL unit 686, and the mapping information is indicated in Figure 25 using reference numeral 708. 25, the mapping information 708 may include, for example, a table assigning to each actually used possible value of the layer indicator field 690 a vector indicating the associated operating point of each possible value. That is, in the case of the type indicator field 696 assuming the first state, the network entity 680 may derive the mapping information 708 from the second NAL unit 686 and may perform a lookup into the mapping information or table 708 for each layer indicator field 690 to retrieve the associated vector to determine the respective / associated operating point in the space 698. The number p of dimensions of the vectors associated with the possible values ​​in the mapping information 708 may be set by default or may be indicated in the data stream, for example, in the second NAL unit 686. It will be shown later that the following information may be indicated to convey information about the mapping information 708: vps_max_layers_minus1 may determine the number M of actually used possible values ​​of the M-bit field 690; num_dimensions_minus1 may determine the number of dimensions.The latter two variables can be transmitted using the syntax element scalability_mask. The table itself can then be indicated on the one hand via a pair of possible values, i.e., layer_id_in_nuh, and by a p-dimensional vector, i.e., dimension_id[i][j]. A mapping 704 is then mapped via mapping information 708 onto this respective vector 710 indicating the operational points between the p-dimensional vectors, i.e., the space 698 associated with the layers of the NAL unit having the layer indicator field 690. It is related to things.

[0185] However, if the type indicator field 696 assumes a second state, the mapping 704 is performed differently. In particular, the mapping is then performed by dividing the layer indicator field 690 into multiple parts, i.e., by dividing the m-bit sequence of the field 690 into a number n of consecutive subsequences of bits. Each part thus obtained is assigned a coordinate x1...x2 of the n-dimensional vector 712. n , which in turn indicate operational points in the scalability space 698 associated with the layers of the NAL unit that constitute the layer indicator field 690. For example, the m-bits of the layer indicator field 690 may be divided into a first part that contributes to the first m1 (most significant) bits of the field 690, a second part that contributes to the next m2 (next most significant) bits of the field 690, m1+...+m n = m. The Th bits of each part can be directly considered as an integer value representation.

[0186] For type indicator fields assuming the second state, the number of dimensions n may be set by default or may be indicated by the data stream. In the specific embodiment described later herein, n is derived based on the same syntax elements used to determine p for type indicator field 696 assuming the first state, i.e., based on scalability_mask. The bit lengths of the portions into which layer indicator field 690 is subdivided, i.e., m1, ..., m n is also exemplarily shown via the syntax element dimension_id_len_minus1, but subdivision can also be achieved by default without explicit transmission.

[0187] It should be noted that the same syntax constructs, e.g., scalability_mask, can be parsed regardless of the type indicator field 696 and indicate the dimensionality and semantic meaning of the scalability space. The allowed states of this syntax element may be restricted—associated with the mapping case—on a specific subset of the available states of the syntax element in the case of the type indicator field 696 that indicates the component-direction interpretation of the layer indicator field. However, for example, the maximum number of dimensions of the scalability space available (or configurable) in the mapping case (first state of the type indicator) may be greater than the maximum number available in the component interpretation case (first state of the type indicator). The encoder therefore adheres to this restriction.

[0188] A typical use case for the embodiment of Figures 23-25 ​​is as follows:

[0189] Holding a large number of meetings with multiple stakeholders In high-volume conversational service scenarios based on RTP, e.g., web conferencing, the video transport between multiple participants is handled by a multipoint control unit (MCU), which must know the parameter sets of each video bitstream. Each participant provides a thumbnail bitstream and two bitstreams with the speaker's enhanced spatial resolution, e.g., 720p and 4K. The MCU makes the decision of which stream to give to which participant. A simple analysis of scalability parameters is therefore a significant relief for the MCU. The partitioning-based method requires fewer computational and memory resources compared to the mapping-based approach of scalability signal transmission.

[0190] Transport System Transport systems, e.g., RTP or MPEG2-TS, that map scalability-related codec information to respective elements may benefit from lower complexity and bit-saving mechanisms such as partitioning compared to mapping-based methods. It is unlikely that a transport system would adopt scalability signaling in the manner of a mapping approach, but would rather break down the mapping indirection to yield dedicated scalability identifiers for each scalability dimension that are explicitly indicated one after the other in a partitioning approach.

[0191] The example of Figure 26 shows a possible implementation as a syntax flag in the VPS extension syntax that allows switching between mapping and partitioning based methods for scalability signaling in HEVC extensions, based on the VPS extension syntax, as reported in [4].

[0192] nuh_layer_id_present_flag, layer_id_in_nuh[i], and dimension_id[i][j] are not indicated. Dedicated_scalability_ids_flag with a value equal to 1 indicates that the bits of the layer_id field in the NAL unit header are split according to the value of dimension_id_len_minus1 and belong to each scalability dimension indicated in scalability_mask. Dedicated_scalability_ids_flag with a value equal to 1 indicates that the syntax elements vps_nuh_layer_id_present_flag, layer_id_in_nuh[i], and dimension_id[i][j] are not indicated. Dedicated_scalability_ids_flag with a value equal to 1 indicates that the derivation of the variables describing the scalable identifiers of the corresponding NAL unit uses only the respective relevant bits of the scalable identifiers of the NAL unit header (compare layer_id). For example:

[0193] DependencyId=layer_id&&0x07

[0194] QualityId=layer_id&&0x38

[0195] }} A dedicated_scalability_ids_flag with a value equal to 0 indicates that the syntax elements vps_nuh_layer_id_present_flag, layer_id_in_nuh[i] and dimension_id[i][j] are indicated and that the bits of layer_id in the NAL unit header are not associated with a specific scalability dimension but are mapped to a scalable identifier in the VPS extension. A dedicated_scalability_ids_flag with a value equal to 0 indicates that the derivation of the variables describing the scalable identifier corresponding to the NAL unit uses the syntax elements dimension_id[i][j] given the indicated layer_id in the NAL unit header, for example in the following way:

[0196] if(layer_id==layer_id_in_nuh[0]){

[0197] DependencyId=dimension_id〔0〕〔0〕

[0198] QualityId=dimension_id〔0〕〔1〕

[0199] When dedicated_scalability_ids_flag is equal to 0, dimension_id_len_minus1[i] indicates the length of dimension_id[i][j]. When dedicated_scalability_ids_flag is equal to 1, dimension_id_len_minus1[i] indicates the following:

[0200] The number of bits of layer_id in the NAL unit header associated with the ith scalability dimension, as indicated by scalability_mask

[0201] The flow chart in Figure 27 further illustrates a possible embodiment of the present invention. The scalable identifier is either derived directly from the bits of layer_id by masked bit copying or is represented in the VPS by association to a specific value of layer_id.

[0202] Another syntax example is shown in Figure 31. Here, the submitted type indicator is indicated by "splitting_flag", while the submitted layer indicator is called nuh_layer_id. Depending on "splitting_flag", a mapping or splitting concept is used to derive the operating point of the layer of each NAL unit in the scalability space from nuh_layer_id. The splitting concept is exemplarily indicated by splitting_flag equal to 1. Then, the scalable identifier, i.e., the vector component related to the scalability dimension of the scalability space, can be derived from the nuh_layer_id syntax element in the NAL unit header by bitmasking it. The respective bitmask for the ith scalable dimension, i.e., the ith component of vector 712 in Figure 25, is defined as follows: In particular, splitting_flag equal to 1 indicates that the dimension_id[i][j] syntax element is not present (i.e., mapping information 708 is not present) and the binary representation of the nuh_layer_id value in the NAL unit header is split into a series of NumScalabilityTypes, i.e., a vector x1 with length n segments according to the value of dimension_id_len_minus1[j]... n , in bits and according to the value of dimension_id[LayerIdxInVps[nuh_layer_id]][j], that is, the components x1... of the vector 712... nindicates that the number of scalability axes of the scalability space is inferred from the NumScalabilityTypes segment of field 690. The semantic meaning and the number of scalability axes of the scalability space are indicated by a series of flags scalability_mask_flag that indicate each of this exemplary predetermined constants as to whether the respective scalability type belongs to any of the scalability dimensions of the respective scalability space 698. In particular, the network entity 680 can derive the scalability space, i.e., the semantic meaning and the number of scalability axes NumScalabilityTypes, from the sequence of flags scalability_mask_flag by the for loop in FIG.

[0203] for( i = 0, NumScalabilityTypes = 0; i < 16; i++ ) [ scalability_mask_flag[ i ] NumScalabilityTypes += scalability_mask_flag[ i ] ]

[0204] scalability_mask_flag[i] equal to 1 indicates that the i-th scalability dimension is present, and scalability_mask_flag[i] equal to 0 indicates that the i-th scalability dimension is absent, where i=1 may mean multiview scalability, i=2 may mean spatial / SNR scalability, and i=0 may mean the addition of depth map information. Other scalability dimension types may exist as well, and of course the just outlined example is merely illustrative. If the length of nuh_layer_id is illustratively 6, the splitting or masked copies may be done as follows:

[0205] The variable dimBitOffset[0] is set equal to 0, and for j in the range from 1 to NumScalabilityTypes-1 inclusive, dimBitOffset[j] is derived as follows:

[0206] JPEG2025179048000003.jpg19150

[0207] The value of -dimension_id_len_minus1[NumScalabilityTypes-1] is inferred to be equal to 5-dimBitOffset[NumScalabilityTypes-1]. The value of -dimBitOffset[NumScalabilityTypes-1] is set equal to 6.

[0208] If NumScalabilityTypes is greater than 0, dimBitOffset[NumScalabilityTypes-1] must be less than 6 for bitstream matching to occur.

[0209] For j from 0 to NumScalabilityTypes-1, dimension_id[i][j] is ((nuh_layer_id&((1<<dimBitOffset[j+1])-1))> > dimBitOffset[j]).

[0210] In summary, the following syntax elements are concerned with relating each first NAL unit with its operational point in the scalability space:

[0211] 1) nuh_layer_id, i.e., the layer indicator field 690 2) scalability_mask_flag, i.e. the number and meaning of the scalability axes 700, 702 in the space 698, and therefore the part x of the field i The sequence of information revealing the number n of 3) dimension_id_len_minus1, i.e., each part of field 690 for each of the axes x i bit length, (all but one, because the remaining one can be assumed to include all bits 706 of field 690.

[0212] JPEG2025179048000004.jpg34170

[0213] If splitting_flag is equal to 0, the mapping concept is used. The mapping information 708 is then transmitted using the following information:

[0214] JPEG2025179048000005.jpg72165

[0215] Thus, when splitting_flag is equal to 0, vector 710 is inherently derived by splitting without explicit signaling, i.e., without signaling the dimension_id[i][j] which is instead inferred.

[0216] Thus, according to the fourth aspect, i.e., the concept of "switchable interpretation of NAL unit layer identifiers," bitstream 40 may contain NAL units, i.e., VPS NAL units, that may be configured with a type indicator field 300 that can be switched between a mapping concept and a bit-segmentation concept to interpret a layer indicator field 302 in a "normal" NAL unit. In this way, the same bit positions of field 302 are used in both modes, but the interpretation of the bits and the interpretation of the provisioning information indicated, i.e., the mapping information or segmentation and semantic information transferred in the bitstream, changes between the modes. Although this requires the additional transmission of a type indicator field, this concept leads to more efficient bitstream transmission entirely as an effect of the mapping concept, and the bit-segmentation concept may be utilized as needed, since both concepts are differently suited to, e.g., different multi-layer data, depending on the number of layers, etc.

[0217] A fifth aspect of the present application relates to a multi-standard multi-layer video decoder interface. The concepts described below describe an interface between a transport layer decoder (such as an MPEG transport stream or RTP) and a scalable video decoder that supports various video coding standards at different layers (e.g., H.264 / AVC at the base layer and HEVC at the enhancement layer).

[0218] A scalable video bitstream consists of multiple layers: it consists of one base layer, which is an independently decodable video signal, and one or more enhancement layers that can only be decoded in combination with the base layer (and potentially other enhancement layers) to provide higher temporal resolution (temporal scalability), spatial resolution (spatial scalability), quality (SNR scalability), greater bit depth (bit depth scalability), or video signals or other camera views (multiview scalability).

[0219] Existing scalable video coding standards, such as H.264 / AVC SVC, define base and enhancement layers of the same standard, which are designed in such a way that a scalable bitstream has the same basic format as a non-scalable bitstream. When a scalable bitstream is input to a non-scalable decoder, it can still detect packet types and discard unknown packets.

[0220] HEVC is the first video coding standard that allows using various video coding standards for the base layer (e.g., H.264 / AVC). The packet formats for both standards are different, so a base layer decoder cannot understand the enhancement layer packets. On the other hand, an enhancement layer decoder can understand the enhancement layer packet format but not the base layer packet format.

[0221] In an audio / video system, a transport layer is used to combine several audio and video streams and provide metadata such as timing and stream type.

[0222] In existing multi-layer transport layer decoders, the access units of the base and enhancement layers are multiplexed into a single video data stream (e.g., Annex B H.264 / AVC byte stream), which is the input to the video decoder.

[0223] If different video coding standards are used for the base and enhancement layers, the base and enhancement layer packets may not be arranged into a single bitstream in all cases.

[0224] According to an embodiment of the fifth aspect, the transport layer decoder distinguishes between the following cases:

[0225] 1. A video decoder reading the output can only decode the base layer. 2. A video decoder reading the output can decode the base and enhancement layers, and the base and enhancement layers are coded using the same video coding standard. 3. A video decoder reading the output may decode the base and enhancement layers, and the base and enhancement layers are coded using different video coding standards.

[0226] In case 1, the transport layer decoder uses the following behavior:

[0227] Only packets containing the base layer are extracted from the transport layer and input to a single-standard single-layer video decoder in a format specified in the video coding standard.

[0228] A specific embodiment is a transport stream decoder that extracts H.264 / AVC NAL units from an MPEG-2 transport stream by selecting only streams with the stream type "AVC video stream conforming to one or more profiles defined in Annex A of ITU-T Rec. H.264 | ISO / IEC 14496-10 or AVC video sub-bit stream" assigned to and input into the H.264 / AVC video decoder in the byte stream format defined in Annex B of the H.264 / AVC specification. NAL units belonging to streams whose stream type is not equal to the stream type "AVC video stream conforming to one or more profiles defined in Annex A of ITU-T Rec. H.264 | ISO / IEC 14496-10 or AVC video sub-bit stream" are discarded within the transport stream decoder. Figure 28 provides an illustration of a specific embodiment.

[0229] In case 2, the transport layer decoder uses the following behavior:

[0230] Packets from the base and enhancement layers are extracted from the transport layer and input to a single-standard multi-layer video decoder in a format specified in the video coding standard.

[0231] A particular embodiment may extract H.264 / AVC NAL units from an MPEG-2 transport stream by selecting a base layer stream having a stream type assigned "AVC video stream conforming to one or more profiles defined in Annex A of ITU-T Rec. H.264 | ISO / IEC 14496-10 or AVC video sub-bit stream" and also one or more enhancement layer streams having a stream type assigned "SVC video sub-bit stream of an AVC video stream conforming to one or more profiles defined in Annex G of ITU-T Rec. H.264 | ISO / IEC 14496-10." The NAL units of the various layers are multiplexed into a byte stream format defined in Annex B of the H.264 / AVC specification and input to an H.264 / AVC SVC video decoder. Figure 29 provides an illustration of a particular embodiment.

[0232] In case 3, the transport layer decoder uses the following behavior:

[0233] Packets from the base and enhancement layers are extracted from the transport layer and passed to a multi-standard multi-layer video decoder in one of the ways described in the following sections.

[0234] Interface A If the enhancement layer standard packet format allows for carrying base layer packets, the base layer packets are encapsulated in the enhancement layer format, which allows a header to be added to each base layer packet that can be understood by an enhancement layer standard video decoder and allows the enhancement layer video decoder to identify the package as a base layer packet of various video encoding standards.

[0235] In a specific embodiment of the present invention, an H.264 / AVC NAL unit is used as the payload of an HEVC NAL unit. That is, an HEVC NAL unit header is added before the H.264 / AVC NAL unit header. A field in the HEVC NAL unit header (e.g., nal_unit_type) is used to identify the payload as an H.264 / AVC NAL unit. The bitstream can be input to a video decoder in the HEVC Annex B byte stream format. Figure 30 provides an illustration of a specific embodiment.

[0236] Interface B A different channel is used for each layer of the scalable bitstream. The video coding standard is determined at the decoder by the channel selection.

[0237] In a particular embodiment of the present invention for two layers, the two layers are delivered in two separate channels: the first channel is used exclusively for H.264 / AVC base layer packets (or Annex B byte stream), while the second channel is used exclusively for HEVC enhancement layer packets.

[0238] Interface C A metadata field indicating the type of video encoding standard is associated with each packet that passes from the transport stream decoder to the multi-standard multi-layer video decoder. Other metadata, such as timing, may be indicated as well.

[0239] In a particular embodiment, each base layer NAL unit is identified as an H.264 / AVC NAL unit according to the associated metadata fields, and each enhancement layer NAL unit is identified as an HEVC NAL unit according to the associated metadata fields.

[0240] Thus, the concept of the fifth aspect can be described as a "multi-standard multi-layer video decoder interface" that provides possibilities regarding how to combine bitstreams of various codecs.

[0241] Thus, according to a fifth aspect of the present application, a transport layer decoder may be configured as outlined below with respect to Figure 44. The transport layer decoder shown in Figure 44 is generally indicated using reference numeral 770. The transport layer decoder 770 is configured to cause an inbound multi-layer video data stream 40, in which scenes are coded in layers, to be decoded by a multi-standard multi-layer decoder 772 to which an output interface of the transport layer decoder 770 is connected. The multi-layer video data stream 40 is composed of NAL units as already described above, although the description may move on to the example of Figure 44 with respect to various other aspects of the application. Each NAL unit is associated with one of the layers. The layers are associated with different codecs, i.e., different standards. For each layer, the NAL units associated with the respective layer are coded using the same codec, i.e., the one with which the respective layer is associated.

[0242] The transport layer decoder 770 is configured to identify, for each NAL unit, which codec it is associated with, and passes the NAL units of the multi-layer video data stream 40 to a multi-standard multi-layer decoder that uses inter-layer prediction between layers associated with the various codecs to decode the multi-layer video data stream.

[0243] As mentioned above, as already outlined above with respect to the fourth aspect of the present application, each NAL unit may be associated with one of the layers of the multi-layer video data stream 40 via a specific layer indicator field. Some, or most, of the NAL units may carry content-related data, i.e., one or more slices. By collecting all NAL units for a particular set of layers, the video content or scene encoded in the data stream 40 can be decoded by the decoder 772 with the amount of information provided by this set of layers. Regarding layer dependency, options with more than two scalability dimensions, etc., reference is made to the description of the fourth aspect of the present application.

[0244] The multi-layer multi-standard decoder 772 may handle various codecs / standards. Examples for various standards are shown above, i.e., H.264, HEVC, but other standards may be mixed as well. Different codecs / standards are not limited to hybrid codecs. Rather, a mixture of different types of codecs may be used as well. The inter-layer prediction used by the multi-layer multi-standard decoder 772 may relate to prediction parameters used in different layers or may refer to image samples of various time-aligned layers. This was already described above with respect to other aspects and examples.

[0245] The transport layer decoder 770 may be configured to perform handing among NAL units belonging to layers of codecs that only the multi-layer multi-standard decoder 772 can address. That is, the handing performed by the transport layer decoder 770 may depend on the transport layer decoder 770's identification of the codec with which each NAL unit is associated. In particular, the transport layer decoder 770 may perform the following for each NAL unit:

[0246] Identify the layer to which the currently examined NAL unit is associated, for example by examining the layer indicator field in the NAL unit header of the NAL unit. Based on the association between the layers of data stream 40 and their codecs / standards derived by transport layer decoder 770 based on an examination of the respective high-level syntax of data stream 40, on the one hand, transport layer decoder 40 determines whether the currently examined NAL unit layer satisfies two criteria: the NAL unit layer belongs to a subset of layers that should be forwarded to decoder 772, which subset is determined by the currently examined NAL unit layer's operating point in the scalability space, and the external prescription as to which operating points in the scalability space can be forwarded to multi-layer multi-standard decoder 772 and which cannot. Furthermore, transport layer decoder 770 checks whether the currently examined NAL unit layer codec belongs to the set of codecs / standards that multi-layer multi-standard decoder 772 can handle. If the check shows that both criteria are satisfied by the currently examined NAL unit, the transport layer decoder 770 forwards the current NAL unit to the decoder 772 for decoding.

[0247] Various possibilities exist for the transport layer decoder 770 to determine the above-mentioned association between the layers included in data stream 40, on the one hand, and their underlying codecs / standards, on the other hand. For example, as explained above with respect to "interface B," various channels may be used to convey data stream 40, i.e., NAL units of layers of one codec / standard on one channel and NAL units of layers coded according to another codec / standard on another channel. In this manner, the transport layer decoder 770 may derive the just-mentioned association between layers, on the one hand, and codecs / standards, on the other, by distinguishing between the various channels. For example, the transport layer decoder 770 determines, for each NAL unit of data stream 40, the channel on which the respective NAL unit arrives in order to ascertain the codec / standard to which the respective NAL unit or layer of the respective NAL unit is associated.

[0248] Additionally or alternatively, the transport layer decoder 770 may forward NAL units of layers belonging to different codecs / standards to the multi-layer multi-standard decoder 772 in a manner dependent on the respective codecs / standards, such that NAL units of layers belonging to one codec / standard are sent on one channel to the decoder 772 and NAL units of layers of different codecs / standards are forwarded to the multi-layer multi-standard decoder 772 on other channels.

[0249] The "different channels" may be provided by the underlying transport layer, i.e., as provided by the transport layer below, which is not shown in Figure 44 for ease of understanding, and differentiation between various channels may be achieved by distinguishing between different channel identifiers.

[0250] Another possibility for passing data stream 40 to multi-layer multi-standard decoder 772 is for transport layer decoder 770 to encapsulate those NAL units identified as being associated with a layer associated with any codec different from the predetermined codec using NAL unit headers of a predetermined codec with NAL unit type indicators set to indicate the codec of the respective layer. This means that the predetermined codec can be, for example, any codec of any enhancement layer of data stream 40. The base layer codec, i.e., the codec associated with the base layer of data stream 40, can be different from the predetermined codec, for example, HEVC. Thus, when passing data stream 40 to multi-layer multi-standard decoder 772, transport layer decoder 770 can convert data stream 40 into a data stream that conforms to the predetermined codec. To this end, the transport layer decoder 770 encapsulates each NAL unit belonging to a layer that is not coded using a given codec using a NAL unit header of the given codec and sets the NAL unit type indicator within that NAL unit header to indicate the codec of the respective layer. For example, a base layer NAL unit may be H.264, and the base layer NAL unit is therefore encapsulated using HEVC and a NAL unit header with the NAL unit type indicator set to indicate H.264. The multi-layer multi-standard 772 then receives the HEVC compliant data stream.

[0251] Of course, as described for interface C, it is alternatively possible for transport layer decoder 770 to provide metadata to each NAL unit of inbound data stream 40 indicating the codec to which the layer to which the respective NAL unit is associated is associated. The 40 NAL units of the data stream are thus sent to decoder 772 in this expanded manner.

[0252] Using the variant just described, however, it is feasible to extend the content encoded in the data stream with further layers encoded using other codecs, e.g. newer codecs, and the existing parts of codec 40 do not need to be modified. A multi-layer multi-standard decoder that can in turn handle newer codecs, i.e. newly added ones, can process this kind of mixture of existing data streams with layers encoded using newer codecs.

[0253] Thus, a parallel / low delay video coding concept for HEVC scalable bitstreams has been presented above.

[0254] The High Efficiency Video Coding (HEVC) standard [1] initially features two dedicated parallelization tools that enable parallel processing on the encoder and decoder side: tile and wavefront parallelism (WPP). These tools enable parallelization within a picture with the goal of improving processing time while minimizing coding efficiency losses when compared to HEVC-coded video that does not feature parallel processing within individual pictures.

[0255] In a scalable [2] or multiview [3] HEVC bitstream, the decoded base layer or base view images are used to predict the enhancement layer or dependent view images. In the above description, the term terminal layer is also used to cover the concept of view at the same time.

[0256] The above embodiments describe a scalable video decoder that can start decoding an enhancement layer image before the decoding of the associated base layer image is finished. The decoding of image areas is pipelined based on the high-level parallelization tools used in each layer. The base and enhancement layer decoders can operate in parallel with each other and within the actual layer. The degree of parallelism within each layer can vary between the base and enhancement layers. Furthermore, it has been described that signaling is required to properly prepare the parallel decoding environment for a particular bitstream.

[0257] As a general remark, please note the following: the above-described embodiments describe decoders and correspondingly designed encoders according to various aspects. As far as all these aspects are concerned, please note that the decoders and encoders all have in common that they may support WPP and / or tile parallelism. Therefore, wherever details are described therein, these details are treated as simultaneously applying to any other aspects, and corresponding descriptions leading to novel embodiments of these other aspects or supplementing descriptions of embodiments of these other aspects are considered to simultaneously apply to any of the other aspects, regardless of whether the respective aspects to which the respective descriptions are transferred are described using the terms "portion," "spatial segment," etc., instead of "tile / substream" to more generally mean parallel-processable segments of an image. The same applies in detail to coding / prediction parameters and descriptions describing possible ways of subdividing an image: all aspects may be implemented to result in a decoder / encoder that uses subdivision into LCUs / CTBs for determining tiles and / or substreams in units of the same. Additionally, in any of these aspects, the LCUs / CTBs may be further subdivided into coding blocks by using recursive multi-tree subdivision, as described above with respect to a subset of the aspects and examples thereof, and, in addition or alternatively, from those aspects with respect to which the relationship between slices and substreams / tiles is described, the slice concept may be employed by examples of all aspects.

[0258] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent descriptions of corresponding methods, where a block or apparatus corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent descriptions of a corresponding block or item or feature of an apparatus. Some or all of the method steps may be performed by (or used in) a hardware apparatus, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of some of the most important method steps may be performed by such an apparatus.

[0259] Depending on specific implementation requirements, embodiments of the present invention may be implemented in hardware or in software. Implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, having electronically readable control signals stored thereon, which cooperates (or may cooperate) with a programmable computer system so that the respective methods are executed. Thus, the digital storage medium may be computer-readable.

[0260] Some embodiments according to the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0261] Typically, embodiments of the present invention may be realized as a computer program product having program code thereon, the program code being operable to perform one of the methods when the computer program product runs on a computer, the program code being for example stored on a machine-readable carrier.

[0262] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0263] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0264] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium or computer-readable medium) comprising, recorded on it, a computer program for performing one of the methods described herein. The data carrier, digital storage medium or recorded medium is typically tangible and / or non-transitory.

[0265] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be arranged to be transmitted via a data communication connection, for example via the Internet.

[0266] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0267] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0268] Further embodiments according to the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0269] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0270] The devices described herein may be implemented using hardware devices, or using computers, or using a combination of hardware devices and computers.

[0271] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0272] The above-described embodiments merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details shown in the specification and description of the embodiments herein.

Claims

1. 1. A video decoder for decoding a multi-layer video data stream (40) in which a scene is coded in a hierarchy of layers using inter-layer prediction from a first layer to a second layer, the video decoder supporting parallel decoding of the multi-layer video data stream in spatial segments (80) into which images (12, 15) of the layers are subdivided, the video decoder comprising: inspecting the long-term syntax element structure (606; e.g., tile_boundaries_aligned_flag) of said multi-layer video data stream for: interpreting the long-term syntax element structure assuming a value from a first set of possible values ​​(e.g., tile_boundaries_aligned_flag=1) as a guarantee that the image (608) of the second layer will be subdivided during the predetermined period (608) so that boundaries between the spatial segments of the image of the second layer cover each boundary of the spatial segments of the image (12) of the first layer, and periodically determining the subdivision of the images of the first and second layers into the spatial segments based on short-term syntax elements (e.g., 602; column_width_minus1[i] and column_width_minus1[i]) of the multi-layer video data stream at time intervals (604) shorter than the predetermined period; a video decoder configured to periodically determine, at a time interval shorter than the predetermined period, the subdivision of the images of the layers from the short-term syntax elements of the multi-layer video data stream into the spatial segments such that, when the long-term syntax element structure assumes values ​​from a second set of possible values ​​(e.g., tile_boundaries_aligned_flag = 0), there are boundaries between the spatial segments of the images of the second layer that do not cover any of the boundaries of the spatial segments of the first layer, at least for a first possible value of the short-term syntax element, and there are boundaries between the spatial segments of the images of the second layer that cover each boundary of the spatial segments of the first layer, at least for a second possible value of the short-term syntax element.

2. decoding the images of the layer using intra-picture spatial prediction, which involves suspending the intra-picture spatial prediction for each spatial segment at the boundary of each of the spatial segments; or 2. The video decoder of claim 1, configured to decode the images of the layers using intra-picture spatial prediction by decoding the spatial segments of the images of the first layer in parallel with supporting the intra-picture spatial prediction that crosses boundaries of the spatial segments of the images of the first layer and that conforms to a decoding delay during the decoding of the spatial segments of the images of the first layer, and by decoding the spatial segments of the images of the second layer in parallel with supporting the intra-picture spatial prediction that crosses boundaries of the spatial segments of the images of the second layer and that conforms to a decoding delay during the decoding of the spatial segments of the images of the second layer.

3. supporting tile-parallel decoding of the multi-layer video data stream in tiles into which images of the layers are subdivided; The decoder interpreting a long-term syntax element assuming a value from the first set of possible values ​​as a guarantee that the image of the second layer will be subdivided for a predetermined period of time such that boundaries between the tiles of the image of the second layer cover each boundary of the tiles of the first layer, and periodically determining, at time intervals shorter than the predetermined period, tile refinements of the subdivision of the image of the second layer relative to the first layer based on the short-term syntax element; and 3. The video decoder of claim 1, configured to periodically determine, at a time interval shorter than the predetermined period, the subdivision of the multi-layer video data stream into tiles of the images of the layers from the short-term syntax elements such that, when the long-term syntax elements assume the values ​​from the second set of possible values, there are boundaries between the tiles of the image of the second layer that do not cover any of the boundaries of the tiles of the first layer, at least for a first possible value of the short-term syntax element, and there are boundaries between the tiles of the image of the second layer that cover each boundary of the tiles of the first layer, at least for a second possible value of the short-term syntax element.

4. 4. The video decoder of claim 3, wherein the video decoder is configured to decode images of a layer using intra-picture spatial prediction that interrupts intra-picture spatial prediction for each tile at a boundary of each of the tiles.

5. the decoder interprets the long-term syntax element structure assuming a value from a first possible value as a guarantee that the image of the second layer is subdivided such that, for a predetermined period of time, each spatial segment of the image of the first layer occupies exactly n spatial segments of the image of the second layer, with n depending on the value of the long-term syntax element structure; 3. The video decoder of claim 1, further comprising: a decoder configured to periodically determine the inter-layer offset within a predetermined period based on the short-term syntax elements of the multi-layer video data stream at a period shorter than the predetermined period when the long-term syntax elements are set to a value in a second set of possible values.

6. 3. The video decoder of claim 1, wherein the decoder is configured to decide whether to start checking or not to start checking for decoding the second layer of the multi-layer video data stream depending on the long-term syntax element structure assuming the value from among the first possible values.

7. 7. A video decoder according to claim 1, wherein the video decoder is a hybrid video decoder.

8. 1. A video encoder for encoding a scene into a multi-layer video data stream in a hierarchy of layers using inter-layer prediction from a first layer to a second layer such that the multi-layer video data stream is decodable in parallel in spatial segments into which images of the layers are subdivided, the encoder comprising: inserting a long-term syntax element structure (606) and short-term syntax elements (602) into the multi-layer video data stream, the short-term syntax elements defining the subdivision of the images of the first layer and the second layer into the spatial segments in the time interval; and Switching between configuring the long-term syntax element structure; and during a predetermined period (608) longer than the time interval, values ​​from a first set of possible values ​​are selected to subdivide the image of the second layer such that boundaries between the spatial segments of the image of the second layer overlap respective boundaries of the spatial segments of the first layer while setting the short-term syntax elements to a proper subset from a set of possible settings; A video encoder configured to switch, during the predetermined period, a value from a second set of possible values ​​to set the short-term syntax element to any of the set of possible settings, wherein the set of possible settings includes at least one setting whereby boundaries exist between the spatial segments of the image of the second layer that do not overlap any of the boundaries of the spatial segments of the first layer, and at least another setting whereby boundaries between the spatial segments of the image of the second layer overlap respective boundaries of the spatial segments of the first layer.

9. the video encoder suspends the intra-picture spatial prediction for each spatial segment at a boundary of the respective spatial segment and encodes the images of the layer using intra-picture spatial prediction; or 9. The video encoder of claim 8, wherein the encoder is configured to encode the images of the layer using intra-picture spatial prediction and perform entropy coding that is compatible with entropy context probabilities, by supporting the intra-picture spatial prediction across boundaries of the spatial segments of the images of the first layer and by individually initializing the entropy context probabilities for the entropy coding of the subsets of the spatial segments of the images of the first layer, or by adopting the entropy context probabilities according to an order among previous subsets of the spatial segments of the images of the first layer in the direct adaptation state, and by supporting the intra-picture spatial prediction across boundaries of the spatial segments of the images of the second layer and by individually initializing the entropy context probabilities for the entropy coding of the subsets of the spatial segments of the images of the second layer, or by adopting the entropy context probabilities according to an order among previous subsets of the spatial segments of the images of the second layer in the direct adaptation state.

10. The spatial segment is a tile, and the encoder When setting the long-term syntax element structure, values ​​from the first set of possible values ​​(tile_boundaries_aligned_flag=1) are set to a suitable subset from a set of possible settings of the short-term syntax element for a predetermined period greater than the time interval, the suitable subset being selected so as to match or refine a subdivision of the second layer image into tiles for the predetermined period; or 10. The video encoder of claim 8, wherein the short-term syntax element is configured to be set to a first possible value of the set of possible settings for at least one time interval during the predetermined period due to the presence of boundaries between the tiles of the image of the second layer that do not overlap any of the boundaries of the tiles of the first layer, and to be set to a second possible value of the set of possible settings for at least other time intervals during the predetermined period according to boundaries between which the tiles of the image of the second layer overlap each boundary of the tiles of the first layer.

11. 11. The video encoder of claim 10, wherein the video encoder is configured to encode the images of the layer using intra-picture spatial prediction, suspending the intra-picture spatial prediction for each tile at each tile boundary.

12. The encoder comprises: The long-term syntax element structure 10. The video encoder of claim 8, further comprising: a video encoder configured to set the short-term syntax elements to a suitable subset from a set of possible values ​​using the predetermined period longer than the time interval when setting the short-term syntax elements to values ​​from the first possible value; and the suitable subset being selected such that, during the predetermined period, each spatial segment of the image of the first layer consists of exactly n spatial segments of the image of the second layer, where n is dependent on the value of the long-term syntax element structure.

13. 1. A decoder for decoding a spatial scalable bitstream (40) in which images are coded in different spatial layers and, for at least one of the spatial layers, in a first spatial segment, the decoder comprising: or, such that any part of a division (622) of the upsampled reference image that depends on the first spatial segment is independent of parts of the image of the first spatial layer that are covered by any other division of the division. so that any part of the divisions (622) of the upsampled reference image is dependent on parts of the image of the first spatial layer that are covered by other divisions of divisions spatially adjacent to the respective division. A decoder configured to: upsample an image (12) of a first spatial layer to obtain an upsampled reference image and predict a second spatial layer using the upsampled reference image; and the decoder is configured to respond to a syntax element (616; e.g., independent_tile_upsampling_idc) in the spatial scalable bitstream to interpolate (620) the image of the first spatial layer depending on the syntax element.

14. The decoder of claim 13 , wherein the decoder is configured to decode different spatial layers in parallel.

15. the decoder is adapted to encode the upsampled reference picture such that any part of the upsampled reference picture that is spatially covered by any of the first spatial segments is independent from any part of the image of the first spatial layer that is covered by any other of the first spatial segments, or 15. A decoder according to claim 13 or 14, responsive to the syntax elements (616) in the spatial scalable bitstream to interpolate (620) the image of the first spatial layer in accordance with the syntax elements such that any portion of the upsampled reference picture spatially covered by any of the first spatial segments depends on portions of the image of the first spatial layer covered by any other spatial segment of the first spatial segment adjacent to the respective spatial segment.

16. Decoder according to any of claims 13 to 15, wherein the spatial scalable bitstream has the images of a second spatial layer encoded therein in the first spatial segments.

17. The decoder while suspending the intra-picture spatial prediction for each first spatial segment at a boundary of the respective first spatial segment, or 17. The decoder of claim 13, configured to perform the decoding using intra-picture spatial prediction, while supporting the intra-picture spatial prediction crossing boundaries of the first spatial segment, entropy decoding each first spatial segment using adaptation of entropy context probabilities, and while initializing the entropy context probabilities of the first spatial segment independently of the other first spatial segments, or adopting the entropy context probabilities according to an order among the first spatial segments of previous first spatial segments of the spatial segment of the image of the first layer so as to adapt up to an intermediate position of the previous first spatial segment.

18. (e.g. independent_tile_upsampling_idc=2) so that any part of the upsampled reference picture spatially covered by any of the second tiles is independent from parts of the image of the first spatial layer spatially covered by any other of the second tiles; or (e.g. independent_tile_upsampling_idc=1) such that any portion of the upsampled reference picture spatially bounded by the spatially co-located boundaries of the first and second tiles is independent from any portion of the image of the first spatial layer covered by any other portion of the upsampled reference picture spatially bounded by the spatially co-located boundaries of the first and second tiles, or (e.g., independent_tile_upsampling_idc=0) such that any portion of the upsampled reference picture spatially covered by any of the second tiles is dependent on portions of the image of the first spatial layer covered by any other of the second tiles adjacent to the respective tile.

15. A decoder according to claim 13 or 14, wherein the spatial scalable bitstream has the images of the first spatial layer encoded therein in the first spatial segment and the images of the second spatial layer encoded therein in the second spatial segment, and the decoder is responsive to the syntax elements (606) in the spatial scalable bitstream to interpolate the images of the first spatial layer depending on the syntax elements.

19. 19. A decoder according to any of claims 13 to 18, wherein the decoder is configured to fill fragments of a filter kernel used in the interpolation of the image of the first spatial layer that are covered by any of the other partitions and that project into any kind of part of the image of the first spatial layer, using an alternative rule according to which the fragments are filled independently of the respective part of the image of the first spatial layer into which the filter kernel projects, in order to achieve said independence.

20. 20. The decoder of claim 19, wherein the decoder is configured to use the substitution rule filling a fraction of the filter kernel that protrudes from an outer boundary of the image of the first spatial layer.

21. The decoder of any of claims 13 to 20, wherein the decoder is a video decoder and configured to respond to the syntax elements (606) in the spatial scalable bitstream on a per picture or per picture-sequence basis.

22. 22. A decoder according to claim 13, wherein the spatially scalable bitstream has the image of the first spatial layer encoded therein in the first spatial segment, and the spatially scalable bitstream has the image of the second spatial layer encoded in a second spatial segment, the boundary of the partition corresponding to a spatial intersection of the boundaries of the first and second spatial segments or to the boundary of the second spatial segment, and the decoder is responsive to the syntax element (606) in the spatially scalable bitstream to project, in response to the syntax element, a fragment of a filter kernel used for interpolating the image of the first spatial layer from one partition to an adjacent partition of the partition using an alternative rule whereby the fragment is filled independently of or using a respective portion of the image of the first spatial layer over which the filter kernel projects.

23. The decoder of any of claims 13 to 22, wherein the decoder is configured to decode the first and second layers in parallel using an inter-layer offset that depends on a syntax element (606).

24. 24. The decoder of claim 13 or 23, wherein the decoder is configured to place the boundaries of the partition to correspond to a logical intersection of a spatial overlap of the boundaries of the first and second spatial segments or the boundary of the second spatial segment depending on the syntax element.

25. 1. An encoder for encoding an image into a spatial scalable bitstream in various spatial layers and in a first spatial segment for at least one of the spatial layers, the encoder comprising: or such that any division of the upsampled reference image division that depends on the first spatial segment is independent of the portion of the image of the first spatial layer that is covered by any other division of the division; or such that any one of the divisions of the upsampled reference image is dependent on the portion of the image of the first spatial layer that is covered by other divisions of the divisions that are spatially adjacent to the respective division. An encoder including: upsampling an image of a first spatial layer to obtain an upsampled reference image and to predict an image of a second spatial layer using the upsampled reference image; and the encoder is configured to set and insert syntax elements (606) into a spatial scalable bitstream and interpolate the image of the first spatial layer depending on the syntax elements.

26. The encoder comprises: or, such that any part of the upsampled reference image spatially covered by any of the first spatial segments is independent of any part of the image of the first spatial layer covered by any other of the first spatial segments; such that any portion of the upsampled reference picture spatially covered by any of the first spatial segments is adjacent to the respective first spatial segment, dependent on any portion of the image of the first spatial layer covered by any other spatial segment of the first spatial segment; 26. The decoder of claim 25, configured to set and insert the syntax elements into a spatial scalable bitstream and to interpolate the images of the first spatial layer according to the syntax elements.

27. 27. The encoder of claim 25 or 26, wherein the encoder is configured to encode the first spatial layer image into the spatial scalable bitstream in the first spatial segment.

28. 28. The encoder of claim 27, wherein the encoder is configured to encode the images of the first spatial layer using intra-picture spatial prediction, the intra-picture spatial prediction being suspended for each first spatial segment at a boundary of the respective first spatial segment.

29. or, such that any part of the upsampled reference image spatially covered by any of the second spatial segments is independent from any part of the image of the first spatial layer spatially covered by any other of the second spatial segments. any portion of the upsampled reference image spatially bounded by the spatially co-located boundaries of the first and second spatial segments is independent from any portion of the image of the first spatial layer covered by any other portion of the upsampled reference image spatially bounded by the spatially co-located boundaries of the first and second spatial segments, or such that any portion of the upsampled reference image spatially covered by any of the second spatial segments is dependent on any portion of the image of the first spatial layer covered by any other of the second spatial segments adjacent to the respective spatial segment.

29. The encoder of claim 27 or 28, wherein the encoder is configured to encode the image of the second spatial layer into the spatial scalable bitstream in the second spatial segment, and the encoder is configured to set and insert the syntax elements into the spatial scalable bitstream and to interpolate the image of the first spatial layer depending on the syntax elements.

30. 30. The encoder of claim 29, wherein the encoder is configured to encode the images of the second spatial layer using intra-picture spatial prediction, wherein the intra-picture spatial prediction is suspended for each second spatial segment at a boundary of the respective second spatial segment.

31. 31. The encoder of claim 25, wherein the encoder is configured to use substitution rules to fill a fraction of a filter kernel used in the interpolation of the image of the first spatial layer, according to which the filter kernel projects into such portions of the image of the first spatial layer, to achieve the independence from any portion of the image of the first spatial layer, and according to which the fraction is filled independently of each portion of the image of the first spatial layer into which the filter kernel projects.

32. 32. The encoder of claim 31, wherein the encoder is also configured to use an alternative rule for filling a portion of a filter kernel that protrudes from an outer boundary of the image of the first spatial layer.

33. The encoder of any of claims 25 to 32, wherein the encoder is a video encoder configured to set and insert the syntax elements into the spatial scalable bitstream on a per image or per image sequence basis.

34. 1. A video decoder for decoding a multi-layer video data stream in which a scene is coded in a hierarchy of layers using inter-layer prediction from a first layer to a second layer, the video decoder supporting parallel decoding of the multi-layer video data stream in spatial segments into images of the layers divided by sequentially traversing the spatial segments in a time-overlapping manner with an inter-layer delay during the traversal of the spatial segments of images of the first layer relative to the traversal of the spatial segments of images of the second layer; The video decoder Examining a long-term syntax element structure (642; e.g., min_spatial_segment_delay) of said multi-layer video data stream; when the long-term syntax element structure (e.g., min_spatial_segment_delay≠0) is set to a value of a first set of possible values, use the value of the long-term syntax element to determine the inter-layer offset in advance for a predetermined period of time, and periodically determine the size and position of the spatial segment of the image of the first layer and the spatial segment of the image of the second layer and the spatial sampling resolution of the image of the first layer and the image of the second layer at time intervals shorter than the predetermined period of time and based on short-term syntax elements (602) of the multi-layer video data stream, respectively; A video decoder configured to periodically determine the inter-layer offset within a predetermined period based on the short-term syntax elements of the multi-layer video data stream at a time interval shorter than the predetermined period when the long-term syntax element is set to a value of a second set of possible values ​​(e.g., min_spatial_segment_delay = 0) that is prime to the first set of possible values.

35. 35. The video decoder of claim 34, wherein the video decoder is configured to decode in parallel the multi-layer video data stream in sub-streams of blocks into which the images of the layer are divided and into which the images of the layer are periodically re-divided using wavefront parallelism by sequentially traversing the sub-streams in a time-overlapping manner with an intra-picture inter-sub-stream delay between the traversal of directly consecutive sub-streams of the same image and the inter-layer offset during the traversal of the sub-streams of images of the first layer.

36. 36. The video decoder of claim 35, wherein the video decoder is configured to decode the sub-streams in parallel, the sub-streams supporting intra-picture spatial prediction across the sub-stream boundaries.

37. 35. The video decoder of claim 34, wherein the video decoder is configured to decode the multi-layer video data stream in tiles into which images of the layers are divided, traversing the tiles in tile order between the tiles within each of the images of the first and second layers, and decoding an immediately subsequent tile of the image of the first layer and an immediately subsequent tile of the image of the second layer in parallel with the inter-layer offset between the traversal of the tile of the image of the first layer relative to the traversal of the tile of the image of the second layer.

38. 38. The video decoder of claim 37, wherein the video decoder is configured to decode the images of the first and second layers using intra-picture spatial prediction that suspends the intra-picture spatial prediction for each tile at a boundary of the respective tile.

39. 39. The video decoder of claim 34, wherein the video decoder is configured to use the value of the long-term syntax element when determining the inter-layer offset, using the value of the long-term syntax element to measure the inter-layer offset in units of spatial segments of the image of the first layer.

40. 40. The video decoder of claim 34, wherein the video decoder is configured to use the value of the long-term syntax element in determining the inter-layer offset by using the value of the long-term syntax element as a number of spatial segments of the picture of the first layer by which the decoding of a first spatial segment of the picture of the second layer should be delayed relative to starting the decoding and traversal of the picture of the first layer.

41. 1. A video encoder that encodes a scene in a hierarchy of layers into a multi-layer video data stream using inter-layer prediction from a first layer to a second layer such that images of the layers can be decodable into divided spatial segments by sequentially traversing the spatial segments in a temporal overlapping manner with an inter-layer offset between the traversals of spatial segments of an image of the first layer relative to the traversals of spatial segments of an image of the second layer, the video encoder comprising: inserting and setting a long-term syntax element structure (min_spatial_segment_delay) and short-term syntax elements into the multi-layer video data stream, the short-term syntax elements of the multi-layer video data stream periodically defining the time interval, size, and position of the spatial segment of the image of the first layer and the spatial segment of the image of the second layer, respectively, and defining the spatial sampling resolution of the image of the first layer and the image of the second layer; the encoder is configured to switch between sets; and setting the long-term syntax element structure (min_spatial_segment_delay≠0) to a value of a first set of possible values, wherein the long-term syntax element structure (min_spatial_segment_delay≠0) has the value indicating the inter-layer offset for a predetermined period longer than the time interval, such that during the predetermined period, the size and the position of the spatial segment of the image of the first layer and the spatial segment of the image of the second layer and the spatial sampling resolution of the image of the first layer and the image of the second layer respectively enable decoding of the multi-layer video data stream by sequentially traversing the spatial segments in the time-overlapping manner with an actual inter-layer offset shorter than or equal to the inter-layer offset indicated by the long-term syntax element during the traversal of the spatial segment of the image of the first layer relative to the traversal of the spatial segment of the image of the second layer; During the predetermined period, the long-term syntax element to a value of a second set of possible values ​​(min_spatial_segment_delay=0) is disjoint from the first set of possible values, and sets the short-term syntax element to one of a series of possible sets, the series of possible sets including at least one set according to the size and the position of the spatial segment of the image of the first layer and the spatial segment of the image of the second layer and the spatial sampling resolution of the image of the first layer and the image of the second layer, respectively, in the time overlapping manner with an actual inter-layer offset shorter than or equal to the inter-layer offset indicated by the long-term syntax element during the traversal of the spatial segment of the image of the first layer relative to the traversal of the spatial segment of the image of the second layer. and setting at least one other set according to the sizes and positions of the spatial segments of the image of the first layer and the spatial segments of the image of the second layer and the spatial sampling resolutions of the image of the first layer and the image of the second layer, respectively, to enable decoding of the multi-layer video data stream by sequentially traversing the spatial segments in the time-overlapping manner having actual inter-layer offsets shorter than or equal to the inter-layer offset indicated by the long-term syntax element during the traversal of the spatial segments of the image of the first layer related to the traversal of the spatial segments of the image of the second layer.

42. The video encoder includes:

42. The video encoder of claim 41, configured to perform encoding such that the spatial segments are sub-streams consisting of rows of blocks into which images of the layer are divided and into which the images of the layer are normally further divided, in a manner that enables the multi-layer video data stream in sub-streams to be decodable in parallel using wavefront parallel processing by sequentially traversing the sub-streams in a time-overlapping manner with an intra-picture intra-sub-stream delay of the sub-streams between immediately successive traversals of the same image and an inter-layer offset between the traversals of sub-streams of images of the first layer relative to the traversals of sub-streams of images of the second layer.

43. The video encoder includes:

43. The video encoder of claim 42, wherein the video encoder is configured to support the intra-picture spatial prediction across the sub-stream boundaries and to encode the sub-streams by entropy coding the sub-streams individually or by initializing the entropy context probabilities to adopt the entropy context probabilities of sub-streams according to an order between the sub-streams prior to an intermediate adaptation state, using intra-picture spatial prediction and performing entropy coding that adapts to the entropy context probabilities.

44. The video encoder includes:

42. The video encoder of claim 41, configured to: perform encoding such that the spatial segments are tiles into which images of the layers are divided, in a manner that enables the multi-layer video data stream to be decodable into tiles by traversing the tiles in the inter-tile order within each of the images of the first and second layers; and perform decoding of directly succeeding tiles of an image of the first layer and directly succeeding tiles of an image of the second layer in parallel with the inter-layer offset during the traversal of the tiles of the image of the first layer relative to the traversal of the tiles of the image of the second layer.

45. 45. The video encoder of claim 44, wherein the video encoder is configured to encode the images of the first and second layers using intra-picture spatial prediction, the intra-picture spatial prediction being suspended for each tile at a respective tile boundary.

46. 46. ​​The video encoder of claim 41, wherein the video encoder is configured such that the values ​​of the long-term syntax elements define a measurement for the inter-layer offset in units of spatial elements of the pictures of the first layer.

47. 47. The video decoder of claim 41, wherein the video encoder is configured to set the value of the long-term syntax element to indicate a number of spatial segments of the image of the first layer by which decoding a first spatial segment of the image of the second layer should be delayed relative to starting the decoding and traversal of the image of the first layer.

48. 1. A network entity for processing a multi-layered video data stream in which scenes are coded in layers such that in each layer, the scene is coded at a different operational point of a scalability space measured by a scalability dimension, the multi-layered video data stream comprising first NAL units each associated with one of the layers and second NAL units interspersed within the first NAL units and representing general information about the multi-layered video data stream, the network entity comprising: In the second NAL unit, inspecting a type indicator field (696, e.g., dedicated_scalability_ids_flag); if the type indicator field has a first state (e.g., dedicated_scalability_ids_flag=0), read mapping information (e.g., layer_id_in_nuh[i], dimension_id[i][j]) that maps possible values ​​of a layer indicator field (e.g., layer_id) in the first NAL unit header from the second NAL unit to the operation point, and associate the first NAL unit with the operation point of the first NAL unit via the layer indicator field and the mapping information; A network entity configured to associate the first NAL unit with the operation point by dividing the layer indicator field in the first NAL unit into one or more parts and locating the operation point of the first NAL unit by using the values ​​of the parts as coordinates of a vector in the scalability space when the type indicator field has a second state (dedicated_scalability_ids_flag = 1).

49. The network entity:

49. The network entity of claim 48, configured to, when the type indicator field has the second state (dedicated_scalability_ids_flag=1), associate the first NAL unit with the operation point by splitting the layer indicator field in the first NAL unit into one or more parts according to a syntax element (dimension_id_len_minus1) in the second NAL unit, position the operation point of the first NAL unit by using the values ​​of the parts as coordinates of a vector in the scalability space, and semantically determine the scalability dimensions according to a further syntax element (scalability_mask) in the second NAL unit.

50. The network entity:

50. A network entity according to claim 48 or 49, wherein the type indicator field is configured to, if the first state (dedicated_scalability_ids_flag = 0), determine the number p of scalability dimensions and semantic meaning from a further syntax element (scalability_mask) in the second NAL unit, and associate the possible values ​​of the layer indicator field with the operational point by reading a list of p-dimensional vectors (708) from the second NAL unit.

51. 51. The network entity of claim 50, wherein if the type indicator field has the second state, the network entity is configured to skip reading a list from the second NAL unit.

52. 52. A network entity as described in claim 49 or 51, wherein the network entity is configured to read the further syntax element from the second NAL unit regardless of the type indicator field having the first or second state, and the size of the layer indicator field is the same regardless of the type indicator field having the first or second state.

53. 53. A network entity according to claim 48 or 52, wherein the network entity comprises a video decoder.

54. 1. A video encoder for encoding a scene into a multi-layered video data stream in layers, such that in each layer the scene is encoded at different operational points of a scalability space measured by a scalability dimension, the multi-layered video data stream including first NAL units each associated with one of the layers and second NAL information interspersed within the first NAL units and indicating general information about the multi-layered video data stream, the video encoder comprising: inserting a type indicator field into the second NAL unit; and The same inserts map information that maps possible values ​​of a layer indicator field in the first NAL unit header to the second NAL unit to the operation point, and sets the type indicator field to have a first state, and the operation point of the first NAL unit sets the layer indicator field in the first NAL unit to be associated with the respective layer indicator field according to the map information; the same as above, setting the layer indicator field in the first NAL unit by dividing the layer indicator field in the first NAL unit into one or more parts, setting the type indicator field to have a second state (dedicated_scalability_ids_flag=1), and setting the above parts such that the values ​​of the parts corresponding to coordinates of a vector within the scalability space indicate the operation point associated with the respective first NAL unit.

10. A video encoder configured to switch between:

55. The video encoder includes:

55. The video encoder of claim 54, further configured to, when setting the type indicator field to have a second state, set and insert syntax elements into the second NAL unit that define the layer indicator field in the first NAL unit as being divided into one or more parts, and set and insert further syntax elements into the second NAL unit that semantically define the scalability dimension.

56. 1. A multi-layered video data stream in which scenes are coded in layers such that in each layer, the scene is coded at different operational points of a scalability space measured by a scalability dimension, the multi-layered video data stream each including first NAL units, each associated with one of the layers, and second NAL units interspersed within the first NAL units and representing general information about the multi-layered video data stream, wherein a type indicator field (696, e.g., dedicated_scalability_ids_flag) indicates: If the type indicator field has a first state (e.g., dedicated_scalability_ids_flag=0), the mapping information in the second NAL unit is responsive to mapping possible values ​​of a layer indicator field (e.g., layer_id) in the first NAL unit header to the operational point; if the type indicator field has a second state (dedicated_scalability_ids_flag=1), then the layer indicator field in the first NAL unit is divided into one or more parts with the operation point of the first NAL unit defined by the values ​​of the parts as coordinates of a vector in the scalability space, A multi-layer video data stream present in the second NAL unit.

57. 1. A method for decoding a multi-layer video data stream, in which a scene is coded into layers, by a multi-standard multi-layer decoder, the multi-layer video data stream including NAL units each associated with one of the layers, the layers being associated with different codecs such that for each layer, the NAL units associated with each layer are coded using the different codecs associated with the respective layer, the transport layer decoder comprising: For each NAL unit, check which codec is associated with it, and a transport layer decoder that passes the NAL units of the multi-layer video data stream to the multi-standard multi-layer decoder that uses inter-layer prediction between layers associated with different codecs to decode the multi-layer video data stream.

58. 58. The video decoder of claim 57, further configured to encapsulate NAL units that have been identified as being associated with a layer that is also associated with any codec different from the predetermined codec using a NAL unit header of the predetermined codec having a NAL unit type indicator set to a state indicating the codec of the respective layer.

59. 59. A video decoder according to claim 57 or 58, further configured to perform discrimination depending on the channel on which the NAL units respectively arrive.

60. 60. The video decoder of claim 57 or 59, further configured to perform handing such that the NAL units associated with different codecs are handed to the multi-standard multi-layer decoder on different channels.

61. 61. A video decoder as claimed in claim 57 or 60, further configured to provide each NAL unit with metadata indicating the codec to which the layer to which the respective NAL unit is associated is associated.

62. 1. A video decoder for decoding a multi-layer video data stream, wherein a scene is coded in a hierarchy of layers using inter-layer prediction from a portion of a first layer to a co-located portion of a second layer, wherein an image of the first layer is subdivided into a first array of blocks and an image of the second layer is subdivided into a second array of blocks, and wherein a raster scan decoding order is defined between the first blocks and the second blocks, respectively, the video decoder comprising: A video decoder configured to determine an inter-layer offset measured in units of the first block for parallel decoding of the images of the first and second layers by sequentially traversing the first and second blocks in a time-overlapping manner with an inter-layer offset between the traversal of a first block of the image of the first layer relative to the traversal of a spatial second block of the image of the second layer in accordance with a syntax element structure (e.g., ctb_delay_enabled_flag, min_spatial_segment_delay) of a multi-layer video data stream.

63. The syntax element structure is a long-term syntax element structure, and the video decoder: A predetermined period of time is determined in advance, and 63. The video decoder of claim 62, configured to periodically determine, at time intervals shorter than the predetermined period, the size and position of the first block of the image of the first layer and the second block of the image of the second layer, and the spatial sampling resolution of the image of the first layer and the image of the second layer, respectively, based on short-term syntax elements of the multi-layer video data stream.

64. The video decoder supports parallel decoding of the multi-layer video data stream in spatial segments arranged sequentially along the raster scan decoding order, and in which images of the layers are divided by sequentially traversing the spatial segments in a time-overlapping manner with the inter-layer offset during the traversal of the spatial segments of the images of the first layer relative to the traversal of the spatial segments of the images of the second layer, and the video decoder when the long-term syntax element structure (e.g., ctb_delay_enabled_flag=0, min_spatial_segment_delay≠0) is set to a value of a first set of possible values, use the value of the long-term syntax element structure to determine the inter-layer offset in advance for a predetermined period of time for measuring the inter-layer offset in units of spatial segments of the image of the first layer, and periodically determine, based on short-term syntax elements of the multi-layer video data stream, sizes and positions of the spatial segments of the image of the first layer and the spatial segments of the image of the second layer and spatial sampling resolutions of the image of the first layer and the image of the second layer at time intervals shorter than the predetermined period; when the long-term syntax element structure is set to a value of a second set of possible values ​​prime to the first set of possible values ​​(e.g., min_spatial_segment_delay=0), periodically determining the inter-layer offset within the predetermined period based on the short-term syntax elements of the multi-layer video data stream at a time interval shorter than the predetermined period; if the long-term syntax element structure is set to values ​​of a third set of possible values ​​prime to the first and second sets of possible values ​​(e.g., ctb_delay_enabled_flag=1, min_spatial_segment_delay≠0), perform the determination of the inter-layer offset in units of the first block, and perform periodic determination of the sizes and positions of the first block of the image of the first layer and the second block of the image of the second layer and the spatial sampling resolution of the image of the first layer and the image of the second layer, respectively; 64. The video decoder of claim 63, wherein the long-term syntax elements of the multi-layer video data stream are examined such that:

65. 65. A video decoder as claimed in claim 63 or 64, wherein the video decoder supports parallel decoding of the multi-layer video data stream in sub-streams other than those into which the images of the layer are divided and composed of rows of the first and second blocks, and uses wavefront parallelism by sequentially traversing the sub-streams in a time-overlapping manner with an intra-picture inter-sub-stream delay between the traversal of directly consecutive sub-streams of the same image and inter-layer offset during the traversal of a sub-stream of an image of the first layer relative to the traversal of a sub-stream of an image of the second layer.

66. the long-term syntax element structure includes a unit flag (compare, for example, ctb_delay_enabled_flag) and a delay indicator (compare, for example, min_spatial_segment_delay), and the video decoder, upon examining the long-term syntax element structure, examining said delay indicator to determine whether same is set to zero or otherwise; if the delay indicator is set to zero, the values ​​of the long-term syntax element structure are set to the second set of values; and 66. A video decoder as claimed in any one of claims 64 to 65, wherein, when the delay indicator is set to a non-zero value, the non-zero value is used to determine the values ​​of the long-term syntax element structure, and when the unit flag is zero, the values ​​of the long-term syntax element structure are set to the first set of values, and when the unit flag is one, the values ​​of the long-term syntax element structure are set to the third set of values.

67. 67. A video decoder according to any of claims 65 to 66, configured to rely on the inter-layer offset when starting parallel decoding of first and second layers.

68. 68. A video decoder as claimed in any one of claims 62 to 67, configured to check whether a number s of spatial segments or coding blocks of the first layer are fully decoded by a number s uniquely dependent on the syntax element structure, and to postpone starting to decode the second layer while decoding the first layer unless the check reveals that at least s spatial segments or coding blocks of the first layer have been fully decoded.

69. 69. A video decoder according to any of claims 62 to 68, configured to rely on an inter-layer offset to start and to handle fully parallel decoding of the first and second layers.

70. 70. The video decoder of claim 62, wherein the decoder is configured to check whether a number s of spatial segments or coding blocks of the first layer are fully decoded by a number s uniquely dependent on the syntax element structure and by a number t-1 of already decoded spatial segments or coding blocks of the second layer, and to postpone starting to decode the t-th spatial segment or coding block of the second layer during decoding of the first layer unless it is found that at least s of the spatial segments or coding blocks of the first layer have been fully decoded.

71. 1. A method for decoding a multi-layer video data stream (40) in which a scene is coded in a hierarchy of layers using inter-layer prediction from a first layer to a second layer, comprising: The video decoder supports parallel decoding of the multi-layer video data stream in spatial segments (80) into which the images (12, 15) of the layers are subdivided, and the method comprises: inspecting a long-term syntax element structure (606; e.g., tile_boundaries_aligned_flag) of said multi-layer video data stream; interpreting the long-term syntax element structure assuming a value from a first set of possible values ​​(e.g., tile_boundaries_aligned_flag=1) as a guarantee that the image (608) of the second layer will be subdivided during the predetermined period (608) so that boundaries between the spatial segments of the image of the second layer overlap with respective boundaries of the spatial segments of the image (12) of the first layer, and periodically determining the subdivision of the images of the first and second layers into the spatial segments based on short-term syntax elements (602; e.g., column_width_minus1[i] and column_width_minus1[i]) of the multi-layer video data stream at time intervals (604) shorter than the predetermined period; and a method for periodically determining, at a time interval shorter than the predetermined period, the subdivision of the images of the layers from the short-term syntax elements of the multi-layer video data stream into the spatial segments such that, when the long-term syntax element structure assumes values ​​from a second set of possible values ​​(e.g., tile_boundaries_aligned_flag = 0), there are, for at least a first possible value of the short-term syntax element, boundaries between the spatial segments of the images of the second layer that do not cover any of the boundaries of the spatial segments of the first layer, and, for at least a second possible value of the short-term syntax element, boundaries between the spatial segments of the images of the second layer that overlap each boundary of the spatial segments of the first layer.

72. 1. A method for encoding a scene into a multi-layer video data stream in a hierarchy of layers using inter-layer prediction from a first layer to a second layer such that the multi-layer video data stream is decodable in parallel in spatial segments into which images of the layers are subdivided, the method comprising: inserting a long-term syntax element structure (606) and short-term syntax elements (602) into the multi-layer video data stream, the short-term syntax elements defining the subdivision of the images of the first layer and the second layer into the spatial segments in time intervals; and Switching between configuring the long-term syntax element structure; and during a predetermined period (608) longer than the time interval, values ​​from a first set of possible values ​​are selected to subdivide the image of the second layer such that boundaries between the spatial segments of the image of the second layer overlap respective boundaries of the spatial segments of the first layer while setting the short-term syntax elements to a proper subset from a set of possible settings; The method includes switching, during the predetermined period, values ​​from a second set of possible values ​​to set the short-term syntax element to any of the set of possible settings, wherein the set of possible settings includes at least one setting whereby there are boundaries between the spatial segments of the image of the second layer that do not overlap any of the boundaries of the spatial segments of the first layer, and at least another setting whereby boundaries between the spatial segments of the image of the second layer overlap respective boundaries of the spatial segments of the first layer.

73. 1. A method for decoding a spatial scalable bitstream (40) in which images are coded in different spatial layers and, for at least one of the spatial layers, in a first spatial segment, comprising: The method comprises: or so that any portion of the divisions (622) of the upsampled reference image is dependent on portions of the image of the first spatial layer that are covered by other divisions of the divisions that are spatially adjacent to the respective division. A method for decoding a spatial scalable bitstream, comprising: upsampling an image (12) of a first spatial layer to obtain an upsampled reference image; and predicting an image (15) of a second spatial layer using the upsampled reference image, the method comprising: responding to a syntax element (616; e.g., independent_tile_upsampling_idc) in the spatial scalable bitstream to interpolate (620) the image of the first spatial layer in dependence on the syntax element.

74. 1. A method for encoding an image into a spatial scalable bitstream in various spatial layers and for at least one of the spatial layers, in a first spatial segment, comprising: The method comprises: such that any part of a division of the upsampled reference image that depends on the first spatial segment is independent of parts of the image of the first spatial layer that are covered by any of the other parts of the division; or any portion of the division of the upsampled reference image is dependent on portions of the image of the first spatial layer that are covered by other portions of the division that are spatially adjacent to the respective portion; A method for upsampling an image of a first spatial layer to obtain an upsampled reference image, and predicting an image of a second spatial layer using the upsampled reference image, the method including setting and inserting a syntax element (606) into a spatial scalable bitstream, and interpolating the image of the first spatial layer depending on the syntax element.

75. 1. A method for decoding a multi-layer video data stream in which a scene is coded in a hierarchy of layers using inter-layer prediction from a first layer to a second layer, wherein the video decoder supports parallel decoding of the multi-layer video data stream in spatial segments into which images of layers are divided by sequentially traversing the spatial segments in a time-overlapping manner with an inter-layer delay during the traversal of a spatial segment of an image of the first layer relative to the traversal of a spatial segment of an image of the second layer, the method comprising: Examining a long-term syntax element structure (642; e.g., min_spatial_segment_delay) of said multi-layer video data stream; when the long-term syntax element structure (e.g., min_spatial_segment_delay≠0) is set to a value of a first set of possible values, use the value of the long-term syntax element to determine the inter-layer offset in advance for a predetermined period of time, and periodically determine the size and position of the spatial segment of the image of the first layer and the spatial segment of the image of the second layer and the spatial sampling resolution of the image of the first layer and the image of the second layer at time intervals shorter than the predetermined period of time and based on short-term syntax elements (602) of the multi-layer video data stream, respectively; and periodically determining the inter-layer offset within a predetermined period based on the short-term syntax elements of the multi-layer video data stream at time intervals shorter than the predetermined period when the long-term syntax elements are set to a value of a second set of possible values ​​that is prime to the first set of possible values ​​(e.g., min_spatial_segment_delay = 0).

76. 1. A method for encoding a scene into a multi-layer video data stream in a hierarchy of layers using inter-layer prediction from a first layer to a second layer such that images of the layers can be decodable into separated spatial segments by sequentially traversing the spatial segments in a temporally overlapping manner with an inter-layer offset between the traversals of spatial segments of an image of the first layer relative to the traversals of spatial segments of an image of the second layer, the method comprising: inserting and setting a long-term syntax element structure (min_spatial_segment_delay) and short-term syntax elements into the multi-layer video data stream, the short-term syntax elements of the multi-layer video data stream periodically defining the time interval, size, and position of the spatial segment of the image of the first layer and the spatial segment of the image of the second layer, respectively, and defining the spatial sampling resolution of the image of the first layer and the image of the second layer; The method includes: setting the short-term syntax elements to an appropriate subset from a set of possible sets for a predetermined time period; wherein the appropriate subset has the value indicating the inter-layer offset for a predetermined period longer than the time interval, and is selected such that during the predetermined period, the size and the position of the spatial segment of the image of the first layer and the spatial segment of the image of the second layer and the spatial sampling resolution of the image of the first layer and the image of the second layer respectively enable decoding of the multi-layer video data stream by sequentially traversing the spatial segments in the time-overlapping manner with respect to the traversal of the spatial segment of the image of the second layer, the actual inter-layer offset being shorter than or equal to the inter-layer offset indicated by the long-term syntax element during the traversal of the spatial segment of the image of the first layer; and the appropriate subset is selected such that the appropriate subset has the value indicating the inter-layer offset for a predetermined period longer than the time interval, and is configured to set the long-term syntax element structure (min_spatial_segment_delay≠0) to a value of a first set of possible values ​​and switch between sets; During the predetermined period, the long-term syntax element to a value of a second set of possible values ​​(min_spatial_segment_delay=0) is disjoint from the first set of possible values, and sets the short-term syntax element to one of a series of possible sets, the series of possible sets including at least one set according to the size and the position of the spatial segment of the image of the first layer and the spatial segment of the image of the second layer and the spatial sampling resolution of the image of the first layer and the image of the second layer, respectively, to the temporal overlap with an actual inter-layer offset shorter than or equal to the inter-layer offset indicated by the long-term syntax element during the traversal of the spatial segment of the image of the first layer relative to the traversal of the spatial segment of the image of the second layer. and setting the sizes and positions of the spatial segments of the image of the first layer and the spatial segments of the image of the second layer and at least one other set according to the spatial sampling resolutions of the image of the first layer and the image of the second layer, respectively, to enable decoding of the multi-layer video data stream by sequentially traversing the spatial segments in the time-overlapping manner having actual inter-layer offsets shorter than or equal to the inter-layer offset indicated by the long-term syntax element during the traversal of the spatial segments of the image of the first layer associated with the traversal of the spatial segments of the image of the second layer.

77. 1. A method for processing a multi-layered video data stream in which scenes are coded in layers such that in each layer the scene is coded at a different operational point in a scalability space measured by a scalability dimension, the multi-layered video data stream comprising first NAL units each associated with one of the layers and second NAL units interspersed within the first NAL units and representing general information about the multi-layered video data stream, the method comprising: In the second NAL unit, inspecting a type indicator field (696, e.g., dedicated_scalability_ids_flag); if the type indicator field has a first state (e.g., dedicated_scalability_ids_flag=0), read mapping information (e.g., layer_id_in_nuh[i], dimension_id[i][j]) that maps possible values ​​of a layer indicator field (e.g., layer_id) in the first NAL unit header from the second NAL unit to the operation point, and associate the first NAL unit with the operation point of the first NAL unit via the layer indicator field and the mapping information; If the type indicator field has a second state (dedicated_scalability_ids_flag=1), associating the first NAL unit with the operation point by dividing the layer indicator field in the first NAL unit into one or more parts and locating the operation point of the first NAL unit by using the values ​​of the parts as coordinates of a vector in the scalability space.

78. 1. A method for encoding a scene into a multi-layered video data stream in layers, such that in each layer the scene is encoded at different operational points of a scalability space measured by a scalability dimension, the multi-layered video data stream comprising first NAL units each associated with one of the layers and second NAL units interspersed within the first NAL units and representing general information about the multi-layered video data stream, the method comprising: inserting a type indicator field into the second NAL unit; and The same inserts map information that maps possible values ​​of a layer indicator field in the first NAL unit header to the second NAL unit to the operation point, and sets the type indicator field to have a first state, and the operation point of the first NAL unit sets the layer indicator field in the first NAL unit to be associated with the respective layer indicator field according to the map information; the same as above, setting the layer indicator field in the first NAL unit by dividing the layer indicator field in the first NAL unit into one or more parts, setting the type indicator field to have a second state (dedicated_scalability_ids_flag=1), and setting the above parts such that the values ​​of the parts corresponding to coordinates of a vector within the scalability space indicate the operation point associated with the respective first NAL unit. The method includes switching between

79. 1. A method for decoding by a multi-standard multi-layer decoder a multi-layer video data stream in which scenes are coded into layers, the multi-layer video data stream being composed of NAL units each associated with one of the layers, the layers being associated with different codecs such that the NAL units associated with each layer are coded using the different codecs associated with each layer, the method comprising: For each NAL unit, check which codec is associated with it, and and passing NAL units of the multi-layer video data stream to the multi-standard multi-layer decoder, the multi-standard multi-layer decoder using inter-layer prediction between layers associated with different codecs to decode the multi-layer video data stream.

80. 1. A method of decoding a multi-layer video data stream, wherein a scene is coded in a hierarchy of layers using inter-layer prediction from a portion of a first layer to a co-located portion of a second layer, wherein an image of the first layer is subdivided into a first array of blocks and an image of the second layer is subdivided into a second array of blocks, and wherein a raster scan decoding order is defined between the first blocks and the second blocks, respectively, the method comprising: a method for determining an inter-layer offset measured in units of the first block for parallel decoding of the images of the first and second layers by sequentially traversing the first and second blocks in a time-overlapping manner with the inter-layer offset during the traversal of a first block of the image of the first layer relative to the traversal of a spatial second block of the image of the second layer in accordance with a syntax element structure (e.g., ctb_delay_enabled_flag, min_spatial_segment_delay) of the multi-layer video data stream.

81. A computer program having a program code for executing when a computer operates in accordance with a method according to any one of claims 71 to 80.

Images