Scalable video coding using inter-layer prediction contributions for enhancement layer prediction

Abstract

To provide a scalable video decoder, encoder and method for achieving higher encoding efficiency.SOLUTION: A scalable video decoder is configured to: reconstruct a base layer signal from an encoded data stream to acquire a reconstructed base layer signal; reconstruct a portion of a currently reconstructed enhancement layer signal from an already reconstructed portion of the enhancement layer signal so as to include spatial or temporal prediction to acquire an enhancement layer internal prediction signal 34; form a weighted average of an inter-layer prediction signal 38 acquired from the reconstructed base layer signal and the enhancement layer internal prediction signal such that weighting between the inter-layer prediction signal and the enhancement layer internal signal varies over different spatial frequency components at a portion currently to be reconstructed to acquire an enhancement layer prediction signal 42; and use the enhancement layer prediction signal to acquire a version 54 obtained by predictively reconstructing the enhancement layer signal.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to scalable video coding. [Background technology]

[0002] In non-scalable coding, intra-coding refers to a coding technique that uses only data of already coded parts of the current picture (e.g. reconstructed samples, coding modes, or symbol statistics) and not reference data of already coded pictures. For example, intra-coded pictures (or intra-pictures) are used in broadcast bitstreams to allow a decoder to tune into the bitstream at so-called random access points. Intra-pictures are also used in error-prone environments to limit error propagation. Generally, the first picture of a coded video sequence must be coded as an intra-picture, since no picture that can be used as a reference picture is available here. Often, intra-pictures are also used in scene cuts where temporal prediction usually cannot provide a suitable prediction signal.

[0003] Furthermore, intra-coding modes are used for certain regions / blocks within so-called inter-pictures, where they perform better than inter-coding modes in terms of rate-distortion efficiency. This is often the case in flat regions as well as in regions where temporal prediction is rather poorly performed (occlusions, partially dissolving or fading objects).

[0004] In scalable coding, the concept of intra coding (coding of intra pictures and coding of intra blocks in inter pictures) can be extended to all pictures belonging to the same access unit or time. Thus, the intra coding mode for spatial or quality enhancement layers can increase coding efficiency and at the same time make it possible to use inter-layer prediction from lower layer pictures instantly. It means that not only can already coded parts in the current enhancement layer picture be used for intra prediction, but also lower layer pictures already coded at the same time can be used. The latter concept is also called inter-layer intra prediction.

[0005] In state-of-the-art hybrid video coding standards (such as H.264 / AVC or HEVC), the images of a video sequence are partitioned into blocks of samples. The size of the blocks is either fixed or the coding scheme is provided with a hierarchical structure that allows the blocks to be further subdivided into smaller block sizes. Usually, the reconstruction of a block is obtained by generating a prediction signal of the block and adding a transmitted residual signal. Usually, the residual signal is transmitted using transform coding, which means that a quantization index list (also called transform coefficient levels) for the transform coefficients is transmitted using an entropy coding technique. Then, at the decoder side, these transmitted transform coefficient levels are scaled and inverse transformed to obtain a residual signal that is added to the prediction signal. The residual signal is generated by intra prediction (using only data already transmitted for the current time instant) or by inter prediction (using data already transmitted for a different time instant).

[0006] If inter prediction is used, the prediction block is derived by motion compensated prediction using samples of already reconstructed frames. This can be done by unidirectional prediction (using one reference picture and one set of motion parameters). Alternatively, the prediction signal can be generated by multi-hypothesis prediction. In the latter case, two or more prediction signals are superimposed, i.e., for each sample, a weighted average is constructed to form the final prediction signal. The (superimposed) multi-prediction signal can be generated using different motion parameters for different hypotheses (e.g. different reference pictures or motion vectors). It is also possible, for unidirectional prediction, to multiply the samples of the motion compensated prediction signal by a constant factor and add a constant offset to form the final prediction signal. Such scaling and offset correction is also used for all hypotheses or for selected hypotheses in multi-hypothesis prediction.

[0007] In current state-of-the-art video coding techniques, an intra prediction signal for a block is obtained by predicting samples from the spatial neighborhood of the current block (that is, blocks reconstructed before the current block according to the block processing order). In the latest standards, various prediction methods are used that perform prediction in the spatial domain. Fine-grained directional prediction modes, which filter or unfilter samples of neighboring blocks, are extended to a specific angle to generate a prediction signal. In addition, there are also plane-based and DC-based prediction modes that use samples of neighboring blocks to generate a flat prediction plane or a DC prediction block.

[0008] In older video coding standards (e.g., H.263, MPEG-4), intra prediction was performed in the transform domain. In this case, the transmitted coefficients were inverse quantized. Then, for a subset of the transform coefficients, the transform coefficient values ​​were predicted using corresponding reconstructed transform coefficients of neighboring blocks. The inverse quantized transform coefficients were added to the predicted transform coefficient values, and the reconstructed transform coefficients were used as input to an inverse transform. The output of the inverse transform formed the final reconstructed signal for the block.

[0009] In scalable video coding, base layer information is also utilized to aid the prediction process for the enhancement layer. In the state-of-the-art video coding standard for scalable coding (SVC extension of H.264 / AVC), there is one additional mode to improve the coding efficiency of the intra prediction process in the enhancement layer. This mode is signaled at the macroblock level (blocks of 16x16 luma samples). This mode is only aided if the collocated samples in the lower layer are coded using an intra prediction mode. If this mode is selected for a macroblock in a quality enhancement layer, the prediction signal is assembled by the collocated samples of the reconstructed lower layer signal before the deblocking filter operation. If an inter-layer intra prediction mode is selected in a spatial enhancement layer, the prediction signal is generated by upsampling the collocated reconstructed base layer signal (after the deblocking filter operation). For the upsampling, an FIR filter is used. Generally, for inter-layer intra prediction modes, an additional residual signal is transmitted by transform coding, and if it is correspondingly signaled in the bitstream, the transmission of the residual signal can be omitted (inferred to be equal to zero). The final reconstructed signal is obtained by adding the reconstructed residual signal (obtained by scaling the transmitted transform coefficient levels and applying an inverse spatial transform) to the prediction signal. Summary of the Invention [Problem to be solved by the invention]

[0010] However, it would be preferable to be able to achieve higher coding efficiency in scalable video coding.

[0011] It is therefore an object of the present invention to provide a concept for scalable video coding that achieves higher coding efficiency. [Means for solving the problem]

[0012] This object is achieved by the subject matter of the independent claims in the co-filed document.

[0013] One embodiment of the present invention is that a better predictor for predictive coding of an enhancement layer signal within scalable video coding can be achieved by forming an enhancement layer prediction signal from an inter-layer prediction signal and an enhancement layer intra prediction signal with different weighting methods for different spatial frequency components (i.e., by forming a weighted average of the inter-layer prediction signal and the enhancement layer intra prediction signal at the currently to be reconstructed portion) to obtain an enhancement layer prediction signal, and the enhancement layer prediction signal is obtained such that the weightings that the inter-layer prediction signal and the enhancement layer intra prediction signal contribute to the enhancement layer prediction signal vary different spatial frequency components. Therefore, it is possible to interpret the enhancement layer prediction signal from the inter-layer prediction signal and the enhancement layer intra prediction signal in a manner optimized for the spectral characteristics of the individual contributing components (i.e., the inter-layer prediction signal on the one hand and the enhancement layer intra prediction signal on the other hand). For example, the inter-layer prediction signal is obtained from the reconstructed base layer signal based on the improvement of resolution or quality. The inter-layer prediction signal may be more accurate at low frequencies compared to high frequencies. As far as the enhancement layer intra prediction signal is concerned, the characteristic may be the opposite, i.e. its accuracy increases for high frequencies compared to low frequencies. In this example, at low frequencies, the contribution of the inter-layer prediction signal to the enhancement layer prediction signal exceeds the contribution of the enhancement layer intra prediction signal to the enhancement layer prediction signal with the respective weighting. And as far as high frequencies are concerned, the contribution of the enhancement layer intra prediction signal to the enhancement layer prediction signal does not exceed. Thus, a more accurate enhancement layer prediction signal may be achieved. As a result, the coding efficiency is increased, resulting in a higher compression ratio.

[0014] According to various embodiments, different possibilities are described for incorporating the just outlined concept into any scalable video coding based concept. For example, the formation of the weighted average can be formed either in the spatial domain or in the transform domain. The performance of the spectral weighted average requires a transform to be performed on the separate contributions (i.e. the inter-layer prediction signal and the enhancement layer intra prediction signal). However, it avoids spectrally filtering either the inter-layer prediction signal or the enhancement layer intra prediction signal in the spatial domain, including, for example, FIR or IIR filtering. However, performing the formation of the spectral weighted average in the spatial domain can avoid the bypassing of the separate contributions to the weighted average via the transform domain. The decision of which domain is actually selected to perform the formation of the spectral weighted average can depend on whether the scalable video data stream contains a residual signal in the form of transform coefficients for the part that is currently to be composed in the enhancement layer signal. If not, the bypassing via the transform domain can be stopped. On the other hand, when a residual signal is present, a detour through the transform domain is more advantageous since it allows the transmitted residual signal in the transform domain to be added directly to the spectrally weighted average in the transform domain.

[0015] One aspect of the present invention is that information available from the base layer encoding / decoding (i.e., base layer hints) can be utilized to make the enhancement layer motion compensation prediction more efficient by encoding the enhancement layer motion parameters more efficiently. In particular, a set of motion parameter candidates collected from adjacent already reconstructed blocks of a frame of the enhancement layer signal can possibly be extended by one or more base layer motion parameter sets of blocks of the base layer signal (base layer signal collocated with a block of the frame of the enhancement layer signal). As a result, the available quality of the motion parameter candidate set is improved based on which the motion compensation prediction of a block of the enhancement layer signal can be performed by selecting one of the motion parameter candidates of the extended motion parameter candidate set and using the selected motion parameter candidate for prediction. Additionally or alternatively, the motion parameter candidate list of the enhancement layer signal can be ordered depending on the base layer motion parameters involved in the base layer encoding / decoding. Thus, the probability distribution for selecting the enhancement layer motion parameters from the ordered motion parameter candidate list is compressed, such that, for example, the explicitly signaled index syntax element can be coded (e.g., using entropy coding, etc.) using fewer bits. Additionally or alternatively, the index used in the encoding / decoding of the base layer can serve as the basis for determining the index into the motion parameter candidate list for the enhancement layer. Thus, any signaling of the index for the enhancement layer can be completely avoided. Or, simply the deviation of the prediction thus determined relative to the index can be transmitted in the enhancement layer substream, which can result in improved coding efficiency.

[0016] One aspect of the present invention is that scalable video coding is made more efficient by deriving / selecting a sub-block sub-partition to be used for enhancement layer prediction within a set of possible sub-block sub-partitions of an enhancement layer block by evaluating spatial variations of base layer coding parameters on the base layer signal. Thus, if and only if, less signaling overhead has to be spent to signal this sub-block sub-partition in the enhancement layer data stream. The sub-block sub-partition selected in this way can be used in predictively encoding / decoding the enhancement layer signal.

[0017] One aspect of the present invention is that sub-block-based coding of the enhancement layer transform coefficient block can be made more efficient if the sub-division of the sub-blocks of each transform coefficient block is controlled based on the base layer residual signal or base layer signal. In particular, by utilizing the respective base layer hints, the sub-blocks can be made longer along the horizontal spatial frequency axis relative to the edge extension observable from the base layer residual signal or base layer signal. Thus, it is possible to adapt the shape of the sub-blocks to the estimated distribution of the energy of the transform coefficients of the enhancement layer transform coefficient block such that, with increasing probability, each sub-block is filled almost completely with either significant transform coefficients (i.e., transform coefficients that are not quantized to zero) or non-significant transform coefficients (i.e., only transform coefficients quantized to zero), while with decreasing probability, any sub-block has the same number of significant transform coefficients on one side and the same number of non-significant transform coefficients on the other side. However, due to the fact that sub-blocks that do not have significant transform coefficients can be efficiently signaled within the data stream, such as by the use of a single flag, and due to the fact that sub-blocks that are almost completely filled with significant transform coefficients do not require the expenditure of signaling capacity to code non-significant transform coefficients that may be interspersed therewith, the coding efficiency for coding the transform coefficient blocks of the enhancement layer is increased.

[0018] One aspect of the present invention is that the coding efficiency of scalable video coding may be increased by substituting missing spatial intra prediction parameter candidates in the spatial neighborhood of a current block of the enhancement layer by using intra prediction parameters of a collocated block of the base layer signal. Thus, the coding efficiency for coding spatial intra prediction parameters is likely to increase, or more precisely, to increase, due to the improved prediction quality of the set of intra prediction parameters of the enhancement layer. It increases the likelihood that a suitable predictor for intra prediction parameters for intra predicted blocks of the enhancement layer is useful, so that the signaling of intra prediction parameters of each enhancement layer block can be performed with fewer bits on average.

[0019] Further advantageous realisations are set out in the dependent claims.

[0020] The preferred embodiments are further explained below with reference to the drawings. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram illustrating a scalable video encoder in which embodiments and aspects described herein may be implemented. [Diagram 2] FIG. 2 is a block diagram illustrating a scalable video decoder compatible with the scalable video encoder of FIG. 1 in which embodiments and aspects described herein may also be implemented. [Diagram 3] FIG. 3 is a block diagram illustrating a more particular embodiment of a scalable video encoder in which embodiments and aspects described herein may be implemented. [Figure 4] FIG. 4 is a block diagram of a scalable video decoder compatible with the scalable video encoder of FIG. 3 in which embodiments and aspects described herein may be implemented. [Diagram 5] FIG. 5 is a schematic diagram showing a video and its base and enhancement layer versions that further illustrates the encoding / decoding ordering. [Figure 6] FIG. 6 is a schematic diagram showing a portion of a layered video signal to illustrate possible prediction modes for an enhancement layer. [Figure 7] FIG. 7 is a block diagram illustrating the formation of an enhancement layer prediction signal using spectrally varying weighting between the enhancement layer intra prediction signal and the inter layer prediction signal according to an embodiment. [Figure 8] FIG. 8 is a schematic diagram of syntax elements possibly included in an enhancement layer sub-stream according to an embodiment. [Figure 9] FIG. 9 is a schematic diagram showing a possible realization of the formation of FIG. 7 according to an embodiment in which the formation / combination is performed in the spatial domain. [Figure 10] FIG. 10 is a schematic diagram illustrating the realization of the formation of FIG. 7 according to an embodiment in which the formation / combination is performed in the spectral domain. [Figure 11] FIG. 11 is a schematic diagram showing a portion of a layered video signal to illustrate spatial intra prediction parameter derivation from a base layer to an enhancement layer signal, according to an embodiment. [Figure 12] FIG. 12 is a schematic diagram illustrating an extension of the derivation of FIG. 11 according to an embodiment. [Figure 13] FIG. 13 is a schematic diagram illustrating a set of candidate spatial intra prediction parameters into which one spatial intra prediction parameter obtained from a base layer is inserted according to an embodiment. [Figure 14] FIG. 14 is a schematic diagram showing a portion taken from a layered video signal to illustrate the granular derivation of prediction parameters from a base layer according to an embodiment. [Figure 15] 15a and 15b are schematic diagrams illustrating how to select an appropriate sub-partition for a current block using spatial variations of base layer motion parameters according to two different examples within the base layer. [Figure 15c] FIG. 15c is a schematic diagram illustrating a first possibility of the coarsest selection method within the possible sub-block sub-divisions for the current enhancement layer block. [Figure 15d] FIG. 15d is a schematic diagram illustrating a second possibility for selecting the coarsest among the possible sub-block sub-divisions for the current enhancement layer block. [Figure 16] FIG. 16 is a schematic diagram showing a portion taken from a layered video signal to illustrate the use of derivation of a sub-block sub-division for a current enhancement layer block according to an embodiment. [Figure 17] FIG. 17 is a schematic diagram showing a portion of a layered video signal to illustrate the use of base layer hints to efficiently encode enhancement layer motion parameter data in accordance with an embodiment. [Figure 18] FIG. 18 is a schematic diagram illustrating a first possibility for increasing the efficiency of the enhancement layer motion parameter signaling. [Figure 19a] FIG. 19a is a schematic diagram illustrating a second possibility of how to utilize base layer hints to make enhancement layer motion parameter signaling more efficient. [Figure 19b] FIG. 19b is a schematic diagram illustrating a first possibility of transmitting the base layer in the order listed in the enhancement layer motion parameter candidates. [Figure 19c] FIG. 19c is a schematic diagram illustrating a second possibility of transmitting the base layer in the order listed in the enhancement layer motion parameter candidates. [Figure 20] FIG. 20 is a schematic diagram illustrating another possibility of utilizing base layer hints to make the signaling of enhancement layer motion parameters more efficient. [Figure 21] FIG. 21 is a schematic diagram illustrating a portion of a layered video signal taken to illustrate an embodiment in which sub-block subdivision of a transform coefficient block is appropriately adjusted to hints obtained from a base layer according to an embodiment. [Figure 22] FIG. 22 is a schematic diagram illustrating different possibilities how to derive a suitable sub-block sub-partition of a transform coefficient block from the base layer. [Figure 23] FIG. 23 is a block diagram illustrating a more detailed embodiment for a scalable video decoder in which the examples and embodiments outlined herein may be implemented. [Figure 24a] FIG. 24a is a block diagram illustrating a scalable video encoder, compatible with the scalable video decoder of FIG. 23, in which examples and embodiments outlined herein may be implemented. [Figure 24b] FIG. 24b is a block diagram illustrating a scalable video encoder, compatible with the scalable video decoder of FIG. 23, in which examples and embodiments outlined herein may be implemented. [Diagram 25] Figure 25 is a schematic diagram to explain the generation of an inter-layer intra prediction signal by summing an (upsampled / filtered) base layer reconstructed signal (BL Reco) and spatial intra prediction using the difference signal (EH Diff) of an already coded adjacent block. [Figure 26] Figure 26 is a schematic diagram to explain the generation of an inter-layer intra prediction signal by summing the (upsampled / filtered) base layer residual signal (BL Resi) and spatial intra prediction using reconstructed enhancement layer samples (EH Reco) of already coded neighboring blocks. [Figure 27] Figure 27 is a schematic diagram to explain the generation of an inter-layer intra prediction signal by a frequency-weighted sum of the (upsampled / filtered) base layer reconstructed signal (BL Reco) and spatial intra prediction using reconstructed enhancement layer samples of already coded neighboring blocks (EH Reco). [Figure 28] FIG. 28 is a schematic diagram for explaining the base layer signal and the enhancement layer signal used in the specification. [Figure 29] FIG. 29 is a schematic diagram illustrating motion compensation prediction of the enhancement layer. [Diagram 30] FIG. 30 is a schematic diagram for explaining prediction using base layer residual and enhancement layer reconstruction. [Diagram 31] FIG. 31 is a schematic diagram for explaining prediction using BL reconstruction and EL difference signals. [Diagram 32] FIG. 32 is a schematic diagram for explaining the prediction using the BL reconstruction and the second hypothesis of the EL difference signal. [Diagram 33] FIG. 33 is a schematic diagram for explaining prediction using BL reconstruction and EL reconstruction. [Diagram 34] FIG. 34 is a schematic diagram for explaining, as an example, the decomposition of an image into square blocks and the corresponding quadtree structure. [Diagram 35] FIG. 35 is a schematic diagram illustrating the allowed decomposition of a square block into sub-blocks in the preferred embodiment. [Diagram 36] 36 is a schematic diagram for explaining the position of motion vector prediction, in which (a) represents the position of spatial candidates, and (b) represents the position of temporal candidates. [Figure 37] FIG. 37 is a schematic diagram illustrating the algorithm and the merging blocks (a) and the redundancy checks performed for spatial candidates (b). [Figure 38] FIG. 38 is a schematic diagram illustrating (a) the block merging algorithm and (b) the redundancy check performed for spatial candidates. [Figure 39] FIG. 39 is a schematic diagram for explaining the scanning directions (diagonal, vertical, horizontal) for a 4×4 transform block. [Diagram 40] Figure 40 is a schematic diagram to explain the scan directions (diagonal, vertical, horizontal) for an 8x8 transform block. The shaded areas define the significant subgroups. [Diagram 41] FIG. 41 is a 16×16 transformation diagram in which only diagonal scans are defined. [Diagram 42]FIG. 42 is a schematic diagram for explaining vertical scanning for 16×16 conversion as proposed in JCTVC-G703. [Diagram 43] 43 is a schematic diagram to explain the implementation of vertical and horizontal scanning for a 16×16 transform block. A coefficient subgroup is defined as one row or one column, respectively. [Diagram 44] FIG. 44 is a schematic diagram for explaining vertical and horizontal scanning for a 16×16 transform block. [Diagram 45] FIG. 45 is a schematic diagram illustrating backward-adaptive enhancement layer intra prediction using neighboring reconstructed enhancement layer samples and reconstructed base layer samples. [Figure 46] FIG. 46 is a schematic diagram showing an enhancement layer image / frame to illustrate spatial insertion of a differential signal according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Fig. 1 illustrates in a general manner an embodiment for a scalable video encoder, in which the embodiments outlined further below may be incorporated. The scalable video encoder of Fig. 1 is generally indicated using the reference sign 2 and receives and encodes a video 4. The scalable video encoder 2 is configured to encode the video 4 in a data stream 6 in a scalable manner. That is to say, the data stream 6 comprises a first portion 6a having the video 4 encoded therein with a first amount of information content and a further portion 6b having the video 4 encoded therein with an amount of information content larger than that of the portion 6a. For example, the information content amounts of the portions 6a and 6b may differ in quality or fidelity, i.e. in the amount of pixel-wise deviation from the original video 4 and / or in the spatial resolution. However, also other forms of different information content amounts may be applied, such as for example color fidelity. The portion 6a may be called a base layer data stream or a base layer substream, whereas the portion 6b may be called an enhancement layer data stream or an enhancement layer substream.

[0023] Scalable video encoder 2 is configured to exploit redundancy between versions 8a and 8b of reconstructable video 4 from base layer substream 6a without enhancement layer substream 6b on the one hand, and from both substreams 6a and 6b on the other hand. To do so, scalable video encoder 2 uses inter-layer prediction.

[0024] As shown in Fig. 1, a scalable video encoder 2 alternatively receives two versions 4a and 4b of a video 4. Both versions 4a and 4b differ from each other in terms of information content, just as base layer sub-stream 6a and enhancement layer sub-stream 6b differ. Thus, for example, scalable video encoder 2 is configured to generate sub-streams 6a and 6b. As a result, base layer sub-stream 6a has version 4a encoded therein, while enhancement layer data stream (sub-stream) 6b has version 4b encoded therein using inter-layer prediction based on base layer sub-stream 6b. The encoding of sub-streams 6a and 6b is both lossy.

[0025] Even if the scalable video encoder 2 only receives the original version of the video 4, the scalable video encoder 2 may be configured to internally derive two versions 4a and 4b therefrom, for example by obtaining a base layer version 4a by spatial downscaling and / or tone mapping from a higher bit depth to a lower bit depth.

[0026] Figure 2 illustrates a scalable video decoder that is compatible with the scalable video encoder 2 of Figure 1 in a similar manner suitable for incorporating the embodiments outlined below. The scalable video decoder of Figure 2 is generally indicated using the reference numeral 10, and is generally configured to decode the encoded data stream 6 to reconstruct an enhancement layer version 8b of the video therefrom if both portions 6a and 6b of the data stream 6 reach the scalable video decoder 10 in a complete manner, or to reconstruct a base layer version 8a of the video therefrom if, for example, portion 6b is unavailable due to transmission loss, etc. That is, the scalable video decoder 10 is configured to be able to reconstruct version 8a from only the base layer sub-stream 6a, and to be able to reconstruct version 8b using inter-layer prediction from both portions 6a and 6b.

[0027] Before the following more detailed embodiments of the present invention (i.e., the embodiments show how the embodiments of Figs. 1 and 2 are implemented) are described in detail, a more detailed implementation of the scalable video encoder and decoder of Figs. 1 and 2 is described with respect to Figs. 3 and 4. Fig. 3 shows a scalable video encoder 2 comprising a base layer encoder 12, an enhancement layer encoder 14 and a multiplexer 16. The base layer encoder 12 is configured to encode a base layer version 4a of the input video, while the enhancement layer encoder 14 is configured to encode an enhancement layer version 4b of the video. Thus, the multiplexer 16 receives a base layer sub-stream 6a from the base layer encoder 12 and an enhancement layer sub-stream 6b from the enhancement layer encoder 14 and multiplexes both into the encoded data stream 6 for output.

[0028] As shown in FIG. 3, both encoders 12 and 14 may be predictive encoders using, for example, spatial and / or temporal prediction to encode the respective input versions 4a and 4b into the respective sub-streams 6a and 6b. In particular, each of the encoders 12 and 14 may be a hybrid video block encoder. That is, each of the encoders 12 and 14 may be configured to encode the respective input versions of the video on a block-by-block basis, during which, for example, an image or frame of the respective video version 4a and 4b is selected between different prediction modes for each block of the respective sub-divided blocks. The different prediction modes of the base layer encoder 12 may include spatial and / or temporal prediction modes. On the other hand, the enhancement layer encoder 14 may additionally support inter-layer prediction modes. The subdivision into blocks may differ between the base layer and the enhancement layer. The prediction modes, prediction parameters for the prediction modes selected for the various blocks, prediction residuals, and, optionally, block subdivisions of the respective video versions may be described by the respective encoders 12, 14 using respective syntaxes including syntax elements that may in turn be encoded into the respective substreams 6a, 6b using entropy coding. Inter-layer prediction may be utilized, for example, one or more times, to predict the enhancement layer video, prediction modes, prediction parameters, and / or block subdivision samples, as just mentioned in a few examples. Thus, both the base layer encoder 12 and the enhancement layer encoder 14 may include a predictive encoder 18a, 18b followed by an entropy encoder 19a, 19b, respectively. The predictive encoders 18a, 18b, in turn, form a syntax element stream using predictive coding from the inbound versions 4a and 4b, respectively. The entropy encoders 19a, 19b entropy code the syntax elements output by the respective predictive encoders 18a, 18b. As just mentioned, the inter-layer prediction of encoder 2 may be relevant at different times within the encoding procedure of the enhancement layer.Predictive encoder 18b is thus shown connected to one or more of predictive encoder 18a, its output, and entropy encoder 19a. Similarly, entropy encoder 19b may optionally utilize inter-layer prediction, for example, by predicting the context used for entropy encoding from the base layer. Entropy encoder 19b is thus shown optionally connected to any of the elements of base layer encoder 12.

[0029] In the same manner as FIG. 2 for FIG. 1, FIG. 4 shows a possible realization of a scalable video decoder 10 that is compatible with the scalable video encoder of FIG. 3. The scalable video decoder 10 of FIG. 4 thus comprises a demultiplexer 40 receiving a data stream 6 to obtain substreams 6a and 6b, a base layer decoder 80 configured to decode the base layer substream 6a, and an enhancement layer decoder 60 configured to decode the enhancement layer substream 6b. As shown, the decoder 60 is connected to the base layer decoder 80 to receive information therefrom for utilizing inter-layer prediction. This allows the base layer decoder 80 to reconstruct a base layer version 8a from the base layer substream 6a. The enhancement layer decoder 60 is then configured to reconstruct an enhancement layer version 8b of the video using the enhancement layer substream 6b. Similar to the scalable video encoder of FIG. 3, each of the enhancement layer decoder 60 and base layer decoder 80 may include an entropy decoder 100, 320, followed by a predictive decoder 102, 322, respectively, therein.

[0030] To simplify the understanding of the following embodiments, FIG. 5 exemplarily illustrates different versions of a video 4, namely base layer versions 4a and 8a, which are simply offset from each other by coding losses. Similarly, enhancement layer versions 4b and 8b, respectively, are simply offset from each other by coding losses. As illustrated, the base layer signal and the enhancement layer signal may be composed of a sequence of pictures 22a and 22b, respectively, which are shown in FIG. 5 as being registered with each other (i.e., picture 22a of the base layer version as well as the temporally corresponding picture 22b of the enhancement layer signal) along a time axis 24. As mentioned above, picture 22b may have a higher spatial resolution and / or represent video 4 with a higher fidelity, etc. (e.g., with a higher bit depth of the sample values ​​of the picture). Using solid and dotted lines, an encoding / decoding order is indicated to be defined between pictures 22a, 22b. According to the example shown in FIG. 5, the encoding / decoding order traverses the pictures 22a and 22b in such a way that the base layer picture 22a of a given time stamp / instance is traversed before the enhancement layer picture 22b of the same time stamp of the enhancement layer signal. With respect to the time axis 24, the pictures 22a, 22b are traversed by the encoding / decoding order 26 in the order of their presentation time. However, an order that deviates from the order of the presentation time of the pictures 22a, 22b is also possible. Neither the encoder 2 nor the decoder 10 need to encode / decode consecutively according to the encoding / decoding order 26. Rather, encoding / decoding can be used in parallel. The encoding / decoding order 26 may define the utilization between parts of the base layer signal and the enhancement layer signal that are adjacent to each other in a spatial, temporal and / or inter-layer sense. As a result, when encoding / decoding a current part of the enhancement layer, the available part of the current enhancement layer part is defined through the encoding / decoding order. Thus, since only the adjacent parts that are available according to this encoding / decoding order 26 are used by the encoder for prediction, the decoder has access to the same sources of information to refine the prediction.

[0031] With respect to the following figures, it is described how the scalable video encoder or decoder described above with respect to figures 1-4 may be implemented to form an embodiment of the present invention in accordance with one aspect of the present invention. Possible implementations of the aspects described below are discussed using the label "Aspect C".

[0032] In particular, Fig. 6 illustrates an image 22b of the enhancement layer signal, denoted using the reference numeral 360, and an image 22a of the base layer signal, denoted using the reference numeral 200. Temporally corresponding images of different layers are shown in the manner indicated relative to one another with respect to the time axis 24. Diagonal lines are used to distinguish portions 200 in the base and enhancement layer signals that have already been coded / decoded according to the coding / decoding order from portions 36 that have not yet been coded or decoded according to the coding / decoding order shown in Fig. 5. Fig. 6 also illustrates a portion 28 of the enhancement layer signal 360 that is currently coded / decoded.

[0033] According to the presently described embodiment, the prediction of portion 28 uses both intra-layer prediction within the enhancement layer itself and inter-layer prediction from the base layer to predict portion 28. However, the predictions are combined such that these predictions contribute to the final prediction of portion 28 in a spectrally varying manner. As a result, in particular, the ratio between both contributions varies spectrally.

[0034] In particular, the portion 28 is predicted spatially or temporally from an already reconstructed portion of the enhancement layer signal 400 (i.e., the portion indicated by the diagonal lines in the enhancement layer signal 400 in FIG. 6). The spatial prediction is illustrated using the arrow 30, whereas the temporal prediction is illustrated using the arrow 32. The temporal prediction may include, for example, a motion compensated prediction, according to which motion vector information is transmitted in the enhancement layer substream for the current portion 28. The motion vector indicates the replacement of a portion of a reference image of the enhancement layer signal 400 to be copied in order to obtain a temporal prediction of the current portion 28. The spatial prediction 30 may include estimating spatially neighboring portions within the current portion 28, already encoded / decoded portions of the image 22b, and spatially neighboring the current portion 28. For this purpose, intra prediction information such as an estimated (or angular) direction is signaled in the enhancement layer substream for the current portion 28. Also, a combination of spatial prediction 30 and temporal prediction 32 may be used as well. In any case, the result is an enhancement layer intra prediction signal 34 as illustrated in FIG.

[0035] Inter-layer prediction is used to obtain another prediction of the current portion 28. For this purpose, the base layer signal 200 undergoes a resolution or quality refinement in order to obtain an inter-layer predicted signal for the current portion 28 of the enhancement layer signal 400, a portion 36 which corresponds spatially and temporally to the current portion 28. The refinement procedure is illustrated using arrow 38 in FIG. 6 and results in an inter-layer predicted signal 39 as shown in FIG. 7.

[0036] Thus, two prediction contributions 34 and 39 are present for the current portion 28. A weighted average of both contributions is then formed to obtain the enhancement layer prediction signal 42 for the current portion 28, in such a way that the weights that the inter-layer prediction signal and the enhancement layer intra prediction signal contribute to the enhancement layer prediction signal 42 vary differently in spatial frequency content, as diagrammatically shown at 44 in Figure 7. Figure 7 exemplarily shows the case where for every spatial frequency component the weights that the prediction signals 34 and 38 contribute to the final prediction signal add the same value 46 for all spectral components, however spectrally varying the ratio between the weights applied to the prediction signal 34 and the weights applied to the prediction signal 39.

[0037] Alternatively, the prediction signal 42 may be used directly by the enhancement layer signal 400 for the current portion 28. Alternatively, the residual signal may be provided in an enhancement layer sub-stream 6b of the current portion 28 resulting in a reconstructed version 54 of the current portion 28 by combination 50 with the prediction signal 42, such as the addition shown in FIG. 7. As an intermediate note, it should be noted that both the scalable video encoder and decoder are hybrid video decoders / encoders that use predictive coding, using transform coding to encode / decode the prediction residual.

[0038] To summarise the description of figures 6 and 7, the enhancement layer sub-stream 6b may include, for the current portion 28, intra prediction parameters 56 for controlling the spatial and / or temporal prediction 30, 32, and optionally weighting parameters 58 for controlling the formation of the spectral weighted average 41, and residual information 59 for signalling the residual signal 48. Meanwhile, the scalable video encoder determines all of these parameters 56, 58, 59 accordingly and plugs the parameters 56, 58, 59 into the enhancement layer sub-stream 6b. The scalable video decoder uses the parameters 56, 58, 59 to reconstruct the current portion 28 as outlined above. All of these elements 56, 58, 59 may be subjected to some quantisation. The scalable video encoder may then determine these parameters / elements (i.e. using a ratio / distortion cost function as quantisation) accordingly. Interestingly, the encoder 2 uses the parameters / elements 56, 58, 59 thus determined to obtain a reconstructed version 54 for the current portion 28, for example to serve as the basis for any prediction for subsequent portions of the enhancement layer signal 400 in the encoding / decoding order.

[0039] Different possibilities exist for the weighting parameters 58 and how they control the formation of the spectral weighted average 41. For example, the weighting parameters 58 may signal only one of two states for the current portion 28, namely one state activating the formation of the spectral weighted average described so far and the other state deactivating the contribution of the inter-layer prediction signal 38. As a result, the final enhancement layer prediction signal 42 is then created only by the enhancement layer intra prediction signal 34. The weighting parameters 58 for the current portion 28 may switch between the activation of the spectral weighted average formation on the one hand and the inter-layer prediction signal 39 forming the enhancement layer prediction signal 42 alone on the other hand. The weighting parameters 58 may also be designed to signal one of the three mentioned states / alternatives. Alternatively / additionally, the weighting parameters 58 may further control the spectrally weighted average formation 41 with respect to the spectral change of the ratio between the weights that the predicted signals 34 and 39 contribute to the final predicted signal 42 for the current portion 28. It will be explained later that the spectrally weighted average formation 41 may involve filtering one or both of the predicted signals 34 and 39 before adding them together, for example using a high-pass filter and / or a low-pass filter. In that case, the weighting parameters 58 may signal filter characteristics for the filter to be used for the prediction of the current portion 28. Alternatively, the weighting parameters 58 may signal / set filter characteristics for the filter to be used for the prediction of the current portion 28. It will be explained below that the spectral weighting in the spectrally weighted average formation 41 may be achieved by individual weighting of the spectral components in the transform domain, and thus in this case the weighting parameters 58 may signal / set values ​​of the individual weightings of these spectral components.

[0040] Additionally or alternatively, the weighting parameters for the current portion 28 may signal whether the spectral weighting in step 41 is performed in the transform domain or in the spatial domain.

[0041] FIG. 9 illustrates an embodiment for performing the spectral weighted averaging in the spatial domain. The prediction signals 39 and 34 are shown to be obtained in the form of respective pixel arrays that correspond to the pixel raster of the current portion 28. To perform the spectral weighted averaging, the pixel arrays of both prediction signals 34 and 39 are shown to undergo filtering. FIG. 9 illustrates the filtering as an example by showing filter kernels 62 and 64 traversing the pixel arrays of the prediction signals 34 and 39, for example to perform FIR filtering. However, IIR filtering is also possible. Furthermore, only one of the prediction signals 34 and 39 may undergo filtering. Since the transfer functions of both filters 62 and 64 are different, the summation 66 of the results of filtering the pixel arrays of the prediction signals 39 and 34 results in the result of the spectral weighted averaging, i.e. the enhancement layer prediction signal 42. In other words, the summation 66 simply sums the collocated samples in the filtered prediction signals 39 and 34 using the filters 62 and 64, respectively. 62-66 result in the spectrally weighted average configuration 41. Figure 9 illustrates that, with residual information 59 present in the form of transform coefficients, an inverse transform 68 can be used to signal the residual signal 48 in the transform domain and result in the spatial domain in the form of a pixel array 70, such that the combination 52 resulting in the reconstructed version 55 is realized by a simple pixel-wise addition of the residual signal array 70 and the enhancement layer prediction signal 42.

[0042] Recall again that prediction is performed by the scalable video encoder and decoder, using the prediction for reconstruction within the decoder and encoder, respectively.

[0043] FIG. 10 exemplarily shows how the spectral weighted average formation is performed in the transform domain. Here, pixel arrays of the predicted signals 39 and 34 are subjected to transforms 72 and 74, respectively. This results in spectral decompositions 76 and 78, respectively. Each spectral decomposition 76 and 78 produces a transform coefficient array, one transform coefficient per spectral component. Each transform coefficient block 76 and 78 is multiplied by a corresponding block of weights, i.e. blocks 82 and 84. As a result, for each spectral component, the transform coefficients of blocks 76 and 78 are weighted individually. For each spectral component, the weight values ​​of blocks 82 and 84 can be summed with a value common to all spectral components, but this is not obligatory. In effect, the multiplier 86 between blocks 76 and 82 and the multiplier 88 between blocks 78 and 84 respectively represent spectral filtering in the transform domain. A transform coefficient / spectral component wise addition 90 then completes the spectral weighted average formation 41 to result in a transform domain version of the enhancement layer prediction signal 42 in the form of a block of transform coefficients. In the case of a residual signal 59 signaling the residual signal 48 in the form of a transform coefficient block, as shown in Figure 10, the residual signal 59 is readily transform coefficient-wise additively combined (or otherwise combined) 52 with the transform coefficient block representing the enhancement layer prediction signal 42 to result in a reconstructed version of the current portion 28 in the transform domain. An inverse transform 84 applied to the additive result of the combination 52 thus results in a pixel array reconstructing the current portion 28, i.e., the reconstructed version 54.

[0044] As mentioned above, the current parameters in the enhancement layer substream 6b for the current portion 28, such as residual information 59 and weighting parameters 58, signal whether the averaging 41 is performed in the transform domain as shown in Figure 10, or in the spatial domain according to Figure 9. For example, suppose the residual information 59 indicates the absence of any transform coefficient blocks for the current portion 28, and the spatial domain is used. Alternatively, the weighting parameters 58 may switch between both domains, with or without the residual information 59 including transform coefficients.

[0045] It is then explained that a differential signal can be calculated and managed between an already reconstructed portion of the enhancement layer signal and the inter-layer prediction signal to obtain an inter-layer enhancement layer prediction signal. A spatial prediction of the differential signal in a first portion juxtaposed to a portion of the enhancement layer signal is now reconstructed from a second portion of the differential signal. A spatially adjacent to the first portion of the enhancement layer signal and belonging to the already reconstructed portion can then be used to spatially predict the differential signal. Alternatively, a temporal prediction of the differential signal in the first portion juxtaposed to a portion of the enhancement layer signal is now reconstructed from a second portion of the differential signal belonging to a previously reconstructed frame of the enhancement layer signal can be used to obtain a temporally predicted differential signal. The combination of the inter-layer prediction signal and the predicted differential signal can be used to obtain an intra-enhancement layer prediction signal, which is then combined with the inter-layer prediction signal.

[0046] With respect to the following figures, it will be described how a scalable video encoder or decoder as described above with respect to FIGS. 1-4 may be implemented to form an embodiment of the present application according to another aspect of the application.

[0047] To explain this, reference is made to Fig. 11, which illustrates a possibility of performing spatial prediction 30 of the current portion 28. As a result, the following description of Fig. 11 is to be combined with the description of Figs. 6 to 10. In particular, the following description is subsequently explained with reference to an example of an illustrated implementation by referring to "aspects" X and Y.

[0048] The situation depicted in FIG. 11 corresponds to that depicted in FIG. 6, i.e. a base layer signal 200 and an enhancement layer signal 400 are shown. The already coded / decoded parts are shown using diagonal lines. In the enhancement layer signal 400, the part currently to be coded / decoded has neighboring blocks 92 and 94. Here, exemplarily, block 92 is depicted on the top side of the current part 28 and block 94 on the left side, for both blocks 92 and 94 having the same size as the current block 28. However, the coincidence of sizes is not obligatory. Rather, the parts of the blocks into which the image 22b of the enhancement layer signal 400 is subdivided may have different sizes. They are not even restricted to squares. They may be rectangular or of other shapes. Furthermore, the current block 28 has neighboring blocks that are not explicitly represented in FIG. 11. However, the neighboring blocks have not yet been decoded / coded. That is, the neighboring blocks follow in the coding / decoding order and are therefore not available for prediction. Beyond this, there are other blocks (such as blocks adjacent to the current block 28, e.g. block 96 diagonally adjacent to the upper left corner of the current block 28) than blocks 92 and 94 that have already been coded / decoded according to the coding / decoding order. However, blocks 92 and 94 are predefined neighboring blocks that serve to predict intra prediction parameters for the current block 28 that is the subject of intra prediction 30 in the example considered here. The number of such predefined neighboring blocks is not limited to two. It may be more or even one.

[0049] The scalable video encoder and the scalable video decoder may determine a set of predefined neighboring blocks (here, blocks 92, 94) from a set of already coded neighboring blocks. Here, blocks 92-96 depend on a predefined sample location 98 in the current portion 28, such as its top-left sample. For example, only those already coded neighboring blocks of the current portion 28 may form a set of "predefined neighboring blocks" including sample locations immediately adjacent to the predefined sample location 98. In any case, the neighboring already coded / decoded blocks include a sample 102 adjacent to the current block 28 based on the sample values ​​of which the region of the current block 28 is to be spatially predicted. For this purpose, spatial prediction parameters such as 56 are signaled in the enhancement layer substream 6b. For example, the spatial prediction parameters for the current block 28 indicate the spatial direction in which the sample values ​​of the sample 102 should be copied into the region of the current block 28.

[0050] In any case, at least as far as the spatially corresponding regions to which the temporally corresponding images 22a relate are concerned, when spatially predicting a current block 28 using block-wise prediction as described above, e.g., using a block-wise selection between spatial and temporal prediction modes, the scalable video decoder / encoder has already reconstructed (in the case of an encoder, encoded) the base layer 200 using the base layer sub-stream 6a.

[0051] 11, several blocks 104 into which the time-ordered image 22a of the base layer signal 200 is subdivided are in and around a region that locally corresponds to the exemplary represented current portion 28. That is exactly the case for the spatially predicted blocks in the enhancement layer signal 400. As in the case of the spatially predicted blocks in the enhancement layer signal 400, the spatial prediction parameters are included or signaled in the base layer substream for those blocks 104 in the base layer signal 200 for which a selection of a spatial prediction mode was signaled.

[0052] Here, illustratively, to enable reconstruction of an enhancement layer signal from the encoded data stream for block 28 for which spatial intra-layer prediction 30 has been selected, the intra-prediction parameters are used and encoded in the following bitstream:

[0053] Intra prediction parameters are often coded using the concept of "most likely intra prediction parameters", which is a fairly small subset of all possible intra prediction parameters. The "most likely intra prediction parameters" may be, for example, a set of most likely intra prediction parameters that includes one, two or three intra prediction parameters. On the other hand, for example, a set of all possible intra prediction parameters may include 35 intra prediction parameters. If an intra prediction parameter is included in the set of most likely intra prediction parameters, it may be signaled in the bitstream with fewer bits. If an intra prediction parameter is not included in the set of most likely intra prediction parameters, its signaling in the bitstream requires more bits. Thus, the amount of bits to be spent on syntax elements to signal intra prediction parameters for the current intra predicted block depends on the quality of the most likely, or perhaps favorable, set of intra prediction parameters. Using this concept, a smaller number of bits is required on average to code the intra prediction parameters, assuming that a set of most likely intra prediction parameters can be properly derived.

[0054] Typically, the set of maximum likelihood intra-prediction parameters is selected in such a way that it includes the intra-prediction parameters of directly neighboring blocks and / or additionally often uses intra-prediction parameters, e.g., in the form of default parameters. For example, it is generally advantageous to include the intra-prediction parameters of neighboring blocks in the set of maximum likelihood intra-prediction parameters, since the dominant gradient direction of the neighboring blocks is the same.

[0055] However, if the neighboring blocks are not coded in a spatial intra-prediction mode, their parameters are not available at the decoder side.

[0056] In scalable coding, it is however possible to use intra-prediction parameters of co-located base layer blocks, and therefore, according to the embodiment outlined below, this situation is exploited by using intra-prediction parameters of co-located base layer blocks in the case of non-coded neighboring blocks in spatial intra-prediction modes.

[0057] As a result, in accordance with FIG. 11, a set of possibly advantageous intra-prediction parameters for the current enhancement layer block is constructed by examining the intra-prediction parameters of predetermined neighboring blocks and by exceptionally reclassifying a predetermined neighboring block to a collocated block in the base layer in any case in which the predetermined neighboring block does not have suitable intra-prediction parameters associated with it, for example because the respective predetermined neighboring block is not coded in an intra-prediction mode.

[0058] First, it is checked whether a predetermined neighboring block, such as block 92 or 94 of the current block 28, was predicted using a spatial intra-prediction mode, i.e., whether a spatial intra-prediction mode was selected for that neighboring block, so that the intra-prediction parameters of that neighboring block are included in the set of possibly favorable intra-prediction parameters for the current block 28 or, alternatively, in the intra-prediction parameters of the collocated block 108 of the base layer, if any. This process can be performed for each of the predetermined neighboring blocks 92 and 94.

[0059] For example, if each predetermined neighboring block is not a spatial intra-prediction block, instead of using a default prediction or the like, the intra-prediction parameters of the block 108 of the base layer signal 200 are included in the set of possibly favorable inter-prediction parameters for the current block 28, which is co-located with the current block 28. For example, the co-located block 108 is determined using a predetermined sample position 98 of the current block 28. That is, the block 108 covers a position 106 that locally corresponds to the predetermined sample position 98 in the temporally aligned image 22a of the base layer signal 200. Of course, a further check can be performed whether this co-located block 108 in the base layer signal 200 is actually a spatial intra-prediction block or not. In the case of FIG. 11, it is exemplarily illustrated that this is the case. However, if the co-located block is also not coded in an intra-prediction mode, the set of possibly favorable intra-prediction parameters can be left without any contribution for the predetermined neighboring block. Or, the default intra-prediction parameters may be used instead as an alternative, i.e., the default intra-prediction parameters are inserted into the set of possibly advantageous intra-prediction parameters.

[0060] Thus, if a block 108 collocated with the current block 28 is spatially intra predicted, its intra prediction parameters signaled in the base layer substream 6a are used, as a sort of alternative, for a predetermined neighboring block 92 or 94 of the current block 28 that does not have any intra prediction parameters because the intra prediction parameters are coded using another prediction mode, such as a temporal prediction mode.

[0061] According to another embodiment, in certain cases, the intra prediction parameters of the predetermined neighboring blocks are replaced by the intra prediction parameters of the co-located base layer block, even if each predetermined neighboring block is of intra prediction mode. For example, a further check, such as whether the intra prediction parameters meet a predetermined criterion, may be performed for any predetermined neighboring block of intra prediction mode. If the predetermined criterion is not met by the intra prediction parameters of the neighboring block, but the same criterion is met by the intra prediction parameters of the co-located base layer block, a substitution is performed despite the very neighboring block being intra-coded. For example, if the intra prediction parameters of the neighboring block do not represent an angular intra prediction mode (but e.g. DC or planar intra prediction mode), but the intra prediction parameters of the co-located base layer block represent an angular intra prediction mode, the intra prediction parameters of the neighboring block may be replaced by the intra prediction parameters of the base layer block.

[0062] The inter prediction parameters for the current block 28 are then determined based on syntax elements present in the encoded data stream, such as the enhancement layer sub-stream 6b and the set of possibly advantageous intra prediction parameters for the current block 28. That is, syntax elements may be coded using fewer bits in the case of an inter prediction parameter for the current block 28 that is a member of the set of possibly advantageous intra prediction parameters than in the case of the remaining members of the set of possible intra prediction parameters that do not lead to a set of possibly advantageous intra prediction parameters.

[0063] The set of possible intra prediction parameters includes several angular modes according to which the current block is filled by copying from neighboring samples already coded / decoded by copying along the angular direction of the respective mode / parameter, one DC mode according to which the samples of the current block are set to a constant value determined, for example by some average, based on neighboring samples already coded / decoded, and so on, and a planar mode according to which the samples of the current block are set to a value distribution following a linear function of x and y slope and cutoff, determined, for example, based on neighboring samples already coded / decoded.

[0064] Figure 12 shows the possibility of how alternative spatial prediction parameters obtained from a collocated block 108 of the base layer can be used together with syntax elements signaled in the enhancement layer substream. Figure 12 shows an expanded view of the current block 28 together with adjacent already coded / decoded samples 102 and predetermined neighboring blocks 92 and 94. Figure 12 also exemplarily shows the angular orientation 112 indicated by the spatial prediction parameters of the collocated block 108.

[0065] The syntax element 114 signaled in the enhancement layer substream 6b for the current block 28 may signal, for example, as shown in FIG. 13, a conditionally coded index 118 into a list 122 of possible favorable intra-prediction parameters, exemplarily illustrated here as an angular direction 124. Alternatively, the candidates in the list 122 may identify the actual intra-prediction parameters 116 if they are not in the most likely set 122, but are indexes 123 in a list 125 of possible intra-prediction modes that are potentially excluded, as shown at 127. The coding of the syntax element may consume less bits for the actual intra-prediction parameters that belong in the list 122. For example, the syntax element may include a flag and an index field. The flag indicates whether to include or exclude members of the list 122, or whether the index points to either the list 122 or the list 125, i.e., to include or exclude members of the list 122. The syntax element includes a field that specifies one of the members 124 of the list 122 or an escape code, and, in the case of an escape code, a second field that specifies a member from a list 125 that includes or excludes the member of the list 122. The order of the members 124 in the list 122 may be determined based on, for example, default rules.

[0066] Thus, the scalable video decoder may obtain or retrieve the syntax element 114 from the enhancement layer substream 6b. The scalable video encoder may then insert the syntax element 114 into the enhancement layer substream 6b. Then, for example, the syntax element 114 is used to index one spatial prediction parameter from the list 122. In forming the list 122, the above-mentioned alternative may be performed by checking whether the predefined neighboring blocks 92 and 94 are of a spatial prediction coding mode type. Otherwise, as mentioned above, it is checked whether the co-located block 108 is, for example, a spatially predicted block in order, and if so, the same spatial prediction parameters, such as the angular direction 112 used to spatially predict this co-located block 108, are included in the list 122. Also, if the base layer block 108 does not include a suitable intra prediction parameter, the list 122 may be left without a contribution from each predefined neighboring block 92 or 94. To avoid the list 122 being empty, at least one of the members 124 may be unconditionally determined using default intra prediction parameters, for example, since both of the predetermined neighboring blocks 92, 98 are intra predicted, as well as the co-located block 108 lacking suitable intra prediction parameters. Alternatively, the list 122 may be allowed to be empty.

[0067] Naturally, the aspects described with respect to Figures 11 to 13 can be coupled with the aspects outlined with respect to Figures 6 to 10. In particular, the intra prediction obtained using spatial intra prediction parameters derived by bypassing the base layer according to Figures 11 to 13 may in particular represent the enhancement layer inter prediction signal 34 of the aspects of Figures 6 to 10, since it is combined in a spectrally weighted manner with the inter layer prediction signal 38 as described above.

[0068] With respect to the following figures, it will be explained how a scalable video encoder or decoder, as described above with respect to Figures 1 to 4, can be implemented to form an embodiment of the present application according to another aspect of the application. Later, for the aspects described below, additional implementation examples are presented with reference to aspects T and U.

[0069] Please refer to FIG. 14, which shows the images 22b and 22a of the enhancement layer signal 400 and the base layer signal 200, respectively, in a temporally registered manner. The portion to be currently coded / decoded is indicated at 28. According to the present embodiment, the base layer signal 200 is predictively coded by a scalable video encoder and predictively reconstructed by a scalable video decoder using base layer coding parameters that are spatially varying the base layer signal. The spatial variation is indicated in FIG. 14 by using a shaded portion 132 where the base layer coding parameters used to predictively code / reconstruct the base layer coding parameters are constant, and when moving from the shaded portion 132 to the non-shaded area, it is surrounded by a non-shaded area where the base layer coding parameters change. According to the embodiment outlined above, the enhancement layer signal 400 is coded / decoded block by block. The current portion 28 is such a block. In accordance with the aspects outlined above, the sub-block sub-division for the current portion 28 is selected from a set of possible sub-block sub-divisions based on spatial variations in base layer coding parameters within the collocated portions 134 of the base layer signal 200, i.e., within the spatially collocated portions of the temporally corresponding images 22a of the base layer signal 200.

[0070] In particular, instead of signaling within the subpartition information of the enhancement layer substream 6b for the current portion 28, the above description presents selecting a subblock subpartition within the set of possible subblock subpartitions of the current portion 28 such that the selected subblock subpartition is the coarsest within the set of possible subblock subpartitions, whereby the base layer signal 200 is subdivided such that, when transferred onto the collocated portion 134 of the base layer signal, the base layer coding parameters are sufficiently similar to each other within each subblock of the respective subblock subpartition. For ease of understanding, please refer to FIG. 15a. FIG. 15a shows the portion 28 illustrating, using diagonal lines, the spatial variation of the base layer coding parameters within the collocated portion 134. In particular, the portion 28 shows three times different subblock subpartitions applied to the block 28. In particular, a quadtree subpartition is exemplarily used in the case of FIG. 15a. That is, the set of possible subblock subpartitions is (or is defined by) a quadtree subpartition. The three examples of subblock subdivisions of the portion 28 depicted in Fig. 15a then belong to different hierarchical levels of the quadtree subdivision of the block 28. From bottom to top, the level or coarseness of the subdivision of the block 28 in subblocks increases. At the highest level, the portion 28 is left as is. At the next lower level, the block 28 is subdivided into four subblocks. Then, at least one of the latter is further subdivided into four subblocks at the next lower level, etc. In Fig. 15a, at each level, the quadtree subdivision is selected where the number of subblocks is the smallest, which is still not a subblock overlapping with a base layer coding parameter change boundary. That is, in the case of Fig. 15a, the quadtree subdivision of the block 28 to be selected for subdividing the block 28 is found to be the lowest one depicted in Fig. 15a. Here, the base layer coding parameters of the base layer are constant in each portion juxtaposed to the respective subblock of the subblock subdivision.

[0071] Therefore, the sub-partition information for block 28 does not need to be signaled in the enhancement layer sub-stream 6b. As a result, coding efficiency is increased. Moreover, as outlined, the method for obtaining the sub-partition is suitable regardless of the position of the current portion 28 with respect to any grid or any registration of the sample array of the base layer signal 200. Also, in particular, the sub-partition derivation works in case of fractional spatial resolution ratios between the base layer and the enhancement layer.

[0072] Based on the thus determined sub-division of the sub-blocks of the portion 28, the portion 28 can be prospectively reconstructed / encoded. It should be noted that, with respect to the above description, different possibilities exist to "measure" the coarseness of the sub-division of the different available sub-blocks of the current block 28. For example, the magnitude of the coarseness is determined based on the number of sub-blocks. The more sub-blocks a respective sub-block sub-division has, the lower the level. This definition obviously does not apply in the case of Fig. 15a, where the "magnitude of the coarseness" is determined by a combination of the number of sub-blocks of the respective sub-block sub-division and the smallest size of all sub-blocks of the respective sub-block sub-division.

[0073] For completeness, Figure 15b illustrates, by way of example, the selection of one possible sub-block sub-division from a set of sub-block sub-divisions available for a current block 28, when exemplarily using the sub-divisions of Figure 35 as the available set. The different hatched (and un-hatched) regions indicate regions where respective collocated regions in the base layer signal have the same base layer coding parameters associated with them.

[0074] As mentioned above, the selection outlined may be implemented by traversing the possible sub-block subdivisions according to some sequential order, such as an order of increasing or decreasing levels of coarseness, and selecting from the possible sub-block subdivisions a possible sub-block subdivision in a situation where, within each sub-block of each sub-block subdivision, the base layer coding parameters are sufficiently similar to one another that the possible sub-block subdivision that no longer applies (in the case of using a traversal according to increasing coarseness) or that happens to apply first (in the case of using a traversal according to decreasing levels of coarseness) may be tested. Alternatively, all possible subdivisions may be tested.

[0075] In the description of Figures 14 and 15a-b, the broad term "base layer coding parameters" is used in the preferred embodiment, however these base layer coding parameters represent base layer prediction parameters, i.e. parameters related to the formation of the prediction of the base layer signal but not to the formation of the prediction residual. Thus, for example, the base layer coding parameters include the prediction mode distinguishing between spatial and temporal prediction, prediction parameters for the blocks / parts of the base layer signal assigned to spatial prediction, such as angular direction, and prediction parameters for the blocks / parts of the base layer signal assigned to temporal prediction, such as motion parameters.

[0076] Interestingly, however, the definition of "sufficient" similarity of base layer coding parameters within a given sub-block only determines / defines a subset of the base layer coding parameters. For example, the similarity may be determined based only on the prediction mode. Alternatively, prediction parameters that further regulate spatial and / or temporal prediction may form the parameters on which the similarity of base layer coding parameters within a given sub-block depends.

[0077] Furthermore, as already outlined, in order to be sufficiently similar to one another, within a given sub-block, the base layer coding parameters need to be exactly equal to one another within each sub-block. Alternatively, the measure of similarity used may need to be within a certain interval to satisfy the "similarity" criterion.

[0078] As outlined above, the subdivision of the selected subblock is not the only quantity that can be predicted or transferred from the base layer signal. Rather, the base layer coding parameters themselves can be transferred to the enhancement layer signal in order to derive enhancement layer coding parameters for the subblocks of the subdivision of the subblock obtained by transferring the subdivision of the selected subblock from the base layer signal to the enhancement layer signal based thereon. As far as the motion parameters are concerned, for example, scaling can be used to transfer from the base layer to the enhancement layer. Preferably, only those parts or syntax elements of the prediction parameters of the base layer are used to set the subblocks of the subdivision of the subblock of the current portion obtained from the base layer that affect the magnitude of similarity. By this magnitude, the fact that these syntax elements of the prediction parameters in each subblock of the subdivision of the selected subblock are somehow similar to each other ensures that the syntax elements of the base layer prediction parameters used to predict the corresponding prediction parameters of the subblocks of the current portion 308 are similar or even equal to each other. As a result, in the first case allowing some variation, some significant "meaning" of syntax elements of the base layer prediction parameters corresponding to the portion of the base layer signal covered by the respective sub-block may be used as predictor for the corresponding sub-block. However, it may also be the case that only the portion of the syntax elements that contributes to the similarity measure is used to predict prediction parameters of the sub-blocks of the enhancement layer sub-division, adding only the transcription of the sub-division itself, so that the mode of the sub-block of the current portion 28 is simply guessed or pre-set, although the mode specific base layer prediction parameters participate in the similarity measure determination.

[0079] One such possibility of not using only base to enhancement layer sub-splitting inter-layer prediction is now illustrated with respect to the next figure (FIG. 16), which shows picture 22b of the enhancement layer signal 400 and picture 22a of the base layer signal 200 in a registered manner along the presentation time axis 24.

[0080] According to the embodiment of Fig. 16, the base layer signal 200 is predictively reconstructed by a scalable video decoder by subdividing the image 22a of the base layer signal 200 into intra-blocks and inter-blocks, and then predictively coded by using a scalable video encoder. According to the example of Fig. 16, the latter subdivision is done in a two-stage method. First, the frame 22a is regularly subdivided into the largest blocks or largest coding units, indicated in Fig. 16 by reference numerals 302, using double lines along their periphery. Then, each largest block 302 is subordinated to a hierarchical quadtree subdivision within the coding units forming said intra-blocks and inter-blocks. As a result, they are the leaves of the quadtree subdivision of the largest block 302. In Fig. 16, reference numerals 304 are used to indicate these leaf blocks or coding units. Usually, solid lines are used to indicate the periphery of these coding units. Meanwhile, spatial intra prediction is used for intra-blocks and temporal inter prediction is used for inter-blocks. Prediction parameters related to spatial intra prediction and temporal inter prediction are set in units of smaller blocks, respectively. However, intra and inter blocks or coding units 304 are sub-divided. Such sub-division is exemplarily shown in FIG. 16 for one of the coding units 304, using reference sign 306 to indicate the smaller block. The smaller block 304 is outlined using a dotted line. That is, for the embodiment of FIG. 16, the spatial video encoder has the opportunity to choose between one spatial prediction and the other temporal prediction for each coding unit 304 of the base layer. However, as far as the enhancement layer signal is concerned, the degree of freedom increases. Here, in particular, frame 22b of the enhancement layer signal 400 is assigned to each one of a set of prediction modes, including not only spatial intra prediction and temporal inter prediction, but also inter-layer prediction as outlined in detail below, within the coding unit into which frame 22b of the enhancement layer signal 400 is sub-divided.Subdivision within these coding units may be done in the same manner as described for the base layer signal: first, frame 22b may be subdivided into rows and columns of the largest blocks outlined using double lines, and then into the outlined coding units using regular solid lines, in a hierarchical quadtree subdivision process.

[0081] One coding unit 308 of the current image 22b of the enhancement layer signal 400 is exemplarily inferred to be assigned to an inter-layer prediction mode and is indicated using diagonal lines. In a manner similar to Figs. 14, 15a and 15b, Fig. 16 shows at 312 how a subdivision of the coding unit 308 is predictively derived by local transfer from the base layer signal. In particular, a local area overlapped by the coding unit 308 is indicated at 312. Within this area, dotted lines indicate the boundaries between adjacent blocks of the base layer signal or, more generally, the boundaries where the base layer coding parameters of the base layer may possibly change. As a result, these boundaries are the boundaries of the prediction blocks 306 of the base layer signal 200 and may partially coincide with the boundaries between adjacent coding units 304 or the largest equally adjacent coding units 302 of the base layer signal 200, respectively. The dotted lines at 312 indicate a subdivision of the current coding unit 308 into predictive blocks derived / selected by local transfer from the base layer signal 200. Details regarding local transfer are provided above.

[0082] 16 embodiment, not only the subdivision into predictive blocks is adopted from the base layer, as already explained. Rather, the prediction parameters of the base layer signal used in region 312 are used to derive prediction parameters to be used to perform prediction for the predictive blocks of coding units 308 of enhancement layer signal 400.

[0083] In particular, according to the embodiment of FIG. 16, not only is the subdivision into prediction blocks derived from the base layer signal, but also prediction modes are used in the base layer signal 200 to code / reconstruct the respective areas locally covered by the respective sub-blocks of the derived subdivision. One example is the following: The prediction modes are used in conjunction with the associated base layer signal 200 to derive the subdivision of the coding unit 308 according to the above. Mode-specific prediction parameters can be used to determine the "similarity" discussed above. Thus, the different diagonal lines shown in FIG. 16 may correspond to different prediction blocks 306 of the base layer. Each of the different prediction blocks 306 may have an intra or inter prediction mode (i.e., a spatial or temporal prediction mode associated with them). As explained above, to be "similar enough", the prediction modes used in the area collocated with the respective sub-blocks of the subdivision of the coding unit 308 and the specific prediction parameters for each prediction mode in the sub-area must be exactly equal to each other. Alternatively, some variation may be tolerated.

[0084] In particular, in accordance with the embodiment of Figure 16, all blocks indicated by diagonal lines extending from the upper left to the lower right may be set to be intra-predicted blocks of coding unit 308 since their locally corresponding portions of the base layer signal are covered by prediction blocks 306 having spatial intra-prediction modes associated with them, while other blocks (i.e., blocks indicated by diagonal lines extending from the lower left to the upper right) may be set to be inter-predicted blocks since their locally corresponding portions of the base layer signal are covered by prediction blocks 306 having temporal inter-prediction modes associated with them.

[0085] On the other hand, for alternative embodiments, the derivation of the prediction within coding unit 308 and the details for performing the prediction are stopped here. That is, the derivation of the subdivision of coding unit 308 within predictive blocks and the allocation of these within predictive blocks coded using non-temporal or spatial prediction and within predictive blocks coded using temporal prediction can be limited, which does not follow the embodiment of FIG. 16 .

[0086] According to the latter embodiment, all prediction blocks of coding unit 308 having a non-temporal prediction mode assigned thereto undergo non-temporal prediction, such as spatial intra prediction, while using prediction parameters derived from prediction parameters of locally matching intra blocks of base layer signal 200 as enhancement layer prediction parameters of blocks of these non-temporal modes. As a result, such derivation may relate to spatial prediction parameters of locally collocated intra blocks of base layer signal 200. For example, such spatial prediction parameters may be an indication of the angular direction in which spatial prediction is performed. As outlined above, a definition of similarity by itself is required, either that the spatial base layer prediction parameters overlap by being the same for each non-temporal prediction block of coding unit 308 as each other, or that for each non-temporal prediction block of coding unit 308, some average of the spatial base layer prediction parameters is used to derive the prediction parameters of each non-temporal prediction block.

[0087] Alternatively, all predictive blocks of coding unit 308 having an assigned non-temporal prediction mode undergo inter-layer prediction in the following manner: first, the base layer signal undergoes decomposition or quality refinement to obtain inter-layer predicted signals, at least in those regions spatially collocated with the non-temporal prediction mode predictive blocks of coding unit 308, and then these predictive blocks of coding unit 308 are predicted using the inter-layer predicted signals.

[0088] Scalable video decoders and encoders, by default, subject all of the coding units 308 to spatial prediction or inter-layer prediction. Alternatively, the scalable video encoder / decoder supports and signals in the coded video data stream signal both alternatives, which version is used as far as non-temporal prediction mode prediction blocks of the coding units 308 are concerned. In particular, the decision between both alternatives may be signaled in the data stream, for example, for any size of the coding units 308 individually.

[0089] As far as another predictive block of coding unit 308 is concerned, coding unit 308 undergoes temporal inter prediction using prediction parameters derived from the prediction parameters of a locally matching inter block, just as it would be the case for a non-temporal prediction mode predictive block, so that the derivation is in turn related to the motion vector assigned to the corresponding part of the base layer signal.

[0090] For every other coding unit having both a spatial intra prediction mode and a temporal inter prediction mode assigned to it, the other coding unit undergoes spatial or temporal prediction in the following manner: In particular, the other coding unit is further subdivided into prediction blocks having a prediction mode assigned to it, where the prediction mode is common to all of the prediction blocks within the coding unit, in particular the same prediction mode assigned to each coding unit. That is, unlike a coding unit having an associated inter-layer prediction mode such as coding unit 308, a coding unit having an associated spatial intra prediction mode or a coding unit having an associated temporal inter prediction mode is subdivided into prediction blocks only of the same prediction mode, i.e. the prediction mode inherited from each coding unit derived by the subdivision of each coding unit.

[0091] All coding unit subdivisions, including 308, may be quadtree subdivisions within a predictive block.

[0092] A further difference between coding units of inter-layer prediction mode such as coding unit 308 and coding units of spatial intra-prediction mode or temporal inter-prediction mode is when the prediction blocks of the spatial intra-prediction mode coding unit or temporal inter-prediction mode coding unit are subjected to spatial and temporal prediction, respectively. The prediction parameters are set independently of the base layer signal 200, for example, by signaling in the enhancement layer substream 6b. Even subdivisions of other coding units having inter-layer prediction modes associated with them, such as coding unit 308, are signaled in the enhancement layer signal 6b. That is, inter-layer prediction mode coding units such as 308 benefit from low bitrate signaling requirements. According to an embodiment, the mode indicator of coding unit 308 itself does not need to be signaled in the enhancement layer substream. Optionally, another parameter can be transmitted for coding unit 308, such as prediction parameter residual, for individual prediction blocks. Additionally or alternatively, the prediction residual for the coding unit may be transmitted / signaled in the enhancement layer sub-stream 6b, while the scalable video decoder retrieves this information from the enhancement layer sub-stream, and the scalable video encoder according to the current embodiment determines these parameters and inserts these parameters in the enhancement layer sub-stream 6b.

[0093] In other words, the prediction of the base layer signal 200 may be made using the base layer coding parameters in such a way that the base layer coding parameters spatially vary the base layer signal 200 within the unit of the base layer block 304. The prediction modes available for the base layer include, for example, spatial and temporal prediction. The base layer coding parameters may further include individual prediction parameters of the prediction modes such as the angular direction (as far as the spatially predicted block 304 is concerned) and the motion vector (as far as the temporally predicted block 304 is concerned). The individual prediction parameters of the latter prediction modes may vary the base layer signal within a unit smaller than the base layer block 304, i.e. within the aforementioned prediction block 306. In order to satisfy the previously outlined requirement of sufficient similarity, it may be necessary that the prediction modes of all base layer blocks 304 overlapping the area of ​​each possible sub-block subdivision are equal to each other. Then, only the respective sub-block subdivisions are shortlisted to obtain the selected sub-block subdivision. However, the requirement is more stringent. That is, the prediction mode individual prediction parameters of the prediction blocks overlapping the common region of the respective sub-block sub-partitions must also be equal to each other, and for each sub-block of this respective sub-block sub-partition and the corresponding region in the base layer signal, only the sub-block sub-partitions that satisfy this requirement can be shortlisted to obtain the final selected sub-block sub-partition.

[0094] In particular, as briefly outlined above, there are various possibilities for how to perform the selection within the set of possible sub-block partitions. To outline this in more detail, refer to Fig. 15c and Fig. 15d. Assume that the set 352 encompasses all possible sub-block sub-partitions 354 of the current block 28. Of course, Fig. 15c merely shows an example. The set 352 of possible or available sub-block sub-partitions of the current block 28 may be known to the scalable video decoder and the scalable video encoder by default, or may be signaled within the encoded data stream, for example a sequence of images or the like. According to the example of Fig. 15c, each member of the set 352, i.e. each available sub-block sub-partition 354, is subjected to a check 356, in which it is checked whether the area within the collocated portion 108 of the base layer signal that is to be subdivided is simply overlapped by the prediction block 306 and the coding unit 304 by transferring the respective sub-block sub-partition 354 from the enhancement layer to the base layer. Then, it is checked whether the base layer coding parameters meet the requirement of sufficient similarity. See for example the exemplary subdivision with reference number 354. According to this exemplary subdivision of available sub-blocks, the current block 28 is sub-divided into four quadrants / sub-blocks 358. And the top left sub-block corresponds to a region 362 in the base layer. Obviously, this region 362 overlaps four blocks of the base layer (i.e., two prediction blocks 306 and two coding units 304, which are not another subdivision in the prediction blocks and thus represent the prediction blocks themselves). Thus, if all base layer coding parameters of these predictive blocks overlapping region 362 satisfy the similarity criterion, and this is furthermore the case for all subblocks / quadrant of a possible subblock subdivision 354 and their corresponding regions overlapping base layer coding parameters, then this possible subblock subdivision 354 satisfies the sufficient requirement for all regions covered by the subblocks of the respective subblock subdivision and belongs to the set of subblock subdivisions 364.Within this set 364 , the coarsest subdivision is then selected, as indicated by arrow 366 , resulting in a subdivision 368 of the selected subblock from set 352 .

[0095] Obviously, it is preferable to try to avoid performing the check 356 for all members of the set 352. Thus, as shown in FIG. 15d and as previously described, the possible subdivisions 354 can be traversed to increase or decrease in size. The traversal is indicated using a double-headed arrow 372. FIG. 15d shows that for at least some of the available sub-block subdivisions, the levels or measures of coarseness are equal to each other. In other words, the ordering according to increasing or decreasing levels of coarseness may be ambiguous. However, this does not prevent the search for the "coarsest sub-block subdivision" belonging to the set 364, since only one such equally coarse possible sub-block subdivision can belong to the set 364. Thus, when traversing in the direction of increasing levels of coarseness, with the second to last traversed possible sub-block subdivision being the sub-block subdivision 354 to be selected, the largest possible sub-block subdivision 368 can be found as soon as the result of the criteria check 356 changes from filled to not filled. Or, when traversing in the direction of decreasing magnitude levels where the sub-block subdivision 368 is the most recently traversed sub-block subdivision, the coarsest possible sub-block subdivision 368 is found while the result of the criteria check 356 switches from unfilled to filled.

[0096] For the following figures, a scalable video encoder or decoder as described above with respect to Figures 1 to 4 is implemented to form an embodiment of the present application according to another embodiment of the present application. Possible implementation forms of the aspects described below are presented below with reference to aspects K, A, and M.

[0097] To explain the embodiment, reference is made to Fig. 17. Fig. 17 shows possibilities for temporal prediction 32 of the current portion 28. As a result, the following description of Fig. 17 may be combined with the descriptions of Figs. 6 to 10 insofar as it relates to combination with inter-layer prediction signals, or with the descriptions of Figs. 11 to 13 insofar as it relates to combination with temporal inter-layer prediction modes.

[0098] The situation shown in FIG. 17 corresponds to the situation shown in FIG. 6. That is, the base layer signal 200 and the enhancement layer signal 400 are shown with the already coded / decoded parts shown using diagonal lines. In the enhancement layer signal 400, the part to be coded / decoded currently has adjacent blocks 92 and 94, here exemplarily depicted as block 92 above and block 94 to the left of the current part 28. Both blocks 92 and 94 exemplarily have the same size as the current block 28. However, the matching of sizes is not mandatory. Rather, the parts of the blocks of the enhancement layer signal 400 that are being subdivided into the image 22b may have different sizes. They are not even limited to being rectangular. They may be oblong or of other shapes. The current block 28 with other adjacent blocks is not explicitly depicted in FIG. 17. However, the other adjacent blocks have not yet been coded / decoded, i.e., they continue in the coding / decoding order and are therefore not available for prediction. Besides, there are other blocks than blocks 92 and 94 already coded / decoded according to the coding / decoding order, illustratively a block 96 next to the current block 28, diagonally above and to the left of the current block 28. However, blocks 92 and 94 are predetermined neighboring blocks that serve to predict inter prediction parameters for the current block 28 undergoing inter prediction 30 in the example considered here. The number of such predetermined neighboring blocks is not limited to two. It may be more than one or may be just one. A discussion of possible embodiments is presented with respect to Figures 36 to 38.

[0099] The scalable video encoder and scalable video decoder may determine a set of predetermined neighboring blocks (here, blocks 92, 94) from a set of already-encoded neighboring blocks (here, blocks 92-96) within the current portion 28, such as its top-left sample, depending on, for example, a predetermined sample location 98. For example, only those already-encoded neighboring blocks of the current portion 28 form a set of "predetermined neighboring blocks" that include sample locations directly adjacent to the predetermined sample location 98. Further possibilities may be described with respect to Figures 36-38.

[0100] In any case, a portion 502 of a previously coded / decoded image 22b of the enhancement layer signal 400, which is displaced from its collocated position of the current block 28 by a motion vector 504 according to the decoding / encoding order, comprises sample values ​​reconstructed on the basis of sample values ​​of the portion 28 predicted by simple copying, by interpolation, etc. For this purpose, the motion vector 504 is signaled in the enhancement layer sub-stream 6b. For example, the temporal prediction parameters for the current block 28 indicate a displacement vector 506 indicating the displacement of the portion 502 from its collocated position of the portion 28 in the reference image 22b, optionally by interpolation, to be copied onto the samples of the portion 28.

[0101] In any case, when temporally predicting the current block 28, the scalable video decoder / encoder has already reconstructed (and, in the case of an encoder, has already encoded) the base layer 200 using the base layer sub-stream 6a, at least insofar as the associated spatially corresponding regions of the temporally corresponding images 22a are so associated, using block-wise prediction, as described above, and, for example, using a block-wise selection between spatial and temporal prediction modes.

[0102] In Fig. 17, the time collocated image 22a of the base layer signal 200 is sub-divided into several blocks 104. The blocks 104 are located around an area that corresponds locally to the exemplarily represented current portion 28. It is just the case with spatially predicted blocks in the enhancement layer signal 400. The spatial prediction parameters are included in the base layer sub-stream 6a for those blocks 104 in the base layer signal 200 or are signaled. The selection of the spatial prediction mode is signaled to the base layer signal 200.

[0103] Here, by way of example, for a block 28 for which temporal intra-layer prediction 32 has been selected, inter prediction parameters, such as motion parameters, are determined using any of the following methods to enable reconstruction of the enhancement layer signal from the encoded data stream:

[0104] The first possibility is illustrated with respect to FIG. 18. In particular, first, a set 512 of motion parameter candidates 514 is collected or generated from adjacent already reconstructed blocks of a frame, such as predetermined blocks 92 and 94. The motion parameters are motion vectors. The motion vectors of blocks 92 and 94 are represented using arrows 516 and 518, in which are marked 1 and 2, respectively. As shown, these motion parameters 516 and 518 may directly form the candidates 514. Several candidates may be formed by combining motion vectors, such as 518 and 516, as shown in FIG. 18.

[0105] Further, a set 522 of one or more base layer motion parameters 524 for blocks 108 of the base layer signal 200 collocated in the portion 28 are assembled or generated from the base layer motion parameters. In other words, the motion parameters associated with the collocated blocks 108 in the base layer are used to derive one or more base layer motion parameters 524.

[0106] One or more base layer motion parameters 524, or scaled versions thereof, are then added 526 to the set 512 of motion parameter candidates 514 to obtain an extended motion parameter candidate set 528 of motion parameter candidates. This can be done in a variety of ways, such as simply adding the base layer motion parameters 524 at the end of the list of candidates 514, or in a different way, an example of which is outlined with respect to FIG. 19a.

[0107] At least one of the motion parameter candidates 532 of the extended motion parameter candidate set 528 is then selected. By motion compensated prediction of portion 28, temporal prediction 32 is performed using the selected one of the motion parameter candidates of the extended motion parameter candidate set. The selection 534 may be signaled in a data stream such as sub-stream 6b for portion 28 by way of an index 536 in list / set 528, or may be performed in another manner as described with respect to FIG. 19a.

[0108] As mentioned above, it may be checked whether the base layer motion parameters 523 have been coded in an encoded data stream, such as the base layer substream 6a, using merging, and if the base layer motion parameters 523 have been coded in an encoded data stream using merging, the addition 526 may be suppressed.

[0109] The motion parameters mentioned according to Fig. 18 may relate to only motion vectors (motion vector prediction) or to a complete set of motion parameters including the number of motion hypotheses per block, reference index, partitioning information (merging). Thus, the "scaled version" may result from scaling the motion parameters used in the base layer signal according to the spatial resolution ratio between the base layer signal and the enhancement layer signal in case of spatial scalability. Depending on the method of the encoded data stream, the encoding / decoding of the base layer motion parameters of the base layer signal may involve, for example, spatial or temporal motion vector prediction or merging.

[0110] The incorporation 526 of the motion parameters 523 used in the collocated portion 108 of the base layer signal into a set 528 of merge / motion vector candidates 532 allows for very efficient indexing among the intra-layer candidates 514 and one or more inter-layer candidates 524. The selection 534 may involve explicit signaling of an index into an extended set / list of motion parameter candidates in the enhancement layer signal 6b, per prediction block, per coding unit, or the like. Alternatively, the selection index 536 may be inferred from other information in the enhancement layer signal 6b or from inter-layer information.

[0111] According to the possibility of Fig. 19a, the formation 542 of the final motion parameter candidate list for the enhancement layer signal for the portion 28 is only optionally performed as outlined with respect to Fig. 18. That is, the formation 542 may be 528 or 512. However, the list 528 / 512 is ordered 544 depending on the base layer motion parameters, such as, for example, the motion parameters (represented by the motion vector 523) of the collocated base layer block 108. For example, the rank of the members (i.e., the motion parameter candidates 532 or 514 of the list 528 / 512) is determined based on the deviation of the respective member with respect to the potentially scaled version of the motion parameter 523. The greater the deviation, the lower the rank of the respective member 532 / 512 in the ordered list 528 / 512'. As a result, the ordering 544 involves the determination of the magnitude of the deviation for each member 532 / 514 of the list 528 / 512. The selection 534 of one candidate 532 / 512 in the ordered list 528 / 512′ is performed and controlled via an explicitly signaled index syntax element 536 in the encoded data stream to obtain the enhancement layer motion parameters from the ordered motion parameter candidate list 528 / 512′ for the portion 28 of the enhancement layer signal. And then, temporal prediction 32 is performed using the selected motion parameters to which the index 536 points 534 by motion compensated prediction of the portion 28 of the enhancement layer signal.

[0112] For the motion parameters mentioned in Fig. 19a, the motion parameters mentioned above with respect to Fig. 18 are applied. The decoding of the base layer motion parameters 520 from the encoded data stream may (optionally) involve spatial or temporal motion vector prediction or merging. The ordering may be done just according to the magnitude of the increase in the difference between the respective enhancement layer motion parameter candidates and the base layer motion parameters of the base layer signal in relation to the block of the base layer signal collocated with the current block of the enhancement layer signal. That is, for the current block of the enhancement layer signal, a list of enhancement layer motion parameter candidates may be determined first. Then, it will be explained that the ordering is performed. In the following, the selection is performed with explicit signaling.

[0113] Alternatively, the ordering 544 may be made according to a magnitude that increases the difference between the base layer motion parameters 523 of the base layer signal associated with the block 108 of the base layer signal collocated with the current block 28 of the enhancement layer signal and the base layer motion parameters 546 of the spatially and / or temporally adjacent blocks 548 in the base layer. The determined ordering in the base layer is then transferred to the enhancement layer. As a result, the enhancement layer motion parameter candidates are ordered in the same way as the determined ordering for the corresponding base layer candidates. In this respect, it may be said that the base layer motion parameters 546 correspond to the enhancement layer motion parameters of the adjacent enhancement layer blocks 92, 94 when the associated base layer block 548 is spatially / temporally collocated with the adjacent enhancement layer blocks 92 and 94 associated with the considered enhancement layer motion parameters. Alternatively, when the neighbor relationship (left neighbor, upper neighbor, A1, A2, B1, B2, B0, or see Figures 36-38 for further examples) between the associated base layer block 548 and the block 108 collocated with the current enhancement layer block 28 is the same as the neighbor relationship between the current enhancement layer block 28 and the respective enhancement layer neighboring blocks 92, 94, the base layer motion parameters 546 may be said to correspond to the enhancement layer motion parameters of the enhancement layer neighboring blocks 92, 94. Based on the base layer ordering, the selection 534 is then performed by explicit signaling.

[0114] To explain this in more detail, refer to FIG. 19b. FIG. 19b shows the first of the outlined alternatives for just obtaining the enhancement layer ordered for the list of motion parameter candidates using the base layer hint. FIG. 19b shows the current block 28 and the positions of three different predefined samples (i.e., illustratively, the top-left sample 581, the bottom-left sample 583, and the top-right sample 585). The example is to be interpreted only as an explanation. Assume that the set of predefined neighboring blocks includes four types of neighbors, illustratively, the neighboring block 94a covering the sample position 587 adjacent to and located above the sample position 581, and the neighboring block 94b including or covering the sample position 589 adjacent to and located directly above the sample position 585. Similarly, the neighboring blocks 92a and 92b are those blocks including the sample positions 591 and 593 adjacently located immediately to the left of the sample positions 581 and 583. Also, note that despite the predetermined number of decision rules, the number of predetermined adjacent blocks may vary, as described with respect to Figures 36-38. Nevertheless, the predetermined adjacent blocks 92a, 92b, 94a, and 94b are distinguishable by their decision rules.

[0115] According to the alternative of Fig. 19b, for each predetermined neighboring block 92a, 92b, 94a, 94b, a collocated block in the base layer is determined. For example, for this, the top left sample 595 of each neighboring block is used. It is exactly the case with the current block 28 with respect to the top left sample 581 formally mentioned in Fig. 19a. This is illustrated in Fig. 19b using the dotted arrows. By this means, for each predetermined neighboring block, a corresponding block 597 is found in addition to the collocated block 108, the collocated current block 28. Using the motion parameters m1, m2, m3, m4 of the collocated base layer block 597 and their respective differences to the base layer motion parameter m of the collocated base layer block 108, the enhancement layer motion parameters M1, M2, M3, M4 of the predetermined neighboring blocks 92a, 92b, 94a, 94b can be ordered in the list 528 or 512. For example, the greater the distance of any of m1-m4, the higher the corresponding enhancement layer motion parameter M1-M4. That is, a higher index may be required to index the same state from the list 528 / 512'. For the magnitude of the distance, the absolute difference is used. In a similar manner, the motion parameter candidates 532 or 514 may be reordered in the list with respect to their rank being the combination of the enhancement layer motion parameters M1-M4.

[0116] Fig. 19c shows an alternative in which corresponding blocks in the base layer are determined in another manner. In particular, Fig. 19c shows the predefined neighboring blocks 92a, 92b, 94a, 94b of the current block 28 and the collocated block 108 of the current block 28. According to the embodiment of Fig. 19c, the base layer blocks corresponding to these of the current block 28 (i.e., 92a, 92b, 94a, 94b) are determined in such a way that these base layer blocks are associated with the enhancement layer neighboring blocks 92a, 92b, 94a, 94b using the same neighboring decision rules to determine these base layer neighboring blocks. In particular, Fig. 19c shows the predefined sample positions of the collocated block 108, i.e., the top left, bottom left, and top right sample positions 601. Based on these sample positions, the four neighboring blocks of block 108 are determined in the same manner as described for enhancement layer neighboring blocks 92a, 92b, 94a, and 94b with respect to predetermined sample positions 581, 583, and 585 of current block 28. Four base layer neighboring blocks 603a, 603b, 605a, and 605b are found in this manner. 603a obviously corresponds to enhancement layer neighboring block 92a. Base layer block 603b corresponds to enhancement layer neighboring block 92b. Base layer block 605a corresponds to enhancement layer neighboring block 94a. Base layer block 605b corresponds to enhancement layer neighboring block 94b. In the same manner as previously described, the base layer motion parameters M1 to M4 of the base layer blocks 903a, 903b, 905a, 905b and their distance to the base layer motion parameter m of the collocated base layer block 108 are used to order the motion parameter candidates in the list 528 / 512 formed from the motion parameters M1 to M4 of the enhancement layer blocks 92a, 92b, 94a, 94b.

[0117] According to the possibilities of Fig. 20, the formation 562 of the final motion parameter candidate list for the enhancement layer signal for portion 28 is merely optionally performed as outlined for Fig. 18 and / or Fig. 19, i.e. the formation 562 is 528 or 512 or 528 / 512'. The reference sign 564 may be used in Fig. 20. According to Fig. 20, the index 566 to point in the motion parameter candidate list 564 is determined depending on the index 567 in the motion parameter candidate list 568 that was used for encoding / decoding the base layer signal, for example for the collocated block 108. For example, when reconstructing a base layer signal at block 108, a list 568 of motion parameter candidates may be determined based on motion parameters 548 of neighboring blocks 548 of block 108 that have the same neighboring relationship (left neighbor, top neighbor, A1, A2, B1, B2, B0, or see Figures 36-38 for further examples) with respect to block 108 that is the same as the neighboring relationship between the predetermined neighboring enhancement layer blocks 92, 94 and the current block 28. Here, the determination 572 of list 567 potentially uses the same construction rules as used in the formation 562, such as the ordering within the list members of lists 568 and 564. More generally, an index 566 for an enhancement layer candidate is determined in such a way that its neighboring enhancement layer blocks 92, 94 are pointed to by index 566 collocated with the base layer block 548 associated with the indexed base layer candidate (i.e., the one pointed to by index 567). As a result, index 567 may serve as a significant predictor of index 566. The enhancement layer motion parameters are then determined using an index 566 into the motion parameter candidate list 564, and the motion compensated prediction of block 28 is performed using the determined motion parameters.

[0118] Regarding the motion parameters mentioned in FIG. 20, the same applies as described above with respect to FIGS.

[0119] With respect to the following figures, it is described how a scalable video encoder or decoder, such as those described above with respect to Figures 1 to 4, can be implemented to form an embodiment of the present application according to another aspect of the application. A detailed implementation of the below-described aspects is described below with reference to Example V.

[0120] This embodiment relates to residual coding in an enhancement layer. In particular, Fig. 21 exemplarily shows an image 22b of an enhancement layer signal 400 and an image 22a of a base layer signal 200 in a temporally registered manner. Fig. 21 shows a method of reconstructing in a scalable video decoder or encoding in a scalable video encoder and an enhancement layer signal, concentrating on a predetermined transform coefficient block and a predetermined portion 404 of transform coefficients 402 representing the enhancement layer signal 400. In other words, the transform coefficient block 402 represents a spatial decomposition of the portion 404 of the enhancement layer signal 400. As already explained above according to the encoding / decoding ordering, the corresponding portion 406 of the base layer signal 200 may already be decoded / encoded when decoding / encoding the transform coefficient block 402. As far as the base layer signal 200 is concerned, predictive encoding / decoding may be used, including signaling the base layer residual signal within an encoded data stream such as base layer substream 6a.

[0121] According to the embodiment described with respect to FIG. 21 , the scalable video decoder / encoder exploits the fact that the estimates 408 of the base layer signal or the base layer residual signal may result in an advantageous selection of a subdivision of the transform coefficient block 402 in a sub-block 412 in the portion 406 collocated with the portion 404. In particular, several possible sub-block subdivisions for subdividing the transform coefficient block 402 into sub-blocks may be supported by the scalable video decoder / encoder. These possible sub-block subdivisions may sub-divide the transform coefficient block 402 in regularly rectangular sub-blocks 412. That is, the transform coefficients 414 of the transform coefficient block 402 may be arranged in columns and columns, and according to the possible sub-block subdivisions, these transform coefficients 414 are regularly clustered in the sub-block 412, so that the sub-block 412 itself is arranged in rows and columns. Using the sub-division of the sub-blocks thus selected, the evaluation 408 allows the ratio between the number of rows and the number of columns of the sub-blocks 412 (i.e., the ratio between their width and height) to be set in such a way that the coding of the transform coefficient block 402 is most efficient. If, for example, the evaluation 408 finds that the reconstructed base layer signal 200 in the collocated portion 406, or at least the base layer residual signal in the corresponding portion 406, is mainly composed of horizontal edges in the spatial domain, then the transform coefficient block 402 is likely to be present with significance (i.e., the transform coefficient levels are non-zero, i.e., the quantized transform coefficients are near the zero horizontal frequency side of the transform coefficient block 402). In the case of vertical edges, the transform coefficient block 402 is likely to be present with non-zero transform coefficient levels at positions near the zero vertical frequency side of the transform coefficient block 402. Thus, first, the sub-blocks 412 are selected to be longer along the vertical direction and smaller along the horizontal direction. And secondly, the sub-blocks are made longer horizontally and smaller vertically. The latter case is shown diagrammatically in FIG.

[0122] That is, the scalable video decoder / encoder selects one sub-block sub-division within a set of possible sub-block sub-divisions based on the base layer residual signal or the base layer signal. Then, the encoding 414 or decoding of the transform coefficient block 402 is performed while applying the selected sub-block sub-division. In particular, since the positions of the transform coefficients 414 are traversed in units of sub-blocks 412, all positions in one sub-block are traversed in a manner that immediately follows the next sub-block in the sub-block ordering defined in the sub-block. For a currently visited sub-block, such as the sub-block 412 exemplarily shown at 22 in FIG. 40, a syntax element is signaled in a data stream, such as the enhancement layer sub-stream 6b, indicating whether the currently visited sub-block has any significant transform coefficients. In FIG. 21, the syntax element 416 is described for two exemplary sub-blocks. If the respective syntax element of the respective sub-block indicates a non-significant transform coefficient, nothing else needs to be transmitted in the data stream or enhancement layer sub-stream 6b. Rather, the scalable video decoder may set the transform coefficients in that sub-block to zero. However, if the syntax element 416 of the respective sub-block indicates that this sub-block has significant transform coefficients, another information related to the transform coefficients in that sub-block is signaled in the data stream or sub-stream 6b. On the decoding side, the scalable video decoder decodes from the data stream or sub-stream 6b a syntax element 418 indicating the level of the transform coefficients in the respective sub-block. The syntax element 418 may indicate the position of the significant transform coefficients in that sub-block according to the scanning order of these transform coefficients in the respective sub-block and, optionally, the scanning order of the transform coefficients in the respective sub-block.

[0123] FIG. 22 illustrates different possibilities, which respectively exist for performing the selection within the subdivision of possible subblocks within the evaluation 408. FIG. 22 again describes a portion 404 of the enhancement layer signal, in relation to which a transform coefficient block 402 subsequently represents a spectral decomposition of the portion 404. For example, the transform coefficient block 402 represents a spectral decomposition of an enhancement layer residual signal, with a scalable video decoder / encoder predictively encoding / decoding the enhancement layer signal. In particular, the transform encoding / decoding is used by the scalable video decoder / encoder to encode the enhancement layer residual signal. The transform encoding / decoding is performed in a block-wise (i.e., within the blocks into which the image 22b of the enhancement layer signal is subdivided) manner. FIG. 22 illustrates a corresponding or collocated portion 406 of the base layer signal. Here, the scalable video decoder / encoder applies predictive encoding / decoding to the base layer signal, using transform encoding / decoding on the prediction residual of the base layer signal (i.e., on the base layer residual signal). In particular, a block-wise transform is used on the base layer residual signal. That is, the base layer residual signal is transformed block-wise, with the individually transformed blocks depicted by dotted lines in FIG. 22. As depicted in FIG. 22, the block boundaries of the base layer transform blocks need not coincide with the contour of the collocated portion 406.

[0124] Nonetheless, to perform the evaluation 408, one or a combination of the following options AC may be used.

[0125] In particular, the scalable video decoder / encoder may perform a transform 422 on the base layer residual signal or the reconstructed base layer signal in portion 406 to obtain a transform coefficient block 424 of transform coefficients that matches in size to the transform coefficient block 402 to be encoded / decoded. Examination of the distribution of values ​​of the transform coefficients in the transform coefficient blocks 424, 426 may be used to appropriately set dimensions of the sub-blocks 412 along the horizontal frequency direction 428 and along the vertical frequency direction 432.

[0126] Additionally or alternatively, the scalable video decoder / encoder may examine all transform coefficient blocks of the base layer transform block 434 depicted by different diagonal lines in FIG. 22 that at least partially overlap the collocated portion 406. In the exemplary case of FIG. 22, there are four base layer transform blocks, whose transform coefficient blocks are then examined. In particular, all of these base layer transform blocks may be of different sizes from each other and may further differ in size with respect to the transform coefficient block 412. Scaling 436 may be performed on these transform coefficient blocks that overlap the base layer transform block 434 to provide an approximation of the transform coefficient block 438 of the spectral decomposition of the base layer residual signal in the portion 406. The distribution of values ​​of the transform coefficients in that transform coefficient block 438 (i.e., 442) may be used in the evaluation 408 to appropriately set the dimensions 428 and 432 of the sub-blocks. As a result, a sub-division of the sub-blocks of the transform coefficient block 402 is selected.

[0127] To perform the evaluation 408, additionally or alternatively, a further alternative may be to examine the base layer residual signal or the reconstructed base layer signal in the spatial domain using edge detection 444 or a determination of a dominant gradient direction, for example based on the extension direction of edges detected in the co-located portion 406 or a determined gradient, to appropriately set the sub-block dimensions 428 and 432.

[0128] Although not explicitly described above, in traversing the positions of the transform coefficients and the units of the sub-blocks 412, it is preferable to traverse the sub-blocks 412 in an order starting from the zero frequency corner of the transform coefficient block (top left corner of FIG. 21) to the highest frequency corner of the block 402 (bottom right corner of FIG. 21). Furthermore, entropy coding may be used to signal the syntax elements in the data stream 6b. That is, the syntax elements 416 and 418 may be conveniently entropy coded, such as arithmetic, variable length coding or another form of entropy coding. The ordering of traversing the sub-blocks 412 may also depend on the sub-block shape selected according to 408. For sub-blocks selected to be wider than their height, the order of traversal may first traverse the sub-blocks in a column direction, then proceed to the next column, and so on. Beyond this, it should be noted again that the base layer information used to select the dimensions of the sub-blocks may be the base layer residual signal or the reconstructed base layer signal itself.

[0129] In the following, different embodiments are described that are combined with the above-described aspects. The embodiments described below make scalable video coding more efficient in relation to many different aspects or means. In part, the above aspects are described in detail below while keeping the general concept, to present alternative derived embodiments thereof. These below-presented descriptions can be used to obtain alternatives or extensions of the above embodiments / aspects. However, most of the embodiments described below relate to sub-aspects and are optionally combined with the aspects already described above, i.e., they are implemented together with the above embodiments in one scalable video decoder / encoder at the same time. However, this is not required.

[0130] In order to make the previous description easier to understand, more detailed embodiments for implementing a suitable scalable video encoder / decoder incorporating embodiments or combinations of embodiments are presented below. The different embodiments described below are enumerated by the use of alphanumeric symbols. The description of some of these aspects refers to elements in the now described figures in which these aspects may be implemented in common according to one embodiment. However, as far as the individual embodiments are concerned, the provision of each element in the implementation of the scalable video decoder / encoder is not necessary as far as every aspect is concerned. Depending on the aspect in question, some elements and some interconnections may be omitted in the drawings described below. Only the elements referenced for each aspect are provided to perform the tasks or functions mentioned in the description of the respective aspect. However, in particular when several elements are enumerated for one function, alternatives may exist.

[0131] However, in order to provide an overview of the functionality of a scalable video decoder / encoder, the following described aspects are implemented: The elements shown in the following figures will now be briefly described.

[0132] 23 illustrates a scalable video decoder for decoding an encoded data stream 6 in which video is encoded in such a way that a main sub-portion (i.e., 6a) of the encoded data stream 6 represents video at a first resolution or quality level. Additional portions 6b of the encoded data stream correspond to representations of video at increasing resolution or quality levels. To keep the amount of data in the encoded data stream 6 low, inter-layer redundancy between sub-streams 6a and 6b is exploited in forming sub-stream 6b. Some of the aspects described below are directed to inter-layer prediction from an associated base layer with which sub-stream 6a is associated, and to an associated enhancement layer with which sub-stream 6b is associated.

[0133] The scalable video decoder includes two block-based predictive decoders 80, 60 running in parallel, receiving the sub-streams 6a and 6b, respectively. As shown in the figure, the demultiplexer 40 may provide separate decoding stages 80 and 60 with the corresponding sub-streams 6a and 6b.

[0134] The internal structure of the block-based predictive coding stages 80 and 60 may be the same, as shown. From the input of each decoding stage 80, 60, the entropy coding module 100; 320, the inverse transformer 560; 580, the adder 180; 340, and the optional filters 120; 300 and 140; 280 are connected in series in the order of description. As a result, at the end of this series connection, a reconstructed base layer signal 600 and a reconstructed enhancement layer signal 360 may be derived, respectively. Meanwhile, the outputs of the adder 180, 340 and the filters 120, 140, 300, 280 provide different versions of the reconstruction of the base layer signal and the enhancement layer signal, respectively. Then, each prediction provider 160; 260 receives a subset or all of these versions and provides a prediction signal based thereon for providing to the residual input of the adder 180; 340, respectively. The entropy decoding stages 100;320 decode from their respective input signals 6a and 6b, respectively, the transform coefficient blocks which go to the inverse transformer 560;580 which encodes parameters including prediction parameters for the prediction provider 160;260.

[0135] Thus, prediction providers 160 and 260 predict blocks of frames of video at their respective resolution / quality levels, and for this purpose, prediction providers 160 and 260 may be selected within a given prediction mode, such as a spatial intra prediction mode or a temporal inter prediction mode, both of which are intra-layer prediction modes (i.e., prediction modes that depend solely on data in the sub-stream in which the respective level resides).

[0136] However, in order to take advantage of the aforementioned inter-layer redundancy, the enhancement layer decoding stage 60 additionally includes a coding parameter inter-layer predictor 240, a resolution / quality improver 220, and / or a prediction provider 260, which is compared to the prediction provider 160. Additionally / alternatively, the enhancement layer decoding stage 60 supports an inter-layer prediction mode that can provide an enhancement layer prediction signal 420 based on data derived from an internal stage of the base layer decoding stage 80. The resolution / quality improver 220 subjects either the reconstructed base layer signal 200a, 200b, 200c or the base layer residual signal 480 to a resolution or quality improvement to obtain an inter-layer prediction signal 380. The coding parameter inter-layer predictor 240 is to predict in some way coding parameters such as prediction parameters and motion parameters, respectively. The prediction provider 260 further supports an inter-layer prediction mode, for example according to a reconstructed portion of the base layer signal such as 200a, 200b, 200c. Alternatively, reconstructed portions of the base layer residual signal 640, potentially refined at increasing resolution / quality levels, are used as a reference / basis.

[0137] As mentioned above, the decoding stages 60 and 80 may be operated in a block-based manner. That is, a frame of video may be subdivided into parts such as blocks. Different coarseness levels may be used to assign the prediction mode performed by the prediction provider 160, 260, the local transform by the inverse transformer 560, 580, the filter coefficient selection by the filter 120, 140, and the prediction parameter setting for the prediction mode by the prediction provider 160, 260. That is, the subdivision of a frame into prediction blocks is in turn a continuation of the subdivision of the frame into blocks (e.g., so-called coding units or prediction units) for which a prediction mode is selected. The subdivision of a frame into blocks (so-called transform units) for transform coding may differ from the partition into prediction units. Some of the inter-layer prediction modes used by the prediction provider 260 are described below for examples. The prediction provider 260 is applied based solely on the state relating to the current level encoding stage 60, 80, respectively, of a number of intra-layer prediction modes, i.e., prediction modes that internally derive the respective prediction signals input to the respective adders 180, 340.

[0138] Further details of some of the illustrated blocks will become apparent from the following description of the individual embodiments. It should be noted that, unless such descriptions are explicitly related to the embodiment provided, these descriptions may equally be transferred generally to the description of other embodiments and figures.

[0139] In particular, the embodiment for the scalable video decoder of FIG. 23 represents a possible realization of a scalable video decoder according to FIG. 2 and FIG. 4. Although the scalable video decoder according to FIG. 23 has been described above, FIG. 23 shows the corresponding scalable video encoder, and the same reference numbers are used for the internal elements of the predictive encoding / decoding scheme in FIG. 23 and FIG. 24. The reasons are as explained above. Also, for the purpose of maintaining a common prediction basis between the encoder and the decoder, reconfigurable versions of the base and enhancement layer signals are used in the encoder to reconstruct the parts already encoded up to this end to obtain a reconfigurable version of the scalable video. Thus, the only difference from the description of FIG. 23 is that the prediction provider 160 and the prediction provider 260, as well as the coding parameter inter-layer predictor 240, determine the prediction parameters within some ratio / distortion optimization process, rather than receiving the prediction parameters from the data stream. Rather, the providers transmit the prediction parameters thus determined to the entropy decoders 19a and 19b. The entropy decoders 19a and 19b in turn transmit the respective base layer sub-stream 6a and enhancement layer sub-stream 6b via the multiplexer 16 for inclusion in the data stream 6. In a similar manner, these entropy encoders 19a and 19b receive the reconstructed base layer signal 200 and the reconstructed enhancement layer signal 400 and the prediction residual between the original base layer and enhancement layer versions 4a, 4b, so that the transformation modules 724, 726 obtain the prediction residual between the reconstructed base layer signal 200 and the reconstructed enhancement layer signal 400 via subsequent subtractors 720 and 722, rather than outputting the entropy decoding result of such residual. However, apart from this, the structure of the scalable video encoder of FIG. 24 corresponds to the structure of the scalable video decoder of FIG. 23. Therefore, with regard to these issues, reference is made to the above description of FIG. 23. Here, as just outlined, any derivation from any data stream must be transformed into a respective determination of the respective element with subsequent insertion into the respective data stream.

[0140] The techniques for intra-coding of enhancement layer signals used in the embodiments described below include multiple methods for generating an intra-prediction signal (using base layer data) for an enhancement layer block, which are provided in addition to a method for generating an intra-prediction signal based solely on reconstructed enhancement layer samples.

[0141] Intra prediction is part of the process of reconstruction of an intra-coded block. The final reconstructed block is obtained by adding the transform coded residual signal (which may be zero) to the intra prediction signal. The residual signal is generated by inverse quantization (scaling) of the transform coefficient levels transmitted in the bitstream followed by an inverse transform.

[0142] The following description applies to scalable coding with quality enhancement layers (where the enhancement layers have the same resolution as the base layer but represent an input video with higher quality or fidelity) and to scalable coding with spatial enhancement layers (where the enhancement layers have a higher resolution, i.e., a larger number of samples, than the base layer). In the case of quality enhancement layers, no upsampling of the base layer signal is required, such as in block 220, although filtering 500, etc. of the reconstructed base layer samples may be applied. In the case of spatial enhancement layers, upsampling of the base layer signal is generally required, for example in block 220.

[0143] The next described aspects support different methods to use reconstructed base layer samples (vs. 200) or base layer residual samples (vs. 640) for intra prediction of enhancement layer blocks. One or more of the methods described below can be supported in addition to intra-layer intra coding (where only reconstructed enhancement layer samples (vs. 400) are used for intra prediction). The use of a particular method may be signaled at the level of the largest supported block size (such as the size of a block / largest coding unit in H.264 / AVC or coding tree in HEVC), or it may be signaled for all supported block sizes, or it may be signaled for a subset of the supported block sizes.

[0144] For all methods described below, the prediction signal may be used directly as the reconstruction signal for the block, i.e. the residual is not transmitted at all. Or the selected method for inter-layer intra prediction may be combined with residual coding. In a particular embodiment, the residual signal is transmitted via transform coding, i.e. the quantized transform coefficients (transform coefficient levels) are transmitted using an entropy coding technique (e.g. variable length coding or arithmetic coding (cf. 19b)). The residual is then obtained by inverse quantizing (scaling) the transmitted transform coefficient levels and applying an inverse transform (cf. 580). In a particular version, the complete residual block corresponding to the block from which the inter-layer intra prediction signal originates is transformed using one transform (cf. 726). That is, the whole block is transformed using one transform of the same size as the prediction block. In another embodiment, the prediction block may be further sub-divided (e.g. using a hierarchical decomposition) into smaller blocks. Then, a separate transform may be applied to each of the smaller blocks (which may also have different block sizes). In another embodiment, the coding unit is partitioned into smaller predictive blocks, and for zero or more of the predictive blocks a prediction signal is generated using one of the methods for inter-layer intra prediction, and then the residual of the entire coding unit is transformed using one transform (as compared to 726), or the coding unit is sub-divided into different transform units, where the sub-division to form the transform units (blocks to which one transform is applied) is different from the sub-division to break down the coding unit into predictive blocks.

[0145] In a particular embodiment, the (upsampled / filtered) reconstructed base layer signal (compared to 380) is directly used as a prediction signal. Multiple methods for using the base layer to intra-predict the enhancement layer include the following methods: The (upsampled / filtered) reconstructed base layer signal (compared to 380) is directly used as an enhancement layer prediction signal. This method is similar to the well-known H.264 / SVC inter-layer intra-prediction mode. In this method, the prediction block for the enhancement layer is formed by collocated samples of the base layer reconstructed signal that are upsampled (compared to 220) and aligned to the corresponding sample positions of the enhancement layer, and optionally filtered before or after upsampling. In contrast to the SVC inter-layer intra-prediction mode, this mode can be supported not only at the macroblock level (or the largest supported block size), but also at any block size. That means that not only can a mode be signaled for the largest supported block size, but also that a block of the largest supported block size (macroblock in MPEG4, H.264 and coding tree block / largest coding unit in HEVC) is hierarchically sub-divided into smaller blocks / coding units, and the use of an inter-layer intra prediction mode can be signaled for any supported block size (for the corresponding block). In a particular embodiment, this mode is only supported for a selected block size. Then, a syntax element signaling the use of this mode can be transmitted only for the corresponding block size. Or, the value of the syntax element signaling the use of this mode (within another coding parameter) can be correspondingly restricted for another block size.Another difference from the inter-layer intra prediction modes in the SVC extension of H.264 / AVC is that the inter-layer intra prediction modes are not only supported when the collocated regions in the base layer are intra-coded, but also when the collocated base layer regions are inter-coded or partially inter-coded.

[0146] In a particular embodiment, spatial intra prediction of the difference signal (see aspect A) is performed. The multiplexing method includes the following method: A (potentially upsampled / filtered) reconstructed base layer signal (compare 380) is combined with a spatial intra prediction signal, where a spatial intra prediction (compare 420) is derived based on difference samples for neighboring blocks (compare 260). The difference samples represent the difference between the reconstructed enhancement layer signal (compare 400) and the (potentially upsampled / filtered) reconstructed base layer signal (compare 380).

[0147] FIG. 25 shows the generation of an inter-layer intra prediction signal by summation 732 of the (upsampled / filtered) base layer reconstructed signal 380 (BL Reco) and spatial intra prediction using a difference signal 734 (EH Diff) of an already coded neighboring block 736. There, a difference signal (EH Diff) for an already coded block 736 is generated by subtracting 738 the (upsampled / filtered) base layer reconstructed signal 380 (BL Reco), where the already coded / decoded part is hatched, from the reconstructed enhancement layer signal (EH Reco) (compared to 400). The current coded / decoded block / region / part is 28. That is, the inter-layer intra prediction method described in FIG. 25 uses two superimposed input signals to generate a prediction block. For this method, a difference signal 734 is required. The difference signal 734 is the difference between the reconstructed enhancement layer signal 400 and the collocated reconstructed base layer signal 200. The base layer signal 200 is upsampled 220 to match the corresponding sample positions of the enhancement layer, and may be optionally filtered before or after upsampling (if no upsampling is applied, it may be filtered, as is the case for quality scalable coding). In particular, for spatial scalable coding, the difference signal 734 usually contains mainly high frequency components. The difference signal 734 is available for all already reconstructed blocks (i.e. all enhancement layer blocks already coded / decoded). The difference signal 734 for neighboring samples 742 of already coded / decoded blocks 736 is used as input to spatial intra prediction techniques (such as spatial intra prediction modes specified in H.264 / AVC or HEVC). The spatial intra prediction, indicated by arrow 744, generates a prediction signal 746 for different components of the block 28 to be predicted. In certain embodiments, any clipping function of the spatial intra prediction process (as known from H.264 / AVC or HEVC) is modified or disabled to match the dynamic range of the difference signal 734.The actually used intra prediction method (one of several provided methods, which may include planar intra prediction with any particular angle, DC intra prediction or directional intra prediction 744) is signaled in the bitstream 6b. It is possible to use spatial intra prediction techniques (methods for generating a prediction signal using samples of already coded neighboring blocks) different from the methods provided in H.264 / AVC and HEVC. The resulting prediction block 746 (using difference samples of the neighboring blocks) is the first part of the final prediction block 420.

[0148] A second part of the prediction signal is generated using the collocated region 28 in the reconstructed signal 200 of the base layer. For the quality enhancement layer, the collocated base layer samples are used directly or are optionally filtered, for example by a low-pass filter or a filter 500 that attenuates high frequency components. For the spatial enhancement layer, the collocated base layer samples are upsampled. For the upsampling 220, a FIR filter or a set of FIR filters can be used. An IIR filter can also be used. Optionally, the reconstructed base layer samples 200 can be filtered before upsampling. Alternatively, the base layer prediction signal (the signal obtained after upsampling the base layer) is filtered after the upsampling stage. The processing of the base layer reconstruction can include one or more additional filters, such as a deblocking filter (compared to 120) or an adaptive loop filter (compared to 140). The base layer reconstruction 200 used for upsampling can be the reconstructed signal before any loop filter (compared to 200c). Alternatively, it may be the reconstructed signal after the deblocking filter but before any other filters (compare 200b), or it may be the reconstructed signal after a particular filter, or after applying all filters used in the base layer decoding process (compare 200a).

[0149] The two generated parts of the prediction signal (the spatially predicted difference signal 746 and the potentially filtered / upsampled base layer reconstruction 380) are added 732 sample-by-sample to form the final prediction signal 420.

[0150] A transfer of the just outlined aspects to the embodiments of Figures 6 to 10 may be that the just outlined possibilities of predicting a current block of the enhancement layer signal are supported by the respective scalable video decoder / encoder as an alternative to the prediction schemes outlined with respect to Figures 6 to 10. The mode used is signaled in the enhancement layer sub-stream 6b via a respective prediction mode identifier, not shown in Figure 8.

[0151] In a particular embodiment, intra prediction follows inter-layer residual prediction (see Example B). Methods for generating an intra prediction signal using base layer data include the following methods: A conventional spatial intra prediction signal (derived using adjacent reconstructed enhancement layer samples) is combined with an (upsampled / filtered) base layer residual signal (the inverse transform of the base layer transform coefficients, or the difference between the base layer reconstruction and the base layer prediction).

[0152] Figure 26 shows the generation of an inter-layer intra prediction signal 420 by summation 752 of an (upsampled / filtered) base layer residual signal 754 (BL Resi) and spatial intra prediction 756 using reconstructed enhancement layer samples 758 (EH Reco) of already encoded neighboring blocks depicted by dotted line 762.

[0153] The concept illustrated in FIG. 26 overlaps two prediction signals to form a prediction block 420, where one prediction signal 764 originates from already reconstructed enhancement layer samples 758 and the other prediction signal 754 originates from base layer residual samples 480. A first part 764 of the prediction signal 420 is derived by applying spatial intra prediction 756 using the reconstructed enhancement layer samples 758. The spatial intra prediction 756 can be one of the methods specified in H.264 / AVC. Or it is one of the methods specified in HEVC. Or it can be another spatial intra prediction technique that generates a prediction signal 764 for the current block 18 that forms the samples 758 of the neighboring block 762. The actually used intra prediction method 756 (which can be one of several provided methods and can include planar intra prediction, DC intra prediction, or directional intra prediction with any particular angle) is signaled in the bitstream 6b. It is possible to use spatial intra prediction techniques (methods for generating a prediction signal using samples of neighboring blocks already coded) different from the methods provided for H.264 / AVC and HEVC. A second part 754 of the prediction signal 420 is generated using the collocated residual signal 480 of the base layer. For the quality enhancement layer, the residual signal may be used to be reconstructed in the base layer. Or the residual signal can be additionally filtered. For the spatial enhancement layer 480, the residual signal is upsampled 220 (to map the base layer sample positions to the enhancement layer sample positions) before it is used as the second part of the prediction signal. Also, the base layer residual signal 480 can be filtered before or after the upsampling stage. To upsample 220 the residual signal, an FIR filter may be applied. The upsampling process may be configured in such a way that no filtering is performed across the transform block boundaries in the base layer that are applied for the purpose of upsampling.

[0154] The base layer residual signal 480 used for inter-layer prediction may be the residual signal from which the transform coefficient levels of the base layer are obtained by scaling and inverse transform 560. Alternatively, the base layer residual signal 480 may be the difference between the reconstructed base layer signal 200 (before or after deblocking and additional filtering, or during any filtering operation) and the prediction signal 660 used in the base layer.

[0155] The two generated signal components (spatial intra prediction signal 764 and inter-layer residual prediction signal 754) are added 752 to form the final enhancement layer intra prediction signal.

[0156] This means that the prediction modes just outlined with respect to Figure 26 can be used or supported by any scalable video decoder / encoder in accordance with Figures 6 to 10 to form, for the currently encoded / decoded portion 28, the alternative prediction modes described above with respect to Figures 6 to 10.

[0157] In a particular embodiment, weighted prediction of spatial intra prediction and base layer reconstruction (see aspect C) is used, which actually represents the specification published above of a particular implementation of the embodiment outlined above with respect to Figures 6 to 10 and therefore the description of such weighted prediction interpreted not only as an alternative to the above embodiment but also as a description of a possible way of implementing the embodiment outlined above with respect to Figures 6 to 10 differently in certain aspects.

[0158] Multiple methods for generating an intra prediction signal using base layer data include the following: The (upsampled / filtered) reconstructed base layer signal is combined with a spatial intra prediction signal, where a spatial intra prediction is derived based on reconstructed enhancement layer samples of neighboring blocks. The final prediction signal is obtained by weighting (vs. 41) the spatial prediction signal and the base layer prediction signal in such a way that different frequency components use different weightings. This can be realized, for example, by filtering (vs. 62) the base layer prediction signal (vs. 38) with a low-pass filter and filtering (vs. 64) the spatial intra prediction signal (vs. 34) with a high-pass filter and adding (vs. 66) the obtained filtered signals. Alternatively, frequency-based weighting can be achieved by transforming (vs. 72, 74) the base layer prediction signal (vs. 38) and the enhancement layer prediction signal (vs. 34) and overlapping the resulting transform block (vs. 76, 78), with different weighting coefficients (vs. 82, 84) being used for different frequency locations. The resulting transform block (vs. 42 in FIG. 10) is then inverse transformed (vs. 84) and used as the enhancement layer prediction signal (vs. 54). Alternatively, the resulting transform coefficients are added (vs. 52) to the scaled transmitted transform coefficient levels (vs. 59) and inverse transformed (vs. 84) to obtain a reconstructed block (vs. 54) before deblocking and in-loop processing.

[0159] Figure 27 shows the generation of an inter-layer intra prediction signal by frequency-weighted summation of the (upsampled / filtered) base layer reconstructed signal (BL Reco) and spatial intra prediction using reconstructed enhancement layer samples of already encoded neighboring blocks (EH Reco).

[0160] The concept of Fig. 27 uses two superimposed signals 772, 774 to form a prediction block 420. A first part 774 of the signal 420 is derived by applying a spatial intra prediction 776, corresponding to 30 in Fig. 6, using reconstructed samples 778 of an already constructed neighboring block in the enhancement layer. A second part 772 of the prediction signal 420 is generated using a collocated reconstructed signal 200 of the base layer. For the quality enhancement layer, the collocated base layer samples 200 are used directly. Or they can be optionally filtered, for example by a low-pass filter or a filter that attenuates high-frequency components. For the spatial enhancement layer, the collocated base layer samples are upsampled 220. For the upsampling, an FIR filter or a set of FIR filters can be used. It is also possible to use an IIR filter. Optionally, the reconstructed base layer samples are filtered before upsampling. Alternatively, the base layer prediction signal (the signal obtained after upsampling the base layer) may be filtered after the upsampling stage. The base layer reconstruction process may include one or more additional filters, such as the deblocking filter 120 and the adaptive loop filter 140. The base layer reconstruction 200 used for upsampling may be the reconstructed signal 200c before any of the loop filters 120, 140. Alternatively, it may be the reconstructed signal 200b after the deblocking filter 120 but before another filter. Alternatively, it may be the reconstructed signal 200a after a particular filter, or the reconstructed signal after applying all the filters 120, 140 used in the base layer decoding process.

[0161] When the reference numbers used in Figures 23 and 24 are compared with the reference numbers used in connection with Figures 6 to 10, block 220 corresponds to reference number 38 used in Figure 6. 39 corresponds to part 380. At least as far as the parts collocated with the current part 28 are concerned, 420 collocated with the current part 28 corresponds to 42. Spatial prediction 776 corresponds to 32.

[0162] The two prediction signals (potentially upsampled / filtered base layer reconstruction 386 and enhancement layer intra prediction 782) are combined to form the final prediction signal 420. The method for combining these signals can have the property that different weighting factors are used for different frequency components. In a particular embodiment, the upsampled base layer reconstruction is filtered with a low-pass filter (vs. 62) (it is also possible to filter the base layer reconstruction before upsampling 220). And the intra prediction signal (vs. 34 obtained by 30) is filtered with a high-pass filter (vs. 64). Both filtered signals are added 784 (vs. 66) to form the final prediction signal 420. The pair of low-pass and high-pass filters may represent a quadrature mirror filter pair, although this is not necessary.

[0163] In another particular embodiment (vs. Fig. 10), the joint processing of the two prediction signals 380 and 782 is realized via a spatial transformation. Both the (potentially upsampled / filtered) base layer reconstruction 380 and the intra prediction signal 782 are transformed (vs. 72, 74) using a spatial transformation. The transform coefficients (vs. 76, 78) of both signals are then scaled with appropriate weighting factors (vs. 82, 84) and then added (vs. 90) to form the transform coefficient block (vs. 42) of the final prediction signal. In one version, the weighting factors (vs. 82, 84) are selected such that for each transform coefficient position, the sum of the weighting coefficients for the components of both signals is equal to 1. In another version, for some or all transform coefficient positions, the sum of the weighting coefficients is not equal to 1. In a particular version, the weighting coefficients are selected in such a way that, for transform coefficients representing low-frequency components, the weighting coefficients for the base layer reconstruction are greater than the weighting coefficients for the enhancement layer intra-prediction signal, and, for transform coefficients representing high-frequency components, the weighting coefficients for the base layer reconstruction are smaller than the weighting coefficients for the enhancement layer intra-prediction signal.

[0164] In one embodiment, the resulting transform coefficient block (vs. 42) (obtained by combining the weighted transformed signals for both components) is inverse transformed (vs. 84) to form the final prediction signal 420 (vs. 54). In another embodiment, the prediction is made directly in the transform domain, i.e. the coded transform coefficient levels (vs. 59) are scaled (i.e. inverse quantized) and added (vs. 52) to the transform coefficients (vs. 42) of the prediction signal (obtained by adding the weighted transformed signals for both components) and then to the resulting block of transform coefficients (not shown in FIG. 10 but inverse transformed (vs. 84) (before the potential deblocking 120 and further in-loop filtering step 140)) to obtain the reconstructed signal 420 for the current block. In other words, in the first embodiment, the transform block obtained by adding the weighted transformed signals for both components can be inverse transformed and used as the enhancement layer prediction signal. Alternatively, in a second embodiment, it may be added to the scaled transmitted transform coefficient levels and inverse transformed to obtain the reconstructed block before deblocking and in-loop processing.

[0165] The selection between base layer reconstruction and the residual signal (see aspect D) may also be used. For the method using the reconstructed base layer signal (as described above), the following version is used.

[0166] · The reconstructed base layer samples 200c before deblocking 120 and further in-loop processing 140 (such as a sample adaptive offset filter or an adaptive loop filter). · The reconstructed base layer samples 200b after deblocking 120 but before further in-loop processing 140 (such as a sample adaptive offset filter or an adaptive loop filter). · Reconstructed base layer samples 200a after deblocking 120 and further in-loop processing 140 (such as a sample adaptive offset filter or an adaptive loop filter) or between multiple in-loop processing steps.

[0167] The selection of the corresponding base layer signal 200a,b,c may be fixed for a particular decoder (and encoder) implementation, or it may be signaled within the bitstream 6. For the latter case, different versions may be used. The use of a particular version of the base layer signal may be signaled at the sequence level, or at the picture level, or at the slice level, or at the largest coding unit level, or at the coding unit level, or at the predictive block level, or at the transform block level, or at any other block level. In another version, the selection may depend on other coding parameters (such as the coding mode) or on characteristics of the base layer signal.

[0168] In another embodiment, multiple versions of the method of using the (upsampled / filtered) base layer signal 200 may be used. For example, two different modes may be provided that directly use the upsampled base layer signal (i.e., 200a), where the two modes may use different interpolation filters. Or, one mode uses additional filtering 500 of the (upsampled) base layer reconstruction signal. Similarly, multiple different versions for other modes mentioned above may be provided. The upsampled / filtered base layer signal 380 employed for the different versions of the mode may differ in the interpolation filters used (including interpolation filters that also filter integer sample positions). Or, the upsampled / filtered base layer signal 380 for the second version may be obtained by filtering 500 the upsampled / filtered base layer signal for the first version. The selection of one of the different versions may be signaled at the sequence, picture, slice, maximum coding unit, coding unit level, prediction block level, or transform block level. Alternatively, it may be inferred from characteristics of the corresponding reconstructed base layer signal or the transmitted coding parameters.

[0169] The same applies to the mode using the reconstructed base layer residual signal, via 480, where different versions of the interpolation filters or additional filtering steps may also be used.

[0170] Different filters may be used for upsampling / filtering the reconstructed base layer signal and the base layer residual signal, which means that a different approach is used for upsampling the base layer residual signal than for upsampling the base layer reconstructed signal.

[0171] For base layer blocks, the residual signal is zero (i.e., no transform coefficient levels are transmitted for the block). The corresponding base layer residual signal may be replaced with another signal derived from the base layer. For example, this may be a high-pass filtered version of the reconstructed base layer block or another differential signal derived from reconstructed base layer samples of adjacent blocks or reconstructed base layer residual samples.

[0172] As far as samples used for spatial intra prediction in the enhancement layer (see aspect H) are concerned, the following special handling is provided: For modes using spatial intra prediction, unavailable neighboring samples in the enhancement layer (neighboring blocks are coded after the current block, so the neighboring samples are unavailable) may be replaced with corresponding samples of the upsampled / filtered base layer signal.

[0173] As far as the coding of intra prediction modes (see aspect X) is concerned, the following special modes and functionalities may be provided: For modes using spatial intra prediction such as 30a, the coding of intra prediction modes may be modified in a way that information about the intra prediction modes in the base layer (if available) is used to code the intra prediction modes in the enhancement layer more efficiently. This may be used, for example, for parameter 56. If a collocated region in the base layer (compared to 36) is intra coded using a particular spatial intra prediction mode, it is likely that a similar intra prediction mode will be used in the enhancement layer block (compared to 28). Intra prediction modes are usually signaled in such a way that within the set of possible intra prediction modes, one or more modes are classified as maximum likelihood modes, where they may be signaled with shorter codewords or fewer arithmetic coding decisions resulting in fewer bits. In HEVC intra prediction, the intra prediction mode of the block above (if available) and the intra prediction mode of the block to the left (if available) are included in the set of maximum likelihood modes. In addition to these modes, one or more additional modes (often used) are included in the list of maximum likelihood modes, where the actual additional mode depends on the availability of the intra prediction modes of the block above the current block and the block to the left of the current block. In HEVC, three modes are classified as maximum likelihood modes exactly. In H.264 / AVC, one mode is classified as maximum likelihood mode. This mode is derived based on the intra prediction mode used for the block above the current block and the block to the left of the current block. Any other concept for classifying intra prediction modes (different from H.264 / AVC and HEVC) is possible and can be used for the following extensions.

[0174] In order to use base layer data for efficient coding of intra-prediction modes in the enhancement layer, the concept of using one or more maximum likelihood modes is modified in a way that the maximum likelihood modes include the intra-prediction modes used in the co-located base layer block (if the corresponding base layer block is intra-coded). In a particular embodiment, the following approach may be used: Given a current enhancement layer block, a co-located base layer block is determined. In a particular version, the co-located base layer block is a base layer block that covers a co-located position of the top-left sample of the enhancement block. In another version, the co-located base layer block is a base layer block that covers a co-located position of the central sample of the enhancement block. In another version, another sample in the enhancement layer block may be used to determine the co-located base layer block. If the determined co-located base layer block is intra-coded, the base layer intra-prediction mode specifies an angular intra-prediction mode, and the intra-prediction mode derived from the enhancement layer block to the left of the current enhancement layer block does not use an angular intra-prediction mode, then the intra-prediction mode derived from the left enhancement layer block is replaced with the corresponding base layer intra-prediction mode. Otherwise, if the determined co-located base layer block is intra-coded, the base layer intra-prediction mode specifies an angular intra-prediction mode, and the intra-prediction mode derived from the enhancement layer block above the current enhancement layer block does not use an angular intra-prediction mode, then the intra-prediction mode derived from the enhancement layer block above the current enhancement layer block is replaced with the corresponding base layer intra-prediction mode. In other versions, a different approach is used to modify the list of (single-element) maximum likelihood modes using the base layer intra-prediction mode.

[0175] Inter-coding techniques for spatial and quality enhancement layers are then presented.

[0176] In state-of-the-art hybrid video coding standards (such as H.264 / AVC or the upcoming HEVC), the images of an image sequence are divided into blocks of samples. The block size may be fixed or the coding scheme may provide a hierarchical structure that allows further subdivision of the blocks into blocks with smaller block sizes. The reconstruction of a block is usually obtained by generating a prediction signal for the block and adding a transmitted residual signal. The residual signal is usually transmitted using transform coding, which means that the quantization indexes for the transform coefficients (also called transform coefficient levels) are transmitted using an entropy coding technique. Then, at the decoder side, these transmitted transform coefficient levels are scaled and inverse transformed to obtain a residual signal that is added to the prediction signal. The residual signal is generated either by intra prediction (using only data already transmitted for the current time instant) or by inter prediction (using data already transmitted for a different time instant).

[0177] In inter prediction, a prediction block is derived by motion compensated prediction using samples of an already reconstructed frame. This is done by unidirectional prediction (using one reference image and one set of motion parameters). Alternatively, the prediction signal can be generated by multi-hypothesis prediction. In the latter case, two or more prediction signals are superimposed. That is, for each sample, a weighted average is constructed to form the final prediction signal. The (superimposed) multi-prediction signal can be generated using different motion parameters (e.g., different reference images or motion vectors) for different hypotheses. It is also possible to multiply samples of the motion compensated prediction signal with a constant coefficient for unidirectional prediction and add a constant offset to form the final prediction signal. Such scaling and offset correction can also be used for all or selected hypotheses in the multi-hypothesis prediction.

[0178] Also in scalable video coding, base layer information can be utilized to support inter prediction processing for enhancement layers. The state-of-the-art video coding standard for scalable coding, the SVC extension of H.264 / AVC, has one additional mode to improve the coding efficiency of inter prediction processing in the enhancement layer. This mode is signaled at the macroblock level (blocks of 16x16 luma samples). In this mode, reconstructed residual samples in the low layer are used to improve the motion compensation prediction signal in the enhancement layer. This mode is also called inter-layer residual prediction. If this mode is selected for a macroblock in a quality enhancement layer, the inter-layer prediction signal is assembled by collocated samples of the reconstructed low layer residual signal. If the inter-layer residual prediction mode is selected in a spatial enhancement layer, the prediction signal is generated by upsampling the collocated reconstructed base layer residual signal. For the upsampling, an FIR filter is used. However, filtering is not applied across transform block boundaries. A prediction signal originating from the reconstructed base layer residual samples is added to a conventional motion compensated prediction signal to form a final prediction signal for the enhancement layer block. In general, for inter-layer residual prediction modes, an additional residual signal is transmitted by transform coding. The transmission of the residual signal may be omitted (inferred to be equal to zero) if it is correspondingly signaled in the bitstream. The final reconstruction signal is obtained by adding a reconstructed residual signal obtained by scaling the transmitted transform coefficient levels and applying an inverse spatial transform to the prediction signal, where the inter-layer residual prediction signal is obtained by adding the motion compensated prediction signal.

[0179] Next, techniques for inter-coding of enhancement layer signals are described. This section describes methods for using a base layer signal in addition to an already reconstructed enhancement layer signal to intra-predict an enhancement layer signal to be coded in a scalable video coding scenario. By using the base layer signal for inter-predicting the enhancement layer signal to be coded, the prediction error can be suppressed sufficiently, which results in an overall bit rate saving for the coding of the enhancement layer. The main focus of this section is to augment the block-based motion compensation of the enhancement layer samples using already coded enhancement layer samples with an additional signal from the base layer. The following description provides possibilities for using various signals from the coded base layer. Although the quadtree block partition is generally adopted as the preferred embodiment, the presented example applies to a general block-based hybrid coding approach without assuming any specific block partition. The use of the base layer reconstruction of the current time index, the base layer residual of the current time index, or even the base layer reconstruction of the already coded picture for the inter-prediction of the enhancement layer block to be coded is described. Also described is how the base layer signal can be combined with an already coded enhancement layer signal to obtain a better prediction for the current enhancement layer. One of the main techniques in the state of the art is inter-layer residual prediction in H.264 / SVC. Inter-layer residual prediction in H.264 / SVC can be employed for all inter-coded macroblocks, regardless of whether they are coded using SVC macroblock types, signaled by using either the base mode flag or conventional macroblock types. A flag is added to the macroblock syntax for spatial and quality enhancement layers, signaling the usage of inter-layer residual prediction. When this residual prediction flag is equal to 1, the residual signal of the corresponding region in the reference layer is block-wise upsampled using a bilinear filter and used as a prediction for the residual signal of the enhancement layer macroblock. As a result, only the corresponding difference signal needs to be coded in the enhancement layer. For the descriptions in this section, the following notation will be used: t0:= time index of the current image t1:=time index of an already reconstructed image EL:=Enhancement Layer BL:= Base Layer EL(t0):=current enhancement layer image to be coded EL_reco:=Enhancement layer reconstruction BL_reco:= Base layer reconstruction BL_resi:= Base layer residual signal (the inverse transform of the base layer transform coefficients, or the difference between the base layer reconstruction and the base layer prediction) EL_diff:=The difference between the enhancement layer reconstruction and the upsampled / filtered base layer reconstruction Different base layer and enhancement layer signals are used in the description illustrated in FIG.

[0180] For the description the following properties of the filter are used: Linearity: Although many of the filters mentioned within the specification are linear, non-linear filters may also be used. Number of output samples: In an upsampling operation, the number of output samples is greater than the number of input samples. Here, filtering of the input data produces more samples than the input values. In conventional filtering, the number of output samples is equal to the number of samples of the input. Such filtering operations can be used, for example, in high-quality scalable coding. Phase delay: For filtering samples at integer positions, the phase delay is usually zero (or an integer number of delays in samples). For samples occurring at fractional positions (e.g. half-pel or quarter-pel positions), a filter with a fractional delay (in units of samples) is usually applied to the samples of the integer lattice.

[0181] Conventional motion compensation prediction, used in all hybrid video coding standards (e.g. MPEG-2, H.264 / AVC or the upcoming HEVC standard), is illustrated in FIG. 29. To predict the signal of the current block, an area of ​​an already reconstructed image is replaced and used as a prediction signal. For the signaling of the replacement, a motion vector is usually coded in the bitstream. For integer sample precision motion vectors, a reference area in a reference image can be directly copied to form the prediction signal. However, it is also possible to transmit fractional sample precision motion vectors. In this case, the prediction signal is obtained by filtering the reference signal with a filter having a fractional sample delay. The reference image used can usually be specified by including a reference image index in the bitstream syntax. In general, it is also possible to superimpose two or more prediction signals to form a final prediction signal. The concept is supported in B slices, for example with two motion hypotheses. In this case, multiple prediction signals are generated using different motion parameters (e.g. different reference images or motion vectors) for the different hypotheses. For unidirectional prediction, it is also possible to multiply samples of the motion compensated prediction signal with a constant factor and add a constant offset to form a final prediction signal. Such scaling and offset correction can also be used for all or selected hypotheses in multi-hypothesis prediction.

[0182] The following description applies to scalable coding with quality enhancement layers (where the enhancement layers have the same resolution as the base layer but represent the input video with higher quality or fidelity) and to scalable coding with spatial enhancement layers (where the enhancement layers have a higher resolution, i.e., a larger number of samples, than the base layer). For quality enhancement layers, no upsampling of the base layer signal is required, although filtering of the reconstructed base layer samples may be applied. In the case of spatial enhancement layers, upsampling of the base layer signal is generally required.

[0183] The embodiments support different methods for using reconstructed base layer samples or base layer residual samples for inter prediction of enhancement layer blocks. In addition to conventional inter prediction and intra prediction, it is possible to support one or more of the methods described below. The usage of a particular method may be signaled at the level of the largest supported block size (such as macroblock in H.264 / AVC, or coding tree block / largest coding unit in HEVC). It may be signaled for all supported block sizes. Or, it may be signaled for a subset of supported block sizes.

[0184] For all methods described below, the prediction signal may be used directly as a reconstruction signal for a block. Alternatively, the selected method for inter-layer inter prediction may be combined with residual coding. In a particular embodiment, the residual signal is transmitted via transform coding, i.e., the quantized transform coefficients (transform coefficient levels) are transmitted using an entropy coding technique (e.g., variable length coding or arithmetic coding), and the residual is obtained by inverse quantizing (scaling) the transmitted transform coefficient levels and applying an inverse transform. In a particular version, the complete residual block corresponding to the block from which the inter-layer inter prediction signal is generated is transformed using a single transform, i.e., the entire block is transformed using a single transform of the same size as the prediction block. In another embodiment, the prediction block may be further sub-divided into smaller blocks, for example using a hierarchical decomposition, and a separate transform is applied to each smaller block with a different block size. In another embodiment, the coding unit may be divided into smaller prediction blocks, and for zero or more of the prediction blocks, a prediction signal is generated using one of the methods for inter-layer inter prediction. The residual of the entire coding unit is then transformed using a single transform, or the coding unit is subdivided into different transform units, where the subdivision to form the transform units (blocks to which a single transform is applied) is different from the subdivision to decompose the coding unit into predictive blocks.

[0185] In the following, the possibility of performing prediction using the base layer residual and the enhancement layer reconstruction is described. Multiplexing methods include the following methods: A conventional inter prediction signal (obtained by interpolation of the already reconstructed motion compensated image of the enhancement layer) is combined with an (upsampled / filtered) base layer residual signal (the inverse transform of the base layer transform coefficients or the difference between the base layer reconstruction and the base layer prediction). This method is also called the "BL_resi" mode (by analogy with Figure 30).

[0186] In summary, the predictions for the enhancement layer samples can be described as follows: EL prediction=filter(BL_resi(t0))+MCP_filter(EL_reco(t1)) It is also possible that more than one hypothesis of the enhancement layer reconstruction signal is used. For example, EL prediction=filter(BL_resi(t0))+MCP_filter1(EL_reco(t1))+MCP_filter2(EL_reco(t2)) The motion compensated prediction (MCP) filter used on the enhancement layer (EL) reference picture may be of integer or fractional sample accuracy. The MCP filter used on the EL reference picture may be the same or different from the MCP filter used on the BL reference picture during the BL decoding process. A motion vector MV(x,y,t) is defined to indicate a specific location in an EL reference picture. The parameters x and y indicate the spatial location in the picture. The parameter t is used to describe the time index of the reference picture, also called the reference index. Often, the term motion vector is used to refer to only the two spatial components (x,y). The integer part of the MV is used to fetch a set of samples from the reference picture, and the fractional part of the MV is used to select an MCP filter from a set of filters. The obtained reference samples are filtered to produce filtered reference samples. Motion vectors are generally coded using differential prediction, which means that a motion vector predictor is derived based on already coded motion vectors (and a syntax element potentially indicates the used one of a set of potential motion vector predictors), and the differential vector is included in the bitstream. The final motion vector is obtained by adding the transmitted motion vector difference to the motion vector predictor. Usually, it is also possible to completely derive the motion parameters for a block. Therefore, usually, a list of potential motion parameter candidates is constructed based on already coded data. This list may include motion parameters of spatially neighboring blocks as well as motion parameters derived based on motion parameters in collocated blocks in a reference frame. The base layer (BL) residual signal can be defined as one of the following: Inverse transformation of the BL transform coefficients, or The difference between the BL reconstruction and the BL prediction, or For BL blocks where the inverse transform of the BL transform coefficients is zero, it can be replaced by another signal derived from BL (e.g. a high-pass filtered version of the reconstructed BL block), or A combination of the above methods. To calculate the EL prediction element from the current BL residual, a region in the BL image collocated with the region considered in the EL image is identified, and the residual signal is taken from the identified BL region. The definition of the collocated region can be made such that it accounts for an integer scaling factor of the BL resolution (e.g., 2× scalability), or a fractional scaling factor of the BL resolution (e.g., 1.5× scalability), or even yields an EL resolution that is the same as the BL resolution (e.g., quality scalability). In the case of quality scalability, the collocated block in the BL image has the same coordinates as the EL block to be predicted. The collocated BL residual may be upsampled / filtered to generate filtered BL residual samples. The final EL prediction is obtained by adding the filtered EL reconstruction samples and the filtered BL residual samples.

[0187] Multiple methods for prediction using base layer reconstruction and enhancement layer difference signal (see embodiment J) include the following method: The (upsampled / filtered) reconstructed base layer signal is combined with a motion compensated prediction signal, where the motion compensated prediction signal is obtained by a motion compensated difference image. The difference image represents the difference between the reconstructed enhancement layer signal and the (upsampled / filtered) reconstructed base layer signal with respect to a reference image. This method is also called BL_reco mode.

[0188] This concept is illustrated in Figure 31. In summary, predictions for the EL samples are described below. EL prediction=filter(BL_reco(t0))+MCP_filter(EL_diff(t1))

[0189] It is also possible that more than one hypothesis of the EL difference signal is used. For example, EL prediction=filter(BL_resi(t0))+MCP_filter1(EL_diff(t1))+MCP_filter2(EL_diff(t2))

[0190] For the EL difference signal the following version can be used: The difference between the EL reconstruction and the upsampled / filtered BL reconstruction, or The difference between the EL reconstruction before or during the loop filtering stage (like deblocking, SAO, ALF) and the upsampled / filtered BL reconstruction.

[0191] The usage of a particular version may be fixed within the decoder, or it may be signaled at the sequence level, picture level, slice level, largest coding unit level, coding unit level, or another partition level, or it may depend on other coding parameters.

[0192] When the EL difference signal is defined using the difference between the EL reconstruction and the upsampled / filtered BL reconstruction, this allows us to store the EL and BL reconstructions and compute the EL difference signal on the fly for blocks using prediction mode, thus saving the memory required to store the EL difference signal. However, it incurs a small computational overhead.

[0193] The MCP filters used on the EL difference images can be of integer or fractional sample accuracy. · The MCP of the difference image can use a different interpolation filter than the MCP of the reconstructed image. For MCP of the difference image, an interpolation filter may be selected based on the characteristics of the corresponding region in the difference image (or based on coding parameters in the bitstream, or based on transmitted information).

[0194] A motion vector MV(x,y,t) is defined to point to a specific location in the EL difference image. The parameters x and y refer to the spatial position within the image, and the parameter t is used to specify the time index of the filtered difference image.

[0195] The integer part of the MV is used to obtain a set of samples from the difference image, and the fractional part of the MV is used to select an MCP filter from a set of filters. The obtained difference samples are filtered to produce filtered difference samples.

[0196] The dynamic range of the difference image can theoretically exceed that of the original image. If we assume an 8-bit representation of the image in the range

[0255] , the difference image can have the range [-255 255]. In practice, however, most of the amplitudes are distributed around positive and negative 0. In the preferred embodiment for storing the difference image, a constant offset of 128 is added and the result is clipped to the range

[0255] and stored as a regular 8-bit image. Then, during the encoding and decoding process, the offset of 128 is subtracted from the difference amplitudes read from the difference image.

[0197] For the method using the reconstructed BL signal, the following versions can be used: it is fixed, or it is signaled at the sequence level, the picture level, the slice level, the maximum coding unit level, the coding unit level, or at another partition level, or it can depend on another coding parameter. Reconstructed base layer samples before deblocking and further in-loop processing (e.g. sample adaptive offset filter or adaptive loop filter). The reconstructed base layer samples after deblocking, but before further in-loop processing (such as a sample adaptive offset filter or adaptive loop filter). The reconstructed base layer samples after deblocking and further in-loop processing (such as a sample-adaptive offset filter or an adaptive loop filter), or between multiple in-loop processing steps.

[0198] To calculate the EL prediction component from the current BL reconstruction, a region in the BL image collocated with the considered region in the EL image is identified. Then, the reconstructed signal is taken from the identified BL region. The definition of the collocated region is made such that it accounts for an integer scaling factor of the BL resolution (e.g., 2× scalability), or a fractional scaling factor of the BL resolution (e.g., 1.5× scalability), or even yields the same EL resolution as the BL resolution (e.g., SNR scalability). In the case of SNR scalability, the collocated block in the BL image has the same coordinates as the EL block to be predicted.

[0199] The final EL prediction is obtained by adding the filtered EL difference samples and the filtered BL reconstruction samples.

[0200] Some possible variations of modes of combining the (upsampled / filtered) base layer reconstruction signal and the motion compensated enhancement layer difference signal are described below. · Multiple versions of the method of using the (upsampled / filtered) BL signal can be used. The upsampled / filtered BL signal employed for these versions can differ from the interpolation filter used (including an interpolation filter that also filters integer sample positions), or the upsampled / filtered BL signal for the second version can be obtained by filtering the upsampled / filtered BL signal for the first version. The selection of one of the different versions can be signaled at the sequence, picture, slice, maximum coding unit, coding unit level or at another level of the picture partition. Alternatively, it can be inferred from the characteristics of the corresponding reconstructed BL signal or the transmitted coding parameters. · Different filters may be used for the upsampled / filtered BL reconstructed signal in case of BL_reco mode and for the BL residual signal in case of BL_resi mode. The upsampled / filtered BL signal can also be combined with two or more hypotheses of the motion compensated difference signal. This is illustrated in FIG.

[0201] Considering the above, prediction may be performed using a combination of base layer reconstruction and enhancement layer reconstruction (see example C). One major difference from the above description with respect to Figs. 11, 12, 13 is the coding mode for obtaining the intra layer prediction 34, which is performed temporally rather than spatially. That is, instead of the spatial prediction 30, the temporal prediction 32 is used to form the intra layer prediction signal 34. Therefore, some examples described below can be easily transferred to the above embodiments of Figs. 6-10 and 11-13, respectively. The multiplexing method includes the following method: The (upsampled / filtered) reconstructed base layer signal is combined into an inter prediction signal, where the inter prediction is derived by motion compensation prediction using the reconstructed enhancement layer image. The final prediction signal is obtained by weighting the inter prediction signal and the base layer prediction signal in a way that different frequency components use different weightings. For example, this can be realized by either of the following: Filtering the base layer prediction signal with a low-pass filter and the inter prediction signal with a high-pass filter and summing the resulting filtered signals. Transforming the base layer prediction signal and the inter prediction signal to superimpose the resulting transform block, where different weighting coefficients are used for different frequency positions. The resulting transform block is then inverse transformed and used as the enhancement layer prediction signal. Alternatively, the resulting transform coefficients are added to the scaled transmitted transform coefficient levels and inverse transformed to obtain a reconstructed block before deblocking and in-loop processing.

[0202] This mode may also be called the "BL_comb" mode described in FIG.

[0203] In summary, the EL prediction can be expressed as follows: EL prediction=BL_weighting(BL_reco(t0))+EL_weighting(MCP_filter(EL_reco(t1)))

[0204] In a preferred embodiment, the weighting is made dependent on the ratio of EL resolution to BL resolution. For example, when the BL is to be scaled up by a factor in the range [1, 1.25), a given set of weightings for EL and BL reconstruction may be used. When the BL is to be scaled up by a factor in the range [1.25 1.75), a different set of weightings may be used. When the BL is to be scaled up by a factor of 1.75 or more, another different set of weightings may be used, etc.

[0205] Scaling factor separation. Specific weighting depending on the base layer and the enhancement layer is also possible in other embodiments for spatial intra-layer prediction.

[0206] In another preferred embodiment, the weighting is done depending on the EL block size to be predicted. For example, for a 4×4 block in the EL, a weighting matrix can be defined that specifies the weighting for the EL reconstructed transform coefficients. And another weighting matrix can be defined that specifies the weighting for the BL reconstructed transform coefficients. The weighting matrix for the BL reconstructed transform coefficients can be, for example, as follows: 64,63,61,49, 63,62,57,40, 61,56,44,28, 49,46,32,15, And the weighting matrix for the EL reconstruction transform coefficients is, for example, as follows: 0,2,8,24, 3,7,16,32, 9,18,20,26, 22,31,30,23,

[0207] Similarly, separate weighting matrices may be defined for block sizes such as 8x8, 16x16, 32x32, etc.

[0208] The actual transform used for the frequency domain weighting may be the same as or different from the transform used to code the prediction residual: for example, an integer approximation for the DCT may be used for both the frequency domain weighting and to compute the transform coefficients of the prediction residual to be coded in the frequency domain.

[0209] In another preferred embodiment, a maximum transform size is defined for the frequency domain weighting in order to limit the amount of computation. If the EL block size under consideration is larger than the maximum transform size, the EL and BL reconstructions are spatially separated into a series of adjacent sub-blocks, the frequency domain weighting is performed on the sub-blocks, and the final prediction signal is formed by assembling the weighted results.

[0210] Moreover, the weighting can be performed on the luma and chroma components, or on a selected subset of the color components.

[0211] In the following, various possibilities for deriving enhancement layer coding parameters are described. The coding (or prediction) parameters to be used to reconstruct the enhancement layer blocks are derived by a multiplexing method from the collocated coding parameters in the base layer. The base layer and the enhancement layer can have different spatial resolutions or they can have the same spatial resolution.

[0212] In the scalable video extension of H.264 / AVC, inter-layer motion prediction is performed for macroblock types signaled by the syntax element base mode flag. If the base mode flag is equal to 1 and the corresponding reference macroblock in the base layer is inter-coded, the enhancement layer macroblock is also inter-coded, and all motion parameters are inferred from the co-located base layer block. Otherwise (base mode flag is equal to 0), for each motion vector, a syntax element called "motion prediction flag" is sent specifying whether the base layer motion vector is used as the motion vector predictor. If the "motion prediction flag" is equal to 1, the motion vector predictor of the co-located reference block of the base layer is scaled according to the resolution ratio and used as the motion vector predictor. If the "motion prediction flag" is equal to 0, the motion vector predictor is calculated as specified in H.264 / AVC.

[0213] In the following, a method for deriving enhancement layer coding parameters is described. A sample array associated with a base layer image is decomposed into blocks, and each block has associated coding (or prediction) parameters. In other words, every sample position within a particular block has a particular associated coding (or prediction) parameter. The coding parameters may include parameters for motion compensated prediction, including the number of motion hypotheses, reference index list, motion vectors, motion vector prediction identifiers, and merge identifiers. The coding parameters may also include intra prediction parameters, such as intra prediction direction.

[0214] It is signaled in the bitstream that blocks in the enhancement layer are coded using collocated information from the base layer.

[0215] For example, derivation of enhancement layer coding parameters (see example T) may be made as follows: For an N×M block of the enhancement layer that is signaled using collocated base layer information, coding parameters associated with sample positions within the block may be derived based on coding parameters associated with collocated sample positions in the base layer sample array.

[0216] In a particular embodiment, this process involves the following steps: 1. Derive coding parameters for each sample position in an NxM enhancement layer block based on the base layer coding parameters. 2. Derive a partition of the N×M enhancement layer blocks in a sub-block, such that all sample positions within a particular sub-block have the same associated coding parameters.

[0217] Also, the second step can be omitted.

[0218] Step 1 is to select the enhancement layer sample position p el Function f c and the coding parameter c, i.e. c=f c (p el ) is given and executed.

[0219] TIFF0007648677000001.tif84169

[0220] The distance between two horizontally or vertically adjacent base layer sample positions is therefore equal to 1. Both the top-left base layer sample and the top-left enhancement layer sample have position p=(0,0).

[0221] As another example, the function f c (p el ) is the base layer sample position p el The base layer sample position p closest to blWe can then return the coding parameter c associated with Also, the function f c (p el ) can interpolate coding parameters in cases where a particular enhancement layer sample position has a fractional component within units of the distance between the base layer sample positions.

[0222] Before restoring the motion parameters, the function f c rounds the spatially displaced components of the motion parameters to the nearest available value within the enhancement layer sampling grid.

[0223] Since each sample position is associated with a prediction parameter after step 1, each enhancement layer sample can be predicted after step 1. Nevertheless, in step 2, block partitions can be derived to perform prediction operations on larger blocks of samples or to transform code prediction residuals within blocks of the derived partitions.

[0224] Step 2 may be performed by grouping the enhancement layer sample locations into square or rectangular blocks, each of which is decomposed into one of a set of possible decompositions into sub-blocks. The square or rectangular blocks correspond to leaves in a quadtree structure that can exist at different levels as represented in FIG. 34.

[0225] The level and decomposition of each square or rectangular block may be determined by performing the following ordered steps. a) Set the highest level to the level corresponding to a block of size NxM. Set the current level to the lowest level (i.e. the level where a square or rectangular block contains a single block of the smallest block size). Go to step b). b) For each square or rectangular block at the current level, if there exists a possible decomposition of the square or rectangular block, such that all sample positions within each sub-block are associated with the same coding parameters or with a small difference (according to the magnitude of any difference). That decomposition is the candidate decomposition. Among all the candidate decompositions, select the one that decomposes the square or rectangular block into the minimum number of sub-blocks. If the current level is the highest level, go to step c). Otherwise, set the current level to the next higher level and go to step b). c) End

[0226] Function f c may be selected in such a way that at some level in step b), there is always at least one candidate decomposition.

[0227] The grouping of blocks with the same coding parameters is not limited to square blocks, but the blocks can be grouped into rectangular blocks. Furthermore, the grouping is not limited to the quadtree structure. It is also possible to use a structure in which a block is decomposed into two equal-sized rectangular blocks or into two different-sized rectangular blocks. It is also possible to use a decomposition structure that uses a quadtree decomposition up to a certain level and then a decomposition into two rectangular blocks. Any other block decomposition is also possible.

[0228] In contrast to the SVC interlayer motion parameter prediction mode, the described mode is not only supported at the macroblock level (or the most supported block size), but also supported for any block size. That means that the mode can be signaled not only for the most supported block size, but also that the block of the most supported block size (macroblock in MPEG4, H.264, and coding tree block / largest coding unit in HEVC) can be hierarchically sub-divided into smaller blocks / coding units, and the use of the interlayer motion mode can be signaled for any supported block size (for the corresponding block). In a particular embodiment, the mode supports only the selected block size. Then, the syntax element signaling the usage of this mode can be transmitted only for the corresponding block size. Or, the value of the syntax element signaling the usage of this mode (in another coding parameter) can be restricted corresponding to another block size. Also, the difference with the inter-layer motion parameter prediction mode in the SVC extension of H.264 / AVC is that blocks coded in this mode are not fully inter-coded: they may contain intra-coded sub-blocks, depending on the collocated base layer signal.

[0229] One of several methods for reconstructing samples of an M×M enhancement layer block using the coding parameters derived by the above described methods may be signaled in the bitstream. Such methods for predicting an enhancement layer block using the derived coding parameters may include: Deriving a prediction signal for the enhancement layer block using the derived motion parameters and the reconstructed enhancement layer reference image for motion compensation. Combining (a) the (upsampled / filtered) base layer reconstruction for the current picture with (b) the motion compensation signal using the derived motion parameters and the enhancement layer reference picture, which is generated by subtracting the (upsampled / filtered) base layer reconstruction from the reconstructed enhancement layer picture. (a) The (upsampled / filtered) base layer residual (the difference between the reconstructed signal and the prediction, or the inverse transform of the coded transform coefficient values) for the current picture combined with (b) a motion compensation signal using the derived motion parameters and the reconstructed enhancement layer reference picture.

[0230] The process for obtaining a partition into smaller blocks for the current block and deriving coding parameters for the sub-blocks can classify some sub-blocks as intra-coded, while other sub-blocks are classified as inter-coded. For inter-coded sub-blocks, motion parameters can be obtained from the collocated base layer block. However, if the collocated base layer block is intra-coded, the corresponding sub-block in the enhancement layer can be classified as intra-coded. For samples of such intra-coded sub-blocks, the enhancement layer signal can be predicted by using information from the base layer. For example, · The (upsampled / filtered) version of the corresponding base layer reconstruction is used as the intra prediction signal. The derived intra prediction parameters are used for spatial intra prediction within the enhancement layer.

[0231] To predict an enhancement layer block using a weighted combination of prediction signals, the following embodiments include a method for generating a prediction signal for an enhancement layer block by combining (a) an enhancement layer intra prediction signal obtained by spatial or temporal (i.e., motion compensated) prediction using reconstructed enhancement layer samples, and (b) a base layer prediction signal that is an (upsampled / filtered) base layer reconstruction for the current picture. The final prediction signal is obtained by weighting the enhancement layer intra prediction signal and the base layer prediction signal in such a way that for each sample, a weighting according to a weighting function is used.

[0232] For example, the weighting function may be realized by the following method: Compare a low-pass filtered version of the original enhancement layer intra prediction signal v with a low-pass filtered version of the base layer reconstruction u. From the comparison, derive a weighting for each sample position to be used to combine the original inter prediction signal and the (upsampled / filtered) base layer reconstruction. For example, the weighting is derived by mapping the difference u v to a weighting w using a transfer function t. That is, t(u v )=w

[0233] Different weighting functions may be used for different block sizes of the current block to be predicted, and the weighting functions may be modified according to the temporal distance of the reference pictures from which the inter-prediction hypotheses are derived.

[0234] In the case of an enhancement layer intra prediction signal, which is an intra prediction signal, a weighting function may be implemented, for example, using different weightings depending on the position within the current block to be predicted.

[0235] In a preferred embodiment, a method for deriving the enhancement layer coding parameters is used, and step 2 of the method uses a set of possible decompositions of a square block, as illustrated in FIG.

[0236] In a preferred embodiment, the function f c (p el ) is the function f p,m × n (p el )

[0237] In an embodiment, the function f c (p el ) returns the encoding parameter c: First, the base layer sample position is p bl =f p,4 ×4(p el ) is derived as Suppose, p bl If c has associated inter prediction parameters obtained by merging with a previously coded base layer block (or has the same motion parameters), then c is equal to the motion parameters of the enhancement layer block corresponding to the base layer block used for merging within the base layer (i.e., the motion parameters are copied from the corresponding enhancement layer block). Otherwise, c is p bl The encoding parameters associated with

[0238] Also, combinations of the above embodiments are possible.

[0239] In another embodiment, using the collocated base layer information, for the enhancement layer block to be signaled, the intra prediction parameters obtained from the initial set of motion parameters and their associated enhancement layer sample positions so that the block can be merged with the block containing these samples (i.e., a copy of the initial set of motion parameters). The initial set of motion parameters consists of an indicator to use one or two hypotheses, a reference index that refers to the first image in the list of reference images, and a motion vector with a zero spatial displacement.

[0240] In another embodiment, using the collocated base layer information, for an enhancement layer block to be signaled, enhancement layer samples with derived motion parameters are predicted and reconstructed first in a certain ordering. Then, samples with derived intra prediction parameters are predicted in intra reconstruction order. As a result, intra prediction may use already reconstructed sample values ​​from (a) neighboring inter prediction blocks and (b) predecessor neighboring intra prediction blocks in the intra reconstruction order.

[0241] In another embodiment, for an enhancement layer block that is being merged (i.e., takes motion parameters derived from another inter-predicted block), the list of merging candidates additionally includes candidates from the corresponding base layer block, and, if the enhancement layer has a higher spatial sampling rate than the base layer, additionally includes up to four candidates derived from the base layer candidates by refining the spatial permutation components only from the available neighboring values ​​in the enhancement layer.

[0242] In another embodiment, the difference measure used in step 2b) asserts that there are small differences within a sub-block only if the differences vanish altogether, i.e. a sub-block can only be formed if all involved sample positions have the same derived coding parameters.

[0243] In another embodiment, the difference measure used in step 2b) is declared to be small within a sub-block if either (a) all included sample positions have derived motion parameters and no pair of sample positions within the block have derived motion parameters that differ by more than a certain value according to a vector norm applied to the corresponding motion vectors, or (b) all included sample positions have derived intra-prediction parameters and no pair of sample positions within the block have directional intra-prediction parameters that differ by more than a certain angle. The derived parameters for the sub-blocks are calculated by an average or median operation. In another embodiment, the partitions obtained by inferring the coding parameters from the base layer can be further refined based on side information signaled in the bitstream. In another embodiment, the residual coding for a block for which coding parameters are inferred from the base layer is independent of the partitions within the block inferred from the base layer. For example, that means that a single transform can be applied to the block, although the inference of coding parameters from the base layer partitions the block into several sub-blocks, each with a separate set of coding parameters. Or, a block for which partitions and coding parameters for the sub-blocks are inferred from the base layer can be divided into smaller blocks for the purpose of transform coding the residual, where the division into transform blocks is independent of the inferred partitions within the block with different coding parameters.

[0244] In another embodiment, the residual coding for blocks whose coding parameters are inferred from the base layer depends on the partitions within the block inferred from the base layer. For example, that means that for transform coding, the division of blocks within the transform block depends on the partitions inferred from the base layer. In one version, a single transform may be applied to each sub-block with different coding parameters. In another version, the partitions may be refined based on side information included in the bitstream. In another version, several sub-blocks may be combined into larger blocks such that the residual signal is signaled in the bitstream for the purpose of transform coding.

[0245] Also embodiments obtained by combining the above-mentioned embodiments are possible.

[0246] In relation to enhancement layer motion vector coding, this next section describes a method for reducing motion information in scalable video coding applications by providing multiple enhancement layer predictors and using motion information coded in the base layer to efficiently code the motion information of the enhancement layer. This idea is appropriate for scalable video coding, including spatial, temporal and quality scalability.

[0247] In the H.264 / AVC inter-layer scalable video extension, motion estimation is performed for macroblock types that are signaled with the syntax element "base mode flag". If the "base mode flag" is equal to 1 and the corresponding reference macroblock in the base layer is inter-coded, the enhancement layer macroblock is also inter-coded. Then, all motion parameters are inferred from the collocated base layer block. Otherwise (if the "base mode flag" is equal to 0), each motion vector (the so-called "motion prediction flag" syntax element) is sent and specified, regardless of whether the base layer motion vector is used as the motion vector predictor. If the "motion prediction flag" is equal to 1, the motion vector predictor of the collocated reference block of the base layer is scaled according to the resolution ratio and used as the motion vector predictor. If the "motion prediction flag" is equal to 0, the motion vector predictor is calculated as specified in H.264 / AVC. In HEVC, motion parameters are predicted by applying Adaptive Motion Vector Prediction (AMVP). AMVP features two spatial motion vector predictors and one temporal motion vector predictor competing with each other. Spatial candidates are selected from the positions of neighboring prediction blocks located to the left or above the current prediction block. Temporal candidates are selected among the collocated positions of the previously coded image. The positions of all spatial and temporal candidates are displayed in Figure 36.

[0248] After the spatial and temporal candidates are inferred, a redundancy check is performed that may introduce the zero motion vector as a candidate into the list. The candidate list describing the index is transmitted to identify the motion vector predictor to be used with the motion vector difference for motion compensated prediction. HEVC further uses a block merging algorithm that aims to reduce the coding redundant motion parameters resulting from the quadtree based coding structure. This is achieved by creating regions consisting of multiple predicted blocks sharing a certain motion parameter. These motion parameters only need to be coded once for the first predicted block of each region seeding new motion information. Similar to AMVP, the block merging algorithm constructs a list containing possible merge candidates for each predicted block. The number of candidates is defined by "NumMergeCands", which is signaled in the slice header and ranges from 1 to 5. Candidates are inferred from spatially neighboring predicted blocks and predicted blocks in the co-located temporal image. The possible sample positions for a predicted block considered to be a candidate are equal to the positions shown in Figure 36. An example of the block merging algorithm with partitions of possible predicted blocks in HEVC is illustrated in Figure 37. The bold lines in Figure 37(a) define all the predicted blocks that are merged into one region and hold a certain motion data. This motion data is sent only to block S. The current predicted block to be coded is indicated by an "X". The prediction blocks in the striped regions are successors of prediction block X in the block scanning order and therefore do not yet have associated prediction data. The dots indicate sample positions of adjacent blocks that are possible spatial merge candidates. Before the possible candidates are inserted into the predictor list, a redundancy check for the spatial candidates is performed as shown in Figure 37(b).

[0249] If the number of spatial and temporal candidates is less than "NumMergeCands", additional candidates are provided by combining with existing candidates or by inserting zero motion vector candidates. If a candidate is added to the list, it has an index that is used to identify the candidate. As new candidates are added to the list, the merge index increases (starting from 0) until the list is completed with the last candidate identified by index "NumMergeCands"-1. A fixed length codeword is used to encode the merge candidate index to ensure independent operations of candidate list derivation and bitstream parsing.

[0250] The following section describes a method for using multiple enhancement layer predictors, including a predictor derived from the base layer, to code the motion parameters of the enhancement layer. The motion information already coded for the base layer can be used to significantly reduce the motion data rate while coding the enhancement layer. The method includes the possibility of deriving all motion data of a prediction block directly from the base layer. In this case, no additional motion data needs to be coded. In the following description, the term prediction block refers to a prediction unit in HEVC (M×N block in H.264 / AVC) and can be understood as a general set of samples in an image.

[0251] The first part of the current section is related to extending the list of motion vector prediction candidates by base layer motion vector predictors (see aspect K). Base layer motion vectors are added to the motion vector predictor list during enhancement layer coding. This is achieved by inferring the motion vector predictor of one or multiple co-located prediction blocks from the base layer and using them as candidates in the list of predictors for motion compensated prediction. The co-located prediction blocks of the base layer are located at the center, left, top, right or bottom of the current block. If the base layer prediction block at the selected location does not contain motion related data or is outside the current range and therefore is not currently accessible, alternative locations can be used to infer the motion vector predictor. These alternative locations are depicted in FIG. 38.

[0252] The inferred motion vectors of the base layer may be scaled according to the resolution ratio before they can be used as predictor candidates. Similar to the motion vector difference, an index describing the candidate list of motion vector predictors is transmitted to the prediction block specifying the final motion vector to be used for motion compensated prediction. In contrast to scalable extensions of the H.264 / AVC standard, the embodiments presented here do not constitute the usage of a motion vector predictor of a collocated block in a reference image - rather it may be available in a list in another predictor and described by a transmitted index. In an embodiment, a motion vector is obtained from the base layer co-located prediction block center position C1 and is added to the head of the candidate list as the first entry. The candidate list of motion vector predictors is extended by one item. If there is no motion data in the base layer available for sample position C1, the list structure is untouched. In another embodiment, any sequence of sample positions in the base layer can be checked for motion data. In case motion data is found, the motion vector predictor of the corresponding position is inserted into the candidate list and is available for enhancement layer motion compensated prediction. Additionally, a motion vector predictor derived from the base layer is inserted into the candidate list at any other position in the list. In another embodiment, a base layer motion predictor can only be inserted into the candidate list if certain constraints are met. These constraints include the value of the merge flag of the co-located reference block, which must be equal to zero. Another constraint can be the width of the prediction block in the enhancement layer being equal to the width of the base co-located prediction block for the resolution ratio. For example, in the application of K× spatial scalability, suppose the width of the collocated blocks in the base layer is equal to N, and the width of the predicted block to be coded in the enhancement layer is K× * Only if it is equal to N can the motion vector predictor be inferred. In another embodiment, one or more motion vector predictors from several sample positions of the base layer may be added to the enhancement layer candidate list. In another embodiment, a candidate with a motion vector predictor inferred from a collocated block may replace a spatial or temporal candidate in the list rather than extending the list. It is also possible to include multiple motion vector predictors derived from base layer data in the motion vector predictor candidate list.

[0253] The second part concerns extending the list of merging candidates by base layer candidates (see aspect K). The motion data of one or more co-located blocks of the base layer are added to the merging candidate list. This method allows the possibility of creating merging regions that share certain motion parameters across the base layer and the enhancement layer. As in the previous section, the base layer block covering the co-located samples at the center position is not limited to this center position, but can be obtained from any position in the immediate vicinity, as depicted in Figure 38. If no motion data is available or accessible for a given position, alternative positions can be selected to infer possible merging candidates. Before the derived motion data is inserted into the merging candidate list, it can be scaled according to the resolution ratio. An index describing the merging candidate list is transmitted to define the motion vector, which is used for motion compensation prediction. However, the method can also suppress possible motion predictor candidates that depend on the motion data of the prediction block in the base layer. In an embodiment, the motion vector predictor of the collocated block in the base layer covering the sample position C1 of FIG. 38 is considered as a possible merging candidate for encoding the current predicted block in the enhancement layer. However, if the "merge_flag" of the reference block is equal to 1 or the collocated reference block does not contain any motion data, the motion vector predictor is not inserted into the list. In any other case, the obtained motion vector predictor is added to the merge candidate list as the second entry. Note that in this embodiment, the length of the merge candidate list is retained and is not extended. In another embodiment, as depicted in FIG. 38, one or more motion vector predictors are derived from the predictor blocks covering any of the sample positions to be added to merge candidate list. In another embodiment, one or several motion vector predictors of the base layer can be added to the merge candidate list at any position. In another embodiment, one or more motion vector predictors are only added to the merge candidate list if certain constraints are met. Such constraints include the size of the enhancement layer prediction block matching the width of the base layer collocated block (with respect to the resolution ratio described in the previous embodiment section for motion vector prediction). Another constraint in another embodiment is the value of "merge_flag" equal to 1. In another embodiment, the length of the merge candidate list can be extended by the number of motion vector predictors inferred from the base layer collocated reference blocks.

[0254] The third part of this specification describes the process of reordering the motion parameter (or merge) candidate list using base layer data (see Example L), which reorders the merge candidate list according to already coded information in the base layer. If the collocated base layer block covering the current block's samples is a motion compensated prediction with a candidate derived from a particular original, the corresponding enhancement layer candidate from the equivalent original (if there is one) is put at the top of the merge candidate list as the first entry. This step is equivalent to writing this candidate with the lowest index. The lowest index assigns the simplest codeword to this candidate. In an embodiment, the collocated base layer block is motion compensated predicted with a candidate originating from a predictive block covering sample position A1, as depicted in FIG. 38. If the merge candidate list of a predictive block in the enhancement layer includes a candidate whose motion vector predictor originates from the corresponding sample position A1 in the enhancement layer, this candidate is placed in the list as the first entry. As a result, this candidate is indexed by index 0 and is therefore assigned the shortest fixed-length codeword. In this embodiment, this step is performed after the derivation of the motion vector predictor of the collocated base layer block for the merge candidate list in the enhancement layer. Thus, the reordering process assigns the lowest index to the candidate originating from the corresponding block as the motion vector predictor of the collocated base layer block. The second lowest index is assigned to the candidate derived from the collocated block in the base layer, as described in the second part of this section. Moreover, the reordering process is performed only if the "merge_flag" of the co-located block in the base layer is equal to 1. In another embodiment, the reordering process can be performed regardless of the value of the "merge_flag" of the co-located prediction block in the base layer. In another embodiment, a candidate with a corresponding original motion vector predictor can be placed in any position in the merge candidate list. In another embodiment, the reordering process can remove all other candidates in the merge candidate list. Here, only candidates whose motion vector predictor has the same original as the motion vector predictor used for motion compensated prediction of the co-located block in the base layer remain in the list. In this case, a single candidate is used and no index is transmitted.

[0255] The fourth part of this specification is directed to reordering the motion vector predictor candidate list using base layer data (see aspect L), and is embodied in a process of reordering the motion vector prediction candidate list using the motion parameters of the base layer block. If the collocated base layer block covering the sample of the current prediction block uses a motion vector from a particular original, then the motion vector predictor from the corresponding original in the enhancement layer is used as the first entry in the motion vector predictor list of the current prediction block. This results in the cheapest codeword being assigned to this candidate. In an embodiment, the co-located base layer block is motion compensated predicted with a candidate originating from the predictor block covering sample position A1, as shown in FIG. 38. If the motion vector predictor candidate list of a block in the enhancement layer includes a candidate whose motion vector predictor originates from the corresponding sample position A1 in the enhancement layer, this candidate is placed in the list as the first entry. As a result, this candidate is indexed by index 0 and is therefore assigned the shortest fixed-length codeword. In this embodiment, this step is performed after the derivation of the motion vector predictor of the co-located base layer block for the motion vector predictor list in the enhancement layer. Thus, the reordering process assigns the lowest index to the candidate originating from the corresponding block as the motion vector predictor of the co-located base layer block. The second lowest index is assigned to the candidate derived from the co-located block in the base layer, as described in the first part of this section. Moreover, the reordering process is performed only if the "merge_flag" of the co-located block in the base layer is equal to 0. In another embodiment, the reordering process can be performed regardless of the value of "merge_flag" of the co-located prediction block in the base layer. In another embodiment, a candidate with a corresponding original motion vector predictor can be placed in any position in the motion vector predictor candidate list.

[0256] The following concerns the enhancement layer coding of transform coefficients.

[0257] In state-of-the-art video and image coding, the residual of the prediction signal is pre-transformed and the resulting quantized transform coefficients are signaled in the bitstream. This coefficient coding follows a fixed scheme.

[0258] Depending on the transform size (for luma residual: 4x4, 8x8, 16x16, 32x32), different scan directions are defined. Given the first and last positions in the scan order, these scans uniquely determine which coefficient positions can be significant and therefore need to be coded. In every scan, the first coefficient is set to be the DC coefficient at position (0,0), which is done by coding the (horizontal) x and (vertical) y positions in the transform block, although the last position must be signaled in the bitstream. Starting from the last position, the signaling of significant coefficients is done in the reverse scan order until the DC position is reached.

[0259] For transform sizes 16x16 and 32x32, only one scan is defined, namely the "diagonal scan". However, transform blocks of sizes 2x2, 4x4 and 8x8 can additionally utilize "vertical" and "horizontal" scans. However, the use of vertical and horizontal scans is restricted to the residuals of intra-predicted coding units. And the scan actually used is derived from the instruction mode of the intra-prediction. Instruction modes with indices in the range 6 and 14 result in vertical scans. Instruction modes with indices in the range 22 and 30 result in horizontal scans. All residual instruction modes result in diagonal scans.

[0260] Figure 39 shows the diagonal, vertical and horizontal scans defined for a 4x4 transform block. The coefficients of the larger transform are divided into subgroups of 16 coefficients. These subgroups allow for hierarchical coding of significant coefficient positions. Subgroups signaled as insignificant do not contain significant coefficients. The transforms for 8x8 and 16x16 are shown with the scans and their associated subgroup divisions in Figures 40 and 41, respectively. The large arrows represent the scan order of the coefficient subgroups.

[0261] In zigzag scanning, for blocks of size greater than 4x4, the subgroups consist of blocks of 4x4 pixels scanned in zigzag scanning. The subgroups are scanned in a zigzag manner. Figure 42 shows the vertical scan for a 16x16 transform as proposed in JCTVC-G703.

[0262] The following paragraphs describe enhancements to the transform coefficient coding. These include the introduction of new scanning modes (methods of assigning scans to transform blocks and modified coding of significant coefficient positions). These enhancements allow a better adaptation of different coefficient distributions within a transform block, and thus achieve coding gains in the rate-distortion function.

[0263] New realizations for vertical-horizontal scan patterns are introduced for 16x16 and 32x32 transform blocks. In contrast to previously proposed scan patterns, the size of the scan subgroups is 16x1 for horizontal scan and 1x16 for vertical scan, respectively. Also, subgroups with sizes of 8x2 and 2x8 are selected, respectively. The subgroups themselves are scanned in the same way.

[0264] Vertical scanning is efficient for transform coefficients that lie within the column-wise extent, which may be found in images that contain horizontal edges.

[0265] Horizontal scanning is efficient due to the transform coefficients being found within the row-wise extent, which is observed in images that contain vertical edges.

[0266] Figure 43 shows the implementation of vertical-horizontal scanning for a 16x16 transform block. The coefficient subgroups are defined as one row or one column, respectively. Vertical-horizontal scanning is the scanning pattern introduced, which allows the coding of coefficients in columns by row-wise scanning. For a 4x4 block, the first column is scanned followed by the remainder of the first row, then the remainder of the second column, then the remainder of the coefficients of the second row, then the remainder of the third row, and finally the remainder of the fourth row and column.

[0267] For larger blocks, the block is partitioned into subgroups of 4 x 4. These 4 x 4 blocks are scanned with a vertical-horizontal scan, and the subgroups are scanned with a vertical-horizontal scan themselves.

[0268] A vertical-horizontal scan can be used when the coefficients are located in the first row and column in a block. In this way, the coefficients are scanned faster than if another scan, such as a diagonal scan, were used. This is found for images that contain both horizontal and vertical edges.

[0269] FIG. 44 shows the vertical and horizontal scans for a 16×16 transform block.

[0270] Other scans are possible as well. For example, all combinations between scans and subgroups can be used. For example, using a horizontal scan for the 4x4 block and a diagonal scan for the subgroups. An appropriate choice of scans can be applied by selecting a different scan for each subgroup.

[0271] It should be mentioned that the different scanning can be achieved in a way that the transform coefficients are reordered after quantization at the encoder side and conventional coding is used. At the decoder side, the transform coefficients are conventionally decoded and reordered before scaling and inverse transform (or after scaling and before inverse transform).

[0272] Different parts of the base layer signal are utilized to derive the coding parameters from the base layer signal. Within the base layer signal there are: · Collocated reconstructed base layer signal Collocated residual base layer signal An estimated enhancement layer residual signal obtained by subtracting the enhancement layer prediction signal from the reconstructed base layer signal. Base layer frame image partition

[0273] [Slope Parameter] The gradient parameters are derived as follows: For each pixel of the examined block, a gradient is calculated. From these gradients, a magnitude and an angle are calculated. The angle that occurs most in the block is related to the block (block angle). The angles are rounded to use only three directions: horizontal (0°), vertical (90°) and diagonal (45°).

[0274] [Edge detection] An edge detector can be applied to the examined blocks as follows: First, the block is smoothed by an n×n smoothing filter (eg, Gaussian). A gradient matrix of size mxm is used to calculate the gradient for each pixel. The magnitude and angle of every pixel are calculated. The angles are rounded to use only three directions: horizontal (0°), vertical (90°) and diagonal (45°). For every pixel with a magnitude greater than a given threshold 1, the neighboring pixels are checked. If a neighboring pixel has a magnitude greater than threshold 2 and has the same angle as the current pixel, the counter for this angle is incremented. For the whole block, the counter with the highest value is selected as the angle of the block.

[0275] [Obtaining the base layer coefficients by previous transformation] To derive coding parameters from the frequency domain of the base layer signal for a particular TU, the examined and collocated signals (reconstructed base layer signal / residual base layer signal / estimated enhancement layer signal) are transformed into the frequency domain. Preferably, this is performed using the same transform used by that particular enhancement layer TU. The resulting base layer transform coefficients may or may not be quantized. To obtain a coefficient distribution comparable to the enhancement layer blocks, rate-distortion quantization with a modified lambda is used.

[0276] [Scan validity score for specific distribution and scan] The scan validity score for a particular important coefficient distribution is defined as follows: Let each location of the examined block be represented by an index in the order of the examined scan. The sum of the index values ​​of the significant coefficient locations is then defined as the effectiveness score of this scan. As a result, the smaller the score a particular distribution has, the better its efficiency.

[0277] [Selecting the appropriate scan pattern for transform coefficient coding] If several scans are available for a particular TU, a rule needs to be defined that uniquely selects one of the scans.

[0278] [Method for selecting scanning pattern] The selected scan can be derived directly from the already decoded signal (without transmitting any additional data), either based on the properties of the collocated base layer signal, or by utilizing only the enhancement layer signal. The scanning pattern can be derived from the EL signal by: · The state-of-the-art derivation rules mentioned above. Using a scanning pattern for the chrominance residual selected for the collocated luminance residual. · Defining a fixed mapping between the coding mode and the scanning pattern used. Obtain the scan pattern from the last significant coefficient position (proportional to the estimated fixed scan pattern). In a preferred embodiment, the scan pattern is selected depending on the last position already decoded as follows:

[0279] The last position is represented as x and y coordinates within the transform block and is already decoded (for the last dependent scan to be coded, a fixed scan pattern is estimated for the last position decoding process, which can be the leading edge scan pattern of that TU). Let T be a defined threshold that depends on the particular transform size. If neither the x nor the y coordinate of the last significant position exceeds T, then a diagonal scan is selected.

[0280] Otherwise, x is compared with y. If x exceeds y, horizontal scanning is selected and vertical scanning is not selected. The preferred value of T for a 4x4 TU is 1. The preferred value of T for a TU larger than 4x4 is 4.

[0281] In another preferred embodiment, the derivation of the scan pattern as described in the previous embodiment is restricted to be done only for TUs of size 16 x 16 and 32 x 32. It can be further restricted to only the luminance signal.

[0282] Also, the scan pattern is derived from the BL signal. Any of the coding parameters mentioned above can be used to derive the selected scan pattern from the base layer signal. In particular, the gradient of the collocated base layer signal can be calculated and compared to a predefined threshold and / or potentially discovered edges can be utilized.

[0283] In a preferred embodiment, the scan direction is derived depending on the block gradient angle as follows: for gradients quantized in the horizontal direction, a vertical scan is used; for gradients quantized in the vertical direction, a horizontal scan is used; otherwise, a diagonal scan is selected.

[0284] In another preferred embodiment, scan patterns are derived as described in the previous embodiment, but only for those transform blocks for which the number of occurrences of the block angles exceeds a threshold. The remaining transform units are decoded using the state-of-the-art scan patterns of the TUs.

[0285] If the base layer coefficients of the collocated blocks are available and are either explicitly signaled in the base layer data stream or calculated by a previous transform, the base layer coefficients may be utilized in the following manner. For each available scan, the cost for encoding the base layer coefficients may be evaluated. The scan with the lowest cost is used to decode the enhancement layer coefficients. An effective score for each available traversal is calculated for the base layer coefficient distribution. The traversal with the smallest score is used to decode the enhancement layer coefficients. The distribution of the base layer coefficients within a transform block is classified into one of a predefined set of distributions that are associated with a particular scanning pattern. The scan pattern is selected depending on the last significant base layer coefficient.

[0286] If the collocated base layer block is predicted using intra prediction, then the intra direction of the prediction may be used to derive the enhancement layer scanning pattern.

[0287] Furthermore, the transform sizes of the collocated base layer blocks are utilized to derive the scanning pattern.

[0288] In a preferred embodiment, the scan pattern is derived from the BL signal only for TUs that represent the residuals of INTRA_COPY mode predicted blocks and whose collocated base layer blocks are intra predicted. For those blocks, a modified state-of-the-art scan selection is used. In contrast to the state-of-the-art scan selection, the intra prediction direction of the collocated base layer blocks is used to select the scan pattern.

[0289] Signaling the scan pattern index in the bitstream (see aspect R). The scanning pattern of the transform blocks can be selected in a rate-distortion sense by the encoder and then signaled in the bitstream.

[0290] A particular scan pattern can be coded by signaling an index into a list of available scan pattern candidates. This list can be a fixed list of scan patterns defined for a particular transform size, or it can be filled dynamically during the decoding process. Filling the list dynamically allows for the selection of the scan pattern that most efficiently codes a particular coefficient distribution. By doing so, the number of available scan patterns for a particular TU can be reduced, and as a result, signaling an index into that list is less costly. If the number of scan patterns in a particular list is reduced to one, no signaling is necessary. For a particular TU, the process of selecting scan pattern candidates may utilize any of the coding parameters mentioned above and / or follows certain rules that utilize certain characteristics of that particular TU. Among them are the following: ·TU represents the residual of the luminance / chrominance signals. · TU has a specific size. · TU represents the residual for a particular prediction mode. The last significant position within a TU is known by the decoder and belongs to a particular subdivision of the TU. · A TU is a part of one I / B / P-slice. The coefficients of a TU are quantized using a specific quantization parameter.

[0291] In a preferred embodiment, the list of scan pattern candidates includes three scans for every TU: "diagonal scan", "vertical scan" and "horizontal scan".

[0292] Alternative embodiments may be obtained by having the candidate list include any combination of scan patterns.

[0293] In a particular preferred embodiment, the list of candidate scan patterns may include any of "diagonal scan", "vertical scan" and "horizontal scan".

[0294] However, the scan pattern selected by the state-of-the-art scan derivation (described above) is set to be in the list first. Another candidate is added to the list only if the particular TU has a size of 16x16 or 32x32. The order of the remaining scan patterns depends on the last significant coefficient position.

[0295] (Note: the diagonal scan is always the first pattern in the list to estimate 16x16 and 32x32 transformations.)

[0296] If the magnitude of the x coordinate exceeds the magnitude of the y coordinate, then the horizontal scan is selected next and the vertical scan is placed in the last position, otherwise the vertical scan is placed in the second position followed by the horizontal scan.

[0297] Another preferred embodiment is obtained by further restricting the conditions to have more than one candidate in the list.

[0298] In another embodiment, the vertical and horizontal scans are only added to the candidate list for 16x16 and 32x32 transform blocks if the coefficients of the transform block represent the residual of the luminance signal.

[0299] In another embodiment, if both the x and y coordinates of the last significant position are greater than a certain threshold, the vertical and horizontal scans are added to the candidate list of transformation blocks. This threshold can be mode and / or TU depending on the size. The preferred threshold is 3 for all sizes larger than 4x4 and 1 for 4x4 TU.

[0300] In another embodiment, a vertical or horizontal scan is only added to the candidate list of transform blocks if either the x and y coordinates of the last significant position are greater than a certain threshold. This threshold can be mode and / or TU depending on the size. A preferred threshold is 3 for all sizes larger than 4x4 and 1 for 4x4 TUs.

[0301] In another embodiment, vertical and horizontal scans are only added to the candidate list for 16x16 and 32x32 transform blocks if both the x and y coordinates of the last significant position are greater than a certain threshold. This threshold can be mode and / or TU depending on the size. The preferred threshold is 3 for all sizes larger than 4x4 and 1 for 4x4 TUs.

[0302] In another embodiment, vertical and horizontal scans are only added to the candidate list for 16x16 and 32x32 transform blocks if either the x and y coordinates of the last significant position are greater than a certain threshold. This threshold may depend on the mode and / or TU size. A preferred threshold is 3 for all sizes greater than 4x4 and 1 for 4x4 TUs.

[0303] For any of the described embodiments, a particular scan pattern is signaled in the bitstream. The signaling itself can be done at different signaling levels. In particular, the signaling can be done at any node of the residual quadtree (all sub-TUs of that node that use the signaled scan and use the same candidate list index), at the CU / LCU level, or at the slice level, for each TU falling within a subgroup of TUs that have the signaled scan pattern.

[0304] An index into the candidate list may be transmitted using fixed-length coding, variable-length coding, arithmetic coding (including context-adaptive binary arithmetic coding), or PIPE coding. If context-adaptive coding is used, the context may be derived based on neighboring blocks, the coding mode described above, and / or characteristics of the particular TU itself.

[0305] In a preferred embodiment, context-adaptive coding is used to signal an index into the scan pattern candidate list of a TU, but a context model is derived based on the transform size and / or position of the last significant position within the TU. Any of the methods described above for deriving a scan pattern may be used to derive a context model for signaling an explicit scan pattern for a particular TU.

[0306] To code the last significant scan position, the following modifications can be used in the enhancement layer: Separate context models are used for all or a subset of coding modes using base layer information. It is also possible to use different context models for different modes with base layer information. The context model can depend on data in a collocated base layer block (e.g., the transform coefficient distribution in the base layer, the gradient information of the base layer, the last scan position in the collocated base layer block). · The last scan position can be encoded as the difference from the last base layer scan position. · If the last scanned position is encoded within a TU by signalling the x and y position, then the context model of the second signalled coordinate can depend on the value of the first signalled. To derive a scanning pattern that is independent of the last significant position, any of the methods described above are used to derive a context model for signaling the last significant position.

[0307] In a particular version, the scan pattern derivation depends on the last significant position: · If the last scan position is encoded within the TU by signalling its x and y position, then the context model for the second coordinate can rely on those scan patterns that are still possible candidates when already knowing the first coordinate. If the last scan position is encoded in the TU by signalling its x and y position, then the context model for the second coordinate may depend on whether the scan pattern is already uniquely selected when already knowing the first coordinate.

[0308] In another version, the scan pattern derivation is independent of the last significant position. The context model can depend on the scanning pattern used within a particular TU. Any of the methods previously described for deriving a scanning pattern may be used to derive a context model for signaling the last significant position.

[0309] To code the significance positions within a TU and the significance flags (subgroup flags and / or significance flags for one transform coefficient), respectively, the following modifications are used in the enhancement layer: Separate context models are used for all or a subset of coding modes that use base layer information. It is also possible to use different context models for different modes that have base layer information. · The context model can depend on data in the collocated base layer block (eg, the number of significant transform coefficients for a particular frequency location). Any of the methods described above for deriving scanning patterns may be used to derive a context model for signaling critical locations and / or their levels. A generalized template is used which estimates both the significant number of already coded transform coefficient levels in the spatial neighborhood of the coefficient to be coded and the significant number of transform coefficients in the collocated base layer signal at similar frequency positions. A generalized template may be used that estimates both the significant number of already coded transform coefficient levels in the spatial neighborhood of the coefficient to be coded and the levels of significant transform coefficients in the collocated base layer signal at similar frequency positions. The context modeled for the subgroup flag may depend on the scan pattern and / or the particular transform size used.

[0310] Different context initialization table usage for the base layer and the enhancement layer may be used. The context model initialization for the enhancement layer may be modified in the following ways: Enhancement layers use a separate set of initialization values. Enhancement layers may use separate sets of initialization values ​​for different operation modes (spatial / temporal or quality scalability). Enhancement layer context models that have counting parts in the base layer may use the state of those counting parts as their initialization state. The algorithm for deriving the initial state of the context model may depend on the base layer QP and / or the delta QP.

[0311] Next, the possibility of backward adaptive enhancement layer coding using base layer data is described. The following section describes a method for generating an enhancement layer prediction signal in a scalable video coding system. The method uses base layer decoded image sample information to infer values ​​of prediction parameters, which are not transmitted in the coded video bitstream but are used to form a prediction signal for the enhancement layer. Thus, the overall bit rate required to code the enhancement layer signal is reduced.

[0312] State-of-the-art hybrid video encoders decompose an original (source) image into blocks of different sizes, usually following a hierarchical structure. For each block, the video signal is predicted from spatially neighboring blocks (intra prediction) or from previously temporally coded pictures (inter prediction). The difference between the prediction and the actual image is the transformation and quantization. The resulting prediction parameters and transform coefficients are entropy coded to form the coded video bitstream. A matching decoder follows the steps in the reverse order... A scalable video coding bitstream consists of different layers: a base layer that provides the complete decodable video, and an enhancement layer that can be used additionally for decoding. The enhancement layer can provide higher spatial resolution (spatial scalability), temporal resolution (temporal scalability) or quality (SNR scalability). In previous standards such as H.264 / AVC SVC, syntax elements such as motion vectors, reference picture indices or intra prediction modes are predicted directly from corresponding syntax elements in the coded base layer. Within the enhancement layer, mechanisms exist to switch between, at the block level, using prediction signals derived from base layer syntax elements or predicted from other enhancement layer syntax elements or decoded enhancement layer samples.

[0313] In the following, the base layer data is used to derive the enhance layer parameters at the decoder side.

[0314] [Method 1: Deriving candidate motion parameters] For a block (a) of the spatial or qualitative enhancement layer image, a corresponding block (b) of the base layer image is determined, which covers the same image region. The inter prediction signal for block (a) of the enhancement layer is formed using the following method: 1. Candidate motion compensation parameter sets are determined, for example, from temporally or spatially neighboring enhancement layer blocks or derivatives thereof. 2. Motion compensation is performed for each candidate motion compensation parameter set to form an inter prediction signal within the enhancement layer. 3. The best motion compensation parameter set is selected by minimizing the magnitude of the error between the predicted signal for the enhancement layer block (a) and the reconstructed signal of the base layer block (b). For spatial scalability, the base layer block (b) is spatially upsampled using an interpolation filter.

[0315] A motion compensation parameter set includes a particular combination of motion compensation parameters.

[0316] The motion compensation parameters may be a motion vector, a reference image index, a choice between one and two predictions and another parameter.

[0317] In an alternative embodiment, a candidate motion compensation parameter set from the base layer block is used. Also, inter prediction is performed in the base layer (using the base layer reference image). To apply the error magnitude, the base layer block (b) reconstructed signal can be used directly without upsampling. The selected optimal motion compensation parameter set is applied to the enhancement layer reference image to form the prediction signal of block (a). When the motion vector is applied in the spatial enhancement layer, the motion vector is scaled according to the resolution change. Both the encoder and the decoder can perform the same prediction step to select the optimal motion compensation parameter set among the available candidates and create the same prediction signal, these parameters are not signaled in the coded video bitstream.

[0318] The choice of prediction method may be signaled in the bitstream and coded using entropy coding. Within a hierarchical block subdivision structure, the coding method may be selected at every sublevel, or alternatively only at a subset of the coding hierarchy. In an alternative embodiment, the encoder may send an improved motion parameter set prediction signal to the decoder. The refinement signal differentially contains coded values ​​of the motion parameters. The refinement signal is entropy coded.

[0319] In an alternative embodiment, the decoder generates a list of the best candidates. The index of the motion parameter set used is signaled in the coded video bitstream. The index is entropy coded. In an embodiment, the list may be ordered by increasing error magnitude.

[0320] The embodiment uses an adaptive motion vector prediction (AMVP) candidate list in HEVC to generate motion compensation parameter set candidates. Another embodiment uses the merge mode candidate list of HEVC to generate motion compensation parameter set candidates.

[0321] [Method 2: Motion vector derivation] For a block (a) of the spatial or quality enhancement layer image, a corresponding block (b) of the base layer image covering the same image region is determined.

[0322] The inter prediction signal for block (a) of the enhancement layer is formed using the following method: 1. Motion vector prediction is selected. 2. Motion estimation for a defined set of search positions is performed on the enhancement layer reference images. 3. For each search position, the magnitude of the error is determined and the motion vector with the smallest error is selected. 4. A prediction for block (a) is formed using the selected motion vector.

[0323] In an alternative embodiment, the search is performed on the reconstructed base layer signal. For spatial scalability, the selected motion vector is scaled according to the spatial resolution change before generating the prediction signal in step 4.

[0324] The search locations may be full resolution or sub-pel resolution, and the search can be performed in multiple steps, e.g., first determining the best full-pel location, followed by another set of candidates based on the selected full-pel location, and the search can be terminated early, e.g., when the error magnitude is below a defined threshold.

[0325] Both the encoder and the decoder can perform the same prediction steps to select the best motion vector among the candidates and generate the same prediction signal, which are not signaled in the coded video bitstream.

[0326] The choice of prediction method may be signaled in the bitstream and coded using entropy coding. Within a hierarchical block subdivision structure, the coding method may be selected within every sublevel, or only a subset of the alternative coding hierarchy. In an alternative embodiment, the encoder may send a refinement motion vector prediction signal to the decoder. The refinement signal is entropy coded.

[0327] The embodiment uses the algorithm described in Method 1 to select the motion vector predictor.

[0328] Another embodiment uses the Adaptive Motion Vector Prediction (AMVP) method of HEVC to select a motion vector predictor from temporally or spatially neighboring blocks of the enhancement layer.

[0329] [Method 3: Intra prediction mode derivation] For each block (a) in the enhancement layer (n) image, a corresponding block (b) covering the same region in the reconstructed base layer (n-1) image is determined.

[0330] In a scalable video decoder, for each base layer block (b), an intra prediction signal is formed using an intra prediction mode (p) that is inferred by the following algorithm: 1) An intra prediction signal is generated for each available intra prediction mode following the rules for intra prediction of the enhancement layer, but using sample values ​​from the base layer. 2) Best prediction mode (p best ) is determined by minimizing the magnitude of the error (eg, the sum of absolute differences) between the intra prediction signal and the decoded base layer block (b). 3) The predictions selected in step 2) (p best ) mode is used to generate a prediction signal for enhancement layer block (a) according to intra prediction rules for the enhancement layer.

[0331] Both the encoder and the decoder use the best prediction mode (p best The same steps can be performed to select the actual intra prediction mode (p best ) is not signaled in the coded video bitstream.

[0332] The choice of prediction method may be signaled in the bitstream and coded using entropy coding. In a hierarchical block subdivision structure, the coding mode may be selected in every sublevel, or alternatively only in a subset of the coding hierarchy. An alternative embodiment uses samples from the enhancement layer in step 2) to generate the intra prediction signal. For spatial scalable enhancement layers, the base layer may be upsampled using an interpolation filter to apply an error magnitude.

[0333] An alternative embodiment is to divide the enhancement layer blocks into smaller block sizes (a i ) into multiple blocks (e.g., a 16×16 block (a) is divided into 16 4×4 blocks (a i The above algorithm can be divided into sub-blocks (a i ) and the corresponding base layer block (b i ) and applies to blocks (a i After the prediction of block (a), residual coding is applied, and the result is i+1 ) is used to predict

[0334] An alternative embodiment is (b) or (b i ) using the surrounding sample values ​​to determine the predicted intra prediction mode (p best For example, for a 4×4 block (a i ) is the corresponding 2 × 2 base layer block (b i ), (b i ) are the samples around the predicted intra prediction mode (pbest ) is used to determine the 4×4 block (c i ) is used to form

[0335] In an alternative embodiment, the encoder can send a refinement intra prediction direction signal to the decoder. For example, in video codecs such as HEVC, most intra prediction modes correspond to angles at which boundary pixels are used to form the prediction signal. An offset to the optimal mode is added to the predicted prediction mode (p best ) The refinement modes are entropy coded.

[0336] Intra prediction modes are usually coded depending on their probability. In H.264 / AVC, the maximum likelihood mode is determined based on the modes used in the (spatial) neighborhood of the block. In the HEVC list, the maximum likelihood modes are created. These maximum likelihood modes can be selected using fewer symbols in the bitstream than the total number of modes would require. An alternative embodiment selects a predicted intra prediction mode (p) for block (a) (determined as described in the previous algorithm) as the maximum likelihood mode or as a member of a list of maximum likelihood modes. best ) to use the

[0337] [Method 4: Intra prediction using border regions] In a scalable video decoder to form an intra prediction signal for a block (a) of a scalable or quality enhancement layer (see FIG. 45), lines of samples (b) from surrounding areas of the same layer are used to fill in the block area. These samples are taken from already coded areas (usually, but not necessarily on the top and left boundaries).

[0338] The following alternative variations of selecting these pixels can be used: a) If a pixel in the surrounding region has not yet been coded, the pixel value is not used to predict the current block. b) If a pixel in the surrounding region has not yet been coded, the pixel value is derived (eg, by repetition) from neighboring pixels that have already been coded. c) If a pixel in the surrounding region has not yet been coded, the pixel value is derived from the pixel in the corresponding region of the decoded base layer image.

[0339] To form an intra prediction of block (a), a neighboring line of pixels (b), derived as described above, is used as a template to estimate the intra prediction of each line (a) of block (a). j ) fill in the gap.

[0340] Block (a) Line (a j ) are filled one by one along the x-axis. To achieve the best possible prediction signal, the columns of template samples (b) are shifted along the y-axis to fill the associated lines (a j ) for the predicted signal (b´ j ) is formed.

[0341] To find the best prediction within each line, we use a shift offset (o j ) is the resulting predicted signal (a j ) and the sample values ​​of the corresponding line in the base layer.

[0342] Suppose, (o j If a is a non-integer value, the interpolation filter is j ) can be used to map the values ​​of (b) to integer sample positions of

[0343] If spatial scalability is used, an interpolation filter may be used to create a matching number of sample values ​​for the corresponding lines of the base layer.

[0344] The fill direction (x-axis) can be horizontal (left-right), vertical (up-down), diagonal or at any other angle. The samples used for template line (b) are the immediately adjacent samples of the block along the x-axis. Template line (b) translates along the y-axis, which forms a 90° angle with the x-axis.

[0345] To find the optimal direction of the x-axis, a full intra prediction signal is generated for block (a). The angle with the smallest error magnitude between the prediction signal and the corresponding base layer block is selected. The number of possible angles is limited.

[0346] Both the encoder and decoder run the same algorithm to determine the best prediction angle and offset. No explicit angle or offset information needs to be signaled in the bitstream. In an alternative embodiment, only the samples of the base layer image are i ) can be used to determine

[0347] In an alternative embodiment, the predicted offset (o i ) is signaled in the bitstream. Entropy coding is used to code the refinement offset value.

[0348] In an alternative embodiment, a refinement of the predicted direction (e.g., a difference value) is signaled in the bitstream. Entropy coding can be used to code the refinement direction value.

[0349] Suppose the line (b´ j If the optimal offset (o) is used for prediction, an alternative embodiment uses a threshold to select it. j If the error magnitude for line (c i ) is the block line (a j ) value. If the optimal offset (oj If the error magnitude for the (upsampled) base layer signal is greater than or equal to a threshold, then the (upsampled) base layer signal is j ) is used to determine the value of

[0350] [Method 5: Alternate forecast parameters] Further prediction information is inferred similarly to methods 1-3, for example for partitions of blocks within subblocks.

[0351] For a block (a) of the spatial or quality enhancement layer image, a representation of the corresponding block (b) of the base layer image is determined, which covers the same image region.

[0352] A prediction signal for enhancement layer block (a) is formed using the following method. 1) A predicted signal is generated for each possible value of the tested parameter. 2) Best prediction mode (p best ) is determined by minimizing the magnitude of the error (eg, the sum of absolute differences) between the prediction signal and the decoded base layer block (b). 3) The predictions selected in step 2) (p best ) mode is used to generate a prediction signal for enhancement layer block (a).

[0353] Both the encoder and the decoder can perform the same prediction step to select the optimal prediction mode among the possible candidates and generate the same prediction signal. The actual prediction mode is not signaled in the coded video bitstream.

[0354] The choice of prediction method may be signaled in the bitstream and coded using entropy coding. Within a hierarchical block subdivision structure, the coding method may be alternatively selected within every sublevel or for only a subset of the coding hierarchy.

[0355] The following description briefly summarizes some of the above embodiments.

[0356] Enhancement layer coding with multiple methods for generating an intra prediction signal using reconstructed base layer samples Main embodiment: For coding blocks in an enhancement layer, multiple methods for generating an intra prediction signal using reconstructed base layer samples are provided in addition to a method for generating a prediction signal based only on reconstructed enhancement layer samples.

[0357] Sub-Example: · Multiplexing methods include the following: The (upsampled / filtered) reconstructed base layer signal is directly used as the enhancement layer prediction signal. · Multiplexing methods include the following: The (upsampled / filtered) reconstructed base layer signal is combined with a spatial intra prediction signal, where the spatial intra prediction is derived based on difference samples for neighboring blocks, which represent the difference between the reconstructed enhancement layer signal and the (upsampled / filtered) reconstructed base layer signal (see aspect A). Multiplexing methods include the following: A conventional spatial intra prediction signal (obtained using neighboring reconstructed enhancement layer samples) is combined with an (upsampled / filtered) base layer residual signal (the inverse transform of the base layer transform coefficients or the difference between the base layer reconstruction and the base layer prediction) (see aspect B). · Multiplexing methods include the following: The (upsampled / filtered) reconstructed base layer signal is combined with a spatial intra prediction signal, where the spatial intra prediction is derived based on reconstructed enhancement layer samples of neighboring blocks. The final prediction signal is obtained by weighting the spatial prediction signal and the base layer prediction signal in such a way that different frequency components use different weightings (see aspect C1). This can be achieved, for example, by any of the following: Filtering the base layer prediction signal with a low-pass filter and the spatial intra prediction signal with a high-pass filter and adding the resulting filtered signals (see aspect C2). o Transform the base layer prediction signal and the enhancement layer prediction signal and overlap the resulting transformation block, where different weighting coefficients are used for different frequency positions (see embodiment C3). The resulting transformation block can be inverse transformed and used as the enhancement layer prediction signal. Alternatively, the resulting transformation coefficients are added to the scaled transmitted transformation coefficient levels and then inverse transformed to obtain a reconstructed block before deblocking and in-loop processing (see embodiment C4). For the method using the reconstructed base layer signal, the following versions can be used: this can be fixed or it can be signaled at sequence level, picture level, slice level, maximum coding unit level, coding unit level, or it can be made dependent on other coding parameters. o Reconstructed base layer samples before deblocking and in-loop processing (e.g. sample adaptive offset filter or adaptive loop filter). o The reconstructed base layer samples after deblocking and before in-loop processing (such as a sample adaptive offset filter or adaptive loop filter). ○ The reconstructed base layer samples after deblocking and further in-loop processing (such as a sample-adaptive offset filter or an adaptive loop filter) or during multiple in-loop processing steps (see aspect D) Multiple versions of the method using the (upsampled / filtered) base layer signal may be used. The upsampled / filtered base layer signals employed for these versions may differ in the interpolation filters used (including those that filter integer sample positions). Or the upsampled / filtered base layer signal for the second version may be obtained by filtering the upsampled / filtered base layer signal for the first version. The selection of one of the different versions may be signaled at sequence level, picture level, slice level, maximum coding unit level, coding unit level. It may be inferred from the characteristics of the corresponding reconstructed base layer signal or the transmitted coding parameters (see aspect E). Different filters may be used to upsample / filter the reconstructed base layer signal (see aspect E) and the base layer residual signal (see aspect F). For base layer blocks where the residual signal is zero, it may be replaced by another signal derived from the base layer (e.g. a high-pass filtered version of the reconstructed base layer block), (see aspect G). For modes using spatial intra prediction, unavailable neighboring samples in the enhancement layer (due to a particular coding order) may be replaced by corresponding samples of the upsampled / filtered base layer signal (see aspect H). For modes that use spatial intra prediction, the coding of the intra prediction modes may be changed. The list of maximum likelihood modes includes the intra prediction modes of the collocated base layer signal. In a particular version, the enhancement layer images are decoded in a two-stage process. In a first stage, only the blocks that only use the base layer signal for prediction (without using neighboring blocks) or the inter prediction signal are decoded and reconstructed. In a second stage, the remaining blocks that use neighboring samples for prediction can be reconstructed. For the blocks reconstructed in the second stage, the spatial intra prediction concept is extended (see aspect I). Depending on the availability of already reconstructed blocks, not only the upper and left neighboring samples of the current block can be used for spatial intra prediction, but also the lower and right neighboring samples.

[0358] Enhancement layer coding with multiple methods for generating an inter prediction signal using reconstructed base layer samples Main embodiment: For coding blocks in an enhancement layer, multiple methods for generating an inter prediction signal using reconstructed base layer samples are provided in addition to a method for generating a prediction signal based only on reconstructed enhancement layer samples.

[0359] Sub-Example: Multiplexing methods include the following: A conventional inter prediction signal (derived by motion compensated interpolation of already reconstructed enhancement layer pictures) is combined with an (upsampled / filtered) base layer residual signal (the inverse transform of the base layer transform coefficients or the difference between the base layer reconstruction and the base layer prediction). · Multiplexing methods include the following: The (upsampled / filtered) reconstructed base layer signal is combined with a motion compensated prediction signal, where the motion compensated prediction signal is obtained by a motion compensated difference image, which represents the difference between the reconstructed enhancement layer signal and the (upsampled / filtered) reconstructed base layer signal with respect to a reference image (see embodiment J). Multiplexing methods include the following: The (upsampled / filtered) reconstructed base layer signal is combined into an inter prediction signal, where the inter prediction is derived by motion compensated prediction using the reconstructed enhancement layer images. The final prediction signal is obtained by weighting the inter prediction signal and the base layer prediction signal in such a way that different frequency components use different weightings (see embodiment C). This can be achieved, for example, by any of the following: Filtering the base layer prediction signal with a low-pass filter and the inter prediction signal with a high-pass filter and adding the resulting filtered signals. o Transform the base layer prediction signal and the inter prediction signal and overlap the resulting transform block, where different weighting coefficients are used for different frequency positions, which is either inverse transformed and used as the enhancement layer prediction signal to obtain a reconstructed block before deblocking and in-loop processing, or the resulting transform coefficients are added to the scaled transmitted transform coefficient levels and then inverse transformed. For the method using the reconstructed base layer signal, the following versions can be used: this can be fixed or it can be signaled at sequence level, picture level, slice level, maximum coding unit level, coding unit level, or it can be made dependent on other coding parameters. o Reconstructed base layer samples before deblocking and in-loop processing (e.g. sample adaptive offset filter or adaptive loop filter). The reconstructed base layer samples after deblocking and before any in-loop processing (such as sample adaptive offset filter or adaptive loop filter). o Reconstructed base layer samples after deblocking and in-loop processing (such as sample adaptive offset filter or adaptive loop filter) or during multiple in-loop processing steps (see aspect D). For base layer blocks where the residual signal is zero, it can be replaced by another signal derived from the base layer (e.g. a high-pass filtered version of the reconstructed base layer block) (see embodiment G). Multiple versions of the method using the (upsampled / filtered) base layer signal may be used. The upsampled / filtered base layer signals employed for these versions differ in the interpolation filters used (including those that filter integer sample positions). Or the upsampled / filtered base layer signal for the second version may be obtained by filtering the upsampled / filtered base layer signal for the first version. The selection of one of the different versions may be signaled at sequence level, picture level, slice level, maximum coding unit level, coding unit level. It may be inferred from the characteristics of the corresponding reconstructed base layer signal or the transmitted coding parameters (see embodiment E). Different filters may be used to upsample / filter the reconstructed base layer signal (see embodiment E) and the base layer residual signal (see embodiment F). For motion compensated prediction of the difference image (the difference between the enhancement layer reconstruction and the upsampled / filtered base layer signal) (see embodiment J), different interpolation filters are used than for the motion compensated prediction of the reconstructed image. For motion compensated prediction of the difference image (the difference between the enhancement layer reconstruction and the upsampled / filtered base layer signal) (see embodiment J), an interpolation filter is selected based on the characteristics of the corresponding region in the difference image (or based on the coding parameters or based on information transmitted in the bitstream).

[0360] [Enhancement layer motion parameter coding] Main aspect: Use of multiple enhancement layer predictors and at least one predictor derived from the base layer for enhancement layer motion parameter coding.

[0361] Sub-Example: Adding the (scaled) base layer motion vector to the motion vector predictor list (see embodiment K). Use of a base layer block that covers the collocated sample at the central position of the current block (another possible derivation). ○ Scale motion vectors according to resolution ratio. Adding the motion data of the collocated base layer blocks to the merge candidate list (see embodiment K). Use of a base layer block that covers the collocated sample at the central position of the current block (another possible derivation). ○ Scaling motion vectors according to resolution ratio. If merge_flag is equal to 1 in the base layer, then do not add. Reordering the merge candidate list based on base layer merge information (see example L) o If a collocated base layer block is to be merged to a particular candidate, the corresponding enhancement layer candidate is used as the first entry in the enhancement layer merge candidate list. Reordering of motion predictor candidate list based on base layer motion predictor information (see embodiment L) o If a collocated base layer block uses a particular motion vector predictor, then the corresponding enhancement layer motion vector predictor is used as the first entry in the enhancement layer motion vector predictor candidate list. The derivation of the merge index (i.e., the candidates with which the current block is to be merged) is based on the base layer information in the collocated block (see embodiment M). As an example, if a base layer block is merged with a particular neighboring block, and it is signaled in the bitstream that it also merges an enhancement layer block, then no merge index is transmitted at all, but instead the enhancement layer block is merged with the same neighboring block (but in the enhancement layer) as the collocated base layer block.

[0362] [Enhancement layer partition and motion parameter inference] Main aspect: Inference of enhancement layer partitions and motion parameters based on base layer partitions and motion parameters (possibly requiring combining this aspect with any of the sub-aspects).

[0363] Sub-aspects: Deriving motion parameters for the NxM sub-blocks of the enhancement layer based on the collocated base layer motion data. Clustering blocks with the same derived parameters (or parameters with small differences) into larger blocks. Determining prediction and coding units (see embodiment T). Motion parameters may include number of motion hypotheses, reference indices, motion vectors, motion vector predictor identifiers, merge identifiers. Signaling one of multiple methods for generating an enhancement layer prediction signal. Such methods may include: ○ Motion compensation using the derived motion parameters and the reconstructed enhancement layer reference images. ○ Combining (a) the (upsampled / filtered) base layer reconstruction for the current picture and (b) a motion compensation signal using the obtained motion parameters and a reference picture of the enhancement layer generated by subtracting the (upsampled / filtered) base layer reconstruction from the reconstructed enhancement layer picture. ○ Combining (a) the (upsampled / filtered) base layer residual (the difference between the reconstructed signal and the prediction, or the inverse transform of the coded transform coefficient values) for the current picture and (b) the motion compensation signal using the obtained motion parameters and the reconstructed enhancement layer reference picture. If a collocated block in the base layer is intra-coded, then the corresponding enhancement layer M×N block (or CU) is also intra-coded, where an intra prediction signal is derived using base layer information (see aspect U). For example, o The (upsampled / filtered) version of the corresponding base layer reconstruction is used as the intra prediction signal (see aspect U). o The intra prediction mode is derived based on the intra prediction mode used in the base layer, and this intra prediction mode is then used for spatial intra prediction in the enhancement layer. If the collocated base layer block for an MxN enhancement layer block (sub-block) is merged with a previously coded base layer block (or has the same motion parameters), then the MxN enhancement layer (sub-)block is also merged with the enhancement layer block corresponding to the base layer block used for merging in the base layer (i.e., the motion parameters are copied from the corresponding enhancement layer block) (see embodiment M).

[0364] [Transform coefficient level coding / context modeling] Main aspects: coding the transform coefficients using different scan patterns, modeling the context for the enhancement layer based on the coding mode and / or base layer data, and different initialization for the context modes.

[0365] Sub-Example: Introducing one or more additional scan patterns, e.g. horizontal and vertical scan patterns. Redefining the sub-blocks for the additional scan patterns. Instead of 4x4 sub-blocks, e.g. 16x1 or 1x16 sub-blocks can be used. Or 8x2 or 8x2 sub-blocks can be used. Additional scan patterns can be introduced only for blocks larger or equal to a certain size, e.g. 8x8 or 16x16 (see embodiment V). The selected scan pattern is signaled in the bitstream (if the coded block flag is equal to 1) (see embodiment N). A fixed context is used to signal the corresponding syntax element. Alternatively, the context derivation for the corresponding syntax element can rely on either: ○ The gradient of the collocated reconstructed base layer signal or the reconstructed base layer residual. Alternatively, the edges detected in the base layer signal. Transform coefficient distribution within collocated base layer blocks. The selected scan can be derived directly from the base layer signal (without any additional data being transmitted) based on the characteristics of the collocated base layer signal (see embodiment N). ○ The gradient of the collocated reconstructed base layer signal or the reconstructed base layer residual. Alternatively, the edges detected in the base layer signal. Transform coefficient distribution within collocated base layer blocks. The different scans can be realized in such a way that the transform coefficients are reordered after quantization at the encoder side and conventional coding is used. At the decoder side, the transform coefficients are decoded conventionally and reordered before scaling and inverse transform (or after scaling and before inverse transform). To code the significance flags (subgroup flags and / or significance flags for a single transform coefficient), the following modifications may be used in the enhancement layer: o A separate context model is used for all or a subset of the coding modes that use base layer information. It is also possible to use different context models for different modes that have base layer information. o Context modeling can rely on data of collocated base layer blocks (eg, the number of significant transform coefficients for a particular frequency location) (see embodiment O). ○ A generalized template may be used that estimates both the number of important transform coefficient levels already coded in the spatial neighborhood of the coefficient to be coded and the number of important transform coefficients in the collocated base layer signal at similar frequency positions (see embodiment O). To encode the last significant scan position, the following modifications can be used in the enhancement layer: o A separate context model is used for all or a subset of the coding modes that use base layer information. It is also possible to use different context models for different modes that have base layer information (see example P). Context modeling can rely on data in collocated base layer blocks (e.g., transform coefficient distribution in the base layer, base layer gradient information, last scan position in the collocated base layer block). o The last scan position can be encoded as the difference relative to the last base layer scan position (see example S). · Use of different context initialization tables for base and enhancement layers.

[0366] [Backward adaptive enhancement layer coding using base layer data] Main Example: Use of base layer data to derive enhancement layer coding parameters.

[0367] Sub-aspects: Deriving merging candidates based on the (potentially upsampled) base layer reconstruction. In the enhancement layer, only the use of merging is signaled. However, in practice, the candidates used to merge the current block are derived based on the reconstructed base layer signal. Thus, for every merging candidate, the error magnitude between the (potentially upsampled) base layer signal for the current enhancement layer block and the corresponding predicted signal (derived using the motion parameters for the merging candidate) is evaluated for every merging candidate (or a subset thereof). The merging candidate associated with the smallest error magnitude is then selected. Also, the error magnitude can be calculated in the base layer using the reconstructed base layer signal and the base layer reference image (see example Q). Obtaining merge candidates based on the (potentially upsampled) base layer reconstruction. Motion vector differentials are not coded but are inferred based on the reconstructed base layer. For the current block, determining a motion vector predictor and evaluating a set of defined searches located around the motion vector predictor. For each search position, determining the error magnitude between the (potentially upsampled) base layer signal for the current enhancement layer block and the replaced reference frame (the replacement is given by the search position). Selecting the search position / motion vector that results in the smallest error magnitude. The search may be divided into several stages. For example, a full pel search is performed first. Then, a half pel search is performed around the full pel vectors. Then, a quarter pel search is performed around the best full / half pel vectors. The search may also be performed in the base layer using the reconstructed base layer signal and the base layer reference images. The found motion vectors are then scaled according to the resolution change between the base layer and the enhancement layer (see embodiment Q). Derive an intra prediction mode based on the (potentially upsampled) base layer reconstruction. The intra prediction mode is inferred based on the reconstructed base layer, although it is not coded. For each possible intra prediction mode (or a subset thereof), determine the error magnitude between the (potentially upsampled) base layer signal for the current enhancement layer block and the intra prediction signal (using the tested prediction node). Select the prediction mode that results in the smallest error magnitude. The error magnitude calculation can also be done in the base layer, using the reconstructed base layer signal and the intra prediction signal in the base layer. Furthermore, the intra block can be decomposed into 4x4 blocks (or another block size). Then, for each 4x4 block, a separate intra prediction mode is determined (see embodiment Q). The intra prediction signal may be determined by row-wise or column-wise alignment of boundary samples with the reconstructed base layer signal. To derive the shift between the neighboring samples and the current line / column, the error magnitude is calculated between the shifted line / column of the neighboring samples and the reconstructed base layer signal. The shift that results in the smallest error magnitude is then selected. As neighboring samples, (upsampled) base layer samples or enhancement layer samples may be used. Also, the error magnitude may be calculated directly in the base layer (see embodiment W). · Using the adaptation methods described below for the derivation of alternative coding parameters such as block partitions.

[0368] A further brief summary of the embodiment is provided below. In particular, the embodiment is described.

[0369] A1) A scalable video decoder is reconstructing (80) a base layer signal (200a, 200b, 200c) from the encoded data stream (6); reconstructing (60) the enhancement layer signal (360); Reconstruction (60) subjecting the reconstructed base layer signal (200a, 200b, 200c) to a resolution or quality refinement (220) to obtain an inter-layer predicted signal (380); Calculating (260) a difference signal between an already reconstructed portion (400a or 400b) of the enhancement layer signal and an inter-layer predicted signal (380); a first portion (440, for example in FIG. 46 ) juxtaposed to a portion of the enhancement layer signal (360) currently to be reconstructed, spatially predicting (260) the difference signal from a second portion (460) of the difference signal, the second portion being spatially adjacent to said first portion and belonging to an already reconstructed portion of the enhancement layer signal (360), to obtain a spatial intra prediction signal; combining (260) the inter-layer prediction signal (380) and the spatial intra prediction signal to obtain an enhancement layer prediction signal (420); The method includes predictively reconstructing (320, 580, 340, 300, 280) an enhancement layer signal (360) using the enhancement layer prediction signal (420). According to embodiment A1, the base layer signal may be reconstructed by the base layer decoding stage 80 from the coded data stream 6 or sub-stream 6a, respectively, for example in the above-mentioned block-based prediction method with transform decoding as far as the base layer residual signal 640 / 480 is concerned. However, other alternative reconstructions are also possible. As far as the reconstruction of the enhancement layer signal 360 by the enhancement layer decoding stage 60 is concerned, the improvement in resolution or quality that the reconstructed base layer signal 200a, 200b or 200c undergoes may mean, for example, upsampling in the case of a resolution improvement, or copying in the case of a quality improvement, or tone mapping from n bits to m bits (m>n) in the case of a bit depth improvement. The calculation of the difference signal is done pixel by pixel, i.e. the juxtaposed pixels of the enhancement layer signal on the one hand and the prediction signal 380 on the other hand are subtracted from each other, and this is done for each pixel location. The spatial prediction of the difference signal can be done by some method, like transmitting intra prediction parameters, like the intra prediction direction, in the coded data stream 6 or in the sub-stream 6b, and copying / interpolating already reconstructed pixels that border the part of the enhancement layer signal 360 currently to be reconstructed along this intra prediction direction in the current part of the enhancement layer signal. Combining can mean addition, weighted sum or more elaborate combination, like combination that weights contributions in the frequency domain differently. Predictive reconstruction of the enhancement layer signal 360 using the enhancement layer prediction signal 420 means entropy decoding and inverse transform of the enhancement layer residual signal 540 and combination 340 of the enhancement layer prediction signal 420 and the latter 540, as shown in the figure.

[0370] B1) A scalable video decoder is Decoding (100) a base layer residual signal (480) from the encoded data stream (6); reconstructing (60) the enhancement layer signal (360); Reconstruction (60) subjecting the reconstructed base layer residual signal (480) to a resolution or quality refinement (220) to obtain an inter-layer residual prediction signal (380); spatially predicting (260) a portion of an enhancement layer signal (360) currently to be reconstructed from an already reconstructed portion of the enhancement layer signal (360) to obtain an enhancement layer intra-prediction signal; combining (260) the inter-layer residual prediction signal and the enhancement layer intra prediction signal to obtain an enhancement layer prediction signal (420); predictively reconstructing (340) an enhancement layer signal (360) using the enhancement layer prediction signal (420). Decoding of the base layer residual signal from the encoded data stream may be performed using entropy decoding and an inverse transform, as shown in the figure. Additionally, the scalable video decoder may optionally perform reconstruction of the base layer signal itself by predictively decoding by deriving a base layer prediction signal 660 and combining this signal with the base layer residual signal 480. As just mentioned, this is merely optional. As far as the reconstruction of the enhancement layer signal is concerned, the resolution or quality improvement is performed as indicated above for embodiment A). Also, as far as the spatial prediction of parts of the enhancement layer signal is concerned, this spatial prediction can be performed as exemplarily outlined in A) for the different signals. As far as the combination and predictive reconstruction is concerned, similar remarks are valid. However, it should be mentioned that the base layer residual signal 480 in embodiment B) is not restricted to be equal to the explicitly signaled version of the base layer residual signal 480. Rather, it is possible for the scalable video decoder to subtract any reconstructed base layer signal version 200 with the base layer prediction signal 660, thereby obtaining a base layer residual signal 480 that deviates from the explicitly signaled one by deviations stemming from a filter function such as filters 120 or 140. The latter condition is also valid for other embodiments in which the base layer residual signal is involved in inter-layer prediction.

[0371] C1) Scalable video decoder: reconstructing (80) a base layer signal (200a, 200b, 200c) from the encoded data stream (6); reconstructing (60) the enhancement layer signal (360); Reconstruction (60) subjecting the reconstructed base layer signal (200) to a resolution or quality refinement (220) to obtain an inter-layer predicted signal (380); spatially or temporally predicting (260) a portion of the enhancement layer signal (360) currently to be reconstructed from an already reconstructed portion of the enhancement layer signal (360) (400a,b in the "spatial" case; 400a,b,c in the "temporal" case) to obtain an enhancement layer intra-prediction signal; forming (260) a weighted average of the inter-layer prediction signal and the enhancement layer intra prediction signal (380) at the portion currently to be reconstructed to obtain the enhancement layer prediction signal (420), such that the weights that the inter-layer prediction signal and the enhancement layer intra prediction signal (380) contribute to the enhancement layer prediction signal (420) vary across different spatial frequency components; predictively reconstructing (340) an enhancement layer signal (360) using the enhancement layer prediction signal (420).

[0372] C2) Here, forming the weighted average (260) involves filtering (260) the inter-layer prediction signal (380) with a low-pass filter and filtering (260) the enhancement layer intra-prediction signal with a high-pass filter to obtain a filtered signal for the portion currently to be reconstructed, and summing the resulting filtered signals. C3) Here, forming (260) the weighted average includes transforming (260) the inter-layer prediction signal and the enhancement layer intra prediction signal to obtain transform coefficients in the currently to-be-reconstructed portion, convolving (260) the obtained transform coefficients using different weighting factors for different spatial frequency components to obtain convolved transform coefficients, and inverse transforming the convolved transform coefficients to obtain the enhancement layer prediction signal. C4)...

Claims

1. 1. A video decoder configured to reconstruct a current block of a current image of a first layer of an encoded data stream, the video decoder comprising: The reconstruction is extracting from the encoded data stream an index to a list of reference pictures, the list including a number of pre-decoded pictures, and selecting one of the reference pictures that belongs to a second layer of the encoded data stream based on the index; extracting, for the current block, displacement information from the encoded data stream indicating a displacement between the current image and the reference image; identifying a reference block in the reference image based on the displacement information; predicting the current block using a prediction signal based on the reference block, the video decoder being configured to weight the prediction signal using a weight; carried out by Video decoder.

2. The video decoder of claim 1 , configured to derive the weights from the encoded data stream.

3. The video decoder of claim 1 , configured to determine the weights based on the current picture and / or the reference pictures.

4. 1. A video encoder configured to encode a current block of a current image into a first layer of an encoded data stream, the video encoder comprising: The encoding is encoding an index into the encoded data stream that points to a list of reference pictures including a plurality of pre-encoded pictures, the index identifying one of the reference pictures belonging to a second layer of the encoded data stream; identifying, for the current block, a reference block in the reference image based on displacement information indicative of a displacement between the current image and the reference image; encoding said displacement information into said encoded data stream; predicting the current block using a prediction signal based on the reference block, the video encoder being configured to weight the prediction signal using a weight; carried out by Video encoder.

5. The video encoder of claim 4 configured to encode the weights into the encoded data stream.

6. The video encoder of claim 4 , configured to determine the weights based on the current picture and / or the reference picture.

7. 1. A video decoding method comprising the steps of reconstructing a current block of a current image of a first layer of an encoded data stream, the method comprising: The reconstructing step includes: extracting from the encoded data stream an index to a list of reference pictures, the list including a number of pre-decoded pictures, and selecting one of the reference pictures that belongs to a second layer of the encoded data stream based on the index; extracting, for the current block, displacement information from the encoded data stream indicative of a displacement between the current image and the reference image; identifying a reference block in the reference image based on the displacement information; a prediction signal based on the reference block, the method including a step of weighting the prediction signal with a weight; and predicting the current block using the prediction signal. Video decoding method.

8. 1. A video encoding method comprising encoding a current block of a current image into a first layer of an encoded data stream, the method comprising: The encoding step includes: encoding an index into the encoded data stream that points to a list of reference pictures including a plurality of pre-encoded pictures, the index identifying one of the reference pictures belonging to a second layer of the encoded data stream; identifying, for the current block, a reference block in the reference image based on displacement information indicative of a displacement between the current image and the reference image; encoding said displacement information into said encoded data stream; a prediction signal based on the reference block, the method comprising the step of weighting the prediction signal with a weight; and predicting the current block using the prediction signal. Video encoding methods.

Images