Scalable video coding using base layer hints for enhancement layer motion parameters

Abstract

To provide a concept for scalable video coding that achieves higher coding efficiency.SOLUTION: A temporally aligned image 22a of a base layer signal 200 is subdivided into several blocks 104. The blocks 104 are located around an area that locally corresponds to the exemplary depicted current portion 28. This is exactly the case with spatially predicted blocks in the enhancement layer signal 400. Spatial prediction parameters are included or signaled in the base layer substream 6a for those blocks 104 in the base layer signal 200. The selection of the spatial prediction mode is signaled to the base layer signal 200. For blocks 28 for which temporal intra-layer prediction 32 is selected, intermediate prediction parameters, such as motion parameters, are determined using one of the following methods to allow reconstruction of the enhancement layer signal from the coded data stream.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001] The present invention relates to scalable video coding. [Background technology]

[0002] In non-scalable coding, intra-coding refers to a coding technique that utilizes only data from already-encoded parts of the current picture (e.g., reconstructed samples, coding modes, or symbol statistics) rather than reference data from previously encoded pictures. For example, intra-coded pictures (intra-pictures) are used in broadcast bitstreams to synchronize decoders to the bitstream at so-called random access points. Intra-pictures are also used to limit error propagation in error-prone environments. Typically, the first picture in a coded video sequence must be coded as an intra-picture, since the picture used as the reference picture is not available there. Intra-pictures are also used in scene cuts where temporal prediction usually cannot provide a suitable prediction signal.

[0003] Furthermore, intra-coding modes are used for specific regions / blocks in so-called intermediate images, where they perform better than inter-coding modes in terms of rate-distortion efficiency. This is often the case in smooth regions as well as in regions where temporal prediction is quite poor (occlusions, partial dissolution or fades).

[0004] In scalable coding, the concept of intra-coding (coding of intra-pictures and coding of intra-blocks in intermediate pictures) is extended to all pictures belonging to the same access unit or time instant. Therefore, the intra-coding mode for spatial or quality enhancement layers can increase coding efficiency and simultaneously use inter-layer prediction from lower-layer pictures at the same time. This means that not only can already-coded parts of the current enhancement layer picture be used for intra-prediction, but also already-coded lower-layer pictures can be used at the same time. The latter concept is also called inter-layer intra-prediction.

[0005] In state-of-the-art hybrid video coding standards (such as H.264 / AVC or HEVC), images of a video sequence are partitioned into blocks of samples. The size of the blocks is either fixed or the coding scheme is provided with a hierarchical structure that allows blocks to be subdivided into smaller block sizes. Reconstruction of a block is usually obtained by generating a prediction signal for the block and adding a transmitted residual signal. The residual signal is usually transmitted using transform coding, which means that a quantization index list (also called transform coefficient levels) for the transform coefficients is transmitted using entropy coding techniques. Then, at the decoder side, these transmitted transform coefficient levels are scaled and inverse transformed to obtain a residual signal that is added to the prediction signal. The residual signal is generated by intra prediction (using only data already transmitted for the current time instant) or by inter prediction (using data already transmitted for a different time instant).

[0006] If inter-prediction is used, the prediction block is obtained by motion-compensated prediction using samples of an already reconstructed frame. This can be done by unidirectional prediction (using one reference picture and one set of motion parameters). Alternatively, the prediction signal can be generated by multi-hypothesis prediction. In the latter case, two or more prediction signals are superimposed. That is, for each sample, a weighted average is formed to form the final prediction signal. The (superimposed) multi-prediction signal is generated using different motion parameters for different hypotheses (e.g., different reference pictures or motion vectors). For unidirectional prediction, it is also possible to multiply the samples of the motion-compensated prediction signal by a constant factor and add a constant offset to form the final prediction signal. In multi-hypothesis prediction, such scaling and offset correction can be used for all hypotheses or for selected hypotheses.

[0007] In current state-of-the-art video coding techniques, an intra prediction signal for a block is obtained by predicting samples from the spatial neighbors of the current block (which are blocks reconstructed before the current block according to the sequentially processed blocks). In the latest standards, various prediction methods are used that perform prediction in the spatial domain. Fine-grained directional prediction modes, in which samples of neighboring blocks are filtered or unfiltered, are extended to a specific angle to generate a prediction signal. In addition, there are also plane-based and DC-based prediction modes that use samples of neighboring blocks to generate a plane prediction plane or a DC prediction block.

[0008] In older video coding standards (e.g., H.263, MPEG-4), intra prediction was performed in the transform domain. In this case, the transmitted coefficients were inverse quantized. Then, for a subset of the transform coefficients, the transform coefficient values ​​were predicted using corresponding reconstructed transform coefficients of neighboring blocks. The inverse quantized transform coefficients were added to the predicted transform coefficient values, and the reconstructed transform coefficients were used as input to an inverse transform. The output of the inverse transform formed the final reconstructed signal for the block.

[0009] In scalable video coding, base layer information is also utilized to aid the prediction process for the enhancement layer. In the state-of-the-art video coding standard for scalable coding (the SVC extension of H.264 / AVC), there is one additional mode to improve the coding efficiency of the intra prediction process in the enhancement layer. This mode is signaled at the macroblock level (a block of 16x16 luma samples). This mode is only supported if the collocated samples in the lower layer are coded using an intra prediction mode. If this mode is selected for a macroblock in a quality enhancement layer, the prediction signal is constructed by the collocated samples of the reconstructed lower layer signal before the deblocking filter operation. If the inter-layer intra prediction mode is selected in the spatial enhancement layer, the prediction signal is generated by extracting the collocated reconstructed base layer signal (after the deblocking filter operation). An FIR filter is used for this extraction. Generally, for inter-layer intra prediction modes, an additional residual signal is transmitted by transform coding. Also, if it is correspondingly signaled in the bitstream, the transmission of the residual signal can be omitted (inferred to be equal to zero). The final reconstructed signal is obtained by adding the reconstructed residual signal (obtained by scaling the transmitted transform coefficient levels and applying an inverse spatial transform) to the prediction signal. Summary of the Invention [Problem to be solved by the invention]

[0010] However, it would be preferable to be able to achieve higher coding efficiency in scalable video coding.

[0011] It is therefore an object of the present invention to provide a concept for scalable video coding that achieves higher coding efficiency. [Means for solving the problem]

[0012] This object is achieved by the subject matter of the independent claims in the concurrently filed documents.

[0013] In one embodiment of the present invention, information available from base layer encoding / decoding (i.e., base layer hints) is utilized to make enhancement layer motion-compensated prediction more efficient by encoding enhancement layer motion parameters more efficiently. In particular, a set of motion parameter candidates collected from adjacent, already reconstructed blocks of a frame of the enhancement layer signal is possibly extended by one or more base layer motion parameter sets of blocks of the base layer signal (base layer signal collocated with the block of the enhancement layer frame). As a result, the available quality of the set of motion parameter candidates is improved based on which motion-compensated prediction of a block of the enhancement layer signal is performed by selecting one of the motion parameter candidates from the extended set of motion parameter candidates and using the selected motion parameter candidate for prediction. Additionally or alternatively, the motion parameter candidate list of the enhancement layer signal is ordered depending on the base layer motion parameters involved in the encoding / decoding of the base layer. For this reason, the probability distribution for selecting the enhancement layer motion parameters from the ordered motion parameter candidate list is compressed, for example, so that an explicitly signaled index syntax element is coded using fewer bits (e.g., using entropy coding, etc.). Additionally or alternatively, the index used in the encoding / decoding of the base layer serves as the basis for determining the index into the motion parameter candidate list for the enhancement layer. For this reason, any signaling of the index for the enhancement layer can be completely avoided. Alternatively, simply the prediction deviation thus determined relative to the index is transmitted in the enhancement layer substream. As a result, coding efficiency is improved.

[0014] One embodiment of the present invention is that in scalable video coding, a better predictor for predictively coding an enhancement layer signal is achieved by forming an enhancement layer prediction signal from an inter-layer prediction signal and an enhancement layer intra-prediction signal with different weighting methods for different spatial frequency components to obtain the enhancement layer prediction signal (i.e., by forming a weighted average of the inter-layer prediction signal and the enhancement layer intra-prediction signal at the currently to be reconstructed portion), so that the weightings that the inter-layer prediction signal and the enhancement layer intra-prediction signal contribute to the enhancement layer prediction signal vary different spatial frequency components. Thus, it is possible to interpret the enhancement layer prediction signal from the inter-layer prediction signal and the enhancement layer intra-prediction signal in a manner optimized for the spectral characteristics of the individual contributing components (i.e., the inter-layer prediction signal on the one hand and the enhancement layer intra-prediction signal on the other hand). For example, the inter-layer prediction signal is obtained from the reconstructed base layer signal based on an improvement in resolution or quality. The inter-layer prediction signal is more accurate at low frequencies compared to high frequencies. As far as the enhancement layer intra-prediction signal is concerned, the characteristic is the opposite: its accuracy increases for high frequencies compared to low frequencies. In this example, at low frequencies, the contribution of the inter-layer prediction signal to the enhancement layer prediction signal exceeds the contribution of the enhancement layer intra-prediction signal to the enhancement layer prediction signal with the respective weightings. And as far as high frequencies are concerned, the contribution of the enhancement layer intra-prediction signal to the enhancement layer prediction signal does not exceed that of the enhancement layer intra-prediction signal. Therefore, a more accurate enhancement layer prediction signal is achieved. As a result, the coding efficiency is increased, resulting in a higher compression ratio.

[0015] According to various embodiments, different possibilities are described for incorporating the just outlined concept into any scalable video coding based concept. For example, the formation of a weighted average can be performed either in the spatial domain or in the transform domain. Performing a spectral weighted average requires a transform to be performed on the separate contributions (i.e., the inter-layer predicted signal and the enhancement layer intra prediction). However, it avoids spectrally filtering either the inter-layer predicted signal or the enhancement layer intra prediction signal in the spatial domain, including, for example, FIR or IIR filtering. However, performing the spectral weighted average in the spatial domain avoids the detour of the separate contributions to the weighted average via the transform domain. The decision of which domain to actually select for performing the spectral weighted average depends on whether the scalable video data stream contains residual signals in the form of transform coefficients for the portion currently to be composed in the enhancement layer signal. If not, the detour via the transform domain is avoided. On the other hand, if a residual signal is present, the detour via the transform domain is more advantageous since it allows the transmitted residual signal in the transform domain to be added directly to the spectrally weighted average in the transform domain.

[0016] One embodiment of the present invention is that scalable video coding is made more efficient by deriving / selecting a subblock subdivision to be used for enhancement layer prediction among a set of possible subblock subdivisions of an enhancement layer block by evaluating spatial variations of base layer coding parameters on the base layer signal. Thus, if so, less signaling overhead must be spent to signal this subblock subdivision in the enhancement layer data stream. The subblock subdivision selected in this way is used in predictively encoding / decoding the enhancement layer signal.

[0017] One embodiment of the present invention is that subblock-based coding of enhancement layer transform coefficient blocks can be made more efficient if the subblock subdivision of each transform coefficient block is controlled based on a base layer residual signal or a base layer signal. In particular, by utilizing each base layer hint, the subblocks become longer along a spatial frequency axis horizontal to the edge extension observable from the base layer residual signal or the base layer signal. Therefore, the shape of the subblocks can be adapted to the estimated distribution of the energy of the transform coefficients of the enhancement layer transform coefficient block so that, with increasing probability, each subblock is filled almost completely with either significant transform coefficients (i.e., transform coefficients not quantized to zero) or non-significant transform coefficients (i.e., only transform coefficients quantized to zero), while with decreasing probability, any subblock has the same number of significant transform coefficients on one side and non-significant transform coefficients on the other side. However, due to the fact that subblocks that do not have significant transform coefficients are efficiently signaled in the data stream, such as by using a single flag, and due to the fact that subblocks that are almost completely filled with significant transform coefficients do not require the waste of signaling capacity to code the non-significant transform coefficients interspersed therein, the coding efficiency for coding the transform coefficient blocks of the enhancement layer is increased.

[0018] One embodiment of the present invention is that the coding efficiency of scalable video coding is increased by substituting missing spatial intra prediction parameter candidates in spatial neighbors of a current block of the enhancement layer by using intra prediction parameters of co-located blocks of the base layer signal. Thus, the coding efficiency for coding spatial intra prediction parameters is likely to increase, or more precisely, to increase, due to the improved prediction quality of the set of intra prediction parameters of the enhancement layer. A suitable predictor for the intra prediction parameters for intra-predicted blocks of the enhancement layer is likely to be useful, thereby increasing the likelihood that signaling of the intra prediction parameters of each enhancement layer block will require fewer bits on average.

[0019] Further advantageous realisations are set out in the dependent claims.

[0020] The preferred embodiments are described in detail below with reference to the drawings. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram illustrating an embodiment of a scalable video encoder. [Figure 2] 2 is a block diagram illustrating one embodiment of a scalable video decoder that matches the scalable video encoder of FIG. 1. [Figure 3] FIG. 2 is a block diagram illustrating a more specific embodiment of a scalable video encoder. [Figure 4] FIG. 4 is a block diagram of a scalable video decoder that matches the scalable video encoder of FIG. 3. [Figure 5] FIG. 1 is a schematic diagram showing a video, its base layer and enhancement layer versions, further illustrating the encoding / decoding ordering. [Figure 6] FIG. 2 is a schematic diagram showing a portion of a layered video signal to illustrate possible prediction modes for an enhancement layer. [Figure 7] 1 is a block diagram illustrating the formation of an enhancement layer prediction signal using spectrally varying weighting between the enhancement layer intra prediction signal and the inter-layer prediction signal. FIG. [Figure 8] FIG. 10 is a schematic diagram of syntax elements possibly contained in an enhancement layer substream. [Figure 9] 8 is a schematic diagram showing a possible realization of the formation of FIG. 7 according to an embodiment in which the forming / combining is performed in the spatial domain. [Figure 10] 8 is a schematic diagram illustrating the realization of the formation of FIG. 7 according to an embodiment in which the formation / combination is performed in the spectral domain. [Figure 11] FIG. 2 is a schematic diagram showing a portion taken from a layered video signal to illustrate spatial intra prediction parameter derivation from a base layer to an enhancement layer signal. [Figure 12] FIG. 12 is a schematic diagram illustrating an extension of the derivation in FIG. [Figure 13] FIG. 2 is a schematic diagram illustrating a set of candidate spatial intra prediction parameters into which one spatial intra prediction parameter obtained from the base layer is inserted. [Figure 14] FIG. 2 is a schematic diagram showing a portion taken from a layered video signal to illustrate the granular derivation of prediction parameters from the base layer; [Figure 15] 15a and 15b are schematic diagrams showing how to select an appropriate subdivision for the current block using spatial variations of base layer motion parameters according to two different samples in the base layer. [Figure 15c] FIG. 10 is a schematic diagram illustrating a first possibility for selecting the coarsest of the possible sub-block subdivisions for the current enhancement layer block; [Figure 15d] FIG. 10 is a schematic diagram illustrating a second possibility for selecting the coarsest of the possible sub-block subdivisions for the current enhancement layer block; [Figure 16]FIG. 10 is a schematic diagram showing a portion taken from a layered video signal to illustrate the use of subblock subdivision derivations for current enhancement layer blocks; [Figure 17] FIG. 1 is a schematic diagram showing a portion taken from a layered video signal to illustrate the use of base layer hints for efficiently encoding enhancement layer motion parameter data. [Figure 18] FIG. 1 is a schematic diagram illustrating a first possibility for increasing the efficiency of enhancement layer motion parameter signaling. [Figure 19a] FIG. 10 is a schematic diagram illustrating a second possibility of utilizing base layer hints to make enhancement layer motion parameter signaling more efficient. [Figure 19b] FIG. 10 is a schematic diagram illustrating a first possibility of transferring base layer motion parameters in the order they appear in the list of enhancement layer motion parameter candidates. [Figure 19c] FIG. 10 is a schematic diagram illustrating a second possibility of transferring base layer motion parameters in the order listed in the enhancement layer motion parameter candidates. [Figure 20] FIG. 10 is a schematic diagram illustrating another possibility of utilizing base layer hints to make enhancement layer motion parameter signaling more efficient. [Figure 21] FIG. 10 is a schematic diagram illustrating a portion of a layered video signal to illustrate an embodiment in which the sub-block subdivision of a transform coefficient block is appropriately adjusted to hints obtained from the base layer. [Figure 22] 5A-5C are schematic diagrams illustrating different possibilities of how to obtain a suitable sub-block subdivision of a transform coefficient block from a base layer; [Figure 23] FIG. 2 is a block diagram illustrating a more detailed embodiment for a scalable video decoder. [Figure 24a] FIG. 24 is a block diagram illustrating a scalable video encoder that matches the scalable video decoder of FIG. [Figure 24b]FIG. 24 is a block diagram illustrating a scalable video encoder that matches the scalable video decoder of FIG. [Figure 25] FIG. 1 is a schematic diagram illustrating the generation of an inter-layer intra prediction signal by summing a (sampled / filtered) base layer reconstruction signal (BL Reco) and spatial intra prediction using the difference signal (EH Diff) of an already coded neighboring block. [Figure 26] Schematic diagram to explain the generation of an inter-layer intra prediction signal by summing the (extracted / filtered) base layer residual signal (BL Resi) and spatial intra prediction using reconstructed enhancement layer samples (EH Reco) of already coded neighboring blocks. [Figure 27] Schematic diagram to explain the generation of an inter-layer intra prediction signal by frequency-weighted summation of a (sampled / filtered) base layer reconstructed signal (BL Reco) and spatial intra prediction using reconstructed enhancement layer samples (EH Reco) of already coded neighboring blocks. [Figure 28] FIG. 2 is a schematic diagram for explaining base layer signals and enhancement layer signals used in the specification. [Figure 29] FIG. 10 is a schematic diagram illustrating motion compensation prediction in an enhancement layer. [Figure 30] FIG. 1 is a schematic diagram illustrating prediction using base layer residual and enhancement layer reconstruction. [Figure 31] FIG. 10 is a schematic diagram illustrating prediction using BL reconstruction and EL difference signals. [Figure 32] FIG. 10 is a schematic diagram illustrating a prediction using BL reconstruction and a second hypothesis of the EL difference signal. [Figure 33] FIG. 1 is a schematic diagram illustrating prediction using BL reconstruction and EL reconstruction. [Figure 34] As an example, a schematic diagram for explaining the decomposition of an image into square blocks and the corresponding four-branch tree structure is shown. [Figure 35] FIG. 10 is a schematic diagram illustrating the allowed decomposition of a square block into sub-blocks in the preferred embodiment. [Figure 36] 1 is a schematic diagram illustrating the positions of motion vector predictions, where (a) represents the positions of spatial candidates and (b) represents the positions of temporal candidates. [Figure 37] Schematic diagram for explaining the block merging algorithm (a) and redundancy check for spatial candidates (b). [Figure 38] Schematic diagram for explaining the block merging algorithm (a) and redundancy check for spatial candidates (b). [Figure 39] FIG. 1 is a schematic diagram for explaining the scanning directions (diagonal, vertical, horizontal) for a 4×4 transform block. [Figure 40] Schematic diagram illustrating scan directions (diagonal, vertical, horizontal) for an 8x8 transform block. The shaded areas define significant subgroups. [Figure 41] A 16x16 transform diagram in which only diagonal scans are defined. [Figure 42] FIG. 1 is a schematic diagram illustrating vertical scanning for 16×16 conversion as proposed in JCTVC-G703. [Figure 43] 1 is a schematic diagram illustrating the implementation of vertical and horizontal scanning for a 16×16 transform block, where coefficient subgroups are defined as either a row or a column, respectively. [Figure 44] FIG. 1 is a schematic diagram illustrating vertical and horizontal scanning for a 16×16 transform block. [Figure 45] FIG. 10 is a schematic diagram illustrating backward-adaptive enhancement-layer intra prediction using neighboring reconstructed enhancement-layer samples and reconstructed base-layer samples. [Figure 46] 1 is a schematic diagram showing an image / frame of the enhancement layer to explain the spatial insertion of the difference signal; FIG. DETAILED DESCRIPTION OF THE INVENTION

[0022] FIG. 1 illustrates in a general manner an embodiment for a scalable video encoder, into which the embodiments outlined below may be incorporated. The scalable video encoder of FIG. 1 is generally designated using reference sign 2 and receives and encodes video 4. The scalable video encoder 2 is configured to encode the video 4 into a data stream 6 in a scalable manner. That is, the data stream 6 includes a first portion 6a having the video 4 encoded therein with a first amount of information content and another portion 6b having the video 4 encoded therein with an amount of information content greater than that of the first portion 6a. For example, the information content amounts of portions 6a and 6b differ in quality or fidelity, i.e., the amount of pixel-wise deviation from the original video 4 and / or in spatial resolution. However, other forms of differing information content may also be applied, such as color fidelity. Portion 6a is referred to as a base layer data stream or base layer substream, while portion 6b is referred to as an enhancement layer data stream or enhancement layer substream.

[0023] Scalable video encoder 2 is configured to exploit redundancy between versions 8a and 8b of reconstructable video 4 from base layer substream 6a without enhancement layer substream 6b on the one hand, and from both substreams 6a and 6b on the other hand. To do so, scalable video encoder 2 uses inter-layer prediction.

[0024] As shown in FIG. 1, scalable video encoder 2 alternatively receives two versions 4a and 4b of video 4. Versions 4a and 4b differ in information content, just as do base layer substream 6a and enhancement layer substream 6b. Thus, for example, scalable video encoder 2 is configured to generate substreams 6a and 6b. As a result, base layer substream 6a has version 4a encoded therein, while enhancement layer data stream (substream) 6b has version 4b encoded therein using inter-layer prediction based on base layer substream 6b. The encoding of both substreams 6a and 6b is lossy.

[0025] Even if the scalable video encoder 2 only receives the original version of the video 4, the scalable video encoder 2 is configured to internally derive two versions 4a and 4b therefrom, for example, by obtaining the base layer version 4a by spatial downscaling and / or tone mapping from a higher bit depth to a lower bit depth.

[0026] Figure 2 illustrates a scalable video decoder that matches scalable video encoder 2 of Figure 1 in a similar manner suitable for incorporating the embodiments outlined below. The scalable video decoder of Figure 2 is generally indicated using reference sign 10. The scalable video decoder is generally configured to decode encoded data stream 6 to reconstruct enhancement layer version 8b of video therefrom if both portions 6a and 6b of data stream 6 reach scalable video decoder 10 in an intact manner, or to reconstruct base layer version 8a of video therefrom if portion 6b is unavailable, for example, due to transmission loss. That is, scalable video decoder 10 is configured to be able to reconstruct version 8a from base layer substream 6a alone, and to be able to reconstruct version 8b using inter-layer prediction from both portions 6a and 6b.

[0027] Before the more detailed following embodiments of the present invention (i.e., the embodiments shown in FIGS. 1 and 2) are explicitly described in detail, a more detailed implementation of the scalable video encoder and decoder of FIGS. 1 and 2 will be described with reference to FIGS. 3 and 4. FIG. 3 shows a scalable video encoder 2 comprising a base layer encoder 12, an enhancement layer encoder 14, and a multiplexer 16. Base layer encoder 12 is configured to encode a base layer version 4a of the input video. Enhancement layer encoder 14 is configured to encode an enhancement layer version 4b of the video. Thus, multiplexer 16 receives base layer substream 6a from base layer encoder 12 and enhancement layer substream 6b from enhancement layer encoder 14, and multiplexes them into encoded data stream 6 for transmission at the output.

[0028] As shown in FIG. 3, both encoders 12 and 14 are predictive encoders that use, for example, spatial and / or temporal prediction to encode their respective input versions 4a and 4b into their respective substreams 6a and 6b. In particular, encoders 12 and 14 are hybrid video block encoders. That is, each of encoders 12 and 14 is configured to encode its respective input version of video on a block-by-block basis, during which, for example, an image or frame of video versions 4a and 4b is selected between different prediction modes for each block into which the respective image or frame is subdivided. The different prediction modes of base layer encoder 12 include spatial and / or temporal prediction modes. Meanwhile, enhancement layer encoder 14 additionally supports inter-layer prediction modes. The subdivision within a block differs between the base layer and the enhancement layer. Prediction modes, prediction parameters for the selected prediction modes for various blocks, prediction residuals, and, optionally, block subdivisions of each video version are described by each encoder 12, 14 using a respective syntax including syntax elements that are, in turn, encoded into the respective substreams 6a, 6b using entropy coding. Inter-layer prediction may be utilized, for example, one or more times, to predict enhancement layer video, prediction modes, prediction parameters, and / or block subdivision samples, as mentioned in a few examples. Thus, both the base layer encoder 12 and the enhancement layer encoder 14 include predictive encoders 18a, 18b followed by entropy encoders 19a, 19b, respectively. Meanwhile, the predictive encoders 18a, 18b form syntax element streams using predictive coding from the inbound versions 4a, 4b, respectively. The entropy encoders 19a, 19b entropy encode the syntax elements output by the respective predictive encoders 18a, 18b. As just mentioned, the inter-layer prediction of encoder 2 involves a different time in the encoding procedure of the enhancement layer, and therefore predictive encoder 18b is shown connected to predictive encoder 18a, its output, and one or more of entropy encoder 19a.Similarly, entropy coder 19b optionally utilizes inter-layer prediction, for example, by predicting the context used for entropy coding from the base layer. Accordingly, entropy coder 19b is optionally shown connected to any of the elements of base layer coder 12.

[0029] In the same manner as Figure 2 for Figure 1, Figure 4 shows a possible implementation of a scalable video decoder 10 that fits into the scalable video encoder of Figure 3. Accordingly, scalable video decoder 10 of Figure 4 includes a demultiplexer 40 that receives data stream 6 to obtain substreams 6a and 6b, a base layer decoder 80 configured to decode base layer substream 6a, and an enhancement layer decoder 60 configured to decode enhancement layer substream 6b. As shown, decoder 60 is connected to base layer decoder 80 to receive information therefrom for utilizing inter-layer prediction. This enables base layer decoder 80 to reconstruct base layer version 8a from base layer substream 6a. Enhancement layer decoder 60 is then configured to reconstruct enhancement layer version 8b of video using enhancement layer substream 6b. Similar to the scalable video encoder of FIG. 3, the enhancement layer decoder 60 and the base layer decoder 80 each include an entropy decoder 100, 320 followed by a predictive decoder 102, 322 therein.

[0030] To simplify understanding of the following embodiments, FIG. 5 exemplarily illustrates different versions of video 4, namely, base layer versions 4a and 8a, which deviate from each other due to coding loss. Similarly, enhancement layer versions 4b and 8b, respectively, deviate from each other due to coding loss. The base layer signal and enhancement layer signal are composed of a sequence of images 22a and 22b, respectively, which are shown in FIG. 5 as being registered with each other along a time axis 24 (i.e., image 22a of the base layer version as well as the temporally corresponding image 22b of the enhancement layer signal). As mentioned above, image 22b represents video 4 with a higher spatial resolution and / or with a higher fidelity (e.g., a higher bit depth of the image sample values). Solid and dotted lines are used to indicate the encoding / decoding order defined between images 22a and 22b. According to the example shown in FIG. 5, the encoding / decoding order traverses pictures 22a and 22b in such a way that base layer picture 22a at a given time instant / instant is traversed before enhancement layer picture 22b at the same time instant of the enhancement layer signal. With respect to time axis 24, pictures 22a and 22b are traversed by encoding / decoding order 26 in their presentation order. However, an order that deviates from the presentation order of pictures 22a and 22b is also possible. Neither encoder 2 nor decoder 10 need sequentially encode / decode according to encoding / decoding order 26. Rather, encoding / decoding can be performed in parallel. The encoding / decoding order 26 defines the availability between portions of the base layer and enhancement layer signals that are adjacent to each other in a spatial, temporal, and / or inter-layer sense. Consequently, when encoding / decoding a current portion of the enhancement layer, the available portions of that current enhancement layer portion are defined through the encoding / decoding order. Thus, since only the adjacent parts that are available according to this encoding / decoding order 26 are used by the encoder for prediction, the decoder has access to the same source of information to refine the prediction.

[0031] With respect to the following figures, it is explained how the scalable video encoder or decoder described above with respect to figures 1 to 4 may form an embodiment of the present invention according to one example. A possible example of the example described below is discussed using the label "Example C."

[0032] In particular, Figure 6 illustrates image 22b of the enhancement layer signal, denoted using reference sign 360, and image 22a of the base layer signal, denoted using reference sign 200. Temporally corresponding images of different layers are shown in a manner indicated relative to one another with respect to time axis 24. Diagonal lines are used to distinguish portions 200 of the base and enhancement layer signals that have already been coded / decoded according to the coding / decoding order from portions 36 that have not yet been coded or decoded according to the coding / decoding order shown in Figure 5. Figure 6 also illustrates portion 28 of the enhancement layer signal 360 that is currently being coded / decoded.

[0033] According to the presently described embodiment, the prediction of portion 28 uses both intra-layer prediction within the enhancement layer itself and inter-layer prediction from the base layer to predict portion 28. However, the predictions are combined such that these predictions contribute to the final prediction of portion 28 in a spectrally varying manner. As a result, in particular, the ratio between both contributions is spectrally varying.

[0034] In particular, portion 28 is spatially or temporally predicted from an already reconstructed portion of enhancement layer signal 400 (i.e., the portion indicated by diagonal lines in enhancement layer signal 400 in FIG. 6). Spatial prediction is illustrated using arrow 30, while temporal prediction is illustrated using arrow 32. Temporal prediction involves motion-compensated prediction, for example, since motion vector information is transmitted in the enhancement layer substream for current portion 28. The motion vector indicates the replacement of a portion of a reference image of enhancement layer signal 400 to be copied to obtain the temporal prediction of current portion 28. Spatial prediction 30 involves estimating spatially neighboring portions within current portion 28, already coded / decoded portions of image 22b, and spatially neighboring current portion 28. For this purpose, intra-prediction information, such as an estimated (or angular) direction, is signaled in the enhancement layer substream for current portion 28. A combination of spatial prediction 30 and temporal prediction 32 may also be used. In any case, the result is an intra prediction signal 34 for the enhancement layer, as illustrated in FIG.

[0035] Inter-layer prediction is used to obtain another prediction of the current portion 28. For this purpose, the base layer signal 200 undergoes a resolution or quality refinement, with the portion 36 of the enhancement layer signal 400 corresponding spatially and temporally to the current portion 28, such that the inter-layer predicted signal for the current portion 28 has increased potential resolution. The refinement procedure is illustrated using arrow 38 in FIG. 6 and results in an inter-layer predicted signal 39 as shown in FIG. 7.

[0036] Thus, two prediction contributions 34 and 39 exist for the current portion 28. A weighted average of both contributions is then formed to obtain the enhancement layer prediction signal 42 for the current portion 28, in such a way that the weights that the inter-layer prediction signal and the enhancement layer intra prediction signal contribute to the enhancement layer prediction signal 42 vary differently in spatial frequency content, as shown diagrammatically at 44 in Figure 7. Figure 7 exemplarily shows the case where, for every spatial frequency component, the weights that the prediction signals 34 and 38 contribute to the final prediction signal add up to the same value 46 for all spectral components, however, spectrally varying the ratio between the weight applied to prediction signal 34 and the weight applied to prediction signal 39.

[0037] Alternatively, the prediction signal 42 is directly used by the enhancement layer signal 400 for the current portion 28. Alternatively, the residual signal is provided in the enhancement layer substream 6b of the current portion 28, resulting in a reconstructed version 54 of the current portion 28 by combining 50 with the prediction signal 42, e.g., by addition as shown in Figure 7. As an intermediate note, it is noted that both the scalable video encoder and decoder are hybrid video decoders / encoders that use predictive coding and that use transform coding to encode / decode the prediction residual.

[0038] 6 and 7, the enhancement layer substream 6b includes, for the current portion 28, intra prediction parameters 56 for controlling the spatial and / or temporal predictions 30 and 32, optionally weighting parameters 58 for controlling the formation of the spectral weighted average 41, and residual information 59 for signaling in the residual signal 48. The scalable video encoder, in turn, determines all of these parameters 56, 58, and 59 accordingly and inserts the parameters 56, 58, and 59 into the enhancement layer substream 6b. The scalable video decoder uses the parameters 56, 58, and 59 to reconstruct the current portion 28, as outlined above. All of these elements 56, 58, and 59 undergo some quantization (i.e., using a ratio / distortion cost function as quantization), and the scalable video encoder determines these parameters / elements accordingly. Interestingly, the encoder 2 uses the parameters / elements 56, 58, 59 determined in this way to obtain a reconstructed version 54 of the current portion 28, for example to serve as the basis for any prediction for subsequent portions of the enhancement layer signal 400 in the encoding / decoding order.

[0039] Different possibilities exist for the weighting parameters 58 and how they control the formation of the spectrally weighted average 41. For example, the weighting parameters 58 signal only one of two states for the current portion 28 (i.e., one state activating the formation of the spectrally weighted average described so far, and the other state deactivating the contribution of the inter-layer prediction signal 38). As a result, the final enhancement layer prediction signal 42 is then generated solely by the enhancement layer intra-prediction signal 34. The weighting parameters 58 for the current portion 28 switch between activation of the spectrally weighted average formation on the one hand and the inter-layer prediction signal 39 solely forming the enhancement layer prediction signal 42 on the other hand. The weighting parameters 58 are also designed to signal one of the three states / alternatives mentioned. Alternatively, the weighting parameters 58 also control the spectrally weighted averaging 41 with respect to the spectral change in the ratio between the weights that the prediction signals 34 and 39 contribute to the final prediction signal 42 for the current portion 28. It will be explained later that the spectrally weighted averaging 41 involves filtering one or both of the prediction signals 34 and 39 before summing them, for example using a high-pass filter and / or a low-pass filter. In that case, the weighting parameters 58 signal the filter characteristics for the filter to be used for the prediction of the current portion 28. Alternatively, it will be explained below that the spectral weighting in the spectrally weighted averaging 41 is achieved by individual weighting of spectral components in the transform domain, and therefore in this case the weighting parameters 58 can signal / set the values ​​of the individual weightings of these spectral components.

[0040] Additionally or alternatively, the weighting parameters for the current portion 28 can signal whether the spectral weighting in step 41 is performed in the transform domain or the spatial domain.

[0041] FIG. 9 illustrates an embodiment for performing spectrally weighted averaging in the spatial domain. Prediction signals 39 and 34 are shown as being obtained in the form of respective pixel arrays that correspond to the pixel raster of current portion 28. To perform spectrally weighted averaging, the pixel arrays of both prediction signals 34 and 39 are shown to undergo filtering. FIG. 9 illustrates filtering as an example by showing filter kernels 62 and 64 moving through the pixel arrays of prediction signals 34 and 39 to perform, for example, FIR filtering. However, IIR filtering is also possible. Furthermore, only one of prediction signals 34 and 39 may undergo filtering. Because the transfer functions of both filters 62 and 64 are different, summation 66 of the results of filtering the pixel arrays of prediction signals 39 and 34 yields the result of spectrally weighted averaging, i.e., enhancement layer prediction signal 42. In other words, summation 66 simply sums collocated samples in prediction signals 39 and 34 that have been filtered using filters 62 and 64, respectively. 62-66 result in the spectrally weighted average configuration 41. Figure 9 illustrates that, with residual information 59 present in the form of transform coefficients, signaling the residual signal 48 in the transform domain, an inverse transform 68 is used to result in the spatial domain in the form of a pixel array 70, such that the combination 52 resulting in the reconstructed version 55 is achieved by a simple pixel-wise addition of the residual signal array 70 and the enhancement layer prediction signal 42.

[0042] Recall again that prediction is performed by the scalable video encoder and decoder, using the prediction for reconstruction in the decoder and encoder, respectively.

[0043] FIG. 10 exemplarily illustrates how spectral weighting averaging is performed in the transform domain. Here, pixel arrays of predicted signals 39 and 34 undergo transforms 72 and 74, respectively, resulting in spectral decompositions 76 and 78, respectively. Each spectral decomposition 76 and 78 produces a transform coefficient array, one transform coefficient per spectral component. Each transform coefficient block 76 and 78 is multiplied by a corresponding block of weights, i.e., blocks 82 and 84. As a result, for each spectral component, the transform coefficients of blocks 76 and 78 are individually weighted. For each spectral component, the weighting values ​​of blocks 82 and 84 sum to a value common to all spectral components. However, this is not mandatory. Effectively, multiplier 86 between blocks 76 and 82 and multiplier 88 between blocks 78 and 84 represent spectral filtering in the transform domain, respectively. Transform coefficient / spectral component addition 90 then completes spectral weighting averaging 41 to yield a transform domain version of enhancement layer prediction signal 42 in the form of a block of transform coefficients. As shown in FIG. 10 , in the case of residual signal 59 signaling residual signal 48 in the form of a transform coefficient block, residual signal 59 is readily transform coefficient-wise additively combined (or otherwise combined) 52 with the transform coefficient block representing enhancement layer prediction signal 42 to yield a reconstructed version of current portion 28 in the transform domain. Thus, an inverse transform 84 applied to the additive result of combination 52 yields a pixel array reconstructing current portion 28, i.e., reconstructed version 54.

[0044] As mentioned above, the current parameters in enhancement layer substream 6b for current portion 28 signal whether averaging 41 is performed in the transform domain as shown in Figure 10, or in the spatial domain according to Figure 9. For example, suppose residual information 59 indicates the absence of any transform coefficient blocks for current portion 28, and the spatial domain is used. Alternatively, weighting parameters 58 switch between both domains, with residual information 59 including transform coefficients or residual information 59 not including transform coefficients.

[0045] It is then described that a difference signal is calculated and managed between an already reconstructed portion of the enhancement layer signal and the inter-layer predicted signal to obtain an inter-layer enhancement layer intra-prediction signal. A spatial prediction for a first portion of the difference signal is performed by reconstructing a second portion of the difference signal from a first portion of the difference signal that is collocated with a portion of the enhancement layer signal. A spatial prediction for a first portion of the difference signal is then reconstructed from a second portion of the difference signal that is spatially adjacent to the first portion of the enhancement layer signal and belongs to a previously reconstructed frame of the enhancement layer signal, and is then used to spatially predict the difference signal. Alternatively, a temporal prediction for the first portion of the difference signal is then reconstructed from a second portion of the difference signal that is collocated with a portion of the enhancement layer signal and belongs to a previously reconstructed frame of the enhancement layer signal, and is used to obtain a temporally predicted difference signal. The inter-layer predicted signal and the predicted difference signal are used to obtain an enhancement layer intra-prediction signal, which is then combined with the inter-layer predicted signal.

[0046] With reference to the following figures, it is described how a scalable video encoder or decoder such as that described above with reference to FIGS. 1-4 may be implemented to form an embodiment of the present invention according to another aspect of its application.

[0047] To explain this, reference is made to Figure 11, which illustrates a possibility of performing spatial prediction 30 of current portion 28. Consequently, the following description of Figure 11 is to be combined with the description of Figures 6 to 10. In particular, the following description will be explained later for an example of an implementation shown by reference to "embodiments" X and Y.

[0048] The situation shown in FIG. 11 corresponds to that shown in FIG. 6 . That is, a base layer signal 200 and an enhancement layer signal 400 are shown. Portions that have already been coded / decoded are indicated using diagonal lines. The portion of the enhancement layer signal 400 currently to be coded / decoded has neighboring blocks 92 and 94. Here, for illustrative purposes, block 92 is depicted above the current portion 28 and block 94 is depicted to the left, with both blocks 92 and 94 having the same size as the current block 28. However, size matching is not mandatory. Rather, the portions of the blocks into which the image 22b of the enhancement layer signal 400 is subdivided have different sizes. They are not even limited to squares; they may be rectangular or have other shapes. Furthermore, the current block 28 has neighboring blocks that are not explicitly depicted in FIG. 11 . However, the neighboring blocks have not yet been decoded / coded. That is, the neighboring blocks follow in coding / decoding order and are therefore unavailable for prediction. Beyond this, there are other blocks (such as block 96 adjacent to current block 28, for example, diagonally adjacent to the upper left corner of current block 28) than blocks 92 and 94 that have already been coded / decoded according to the coding / decoding order. However, blocks 92 and 94 are predetermined neighboring blocks that serve to predict intra prediction parameters for current block 28, which undergoes intra prediction 30 in the example considered here. The number of such predetermined neighboring blocks is not limited to two; it may be more or even one.

[0049] The scalable video encoder and scalable video decoder determine a set of predetermined neighboring blocks (here, blocks 92 and 94) from a set of already-encoded neighboring blocks. Here, blocks 92-96 depend on a predetermined sample location 98 in the current portion 28, such as its upper-left sample. For example, only those already-encoded neighboring blocks of the current portion 28 form a set of "predetermined neighboring blocks" that includes sample locations immediately adjacent to the predetermined sample location 98. In any case, the neighboring already-encoded / decoded blocks include a sample 102 adjacent to the current block 28 based on the sample values ​​from which an area of ​​the current block 28 is to be spatially predicted. For this purpose, spatial prediction parameters, such as 56, are signaled in the enhancement layer substream 6b. For example, the spatial prediction parameters for the current block 28 indicate the spatial direction in which the sample value of sample 102 should be copied into the area of ​​the current block 28.

[0050] In any case, at least as far as the relevant spatially corresponding region of the temporally corresponding image 22a is concerned, when spatially predicting the current block 28 using block-wise prediction, e.g., block-wise selection between spatial and temporal prediction modes, as described above, the scalable video decoder / encoder has already reconstructed (or, in the case of an encoder, encoded) the base layer 200 using the base layer sub-stream 6a.

[0051] 11, several blocks 104 into which the time-aligned image 22a of the base layer signal 200 is subdivided lie in and around an area that locally corresponds to the illustratively represented current portion 28. That is exactly the case for spatially predicted blocks in the enhancement layer signal 400. Spatial prediction parameters are included in or signaled in the base layer substream for those blocks 104 in the base layer signal 200. The selection of the spatial prediction mode is signaled to the base layer signal.

[0052] Here, illustratively, to allow reconstruction of an enhancement layer signal from the coded data stream for block 28 for which spatial intra-layer prediction 30 has been selected, the intra-prediction parameters are used and coded in the following bitstream:

[0053] Intra prediction parameters are often coded using the concept of "most likely intra prediction parameters," which is a fairly small subset of all possible intra prediction parameters. The "most likely intra prediction parameters" may be: For example, a most likely intra prediction parameter set may include one, two, or three intra prediction parameters. Alternatively, a set of all possible intra prediction parameters may include 35 intra prediction parameters. If an intra prediction parameter is included in the most likely intra prediction parameter set, it is signaled with fewer bits in the bitstream. If an intra prediction parameter is not included in the most likely intra prediction parameter set, its signaling in the bitstream requires more bits. Therefore, the amount of bits to be spent on syntax elements to signal the intra prediction parameters for the current intra-predicted block depends on the quality of the most likely, or possibly advantageous, intra prediction parameter set. Using this concept, on average, a low number of bits is required to code the intra prediction parameters. It is appreciated that the most likely intra prediction parameter set is appropriately obtained.

[0054] Typically, the set of most likely intra prediction parameters is selected in such a way that it includes the intra prediction parameters of directly neighboring blocks and / or additionally uses intra prediction parameters, often in the form of, for example, initialization parameters. For example, it is generally advantageous to include the intra prediction parameters of neighboring blocks in the set of most likely intra prediction parameters because the dominant gradient direction of the neighboring blocks is the same.

[0055] However, if the neighboring blocks are not coded in spatial intra prediction mode, their parameters are not available at the decoder side.

[0056] In scalable coding, it is however possible to use intra prediction parameters of co-located base layer blocks, and therefore, according to the embodiment outlined below, this situation is exploited by using intra prediction parameters of co-located base layer blocks in the case of neighboring blocks that are not coded in spatial intra prediction mode.

[0057] As a result, in accordance with FIG. 11, a set of likely advantageous intra prediction parameters for the current enhancement layer block is constructed by inspecting the intra prediction parameters of predetermined neighboring blocks and by exceptionally repartitioning to co-located blocks in the base layer if the predetermined neighboring block does not have suitable intra prediction parameters associated with it, for example, because each predetermined neighboring block is not coded in an intra prediction mode.

[0058] First, a check is made to see if a predetermined neighboring block, such as block 92 or 94 of current block 28, was predicted using a spatial intra prediction mode, i.e., whether a spatial intra prediction mode was selected for that neighboring block, so that the intra prediction parameters of that neighboring block are included in the set of likely intra prediction parameters for current block 28 or, alternatively, the intra prediction parameters of co-located block 108 of the base layer, if any. This process can be performed for each of predetermined neighboring blocks 92 and 94.

[0059] For example, if each predetermined neighboring block is not a spatially intra-predicted block, then the intra-prediction parameters of block 108 of base layer signal 200, which is co-located with current block 28, are included in the set of likely prediction parameters for current block 28, using something like default prediction. For example, co-located block 108 is determined using a predetermined sample position 98 of current block 28. That is, block 108 covers position 106, which locally corresponds to predetermined sample position 98 in temporally aligned image 22a of base layer signal 200. Naturally, a further check is performed to determine whether this co-located block 108 in base layer signal 200 is actually a spatially intra-predicted block. The case of FIG. 11 exemplarily illustrates this being the case. However, if the co-located block is also not coded in intra-prediction mode, the set of likely intra-prediction parameters for that predetermined neighboring block is left without any contribution. Alternatively, default intra-prediction parameters are used instead. That is, the default intra prediction parameters are inserted into the set of possibly advantageous intra prediction parameters.

[0060] Thus, if a block 108 collocated with the current block 28 is spatially intra-predicted, its intra-prediction parameters signaled in the base layer substream 6a are used, as a sort of substitute, for a predetermined neighboring block 92 or 94 of the current block 28 that does not have any intra-prediction parameters because the intra-prediction parameters are coded using another prediction mode, such as a temporal prediction mode.

[0061] According to another embodiment, in certain cases, the intra prediction parameters of a predetermined neighboring block are substituted by the intra prediction parameters of the co-located base layer block, even if each predetermined neighboring block is of an intra prediction mode. For example, a further check is performed for any predetermined neighboring block of an intra prediction mode, such as whether the intra prediction parameters meet a predetermined criterion. If the predetermined criterion is not met by the intra prediction parameters of a neighboring block, but the same criterion is met by the intra prediction parameters of the co-located base layer block, substitution is performed regardless of whether the closely neighboring block is intra-coded. For example, if the intra prediction parameters of a neighboring block do not represent an angular intra prediction mode (but, for example, a DC or planar intra prediction mode), but the intra prediction parameters of the co-located base layer block represent an angular intra prediction mode, the intra prediction parameters of the neighboring block are replaced by the intra prediction parameters of the base layer block.

[0062] The intermediate prediction parameters for the current block 28 are then determined based on syntax elements present in the encoded data stream, such as the enhancement layer substream 6b and the likely favorable intra prediction parameter set for the current block 28. That is, syntax elements are coded using fewer bits for intermediate prediction parameters for the current block 28 that are members of the likely favorable intra prediction parameter set than for the remaining members of the possible intra prediction parameter sets that do not lead to a likely favorable intra prediction parameter set.

[0063] The set of possible intra prediction parameters includes several angular modes according to which the current block is filled by copying from neighboring samples already coded / decoded by copying along the angular direction of the respective mode / parameter; one DC mode according to which the samples of the current block are set to a constant value determined, for example, by some average, based on neighboring samples already coded / decoded; and a planar mode according to which the samples of the current block are set to a value distribution following a linear function of slope and intercept of x and y, determined, for example, based on neighboring samples already coded / decoded.

[0064] Figure 12 shows the possibility of how spatial prediction parameter alternatives obtained from a collocated block 108 of the base layer can be used in conjunction with syntax elements signaled in the enhancement layer substream. Figure 12 shows, in an expanded manner, the current block 28 together with adjacent already coded / decoded samples 102 and predetermined neighboring blocks 92 and 94. Figure 12 also exemplarily shows the angular direction 112 indicated by the spatial prediction parameters of the collocated block 108.

[0065] The syntax element 114 signaled in the enhancement layer substream 6b for the current block 28 can signal, for example, as shown in FIG. 13, a conditionally coded index 118 into a list 122 of possible favorable intra prediction parameter results, illustrated here illustratively as angular directions 124. Alternatively, if the actual intra prediction parameters 116 are not in the most likely set 122, a candidate in list 122 may signal an index 123 into a list 125 of possibly excluded intra prediction modes, as shown at 127, thereby identifying the actual intra prediction parameters 116. The coding of the syntax element consumes fewer bits for the actual intra prediction parameters that belong to list 122. For example, the syntax element includes a flag and an index portion. The flag indicates whether to include or exclude a member of list 122, and whether the index points to either list 122 or list 125, i.e., whether to include or exclude a member of list 122. The syntax element includes a field that specifies one of the members 124 of the list 122 or an escape code, and, in the case of an escape code, a second field that specifies a member from a list 125 that includes or excludes the member of the list 122. The order among the members 124 in the list 122 is determined, for example, based on a default rule.

[0066] Therefore, the scalable video decoder obtains or retrieves syntax element 114 from enhancement layer substream 6b. Then, the scalable video encoder inserts syntax element 114 into enhancement layer substream 6b. Then, for example, syntax element 114 is used to index a spatial prediction parameter from list 122. In forming list 122, the aforementioned alternative is performed by checking whether predetermined neighboring blocks 92 and 94 are of a spatial prediction coding mode type. Otherwise, as described above, it is checked whether co-located block 108 is a spatially predicted block in turn, and if so, the same spatial prediction parameters, such as the angular direction 112 used to spatially predict this co-located block 108, are included in list 122. Also, if base layer block 108 does not contain suitable intra prediction parameters, list 122 is omitted from each predetermined neighboring block 92 or 94 without a contribution. To avoid the list 122 being empty, for example, because both predetermined neighboring blocks 92, 98 are intra-predicted, as well as the co-located block 108 lacking suitable intra-prediction parameters, at least one of the members 124 is unconditionally determined using default intra-prediction parameters. Alternatively, the list 122 is allowed to be empty.

[0067] Naturally, the embodiments described with respect to Figures 11-13 can be coupled with the embodiments outlined with respect to Figures 6-10. In particular, the intra prediction obtained using spatial intra prediction parameters derived by detouring through the base layer according to Figures 11-13 represents the enhancement layer inter prediction signal 34 in the embodiments of Figures 6-10, since it is combined in a spectrally weighted manner with the inter-layer prediction signal 38 as described above.

[0068] With respect to the following figures, it will be explained how a scalable video encoder or decoder, as described with respect to Figures 1 to 4, may form an embodiment of the present application according to another example of the application. Later, for the examples described below, additional implementation examples will be presented with reference to Examples T and U.

[0069] FIG. 14 shows images 22b and 22a of an enhancement layer signal 400 and a base layer signal 200, respectively, in a temporally registered manner. The portion to be currently coded / decoded is indicated at 28. According to the current embodiment, the base layer signal 200 is predictively coded by a scalable video encoder and predictively reconstructed by a scalable video decoder using base layer coding parameters that spatially vary the base layer signal. The spatial variation is illustrated in FIG. 14 using a shaded portion 132 surrounded by a non-shaded area. Within the shaded portion 132, the base layer coding parameters used to predictively code / reconstruct the base layer signal 200 remain constant. When moving from the shaded portion 132 to the non-shaded area, the base layer coding parameters change. According to the embodiment outlined above, the enhancement layer signal 400 is coded / reconstructed in units of blocks. The current portion 28 is one such block. According to the embodiment outlined above, the subblock subdivision for the current portion 28 is selected from a set of possible subblock subdivisions based on spatial variations of base layer coding parameters within the collocated portions 134 of the base layer signal 200, i.e., within the spatially collocated portions of the temporally corresponding images 22a of the base layer signal 200.

[0070] In particular, instead of signaling subdivision information in the enhancement layer substream 6b for the current portion 28, the above description suggests selecting a subblock subdivision from the set of possible subblock subdivisions of the current portion 28 such that the selected subblock subdivision is the coarsest among the set of possible subblock subdivisions. Thereby, subdividing the base layer signal 200 such that, when transferred onto the collocated portion 134 of the base layer signal, the base layer coding parameters are sufficiently similar to each other within each subblock of each subblock subdivision. For ease of understanding, refer to FIG. 15a, which shows the portion 28 illustrating, using diagonal lines, the spatial variation of the base layer coding parameters within the collocated portion 134. In particular, the portion 28 is shown three times with different subblock subdivisions applied to the block 28. In particular, a four-branch tree subdivision is exemplarily used in the case of FIG. 15a. That is, the set of possible subblock subdivisions is (or is defined by) a 4-branch tree subdivision. The three specific examples of subblock subdivisions of portion 28 depicted in FIG. 15a belong to different hierarchical levels of the 4-branch tree subdivision of block 28. From bottom to top, the level or coarseness of the subdivision of block 28 within subblocks increases. At the highest level, portion 28 is left as is. At the next lower level, block 28 is subdivided into four subblocks. At least one of the latter subblocks is then further subdivided into four subblocks at the next lower level, and so on. At each level in FIG. 15a, the 4-branch tree subdivision is selected to have the smallest number of subblocks, which is still not a subblock that overlaps with a base layer coding parameter change boundary. That is, in the case of FIG. 15a, the 4-branch tree subdivision of block 28 to be selected for subdividing block 28 is deemed to be the lowest one depicted in FIG. 15a. Here, the base layer coding parameters of the base layer are constant within each of the subblock-coordinated portions of the subblock subdivision.

[0071] Therefore, subdivision information for block 28 does not need to be signaled in enhancement layer substream 6b, resulting in increased coding efficiency. Moreover, as outlined, the method for deriving the subdivision is appropriate regardless of the position of current portion 28 with respect to any grid or any registration of the sample array of base layer signal 200. Also, in particular, subdivision derivation works in the case of fractional spatial resolution ratios between the base layer and enhancement layer.

[0072] Based on the subblock subdivision of portion 28 determined in this way, portion 28 is predictively reconstructed / encoded. It should be noted that, in relation to the above description, different possibilities exist for "measuring" the coarseness of the different available subblock subdivisions of current block 28. For example, the magnitude of the coarseness is determined based on the number of subblocks. The more subblocks each subblock subdivision has, the lower the level. This definition obviously does not apply in the case of FIG. 15a, where the "magnitude of the coarseness" is determined by a combination of the number of subblocks in each subblock subdivision and the smallest size of all subblocks in each subblock subdivision.

[0073] For completeness, Figure 15b shows, by way of example, the selection of one possible sub-block subdivision from a set of sub-block subdivisions available for the current block 28, when exemplarily using the subdivisions of Figure 35 as the available set. Different diagonal shading (and non-diagonal shading) indicates regions in the base layer signal where adjacent regions have the same base layer coding parameters associated with them.

[0074] As mentioned above, the outlined selection is performed by traversing the possible sub-block subdivisions according to some sequential order, such as an order of increasing or decreasing levels of coarseness, and selecting from the possible sub-block subdivisions a possible sub-block subdivision where the base layer coding parameters within each sub-block of each sub-block subdivision are sufficiently similar to one another that it no longer applies (when using traversals according to increasing levels of coarseness) or that it happens to apply first (when using traversals according to decreasing levels of coarseness). Alternatively, all possible subdivisions are tested.

[0075] In the description of Figures 14 and 15a-b, the broad term "base layer coding parameters" is used in the preferred embodiment, but these base layer coding parameters refer to base layer prediction parameters, i.e., parameters that relate to the formation of the prediction of the base layer signal but not to the construction of the prediction residual. Thus, for example, the base layer coding parameters include a prediction mode that distinguishes between spatial prediction (prediction parameters for the block / portion of the base layer signal assigned to spatial prediction, such as angular direction) and temporal prediction (prediction parameters for the block / portion of the base layer signal assigned to temporal prediction, such as motion parameters).

[0076] However, the definition of "sufficient" similarity of base layer coding parameters within a given sub-block only determines / defines a subset of the base layer coding parameters. For example, the similarity is determined based only on the prediction mode. Alternatively, prediction parameters that further regulate spatial and / or temporal prediction form the parameters on which the similarity of base layer coding parameters within a given sub-block depends.

[0077] Furthermore, as already outlined, in order to be sufficiently similar to one another, within a given subblock, the base layer coding parameters must be exactly equal to one another within each subblock, or alternatively, the measure of similarity used must be within a predetermined interval to satisfy the "similarity" criterion.

[0078] As outlined above, the selected subblock subdivision is not the only quantity predicted or transported from the base layer signal. Rather, the base layer coding parameters themselves are transported to the enhancement layer signal to obtain enhancement layer coding parameters for the subblocks of the subblock subdivision obtained by transporting the selected subblock subdivision from the base layer signal to the enhancement layer signal. As far as motion parameters are concerned, for example, scaling is used for transporting from the base layer to the enhancement layer. Preferably, only those parts or syntax elements of the base layer prediction parameters are used to set the subblocks of the subblock subdivision of the current portion obtained from the base layer that affect the degree of similarity. By this measure, the fact that these syntax elements of the prediction parameters in each subblock of the selected subblock subdivision are somehow similar to each other ensures that the syntax elements of the base layer prediction parameters used to predict the corresponding prediction parameters of the subblocks of the current portion 308 are similar or even equal to each other. As a result, for the first time, allowing some variation, some significant "meaning" of the syntax elements of the base layer prediction parameters corresponding to the portion of the base layer signal covered by each sub-block is used as a predictor for the corresponding sub-block. However, it may also be that only the portion of the syntax elements that contribute to the magnitude of similarity are used to predict the prediction parameters of the sub-blocks of the enhancement layer subdivision, adding only the transcription of the subdivision itself, so as to infer or pre-set the mode of the sub-blocks of the current portion 28, although the mode-specific base layer prediction parameters participate in the similarity magnitude determination.

[0079] One such possibility of not using only inter-layer prediction subdivision from base layer to enhancement layer is now described with respect to the next figure (FIG. 16). FIG. 16 shows picture 22b of the enhancement layer signal 400 and picture 22a of the base layer signal 200 in a registered manner along the presentation time axis 24.

[0080] According to the embodiment of FIG. 16, the base layer signal 200 is predictively reconstructed by a scalable video decoder and predictively coded using a scalable video encoder by subdividing the image 22a of the base layer signal 200 into intra blocks and intermediate blocks. According to the example of FIG. 16, the latter subdivision is performed in a two-stage manner. First, the frame 22a is regularly subdivided into the largest blocks or largest coding units, denoted by reference numerals 302 in FIG. 16, using double lines along their periphery. Then, each largest block 302 is subjected to a hierarchical quad-tree subdivision among the coding units forming the aforementioned intra blocks and intermediate blocks. Consequently, they are the leaves of the quad-tree subdivision of the largest block 302. In FIG. 16, reference numerals 304 are used to denote these leaf blocks or coding units. Typically, solid lines are used to denote the periphery of these coding units. Meanwhile, spatial intra prediction is used for the intra blocks, while temporal inter prediction is used for the intermediate blocks. Prediction parameters related to spatial intra prediction and temporal inter prediction are set within the units of smaller blocks, respectively. However, the intra and inter blocks or coding units 304 are subdivided. Such subdivision is exemplarily shown in FIG. 16 for one of the coding units 304, using reference sign 306 to indicate the smaller block. The smaller block 304 is outlined using a dotted line. That is, in the embodiment of FIG. 16, the spatial video encoder has the opportunity to choose between one spatial prediction and the other temporal prediction for each coding unit 304 of the base layer. However, as far as the enhancement layer signal is concerned, the degrees of freedom are increased. Here, in particular, frame 22b of the enhancement layer signal 400 is assigned to each of a set of prediction modes, including not only spatial intra prediction and temporal inter prediction, but also inter-layer prediction as outlined in detail below, within the coding units into which frame 22b of the enhancement layer signal 400 is subdivided.Subdivision into these coding units is done in the same manner as described for the base layer signal. First, frame 22b is regularly subdivided into rows and columns of the largest block outlined using double lines, which are then subdivided into coding units outlined using solid lines in a hierarchical quadtree subdivision process.

[0081] One coding unit 308 of the current image 22b of the enhancement layer signal 400 is illustratively inferred to be assigned to inter-layer prediction mode and is indicated using diagonal lines. In a manner similar to FIGS. 14, 15a, and 15b, FIG. 16 shows at 312 how a subdivision of the coding unit 308 is predictively derived by local translation from the base layer signal. In particular, a local region overlapped by the coding unit 308 is indicated at 312. Within this region, dotted lines indicate boundaries between adjacent blocks of the base layer signal, or more generally, boundaries through which base layer coding parameters of the base layer may change. Consequently, these boundaries are boundaries of the prediction blocks 306 of the base layer signal 200 and partially coincide with the boundaries between adjacent coding units 304 or the largest equally adjacent coding units 302 of the base layer signal 200, respectively. The dotted lines at 312 indicate the subdivision of the current coding unit 308 among the prediction blocks derived / selected by local translation from the base layer signal 200. Details regarding local translation are provided above.

[0082] 16, as already explained, only subdivision within the prediction block is not taken from the base layer. Rather, the prediction parameters of the base layer signal used in region 312 are used to derive prediction parameters to be used to perform prediction for the prediction block of coding unit 308 of enhancement layer signal 400.

[0083] In particular, according to the embodiment of FIG. 16, not only are subdivisions within the prediction block obtained from the base layer signal, but prediction modes are also used in the base layer signal 200 to encode / reconstruct each area locally covered by each subblock of the obtained subdivision. One example is as follows: To obtain the subdivision of the coding unit 308 as described above, the prediction modes are used in conjunction with the associated base layer signal 200. Mode-specific prediction parameters are used to determine the "similarity" discussed above. Accordingly, the different diagonal lines shown in FIG. 16 correspond to different prediction blocks 306 of the base layer. Each of the different prediction blocks 306 has an intra- or inter-prediction mode (i.e., a spatial or temporal prediction mode associated with it). As explained above, to be "sufficiently similar," the prediction modes used in the collocated areas of each subblock of the subdivision of the coding unit 308 and the specific prediction parameters for each prediction mode within the subarea must be exactly equal to each other. Alternatively, some variation may be tolerated.

[0084] In particular, according to the embodiment of Figure 16, all blocks indicated by diagonal lines extending from the upper left to the lower right are set to be intra prediction blocks of the coding unit 308 because their locally corresponding portions of the base layer signal are covered by prediction blocks 306 having the spatial intra prediction mode associated with them, while other blocks (i.e., blocks indicated by diagonal lines extending from the lower left to the upper right) are set to inter prediction blocks because their locally corresponding portions of the base layer signal are covered by prediction blocks 306 having the temporal inter prediction mode associated with them.

[0085] On the other hand, for the alternative embodiment, the derivation of the prediction stops here in the details for performing the prediction within the coding unit 308, i.e., the derivation of the subdivision of the coding unit 308 among the prediction blocks and the allocation of these prediction blocks among the prediction blocks coded using non-temporal or spatial prediction and among the prediction blocks coded using temporal prediction are limited, which does not follow the embodiment of Figure 16.

[0086] According to the latter embodiment, all prediction blocks of the coding unit 308 having a non-temporal prediction mode assigned thereto undergo non-temporal prediction, such as spatial intra prediction, while using, as the enhancement layer prediction parameters for these non-temporal mode blocks, prediction parameters derived from the prediction parameters of locally matching intra blocks of the base layer signal 200. Consequently, such derivation relates to the spatial prediction parameters of locally co-located intra blocks of the base layer signal 200. For example, such spatial prediction parameters may indicate the angular direction along which the spatial prediction is performed. As outlined above, a definition of similarity is required: either the spatial base layer prediction parameters overlap for each non-temporal prediction block of the coding unit 308 by being the same as each other, or, for each non-temporal prediction block of the coding unit 308, an average of the spatial base layer prediction parameters is used to derive the prediction parameters for each non-temporal prediction block.

[0087] Alternatively, all prediction blocks of the coding unit 308 that have an assigned non-temporal prediction mode are inter-layer predicted in the following manner: first, the base layer signal undergoes decomposition or quality refinement to obtain inter-layer prediction signals at least in those regions spatially collocated with the non-temporal prediction mode prediction blocks of the coding unit 308, and then these prediction blocks of the coding unit 308 are predicted using the inter-layer prediction signals.

[0088] Scalable video decoders and encoders, by default, subject all of the coding units 308 to spatial prediction or inter-layer prediction. Alternatively, the scalable video encoder / decoder supports both alternatives and signals them in the coded video data stream signal, whichever version is used as far as non-temporal prediction mode prediction blocks of the coding units 308 are concerned. In particular, the decision between both alternatives is signaled in the data stream, e.g., individually for any size of the coding units 308.

[0089] As far as other prediction blocks of the coding unit 308 are concerned, the coding unit 308 undergoes temporal inter-prediction using prediction parameters derived from the prediction parameters of the locally matching inter-block, just as it would be for a non-temporal prediction mode prediction block, so that the derivation, in turn, relates to the motion vector assigned to the corresponding part of the base layer signal.

[0090] For all other coding units that have both a spatial intra prediction mode and a temporal inter prediction mode assigned to them, the other coding units undergo spatial or temporal prediction in the following manner. In particular, the other coding units are further subdivided into prediction blocks that have a prediction mode assigned to them. The prediction mode is common for all of the prediction blocks within a coding unit, and in particular, the same prediction mode assigned to each coding unit. That is, coding units that are different from coding units such as coding unit 308 and have a spatial intra prediction mode associated with them or a temporal inter prediction mode associated with them are subdivided into prediction blocks of the same prediction mode. That is, the prediction modes are simply inherited from the respective coding units from which they are derived by the subdivision of the respective coding units.

[0091] All coding unit subdivisions, including 308, are subdivisions of the quadtree within the prediction block.

[0092] A further difference between coding units of inter-layer prediction modes, such as coding unit 308, and coding units of spatial intra prediction mode or temporal inter prediction mode, is that the prediction blocks of the spatial intra prediction mode coding unit or temporal inter prediction mode coding unit undergo spatial prediction and temporal prediction, respectively. Prediction parameters are set independently of the base layer signal 200, for example, by signaling in the enhancement layer substream 6b. Even subdivisions of coding units other than coding unit 308 that have an associated inter-layer prediction mode are signaled in the enhancement layer signal 6b. That is, inter-layer prediction mode coding units, such as 308, benefit from lower bit-rate signaling requirements. According to an embodiment, the mode indicator of coding unit 308 itself does not need to be signaled in the enhancement layer substream. Optionally, additional parameters, such as prediction parameter residuals, are transmitted to coding unit 308 for individual prediction blocks. Additionally or alternatively, the prediction residual for the encoding unit 308 is transmitted / signaled in the enhancement layer substream 6b, while the scalable video decoder retrieves this information from the enhancement layer substream, and the scalable video encoder according to the current embodiment determines these parameters and inserts them into the enhancement layer substream 6b.

[0093] In other words, the base layer signal 200 is predicted using base layer coding parameters in such a way that the base layer coding parameters spatially vary the base layer signal 200 within the unit of the base layer block 304. Prediction modes available for the base layer include, for example, spatial and temporal prediction. The base layer coding parameters further include prediction mode-specific prediction parameters such as the angular direction (as far as the spatially predicted block 304 is concerned) and the motion vector (as far as the temporally predicted block 304 is concerned). The latter prediction mode-specific prediction parameters vary the base layer signal within units smaller than the base layer block 304, i.e., within the aforementioned prediction block 306. To satisfy the previously outlined requirement of sufficient similarity, it is necessary that the prediction modes of all base layer blocks 304 overlapping the area of ​​each possible subblock subdivision are equal to each other. Then, only each subblock subdivision is shortlisted to obtain the selected subblock subdivision. However, the requirement is more stringent. That is, the prediction mode individual prediction parameters of the prediction blocks overlapping the common area of ​​each subblock subdivision must also be equal to each other. For each subblock of this respective subblock subdivision and the corresponding area in the base layer signal, only the subblock subdivisions that satisfy this requirement are put into a shortlist to obtain the final selected subblock subdivision.

[0094] In particular, as briefly outlined above, there are different possibilities for how to select among the set of possible subblock subdivisions. To outline this in more detail, refer to Figures 15c and 15d. Imagine that a set 352 encompasses all possible subblock subdivisions 354 of the current block 28. Of course, Figure 15c merely illustrates an example. The set 352 of possible or available subblock subdivisions of the current block 28 is known to the scalable video decoder and scalable video encoder by default, or is signaled in an encoded data stream, such as a sequence of images or the like. Following the example of Figure 15c, each member of the set 352, i.e., each available subblock subdivision 354, is subjected to a check 356 to check whether the area within the collocated portion 108 of the base layer signal that is to be subdivided by transferring each subblock subdivision 354 from the enhancement layer to the base layer is simply overlapped by the prediction block 306 and the encoding unit 304. Then, it is checked whether the base layer coding parameters meet the sufficient similarity requirement. See, for example, the exemplary subdivision marked with reference number 354. According to this exemplary subdivision of available subblocks, the current block 28 is subdivided into four quadrants / subblocks 358, and the top left subblock corresponds to region 362 in the base layer. Obviously, this region 362 overlaps four blocks of the base layer (i.e., two prediction blocks 306 and two coding units 304, which are not separate subdivisions within the prediction blocks, but thus represent the prediction blocks themselves). Therefore, if all base layer coding parameters of these predictive blocks overlapping region 362 satisfy the similarity criterion, and this is also the case for all subblocks / quadrant of a possible subblock subdivision 354 and their corresponding region overlapping base layer coding parameters, then this possible subblock subdivision 354 satisfies the sufficient requirement for all regions covered by the subblocks of the respective subblock subdivision and belongs to the subblock subdivision set 364.Within this set 364 , the coarsest subdivision is then selected as indicated by arrow 366 , resulting in a selected subblock subdivision 368 from set 352 .

[0095] Clearly, it is preferable to try to avoid performing check 356 for all members of set 352. Therefore, as shown in FIG. 15d and as previously described, possible subdivisions 354 are traversed to increase or decrease in size. Traversal is indicated using double-headed arrows 372. FIG. 15d shows that for at least some of the available subblock subdivisions, the size levels or magnitudes are equal to each other. In other words, the ordering according to increasing or decreasing size levels is ambiguous. However, since only one such equally large possible subblock subdivision belongs to set 364, this does not prevent a search for the "largest subblock subdivision" belonging to set 364. Thus, when traversing in the direction of increasing size levels, with the second-to-last traversed possible subblock subdivision being the subblock subdivision 354 to be selected, the largest possible subblock subdivision 368 is found as soon as the result of criteria check 356 changes from filled to unfilled. Alternatively, when traversing in the direction of a level of decreasing magnitude, where the subblock subdivision 368 is the most recently traversed subblock subdivision, the result of the criteria check 356 switches from not filled to filled.

[0096] For the following figures, a scalable video encoder or decoder such as those described above for Figures 1-4 may be implemented to form embodiments of the present application according to another aspect of the application. Possible implementations of the examples described below are presented with reference to Examples K, A, and M.

[0097] To explain the example, reference is made to Fig. 17, which shows possibilities for temporal prediction 32 of the current portion 28. Consequently, the following description of Fig. 17 may be combined with the descriptions relating to Figs. 6 to 10 insofar as it relates to combination with inter-layer prediction signals, or with the descriptions relating to Figs. 11 to 13 insofar as it relates to combination with temporal inter-layer prediction modes.

[0098] The situation shown in FIG. 17 corresponds to the situation shown in FIG. 6 . That is, the base layer signal 200 and the enhancement layer signal 400 are shown, with the already coded / decoded portions indicated using diagonal lines. The portion of the enhancement layer signal 400 currently to be coded / decoded includes adjacent blocks 92 and 94, illustratively depicted here as blocks 92 above and 94 to the left of the current portion 28. Both blocks 92 and 94 illustratively have the same size as the current block 28. However, size matching is not mandatory. Rather, the portions of the blocks in the image 22b of the enhancement layer signal 400 being subdivided have different sizes. They are not even limited to rectangular shapes. They may be rectangular or have other shapes. The current block 28 with its adjacent blocks is not explicitly depicted in FIG. 17 . However, the adjacent blocks have not yet been coded / decoded; that is, they follow in coding / decoding order and are therefore unavailable for prediction. In addition to the blocks 92 and 94 that have already been coded / decoded according to the coding / decoding order, there is another block, illustratively a block 96 adjacent to the current block 28 and located diagonally above and to the left of the current block 28. However, in the example considered here, blocks 92 and 94 are predetermined neighboring blocks that serve to predict inter-prediction parameters for the current block 28 that undergoes inter-prediction 30. The number of such predetermined neighboring blocks is not limited to two; it may be greater than one or simply one. A discussion of possible embodiments is presented with reference to Figures 36 to 38.

[0099] The scalable video encoder and scalable video decoder determine the set of predetermined neighboring blocks (here, blocks 92 and 94) from the set of already-encoded neighboring blocks (here, blocks 92-96) within the current portion 28, such as its top-left sample, depending on, for example, a predetermined sample position 98. For example, only those already-encoded neighboring blocks of the current portion 28 form a set of "predetermined neighboring blocks" that includes sample positions immediately adjacent to the predetermined sample position 98. Another possibility is described with respect to Figures 36-38.

[0100] In any case, a portion 502 of a previously coded / decoded image 22b of the enhancement layer signal 400, which is displaced from its collocated position with the current block 28 according to the decoding / coding order by a motion vector 504, comprises sample values ​​reconstructed on the basis of sample values ​​of the portion 28 that are predicted by simple copying, interpolation, etc. For this purpose, the motion vector 504 is signaled in the enhancement layer substream 6b. For example, the temporal prediction parameters for the current block 28 indicate a displacement vector 506 indicating the displacement of the portion 502 from its collocated position with the portion 28 in the reference image 22b, optionally by interpolation, to be copied onto the samples of the portion 28.

[0101] In any case, when temporally predicting the current block 28, the scalable video decoder / encoder has already reconstructed (and, in the case of an encoder, has already encoded) the base layer 200 using the base layer substream 6a, and, at least insofar as the spatially corresponding regions of the temporally corresponding images 22a concerned are so related, uses block-wise prediction, as described above, and, for example, uses block-wise selection between spatial and temporal prediction modes.

[0102] 17, the time-coordinated image 22a of the base layer signal 200 is subdivided into several blocks 104. The blocks 104 are located around an area that locally corresponds to the illustratively represented current portion 28, just as with spatially predicted blocks in the enhancement layer signal 400. Spatial prediction parameters are included or signaled in the base layer substream 6a for those blocks 104 in the base layer signal 200. The selection of the spatial prediction mode is signaled to the base layer signal 200.

[0103] Here, illustratively, for blocks 28 for which temporal intra-layer prediction 32 is selected, intermediate prediction parameters, such as motion parameters, are determined using any of the following methods to allow reconstruction of the enhancement layer signal from the coded data stream:

[0104] The first possibility is illustrated with respect to FIG. 18. In particular, first, a set 512 of motion parameter candidates 514 is collected or generated from adjacent, already reconstructed blocks of a frame, such as predetermined blocks 92 and 94. The motion parameters are motion vectors. The motion vectors of blocks 92 and 94 are represented using arrows 516 and 518, with the numbers 1 and 2 respectively. As shown, these motion parameters 516 and 518 directly form candidate 514. Some candidates are formed by combining motion vectors, such as 518 and 516, as shown in FIG. 18.

[0105] Furthermore, a set 522 of base layer motion parameters 524 for one or more blocks 108 of the base layer signal 200 collocated in the portion 28 is compiled or generated from the base layer motion parameters. In other words, the motion parameters associated with the collocated blocks 108 in the base layer are used to obtain one or more base layer motion parameters 524.

[0106] One or more base layer motion parameters 524, or scaled versions thereof, are then added 526 to the set 512 of motion parameter candidates 514 to obtain an extended set 528 of motion parameter candidates. This can be done in a variety of ways, such as simply adding the base layer motion parameters 524 at the end of the list of candidates 514, or in a different way, an example of which is outlined for Figure 19a.

[0107] At least one of the motion parameter candidates 532 of the set of extended motion parameter candidates 528 is then selected. By motion compensated prediction of portion 28, temporal prediction 32 is performed using the selected one of the motion parameter candidates of the set of extended motion parameter candidates. Selection 534 may be signaled in a data stream such as substream 6b for portion 28 by way of an index 536 into list / set 528, or may be performed in another manner as described with respect to FIG. 19a.

[0108] As previously mentioned, it is checked whether the base layer motion parameters 523 are coded in an encoded data stream, such as the base layer substream 6a, using merging, and if the base layer motion parameters 523 are coded in an encoded data stream using merging, then addition 526 is suppressed.

[0109] The motion parameters described according to Figure 18 relate to only motion vectors (motion vector prediction) or to the complete set of motion parameters including the number of motion hypotheses per block, reference index list, and segmentation information (merging). Thus, in the case of spatial scalability, the "scaled version" comes from scaling the motion parameters used in the base layer signal according to the spatial resolution ratio between the base layer signal and the enhancement layer signal. Depending on the method of the coded data stream, the coding / decoding of the base layer motion parameters of the base layer signal involves motion vector prediction, such as spatial or temporal, or merging.

[0110] The incorporation 526 of the motion parameters 523 used in the collocated portion 108 of the base layer signal into a set 528 of merged / motion vector candidates 532 allows for very efficient indexing among the intra-layer candidates 514 and one or more inter-layer candidates 524. The selection 534 may involve explicit signaling, per prediction block, per coding unit, etc., of an index into an expanded set / list of motion parameter candidates in the enhancement layer signal 6b. Alternatively, the selection index 536 may be inferred from other information in the enhancement layer signal 6b or from inter-layer information.

[0111] According to the possibility of FIG. 19a, the formation 542 of the final motion parameter candidate list for the enhancement layer signal for portion 28 is only optionally performed as outlined with respect to FIG. 18, i.e., it could be 528 or 512. However, list 528 / 512 is ordered 544 depending on the base layer motion parameters, such as the motion parameters of the co-located base layer block 108 represented by motion vector 523. For example, the rank of the members of list 528 / 512, i.e., motion parameter candidates 532 or 514, is determined based on their respective deviations from a potentially scaled version of motion parameter 523. The greater the deviation, the lower the rank of each member 532 / 512 in ordered list 528 / 512'. Thus, ordering 544 may involve determining the magnitude of deviation for each member 532 / 514 of list 528 / 512. Then, a selection 534 of one candidate 532 / 512 in the ordered list 528 / 512′ is performed, controlled by an explicitly signaled index syntax element 536 in the coded data stream, to obtain enhancement layer motion parameters from the ordered motion parameter candidate list 528 / 512′ for the portion 28 of the enhancement layer signal, and then a temporal prediction 32 of the portion 28 of the enhancement layer signal by motion compensated prediction is performed using the selected motion parameters pointed to 534 by the index 536.

[0112] The motion parameters mentioned above with reference to FIG. 18 are applied to the motion parameters mentioned in FIG. 19a. The decoding of the base layer motion parameters 520 from the coded data stream involves (optionally) spatial or temporal motion vector prediction or merging. The ordering is done according to the magnitude of the difference between each enhancement layer motion parameter candidate and the base layer motion parameters of the base layer signal, just relative to the block of the base layer signal collocated with the current block of the enhancement layer signal. That is, for the current block of the enhancement layer signal, a list of enhancement layer motion parameter candidates is first determined. Next, the ordering is performed, as will be explained below. The selection is performed with explicit signaling.

[0113] Alternatively, the ordering 544 may be performed according to a measure of the difference between the base layer motion parameters 523 of the base layer signal relating to the block 108 of the base layer signal collocated with the current block 28 of the enhancement layer signal and the base layer motion parameters 546 of spatially and / or temporally neighboring blocks 548 within the base layer. The determined ordering within the base layer is then transferred to the enhancement layer, with the enhancement layer motion parameter candidates being ordered in the same way as the ordering determined for the corresponding base layer candidates. In this regard, the base layer motion parameters 546 may be said to correspond to the enhancement layer motion parameters of the neighboring enhancement layer blocks 92, 94 if the associated base layer block 548 is spatially / temporally collocated with the neighboring enhancement layer blocks 92, 94 associated with the considered enhancement layer motion parameters. Alternatively, if the neighbor relationship (left neighbor, above neighbor, A1, A2, B1, B2, B0, or see Figures 36-38 for other examples) between the associated base layer block 548 and the block 108 co-located with the current enhancement layer block 28 is the same as the neighbor relationship between the current enhancement layer block 28 and the respective enhancement layer neighboring blocks 92, 94, then the base layer motion parameters 546 can be said to correspond to the enhancement layer motion parameters of the neighboring enhancement layer blocks 92, 94. Thereafter, by explicit signaling, the selection 534 is performed based on the base layer ordering.

[0114] To explain this in more detail, refer to FIG. 19b. FIG. 19b shows the first of the outlined alternatives for obtaining an enhancement layer ordered list of motion parameter candidates using base layer hints. FIG. 19b shows an alternative current block 28 and the positions of three different predetermined samples (i.e., illustratively, upper-left sample 581, lower-left sample 583, and upper-right sample 585). The example is inserted for illustrative purposes only. The set of predetermined neighboring blocks illustratively includes four types of neighbors. Neighboring block 94a covers sample position 587, which is immediately above and adjacent to sample position 581. Neighboring block 94b includes or covers sample position 589, which is immediately above and adjacent to sample position 585. Similarly, neighboring blocks 92a and 92b include sample positions 591 and 593, which are immediately to the left of sample positions 581 and 583. 36-38, the number of predetermined adjacent blocks may vary despite the predetermined number of decision rules. Nevertheless, the predetermined adjacent blocks 92a, 92b, 94a, and 94b are distinguishable by their decision rules.

[0115] According to the alternative of FIG. 19b, a collocated block in the base layer is determined for each predetermined neighboring block 92a, 92b, 94a, 94b. For example, the upper-left sample 595 of each neighboring block is used for this purpose. This is exactly the case with the current block 28 with respect to the upper-left sample 581 formally referred to in FIG. 19a. This is illustrated using dotted arrows in FIG. 19b. By this means, for each predetermined neighboring block, a corresponding block 597 is found in addition to the collocated block 108 and the collocated current block 28. Using the motion parameters m1, m2, m3, m4 of the collocated base layer block 597 and their respective differences with respect to the base layer motion parameter m of the collocated base layer block 108, the enhancement layer blocks M1, M2, M3, M4 of the predetermined neighboring blocks 92a, 92b, 94a, 94b are ordered in the list 528 or 512. For example, the greater the distance between any of m1-m4, the higher the corresponding enhancement layer motion parameter M1-M4. That is, a higher index list is required to index the same state from list 528 / 512'. Absolute differences are used for distance measurements. Similarly, motion parameter candidates 532 or 514 are reordered in the list with their rank being the combination of enhancement layer motion parameters M1-M4.

[0116] Figure 19c illustrates an alternative in which corresponding blocks in the base layer are determined in another manner. In particular, Figure 19c illustrates predetermined neighboring blocks 92a, 92b, 94a, and 94b of current block 28 and co-located block 108 of current block 28. In accordance with the embodiment of Figure 19c, the corresponding base layer blocks of current block 28 (i.e., 92a, 92b, 94a, and 94b) are determined in such a way that these base layer blocks are related to enhancement layer neighboring blocks 92a, 92b, 94a, and 94b using the same neighbor determination rules for determining these base layer neighboring blocks. In particular, Figure 19c illustrates predetermined sample positions 601 of co-located block 108, i.e., the top-left, bottom-left, and top-right sample positions. Based on these sample positions, the four neighboring blocks of block 108 are determined in the same manner as described for enhancement layer neighboring blocks 92a, 92b, 94a, and 94b relative to predetermined sample positions 581, 583, and 585 of current block 28. Four base layer neighboring blocks 603a, 603b, 605a, and 605b are found in this manner. 603a clearly corresponds to enhancement layer neighboring block 92a. Base layer block 603b corresponds to enhancement layer neighboring block 92b. Base layer block 605a corresponds to enhancement layer neighboring block 94a. Base layer block 605b corresponds to enhancement layer neighboring block 94b. In the same manner as previously described, the base layer motion parameters M1 to M4 of the base layer blocks 903a, 903b, 905a, 905b and their distance to the base layer motion parameter m of the co-located base layer block 108 are used to order the motion parameter candidates in the list 528 / 512 formed from the motion parameters M1 to M4 of the enhancement layer blocks 92a, 92b, 94a, 94b.

[0117] According to the possibility of Fig. 20, the formation 562 of the final motion parameter candidate list for the enhancement layer signal for portion 28 is simply optionally performed as outlined for Fig. 18 and / or Fig. 19, i.e., formation 562 is 528 or 512 or 528 / 512'. Reference sign 564 is used in Fig. 20. According to Fig. 20, an index 566 to point into motion parameter candidate list 564 is determined depending on an index 567 into motion parameter candidate list 568 that was used to encode / decode the base layer signal, for example for collocated block 108. For example, when reconstructing a base layer signal at block 108, a list 568 of motion parameter candidates is determined based on motion parameters 548 of neighboring blocks 548 of block 108 that have the same neighboring relationship (left neighbor, above neighbor, A1, A2, B1, B2, B0, or see Figures 36-38 for another example) with respect to block 108 as the neighboring relationship between predetermined neighboring enhancement layer blocks 92, 94 and current block 28. Here, determination 572 of list 567 potentially uses the same construction rules used in formation 562, such as the ordering among the list members of lists 568 and 564. More generally, index 566 for the enhancement layer candidate is determined in such a way that its neighboring enhancement layer block 92, 94 is pointed to by index 566 collocated with the base layer block 548 related to the indexed base layer candidate (i.e., the one pointed to by index 567). As a result, index 567 serves as a significant predictor of index 566. The enhancement layer motion parameters are then determined using the index 566 into the motion parameter candidate list 564, and the motion compensated prediction of block 28 is performed using the determined motion parameters.

[0118] For the motion parameters mentioned in FIG. 20, the same applies as described above with respect to FIGS.

[0119] With respect to the following figures, it will be described how a scalable video encoder or decoder, such as those described above with respect to Figures 1 to 4, can be implemented to form an embodiment of the present application according to another example of the application. A detailed implementation of the example described below is explained below with reference to Example V.

[0120] This embodiment relates to residual coding in an enhancement layer. In particular, FIG. 21 exemplarily illustrates an image 22b of an enhancement layer signal 400 and an image 22a of a base layer signal 200 in a temporally registered manner. FIG. 21 illustrates a method for reconstructing in a scalable video decoder or encoding in a scalable video encoder an enhancement layer signal, focusing on a predetermined transform coefficient block and a predetermined portion 404 of transform coefficients 402 representing the enhancement layer signal 400. In other words, the transform coefficient block 402 represents a spatial decomposition of the portion 404 of the enhancement layer signal 400. As already explained above according to the encoding / decoding ordering, the corresponding portion 406 of the base layer signal 200 has already been decoded / encoded when decoding / encoding the transform coefficient block 402. As far as the base layer signal 200 is concerned, predictive encoding / decoding is used, including signaling the base layer residual signal in an encoded data stream such as base layer substream 6a.

[0121] 21 , the scalable video decoder / encoder exploits the fact that an estimate 408 of the base layer signal or the base layer residual signal leads to an advantageous selection of a subdivision of the transform coefficient block 402 into subblocks 412 in portion 406 collocated with portion 404. In particular, several possible subblock subdivisions for subdividing the transform coefficient block 402 into subblocks are supported by the scalable video decoder / encoder. These possible subblock subdivisions subdivide the transform coefficient block 402 into regularly rectangular subblocks 412. That is, the transform coefficients 414 of the transform coefficient block 402 are arranged in columns and columns, and, according to the possible subblock subdivisions, these transform coefficients 414 are regularly packed into the subblocks 412, so that the subblocks 412 themselves are arranged in rows and rows. Using the subblock subdivision thus selected, the evaluation 408 allows the ratio between the number of rows and the number of columns of the subblocks 412 (i.e., the ratio between their width and height) to be set in such a way that the encoding of the transform coefficient block 402 is most efficient. If, for example, the evaluation 408 determines that the reconstructed base layer signal 200 in the collocated portion 406, or at least the base layer residual signal in the corresponding portion 406, is primarily composed of horizontal edges in the spatial domain, then the transform coefficient block 402 is likely to be present with importance (i.e., non-zero transform coefficient levels, i.e., quantized transform coefficients near the zero horizontal frequency side of the transform coefficient block 402). In the case of vertical edges, the transform coefficient block 402 is likely to be present with non-zero transform coefficient levels at a position near the zero vertical frequency side of the transform coefficient block 402. Therefore, the subblocks 412 are first selected to be longer along the vertical direction and smaller along the horizontal direction. And secondly, the sub-blocks are made longer horizontally and smaller vertically, the latter case being shown diagrammatically in FIG.

[0122] That is, the scalable video decoder / encoder selects one subblock subdivision from a set of possible subblock subdivisions based on the base layer residual signal or base layer signal. Then, encoding 414 or decoding of the transform coefficient block 402 is performed applying the selected subblock subdivision. In particular, since the positions of the transform coefficients 414 are traversed in units of subblocks 412, all positions within one subblock are traversed in a manner that immediately follows the next subblock in the subblock ordering defined within the subblock. A currently visited subblock, such as subblock 412, is illustratively shown in Figure 40. A syntax element is signaled in a data stream, such as enhancement layer substream 6b, indicating whether the currently visited subblock has significant transform coefficients. In Figure 21, syntax element 416 is illustrated for two exemplary subblocks. If each syntax element of each subblock indicates non-significant transform coefficients, nothing else needs to be transmitted in the data stream or enhancement layer substream 6b. Rather, the scalable video decoder sets the transform coefficients in that subblock to zero. However, if syntax element 416 of a respective subblock indicates that this subblock has significant transform coefficients, then additional information related to the transform coefficients in that subblock is signaled in the data stream or substream 6b. On the decoding side, the scalable video decoder decodes from the data stream or substream 6b a syntax element 418 that indicates the level of the transform coefficients in each subblock. Syntax element 418 indicates the position of the significant transform coefficients in that subblock according to the scanning order among these transform coefficients in each subblock and, optionally, according to the scanning order among the transform coefficients in each subblock.

[0123] FIG. 22 illustrates different possibilities for selecting among possible subblock subdivisions in estimate 408. FIG. 22 again illustrates portion 404 of an enhancement layer signal, with transform coefficient block 402 representing a spectral decomposition of portion 404. For example, transform coefficient block 402 represents a spectral decomposition of an enhancement layer residual signal, with a scalable video decoder / encoder predictively encoding / decoding the enhancement layer signal. In particular, transform encoding / decoding is used by the scalable video decoder / encoder to encode the enhancement layer residual signal. The transform encoding / decoding is performed in a block-wise manner (i.e., of the blocks into which image 22b of the enhancement layer signal is subdivided). FIG. 22 illustrates a corresponding or collocated portion 406 of a base layer signal. Here, the scalable video decoder / encoder applies predictive encoding / decoding to the base layer signal while using transform encoding / decoding on the predictive residual of the base layer signal (i.e., on the base layer residual signal). In particular, a block-wise transform is used for the base layer residual signal, i.e., the base layer residual signal is block-wise transformed with individually transformed blocks depicted by dotted lines in Figure 22. As depicted in Figure 22, the block boundaries of the base layer transform blocks need not coincide with the outline of the collocated portion 406.

[0124] Nevertheless, to perform the evaluation 408, one or a combination of the following options A to C may be used.

[0125] In particular, the scalable video decoder / encoder performs a transform 422 on the base layer residual signal or the reconstructed base layer signal in portion 406 to obtain a transform coefficient block 424 of transform coefficients that matches in size the transform coefficient block 402 to be coded / decoded. Examination of the distribution of values ​​of the transform coefficients in the transform coefficient blocks 424, 426 is used to appropriately set the dimensions of the sub-blocks 412 along the horizontal frequency direction 428 and along the vertical frequency direction 432.

[0126] Additionally or alternatively, the scalable video decoder / encoder examines all transform coefficient blocks of the base layer transform block 434, depicted by different diagonal lines in FIG. 22, that at least partially overlap the collocated portion 406. In the exemplary case of FIG. 22, there are four base layer transform blocks, and their transform coefficient blocks are then examined. In particular, all of these base layer transform blocks are different sizes from each other and further different in size from the transform coefficient block 412. Scaling 436 is performed on these transform coefficient blocks that overlap the base layer transform block 434 to produce a transform coefficient block 438 approximation of the spectral decomposition of the base layer residual signal in the portion 406. The distribution of transform coefficient values ​​within that transform coefficient block 438 (i.e., 442) is used in evaluation 408 to appropriately set the subblock dimensions 428 and 432. As a result, a subblock subdivision of the transform coefficient block 402 is selected.

[0127] Additionally or alternatively, evaluation 408 is performed by inspecting the base layer residual signal or the reconstructed base layer signal using edge detection 444 or determining the dominant gradient direction in the spatial domain (e.g., based on the extension direction of the detected edge or the gradient determined in the collocated portion 406 to appropriately set the subblock dimensions 428 and 432).

[0128] Although not explicitly explained above, in traversing the locations of the transform coefficients and units of the subblocks 412, it is preferable to traverse the subblocks 412 in an order starting from the zero-frequency corner of the transform coefficient block (the upper left corner in FIG. 21) to the highest-frequency corner of the block 402 (the lower right corner in FIG. 21). Furthermore, entropy coding is used to signal the syntax elements in data stream 6b. That is, syntax elements 416 and 418 are conveniently entropy coded, such as arithmetic, variable-length coding, or another form of entropy coding. The ordering of traversing the subblocks 412 also depends on the subblock shape selected according to 408. For subblocks selected to be wider than their height, the order of traversal traverses the subblocks column-wise first, then proceeds to the next column, and so on. Beyond this, note again that the base layer information used to select the dimensions of the subblocks is the reconstructed base layer residual signal or the base layer signal itself.

[0129] Below, different embodiments are described that can be combined with the above-described embodiments. The embodiments described below relate to many different implementations or measures for making scalable video coding more efficient. In part, the above embodiments are described in detail below while retaining the general concept in order to present alternative embodiments derived therefrom. These descriptions presented below can be used to obtain alternatives or extensions of the above embodiments / implementations. However, most of the embodiments described below can be optionally combined with the embodiments already described above. That is, they can be simultaneously implemented together with the above embodiments in one scalable video decoder / encoder. However, this is not required.

[0130] To make the previous description easier to understand, more detailed embodiments for implementing a suitable scalable video encoder / decoder incorporating an embodiment or combination of embodiments are presented next. The different embodiments described below are enumerated by using alphanumeric symbols. These embodiments generally implement, according to one embodiment, the description of some of the elements of these embodiments in the drawings just described. However, as far as individual embodiments are concerned, the presence of elements in the implementation of a scalable video decoder / encoder is not necessary as far as every embodiment is concerned. Depending on the embodiment in question, some elements and some interconnections may be omitted in the drawings described below. Only the elements cited for each embodiment are provided to perform the tasks or functions mentioned in the description of each embodiment. However, here, when several elements are cited for one function, sometimes substitutions specifically exist.

[0131] However, to provide an overview of the functionality of a scalable video decoder / encoder, the following described example is implemented. The elements shown in the following figures will now be briefly described.

[0132] 23 illustrates a scalable video decoder for decoding an encoded data stream 6 in which video is encoded in such a way that a main sub-portion (i.e., 6a) of the encoded data stream 6 represents the video at a first resolution or quality level. Additional portions 6b of the encoded data stream correspond to representations of the video at increasing resolution or quality levels. To keep the amount of data in the encoded data stream 6 low, inter-layer redundancy between sub-streams 6a and 6b is exploited in forming sub-stream 6b. Some of the embodiments described below are directed to inter-layer prediction from the base layer to which sub-stream 6a pertains, and the enhancement layer to which sub-stream 6b pertains.

[0133] The scalable video decoder includes two block-based predictive decoders 80, 60 operating in parallel, receiving sub-streams 6a and 6b, respectively. As shown in the figure, a demultiplexer 40 provides separate decoding stages 80 and 60 with the corresponding sub-streams 6a and 6b.

[0134] The internal structure of the block-based predictive coding stages 80 and 60 is the same, as shown in the figure. From the input of each of the entropy decoding modules 100, 60, the entropy coding module 100; 320, the inverse transformer 560; 580, the adder 180; 340, and the optional filters 120; 300, 140; 280 are serially connected in the order shown. At the end of this serial connection, a reconstructed base layer signal 600 and a reconstructed enhancement layer signal 360 are obtained, respectively. Meanwhile, the outputs of the adder 180; 340 and the filters 120; 140; 300; 280 provide different reconstruction versions of the base layer signal and the enhancement layer signal, respectively. Each prediction provider 160; 260 then receives a subset or all of these versions and, based on them, provides a prediction signal for providing to the residual input of the adder 180; 340. The entropy decoding stage 100;320 decodes from the respective input signals 6a and 6b, transform coefficient blocks which go to the inverse transformer 560;580 to encode parameters, including prediction parameters for the prediction provider 160;260.

[0135] Thus, prediction providers 160 and 260 predict blocks of video frames at their respective resolution / quality levels, and for this purpose, prediction providers 160 and 260 select among predefined prediction modes, such as spatial intra prediction mode and temporal inter prediction mode, both of which are intra-layer prediction modes (i.e., prediction modes that depend solely on data in the substream containing the respective level).

[0136] However, to take advantage of the aforementioned inter-layer redundancy, the enhancement layer decoding stage 60 additionally includes a coding parameter inter-layer predictor 240, a resolution / quality improver 220, and / or a prediction provider 260, which is compared to the prediction provider 160. Additionally / alternatively, the enhancement layer decoding stage 60 supports an inter-layer prediction mode that can provide an enhancement layer prediction signal 420 based on data obtained from an internal stage of the base layer decoding stage 80. The resolution / quality improver 220 subjects either the reconstructed base layer signal 200a, 200b, 200c or the base layer residual signal 480 to resolution or quality improvement to obtain an inter-layer prediction signal 380. The coding parameter inter-layer predictor 240 somehow predicts coding parameters, such as prediction parameters and motion parameters, respectively. The prediction provider 260 further supports the inter-layer prediction mode, for example, according to a reconstructed portion of the base layer signal, such as 200a, 200b, 200c. Alternatively, reconstructed portions of the base layer residual signal 640, potentially refined to increasing resolution / quality levels, are used as a reference / basis.

[0137] As mentioned above, the decoding stages 60 and 80 operate in a block-based manner. That is, a frame of video is subdivided into portions such as blocks. Different coarseness levels are used to assign prediction modes performed by the prediction providers 160 and 260, local transforms by the inverse transformers 560 and 580, filter coefficient selection by the filters 120 and 140, and prediction parameter settings for the prediction modes by the prediction providers 160 and 260. That is, the subpartitioning of a frame into prediction blocks is, in turn, a continuation of the subpartitioning of the frame into blocks (e.g., so-called coding units or prediction units) for which a prediction mode is selected. For transform coding, the subpartitioning of a frame into blocks (so-called transform units) is different from the partitioning into prediction units. Some of the inter-layer prediction modes used by the prediction provider 260 are described below with reference to examples. The prediction provider 260 is applied based solely on the state of the current level encoding stage 60, 80, respectively, of several intra-layer prediction modes, i.e., prediction modes that internally derive the respective prediction signals input to the respective adders 180, 340.

[0138] Further details of some of the blocks shown in the figures will become apparent from the following description of the individual embodiments. Note that unless such descriptions explicitly relate to the embodiment provided, these descriptions may be equally generally transferred to the description of other embodiments and figures.

[0139] In particular, the embodiment for the scalable video decoder of Figure 23 represents a possible implementation of a scalable video decoder according to Figures 2 and 4. Although the scalable video decoder according to Figure 23 has been described above, Figure 23 also illustrates the corresponding scalable video encoder, and the same reference signs are used for the internal elements of the predictive encoding / decoding scheme in Figures 23 and 24 for the reasons explained above. Also, for the purpose of maintaining a common prediction basis between the encoder and decoder, reconfigurable versions of the base and enhancement layer signals are used in the encoder to reconstruct the already coded portions to obtain a reconfigurable version of the scalable video. Thus, the only difference from the description of Figure 23 is that prediction provider 160 and prediction provider 260, as well as coding parameter inter-layer predictor 240, determine their prediction parameters during some rate / distortion optimization process rather than receiving them from the data stream. Rather, the providers transmit the prediction parameters thus determined to entropy decoders 19a and 19b. Entropy decoders 19a and 19b sequentially transmit their respective base layer substreams 6a and enhancement layer substreams 6b via multiplexer 16 for inclusion in data stream 6. In a similar manner, these entropy encoders 19a and 19b receive prediction residuals between reconstructed base layer signal 200 and reconstructed enhancement layer signal 400 and the original base layer and enhancement layer versions 4a and 4b, as obtained via subtractors 720 and 722 followed by transformation modules 724 and 726, rather than outputting the entropy decoding results of such residuals. However, apart from this, the structure of the scalable video encoder of FIG. 24 corresponds to the structure of the scalable video decoder of FIG. 23. Therefore, with regard to these issues, reference is made to the above description of FIG. 23. Here, as just outlined, any derivation from any data stream must be translated into a respective determination of the respective element with subsequent insertion into the respective data stream.

[0140] The techniques for intra-coding enhancement layer signals used in the embodiments described below include multiple methods for generating intra-prediction signals (using base layer data) for enhancement layer blocks, which are provided in addition to methods for generating intra-prediction signals based solely on reconstructed enhancement layer samples.

[0141] Intra prediction is part of the process of reconstructing an intra-coded block. The final reconstructed block is obtained by adding the transform-coded residual signal (which may be zero) to the intra prediction signal. The residual signal is generated by inverse quantization (scaling) of the transform coefficient levels transmitted in the bitstream followed by an inverse transform.

[0142] The following description applies to scalable coding with quality enhancement layers (which represent input video with the same resolution as the base layer but with higher quality or fidelity) and to scalable coding with spatial enhancement layers (which have a higher resolution, i.e., a larger number of samples, than the base layer). In the case of quality enhancement layers, extraction of the base layer signal is not required, such as in block 220, but filtering 500, etc. of the reconstructed base layer samples is applied. In the case of spatial enhancement layers, extraction of the base layer signal is generally required, such as in block 220.

[0143] The embodiments described below support different methods for using reconstructed base layer samples (say 200) or base layer residual samples (say 640) for intra prediction of enhancement layer blocks. One or more of the methods described below can be supported in addition to intra-layer intra coding (where only reconstructed enhancement layer samples (say 400) are used for intra prediction). The use of a particular method may be signaled at the level of the largest supported block size (such as the size of a block / largest coding unit in H.264 / AVC or a coding tree in HEVC), or it may be signaled for all supported block sizes, or it may be signaled for a subset of supported block sizes.

[0144] For all methods described below, the prediction signal is used directly as the reconstruction signal for the block, i.e., no residual is transmitted at all. Alternatively, the selected method for inter-layer intra prediction is combined with residual coding. In a particular embodiment, the residual signal is transmitted via transform coding. That is, quantized transform coefficients (transform coefficient levels) are transmitted using an entropy coding technique (e.g., variable length coding or arithmetic coding (cf. 19b)). The residual is then obtained by inverse quantizing (scaling) the transmitted transform coefficient levels and applying an inverse transform (cf. 580). In a particular version, the complete residual block corresponding to the block from which the inter-layer intra prediction signal originates is transformed using a single transform (cf. 726). That is, the entire block is transformed using a single transform of the same size as the prediction block. In another embodiment, the prediction block is further subdivided (e.g., using hierarchical decomposition) into smaller blocks. Then, a separate transform is applied to each of the smaller blocks (which may also have different block sizes). In another embodiment, the coding unit is divided into smaller prediction blocks. Then, for zero or more of the prediction blocks, a prediction signal is generated using one of the methods for inter-layer intra prediction, and then the residual of the entire coding unit is transformed using a single transform (similar to 726), or the coding unit is subdivided into different transform units, where the subdivision to form the transform units (blocks to which a single transform is applied) is different from the subdivision to decompose the coding unit into prediction blocks.

[0145] In a specific embodiment, the (sampled / filtered) reconstructed base layer signal (illustrated as 380) is directly used as the prediction signal. Multiple methods for using the base layer to intra-predict the enhancement layer include the following: The (sampled / filtered) reconstructed base layer signal (illustrated as 380) is directly used as the enhancement layer prediction signal. This method is similar to the well-known H.264 / SVC inter-layer intra prediction mode. In this method, prediction blocks for the enhancement layer are formed by collocated samples of the base layer reconstructed signal, which are sampled (illustrated as 220) to match corresponding sample positions in the enhancement layer, and optionally filtered before or after the sampling. In contrast to the SVC inter-layer intra prediction mode, this mode is supported not only at the macroblock level (or the largest supported block size), but also at any block size. That means that the mode is not only signaled for the largest supported block size, but also that the block of the largest supported block size (macroblock in MPEG4, H.264, and coding tree block / largest coding unit in HEVC) is hierarchically subdivided into smaller blocks / coding units, and the use of the inter-layer intra prediction mode is signaled (for the corresponding block) for any supported block size. In certain embodiments, this mode is only supported for selected block sizes. Then, syntax elements signaling the use of this mode are transmitted only for the corresponding block size. Alternatively, the values ​​of syntax elements signaling the use of this mode (among other coding parameters) are correspondingly restricted for other block sizes. Another difference from the inter-layer intra prediction mode in the SVC extension of H.264 / AVC is that the inter-layer intra prediction mode is supported not only when co-located regions in the base layer are intra-coded, but also when co-located base layer regions are inter-coded or partially inter-coded.

[0146] In a particular embodiment, spatial intra prediction of the difference signal (see Example A) is performed. Multiplexing methods include the following: A (potentially sampled / filtered) reconstructed base layer signal (illustrated as 380) is combined with a spatial intra prediction signal, where a spatial intra prediction (illustrated as 420) is obtained based on difference samples for neighboring blocks (illustrated as 260). The difference samples represent the difference between the reconstructed enhancement layer signal (illustrated as 400) and the (potentially sampled / filtered) reconstructed base layer signal (illustrated as 380).

[0147] FIG. 25 illustrates the generation of an inter-layer intra prediction signal by summing 732 the (extracted / filtered) base layer reconstructed signal 380 (BL Reco) with spatial intra prediction using a difference signal 734 (EH Diff) of an already coded neighboring block 736. Here, the difference signal (EH Diff) for the already coded block 736 is generated by subtracting 738 the (extracted / filtered) base layer reconstructed signal 380 (BL Reco) from the reconstructed enhancement layer signal (EH Reco) (similar to 400). The currently coded / decoded block / region / portion is 28. That is, the inter-layer intra prediction method described in FIG. 25 uses two overlapped input signals to generate a prediction block. For this method, a difference signal 734 is required. The difference signal 734 is the difference between the reconstructed enhancement layer signal 400 and the collocated reconstructed base layer signal 200. The base layer signal 200 is sampled 220 to match the corresponding sample positions of the enhancement layer, and is optionally filtered before or after the sampling (if sampling is not applied, it is filtered, as is the case with quality scalable coding). In particular, for spatial scalable coding, the difference signal 734 typically contains mainly high-frequency components. The difference signal 734 is available for all already reconstructed blocks (i.e., all already coded / decoded enhancement layer blocks). The difference signals 734 for neighboring samples 742 of already coded / decoded blocks 736 are used as input to spatial intra prediction techniques (such as the spatial intra prediction mode specified in H.264 / AVC or HEVC). Spatial intra prediction, indicated by arrow 744, generates prediction signals 746 for different components of the block 28 to be predicted. In particular embodiments, any cropping functionality in the spatial intra prediction process (as known from H.264 / AVC or HEVC) is modified or disabled to match the dynamic range of the difference signal 734.The actually used intra prediction method (one of several provided methods, which can include planar intra prediction with any particular angle, DC intra prediction, or directional intra prediction 744) is signaled in the bitstream 6b. It is possible to use spatial intra prediction techniques (methods for generating a prediction signal using samples of already coded neighboring blocks) different from the methods provided in H.264 / AVC and HEVC. The resulting prediction block 746 (using difference samples of neighboring blocks) is the first part of the final prediction block 420.

[0148] A second portion of the prediction signal is generated using the collocated region 28 in the base layer reconstructed signal 200. For the quality enhancement layer, the collocated base layer samples are used directly or are optionally filtered, for example, by a low-pass filter or a filter 500 that attenuates high-frequency components. For the spatial enhancement layer, the collocated base layer samples are extracted. For the extraction 220, an FIR filter or a set of FIR filters is used. An IIR filter can also be used. Optionally, the reconstructed base layer samples 200 are filtered before extraction. Alternatively, the base layer prediction signal (the signal obtained after extracting the base layer) is filtered after the extraction stage. The base layer reconstruction process can include one or more additional filters, such as a deblocking filter (e.g., 120) or an adaptive loop filter (e.g., 140). The base layer reconstruction 200 used for extraction is the reconstructed signal before any loop filters (e.g., 200c). Alternatively, it may be the reconstructed signal after the deblocking filter but before any other filters (e.g., 200b), or it may be the reconstructed signal after a particular filter, or after applying all filters used in the base layer decoding process (e.g., 200a).

[0149] The two generated parts of the prediction signal (the spatially predicted difference signal 746 and the potentially filtered / sampled base layer reconstruction 380) are added 732 sample by sample to form the final prediction signal 420.

[0150] Transferring the just-outlined example to the embodiments of Figures 6 to 10 is that the just-outlined possibilities of predicting the current block of the enhancement layer signal are supported by the respective scalable video decoder / encoder as alternatives to the prediction schemes outlined with respect to Figures 6 to 10. The mode used is signaled in the enhancement layer sub-stream 6b via a respective prediction mode identifier, which is not shown in Figure 8.

[0151] In a particular embodiment, intra prediction follows inter-layer residual prediction (see Example B). Multiple methods for generating an intra prediction signal using base layer data include the following: A conventional spatial intra prediction signal (obtained using adjacent reconstructed enhancement layer samples) is combined with a (sampled / filtered) base layer residual signal (the inverse transform of the base layer transform coefficients or the difference between the base layer reconstruction and the base layer prediction).

[0152] Figure 26 shows the generation of the inter-layer intra prediction signal 420 by the sum 752 of a (extracted / filtered) base layer residual signal 754 (BL Resi) and spatial intra prediction 756 using reconstructed enhancement layer samples 758 (EH Reco) of already coded neighboring blocks, depicted by the dotted line 762.

[0153] The concept shown in Figure 26 overlaps two prediction signals to form a prediction block 420, where one prediction signal 764 is generated from already reconstructed enhancement layer samples 758 and the other prediction signal 754 is generated from base layer residual samples 480. A first portion 764 of the prediction signal 420 is obtained by applying spatial intra prediction 756 using the reconstructed enhancement layer samples 758. The spatial intra prediction 756 is one of the methods specified in H.264 / AVC, or one of the methods specified in HEVC, or it is another spatial intra prediction technique that generates the prediction signal 764 for the current block 18 that forms the samples 758 of the neighboring block 762. The actually used intra prediction method 756 (which may be one of several provided methods, including planar intra prediction, DC intra prediction, or directional intra prediction with any particular angle) is signaled in the bitstream 6b. It is possible to use spatial intra prediction techniques (methods for generating a prediction signal using samples of neighboring blocks that have already been coded) different from those provided for H.264 / AVC and HEVC. The second part 754 of the prediction signal 420 is generated using the collocated residual signal 480 of the base layer. For quality enhancement layers, the residual signal can be used to reconstruct the base layer. Alternatively, the residual signal can be additionally filtered. For spatial enhancement layers 480, the residual signal is extracted 220 (to map base layer sample positions to enhancement layer sample positions) before it is used as the second part of the prediction signal. The base layer residual signal 480 can also be filtered before or after the extraction stage. An FIR filter is applied to extract the residual signal 220. The extraction process is configured in such a way that no filtering is performed across the transform block boundaries in the base layer that are applied for extraction purposes.

[0154] The base layer residual signal 480 used for inter-layer prediction is the residual signal from which the transform coefficient levels of the base layer are obtained by scaling and inverse transform 560. Alternatively, the base layer residual signal 480 is the difference between the reconstructed base layer signal 200 (before or after deblocking and additional filtering, or during any filtering operation) and the prediction signal 660 used in the base layer.

[0155] The two generated signal components (spatial intra prediction signal 764 and inter-layer residual prediction signal 754) are added 752 to form the final enhancement layer intra prediction signal.

[0156] This means that the prediction modes just outlined with respect to Figure 26 are used or supported by any scalable video decoder / encoder in accordance with Figures 6 to 10 to form the alternative prediction modes described above with respect to Figures 6 to 10 for the currently encoded / decoded portion 28.

[0157] In a particular embodiment, weighted prediction of spatial intra prediction and base layer reconstruction (see example C) is used. This actually represents a specification announcing a particular realization of the embodiment outlined above with respect to Figures 6-10, and therefore a description of such weighted prediction to be interpreted not only as an alternative to the above embodiment, but also as a description of a possible way of implementing the embodiment outlined above with respect to Figures 6-10 differently from the given example.

[0158] Multiple methods for generating an intra prediction signal using base layer data include the following: the (extracted / filtered) reconstructed base layer signal is combined into a spatial intra prediction signal, where the spatial intra prediction is obtained based on reconstructed enhancement layer samples of neighboring blocks; the final prediction signal is obtained by weighting (e.g., 41) the spatial prediction signal and the base layer prediction signal in such a way that different frequency components use different weightings; this can be achieved, for example, by filtering (e.g., 62) the base layer prediction signal (e.g., 38) with a low-pass filter and filtering (e.g., 64) the spatial intra prediction signal (e.g., 34) with a high-pass filter, and then adding (e.g., 66) the resulting filtered signals. Alternatively, frequency-based weighting can be achieved by transforming (e.g., 72, 74) the base layer prediction signal (e.g., 38) and the enhancement layer prediction signal (e.g., 34) to form overlapping transform blocks, and then overlapping the resulting transform blocks (e.g., 76, 78), where different weighting coefficients (e.g., 82, 84) are used for different frequency locations. The resulting transform block (e.g., 42 in FIG. 10) is then inverse transformed (e.g., 84) and used as the enhancement layer prediction signal (e.g., 54). Alternatively, the resulting transform coefficients can be added (e.g., 52) to the scaled transmitted transform coefficient levels (e.g., 59) and then inverse transformed (e.g., 84) to obtain a reconstructed block (e.g., 54) before the deblocking and inner-loop process.

[0159] Figure 27 shows the generation of an inter-layer intra prediction signal by frequency-weighted summation of the (extracted / filtered) base layer reconstructed signal (BL Reco) and spatial intra prediction using reconstructed enhancement layer samples (EH Reco) of adjacent blocks already coded.

[0160] The concept of Figure 27 uses two superimposed signals 772, 774 to form a prediction block 420. A first portion 774 of the signal 420 is obtained by applying spatial intra prediction 776, corresponding to 30 in Figure 6, using reconstructed samples 778 of an already constructed adjacent block in the enhancement layer. A second portion 772 of the prediction signal 420 is generated using a collocated reconstructed signal 200 of the base layer. For the quality enhancement layer, the collocated base layer samples 200 are used directly. Alternatively, they are optionally filtered, for example, by a low-pass filter or a filter that attenuates high-frequency components. For the spatial enhancement layer, the collocated base layer samples are extracted 220. For the extraction, an FIR filter or a set of FIR filters is used. It is also possible to use an IIR filter. Optionally, the reconstructed base layer samples are filtered before extraction. Alternatively, the base layer prediction signal (the signal obtained after extracting the base layer) is filtered after the extraction stage. The base layer reconstruction process can include one or more additional filters, such as a deblocking filter 120 and an adaptive loop filter 140. The base layer reconstruction 200 used for extraction can be the reconstructed signal 200c before any of the loop filters 120, 140. Alternatively, it can be the reconstructed signal 200b after the deblocking filter 120 but before another filter. Alternatively, it can be the reconstructed signal 200a after a particular filter, or the reconstructed signal after applying all of the filters 120, 140 used in the base layer decoding process.

[0161] When comparing the references used in Figures 23 and 24 with those used in connection with Figures 6-10, block 220 corresponds to reference 38 used in Figure 6. 39 corresponds to part 380. At least as far as the parts collocated with current part 28 are concerned, 420 collocated with current part 28 corresponds to 42. Spatial prediction 776 corresponds to 32.

[0162] The two prediction signals (potentially extracted / filtered base layer reconstruction 386 and enhancement layer intra prediction 782) are combined to form the final predicted signal 420. The method for combining these signals can have the property that different weighting factors are used for different frequency components. In particular embodiments, the extracted base layer reconstruction is filtered with a low-pass filter (illustrated as 62) (it is also possible to filter the base layer reconstruction before extraction 220). The intra prediction signal (illustrated as 34 obtained by 30) is filtered with a high-pass filter (illustrated as 64). Both filtered signals are added 784 (illustrated as 66) to form the final predicted signal 420. Although the pair of low-pass and high-pass filters represents a pair of quadrature mirror filters, this is not necessary.

[0163] In another particular embodiment (illustrated in FIG. 10 ), the process of combining the two prediction signals 380 and 782 is achieved via a spatial transform. Both the (potentially extracted / filtered) base layer reconstruction 380 and the intra prediction signal 782 are transformed (illustrated as 72 and 74) using a spatial transform. Next, the transform coefficients (illustrated as 76 and 78) of both signals are scaled by appropriate weighting factors (illustrated as 82 and 84) and then added (illustrated as 90) to form the transform coefficient block (illustrated as 42) of the final prediction signal. In one version, the weighting factors (illustrated as 82 and 84) are selected in such a way that, for each transform coefficient position, the sum of the weighting factors for the components of both signals equals 1. In another version, for some or all transform coefficient positions, the sum of the weighting factors does not equal 1. In a particular version, the weighting factors are selected in such a way that for transform coefficients representing low frequency components, the weighting factors for the base layer reconstruction are greater than the weighting factors for the enhancement layer intra-prediction signal, and for transform coefficients representing high frequency components, the weighting factors for the base layer reconstruction are less than the weighting factors for the enhancement layer intra-prediction signal.

[0164] In one embodiment, the transform coefficient block (illustrated as 42) obtained (by combining the weighted transformed signals for both components) is inverse transformed (illustrated as 84) to form the final predicted signal 420 (illustrated as 54). In another embodiment, the prediction is made directly in the transform domain. That is, the coded transform coefficient levels (illustrated as 59) are scaled (i.e., inverse quantized) and added (illustrated as 52) to the transform coefficients (illustrated as 42) of the predicted signal (obtained by combining the weighted transformed signals for both components) and to the resulting block of transform coefficients (illustrated as 84) to obtain the reconstructed signal 420 for the current block (before the potential deblocking 120 and further intra-loop filtering stage 140, although not shown in FIG. 10). In other words, in the first embodiment, the transform block obtained by combining the weighted transformed signals for both components is inverse transformed and used as the enhancement layer predicted signal. Alternatively, in a second embodiment, the scaled transmitted transform coefficient levels are added and inverse transformed to obtain the reconstructed block before the deblocking and inner-loop process.

[0165] The choice between base layer reconstruction and residual signal (see Example D) is also used. For the method using the reconstructed base layer signal (described above), the following version is used:

[0166] The reconstructed base layer samples 200c before deblocking 120 and further in-loop processing 140 (such as a sample adaptive offset filter or adaptive loop filter). The reconstructed base layer samples 200b after deblocking 120 but before another in-loop process 140 (such as a sample-as-adaptive offset filter or adaptive loop filter). Reconstructed base layer samples 200a after deblocking 120 and further inner-loop processing 140 (such as a sample adaptive offset filter or adaptive loop filter) or between multiple inner-loop processing stages.

[0167] The selection of the corresponding base layer signal 200a, b, c may be fixed for a particular decoder (and encoder) implementation, or it may be signaled in the bitstream 6. For the latter case, different versions may be used. The usage of a particular version of the base layer signal may be signaled at the sequence level, or at the picture level, or at the slice level, or at the largest coding unit level, or at the coding unit level, or at the predictive block level, or at the transform block level, or at any other block level. In other versions, the selection may depend on other coding parameters (such as the coding mode) or on characteristics of the base layer signal.

[0168] In another embodiment, multiple versions of the method for using the (sampled / filtered) base layer signal 200 are used. For example, two different modes are provided that directly use the sampled base layer signal (i.e., 200a), where the two modes use different interpolation filters. Alternatively, one mode uses additional filtering 500 of the (sampled) base layer reconstruction signal. Similarly, multiple different versions are provided for the other modes mentioned above. The employed sampled / filtered base layer signal 380 for the different versions of the mode differ in the interpolation filters used (including interpolation filters that also filter integer sample positions). Alternatively, the sampled / filtered base layer signal 380 for the second version is obtained by filtering the sampled / filtered base layer signal 500 for the first version. The selection of one of the different versions is signaled at the sequence, picture, slice, largest coding unit, coding unit level, prediction block level, or transform block level. Alternatively, it may be inferred from properties of the corresponding reconstructed base layer signal or transmitted coding parameters.

[0169] The same applies to the mode using the reconstructed base layer residual signal, via 480, where a different version of the interpolation filter or additional filtering stage is also used.

[0170] Different filters are used to extract / filter the reconstructed base layer signal and the base layer residual signal, which means that a different approach is used to extract the base layer residual signal than to extract the base layer reconstructed signal.

[0171] For base layer blocks, the residual signal is zero (i.e., no transform coefficient levels are transmitted for the block). The corresponding base layer residual signal is replaced with another signal derived from the base layer. For example, this could be a high-pass filtered version of the reconstructed base layer block, or another different signal derived from samples of the reconstructed base layer residual of an adjacent block.

[0172] As far as samples used for spatial intra prediction in the enhancement layer (see embodiment H) are concerned, the following special processing is provided: For modes using spatial intra prediction, unavailable neighboring samples in the enhancement layer (neighboring blocks are coded after the current block, so neighboring samples are unavailable) are replaced with corresponding samples of the extracted / filtered base layer signal.

[0173] As far as encoding of intra prediction modes (see Example X) is concerned, the following special modes and functionality are provided: For modes using spatial intra prediction, such as 30a, the encoding of the intra prediction mode is modified in such a way that information about the intra prediction mode in the base layer (if available) is used to more efficiently encode the intra prediction mode in the enhancement layer. This is used, for example, for parameter 56. If a collocated region in the base layer (e.g., 36) is intra-coded using a particular spatial intra prediction mode, it is likely that a similar intra prediction mode will be used in the enhancement layer block (e.g., 28). The intra prediction mode is usually signaled in such a way that one or more modes are classified as the most likely mode among a set of possible intra prediction modes, whereby it is signaled with a shorter word. Alternatively, the shorter the arithmetic encoding, the fewer bits the alternative results in. In HEVC intra prediction, the intra prediction mode of the block above (if available) and the intra prediction mode of the block to the left (if available) are included in the set of most likely modes. In addition to these modes, one or more additional modes (which are often used) are included in the list of most likely modes, where the actual additional mode depends on the availability of the intra prediction modes of the block above the current block and the block to the left of the current block. In HEVC, three modes are classified as the most likely modes. In H.264 / AVC, one mode is classified as the most likely mode. This mode is obtained based on the intra prediction modes used for the block above the current block and the block to the left of the current block. Any other concept for classifying intra prediction modes (different from H.264 / AVC and HEVC) is possible and will be used for the following extensions.

[0174] To use base layer data for efficient coding of intra-prediction modes in the enhancement layer, the concept of using one or more most likely modes is modified in such a way that the most likely modes include the intra-prediction modes used in the co-located base layer blocks (if the corresponding base layer blocks are intra-coded). In a particular embodiment, the following approach is used: Given a current enhancement layer block, a co-located base layer block is determined. In a particular version, the co-located base layer block is the base layer block that covers the co-located position of the top-left sample of the enhancement block. In another version, the co-located base layer block is the base layer block that covers the co-located position of the central sample of the enhancement block. In another version, another sample in the enhancement layer block is used to determine the co-located base layer block. If the determined co-located base layer block is intra-coded, the base layer intra-prediction mode specifies an angular intra-prediction mode, and the intra-prediction mode obtained from the enhancement layer block to the left of the current enhancement layer block does not use an angular intra-prediction mode, then the intra-prediction mode obtained from the left enhancement layer block is replaced with the corresponding base layer intra-prediction mode. Otherwise, if the determined co-located base layer block is intra-coded, the base layer intra-prediction mode specifies an angular intra-prediction mode, and the intra-prediction mode obtained from the enhancement layer block above the current enhancement layer block does not use an angular intra-prediction mode, then the intra-prediction mode obtained from the above enhancement layer block is replaced with the corresponding base layer intra-prediction mode. In other versions, a different approach is used to modify the list of most likely modes (consisting of a single element) using the base layer intra-prediction mode.

[0175] Inter-coding techniques for spatial and quality enhancement layers are then presented.

[0176] In state-of-the-art hybrid video coding standards (such as H.264 / AVC or the upcoming HEVC), images of an image sequence are divided into blocks of samples. The block size is either fixed or the coding scheme provides a hierarchical structure that allows further subdivision of blocks into blocks with smaller block sizes. Reconstruction of a block is usually obtained by generating a prediction signal for the block and adding a transmitted residual signal. The residual signal is usually transmitted using transform coding, which means that a quantization index list for the transform coefficients (also called transform coefficient levels) is transmitted using entropy coding techniques. Then, at the decoder side, these transmitted transform coefficient levels are scaled and inverse transformed to obtain a residual signal that is added to the prediction signal. The residual signal is generated either by intra prediction (using only data already transmitted for the current time instant) or by inter prediction (using data already transmitted for a different time instant).

[0177] In inter-prediction, a prediction block is obtained by motion-compensated prediction using samples of an already reconstructed frame. This is done by unidirectional prediction (using one reference picture and one set of motion parameters). Alternatively, a prediction signal can be generated by multi-hypothesis prediction. In the latter case, two or more prediction signals are overlapped. That is, for each sample, a weighted average is formed to form the final prediction signal. The (overlapped) multi-prediction signals are generated using different motion parameters (e.g., different reference pictures or motion vectors) for different hypotheses. For unidirectional prediction, it is also possible to multiply samples of the motion-compensated prediction signal with a constant coefficient and add a constant offset to form the final prediction signal. Such scaling and offset correction are also used for all or selected hypotheses in the multi-hypothesis prediction.

[0178] In scalable video coding, base layer information is utilized to support the inter-prediction process for the enhancement layer. The state-of-the-art video coding standard for scalable coding, the SVC extension of H.264 / AVC, adds an additional mode to improve the coding efficiency of the inter-prediction process in the enhancement layer. This mode is signaled at the macroblock level (a block of 16x16 luma samples). In this mode, reconstructed residual samples in the lower layer are used to refine the motion-compensated prediction signal in the enhancement layer. This mode is also called inter-layer residual prediction. If this mode is selected for a macroblock in a quality enhancement layer, the inter-layer prediction signal is constructed by collocated samples of the reconstructed lower layer residual signal. If the inter-layer residual prediction mode is selected in a spatial enhancement layer, the prediction signal is generated by extracting the collocated reconstructed base layer residual signal. An FIR filter is used for the extraction. However, filtering is not applied across transform block boundaries. A prediction signal generated from the reconstructed base layer residual samples is added to a conventional motion-compensated prediction signal to form a final prediction signal for the enhancement layer block. Generally, for inter-layer residual prediction modes, an additional residual signal is transmitted by transform coding. The transmission of the residual signal is omitted (inferred to be equal to zero) if it is correspondingly signaled in the bitstream. The final reconstructed signal is obtained by scaling the transmitted transform coefficient levels and applying an inverse spatial transform to the prediction signal, where the inter-layer residual prediction signal is obtained by adding the motion-compensated prediction signal.

[0179] Next, techniques for inter-coding enhancement layer signals are described. This section describes methods for employing a base layer signal, in addition to an already reconstructed enhancement layer signal, to intra-predict an enhancement layer signal to be coded in a scalable video coding scenario. By employing a base layer signal for inter-predicting an enhancement layer signal to be coded, prediction errors are significantly reduced, resulting in overall bit rate savings for enhancement layer coding. The main focus of this section is augmenting block-based motion compensation of enhancement layer samples using already-coded enhancement layer samples with additional signals from the base layer. The following description provides possibilities for using various signals from the coded base layer. Although a four-branch tree block partitioning is generally adopted as a preferred embodiment, the presented examples apply to a general block-based hybrid coding approach without assuming any specific block partitioning. The use of base layer reconstruction of the current time index, base layer residual of the current time index, or even base layer reconstruction of an already-coded image for inter-prediction of an enhancement layer block to be coded is described. Also described is a method for combining the base layer signal with an already coded enhancement layer signal to obtain a better prediction for the current enhancement layer. One of the main state-of-the-art techniques is inter-layer residual prediction in H.264 / SVC. Inter-layer residual prediction in H.264 / SVC is employed for all inter-coded macroblocks, regardless of whether they are coded using the SVC macroblock type, signaled by using either the base mode flag or a conventional macroblock type. A flag is added to the macroblock syntax for spatial and quality enhancement layers, signaling the usage of inter-layer residual prediction. When this residual prediction flag is equal to 1, the residual signal of the corresponding region in the reference layer is block-wise extracted using a bilinear filter and used as the prediction for the residual signal of the enhancement layer macroblock. As a result, only the corresponding difference signal needs to be coded in the enhancement layer. In the descriptions in this section, the following notation will be used: t0:= time index of the current image t1:=time index of an already reconstructed image EL:=Enhancement Layer BL:= Base Layer EL(t0):=Current enhancement layer image to be coded EL_reco:=Enhancement layer reconstruction BL_reco:= Base layer reconstruction BL_resi:= Base layer residual signal (the inverse transform of the base layer transform coefficients, or the difference between the base layer reconstruction and the base layer prediction) EL_diff:=Difference between the enhancement layer reconstruction and the extracted / filtered base layer reconstruction The different base layer and enhancement layer signals are used in the description illustrated in FIG.

[0180] For the description, the following properties of the filter are used: Linearity: Although many of the filters mentioned in the description are linear, non-linear filters are also used. Number of output samples: In a subtraction operation, the number of output samples is greater than the number of input samples. Here, filtering of the input data produces more samples than the input values. In conventional filtering, the number of output samples is equal to the number of input samples. Such filtering operations are used, for example, in quality scalable coding. Phase delay: For filtering samples at integer positions, the phase delay is usually zero (or an integer delay in samples). For samples occurring at fractional positions (e.g., half-pel or quarter-pel positions), a filter with a fractional delay (in units of samples) is usually applied to the samples of the integer lattice.

[0181] Conventional motion compensation prediction, used in all hybrid video coding standards (e.g., MPEG-2, H.264 / AVC, or the upcoming HEVC standard), is illustrated in Figure 29. To predict the signal of a current block, an area of ​​an already reconstructed image is replaced and used as a prediction signal. To signal the replacement, a motion vector is usually coded in the bitstream. For integer-sample precision motion vectors, the reference area in the reference picture is directly copied to form the prediction signal. However, it is also possible to transmit fractional-sample precision motion vectors. In this case, the prediction signal is obtained by filtering the reference signal with a filter having a fractional-sample delay. The reference picture used is usually specified by including a reference picture index in the bitstream syntax. In general, it is also possible to overlap two or more prediction signals to form the final prediction signal. This concept is supported, for example, in B slices with two motion hypotheses. In this case, multiple prediction signals are generated using different motion parameters (e.g., different reference pictures or motion vectors) for the different hypotheses. For unidirectional prediction, it is also possible to multiply the samples of the motion compensated prediction signal with a constant factor and add a constant offset to form the final prediction signal. Such scaling and offset correction is also used for all or selected hypotheses in multi-hypothesis prediction.

[0182] The following description applies to scalable coding with quality enhancement layers (the enhancement layers have the same resolution as the base layer but represent input video with higher quality or fidelity) and to scalable coding with spatial enhancement layers (the enhancement layers have a higher resolution, i.e., a larger number of samples, than the base layer). For quality enhancement layers, extraction of the base layer signal is not required, although filtering of the reconstructed base layer samples is applied. For spatial enhancement layers, extraction of the base layer signal is generally required.

[0183] Embodiments support different methods for using reconstructed base layer samples or base layer residual samples for inter prediction of enhancement layer blocks. In addition to conventional inter prediction and intra prediction, it is possible to support one or more of the methods described below. The usage of a particular method is signaled at the level of the largest supported block size (such as macroblock in H.264 / AVC or coding treeblock / largest coding unit in HEVC), it is signaled for all supported block sizes, or it is signaled for a subset of supported block sizes.

[0184] For all methods described below, the prediction signal is directly used as the reconstruction signal for the block. Alternatively, the selected method for inter-layer intra prediction is combined with residual coding. In a particular embodiment, the residual signal is transmitted via transform coding. That is, quantized transform coefficients (transform coefficient levels) are transmitted using an entropy coding technique (e.g., variable length coding or arithmetic coding), and the residual is obtained by inverse quantizing (scaling) the transmitted transform coefficient levels and applying an inverse transform. In a particular version, the complete residual block corresponding to the block from which the inter-layer intra prediction signal is generated is transformed using a single transform. That is, the entire block is transformed using a single transform of the same size as the prediction block. In another embodiment, the prediction block is further subdivided into smaller blocks, for example, using hierarchical decomposition. A separate transform is applied to each smaller block with a different block size. In another embodiment, the coding unit is divided into smaller prediction blocks. Then, for zero or more of the prediction blocks, prediction signals are generated using one of the methods for inter-layer intra prediction. The remainder of the entire coding unit is then transformed using a single transform, or the coding unit is subdivided into different transform units, where the subdivision to form the transform units (blocks to which a single transform is applied) is different from the subdivision to decompose the coding unit into prediction blocks.

[0185] In the following, the possibility of performing prediction using base layer residual and enhancement layer reconstruction is described. Multiple methods include the following: A conventional inter-prediction signal (obtained by motion-compensated interpolation of already reconstructed enhancement layer images) is combined with a (extracted / filtered) base layer residual signal (the inverse transform of the base layer transform coefficients or the difference between the base layer reconstruction and the base layer prediction). This method is also called "BL_resi" mode (by analogy with Figure 30).

[0186] In summary, predictions for enhancement layer samples are described below. EL prediction=filter(BL_resi(t0))+MCP_filter(EL_reco(t1)) It is also possible that more than one hypothesis of the enhancement layer reconstruction signal is used. For example, EL prediction=filter(BL_resi(t0))+MCP_filter1(EL_reco(t1))+MCP_filter2(EL_reco(t2)) The motion compensated prediction (MCP) filter used on the enhancement layer (EL) reference picture is of integer or fractional sample accuracy. The MCP filter used on the EL reference picture may be the same as or different from the MCP filter used on the BL reference picture during the BL decoding process. A motion vector MV(x, y, t) is defined to indicate a specific location within an EL reference image. The parameters x and y indicate the spatial location within the image. The parameter t is used to describe the temporal index of the reference image and is also called the reference index. Often, the term motion vector is used to refer to only two spatial components (x, y). The integer part of the MV is used to fetch a set of samples from the reference image, and the fractional part of the MV is used to select an MCP filter from a set of filters. The fetched reference samples are filtered to produce filtered reference samples. Motion vectors are generally coded using differential prediction, which means that a motion vector predictor is derived based on already coded motion vectors (and a syntax element potentially indicates the used one of a set of potential motion vector predictors), and a differential vector is included in the bitstream. The final motion vector is obtained by adding the transmitted motion vector difference to the motion vector predictor. Usually, it is also possible to completely derive the motion parameters for a block. Therefore, a list of potential motion parameter candidates is usually constructed based on already coded data, and this list may include motion parameters of spatially adjacent blocks as well as motion parameters obtained based on motion parameters in collocated blocks in the reference frame. The base layer (BL) residual signal can be defined as one of the following: Inverse transformation of BL transformation coefficients, or The difference between the BL reconstruction and the BL prediction, or For BL blocks where the inverse transform of the BL transform coefficients is zero, it can be replaced by another signal derived from BL (e.g., a high-pass filtered version of the reconstructed BL block), or A combination of the above methods. To calculate the EL predictor from the current BL residual, a region in the BL image collocated with the considered region in the EL image is identified, and the residual signal is extracted from the identified BL region. The definition of the collocated region is made such that it accounts for an integer scaling factor of the BL resolution (e.g., 2x scalability) or a fractional order scaling factor of the BL resolution (e.g., 1.5x scalability). Or it can even produce an EL resolution that is the same as the BL resolution (e.g., quality scalability). In the case of quality scalability, the collocated block in the BL image has the same coordinates as the EL block to be predicted. The collocated BL residual can be extracted / filtered to generate a filtered BL residual sample. The final EL prediction is obtained by adding the filtered EL reconstruction samples and the filtered BL residual samples.

[0187] Multiple methods for prediction using base layer reconstruction and enhancement layer difference signal (see embodiment J) include the following method: The (sampled / filtered) reconstructed base layer signal is combined with a motion compensated prediction signal, where the motion compensated prediction signal is obtained by a motion compensated difference image. The difference image represents the difference between the reconstructed enhancement layer signal and the (sampled / filtered) reconstructed base layer signal with respect to a reference image. This method is also called BL_reco mode.

[0188] This concept is illustrated in Figure 31. In summary, predictions for EL samples are described below. EL prediction=filter(BL_reco(t0))+MCP_filter(EL_diff(t1))

[0189] It is also possible that more than one hypothesis of the EL difference signal is used. For example, EL prediction=filter(BL_resi(t0))+MCP_filter1(EL_diff(t1))+MCP_filter2(EL_diff(t2))

[0190] For the EL difference signal the following version is used: the difference between the EL reconstruction and the extracted / filtered BL reconstruction, or · The difference between the EL reconstruction before or during the loop filtering stage (like deblocking, SAO, ALF) and the extracted / filtered BL reconstruction.

[0191] The usage of a particular version may be fixed in the decoder, or it may be signaled at the sequence level, picture level, slice level, largest coding unit level, coding unit level, or another partition level, or it may depend on other coding parameters.

[0192] When the EL difference signal is defined using the difference between the EL reconstruction and the extracted / filtered BL reconstruction, it is amenable to just store the EL reconstruction and the BL reconstruction and use prediction mode to calculate the EL difference signal on the fly for the block, thereby saving the memory required to store the EL difference signal. However, it incurs a small computational complexity overhead.

[0193] The MCP filters used on the EL difference images can be of integer or fractional sample accuracy. · The MCP of the difference image can use a different interpolation filter than the MCP of the reconstructed image. For the MCP of the difference image, an interpolation filter is selected based on the characteristics of the corresponding region in the difference image (or based on coding parameters in the bitstream, or based on transmitted information).

[0194] A motion vector MV(x,y,t) is defined to point to a specific location in the EL difference image. The parameters x and y indicate the spatial position within the image, and the parameter t is used to describe the time index of the filtered difference image.

[0195] The integer part of the MV is used to fetch a set of samples from the difference image, and the fractional part of the MV is used to select an MCP filter from a set of filters. The fetched difference samples are filtered to produce filtered difference samples.

[0196] The dynamic range of the difference image can theoretically exceed that of the original image. Assuming an 8-bit representation of the image in the range

[0255] , the difference image can have a range [-255 255]. In practice, however, most of the amplitudes are distributed around ±0. In a preferred embodiment for storing the difference image, a constant offset of 128 is added, and the result is clipped to the range

[0255] and stored as a regular 8-bit image. Then, during the encoding and decoding process, the offset of 128 is subtracted back from the difference amplitude read from the difference image.

[0197] For the method using the reconstructed BL signal, the following versions can be used: it is fixed, or it is signaled at the sequence level, picture level, slice level, largest coding unit level, coding unit level, or another partition level, or it can depend on other coding parameters. Reconstructed base layer samples before deblocking and further in-loop processing (such samples as the adaptive offset filter or adaptive loop filter). Reconstructed base layer samples after deblocking, but before further in-loop processing (such as adaptive offset filters or adaptive loop filters). Reconstructed base layer samples after deblocking and further in-loop processing (such as adaptive offset filters or adaptive loop filters), or reconstructed base layer samples between multiple in-loop processing steps.

[0198] To calculate the EL prediction component from the current BL reconstruction, a region in the BL image collocated with the considered region in the EL image is identified. Then, the reconstructed signal is extracted from the identified BL region. The definition of the collocated region is made such that it accounts for an integer scaling factor of the BL resolution (e.g., 2× scalability), or a fractional scaling factor of the BL resolution (e.g., 1.5× scalability), or even yields an EL resolution that is the same as the BL resolution (e.g., SNR scalability). In the case of SNR scalability, the collocated block in the BL image has the same coordinates as the EL block to be predicted.

[0199] The final EL prediction is obtained by adding the filtered EL difference samples and the filtered BL reconstruction samples.

[0200] Some possible variations of the mode of combining the (sampled / filtered) base layer reconstruction signal and the motion compensated enhancement layer difference signal are described below. Multiple versions of the method using the (sampled / filtered) BL signal can be used. The sampled / filtered BL signal employed for these versions can differ from the interpolation filter used (including interpolation filters that also filter integer sample positions), or the sampled / filtered BL signal for the second version can be obtained by filtering the sampled / filtered BL signal for the first version. The selection of one of the different versions can be signaled at the sequence, picture, slice, largest coding unit, coding unit level, or another level of picture partitioning, or it can be inferred from the characteristics of the corresponding reconstructed BL signal or the transmitted coding parameters. · Different filters are used for the extracted / filtered BL reconstructed signal in BL_reco mode and for the BL residual signal in BL_resi mode. The extracted / filtered BL signal can also be combined with two or more hypotheses of the motion compensated difference signal. This is illustrated in FIG.

[0201] Considering the above, prediction is performed using a combination of base layer reconstruction and enhancement layer reconstruction (see Example C). One major difference from the above description of Figures 11, 12, and 13 is the coding mode for obtaining the intra-layer prediction 34, which is performed temporally rather than spatially. That is, instead of spatial prediction 30, temporal prediction 32 is used to form the intra-layer prediction signal 34. Therefore, some examples described below can be easily transferred to the above embodiments of Figures 6 to 10 and Figures 11 to 13. Multiplexing methods include the following: The (extracted / filtered) reconstructed base layer signal is combined into an inter-prediction signal, where the inter-prediction is obtained by motion-compensated prediction using the reconstructed enhancement layer image; The final prediction signal is obtained by weighting the inter-prediction signal and the base layer prediction signal in a way that different frequency components use different weightings. For example, this can be achieved by any of the following: Filtering the base layer prediction signal with a low-pass filter, filtering the intermediate prediction signal with a high-pass filter, and summing the resulting filtered signals. Transforming the base layer prediction signal and the inter prediction signal and convolving the resulting transform block, where different weighting coefficients are used for different frequency positions. The resulting transform block is then inverse transformed and used as the enhancement layer prediction signal. Alternatively, the resulting transform coefficients are added to the scaled transmitted transform coefficient levels and inverse transformed to obtain a reconstructed block before the deblocking and inner-loop process.

[0202] This mode is also called the "BL_comb" mode described in FIG.

[0203] Briefly, the EL predictions are described below. EL prediction=BL_weighting(BL_reco(t0))+EL_weighting(MCP_filter(EL_reco(t1)))

[0204] In a preferred embodiment, the weighting is done depending on the ratio of EL resolution to BL resolution. For example, if the BL is to be scaled up by a factor in the range [1 1.25], a given set of weightings for EL and BL reconstruction is used. If the BL is to be scaled up by a factor in the range [1.25 1.75], a different set of weightings is used. If the BL is to be scaled up by a factor greater than or equal to 1.75, another different set of weightings is used, etc.

[0205] Scaling factor separation. Specific weighting depending on the base layer and the enhancement layer is also possible in other embodiments for spatial intra-layer prediction.

[0206] In another preferred embodiment, weighting is performed depending on the EL block size to be predicted. For example, for a 4x4 block in the EL, a weighting matrix is ​​defined that specifies weightings for the EL reconstructed transform coefficients. And another weighting matrix is ​​defined that specifies weightings for the BL reconstructed transform coefficients. The weighting matrix for the BL reconstructed transform coefficients is, for example, as follows: 64,63,61,49, 63,62,57,40, 61,56,44,28, 49,46,32,15, And the weighting matrix for the EL reconstruction transform coefficients is, for example: 0,2,8,24, 3,7,16,32, 9,18,20,26, 22,31,30,23,

[0207] Similarly, separate weighting matrices are defined for block sizes such as 8x8, 16x16, 32x32, etc.

[0208] The actual transform used for frequency domain weighting may be the same as or different from the transform used to code the prediction residual. For example, an integer approximation for the DCT may be used for both frequency domain weighting and to calculate the transform coefficients of the prediction residual to be coded in the frequency domain.

[0209] In another preferred embodiment, a maximum transform size is defined for the frequency domain weighting in order to limit the amount of computation. If the EL block size under consideration is larger than the maximum transform size, the EL and BL reconstructions are spatially separated into a series of adjacent sub-blocks. The frequency domain weighting is performed on the sub-blocks. The final prediction signal is formed by assembling the weighted results.

[0210] Additionally, weighting can be performed on the luma and chroma components, or on a selected subset of the color components.

[0211] In the following, different possibilities for obtaining enhancement layer coding parameters are described. The coding (or prediction) parameters to be used to reconstruct the enhancement layer blocks are obtained by multiplexing from the collocated coding parameters in the base layer. The base layer and the enhancement layer can have different spatial resolutions, or they can have the same spatial resolution.

[0212] In the scalable video extension of H.264 / AVC, inter-layer motion prediction is performed for a macroblock type signaled by the syntax element base mode flag. If the base mode flag is equal to 1 and the corresponding reference macroblock in the base layer is inter-coded, the enhancement layer macroblock is also inter-coded, and all motion parameters are inferred from the co-located base layer block. Otherwise (base mode flag is equal to 0), for each motion vector, a syntax element called "motion prediction flag" is sent specifying whether the base layer motion vector is used as the motion vector predictor. If the "motion prediction flag" is equal to 1, the motion vector predictor of the co-located reference block of the base layer is scaled according to the resolution ratio and used as the motion vector predictor. If the "motion prediction flag" is equal to 0, the motion vector predictor is calculated as specified in H.264 / AVC.

[0213] The following describes a method for obtaining enhancement layer coding parameters. A sample array associated with a base layer image is decomposed into blocks, and each block is associated with coding (or prediction) parameters. In other words, every sample position within a particular block has a particular associated coding (or prediction) parameter. The coding parameters include parameters for motion-compensated prediction, including the number of motion hypotheses, reference index lists, motion vectors, motion vector predictor identifiers, and merge identifiers. The coding parameters also include intra-prediction parameters, such as intra-prediction direction.

[0214] It is signaled in the bitstream that blocks in the enhancement layer are coded using collocated information from the base layer.

[0215] For example, the derivation of enhancement layer coding parameters (see Example T) is made as follows: For an NxM block of the enhancement layer that is signaled using collocated base layer information, coding parameters related to sample positions within the block are obtained based on coding parameters related to collocated sample positions in the base layer sample array.

[0216] In a particular embodiment, this process involves the following steps: 1. Derive coding parameters for each sample position in the NxM enhancement layer block based on the base layer coding parameters. 2. Derive a partition of the NxM enhancement layer blocks into subblocks so that all sample positions within a particular subblock have the same associated coding parameters.

[0217] Also, the second step can be omitted.

[0218] Step 1 is to find the enhancement layer sample position p el function f c and given the encoding parameter c, i.e., c=f c (p el )

[0219] For example, to ensure a minimum block size m×n in the enhancement layer, the function fc is calculated by the function f in the following relation: p,m × n p given by bl It can return the encoding parameter c related to JPEG2025169276000002.jpg99147

[0220] The distance between two horizontally or vertically adjacent base layer sample positions is therefore equal to 1. Both the top-left base layer sample and the top-left enhancement layer sample have position p=(0,0).

[0221] Another example is the function f c (p el ) is the base layer sample position p el , can be reverted to the coding parameter c associated with the base layer sample position pb1 that is closest to . Also, the function f c (p el ) can interpolate coding parameters in cases where a particular enhancement layer sample position has a fractional component in units of the distance between base layer sample positions.

[0222] Before returning the motion parameters, the function f c The spatially displaced components of the motion parameters are cycled around to the nearest available value in the enhancement layer sampling lattice.

[0223] After step 1, each enhancement layer sample is predicted, since each sample position is associated with a prediction parameter after step 1. Nevertheless, in step 2, block partitions are obtained in order to perform prediction operations on larger blocks of samples or to transform code prediction residuals within blocks of derived partitions.

[0224] Step 2 is performed by grouping the enhancement layer sample locations into square or rectangular blocks, each of which is decomposed into one of a set of allowed decompositions into sub-blocks. The square or rectangular blocks correspond to leaves in a four-branch tree structure that can exist at different levels, as depicted in Figure 34.

[0225] The level and decomposition of each square or rectangular block is determined by performing the following ordered steps: a) Set the highest level to the level corresponding to a block of size NxM. Set the current level to the lowest level (i.e. the level containing a single block of square or rectangular blocks with the smallest block size). Go to step b). b) For each square or rectangular block at the current level, if there is a permissible decomposition of the square or rectangular block, then all sample positions within each subblock are associated with the same coding parameters, or with small differences (according to the magnitude of any differences). That decomposition is the candidate decomposition. Of all the candidate decompositions, select the one that decomposes the square or rectangular block into the smallest number of subblocks. If the current level is the highest level, go to step c). Otherwise, set the current level to the next higher level and go to step b). c) End

[0226] Function f c is selected in such a way that at some level in step b), there is always at least one candidate decomposition.

[0227] The grouping of blocks with the same coding parameters is not limited to square blocks, but the blocks can be grouped into rectangular blocks. Furthermore, the grouping is not limited to a quadtree structure. It is also possible to use a decomposition structure in which a block is decomposed into two rectangular blocks of the same size or into two rectangular blocks of different sizes. It is also possible to use a decomposition structure that uses a quadtree decomposition up to a certain level and then a decomposition into two rectangular blocks. Any other block decomposition is also possible.

[0228] In contrast to the SVC inter-layer motion parameter prediction mode, the described mode is not only supported at the macroblock level (or the largest supported block size), but also supported for any block size. This means that the mode is not only signaled for the largest supported block size, but also that blocks of the largest supported block size (macroblocks in MPEG4, H.264, and coding tree blocks / largest coding units in HEVC) are hierarchically subdivided into smaller blocks / coding units, and the usage of the inter-layer motion mode is signaled for any supported block size (for the corresponding block). In certain embodiments, this mode supports only selected block sizes. Then, the syntax element signaling the usage of this mode is transmitted only for the corresponding block size. Alternatively, the value of the syntax element signaling the usage of this mode (among other coding parameters) is restricted to correspond to other block sizes. Another difference from the inter-layer motion parameter prediction mode in the SVC extension of H.264 / AVC is that blocks coded in this mode are not fully inter-coded: they may contain intra-coded sub-blocks, depending on the collocated base layer signal.

[0229] One of several methods for reconstructing samples of the M×M enhancement layer block using the coding parameters obtained by the above-described methods is signaled in the bitstream. Such methods for predicting the enhancement layer block using the obtained coding parameters include: For motion compensation, using the obtained motion parameters and the reconstructed enhancement layer reference picture to obtain a prediction signal for the enhancement layer block. Combining (a) the (extracted / filtered) base layer reconstruction for the current image with (b) the motion compensation signal using the resulting motion parameters and enhancement layer reference image, which is generated by subtracting the (extracted / filtered) base layer reconstruction from the reconstructed enhancement layer image. Combination of (a) the (extracted / filtered) base layer residual image (the difference between the reconstructed signal and the prediction, or the inverse transform of the coded transform coefficient values) with (b) the motion compensated signal using the obtained motion parameters and the reconstructed enhancement layer reference image.

[0230] The process of obtaining a partition within a smaller block for the current block and obtaining coding parameters for the subblocks can classify some subblocks as intra-coded, while other subblocks are classified as inter-coded. For inter-coded subblocks, motion parameters are obtained from the co-located base layer block. However, if the co-located base layer block is intra-coded, the corresponding subblock in the enhancement layer is classified as intra-coded. For samples of such intra-coded subblocks, the enhancement layer signal is predicted by using information from the base layer. For example, A (sampled / filtered) version of the corresponding base layer reconstruction is used as the intra prediction signal. The obtained intra prediction parameters are used for spatial intra prediction within the enhancement layer.

[0231] To predict an enhancement layer block using a weighted combination of prediction signals, the following embodiments include a method for generating a prediction signal for an enhancement layer block by combining (a) an enhancement layer intra prediction signal obtained by spatial or temporal (i.e., difference-compensated) prediction using reconstructed enhancement layer samples, and (b) a base layer prediction signal that is the (extracted / filtered) base layer reconstruction for the current picture. The final prediction signal is obtained by weighting the enhancement layer intra prediction signal and the base layer prediction signal, in such a way that, for each sample, a weighting according to a weighting function is used.

[0232] For example, the weighting function can be realized in the following way: Compare a low-pass filtered version of the original enhancement layer intra-predicted signal v with a low-pass filtered version of the base layer reconstruction u. From the comparison, obtain a weighting for each sample position to be used to combine the original inter-predicted signal and the (sampled / filtered) base layer reconstruction. For example, the weighting can be obtained by mapping the difference u v to a weighting w using a transfer function t. That is, t(uv)=w

[0233] Different weighting functions are used for different block sizes of the current block to be predicted, and the weighting functions are modified according to the temporal distance of the reference images from which the intermediate prediction hypotheses are derived.

[0234] In the case of an enhancement layer intra prediction signal that is an intra prediction signal, a weighting function is implemented, for example, using different weights depending on the position within the current block to be predicted.

[0235] In a preferred embodiment, a method is used to obtain the enhancement layer coding parameters, and step 2 of the method uses the set of allowed decompositions of a square block, as described in Figure 35.

[0236] In a preferred embodiment, the function f c (p el ) is the function f described above for m=4 and n=4. p,m × n( p el )

[0237] In an embodiment, the function f c (p el ) returns the encoding parameter c: First, the base layer sample position is p bl =f p,4 ×4(p el ) is obtained. · Suppose, p bl If c has related intermediate prediction parameters obtained by merging with a previously coded base layer block (or has the same motion parameters), then c is equal to the motion parameters of the enhancement layer block corresponding to the base layer block used for merging in the base layer (i.e., the motion parameters are copied from the corresponding enhancement layer block). Otherwise, c is p bl The encoding parameters associated with

[0238] Furthermore, the above embodiments can be combined.

[0239] In another embodiment, for enhancement layer blocks to be signaled using collocated base layer information, intra prediction parameters obtained from a default set of motion parameters and their associated enhancement layer sample locations are merged with a block containing these samples (i.e., a copy of the default set of motion parameters). The default set of motion parameters consists of an indicator for using one or two hypotheses, a reference index list referencing the first image in a list of reference images, and a motion vector with a null spatial displacement.

[0240] In another embodiment, for an enhancement layer block to be signaled, using the collocated base layer information, enhancement layer samples with the obtained motion parameters are predicted and reconstructed first in a certain order. Then, samples with the obtained intra prediction parameters are predicted in intra reconstruction order. As a result, intra prediction can use already reconstructed sample values ​​from (a) neighboring intermediate prediction blocks and (b) previous neighboring intra prediction blocks in the intra reconstruction order.

[0241] In another embodiment, for an enhancement layer block that is being merged (i.e., takes motion parameters from another intermediate prediction block), the list of merge candidates additionally includes candidates from the corresponding base layer block, and, if the enhancement layer has a higher spatial sampling ratio than the base layer, additionally includes up to four candidates derived from the base layer candidates by refining the spatial permutation components with only the neighboring values ​​available in the enhancement layer.

[0242] In another embodiment, the difference magnitude used in step 2b) is such that a sub-block is declared to have small differences only if the differences vanish altogether, i.e. a sub-block is formed only if all included sample positions have the same resulting coding parameters.

[0243] In another embodiment, if (a) all included sample positions have derived motion parameters and no pair of sample positions within the block has derived motion parameters that differ by more than a specific value according to the vector standard applied to the corresponding motion vectors, or (b) all included sample positions have derived intra-prediction parameters and no pair of sample positions within the block has derived intra-prediction parameters that differ by more than a specific angle of intra-prediction direction, the difference magnitude used in step 2b) is declared to be small within the sub-block. The derived parameters for the sub-blocks are calculated by a meaningful or central operation. In another embodiment, the partition obtained by inferring coding parameters from the base layer is further refined based on side information signaled in the bitstream. In another embodiment, the residual coding for a block whose coding parameters are inferred from the base layer is independent of the partitions within the block inferred from the base layer. For example, that means that inferring coding parameters from the base layer partitions the block into several sub-blocks, each with a separate set of coding parameters, but a single transform is applied to the block. Or, a block whose partitions and coding parameters for the sub-blocks are inferred from the base layer is divided into smaller blocks for the purpose of transform coding the residual, where the division into transform blocks is independent of the inferred partitions within the block with different coding parameters.

[0244] In another embodiment, the residual coding for blocks whose coding parameters are inferred from the base layer depends on the partitioning within the block inferred from the base layer. For example, that means that for transform coding, the division of blocks within transform blocks depends on the partitioning inferred from the base layer. In one version, a single transform is applied to each sub-block with different coding parameters. In another version, the partitioning is refined based on side information included in the bitstream. In another version, several sub-blocks are aggregated into larger blocks so that the residual signal is signaled in the bitstream for the purpose of transform coding.

[0245] Also, embodiments obtained by combining the above-described embodiments are possible.

[0246] Regarding enhancement layer motion vector coding, this next section describes a method for reducing motion information in scalable video coding applications by providing multiple enhancement layer predictors and using motion information coded in the base layer to efficiently code the motion information of the enhancement layer. This idea is appropriate for scalable video coding, including spatial, temporal, and quality scalability.

[0247] In the scalable video extension of H.264 / AVC, inter-layer motion prediction is performed for macroblock types signaled by the syntax element "base mode flag". If the "base mode flag" is equal to 1 and the corresponding reference macroblock in the base layer is inter-coded, the enhancement layer macroblock is also inter-coded. All motion parameters are inferred from the co-located base layer block. Otherwise (if the "base mode flag" is equal to 0), each motion vector (the so-called "motion prediction flag" syntax element) is transmitted and specified, regardless of whether the base layer motion vector is used as the motion vector predictor. If the "motion prediction flag" is equal to 1, the motion vector predictor of the co-located reference block of the base layer is scaled according to the resolution ratio and used as the motion vector predictor. If the "motion prediction flag" is equal to 0, the motion vector predictor is calculated as specified in H.264 / AVC. In HEVC, motion parameters are predicted by applying Advanced Motion Vector Competition (AMVP). AMVP features two spatial motion vector predictors and one temporal motion vector predictor that compete with each other. Spatial candidates are selected from the positions of neighboring prediction blocks located to the left or above the current prediction block. Temporal candidates are selected among the collocated positions of the previously coded image. The positions of all spatial and temporal candidates are displayed in Figure 36.

[0248] After spatial and temporal candidates are inferred, a redundancy check is performed to introduce the zero motion vector as a candidate into the list. The candidate list describing the index is transmitted to identify the motion vector predictor to be used with the motion vector difference for motion compensated prediction. HEVC also uses a block merging algorithm that aims to reduce the coding redundant motion parameters resulting from the quad-tree based coding structure. This is achieved by creating regions consisting of multiple prediction blocks that share specific motion parameters. These motion parameters only need to be coded once for the first prediction block of each region, seeding new motion information. Similar to AMVP, the block merging algorithm constructs a list containing possible merge candidates for each prediction block. The number of candidates is defined by "NumMergeCands," which is signaled in the slice header and ranges from 1 to 5. Candidates are inferred from spatially neighboring prediction blocks and prediction blocks in the co-located temporal image. The possible sample locations for a prediction block considered as a candidate are equivalent to those shown in Figure 36. An example of the block merging algorithm with possible prediction block partitions in HEVC is illustrated in Figure 37. The thick lines in Figure 37(a) define all prediction blocks that are merged into one region and hold specific motion data. This motion data is sent only for block S. The current prediction block to be coded is indicated by an "X." The prediction blocks in the removed region are successors of prediction block X in the block scanning order and therefore do not yet have associated prediction data. The dots indicate sample positions of neighboring blocks that are possible spatial merge candidates. Before a possible candidate is inserted into the predictor list, a redundancy check for the spatial candidate is performed as indicated in Figure 37(b).

[0249] If the number of spatial and temporal candidates is less than "NumMergeCands", additional candidates are provided by combining with existing candidates or by inserting zero motion vector candidates. If a candidate is added to the list, it has an index that is used to identify the candidate. As new candidates are added to the list, the merge index (starting from 0) increases until the list is completed with the last candidate identified by index "NumMergeCands" - 1. Fixed-length codewords are used to encode the merge candidate indexes to ensure independent operation of candidate list derivation and bitstream parsing.

[0250] The following section describes a method for using multiple enhancement layer predictors, including a predictor obtained from the base layer, to encode enhancement layer motion parameters. The motion information already coded for the base layer can be used to significantly reduce the motion data rate while coding the enhancement layer. This method includes the possibility of directly obtaining all motion data of a prediction block from the base layer. In this case, no additional motion data needs to be coded. In the following description, the term prediction block refers to a prediction unit in HEVC (an M×N block in H.264 / AVC) and is understood as a general set of samples in an image.

[0251] The first part of the current section concerns extending the list of motion vector prediction candidates by a base layer motion vector predictor (see Example K). Base layer motion vectors are added to the motion vector predictor list during enhancement layer coding. This is achieved by inferring one or more motion vector predictors for co-located prediction blocks from the base layer and using them as candidates in the list of predictors for motion-compensated prediction. Co-located prediction blocks in the base layer are located at the center, left, top, right, or bottom of the current block. If the base layer prediction block at the selected location does not contain motion-related data or is outside the current range and therefore not currently accessible, two alternative locations can be used to infer the motion vector predictor. These alternative locations are depicted in Figure 38.

[0252] The inferred motion vectors of the base layer are scaled according to the resolution ratio before they are used as predictor candidates. As well as the motion vector difference, an index describing the candidate list of motion vector predictors is transmitted to the prediction block specifying the final motion vector to be used for motion compensated prediction. In contrast to scalable extensions of the H.264 / AVC standard, the embodiment presented here does not account for the usage of a motion vector predictor of a co-located block in a reference image - rather, it is available in a list among other predictors and is described by a transmitted index. In one embodiment, a motion vector is obtained from the center position C1 of the co-located prediction block in the base layer and added to the beginning of the candidate list as the first entry. The candidate list of motion vector predictors is extended by one item. If no motion data is available in the base layer for sample position C1, the list structure is untouched. In another embodiment, any sequence of sample positions in the base layer is checked for motion data. If motion data is found, the motion vector predictor at the corresponding position is inserted into the candidate list and is available for motion-compensated prediction in the enhancement layer. Additionally, motion vector predictors obtained from the base layer are inserted into the candidate list at any other position in the list. In another embodiment, a base layer motion predictor is only inserted into the candidate list if certain constraints are met. These constraints include the value of the merge flag of the co-located reference block, which must be equal to zero. Another constraint is that the width of the prediction block in the enhancement layer must be equal to the width of the co-located prediction block in the base layer relative to the resolution ratio. For example, in the application of K× spatial scalability, if the width of the collocated block in the base layer is equal to N and the width of the predictive block to be coded in the enhancement layer is equal to K*N, then the motion vector predictor is only inferred. In another embodiment, one or more motion vector predictors from several sample positions of the base layer are added to the enhancement layer candidate list. In another embodiment, candidates with motion vector predictors inferred from co-located blocks replace spatial and temporal candidates in the list rather than extending the list. It is also possible to include multiple motion vector predictors derived from base layer data in the motion vector predictor candidate list.

[0253] The second part concerns extending the list of merge candidates with base layer candidates (see Example K). Motion data of one or more co-located blocks of the base layer are added to the merge candidate list. This method enables the possibility of creating merge regions that share specific motion parameters across the base layer and enhancement layer. As in the previous section, as depicted in Figure 38, the base layer block covering the co-located sample at the central location is not limited to this central location but can be obtained from any location in the immediate vicinity. If no motion data is available or accessible for a given location, alternative locations can be selected to infer possible merge candidates. Before the obtained motion data is inserted into the merge candidate list, it is scaled according to the resolution ratio. An index describing the merge candidate list is transmitted to define the motion vector, which is used for motion-compensated prediction. However, the method also suppresses possible motion predictor candidates that depend on the motion data of the prediction block in the base layer. In an embodiment, the motion vector predictor of the co-located block in the base layer covering sample position C1 in Figure 38 is considered as a possible merge candidate for encoding the current predicted block in the enhancement layer. However, if the "merge_flag" of the referenced block is equal to 1 or if the co-located referenced block does not contain any motion data, the motion vector predictor is not inserted into the list. In any other case, the obtained motion vector predictor is added to the merge candidate list as a second entry. Note that in this embodiment, the length of the merge candidate list is maintained and not extended. In another embodiment, as depicted in Figure 38, one or more motion vector predictors are obtained from the predictor blocks covering any of the sample positions to be added to the merge candidate list. In another embodiment, one or several motion vector predictors from the base layer are added to the merge candidate list at any position. In another embodiment, one or more motion vector predictors are only added to the merge candidate list if certain constraints are met. Such constraints include the width of the prediction block of the enhancement layer matching the width of the collocated block of the base layer (with respect to the resolution ratio described in the previous embodiment section for motion vector prediction). Another constraint in another embodiment is the value of "merge_flag" equal to 1. In another embodiment, the length of the merge candidate list is extended by the number of motion vector predictors inferred from the collocated reference block of the base layer.

[0254] The third part of this specification describes the process of reordering the motion parameter (or merge) candidate list using base layer data (see Example L), which reorders the merge candidate list according to information already coded in the base layer. If the co-located base layer block covering the current block's samples is motion compensated with a candidate derived from a particular original, the corresponding enhancement layer candidate from the equivalent original (if any) is placed at the beginning of the merge candidate list as the first entry. This step is equivalent to listing this candidate with the lowest index. The lowest index assigns the simplest codeword to this candidate. In an embodiment, a co-located base layer block is motion-compensated predicted with a candidate originating from a predictor block covering sample position A1, as depicted in Figure 38. If the merge candidate list of a predictor block in the enhancement layer includes a candidate whose motion vector predictor originates from the corresponding sample position A1 in the enhancement layer, this candidate is placed in the list as the first entry. As a result, this candidate is indexed by index 0 and is therefore assigned the shortest fixed-length codeword. In this embodiment, this step is performed after derivation of the motion vector predictor of the co-located base layer block for the merge candidate list in the enhancement layer. Therefore, the reordering process assigns the lowest index to the candidate originating from the corresponding block as the motion vector predictor of the co-located base layer block. The second lowest index is assigned to the candidate derived from the co-located block in the base layer, as described in the second part of this section. Furthermore, the reordering process is performed only if the "merge_flag" of the co-located block in the base layer is equal to 1. In another embodiment, the reordering process is performed regardless of the value of "merge_flag" of the co-located prediction block in the base layer. In another embodiment, a candidate with a corresponding original motion vector predictor is placed in any position in the merge candidate list. In another embodiment, the reordering process removes all other candidates in the merge candidate list. Here, only candidates whose motion vector predictor has the same original as the motion vector predictor used for motion compensated prediction of the co-located block in the base layer remain in the list. In this case, a single candidate is used, and no index is transmitted.

[0255] The fourth part of this specification is directed to reordering the motion vector predictor candidate list using base layer data (see embodiment L), which embodies a process of reordering the motion vector prediction candidate list using the motion parameters of the base layer block. If the co-located base layer block covering the sample of the current prediction block uses a motion vector from a particular original, the motion vector predictor from the corresponding original in the enhancement layer is used as the first entry in the motion vector predictor list of the current prediction block. This results in assigning the cheapest codeword to this candidate. In an embodiment, a co-located base layer block is motion-compensated predicted with a candidate originating from the predictor block covering sample position A1, as shown in Figure 38. If the motion vector predictor candidate list of a block in the enhancement layer includes a candidate whose motion vector predictor originates from the corresponding sample position A1 in the enhancement layer, this candidate is placed in the list as the first entry. As a result, this candidate is indexed by index 0 and is therefore assigned the shortest fixed-length codeword. In this embodiment, this step is performed after derivation of the motion vector predictor of the co-located base layer block for the motion vector predictor list in the enhancement layer. Therefore, the reordering process assigns the lowest index to the candidate originating from the corresponding block as the motion vector predictor for the co-located base layer block. The second lowest index is assigned to the candidate derived from the co-located block in the base layer, as described in the first part of this section. Furthermore, the reordering process is performed only if the "merge_flag" of the co-located block in the base layer is equal to 0. In another embodiment, the reordering process is performed regardless of the value of "merge_flag" of the co-located prediction block in the base layer. In another embodiment, a candidate with a corresponding original motion vector predictor is placed in any position in the motion vector predictor candidate list.

[0256] The following concerns enhancement layer coding of transform coefficients.

[0257] In state-of-the-art video and image coding, the residual of the prediction signal is pre-transformed and the resulting quantized transform coefficients are signaled in the bitstream. This coefficient coding follows a fixed scheme.

[0258] Depending on the transform size (for luma residual: 4x4, 8x8, 16x16, 32x32), different scan directions are defined. Given the first and last positions in the scan order, these scans uniquely determine which coefficient positions may be significant and therefore need to be coded. In every scan, the first coefficient is set to be the DC coefficient at position (0,0), while the last position must be signaled in the bitstream. The bitstream is done by coding the (horizontal) x and (vertical) y positions within the transform block. Starting from the last position, signaling of significant coefficients is done in the reverse scan order until the DC position is reached.

[0259] For transform sizes 16x16 and 32x32, only one scan, i.e., "diagonal scan," is defined. However, transform blocks of sizes 2x2, 4x4, and 8x8 can also use "vertical" and "horizontal" scans. However, the use of vertical and horizontal scans is limited to the residual of the intra-prediction coding unit. The actual scan used is derived from the instruction mode of the intra-prediction. Instruction modes with indices in the range 6 and 14 result in vertical scans. Instruction modes with indices in the range 22 and 30 result in horizontal scans. All residual instruction modes result in diagonal scans.

[0260] Figure 39 shows the diagonal, vertical, and horizontal scans defined for a 4x4 transform block. The coefficients of larger transforms are subdivided into subgroups of 16 coefficients. These subgroups allow for hierarchical coding of significant coefficient positions. Subgroups signaled as non-significant do not contain significant coefficients. The transforms for 8x8 and 16x16 are shown with the scans and their associated subgroup divisions in Figures 40 and 41, respectively. The large arrows indicate the scan order of the coefficient subgroups.

[0261] In zigzag scanning, for blocks of size greater than 4x4, a subgroup consists of a 4x4 pixel block scanned with zigzag scanning. The subgroups are scanned in a zigzag manner. Figure 42 shows the vertical scan for a 16x16 transform as proposed in JCTVC-G703.

[0262] The following paragraphs describe extensions for transform coefficient coding. These include the introduction of new scanning modes (methods of assigning scans to transform blocks and modified coding of significant coefficient positions). These extensions allow a better adaptation of different coefficient distributions within a transform block, thereby achieving coding gains in rate-distortion functions.

[0263] New implementations for vertical and horizontal scan patterns are introduced for 16x16 and 32x32 transform blocks. In contrast to previously proposed scan patterns, the size of the scan subgroups is 16x1 for horizontal scans and 1x16 for vertical scans. Also, subgroups with sizes of 8x2 and 2x8 are selected, respectively. The subgroups themselves are scanned in the same way.

[0264] Vertical scanning is efficient for transform coefficients that lie in such spread-out rows, which is noticeable in images that contain horizontal edges.

[0265] Horizontal scanning is efficient for transform coefficients that lie in such spread-out rows, which is noticeable in images that contain vertical edges.

[0266] Figure 43 shows the implementation of vertical and horizontal scanning for a 16x16 transform block. Coefficient subgroups are defined as one row or one column, respectively. Vertical and horizontal scanning is the scanning pattern implemented. The scanning pattern allows encoding of coefficients within a row by column-like scanning. For a 4x4 block, the first row is scanned followed by the remainder of the first column, then the remainder of the second row, then the remainder of the coefficients in the second column, then the remainder of the third row, and finally the remainder of the fourth column and row.

[0267] For larger blocks, the block is partitioned into 4x4 subgroups. These 4x4 blocks are scanned with a vertical and horizontal scan, and the subgroups are scanned with a vertical and horizontal scan themselves.

[0268] Vertical and horizontal scanning is used when the coefficients are located in the first row and column of a block. In this way, the coefficients are scanned faster than if a different scan (e.g., diagonal scanning) were used. This is allowed for images that contain both horizontal and vertical edges.

[0269] Figure 44 shows the vertical and horizontal scan for a 16x16 transform block.

[0270] Other scans are possible as well, for example all combinations between scans and subgroups can be used, for example using horizontal scan for 4x4 blocks and diagonal scan for subgroups, an appropriate choice of scan is applied by selecting a different scan for each subgroup.

[0271] It should be mentioned that different scanning can be achieved in a way that the transform coefficients are reordered after quantization at the encoder side and conventional coding is used. At the decoder side, the transform coefficients are reordered before scaling and inverse transform (or after scaling and before the inverse transform) with conventional decoding.

[0272] Different parts of the base layer signal are used to derive the coding parameters from the base layer signal. Among the base layer signals are: Collocated reconstructed base layer signal Collocated residual base layer signal An estimated enhancement layer residual signal obtained by subtracting the enhancement layer prediction signal from the reconstructed base layer signal. Base layer frame image divider

[0273] Slope Parameters The gradient parameters are obtained as follows: For each pixel of the examined block, a gradient is calculated. From these gradients, a magnitude and an angle are calculated. The most raised angle in the block is related to the block (block angle). The angles are oriented using only three directions: horizontal (0°), vertical (90°), and diagonal (45°).

[0274] Edge Detection The edge detector is applied to the examined blocks as follows: First, the block is smoothed by an n×n smoothing filter (eg, Gaussian). A gradient matrix of size mxm is used to calculate the gradient for each pixel. The magnitude and angle of every pixel are calculated. The angles are oriented to use only three directions: horizontal (0°), vertical (90°), and diagonal (45°). For every pixel with a magnitude greater than a given threshold 1, the neighboring pixels are checked. If a neighboring pixel has a magnitude greater than threshold 2 and has the same angle as the current pixel, the counter for this angle is incremented. For the whole block, the counter with the highest value is selected as the angle for the block.

[0275] [Obtaining base layer coefficients by previous transformation] To obtain coding parameters from the frequency domain of the base layer signal for a particular TU, the examined and collocated signals (reconstructed base layer signal / residual base layer signal / estimated enhancement layer signal) are transformed in the frequency domain. Preferably, this is performed using the same transform used by that particular enhancement layer TU. The resulting base layer transform coefficients may or may not be quantized. To obtain a coefficient distribution comparable to the enhancement layer blocks, rate-distortion quantization with a modified lambda is used.

[0276] [Scan validity score for specific distribution and scan] The scan validity score for a particular significant coefficient distribution is defined as follows: Let each location of the examined block be represented by an index in the order of the examined scan. Then, the sum of the index values ​​of the significant coefficient locations is defined as the effectiveness score of this scan. As a result, the smaller the score of a scan, the more effective a particular distribution is.

[0277] [Selecting an appropriate scan pattern for transform coefficient coding] If several scans are available for a particular TU, a rule needs to be defined that uniquely selects one of the scans.

[0278] [Method for selecting scanning patterns] The selected scan can be derived directly from the already decoded signal (without transmitting any additional data), either based on the characteristics of the collocated base layer signal, or by utilizing only the enhancement layer signal. The scanning pattern can be obtained from the EL signal by: · The state-of-the-art derivation rules mentioned above. Using a scanning pattern for the chrominance residual selected for the juxtaposed luminance residual. · Defining a fixed mapping between the coding mode and the scanning pattern used. Obtaining the scan pattern from the last significant coefficient position (proportional to the estimated fixed scan pattern). In a preferred embodiment, the scanning pattern is selected depending on the last position already decoded as follows:

[0279] The last position is represented as x and y coordinates within the transform block and is already decoded (for the last dependent scan to be coded, a fixed scan pattern is assumed for the last position decoding process, which is the leading scan pattern of that TU). Let T be a defined threshold that depends on the specific transform size. If neither the x nor the y coordinate of the last significant position exceeds T, then diagonal scanning is selected.

[0280] Otherwise, x is compared with y. If x exceeds y, horizontal scanning is selected and vertical scanning is not selected. The preferred value of T for a 4x4 TU is 1. The preferred value of T for a TU larger than 4x4 is 4.

[0281] In another preferred embodiment, the derivation of the scanning pattern described in the previous embodiment is restricted to be done only for TUs of size 16x16 and 32x32, and it is further restricted to only the luminance signal.

[0282] Alternatively, the scan pattern can be derived from the BL signal. Any of the coding parameters described above can be used to derive the selected scan pattern from the base layer signal. In particular, the gradient of the collocated base layer signal can be calculated and compared with a predefined threshold, and / or potentially detected edges can be utilized.

[0283] In a preferred embodiment, the scan direction is obtained depending on the block gradient angle as follows: for gradients quantized in the horizontal direction, a vertical scan is used; for gradients quantized in the vertical direction, a horizontal scan is used; otherwise, a diagonal scan is selected.

[0284] In another preferred embodiment, scan patterns are obtained as described in the previous embodiment, but only for those transform blocks where the number of block angle occurrences exceeds a threshold. The remaining transform units are decoded using the TU's state-of-the-art scan patterns.

[0285] If the base layer coefficients of the collocated block are available and either explicitly signaled in the base layer data stream or calculated by a previous transform, the base layer coefficients can be used in the following manner. For each available scan, the cost for coding the base layer coefficients is evaluated. The scan with the lowest cost is used to decode the enhancement layer coefficients. The effectiveness score of each available scan is calculated for the base layer coefficient distribution. The scan with the smallest score is used to decode the enhancement layer coefficients. The distribution of base layer coefficients within a transform block is classified into one of a predefined set of distributions that are associated with a particular scanning pattern. The scan pattern is selected depending on the last significant base layer coefficient.

[0286] If the collocated base layer block is predicted using intra prediction, the intra direction of that prediction is used to derive the enhancement layer scanning pattern.

[0287] Furthermore, the transform size of the collocated base layer blocks is utilized to obtain the scanning pattern.

[0288] In a preferred embodiment, the scan pattern is derived from the BL signal only for TUs that represent the residuals of INTRA_COPY mode predicted blocks, and their collocated base layer blocks are intra predicted. For those blocks, a modified state-of-the-art scan selection is used. In contrast to the state-of-the-art scan selection, the intra prediction direction of the collocated base layer blocks is used to select the scan pattern.

[0289] Signaling the scan pattern index in the bitstream (see embodiment R). The scanning pattern of the transform blocks is selected by the encoder at a rate-distortion value and then signaled in the bitstream.

[0290] A particular scan pattern can be coded by signaling an index into a list of available scan pattern candidates. This list can be a fixed list of scan patterns defined for a particular transform size, or it can be actively filled during the decoding process. Actively filling the list allows for the optimal selection of those scan patterns. The scan pattern likely most efficiently codes a particular coefficient distribution. By doing so, the number of available scan patterns for a particular TU can be reduced, and as a result, signaling an index into that list is less expensive. If the number of scan patterns in a particular list is reduced to one, no signaling is necessary. For a particular TU, the process of selecting scan pattern candidates may utilize any of the coding parameters described above and / or follow predetermined rules that utilize specific characteristics of that particular TU, among them: ·TU represents the residual of the luminance / chrominance signals. · TU has a specific size. ·TU represents the residual of a particular prediction mode. The last significant position within a TU is known by the decoder and belongs to a particular subdivision of the TU. A TU is a part of one I / B / P-slice. The coefficients of a TU are quantized using a specific quantization parameter.

[0291] In a preferred embodiment, the list of scan pattern candidates includes three scans for every TU: "diagonal scan", "vertical scan" and "horizontal scan".

[0292] Another embodiment is obtained by including any combination of scan patterns in the candidate list.

[0293] In a particular preferred embodiment, the list of candidate scan patterns includes any of "diagonal scan," "vertical scan," and "horizontal scan."

[0294] However, the scan pattern selected by the state-of-the-art scan derivation (described above) is set to be in the list first. Another candidate is added to the list only if the particular TU has a size of 16x16 or 32x32. The order of the remaining scan patterns depends on the last significant coefficient position.

[0295] (Note: The diagonal scan is always the first pattern in the list to estimate 16x16 and 32x32 transformations.)

[0296] If the magnitude of the x coordinate exceeds the magnitude of the y coordinate, the horizontal scan is chosen next and the vertical scan is placed in the last position, otherwise the vertical scan is placed in the second position followed by the horizontal scan.

[0297] Another preferred embodiment is obtained by further restricting the conditions to have more than one candidate in the list.

[0298] In another embodiment, vertical and horizontal scans are only added to the candidate list for 16x16 and 32x32 transform blocks if the coefficients of the transform block represent residuals of the luminance signal.

[0299] In another embodiment, if both the x and y coordinates of the last significant position are greater than a certain threshold, the vertical and horizontal scans are added to the candidate list of transformation blocks. This threshold is mode and / or TU dependent on the size. The preferred threshold is 3 for all sizes greater than 4x4 and 1 for 4x4 TUs.

[0300] In another embodiment, a vertical or horizontal scan is only added to the candidate list of transformation blocks if either the x or y coordinate of the last significant position is greater than a certain threshold. This threshold is mode and / or TU dependent on the size. A preferred threshold is 3 for all sizes greater than 4x4 and 1 for 4x4 TUs.

[0301] In another embodiment, vertical and horizontal scans are only added to the candidate list for 16x16 and 32x32 transform blocks if both the x and y coordinates of the last significant position are greater than a certain threshold. This threshold is mode and / or TU dependent on the size. A preferred threshold is 3 for all sizes greater than 4x4 and 1 for 4x4 TUs.

[0302] In another embodiment, vertical and horizontal scans are only added to the candidate list for 16x16 and 32x32 transform blocks if either the x or y coordinate of the last significant position is greater than a certain threshold. This threshold is mode and / or TU dependent on the size. A preferred threshold is 3 for all sizes greater than 4x4 and 1 for 4x4 TUs.

[0303] For any of the described embodiments, a particular scan pattern is signaled in the bitstream. The signaling itself can be done at different signaling levels. In particular, the signaling can be done at any node of the residual quad-branch tree (all sub-TUs of that node that use the signaled scan and use the same candidate list index), at the CU / LCU level, or at the slice level for each TU falling within a sub-group of TUs that has the signaled scan pattern.

[0304] An index into the candidate list is transmitted using fixed length coding, variable length coding, arithmetic coding (including context-appropriate binary arithmetic coding), or PIPE coding. If context-appropriate coding is used, the context is derived based on parameters of neighboring blocks, the coding mode described above, and / or characteristics of the particular TU itself.

[0305] In a preferred embodiment, context-appropriate coding is used to signal the TU's index into the scan pattern candidate list, but the context model is derived based on the transform size and / or position of the last significant position within the TU. Any of the methods described above for deriving a scan pattern may be used to derive a context model for signaling an explicit scan pattern for a particular TU.

[0306] To encode the last significant scan position, the following modifications are used in the enhancement layer: Separate context models are used for all or a subset of coding modes using base layer information. It is also possible to use different context models for different modes with base layer information. The context model can depend on data in the collocated base layer block (e.g., transform coefficient distribution in the base layer, gradient information in the base layer, last can position in the collocated base layer block). The last scan position can be coded as the difference from the last base layer scan position. If the last scan position is coded in the TU by signalling the x and y position, then the context model for the second signalled coordinate can depend on the value of the first signalled coordinate. To obtain a scanning pattern that is independent of the last significant position, any of the methods described above is used to obtain a context model for signaling the last significant position.

[0307] In certain versions, the scan pattern derivation depends on the last significant position: If the last scan position is encoded in the TU by signaling its x and y position, then the context model for the second coordinate can rely on those scan patterns that are still possible candidates when already knowing the first coordinate. If the last scan position is encoded in the TU by signaling its x and y position, then the context model for the second coordinate can depend on whether the scan pattern is already uniquely selected when already knowing the first coordinate.

[0308] In another version, the scan pattern derivation is independent of the last significant position. The context model can depend on the scanning pattern used within a particular TU. Any of the methods mentioned above for obtaining the scanning pattern is then used to obtain the context model for signaling the final critical position.

[0309] To code the significance position within a TU and the significance flag (subgroup flag and / or significance flag for one transform coefficient), respectively, the following modifications are used in the enhancement layer: Separate context models are used for all or a subset of coding modes that use base layer information. It is also possible to use different context models for different modes that have base layer information. The context model can depend on data in the collocated base layer block (e.g., the number of significant transform coefficients for a particular frequency location). Any of the methods described above for obtaining scanning patterns are used to obtain context models for signaling critical locations and / or their levels. A generalized template is used that estimates both the significant number of already coded transform coefficient levels in the spatial neighborhood of the coefficient to be coded and the significant number of transform coefficients in the collocated base layer signal at similar frequency positions. A generalized template is used that estimates both the significant number of levels of already coded transform coefficients in the spatial neighborhood of the coefficient to be coded and the significant levels of transform coefficients in the collocated base layer signal at similar frequency positions. The context modeled for the subgroup flag depends on the scan pattern and / or the particular transform size used.

[0310] Different context initialization table usage for the base layer and enhancement layer is used. The context model initialization for the enhancement layer is modified in the following way: Enhancement layers use separate sets of initialization values. Enhancement layers use separate sets of initialization values ​​for different operation modes (spatial / temporal or quality scalability). Enhancement layer context models that have counting parts in the base layer use the states of those counting parts as their initialization states. The algorithm for obtaining the initial context state is base layer QP and / or delta QP dependent.

[0311] Next, the possibility of encoding a suitable subsequent enhancement layer using base layer data is described. The following section describes a method for generating an enhancement layer prediction signal in a scalable video coding system. The method uses base layer decoded image sample information to infer values ​​of prediction parameters. The values ​​of the prediction parameters are not transmitted in the coded video bitstream, but are used to form the prediction signal for the enhancement layer. Thus, the overall bit rate required to encode the enhancement layer signal is reduced.

[0312] State-of-the-art hybrid video encoders decompose an original (source) image into blocks of different sizes, usually according to a hierarchical structure. For each block, the video signal is predicted from spatially neighboring blocks (intra prediction) or from previously temporally coded pictures (inter prediction). The difference between the prediction and the actual image is the transformation and quantization. The resulting prediction parameters and transform coefficients are entropy coded to form the coded video bitstream. A matching decoder follows the steps in reverse order... A scalable video coding bitstream consists of different layers: a base layer that provides the complete decodable video, and an enhancement layer that is additionally used for decoding. The enhancement layer can provide higher spatial resolution (spatial scalability), temporal resolution (temporal scalability), or quality (SNR scalability). In previous standards such as H.264 / AVC SVC, syntax elements such as motion vectors, reference picture indices or intra prediction modes are predicted directly from the corresponding syntax elements in the coded base layer. Within the enhancement layer, mechanisms exist to switch between using prediction signals, at the block level, derived from base layer syntax elements or predicted from other enhancement layer syntax elements or decoded enhancement layer samples.

[0313] In the following part, the base layer data is used to derive the enhancement layer parameters at the decoder side.

[0314] [Method 1: Derivation of motion parameter candidates] For a block (a) of the spatial or qualitative enhancement layer image, a corresponding block (b) of the base layer image is determined, which covers the same image region. The inter prediction signal for block (a) of the enhancement layer is formed using the following method: 1. Candidate motion compensation parameter sets are determined, for example, from temporally or spatially adjacent enhancement layer blocks or derivatives thereof. 2. Motion compensation is performed for each candidate motion compensation parameter set to form an inter-prediction signal within the enhancement layer. 3. The best set of motion compensation parameters is selected by minimizing the magnitude of the error between the predicted signal for the enhancement layer block (a) and the reconstructed signal for the base layer block (b). In spatial scalability, the base layer block (b) is spatially extracted using an interpolation filter.

[0315] The motion compensation parameter set includes a particular combination of motion compensation parameters.

[0316] The motion compensation parameters are the motion vector, the reference picture index, the choice between one and two predictions and other parameters.

[0317] In an alternative embodiment, a candidate motion compensation parameter set is used from the base layer block. Inter-prediction is also performed in the base layer (using a reference image from the base layer). To apply the error magnitude, the reconstructed signal of base layer block (b) is used directly without extraction. The selected optimal motion compensation parameter set is applied to the reference image of the enhancement layer to form the prediction signal of block (a). When motion vectors are applied in the spatial enhancement layer, they are scaled according to the resolution change. Both the encoder and decoder can perform the same prediction step to select the optimal motion compensation parameter set among the available candidates and create the same prediction signal, without these parameters being signaled in the coded video bitstream.

[0318] The choice of prediction method is signaled in the bitstream and coded using entropy coding. Within a hierarchical block subdivision structure, the coding method can be selected at any sublevel, or alternatively, only for a subset of the coding hierarchy. In an alternative embodiment, the encoder can send an improved motion parameter set prediction signal to the decoder. The improved signal includes differentially coded values ​​of the motion parameters. The improved signal is entropy coded.

[0319] In an alternative embodiment, the decoder generates a list of best candidates. The indices of the motion parameter sets used are signaled in the coded video bitstream. The indices are entropy coded. In an embodiment, the list is ordered by increasing error magnitude.

[0320] The embodiment uses an HEVC adaptive motion vector prediction (AMVP) candidate list to generate motion compensation parameter set candidates. Another embodiment uses the merge mode candidate list of HEVC to generate candidate motion compensation parameter sets.

[0321] [Method 2: Motion vector derivation] For a block (a) of the spatial or quality enhancement layer image, a corresponding block (b) of the base layer image covering the same image region is determined.

[0322] The inter prediction signal for block (a) of the enhancement layer is formed using the following method: 1. Motion vector prediction is selected. 2. Motion estimation for a defined set of search positions is performed on the reference image of the enhancement layer. 3. For each search position, the magnitude of the error is determined and the motion vector with the smallest error is selected. 4. A prediction for block (a) is formed using the selected motion vector.

[0323] In an alternative embodiment, the search is performed on the reconstructed base layer signal. For spatial scalability, the selected motion vector is scaled according to the spatial resolution change before generating the prediction signal in step 4.

[0324] The search locations can be at full resolution or sub-pel resolution, and the search can be performed in multiple steps, e.g., first determining the best full pel location, followed by another set of candidates based on the selected full pel location. For example, the search can terminate early when the magnitude of the error is below a defined threshold.

[0325] Both the encoder and the decoder can perform the same prediction step to select the best motion vector among the candidates and generate the same prediction signal, which is not signaled in the coded video bitstream.

[0326] The choice of prediction method is signaled in the bitstream and coded using entropy coding. Within a hierarchical block subdivision structure, the coding method may be selected within every sublevel, or only within a subset of alternate coding layers. In an alternative embodiment, the encoder can send an improved motion vector prediction signal to the decoder. The improved signal is entropy coded.

[0327] The embodiment uses the algorithm described in Method 1 to select the motion vector predictor.

[0328] Another embodiment uses the adaptive motion vector prediction (AMVP) method of HEVC to choose a motion vector predictor from temporally or spatially neighboring blocks of the enhancement layer.

[0329] [Method 3: Intra prediction mode] For each block (a) in the enhancement layer (n) image, a corresponding block (b) covering the same region in the reconstructed base layer (n-1) image is determined.

[0330] In a scalable video decoder, for each base layer block (b), an intra prediction signal is formed using an intra prediction mode (p) that is inferred by the following algorithm: 1) An intra prediction signal is generated for each available intra prediction mode following the rules for intra prediction of the enhancement layer, but using sample values ​​from the base layer. 2) Best prediction mode (p best ) is determined by minimizing the magnitude of the error (eg, the sum of absolute differences) between the intra prediction signal and the decoded base layer block (b). 3) The predictions (p best ) mode supports intra prediction rules for the enhancement layer and is used to generate a prediction signal for the enhancement layer block (a).

[0331] Both the encoder and decoder select the best prediction mode (p best The same steps can be performed to select the actual intra prediction mode (p best ) is not thus signaled in the coded video bitstream.

[0332] The choice of prediction method is signaled in the bitstream and coded using entropy coding. In a hierarchical block subdivision structure, this coding mode may be selected in every sublevel, or alternatively only for a subset of the coding hierarchy. An alternative embodiment uses samples from the enhancement layer in step 2) to generate the intra prediction signal. For spatial scalable enhancement layers, the base layer is extracted using an interpolation filter to apply an error magnitude.

[0333] An alternative embodiment is to divide the enhancement layer blocks into smaller block sizes (a i ) into multiple blocks (e.g., a 16x16 block (a) is divided into 16 4x4 blocks (a i The algorithm described above divides each sub-block (ai) into its corresponding base layer block (b i ) and applies to the block (a i After the prediction of block (a), residual coding is applied, and the result is i+1 ) is used to predict

[0334] An alternative embodiment is to use the predicted intra prediction mode (p best To determine (b) or (b i ) surrounding sample values. For example, the 4x4 block (a i ) is the corresponding 2 × 2 base layer block (b i ), then (b i ) are predicted by the intra prediction mode (p best ) is used to determine the 4x4 block (ci ) is used to form

[0335] In an alternative embodiment, the encoder can send an improved intra prediction direction signal to the decoder. For example, in video codecs such as HEVC, most intra prediction modes correspond to the angle at which boundary pixels are used to form the prediction signal. An offset to the optimal mode is added to the predicted intra prediction mode (p best ) The refinement mode is entropy coded.

[0336] Intra prediction modes are usually coded depending on their probability. In H.264 / AVC, the most likely mode is determined based on the modes used in the block's (spatial) neighborhood. In the HEVC list of most likely modes, a list of most likely modes is created. These most likely modes are selected using fewer symbols in the bitstream than the total number of modes requires. An alternative embodiment selects a predicted intra prediction mode (p) for block (a) (determined as described in the previous algorithm) as the most likely mode or as a member of a list of most likely modes. best ) to use.

[0337] Method 4: Intra prediction using boundary regions In a scalable video decoder to form an intra prediction signal for a block (a) of a scalable or quality enhancement layer (see Figure 45), lines of samples (b) from surrounding areas of the same layer are used to fill in the block area. These samples are taken from areas that have already been coded (usually, but not necessarily on the top and left borders).

[0338] The following alternative variations of selecting these pixels are used: a) If a pixel in the surrounding region has not yet been coded, the pixel value is not used to predict the current block. b) If a pixel in the surrounding region has not yet been coded, the pixel value is obtained (eg, by repetition) from neighboring pixels that have already been coded. c) If a pixel in the surrounding region has not yet been coded, the pixel value is obtained from the pixel in the corresponding region of the decoded base layer image.

[0339] To form an intra prediction of block (a), adjacent lines of pixels (b) obtained as described above are added to each line (a) of block (a). j ) is used as a template to fill in the

[0340] Block (a) line (a j ) are filled one by one along the x-axis. To achieve the best possible predicted signal, the columns of template samples (b) are aligned along the relation line (a j ) for the predicted signal (b' j ) along the y-axis to form

[0341] To find the best prediction within each line, the transition offset (o j ) is the resulting predicted signal (a j ) and the sample values ​​of the corresponding line in the base layer.

[0342] Suppose, (o j If ) is a non-integer value, the interpolation filter is j ) to integer sample positions in (b).

[0343] If spatial scalability is used, an interpolation filter is used to create a matching number of sample values ​​for the corresponding line of the base layer.

[0344] The filling direction (x-axis) can be horizontal (left to right), vertical (up to down), diagonal, or at any other angle. The samples used for template line (b) are the samples directly adjacent to the block along the x-axis. Template line (b) migrates along the y-axis, which forms a 90° angle with the x-axis.

[0345] To find the optimal direction of the x-axis, a fill-in intra prediction signal is generated for block (a). The angle with the smallest error magnitude between the prediction signal and the corresponding base layer block is selected. The number of possible angles is limited.

[0346] Both the encoder and decoder run the same algorithm to determine the best prediction angle and offset, and no explicit angle or offset information needs to be signaled in the bitstream. In an alternative embodiment, only the base layer image samples are taken at the offset (o i ) is used to determine

[0347] In an alternative embodiment, the predicted offset (o i ) is signaled in the bitstream. Entropy coding is used to encode the refinement offset value.

[0348] In an alternative embodiment, the refinement of the predicted direction (e.g., a difference value) is signaled in the bitstream, and entropy coding is used to encode the refinement direction value.

[0349] Suppose the line (b' j If the optimal offset (o j If the error magnitude for line (c i ) is a block line (a j ) value. If the optimal offset (oj If the error magnitude for the block line (a j ) is used to determine the value of

[0350] Method 5: Alternative prediction parameters Further prediction information is inferred in a similar manner to methods 1-3, eg for block partitioning within subblocks.

[0351] For a block (a) of the spatial or quality enhancement layer image, a corresponding block (b) of the base layer image is determined, which covers the same image region.

[0352] The prediction signal for block (a) of the enhancement layer is formed using the following method. 1) A predicted signal is generated for each possible value of the tested parameter. 2) Best prediction mode (p best ) is determined by minimizing the error measure (eg, the sum of absolute differences) between the prediction signal and the decoded base layer block (b). 3) The predictions (p best ) mode is used to generate the prediction signal for enhancement layer block (a).

[0353] Both the encoder and decoder can perform the same prediction step to select the best prediction mode among the possible candidates and generate the same prediction signal. The actual prediction mode is not signaled in the coded video bitstream.

[0354] The choice of prediction method is signaled in the bitstream and coded using entropy coding. In a hierarchical block subdivision structure, this coding method can be alternatively selected within every sublevel or for only a subset of the coding hierarchy.

[0355] The following description briefly summarizes some of the above embodiments.

[0356] Enhancement layer coding with multiple methods for generating an intra-prediction signal using reconstructed base layer samples Main embodiment: For coding blocks in the enhancement layer, multiple methods are provided for generating an intra prediction signal using reconstructed base layer samples, in addition to methods for generating a prediction signal based solely on reconstructed enhancement layer samples.

[0357] Sub-Example: · Multiplexing methods include the following methods: The (sampled / filtered) reconstructed base layer signal is directly used as the enhancement layer prediction signal. Multiplexing methods include the following: the (sampled / filtered) reconstructed base layer signal is combined with a spatial intra prediction signal, where the spatial intra prediction is obtained based on difference samples for neighboring blocks, which represent the difference between the reconstructed enhancement layer signal and the (sampled / filtered) reconstructed base layer signal (see embodiment A). Multiplexing methods include the following: A conventional spatial intra prediction signal (obtained using neighboring reconstructed enhancement layer samples) is combined with a (sampled / filtered) base layer residual signal (the inverse transform of the base layer transform coefficients or the difference between the base layer reconstruction and the base layer prediction) (see embodiment B). Multiplexing methods include the following: the (sampled / filtered) reconstructed base layer signal is combined into a spatial intra prediction signal, where the spatial intra prediction is obtained based on reconstructed enhancement layer samples of neighboring blocks, and the final prediction signal is obtained by weighting the spatial prediction signal and the base layer prediction signal in such a way that different frequency components use different weightings (see example C1). This can be achieved, for example, by one of the following: Filtering the base layer prediction signal with a low-pass filter and the spatial intra prediction signal with a high-pass filter and adding the resulting filtered signals (see embodiment C2). o Transform the base layer prediction signal and the enhancement layer prediction signal and overlap the resulting transform block, where different weighting coefficients are used for different frequency positions (see example C3). The resulting transform block is inverse transformed and used as the enhancement layer prediction signal. Alternatively, the resulting transform coefficients are added to the scaled transmitted transform coefficient levels and then inverse transformed to obtain a reconstructed block before the deblocking and inner-loop process (see example C4). For the method using the reconstructed base layer signal, the following versions can be used: it is fixed, or it is signaled at the sequence level, picture level, slice level, largest coding unit level, coding unit level, or it is generated depending on other coding parameters. ○ Reconstructed base layer samples before deblocking and inner-loop processing (as samples, adaptive offset filter or adaptive loop filter). ○ The reconstructed base layer samples after deblocking and before the inner-loop process (as samples, adaptive offset filter or adaptive loop filter). ○ Reconstructed base layer samples after deblocking and inner-loop processing (as samples, adaptive offset filter or adaptive loop filter), or reconstructed base layer samples during multiple inner-loop processing steps (see embodiment D). Multiple versions of the method using the (sampled / filtered) base layer signal are used. The sampled / filtered base layer signals employed for these versions differ in the interpolation filters used (including interpolation filters that filter integer sample positions). Alternatively, the sampled / filtered base layer signal for the second version is obtained by filtering the base layer signal sampled / filtered for the first version. The selection of one of the different versions is signaled at the sequence level, picture level, slice level, largest coding unit level, or coding unit level. It is inferred from the characteristics of the corresponding reconstructed base layer signal or the transmitted coding parameters (see embodiment E). Different filters are used to extract / filter the reconstructed base layer signal (see example E) and the base layer residual signal (see example F). For base layer blocks where the residual signal is zero, it is replaced by another signal derived from the base layer (e.g., a high-pass filtered version of the reconstructed base layer block) (see embodiment G). For modes using spatial intra prediction, unavailable neighboring samples in the enhancement layer (due to the specific coding order) are replaced by corresponding samples of the extracted / filtered base layer signal (see embodiment H). For modes that use spatial intra prediction, the coding of the intra prediction modes is changed. The list of most likely modes includes the intra prediction modes of the collocated base layer signal. In a specific version, enhancement layer images are decoded in a two-stage process. In the first stage, only blocks that use the base layer signal (without using neighboring blocks) for prediction, or the inter-prediction signal, are decoded and reconstructed. In the second stage, residual blocks that use neighboring samples for prediction are reconstructed. For the blocks reconstructed in the second stage, the spatial intra prediction concept is extended (see Example I). Depending on the availability of already reconstructed blocks, not only the upper and left neighboring samples of the current block, but also the lower and right neighboring samples are used for spatial intra prediction.

[0358] Enhancement layer coding with multiple methods for generating an inter-prediction signal using reconstructed base layer samples Main embodiment: For coding blocks in the enhancement layer, multiple methods for generating an intermediate prediction signal using reconstructed base layer samples are provided in addition to methods for generating a prediction signal based solely on reconstructed enhancement layer samples.

[0359] Sub-Example: Multiplexing methods include the following: A conventional inter-prediction signal (obtained by motion compensated interpolation of already reconstructed enhancement layer pictures) is combined with a (sampled / filtered) base layer residual signal (the inverse transform of the base layer transform coefficients or the difference between the base layer reconstruction and the base layer prediction). Multiplexing methods include the following: the (sampled / filtered) reconstructed base layer signal is combined with a motion compensated prediction signal, where the motion compensated prediction signal is obtained by a motion compensated difference image, which represents the difference between the reconstructed enhancement layer signal and the (sampled / filtered) reconstructed base layer signal relative to a reference image (see embodiment J). Multiplexing methods include: the (extracted / filtered) reconstructed base layer signal is combined into an inter prediction signal, where the inter prediction is obtained by motion compensated prediction using the reconstructed enhancement layer images, and the final prediction signal is obtained by weighting the inter prediction signal and the base layer prediction signal in such a way that different frequency components use different weightings (see embodiment C). This can be achieved, for example, by one of the following: Filtering the base layer prediction signal with a low pass filter, filtering the inter prediction signal with a high pass filter and adding the resulting filtered signals. Transform the base layer prediction signal and the inter prediction signal and overlap the resulting transform block, where different weighting coefficients are used for different frequency positions. The resulting transform block is either inverse transformed and used as the enhancement layer prediction signal to obtain a reconstructed block before the deblocking and inner-loop process, or the resulting transform coefficients are added to the scaled transmitted transform coefficient levels and then inverse transformed. For the method using the reconstructed base layer signal, the following versions can be used: it is fixed, or it is signaled at the sequence level, picture level, slice level, largest coding unit level, coding unit level, or it is made dependent on other coding parameters. ○ Reconstructed base layer samples before deblocking and inner-loop processing (as samples, adaptive offset filter or adaptive loop filter). ○ The reconstructed base layer samples after deblocking and before the inner-loop process (as samples, adaptive offset filter or adaptive loop filter). ○ Reconstructed base layer samples after deblocking and inner-loop processing (as samples, adaptive offset filter or adaptive loop filter), or reconstructed base layer samples during multiple inner-loop processing steps (see embodiment D). For base layer blocks where the residual signal is zero, it is replaced by another signal derived from the base layer (e.g., a high-pass filtered version of the reconstructed base layer block) (see embodiment G). Multiple versions of the method using the (sampled / filtered) base layer signal are used. The sampled / filtered base layer signals employed for these versions differ in the interpolation filters used (including interpolation filters that filter integer sample positions). Alternatively, the sampled / filtered base layer signal for the second version is obtained by filtering the base layer signal sampled / filtered for the first version. The selection of one of the different versions is signaled at the sequence level, picture level, slice level, largest coding unit level, or coding unit level. It is inferred from the characteristics of the corresponding reconstructed base layer signal or the transmitted coding parameters (see embodiment E). Different filters are used to extract / filter the reconstructed base layer signal (see example E) and the base layer residual signal (see example F). For motion compensated prediction of the difference image (the difference between the enhancement layer reconstruction and the extracted / filtered base layer signal) (see embodiment J), different interpolation filters are used than for the motion compensated prediction of the reconstructed image. For motion compensated prediction of the difference image (the difference between the enhancement layer reconstruction and the extracted / filtered base layer signal) (see embodiment J), the interpolation filter is selected based on the characteristics of the corresponding region in the difference image (or based on coding parameters or based on information transmitted in the bitstream).

[0360] [Enhancement layer motion parameter coding] Main embodiment: Use of multiple enhancement layer predictors and at least one predictor derived from the base layer for enhancement layer motion parameter coding.

[0361] Sub-Example: Adding the (scaled) base layer motion vector to the motion vector predictor list (see embodiment K). Use of a base layer block that covers a collocated sample at the center position of the current block (another possible derivation). ○ Scale motion vectors according to resolution ratio. Adding the motion data of the collocated base layer blocks to the merge candidate list (see embodiment K). Use of a base layer block that covers a collocated sample at the center position of the current block (another possible derivation). ○ Scale motion vectors according to resolution ratio. If "merge_flag" is equal to 1 in the base layer, then do not add. Reordering the merge candidate list based on base layer merge information (see Example L) If a collocated base layer block is merged with a particular candidate, the corresponding enhancement layer candidate is used as the first entry in the enhancement layer merge candidate list. Reordering of motion predictor candidate list based on base layer motion predictor information (see Example L) If a co-located base layer block uses a particular motion vector predictor, then the corresponding enhancement layer motion vector predictor is used as the first entry in the enhancement layer motion vector predictor candidate list. The derivation of the merge index (i.e., the candidates with which the current block is merged) is based on the base layer information among the collocated blocks (see embodiment M). As an example, if a base layer block is merged into a particular neighboring block, and it is signaled in the bitstream that an enhancement layer block is also merged, no merge index is transmitted, but instead the enhancement layer block is merged into the same neighboring block (but in the enhancement layer) as the collocated base layer block.

[0362] [Enhancement layer partitioning and motion parameter inference] Main embodiment: Inference of enhancement layer partition and motion parameters based on base layer partition and motion parameters (possibly requiring combining this embodiment with any of the sub-embodiments).

[0363] Sub-Example: Deriving motion parameters for NxM subblocks of the enhancement layer based on collocated base layer motion data. Clustering blocks with the same derived parameters (or parameters with small differences) into larger blocks. Determining prediction and coding units (see embodiment T). The motion parameters include the number of motion hypotheses, the reference index list, the motion vector, the motion vector predictor identifier, and the merge identifier. Signaling one of multiple methods for generating an enhancement layer prediction signal, including: Motion compensation using the obtained motion parameters and the reference image of the reconstructed enhancement layer. ○ Combining (a) the (extracted / filtered) base layer reconstruction for the current picture and (b) a motion compensation signal using the obtained motion parameters and a reference picture of the enhancement layer generated by subtracting the (extracted / filtered) base layer reconstruction from the reconstructed enhancement layer picture. ○ Combining (a) the (extracted / filtered) base layer residual (the difference between the reconstructed signal and the prediction, or the inverse transform of the coded transform coefficient values) for the current picture and (b) the motion compensation signal using the obtained motion parameters with the reconstructed enhancement layer reference picture. If a collocated block in the base layer is intra-coded, the corresponding enhancement layer MxN block (or CU) is also intra-coded, where an intra prediction signal is obtained using base layer information (see embodiment U). For example, o A (sampled / filtered) version of the corresponding base layer reconstruction is used as the intra prediction signal (see embodiment U). The intra prediction mode is derived based on the intra prediction mode used in the base layer, and this intra prediction mode is then used for spatial intra prediction in the enhancement layer. If the collocated base layer block for an MxN enhancement layer block (sub-block) is merged with (or has the same motion parameters as) a previously coded base layer block, then the MxN enhancement layer (sub-)block is also merged with the enhancement layer block corresponding to the base layer block used for merging in the base layer (i.e., the motion parameters are copied from the corresponding enhancement layer block) (see embodiment M).

[0364] [Transform coefficient level coding / context modeling] Main examples: coding transform coefficients using different scan patterns, modeling context for enhancement layers based on coding mode and / or base layer data, and different initialization for context modes.

[0365] Sub-Example: Introducing one or more additional scan patterns, e.g., horizontal and vertical scan patterns. Redefining subblocks for the additional scan patterns. Instead of 4x4 subblocks, e.g., 16x1 or 1x16 subblocks are used. Or, 8x2 or 8x2 subblocks are used. Additional scan patterns are introduced only for blocks larger than or equal to a certain size, e.g., 8x8 or 16x16 (see embodiment V). The selected scan pattern is signaled in the bitstream (if the coded block flag is equal to 1) (see embodiment N). A fixed context is used to signal the corresponding syntax element. Alternatively, the context derivation for the corresponding syntax element can depend on one of the following: The gradient of the collocated reconstructed base layer signal or the reconstructed base layer residual, or the detected edges in the base layer signal. Transform coefficient distribution among collocated base layer blocks. The selected scan is derived directly from the base layer signal (without any additional data being transmitted) based on the characteristics of the collocated base layer signal (see embodiment N). The gradient of the collocated reconstructed base layer signal or the reconstructed base layer residual, or the detected edges in the base layer signal. Transform coefficient distribution among collocated base layer blocks. The different scanning is realized in such a way that the transform coefficients are reordered after quantization at the encoder side and conventional coding is used. At the decoder side, the transform coefficients are decoded conventionally and reordered before scaling and inverse transform (or after scaling and before inverse transform). To code the significance flags (subgroup flags and / or significance flags for a single transform coefficient), the following modifications are used in the enhancement layer: o A separate context model is used for all or a subset of the coding modes that use base layer information. It is also possible to use different context models for different modes that have base layer information. o Context modeling can rely on data of collocated base layer blocks (eg, the number of significant transform coefficients for a particular frequency position) (see embodiment O). A generalized template is used that estimates both the number of significant transform coefficient levels already coded in the spatial neighborhood of the coefficient to be coded and the number of significant transform coefficients in the co-located base layer signal at the same frequency position (see embodiment O). To encode the last significant scan position, the following modifications are used in the enhancement layer: o Separate context models are used for all or a subset of coding modes that use base layer information. It is also possible to use different context models for different modes that have base layer information (see example P). Context modeling can depend on data in collocated base layer blocks (e.g., transform coefficient distribution in the base layer, base layer gradient information, last scan position in the collocated base layer block). o The last scan position is encoded as the difference relative to the last base layer scan position (see example S). · How to use different context initialization tables for base and enhancement layers.

[0366] Backward adaptive enhancement layer coding using base layer data Main example: Use of base layer data to derive enhancement layer coding parameters.

[0367] Sub-Example: Obtaining merge candidates based on the (potentially derived) base layer reconstruction. In the enhancement layer, only the use of merge is signaled. However, in practice, the candidate used to merge the current block is obtained based on the reconstructed base layer signal. Therefore, for every merge candidate, the error magnitude between the (potentially derived) base layer signal for the current enhancement layer block and the corresponding prediction signal (obtained using the motion parameters for the merge candidate) is evaluated for every merge candidate (or a subset thereof). The merge candidate associated with the smallest error magnitude is then selected. The error magnitude is also calculated in the base layer using the reconstructed base layer signal and the base layer reference image (see Example Q). Obtain merge candidates based on the (potentially extracted) base layer reconstruction. The motion vector difference is not coded, but is inferred based on the reconstructed base layer. Determine a motion vector predictor for the current block and evaluate a set of defined searches located around the motion vector predictor. For each search position, determine the error magnitude between the (potentially extracted) base layer signal for the current enhancement layer block and the replaced reference frame (the replacement is given by the search position). Select the search position / motion vector that results in the smallest error magnitude. The search is divided into several stages. For example, a full-pel search is performed first. Then, a half-pel search is performed around the full-pel vector. Then, a quarter-pel search is performed around the best full / half-pel vector. A search is also performed in the base layer using the reconstructed base layer signal and the base layer reference image. The found motion vector is then scaled according to the resolution change between the base layer and the enhancement layer (see Example Q). Obtain an intra prediction mode based on the (potentially extracted) base layer reconstruction. The intra prediction mode is inferred based on the reconstructed base layer, although it is not coded. For each possible intra prediction mode (or a subset thereof), determine the error magnitude between the (potentially extracted) base layer signal for the current enhancement layer block and the intra prediction signal (using the tested prediction node). Select the prediction mode that results in the smallest error magnitude. The error magnitude calculation is also performed in the base layer using the reconstructed base layer signal and the intra prediction signal in the base layer. Furthermore, the intra block can be decomposed into 4x4 blocks (or another block size). Then, a separate intra prediction mode is determined for each 4x4 block (see Example Q). The intra prediction signal is determined by the column alignment or column alignment of boundary samples with the reconstructed base layer signal. To obtain the shift between the neighboring samples and the current line / column, the error magnitude is calculated between the shifted line / column of neighboring samples and the reconstructed base layer signal. The shift resulting in the smallest error magnitude is then selected. As neighboring samples, (extracted) base layer samples or enhancement layer samples are used. Alternatively, the error magnitude can be calculated directly in the base layer (see embodiment W). · Use backward adaptive techniques for the derivation of additional coding parameters such as block partitioning.

[0368] A further brief summary of the embodiment is presented below. In particular, the embodiment is described.

[0369] A1) A scalable video decoder is Reconstructing (80) a base layer signal (200a, 200b, 200c) from the coded data stream (6); reconstructing (60) the enhancement layer signal (360); Reconstruction (60) is subjecting (220) the reconstructed base layer signal (200a, 200b, 200c) to a resolution or quality refinement to obtain an inter-layer predicted signal (380); Calculating (260) a difference signal between an already reconstructed portion (400a or 400b) of the enhancement layer signal and an inter-layer predicted signal (380); spatially predicting (260) a difference signal from a second portion (460) of the difference signal that is spatially adjacent to a first portion (440, for example in FIG. 46) juxtaposed to a portion of the enhancement layer signal (360) currently to be reconstructed and that belongs to an already reconstructed portion of the enhancement layer signal (360), to obtain a spatial intra-predicted signal; combining (260) the inter-layer prediction signal (380) and the spatial intra prediction signal to obtain an enhancement layer prediction signal (420); The method includes predictively reconstructing (320, 580, 340, 300, 280) an enhancement layer signal (360) using the enhancement layer prediction signal (420). According to embodiment A1, the base layer signal is reconstructed by the base layer decoding stage 80 from the coded data stream 6 or substream 6a, respectively, for example in the above-mentioned block-based prediction method with transform decoding as far as the base layer residual signal 640 / 480 is concerned, although other alternative reconstructions are also possible. As far as the reconstruction of the enhancement layer signal 360 by the enhancement layer decoding stage 60 is concerned, the improvement in resolution or quality that the reconstructed base layer signal 200a, 200b or 200c undergoes means, for example, extraction in the case of a resolution improvement, or copying in the case of a quality improvement, or tone mapping from n bits to m bits (m>n) in the case of a bit depth improvement. The calculation of the difference signal is done pixel-wise, i.e. the juxtaposed pixels of the enhancement layer signal on the one hand and the prediction signal 380 on the other hand are subtracted from each other, and this is done pixel-by-pixel. The spatial prediction of the difference signal is done by some method, such as transmitting intra-prediction parameters, such as the intra-prediction direction, in the coded data stream 6 or in the sub-stream 6b, and copying / interpolating already reconstructed pixels that border the portion of the enhancement layer signal 360 currently to be reconstructed along this intra-prediction direction in the current portion of the enhancement layer signal. Combining means addition, weighted sum or more elaborate combination, such as combination that weights contributions differently in the frequency domain. Predictive reconstruction of the enhancement layer signal 360 using the enhancement layer prediction signal 420 means entropy decoding and inverse transform of the enhancement layer residual signal 540 and combination 340 of the enhancement layer prediction signal 420 and the latter 540, as shown in the figure.

[0370] B1) Scalable video decoder: Decoding (100) a base layer residual signal (480) from the coded data stream (6); reconstructing (60) the enhancement layer signal (360); Reconstruction (60) is subjecting the reconstructed base layer residual signal (480) to a resolution or quality refinement (220) to obtain an inter-layer residual prediction signal (380); spatially predicting (260) a portion of the enhancement layer signal (360) currently to be reconstructed from an already reconstructed portion of the enhancement layer signal (360) to obtain an enhancement layer intra-predicted signal; combining (260) the inter-layer residual prediction signal and the enhancement layer intra prediction signal to obtain an enhancement layer prediction signal (420); and predictively reconstructing (340) an enhancement layer signal (360) using the enhancement layer prediction signal (420). Decoding of the base layer residual signal from the encoded data stream is performed using entropy decoding and an inverse transform, as shown in the figure. Additionally, the scalable video decoder optionally performs reconstruction of the base layer signal itself by predictively decoding a base layer prediction signal 660 and combining this signal with the base layer residual signal 480. As just mentioned, this is merely optional. As far as the reconstruction of the enhancement layer signal is concerned, the resolution or quality improvement is performed as indicated above for embodiment A). Also, as far as the spatial prediction of parts of the enhancement layer signal is concerned, this spatial prediction is performed as exemplarily outlined in A) for the different signals. As far as the combination and predictive reconstruction are concerned, the same remarks are valid. However, it is noted that the base layer residual signal 480 in embodiment B) is not restricted to be equal to the explicitly signaled version of the base layer residual signal 480. Rather, it is possible for the scalable video decoder to subtract any reconstructed base layer signal version 200 with the base layer prediction signal 660, resulting in a base layer residual signal 480 that deviates from the explicitly signaled one by deviations stemming from a filter function such as filter 120 or 140. The latter condition is also valid for other embodiments in which the base layer residual signal is involved in inter-layer prediction.

[0371] C1) Scalable video decoder: Reconstructing (80) a base layer signal (200a, 200b, 200c) from the coded data stream (6); reconstructing (60) the enhancement layer signal (360); Reconstruction (60) is subjecting the reconstructed base layer signal (200) to a resolution or quality refinement (220) to obtain an inter-layer predicted signal (380); spatially or temporally predicting (260) a portion of the enhancement layer signal (360) to be currently reconstructed from an already reconstructed portion of the enhancement layer signal (360) (400a, b in the "spatial" case; 400a, b, c in the "temporal" case) to obtain an enhancement layer intra-predicted signal; forming (260) a weighted average of the inter-layer prediction signal and the enhancement layer intra prediction signal (380) at the portion currently to be reconstructed to obtain the enhancement layer prediction signal (420), such that the weights by which the inter-layer prediction signal and the enhancement layer intra prediction signal (380) contribute to the enhancement layer prediction signal (420) vary across different spatial frequency components; and predictively reconstructing (340) an enhancement layer signal (360) using the enhancement layer prediction signal (420).

[0372] C2) Here, forming (260) the weighted average includes filtering (260) the inter-layer predicted signal (380) with a low-pass filter and filtering (260) the enhancement layer intra-predicted signal with a high-pass filter to obtain a filtered signal for the portion currently to be reconstructed, and summing the resulting filtered signals. C3) where forming (260) the weighted average includes transforming (260) the inter-layer prediction signal and the enhancement layer intra-prediction signal to obtain transform coefficients in the part currently to be reconstructed, convolving (260) the obtained transform coefficients using different weighting factors for different spatial frequency components to obtain convolved transform coefficients, and inverse transforming the convolved transform coefficients to obtain the enhancement layer prediction signal. C4) where, using the enhancement layer prediction signal (420), a predictive reconstruction (320, 340) of the enhancement layer signal derives (320) transform coefficient levels for the enhancement layer signal from the coded data stream (6) and , and performing a summation (340) of the transform coefficient levels and the convolved transform coefficients to obtain a transformed version of the enhancement layer signal (360), and subjecting the transformed version of the enhancement layer signal to an inverse transform (i.e., the inverse transform T in the figure) to obtain an enhancement layer signal (360). -1 is placed downstream of adder 340, at least for that encoding mode. As far as the reconstruction of the base layer signal is concerned, reference is made to the figures in general and to the foregoing, as to examples A) and B). The same applies to the resolution or quality improvements mentioned in C) as well as spatial prediction. The temporal prediction referred to in C) involves a prediction provider 160 obtaining motion prediction parameters from the coded data stream 6 and the substream 6a, respectively. The motion parameters include motion vectors, reference frame indices, or a combination of motion subdivision information and motion vectors for each subblock of the currently reconstructed portion. As previously mentioned, the formation of the weighted average can be completed in either the spatial domain or the transform domain. Thus, the summation in summer 340 can be performed in either the spatial domain or the transform domain. In the latter case, inverse transformer 580 applies an inverse transform to the weighted average.

[0373] D1) Scalable video decoder: Reconstructing (80) a base layer signal (200a, 200b, 200c) from the coded data stream (6); reconstructing (60) the enhancement layer signal (380); Reconstruction (60) is subjecting the reconstructed base layer signal to a resolution or quality refinement (220) to obtain an inter-layer predicted signal (380); predictively reconstructing (320, 340) an enhancement layer signal (360) using the inter-layer predicted signal (380); Here, the reconstruction (60) of the enhancement layer signal is performed such that the inter-layer prediction signal (380) evolves and is configured to control via side information in the coded bitstream from different ones of none (200a), one, or all (200b, 200c) of the deblocking and intra-loop filtering (140) for different portions of the video scaled and represented by the base layer signal and the enhancement layer signal, respectively.

[0374] As far as the reconstruction of the base layer signal is concerned, reference is made to the figures in general and to the above, as to examples A) and B). The same applies to improvements in resolution or quality. The prediction reconfiguration referred to in D) involves the prediction provider 160, as previously mentioned. spatially or temporally predicting (260) a portion of the enhancement layer signal (360) currently to be reconstructed from an already reconstructed portion of the enhancement layer signal (380) to obtain an enhancement layer intra-predicted signal; It involves combining (260) an inter-layer prediction signal (380) and an enhancement layer intra-prediction signal to obtain an enhancement layer prediction signal (420). The fact that the inter-layer prediction signal (380) evolves is controlled via side information in the coded bitstream (360) from different ones of none, one, or all of the deblocking filters (200a, 200b, 200c), and the inner loop filter (140) is applied to different parts of the video as described below. Naturally, the base layer substream 6a itself (optionally) signals the use of different means to arrive at the final base layer signal 600 (deblocking only, or inner-loop filtering only, or both deblocking and inner-loop filtering), so as to bypass all filters 120, 140. Even the filter transfer functions are signaled and changed by side information in the base layer substream 6a. The dimensions defining the different portions at which these changes are made may be defined by the aforementioned coding unit, prediction block, or any other dimension. Consequently, the scalable video decoder (encoding stage 80) would apply these changes if only the base layer signal were to be reconstructed. However, independently therefrom (i.e., independently of the just-mentioned side information in base layer signal 6a), substream 6b includes side information signaling new changes in which filtering combinations are used to obtain the base layer signal. There, all filters 120, 140 are bypassed and used in the predictive reconstruction of the enhancement signal (either using only deblocking, or using only inner-loop filtering, or using both deblocking and inner-loop filtering), i.e. even the filter transfer functions are signaled and changed by side information in the substream 6b. The dimensions defining the different parts where these changes are made are defined by the aforementioned coding units, or prediction blocks, or any other dimensions, and this signaling differs from the dimensions used in the base layer signal 6a.

[0375] E1) Scalable Video Decoder Reconst...

Claims

1. Decoding base layer motion parameters from the encoded data stream (6); Reconstructing (60) the enhancement layer signal 1. A scalable video decoder configured to: The reconstruction (60) of the enhancement layer signal comprises: collecting, for a block (28) of a frame of the enhancement layer signal, a set (512) of motion parameter candidates (514) from adjacent already reconstructed blocks (92, 94) of the frame; collecting a set (522) of one or more base layer motion parameters (524) for a block (108) of the base layer signal collocated with the block (28) of the frame of the enhancement layer signal; adding (526) the one or more base layer motion parameters or the set of scaled versions of base layer motion parameters (522) to the set of enhancement layer motion parameter candidates (512) to obtain a set of enhanced motion parameter candidates (528) for the enhancement layer; selecting (534) one of the motion parameter candidates of the extended set of motion parameter candidates; and predicting (32) the enhancement layer signal by motion compensated prediction using a selected one of the motion parameter candidates of the set of extended motion parameter candidates.

2. 2. The scalable video decoder of claim 1, configured to scale the base layer motion parameters according to a spatial resolution ratio between the base layer signal and the enhancement layer signal to obtain scaled versions of the base layer motion parameters.

3. 3. A scalable video decoder according to claim 1 or claim 2, configured to check whether the base layer motion parameters are coded in the coded data stream using merging, and to suppress the addition if the base layer motion parameters are coded in the coded data stream using merging.

4. 4. A scalable video decoder according to claim 1, configured to perform the selection using an index into the set of extended motion parameter candidates signaled in the encoded data stream.

5. 5. A scalable video decoder according to claim 1, wherein the decoder is configured to assemble the set of enhancement layer motion parameter candidates for the block of the frame of the enhancement layer signal by copying and / or combining motion parameters of temporally and / or spatially adjacent already reconstructed blocks (92, 94) of the enhancement layer from the adjacent already reconstructed blocks.

6. 6. A scalable video decoder as claimed in claim 1, configured to identify the blocks of the base layer signal juxtaposed with the blocks of the frames of the enhancement layer signal by checking which blocks of temporally corresponding frames of the base layer signal overlap at positions that spatially correspond to predetermined sample positions of the blocks of the frames of the enhancement signal.

7. 7. A scalable video decoder according to claim 1, configured to order the set of extended motion parameter candidates in dependence on the base layer motion parameters.

8. Decoding (100) base layer motion parameters from the encoded data stream (6); Reconstructing (60) the enhancement layer signal 1. A scalable video decoder configured to: The reconstruction (60) of the enhancement layer signal comprises: ordering a motion parameter candidate list for the enhancement layer signal in dependence on the base layer motion parameters; selecting an enhancement layer motion parameter from the ordered motion parameter candidate list for the enhancement layer signal; predicting the enhancement layer signal with motion compensation parameters using the selected motion parameters; a scalable video decoder comprising:

9. 9. The scalable video decoder of claim 8, configured to perform the selection under control via an explicitly signaled index syntax element in the encoded data stream.

10. 10. A scalable video decoder as claimed in claim 8 or claim 9, characterized in that the decoder is configured to perform the ordering, for each enhancement layer motion parameter candidate in the motion parameter candidate list, according to a magnitude measuring a difference between the respective enhancement layer motion parameter candidate and a base layer motion parameter of the base layer signal that is associated with a block of the base layer signal that is co-located with the current block of the enhancement layer signal, among the base layer motion parameters of the base layer signal.

11. 11. The scalable video decoder of claim 10, configured to scale the base layer motion parameters according to a spatial resolution ratio between the base layer signal and the enhancement layer signal, with reference to the difference from the scaled base layer motion parameters.

12. 11. A scalable video decoder as claimed in claim 8, wherein the decoder is configured to perform, for each set of spatially and / or temporally adjacent blocks of the base layer signal that are spatially and / or temporally adjacent to a block of the base layer signal that is co-located with the current block of the enhancement layer signal, the ordering according to a measure of a difference between the base layer motion parameters of the base layer signal related to the block of the base layer signal that is co-located with the current block of the enhancement layer signal and the base layer motion parameters of each of the spatially and / or temporally adjacent blocks.

13. forming the motion parameter candidate list for the enhancement layer signal based on enhancement layer motion parameters of spatially and / or temporally neighboring blocks of the enhancement layer signal that are spatially and / or temporally neighboring to the current block of the enhancement layer signal; and for the current block of the enhancement layer, identifying the spatially and / or temporally neighboring blocks of the enhancement layer signal by applying a predetermined neighboring rule, and for the block of the base layer signal collocated with the current block of the enhancement layer signal, identifying the spatially and / or temporally neighboring blocks (603a, 603b, 605a, 605b) of the base layer signal by applying the predetermined neighboring rule; or identifying spatially and / or temporally adjacent blocks (597) of the base layer signal such that each of the spatially and / or temporally adjacent blocks (597) of the base layer signal is juxtaposed with each of the spatially and / or temporally adjacent blocks of the enhancement layer signal; performing an ordering of the motion parameter candidate list for the enhancement layer signal depending on a base layer ordering between the base layer motion parameters of the spatially and / or temporally adjacent blocks defined by a magnitude of a difference between the base layer motion parameters of the base layer signal related to a block of the base layer signal juxtaposed to the current block of the enhancement layer signal and the base layer motion parameters of the spatially and / or temporally adjacent blocks; 13. A scalable video decoder according to claim 12, configured to:

14. decoding (100) base layer motion parameters (520) from the coded data stream (6) using an index into a motion parameter candidate list for the base layer signal (200); Reconstructing (60) the enhancement layer signal (360) 1. A scalable video decoder configured to: The reconstruction (60) of the enhancement layer signal (360) comprises: determining an index into a motion parameter candidate list for the enhancement layer signal depending on the index into the motion parameter candidate list for the base layer signal; determining enhancement layer motion parameters using the index into the motion parameter candidate list for the enhancement layer signal; and predicting the enhancement layer signal by motion compensated prediction using the determined enhancement layer motion parameters; 2. A scalable video decoder comprising:

15. checking a flag signaling whether motion parameters for a block of the base layer signal collocated with a current block of the enhancement layer signal are signaled in the base layer substream with or without merging; If the motion parameters for a block of the base layer signal collocated with the current block of the enhancement layer signal are coded using merging, determine the index into the motion parameter candidate list for the enhancement layer signal such that the block of the enhancement layer signal associated with the enhancement layer motion parameter determined using the index into the motion parameter candidate list for the enhancement layer signal is collocated with the block of the base layer signal associated with a base layer motion parameter in a list of base layer motion parameters composed of motion parameters of neighboring blocks adjacent to the block of the base layer signal collocated with the current block of the enhancement layer signal, indexed by an index into a list of base layer motion parameters obtained from the coded data stream.

15. A scalable video decoder according to claim 14, configured to:

16. identifying spatially and / or temporally neighboring blocks of the enhancement layer signal by applying predetermined neighboring rules to a current block of the enhancement layer signal, and forming the motion parameter candidate list for the enhancement layer signal based on enhancement layer motion parameters of the spatially and / or temporally neighboring blocks of the enhancement layer signal that are spatially and / or temporally neighboring to the current block of the enhancement layer signal; identifying spatially and / or temporally neighboring blocks of the base layer signal by applying the predetermined neighboring rules to blocks of the base layer signal collocated with the current block of the enhancement layer signal when decoding the base layer motion parameters from the encoded data stream; forming a motion parameter candidate list for the base layer signal from the identified base layer motion parameters of the spatially and / or temporally adjacent blocks, and reconstructing the base layer signal at the block of the base layer signal collocated with the current block of the enhancement layer signal by indexing into the motion parameter candidate list for the base layer signal using a base layer index; determining an index into a motion parameter candidate list for the enhancement layer signal to be equal to the base layer index; 15. A scalable video decoder according to claim 14, configured to:

17. 17. A scalable video decoder according to claim 1, wherein the motion parameter candidates of the set of extended motion parameter candidates relate to a plurality of motion vectors or a set of motion parameters comprising a motion hypothesis, a reference index, partition information and / or the number of one motion vector.

18. decoding base layer motion parameters from the encoded data stream (6); Reconstructing (60) the enhancement layer signal; 1. A scalable video decoding method comprising: The step of reconstructing (60) the enhancement layer signal comprises: collecting, for a block of a frame of the enhancement layer signal, a set of motion parameter candidates from adjacent already reconstructed blocks of the frame; collecting one or more sets of base layer motion parameters for a block of the base layer signal collocated with the block of the frame of the enhancement layer signal; adding one or more base layer motion parameters or a set of scaled versions of base layer motion parameters to the set of enhancement layer motion parameter candidates to obtain a set of enhanced motion parameter candidates for the enhancement layer; selecting one of the motion parameter candidates of the extended set of motion parameter candidates; predicting the enhancement layer signal by motion compensated prediction using the selected one of the motion parameter candidates of the set of extended motion parameter candidates; 2. A scalable video decoding method comprising:

19. Decoding (100) base layer motion parameters from the encoded data stream (6); Reconstructing (60) the enhancement layer signal; 1. A scalable video decoding method comprising: The step of reconstructing (60) the enhancement layer signal comprises: ordering a motion parameter candidate list for the enhancement layer signal in dependence on the base layer motion parameters; selecting an enhancement layer motion parameter from the motion parameter candidate list for the ordered enhancement layer signal; predicting the enhancement layer signal with motion compensation parameters using the selected enhancement layer motion parameters; 2. A scalable video decoding method comprising:

20. decoding (100) base layer motion parameters (520) from the coded data stream (6) using an index into a motion parameter candidate list for the base layer signal (200); Reconstructing (60) an enhancement layer signal (360); 1. A scalable video decoding method comprising: The step of reconstructing (60) the enhancement layer signal comprises: determining an index into a motion parameter candidate list for the enhancement layer signal depending on the index into the motion parameter candidate list for the base layer signal; determining enhancement layer motion parameters using the index into the motion parameter candidate list for the enhancement layer signal; predicting the enhancement layer signal by motion compensated prediction using the determined enhancement layer motion parameters; 2. A scalable video decoding method comprising:

21. encoding the base layer motion parameters into the encoded data stream (6); encoding the enhancement layer signal; 1. A scalable video encoder configured to: The encoding of the enhancement layer signal comprises: collecting, for a block of a frame of the enhancement layer signal, a set of motion parameter candidates from adjacent already reconstructed blocks of the frame; collecting one or more sets of base layer motion parameters for a block of the base layer signal collocated with the block of the frame of the enhancement layer signal; adding one or more base layer motion parameters or a set of scaled versions of base layer motion parameters to the set of enhancement layer motion parameter candidates to obtain a set of extended motion parameter candidates for the enhancement layer; selecting one of the motion parameter candidates of the extended set of motion parameter candidates; predicting the enhancement layer signal by motion compensated prediction using the selected one of the motion parameter candidates of the set of extended motion parameter candidates; a scalable video encoder configured to include:

22. encoding the base layer motion parameters into the encoded data stream (6); encoding the enhancement layer signal; 1. A scalable video encoder configured to: The encoding of the enhancement layer signal comprises: ordering a motion parameter candidate list for the enhancement layer signal in dependence on the base layer motion parameters; selecting an enhancement layer motion parameter from the ordered motion parameter candidate list for the enhancement layer signal; predicting the enhancement layer signal with motion compensation parameters using the selected enhancement layer motion parameters; 1. A scalable video encoder comprising:

23. encoding (100) base layer motion parameters (520) into the coded data stream (6) using an index into a motion parameter candidate list for the base layer signal (200); encoding (60) an enhancement layer signal (360); 1. A scalable video encoder configured to: The encoding (60) of the enhancement layer signal (360) comprises: determining an index into a motion parameter candidate list for the enhancement layer signal depending on the index into the motion parameter candidate list for the base layer signal; determining enhancement layer motion parameters using the index into the motion parameter candidate list for the enhancement layer signal; and predicting the enhancement layer signal by motion compensated prediction using the determined enhancement layer motion parameters; 1. A scalable video encoder comprising:

24. encoding base layer motion parameters into an encoded data stream (6); encoding an enhancement layer signal; 1. A scalable video coding method, comprising: The step of encoding the enhancement layer signal comprises: collecting, for a block of a frame of the enhancement layer signal, a set of motion parameter candidates from adjacent already reconstructed blocks of the frame; collecting one or more sets of base layer motion parameters for a block of the base layer signal collocated with the block of the frame of the enhancement layer signal; adding one or more base layer motion parameters or a set of scaled versions of base layer motion parameters to the set of enhancement layer motion parameter candidates to obtain a set of enhanced motion parameter candidates for the enhancement layer; selecting one of the motion parameter candidates of the extended set of motion parameter candidates; predicting the enhancement layer signal by motion compensated prediction using the selected one of the motion parameter candidates of the set of extended motion parameter candidates; 1. A scalable video coding method comprising:

25. encoding base layer motion parameters into an encoded data stream (6); encoding an enhancement layer signal; 1. A scalable video coding method, comprising: The step of encoding the enhancement layer signal comprises: ordering a motion parameter candidate list for the enhancement layer signal in dependence on the base layer motion parameters; selecting an enhancement layer motion parameter from the ordered motion parameter candidate list for the enhancement layer signal; predicting the enhancement layer signal with motion compensation parameters using the selected enhancement layer motion parameters; 1. A scalable video coding method comprising:

26. encoding (100) base layer motion parameters (520) into an encoded data stream (6) using an index into a motion parameter candidate list for the base layer signal (200); encoding (60) an enhancement layer signal (360); 1. A scalable video coding method, comprising: The step of encoding (60) the enhancement layer signal (360) comprises: determining an index into a motion parameter candidate list for the enhancement layer signal depending on the index into the motion parameter candidate list for the base layer signal; determining enhancement layer motion parameters using the index into the motion parameter candidate list for the enhancement layer signal; predicting the enhancement layer signal by motion compensated prediction using the determined enhancement layer motion parameters; 1. A scalable video coding method comprising:

27. A computer program having a program code that, when run on a computer, causes the computer to perform the scalable video decoding method of any one of claims 17 to 20 or the scalable video encoding method of any one of claims 24 to 26.

Images