Sample region merging

Abstract

To provide an encoder, a method, a computer program, and a data stream that specify a simply connected region in a plurality of simply connected regions in a predetermined relative positional relationship, relative to a predetermined simply connected region.SOLUTION: In a decoder, a sub-division unit 700 performs spatial sub-division of the arrangement of samples representing spatial sampling of a two-dimensional information signal into a plurality of non-overlapping simply connected regions of different sizes by performing recursive multi-partitioning depending on a subset of syntax elements contained in a data stream. A combination unit 702 performs combination of spatially neighboring simply connected regions, of the plurality of simply connected regions depending on a second subset of syntax elements within the data stream separated from the first subset, to obtain an intermediate sub-division of the arrangement of samples into disjoint sets of simply connected regions, the union of which is the plurality of simply connected regions. A reconstruction unit 704 reconstructs the arrangement of samples from the data stream by using the intermediate sub-division.SELECTED DRAWING: Figure 20

Description

[Technical field]

[0001] The present invention relates to a coding scheme for two-dimensionally sampled information signals, such as moving or still images. [Background technology]

[0002] Typically, in image and video coding, an image, or a set of certain sample arrays within an image, is decomposed into blocks and associated with certain coding parameters. Typically, an image is composed of several sample arrays. In addition, an image may be associated with additional auxiliary sample arrays that specify, for example, transparency information, depth maps, etc. The sample arrays of an image (including the auxiliary sample arrays) can be grouped into one or more so-called plane groups, each plane group being composed of one or more sample arrays. A plane group of an image can be coded independently or, if the image is associated with several plane groups, can be coded by prediction from other plane groups of the same image. Typically, each plane group is decomposed into blocks. The blocks (or the corresponding blocks of sample arrays) are predicted either by inter-picture prediction or by intra-picture prediction. The blocks can be of various sizes and are either square or rectangular. The partitioning of an image into blocks can be fixed by the syntax or (at least in part) can be signaled in the bitstream. Often syntax elements are transmitted that signal the subdivision of blocks of a given size. These syntax elements allow to specify whether and how a block is to be subdivided into smaller blocks and associated with coding parameters for prediction purposes etc. For every sample of a block (or for the corresponding block of the sample array), the decoding of the associated coding parameters is specified in a certain way. For example, all samples in the block are predicted using the same set of prediction parameters, such as reference indexes (identifying a reference image in the set of previously coded images), motion parameters (specifying the amount of motion of the block between the reference image and the current image), parameters for specifying the interpolation filter, intra prediction mode etc. The motion parameters can be represented by a displacement vector with horizontal and vertical components or by higher-order motion parameters, such as affine motion parameters consisting of six components. It is also possible to associate more than one set of a particular prediction parameters (such as reference indexes and motion parameters) to a single block.Then, for each set of these specific prediction parameters, a single intermediate prediction signal for the block (or the corresponding block of the sample array) is generated, and the final prediction signal is formed by a combination, including superposition, of the intermediate prediction signals. The corresponding weighting parameters, possibly with a constant offset (added to the weighted sum), can be fixed for either the image, the reference image or the set of reference images, or can be included in the set of prediction parameters for the corresponding block. The difference between the original block (or the corresponding block of the sample array) and the prediction signal, also called the residual signal, is usually transformed and quantized. Often, a two-dimensional transform is applied to the residual signal (or the corresponding sample array of the residual block). For transform coding, the block (or the corresponding block of the sample array) for which a specific set of prediction parameters is used can be further divided before applying the transform. The transform block can be the same size as the block used for prediction, or smaller. It is also possible that the transform block contains several blocks used for prediction. Each transform block can be of various sizes, resulting in a square or rectangular block. After transformation, the resulting transform coefficients are quantized to obtain the so-called transform coefficient levels. The transform coefficient levels and the prediction parameters and, if present, the subdivision information are entropy coded. Summary of the Invention [Problem to be solved by the invention]

[0003] In image and video coding standards, the possibilities for subdivision of an image (or a group of planes) into blocks provided by the syntax are very limited. Usually, it is only possible to specify whether and (possibly how) a block of a given size can be subdivided into smaller blocks. For example, in H.264 the maximum block size is 16x16. A 16x16 block is also called a macroblock, and each image is partitioned into macroblocks in a first step. For each 16x16 macroblock, it can be signaled whether it is coded as a 16x16 block, two 16x8 blocks, two 8x16 blocks, or four 8x8 blocks. If a 16x16 block is subdivided into four 8x8 blocks, each of these 8x8 blocks can be coded as either one 8x8 block, two 8x4 blocks, two 4x8 blocks, or four 4x4 blocks. In the current standards for image and video coding, the reduced possibilities for specifying the partitioning into blocks have the advantage of keeping the rate of the ancillary information for signaling the subdivision information small, but the disadvantage, as explained below, of the potentially large bit rate required for the transmission of the prediction parameters of the blocks. The rate of the ancillary information for signaling the prediction information is usually a significant part of the overall bit rate of the block. Therefore, if this ancillary information is reduced, the coding efficiency can be improved, which can be achieved, for example, by using a larger block size. Real images and images of video sequences consist of objects of arbitrary shapes with specific properties. For example, such objects or object parts are characterized by a specific texture or a specific motion. Usually, the same set of prediction parameters is applied to such objects or object parts. However, the boundaries of the objects usually do not coincide with the boundaries of the possible blocks of a larger prediction block (for example, a 16x16 macroblock in H.264). Usually, the encoder makes the subdivision decision (among a limited number of possibilities) such that a certain amount of rate / distortion loss is minimized.This can result in a large number of small blocks for an object of any shape, each of which is associated with a set of prediction parameters that need to be transmitted, and the associated information rate can become a significant fraction of the overall bit rate, but since some of the small blocks still represent the same object or some regions of the object, the prediction parameters of the resulting large number of blocks will be identical or very similar.

[0004] That is, subdivision or tiling of an image into smaller parts or tiles or blocks significantly impacts the coding efficiency and coding complexity. As outlined above, the more subdivision of an image into smaller blocks allows for more spatial detail in the coding parameter settings, and the better these coding parameters adapt to the image / video material. On the other hand, the more granular the coding parameter settings, the more burdensome the amount of accompanying information required to inform the decoder of the required settings. It should also be noted that allowing the encoder to (further) spatially sub-divide an image / video into blocks significantly increases the amount of possible coding parameter settings, and as a result, in general, makes it more difficult to find the best rate / distortion compromise for the coding parameter settings. [Means for solving the problem]

[0005] The present invention provides a coding scheme for coding an array of information samples representing a spatially sampled two-dimensional information signal, such as, but not limited to, moving image or still image, which allows for an improved trade-off between encoding complexity and achievable rate / distortion ratio and / or allows for improved rate / distortion ratios to be achieved.

[0006] This object is achieved by a decoder, an encoder, a method, a computer program and a data stream according to the independent claims.

[0007] In an embodiment, a desired combination or grouping of simply connected regions resulting from a subdivision of the sequence of information samples is coded to reduce the amount of data. For this purpose, a predetermined relative positional relationship is defined for the simply connected regions. This allows for a given simply connected region to identify a simply connected region among the plurality of simply connected regions that is in a predetermined relative positional relationship with respect to the given simply connected region. That is, if the number is zero, no merge indicator for the given simply connected region may be present in the data stream. Furthermore, if the number of simply connected regions in a predetermined relative positional relationship with respect to the given simply connected region is one, the coding parameters of the simply connected region may be employed or used for predicting the coding parameters for the given simply connected region, and no further syntax elements are required. Otherwise, that is, if the number of simply connected regions in a predetermined relative positional relationship with respect to the given simply connected region is greater than one, the introduction of further syntax elements may be suppressed even if the coding parameters associated with these identified simply connected regions are identical to each other.

[0008] In an embodiment, if the coding parameters of adjacent simply connected regions are not equal to each other, a proper subset of the number of simply connected regions in a given relative position to a given simply connected region may be identified by a reference adjacent identifier, and this proper subset is used when adopting the coding parameters or predicting the coding parameters of a given simply connected region.

[0009] In yet another embodiment, the region of samples representing a spatial sampling of the two-dimensional information signal is spatially sub-divided by multiple recursive partitioning into a number of simply connected regions of different sizes according to a first subset of syntax elements contained in the data stream. Then, according to a second subset of syntax elements in the data stream that are separated from the first subset, the spatially adjacent simply connected regions are combined to perform an intermediate sub-division of the sequence of samples into disjoint sets of simply connected regions whose union is a number of simply connected regions. The intermediate sub-division is used in reconstructing the sequence of samples from the data stream. This reduces the importance of optimizing the sub-division, since if the sub-division is too dense, it can be corrected by combining later. Furthermore, the combination of sub-division and combining allows the realization of intermediate sub-divisions that are not possible by multiple recursive partitioning alone, and the sub-division and combining can be combined using a set of separated syntax elements to improve the fit of the effective or intermediate sub-division to the actual content of the two-dimensional information signal. The additional overhead caused by the additional subset of syntax elements to indicate the details of the combining is negligible compared to the advantages.

[0010] In an embodiment, the array of information samples, which represents the spatially sampled information signal, is first spatially divided into tree root regions and then subdivided according to multi-tree subdivision information extracted from the data stream. At least a subset of the tree root regions is subdivided into smaller simply connected regions of different sizes by performing multiple partitionings recursively on the subset of the tree root regions. In order to be able to find a desirable compromise between too fine and too coarse subdivision in terms of rate / distortion with a reasonable encoding complexity, the size of the largest region of the tree root regions, into which the array of information samples is spatially divided, is included in the data stream and extracted from the data stream on the decoding side. The decoder may therefore comprise an extraction unit capable of extracting the maximum region size and multi-tree subdivision information from the data stream, a subdivision unit capable of spatially dividing an array of information samples representing a spatially sampled information signal into tree root regions of the maximum region size, subdividing according to the multi-tree subdivision information and subdividing at least a subset of the tree root regions into smaller simply connected regions of various sizes by recursively partitioning the subset of the tree root regions multiple times, and a reconstruction unit capable of reconstructing the array of information samples from the data stream using the subdivision into smaller simply connected regions.

[0011] In an embodiment, the data stream also includes the maximum hierarchical level at which a subset of the root region of a tree is subject to multiple recursive partitioning, making it easier to convey multi-tree subdivision information and requiring fewer bits to encode.

[0012] Furthermore, the reconstruction unit may be configured to perform one or more of the following with a granularity depending on the intermediate subdivision: determining a prediction mode to use from at least intra prediction modes and inter prediction modes; converting from the spectral domain to the spatial domain; performing inter prediction and / or setting inter prediction parameters; performing intra prediction and / or setting intra prediction parameters.

[0013] Furthermore, the extractor may be configured to extract syntax elements associated with the leaf regions of the partitioned treeblock from the data stream in a depth-order traversal order. In this way, the extractor can use the statistics of the syntax elements of the adjacent leaf regions already coded with a higher probability than if a width-order traversal order was used. In another embodiment, a further subdivision unit is used to subdivide at least a subset of the smaller simply connected regions into smaller simply connected regions according to further multi-tree subdivision information. The first stage subdivision may be used by the reconstruction unit to perform a prediction of the domain of information samples, and the second stage subdivision may be used by the reconstruction unit to perform a retransformation from the spectral domain to the spatial domain. Defining the residual subdivision at a lower level with respect to the prediction subdivision reduces the bit consumption of the overall coding of the subdivision. On the other hand, the restrictions and degrees of freedom of the residual subdivision due to the lower definition have a small negative impact on the coding efficiency. This is because in most cases the parts of an image that have similar motion compensation parameters are larger than the parts that have similar spectral characteristics.

[0014] In yet another embodiment, a further maximum region size is included in the data stream, which defines the size of the tree-root sub-region into which the smaller simply-connected regions are first divided before at least a subset of the tree-root sub-regions are further subdivided into smaller simply-connected regions according to the further multi-tree subdivision information. This allows for a unique setting of the maximum region size of the prediction subdivision, while the residual subdivision allows for an improved rate / distortion trade-off.

[0015] In yet another embodiment of the invention, a first subset of syntax elements, which is separated from a second subset of syntax elements forming the multi-tree subdivision information, is included in the data stream, and a combination unit at the decoding side can combine smaller spatially adjacent simply connected regions of the multi-tree subdivision according to the first subset of syntax elements to obtain an intermediate subdivision of the sample sequence. The reconstruction unit may be configured to reconstruct the sample sequence using the intermediate subdivision. This allows the encoder to perform a subdivision that is effective for the spatial distribution of the characteristics of the information sample sequence, making it easier to find an optimal rate / distortion compromise. For example, if the size of the largest region is large, the multi-tree subdivision information is likely to be complex due to the large tree root region. On the other hand, if the size of the largest region is small, it is likely that adjacent tree root regions are related to information contents with similar characteristics and that these tree root regions are also processed together. The combination eliminates the gap between the two extreme cases, resulting in a subdivision with a granularity close to the optimum. From the encoder's point of view, the combination syntax element simplifies the encoding procedure or reduces the computational complexity, since if the encoder mistakenly uses too fine a subdivision, it will correct the mistake by subsequently setting the combination syntax element, with or without adaptation of a small part of the syntax elements that were set before the setting of the combination syntax element.

[0016] In yet another embodiment, the maximum region size and multi-tree subdivision information is used for residual subdivision rather than prediction subdivision.

[0017] When dealing with simply connected regions of a quadtree subdivision of an array of information samples representing a spatially sampled information signal, a depth-ordered traversal order is used in accordance with an embodiment, rather than a width-ordered traversal order. Using a depth-ordered traversal order increases the probability that each simply connected region has neighboring simply connected regions that have been traversed, and ensures that information about these neighboring simply connected regions is available when reconstructing each current simply connected region.

[0018] If the array of information samples is first partitioned into a regular array of tree root regions of zeroth order hierarchical size, and then at least a subset of the tree root regions is subdivided into smaller simply connected regions of different sizes, the reconstruction unit may use a zigzag scan to scan the tree root regions, and for each partitioned tree root region, process the simply connected leaf regions in a depth-order traversal order, and then proceed to the next tree root region in a zigzag scan order. Furthermore, according to the depth-order traversal order, simply connected leaf regions at the same hierarchical level may be traversed in a zigzag scan order. In this way, an increased probability of having adjacent simply connected leaf regions is maintained.

[0019] In an embodiment, the flags associated with the nodes of the multi-tree structure are arranged sequentially in the depth-order traversal order, but the sequential coding of the flags uses a context of probability estimation that is the same for flags associated with nodes of the multi-tree structure located in the same hierarchical level of the multi-tree structure, but different for nodes of the multi-tree structure located in different hierarchical levels of the multi-tree structure, thus allowing, on the one hand, an improved compromise between the number of contexts provided and adaptation to the statistics of the actual symbols of the flags.

[0020] In an embodiment, the context of the probability estimation for a given flag used is different depending on the flag corresponding to a region of the root region of the tree that precedes the given flag in the depth-order traversal order and has a predetermined relative position to the region to which the given flag corresponds. As with the idea underlying the described aspect, the use of a depth-order traversal order ensures with a high probability that the encoded flags include flags corresponding to regions adjacent to the region to which the given flag corresponds. This knowledge is exploited to improve the adaptability of the given flag to the context to which it is used.

[0021] The flags used to set the context for a given flag may correspond to a region that is located above and / or to the left of the region to which the given flag corresponds. Furthermore, the flags used to select a context may be restricted to those flags that belong to the same hierarchical level as the node with which a given flag is associated.

[0022] In an embodiment, the coded signalling information comprises an indication of the highest hierarchical level and a set of flags associated with nodes of the multi-tree structure other than the highest hierarchical level. Each flag specifies whether the associated node is an intermediate node or a child node. The set of flags are decoded sequentially from the data stream in a depth-ordered or width-ordered traversal order, skipping the nodes at the highest hierarchical level and automatically designating the same leaf node. As a result, the coding rate is reduced.

[0023] In another embodiment, the highest hierarchical level may be indicated in the coded signaling of the multi-tree structure, making it possible to restrict the presence of flags to hierarchical levels other than the highest hierarchical level, preventing further partitioning of blocks at the highest hierarchical level.

[0024] If a spatial multitree subdivision is part of a secondary subdivision of a leaf node and the root region of the tree of the primary multitree subdivision is not partitioned, the context used to encode the flags of the secondary subdivision may be selected such that the context is the same for flags associated with regions of the same size. Preferred embodiments of the present invention will now be described with reference to the following figures. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a configuration diagram showing an encoder according to an embodiment of the present application. [Diagram 2] FIG. 2 is a block diagram showing a decoder according to an embodiment of the present invention; [Figure 3A] FIG. 2 is a schematic diagram illustrating an example of a subdivision of a quadtree, showing the first hierarchical level. [Figure 3B] FIG. 2 is a schematic diagram illustrating an example of a subdivision of a quadtree, showing the second hierarchical level. [Figure 3C] FIG. 2 is a schematic diagram illustrating an example of a subdivision of a quadtree, showing the third hierarchical level. [Figure 4] FIG. 4 is a schematic diagram illustrating a tree structure of a subdivision of the quadtree of FIGS. 3A to 3C according to an embodiment. [Figure 5A] FIG. 4 is a schematic diagram showing a subdivision of the quadtree of FIGS. 3A to 3C. [Figure 5B] FIG. 4 is a schematic diagram showing an indexed tree structure of the individual leaf blocks of FIGS. 3A to 3C. [Figure 6A] 5 is a schematic diagram illustrating a sequence of binary strings or flags representing a subdivision of the tree structure of FIG. 4 and the quadtree of FIGS. 3A-3C according to different embodiments. [Figure 6B] 5 is a schematic diagram illustrating a sequence of binary strings or flags representing a subdivision of the tree structure of FIG. 4 and the quadtree of FIGS. 3A-3C according to different embodiments. [Figure 7] 4 is a flow chart showing steps performed by a data stream extraction unit according to an embodiment; [Figure 8]5 is a flow chart showing the functionality of a data stream extraction unit according to another embodiment; [Figure 9A] FIG. 2 is a schematic diagram illustrating a quadtree subdivision according to an embodiment, showing candidate blocks neighboring a given block highlighted; [Figure 9B] FIG. 2 is a schematic diagram illustrating a quadtree subdivision according to an embodiment, showing candidate blocks neighboring a given block highlighted; [Figure 10] 5 is a flow chart showing the functionality of a data stream extraction unit according to another embodiment; [Figure 11] FIG. 2 is a schematic diagram illustrating the construction of an image from planes and plane groups and illustrating coding using inter-plane adaptation / prediction according to an embodiment. [Figure 12A] FIG. 2 is a schematic diagram showing a subtree structure to explain an inheritance scheme according to an embodiment. [Figure 12B] FIG. 2 is a schematic diagram showing subdivisions corresponding to subtree structures to explain an inheritance scheme according to an embodiment. [Figure 12C] FIG. 2 is a schematic diagram showing a subtree structure to explain the inheritance scheme of adoption and prediction according to an embodiment. [Figure 12D] FIG. 2 is a schematic diagram showing a subtree structure to explain the inheritance scheme of adoption and prediction according to an embodiment. [Figure 13] 4 is a flow chart illustrating steps performed by an encoder to implement an inheritance scheme in accordance with an embodiment. [Figure 14] FIG. 14A is a diagram showing a first order subdivision to illustrate the possibility of implementing an inheritance scheme in relation to inter-plane prediction in accordance with an embodiment, and FIG. 14B is a diagram showing a lower order subdivision to illustrate the possibility of implementing an inheritance scheme in relation to inter-plane prediction in accordance with an embodiment. [Figure 15] FIG. 2 is a block diagram illustrating the process of decoding in conjunction with an inheritance scheme, according to an embodiment. [Figure 16]FIG. 2 is a schematic diagram illustrating the order of sub-region traversal of a multi-tree subdivision for a sub-region subject to intra prediction according to an embodiment; [Figure 17] FIG. 2 is a configuration diagram showing a decoder according to an embodiment. [Figure 18A] FIG. 4 is a schematic diagram showing different subdivision possibilities according to another embodiment; [Figure 18B] FIG. 4 is a schematic diagram showing different subdivision possibilities according to another embodiment; [Figure 18C] FIG. 4 is a schematic diagram showing different subdivision possibilities according to another embodiment; [Figure 19] FIG. 1 is a configuration diagram showing an encoder according to an embodiment. [Figure 20] FIG. 13 is a block diagram showing a decoder according to another embodiment of the present invention; [Figure 21] FIG. 13 is a configuration diagram showing an encoder according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] In the following description of each figure, elements that appear in multiple figures are designated by common reference numerals, and duplicate descriptions of these elements are omitted, i.e., a description of an element that appears in one figure also applies to other figures in which the element appears, to the extent that the description is relevant.

[0027] Furthermore, the following description begins with the encoder and decoder embodiments described in Figures 1 through 11. The embodiments described for these figures combine many aspects of the present application, but also benefit when implemented separately within a coding scheme. Thus, for the following figures, embodiments utilizing the above aspects using each of these embodiments separately are briefly described, and represent different abstractions of the embodiments described for Figures 1 and 11.

[0028] FIG. 1 shows an encoder according to an embodiment of the invention. The encoder 10 of FIG. 1 comprises a prediction unit 12, a residual precoder 14, a residual reconstruction unit 16, a data stream insertion unit 18 and a block division unit 20. The encoder 10 is for encoding a temporally and spatially sampled information signal into a data stream 22. The temporally and spatially sampled information signal may be, for example, a video, i.e. a sequence of images. Each image represents an array of image samples. Other examples of temporally and spatial information signals include, for example, depth images captured by a time-of-light camera. It should be noted further that the spatially sampled information signal may contain more than one array per frame or timestamp, for example in the case of a color video, where one array of luma samples is included per frame together with two arrays of chroma samples. Also, the temporal sampling rates of the different components of the information signal, i.e. luma and chroma, may be different. The same applies to the spatial resolution. The video may also be accompanied by further spatially sampled information, such as depth or transparency information. However, in the following description, in order to facilitate understanding of the main teachings of the present application, the description will first focus on the processing of one of these arrays, and then the handling of multiple planes.

[0029] The encoder 10 of Fig. 1 is configured to create a data stream 22 such that syntax elements of the data stream 22 describe images at a granularity lying between the entire image and individual image representatives. To that end, the segmenter 20 is configured to subdivide each image 24 into simply connected regions 26 of various sizes. In the following, these regions will be referred to simply as blocks or subregions 26.

[0030] As will be explained in more detail below, the segmentation unit 20 uses a multi-tree subdivision to subdivide the image 24 into blocks 26 of various sizes. More precisely, in the particular embodiment outlined below with respect to Figures 1 to 11, a quad-tree subdivision is mostly used. The segmentation unit 20 may comprise an internally connected subdivision unit 28, as will be explained in more detail below. The subdivision unit 28 subdivides the image 24 into said blocks 26. The subdivision unit 28 is connected to a combination unit 30. The combination unit 30 can combine groups of these blocks 26 to obtain an effective subdivision or granularity lying between the non-subdivision of the image 24 and the subdivision defined by the subdivision unit 28.

[0031] 1, the predictor 12, the residual precoder 14, the residual reconstruction unit 16 and the data stream insertion unit 18 perform a subdivision of the image defined by a partitioning unit 20. For example, as described in more detail below, the predictor 12 uses the prediction subdivision defined by the partitioning unit 20 to: set prediction parameters corresponding to each sub-region according to a selected prediction mode and make a decision for each individual sub-region of the prediction subdivision as to whether the respective sub-region should be subjected to intra-picture or inter-picture prediction;

[0032] The residual precoder 14 may then use the subdivision of the residual of the image 24 for encoding the residual of the prediction of the image 24 provided by the predictor 12. When the residual reconstruction unit 16 reconstructs the residual from the syntax elements output by the residual precoder 14, the residual reconstruction unit 16 also performs said subdivision of the residual. The data stream insertion unit 18 may use said division, i.e. the prediction and residual subdivision, to determine the order and adjacent relationship of the syntax elements for insertion, for example using entropy encoding, of the syntax elements output by the residual precoder 14 and the predictor 12 into the data stream 22.

[0033] As shown in Fig. 1, the encoder 10 comprises an input 32 at which the original information signal is input to the encoder 10. A subtractor 34, a residual precoder 14 and a data stream inserter 18 are connected in series, in this order, between the input 32 and an output of the data stream inserter 18 at which the encoded data stream 22 is output. The subtractor 34 and the residual precoder 14 are part of a prediction loop. The prediction loop is closed with a residual construction unit 16, an adder 36 and a prediction unit 12, which are connected in series, in the following order, between the output of the residual precoder 14 and an inverted input of the subtractor 34. The output of the prediction unit 12 is also connected to another input of the adder 36. The prediction unit 12 may also have an input connected directly to the input 32 and another input connected to the output of the adder 36 via an optional in-loop filter 38. Furthermore, since the prediction unit 12 generates associated information during operation, an output of the prediction unit 12 is also connected to a data stream insertion unit 18. Similarly, the splitting unit 20 has an output connected to another input of the data stream insertion unit 18. Having described the structure of encoder 10, its mode of operation will now be described in more detail.

[0034] As explained above, the partitioning unit 20 determines for each image 24 its method of subdivision into sub-regions 26. According to the subdivision of the image 24 used for the prediction, the prediction unit 12 determines for each sub-region corresponding to this subdivision its method of prediction. The prediction unit 12 outputs a prediction of the sub-region to an inverting input of the subtractor 34 and to another input of the adder 36, and outputs prediction information to the data stream insertion unit 18 reflecting how the prediction unit 12 obtains this prediction from the encoded part of the video.

[0035] At the output of the subtractor 34, the residual precoder 14 processes this prediction residual according to the residual subdivision indicated by the division unit 20 to obtain a prediction residual. As will be explained in more detail below using figures 3 to 10, the residual subdivision of the image 24 used by the residual precoder 14 may be related to the prediction subdivision used by the predictor 12, such that each prediction subregion is adopted as a residual subregion or is further subdivided into smaller residual subregions, or there may be a completely independent prediction and residual subdivision.

[0036] The residual precoder 14 performs a spatial to spectral domain transformation by a two-dimensional transform on each residual sub-region, followed by or essentially involving a quantization of the resulting transform coefficients of the resulting transform block, whereby distortions occur as a result of quantization noise. For example, the data stream inserter 18 may losslessly encode syntax elements describing said transform coefficients into the data stream 22, for example using entropy encoding.

[0037] The residual reconstruction unit 16 then retransforms the transform coefficients into a residual signal using requantization with retransformation. To obtain the residual of the prediction, the residual signal is combined in an adder 36 with the prediction used by the subtractor 34. As a result, at the output of the adder 36, a reconstructed part or sub-region of the current image is obtained. The prediction unit 12 may directly use the sub-regions of the reconstructed image for intra prediction, in order to make a prediction of a certain prediction sub-region by estimation from neighbouring reconstructed prediction sub-regions. However, intra prediction performed in the spectral domain by directly predicting the spectrum of the current sub-region from the spectra of neighbouring sub-regions is also theoretically possible.

[0038] For inter-plane prediction, the predictor 12 may use a version of the encoded and reconstructed image that has been filtered by an optional in-loop filter 38. For example, the in-loop filter 38 may comprise a deblocking filter and / or an adaptive filter having a transfer function that benefits from shaping the quantization noise as described above.

[0039] The predictor 12 selects prediction parameters which indicate how to make a prediction of a particular prediction sub-region compared to the original exemplars in the image 24. As explained in more detail below, the prediction parameters may include an indication of a prediction mode, such as intra or inter, for each prediction sub-region. In the case of intra prediction, the prediction parameters may include an indication of the angle along which the boundary within the prediction sub-region within which the intra prediction is performed primarily extends. In the case of inter prediction, the prediction parameters may include motion vectors, video indices and also higher order motion transformation parameters. In the case of both intra and / or inter prediction, the prediction parameters may include optional filter information for filtering the reconstructed image exemplars on which the prediction of the current prediction sub-region is based.

[0040] As will be explained in more detail below, said subdivision defined by the partitioning unit 20 significantly affects the rate / distortion ratio that can be maximally achieved by the residual precoder 14, the predictor 12 and the data stream insertion unit 18. A finer subdivision may result in a better prediction obtained by the predictor 12, a smaller residual signal coded by the residual precoder 14 may be coded with fewer bits, but a much larger coding rate is required for the prediction parameters 40 output by the predictor 12 and inserted into the data stream 22. The opposite is true for a coarser subdivision. The above ideas also apply equally to the residual subdivision. Transforming an image with a finer granularity for the individual transform blocks reduces the computational complexity of the transform and increases the spatial resolution of the resulting transform, i.e. smaller residual subregions result in a more consistent spectral distribution of the contents of the individual residual subregions. However, the spectral resolution decreases and the ratio between significant coefficients and insignificant coefficients, i.e. quantized to zero, deteriorates. That is to say, the granularity of the transformation must be locally adapted to the image content. Moreover, apart from the positive effects of a finer granularity, a finer granularity usually requires an increased amount of accompanying information to indicate the selected subdivision to the decoder. As will be explained in more detail below, the following embodiment describes an encoder 10 that is able to adapt the subdivision very effectively to the content of the encoded information signal and to communicate the used subdivision information to the decoding side by instructing a data stream insertion unit 18 to insert subdivision information in the coded data stream 22. The details are given below.

[0041] However, before defining the subdivision of the divider 20 in detail, a decoder according to an embodiment of the present application will be described in detail with reference to FIG.

[0042] The decoder of Fig. 2 is indicated with the reference numeral 100 and comprises an extraction unit 102, a segmentation unit 104, a residual reconstruction unit 106, an adder 108, a prediction unit 110, an optional in-loop filter 112 and an optional post-filter 114. The extraction unit 102 receives an encoded data stream at an input 116 of the decoder 100 and extracts from the subdivision information 118 of the encoded data stream prediction parameters 120 and residual data 122 which the extraction unit 102 outputs to the image segmentation unit 104, the prediction unit 110 and the residual reconstruction unit 106, respectively. An output of the residual reconstruction unit 106 is connected to a first input of the adder 108. Another input and an output of the adder 108 are connected to a prediction loop, in which the optional in-loop filter 112 and the prediction unit 110 are connected in series in this order, with a bypass path directly from the output of the adder 108 to the prediction unit 110. 1 above, i.e. one for intra-picture prediction and the other for inter-picture prediction. Either the output of the adder 108 or the output of the in-loop filter 112 may be connected to an output 124 of the decoder 100, where the reconstructed information signal is output, e.g. to a reproduction device. An optional post-filter 114 may be connected in the path to the output 124 in order to improve the visual quality of the reconstructed signal at the output 124.

[0043] Generally, the residual reconstruction unit 106, the adder 108 and the prediction unit 110 work similarly to the elements 16, 36 and 12 of FIG. 1. In other words, they perform operations similar to those of the above elements of FIG. 1. To that end, the residual reconstruction unit 106 and the prediction unit 110 are controlled by the prediction parameters 120 and the subdivision indicated by the image division unit 104, respectively, according to the subdivision information 118 from the extraction unit 102. The aim is to predict a small region of prediction in the same way as the prediction unit 12 did or decided to do, and to retransform the received transform coefficients with the same granularity as the residual precoder 14 did the transformation. The image division unit 104 then reorganizes in a synchronized manner the subdivision selected by the division unit 20 of FIG. 1 by referring to the subdivision information 118. The extraction unit may then use the subdivision information to control the data extraction, such as with regard to the selection of contexts, the determination of adjacency relations, the estimation of probabilities, the parsing of the syntax of the data stream, etc.

[0044] There are several possible variations of the above embodiment. Some of them, the subdivision performed by the subdivision unit 28 and the combination performed by the combination unit 30, are described in detail below. Others are described with reference to Figs. 12 to 16. All of these variations of the embodiment may be applied individually or in a subset to the above description of Figs. 1 and 2, respectively, wherever possible. For example, the division units 20 and 104 may not only determine the prediction subdivision and the residual subdivision for each image. Instead, they may determine the subdivision of each of the optional in-loop filters 38 and 112 either independently or dependently on other subdivisions for prediction or residual coding. Also, the subdivision determination by these elements may not be performed on a frame-by-frame basis. Instead, the subdivision determined for a particular frame may be reused or adopted for a certain number of subsequent frames before moving on to a new subdivision.

[0045] To provide further details regarding the division of an image into sub-regions, the following description first focuses on the subdivision performed by subdivision units 28 and 104a, followed by the combining process performed by combiner 30 and combiner 104b, and finally the inter-plane matching / prediction.

[0046] When the subdivision unit 28, 104a divides the image, it can divide it into a number of blocks, which can be of different sizes, for the purpose of coding predictions and residuals of image or video data. As mentioned above, the image 24 can be used as one or more arrays of image sample values. In the case of YUV / YCbCr color space, for example, a first array can represent the luma channel and the other two arrays can represent the chroma channels. These arrays can have different dimensions. All the arrays can be grouped into one or more plane groups, each plane group consisting of one or more consecutive planes, such that each plane is included in only one plane group. For each plane group, the following applies: The first array of a particular plane group is called the primary array of this plane group. Any other arrays that follow are subarrays. The division of the blocks of the primary arrays can be based on a quadtree approach. An optimal approach is described below. The division of the blocks of the subarrays can be derived based on the division of the primary arrays.

[0047] In the embodiment described below, the subdivision unit 28, 104a is configured to divide the primary array into a number of equal-sized square blocks, so-called treeblocks, as follows: If a quadtree is used, the length of the boundary of the treeblocks is typically a power of two, such as 16, 32, 64, etc. However, it should be noted that strictly speaking, other tree types can also be used, such as binary trees or trees with any number of leaves. Furthermore, the number of children of a tree can vary depending on the level of the tree and the signal it represents.

[0048] Alternatively, as mentioned above, the array of samples may represent information other than a video sequence, such as a depth map or a light field. For simplicity, the following description focuses on a quadtree as a representative example of a multi-tree. A quadtree is a tree with exactly four children at each internal node. Each treeblock constitutes a primary quadtree, and each leaf of the primary quadtree constitutes a sub-quadtree. The primary quadtree determines the subdivision of a given treeblock for prediction, and the sub-quadtree determines the subdivision of a given prediction block for residual coding.

[0049] The root node of a primary quadtree corresponds to a complete treeblock. For example, Figure 3A illustrates treeblock 150. We assume that each image is divided into a regular grid of lines and columns, such as treeblock 150, such that the array of samples is covered without gaps. However, it should be noted that not all of the subdivisions of blocks described below require seamless subdivisions without overlaps. Instead, adjacent blocks may overlap each other, as long as no leaf block is a proper subpart of an adjacent leaf block.

[0050] Along with the quadtree structure of the treeblock 150, each node can be further divided into four child nodes. For a first-order quadtree, this means that each treeblock 150 can be divided into four subblocks, each half the width and height of the treeblock 150. In FIG. 3A, these subblocks are indicated by reference numbers 152a to 152d. In a similar manner, these subblocks can be further divided into four smaller subblocks, each half the width and height of the original subblock. In FIG. 3d, an example of a subblock 152c is shown, which is subdivided into four smaller subblocks 154a to 154d. In FIGS. 3A to 3C, an example of a treeblock 150 being divided as follows: The treeblock 150 is first divided into four subblocks 152a to 152d. The bottom-left subblock 152c is then further divided into four smaller subblocks 154a to 154d. Finally, as shown in Figure 3C, the top right of these smaller sub-blocks, block 154b, is again divided into four blocks having 1 / 8 the width and height of the original treeblock 150. These smaller blocks are designated 156a through 156d.

[0051] Figure 4 shows the tree structure that is the basis for the example of quadtree-based partitioning shown in Figures 3A to 3d. The numbers next to the nodes of the tree are values ​​called subdivision flags, which will be explained in more detail later when explaining the signaling of the quadtree structure. The root node of the quadtree is shown at the top of the figure ("level 0"). The four branches of this root node at level 1 correspond to the four subblocks shown in Figure 3A. The third of these subblocks is further subdivided into four subblocks in Figure 3B, so the third node at level 1 in Figure 4 also has four branches. In addition, there are four subordinate branches connecting to the second node at level 2 of the quadtree hierarchy, corresponding to the subdivision of the second (top right) child node in Figure 3C. The node at level 3 is not further subdivided.

[0052] Each leaf of the primary quadtree corresponds to a variable-sized block for which individual prediction parameters can be specified (i.e. intra- or inter-plane prediction mode, motion parameters, etc.). In the following, these blocks are referred to as prediction blocks. These leaf blocks are specifically illustrated in FIG. 3C. Referring briefly now to the description of FIG. 1 and FIG. 2, the partitioning unit 20 or subdivision unit 28 performs the above-mentioned quadtree subdivision decision. The subdivision units 152a-152d perform the decision of which of the tree block 150, the sub-blocks 152a-152d, the smaller sub-blocks 154a-154d, etc., are to be subdivided or further partitioned. This aims to find an optimal trade-off between fine prediction subdivision and coarse prediction subdivision, as already indicated above. The prediction unit 12 then uses the indicated prediction subdivision to determine the above-mentioned prediction parameters with a granularity according to the prediction subdivision. Alternatively, for each prediction sub-region represented by a block shown in FIG. 3C or the like, a indicated prediction subdivision is used.

[0053] The prediction block shown in Fig. 3C can be further divided into smaller blocks for the purpose of residual coding. For each prediction block, i.e., for each leaf node of the primary quadtree, the corresponding subdivision is determined by one or more sub-quadtrees for residual coding. For example, if the size of the residual block is at most 16x16, a given 32x32 prediction block can be divided into four 16x16 blocks, each determined by a sub-quadtree for residual coding. Each 16x16 block in this example corresponds to a root node of a sub-quadtree.

[0054] Similar to the description for the subdivision of a given treeblock into predictive blocks, each predictive block can be split into multiple residual blocks by decomposing the sub-quadtree. Each leaf of the sub-quadtree corresponds to a residual block. The residual blocks can be assigned individual residual coding parameters (i.e., transform mode, transform coefficients, etc.) by the residual precoder 14, which is controlled by the residual coding parameters, followed by the residual reconstruction unit 16, 106.

[0055] In other words, the subdivision unit 28 may be configured to determine a prediction subdivision and a sub-residual subdivision for each image or each group of images. This is done as follows: first, the image is divided into a regular array of tree blocks 150, and a subset of these tree blocks is partitioned recursively by a quadtree subdivision to obtain a prediction subdivision into prediction blocks. If no partitioning is performed on each tree block, or on the leaf blocks of the quadtree subdivision, the prediction blocks may be tree blocks. Then, the subset of these prediction blocks is further sub-divided in a similar manner as follows: If the prediction block is larger than the maximum size of the sub-residual subdivision, first, the respective prediction block is partitioned into a regular array of sub-tree blocks. Then, the subset of these sub-tree blocks is sub-divided according to the quadtree subdivision procedure to obtain a residual block. If no partitioning is performed on each prediction block into sub-tree blocks, the residual block may be a prediction block. It may also be a subtree block if no further division into smaller regions is performed for each subtree block or subdivision leaf block of the residual quadtree.

[0056] As outlined above, the subdivision selected for the primary array may be mapped to the subarray. This is easy when considering a subarray with the same dimensions as the primary array. However, special methods are required when the dimensions of the primary array and the subarray are different. In general, mapping of the subdivision of the primary array to the subarray with different dimensions is possible by spatial mapping, i.e. by spatially mapping the boundaries of the blocks of the subdivision of the primary array to the subarray. In particular, for each subarray, horizontal and vertical scaling factors may be used that determine the ratio of the dimensions of the primary array to the subarray. The division of the subarray into subblocks for prediction and residual coding may be determined using the treeblocks of the subarray resulting from applying the relative scaling factors by the primary quadtree and the subquadtree of each aligned treeblock of the primary array. If the horizontal and vertical scaling factors are different (e.g. 4:2:2 chroma subsampling), the resulting prediction and residual blocks of the subarray will not be square. In this case, it is possible to predetermine or select whether a non-square residual block (for the whole sequence, for one image of the sequence, or for a single prediction or residual block, respectively) is to be divided into square blocks or not, depending on the situation. For example, in the first case, the encoder and decoder accept the subdivision into square blocks if the mapped block is not square. In the second case, the subdivision unit 28 signals the selection via the data stream insertion unit 18 and the data stream 22 to the subdivision unit 104a. For example, in the case of 4:2:2 chroma subsampling, if the subarray is half as wide as the primary array and has the same height, the residual block is twice as high as it is wide. By dividing this block vertically, two square blocks are obtained again from one block.

[0057] As mentioned above, the subdivision unit 28 or the division unit 20, respectively, communicates the quadtree-based division to the subdivision unit 104a via the data stream 22. To that end, the subdivision unit 28 informs the data stream insertion unit 18 of the subdivision selected for the image 24. The data stream insertion unit then transmits to the decoding side the structures of the primary and secondary quadtrees, i.e. the division of the image array into blocks of variable size for prediction or residual coding in the data stream or bitstream 22.

[0058] The minimum and maximum allowed block sizes are signaled as side information and may vary from picture to picture. Also, the minimum and maximum allowed block sizes can be fixed in the encoder and decoder. These minimum and maximum block sizes may be different for predicted and residual blocks. To signal the structure of the quadtree, the quadtree needs to be traversed and for each node it needs to be specified whether this particular node is a leaf node of the quadtree (i.e. the corresponding block is not further subdivided) or branches into four child nodes (i.e. the corresponding block is divided into four sub-blocks of half size).

[0059] The information propagation within an image is done treeblock by treeblock in a raster scan order, such as left to right and top to bottom, as shown at 140 in Figure 5A. The scan order can also be other orders, such as bottom right to top left, checkerboard, etc. In the preferred embodiment, to propagate the subdivision information, each treeblock, i.e., each quadtree, is traversed in depth order.

[0060] In a preferred embodiment, not only the subdivision information, i.e. the structure of the tree, but also the prediction data etc., i.e. the payload associated with the leaf nodes of the tree, are transmitted / processed in depth-order. The reason for this is that a depth-order traversal has significant advantages over width-order. In Fig. 5B, the structure of a quadtree is shown with leaf nodes denoted a, b, ..., j. Fig. 5A shows the resulting partitioning of blocks. If the blocks / leaf nodes are traversed in width-order, they are obtained in the order abjchidefg, whereas in depth-order it is abc...ij. As can be seen in Fig. 5A, in the depth-order case, the left and top neighboring blocks are always transmitted / processed before the current block. In this way, the parameters specified for the left and top neighboring blocks can always be used for motion vector prediction and context modeling, resulting in improved coding efficiency. This is not the case in the width-order case, since for example block j is transmitted before blocks e, g, i.

[0061] Finally, the signaling is performed recursively for each treeblock along the structure of the primary quadtree so that for each node, a flag is transmitted specifying whether the corresponding block is divided into four subblocks or not. If this flag has a value of "1" (indicating "true"), this signaling process is repeated recursively for all four child nodes or subblocks in raster scan order (top left, top right, bottom left, bottom right) until a leaf node of the primary quadtree is reached. The leaf node is characterized by having a subdivision flag with a value of "0". If the node is at the lowest hierarchical level of the primary quadtree and corresponds to the smallest allowed size of the prediction block, no subdivision flag needs to be transmitted. In the example of Figures 3A to 3C, a "1" is transmitted first, as shown at 190 in Figure 6A, which specifies that the treeblock 150 is divided into four subblocks 152a to 152d. Then, in the raster scan order 200, the encoding of the subdivision information for all four sub-blocks 152a-152d is performed recursively. For the first two sub-blocks 152a and 152b, a "0" is sent (see 202 in FIG. 6A), specifying that no subdivision is performed. For the third sub-block 152c (bottom left), a "1" is sent (see 204 in FIG. 6A), specifying that this block is sub-divided. This causes the four sub-blocks 154a-154d of this block to be processed using a recursive approach. Now, for the first sub-block, a "0" is sent (206), and for the second (top right) sub-block, a "1" is sent (208). This causes the four blocks 156a-156d, which are the smallest block size in FIG. 3C, to be processed. If the smallest allowed block size in this example is reached, no further subdivision can be performed and no more data needs to be transmitted. Alternatively, "0000" is transmitted to specify that these blocks are not further divided, as shown in Fig. 6A at 210. After this, "00" is transmitted for the bottom two blocks of Fig. 3B (see Fig. 6A at 212), and finally, for the bottom right block of Fig. 3A, a "0" is transmitted (see 214).Thus, the complete binary sequence representing the quadtree structure, in its entirety, looks like that shown in Figure 6A.

[0062] In this binary sequence shown in Fig. 6A, the different background shadings correspond to different levels of the hierarchy of the quadtree-based subdivision: shading 216 represents level 0 (corresponding to a block size equal to the original treeblock size), shading 218 represents level 1 (corresponding to a block size equal to half the original treeblock size), shading 220 represents level 2 (corresponding to a block size equal to 1 / 4 the original treeblock size) and shading 222 represents level 3 (corresponding to a block size equal to 1 / 8 the original treeblock size). All subdivision flags of the same hierarchical level (corresponding to the same block size and thus to the same representation in the illustrated binary sequence) may be entropy coded, for example by the insertion unit 18, using one and the same probability model. In the case of width-order traversal, the subdivision information is transmitted in a different order as shown in FIG. 6b.

[0063] The resulting division of each prediction block into residual blocks, as well as the subdivision of each tree block for prediction purposes, must be transmitted in the bitstream. There may also be a maximum and minimum block size for residual coding transmitted as side information, which may vary from image to image. The maximum and minimum block sizes for residual coding may also be fixed in the encoder and decoder. At each leaf node of the primary quadtree as shown in Fig. 3C, the corresponding prediction block may be divided into residual blocks of the maximum allowed size. These blocks are the root nodes that constitute the structure of the lower quadtree for residual coding. For example, if the maximum size of the residual blocks of an image is 64x64 and the size of the prediction blocks is 32x32, then the entire prediction block corresponds to one root node of the lower (residual) quadtree of size 32x32. On the other hand, if the maximum size of the residual blocks of an image is 16x16, then the 32x32 prediction block will be composed of four root nodes of the residual quadtree of size 16x16. Within each prediction block, the propagation of the sub-quadtree structure is performed for each root node in raster scan order (left to right, top to bottom). As in the case of the primary (prediction) quadtree structure, for each node, a flag is coded specifying whether this particular node is split into four child nodes or not. Then, if the value of this flag is "1", this procedure is repeated recursively for all four corresponding child nodes and corresponding sub-blocks in raster scan order (top left, top right, bottom left, bottom right) until a leaf node of the sub-quadtree is reached. As in the case of the primary quadtree, there is no need to propagate for the nodes of the lowest hierarchical level of the sub-quadtree, since these nodes correspond to the minimum size blocks of the residual block and cannot be further divided.

[0064] For entropy coding, the subdivision flags of residual blocks belonging to residual blocks of the same block size may be encoded using one and the same probability model.

[0065] Thus, according to the example given above on the basis of figures 3A to 6A, the subdivision unit 28 defined a primary subdivision for prediction purposes and sub-subdivisions of blocks of different sizes of the primary subdivision for residual coding purposes. The data stream insertion unit 18 performed the coding of the primary subdivision by transmitting, for each tree block in zigzag scan order, a bit sequence constructed according to figure 6A, as well as the coding of the maximum size of the primary blocks and the maximum hierarchical level of the primary subdivision. For each prediction block thus defined, the associated prediction parameters are included in the data stream. Also, for each prediction block that was less than or equal to the maximum size of the residual subdivision and for each block of the root of the residual tree divided from the prediction block that exceeds the maximum size defined for the residual block, the coding of similar information, i.e. the maximum size, the maximum hierarchical level and the bit sequence as in figure 6A, was performed. For each residual block thus defined, the residual data is inserted into the data stream.

[0066] The extraction unit 102 extracts each bit sequence from the data stream at the input unit 116 and informs the obtained subdivision information to the division unit 104. Furthermore, the data stream insertion unit 18 and the extraction unit 102 may further transmit syntax elements, such as the residual data output by the residual precoder 14 and the prediction parameters output by the prediction unit 12, using the above order between the prediction blocks and the residual blocks. Using this order has the advantage that it may be possible to select a suitable context for the encoding of the individual syntax elements of a certain block by utilizing the already coded / decoded syntax elements of the neighboring blocks. Furthermore, similar to the residual precoder 14 and the prediction unit 12, the residual reconstruction unit 106 and the precoder 110 may also process the individual prediction blocks and the residual blocks in the order outlined above.

[0067] Figure 7 is a flow chart showing the steps performed by the extractor 102 to extract subdivision information from the data stream 22 when encoding is performed in the manner outlined above. In a first step, the extractor 102 divides the image 24 into tree root blocks 150. This step is shown as step 300 in Figure 7. In step 300, the extractor 102 may extract the maximum size of the prediction blocks from the data stream 22. Alternatively, in step 300, the extractor 102 may extract the maximum hierarchical level from the data stream 22. Next, in step 302, the extractor 102 decodes flags or bits from the data stream. When step 302 is performed for the first time, the extractor 102 recognizes that each flag is the first flag of the bit sequence belonging to the first block 150 of the root of the tree in the order 140 of scanning the blocks of the root of the tree. Since this flag is a flag of the hierarchical level 0, the extractor 102 can determine the context using the context modeling associated with that hierarchical level 0 in step 302. For each context, a respective probability estimate may be made for the entropy decoding of the associated flag. The context probability estimate may be adapted per context to the statistics of the symbols of the respective context. For example, to determine a suitable context for decoding the flag of the hierarchical level 0 in step 302, the extractor 102 may select one of the following contexts from the set of contexts: The context is associated with that hierarchical level 0 depending on the flags of the neighboring treeblocks at hierarchical level 0, and further depending on the information contained in the bit string defining the quadtree subdivision of the treeblocks neighboring the currently processed treeblock, such as the top and left neighboring treeblocks.

[0068] In the next step, i.e., step 304, the extractor 102 checks whether the last decoded flag suggests a partition. If so, the extractor 102 performs a partition of the current block (here a treeblock) in step 306 or indicates this partition to the subdivision unit 104a. Then, in step 308, it checks whether the current hierarchical level is equal to the maximum hierarchical level minus one. For example, the extractor 102 may keep the maximum hierarchical level extracted from the data stream in step 300. If the current hierarchical level is not equal to the maximum hierarchical level minus one, the extractor 102 increments the current hierarchical level by one in step 310 and returns to step 302 to decode the next flag from the data stream. Now, since the flag decoded in step 302 belongs to another hierarchical level, according to an embodiment, the extractor 102 may select one of the other sets of contexts belonging to the current hierarchical level. This selection may be based on the bit sequence of the subdivisions, such as in FIG. 6A, of the decoded neighboring treeblocks.

[0069] If the flag is decoded and it is determined in step 304 that the flag does not indicate partitioning of the current block, the extractor 102 proceeds to step 312 to check whether the current hierarchical level is 0. If so, the extractor 102 proceeds to step 314 to process the next tree root block in the traversal order 140, or stops the process of extracting subdivision information if there are no tree root blocks remaining to be processed.

[0070] It should be noted that the description of Figure 7 focuses only on the decoding of the flag indicating subdivision for the prediction subdivision. In practice, step 314 may also include decoding of other bins or syntax elements related to the current treeblock, etc. In any case, if there is a further next tree root block, the extractor 102 proceeds from step 314 to step 302 and decodes the next flag from the subdivision information, i.e. the first flag in the flag sequence for the new tree root block.

[0071] If it is found in step 312 that the hierarchical level is not 0, the process proceeds to step 316, where it is checked whether there are any further child nodes related to the current node. That is, when the extraction unit 102 checks in step 316, it has already been checked in step 312 that the current hierarchical level is a hierarchical level other than 0. There is a parent node that belongs to the tree root block 150 or one of the smaller blocks 152a-152d, or even to the smaller blocks 152a-152d, etc. The node of the tree structure to which the most recently decoded flag belongs has a parent node, which is also common to the other three nodes of the current tree structure. The order of traversal of such child nodes having a common parent node is illustrated in FIG. 3A by reference numeral 200 for the case of hierarchical level 0. Thus, in step 316, the extraction unit 102 checks whether all these four child nodes have been traversed in the process of FIG. 7. If not, i.e., if there are further child nodes together with the current parent node, the process of FIG. 7 proceeds to step 318. The next child node in the current hierarchical level is traversed according to the zigzag traversal order 200, so that the corresponding sub-block represents the current block for process 7. A flag is then decoded from the data stream for the current block or current node in step 302. However, if there are no more child nodes of the current parent node in step 316, the process of Figure 7 proceeds to step 320, where the current hierarchical level is decremented by one, and the process proceeds to step 312.

[0072] The extraction unit 102 and the subdivision unit 104a work together to obtain the encoder-side selected subdivision from the data stream by performing the steps shown in Figure 7. The process of Figure 7 summarizes the prediction subdivision for the above mentioned case. Figure 8, which is combined with the flow chart of Figure 7, shows how the extraction unit 102 and the subdivision unit 104a work together to obtain the residual subdivision from the data stream.

[0073] 8 shows in particular the steps performed by the extraction unit 102 and the subdivision unit 104a for each prediction block resulting from the prediction subdivision. These prediction blocks are traversed in a depth-order traversal order within each tree block 150 currently scanned to traverse a leaf block, as shown in FIG. 3C, according to the zigzag scan order 140 within the prediction subdivision tree block 150 as described above. The leaf blocks of the partitioned primary tree block are scanned according to the depth-order traversal order. The subblocks of the particular hierarchical level to which the common current node belongs are scanned in a zigzag scan order 200, scanning the subdivisions of each of these subblocks first, and then proceeding to the next subblock in the zigzag scan order 200. In the example of FIG. 3C, the resulting traversal order among the leaf nodes of treeblock 150 is indicated at 350 .

[0074] For the currently scanned prediction block, the process of Fig. 8 starts at step 400. In step 400, an internal parameter representing the current size of the current block is set equal to the size of hierarchical level 0 of the residual subdivision, i.e. the maximum block size of the residual subdivision. It should be noted that the maximum size of the residual block may be smaller than the minimum block size of the prediction subdivision or may be larger than it. In other words, in an embodiment, the encoder is free to choose between the above possibilities.

[0075] In the next step, step 402, a check is made as to whether the size of the prediction block of the currently scanned block is larger than an internal parameter representing the current size. If so, the currently scanned prediction block is larger than the maximum size of a residual block. This prediction block is a leaf block of a prediction subdivision or a tree block of a prediction subdivision and is not further partitioned. In this case, the process of FIG. 8 proceeds to step 300 of FIG. 7. That is, the currently scanned prediction block is partitioned into blocks of residual tree roots, and the first flag of the flag sequence of the first residual tree block in this currently scanned prediction block is decoded, e.g., in step 302.

[0076] However, if the size of the currently scanned prediction block is less than or equal to the internal parameter representing the current size, the process of Figure 8 proceeds to step 404, where the size of the prediction block is checked to determine whether it is equal to the internal parameter representing the current size, if so, the splitting step 300 may be skipped and the process proceeds directly to step 302 of Figure 7.

[0077] However, if the size of the currently scanned prediction block is smaller than the internal parameter representing the current size, the process of FIG. 8 proceeds to step 406, where the hierarchical level is incremented by one and the current size is set to the size of the new hierarchical level, such as divided by two (in both axes in case of a quadtree subdivision). The check of step 404 is then performed again. The effect of the loop formed by steps 404 and 406 is that the hierarchical level always corresponds to the size of the corresponding block to be partitioned, regardless of whether each prediction block is smaller or larger than the maximum size of the residual block. Thus, when decoding the flag in step 302, the context modeling is performed depending on both the hierarchical level to which the flag refers and the size of the block. The use of different contexts for flags of different hierarchical levels or block sizes has the following advantages: the probability estimates are more closely aligned to the distribution of real probabilities for the occurrence of the flag's value; and on the other hand, the number of contexts managed is relatively small. This results in a reduced overhead of context management and a better fit of the contexts to the statistics of the real symbols.

[0078] As mentioned above, there may be several sample arrays, which may be grouped into one or more plane groups. The input signal received at the input unit 32 and encoded may be, for example, an image of a video sequence or a still image. Thus, the image may be provided in the form of one or more sample arrays. In the context of encoding an image of a video sequence or a still image, a sample array may refer to three color planes such as red, green and blue, or luma and chroma planes such as YUV or YCbCr color representations. There may also be a sample array representing alpha, i.e. transparency, and / or depth information for 3D video material. A number of such sample arrays may be grouped together in so-called plane groups. For example, luma (Y) may be one plane group with only one sample array and chroma, such as CbCr, another plane group with two sample arrays. In another example, YUV may be one plane group with three matrices and depth information for 3D video material may be another plane group with only one sample array. For every plane group, one primary quadtree structure may be coded in the data stream 22 to represent the division into prediction blocks, and for each prediction block, a secondary quadtree structure may be coded representing the division into residual blocks. Thus, following the first example above, where the luma components form one plane group and the chroma components form the other plane group, there is one quadtree structure for prediction blocks of the luma plane, one quadtree structure for residual blocks of the luma plane, one quadtree structure for prediction blocks of the chroma plane and one quadtree structure for residual blocks of the chroma plane. However, in the second example above, there is one quadtree structure for prediction blocks of both luma and chroma (YUV), one quadtree structure for residual blocks of both luma and chroma (YUV), one quadtree structure for prediction blocks of depth information for 3D video material and one quadtree structure for residual blocks of depth information for 3D video material.

[0079] Furthermore, the above description describes how the input signal is divided into prediction blocks using a first-order quadtree structure and these prediction blocks are further subdivided into residual blocks using a sub-quadtree structure. In an alternative embodiment, the subdivision does not have to end at the sub-quadtree stage, i.e. the blocks resulting from the division using the sub-quadtree structure may be further subdivided using a third quadtree structure. This division may be performed with the aim of using further coding tools that facilitate the encoding of the residual signal.

[0080] The above description focuses on the subdivision performed by the subdivision units 28 and 104a, respectively. As mentioned above, the subdivision is defined by the subdivision units 28, 104a, respectively, which control the granularity at which the above-mentioned modules of the encoder 10 and decoder 100 process. However, in the embodiment described below, the subdivision units 228, 104a are followed by a combiner 30 and a combiner 104b, respectively. However, the combiner units 30, 104b are optional and may be omitted.

[0081] However, in practice, as will be explained in more detail below, the combiner offers the encoder the opportunity to combine some of the prediction or residual blocks into groups or clusters, so that other modules, at least some of them, can handle these groups of blocks together. For example, if the signaling of a group of prediction blocks with the transmission of common parameters for all blocks belonging to this group is more promising in terms of rate / distortion than the signaling of prediction parameters for all of these prediction blocks individually, the predictor 12 may sacrifice small deviations between the prediction parameters of some of the prediction blocks, as determined by the optimization using the subdivision of the subdivision unit 28, and instead use prediction parameters common to all of these prediction blocks. However, the process of obtaining a prediction in the predictors 12 and 110 themselves based on these common prediction parameters may be performed for the prediction blocks. However, it is also possible that the predictor 12 and 110 performs the prediction process once for the entire group of prediction blocks.

[0082] As will be explained in more detail below, it is also possible that the grouping of prediction blocks is not only aimed at using the same or common prediction parameters for a group of prediction blocks, but also or in addition this allows the encoder 10 to transmit one prediction parameter for this group together with the residuals of the predictions of the prediction blocks belonging to this group, reducing the overhead of the signaling of the prediction parameters of this group. The process of coupling to the data stream insertion unit 18 has less impact than the decisions made by the residual precoder 14 and the prediction unit 12, but will be explained in more detail below. Strictly speaking, it should be noted that the above aspects also apply to other subdivisions, such as the subdivision of the residual and the subdivision of the filters mentioned above.

[0083] First, the combination of sets of samples such as the above-mentioned prediction block and residual block can be realized in various ways, i.e., not limited to the above-mentioned multi-tree subdivision, but the following description will focus on the combination of blocks resulting from the multi-tree subdivision in the above-mentioned embodiment.

[0084] In general, combining syntax elements associated with a particular set of samples for the purpose of transmitting associated coding parameters allows the rate of ancillary information to be reduced in image and video coding. For example, the sample array of the signal to be encoded is usually partitioned into a particular set of samples. The particular set of samples may represent a rectangular or square block, or another set of samples including an area of ​​any shape, such as a triangle. In the above embodiment, the simply connected regions were the prediction blocks and residual blocks resulting from the subdivision of the multi-tree. The subdivision of the sample array may be fixed by the syntax, and the subdivision information may be transmitted, at least in part, in the bitstream, as described above. In order to keep the rate of ancillary information for transmitting the subdivision information small, the syntax usually allows only a limited number of simple partitioning choices, such as subdivision of blocks into smaller blocks. The sample sets are associated with particular coding parameters, which specify the information of the prediction, the mode of coding of the residual, etc. More details on this are given above. For each sample set, separate coding parameters may be transmitted, specifying the coding of the prediction and / or the residual. To achieve improved coding efficiency, the later described combining aspect, i.e. combining two or more sample sets into so-called groups of sample sets, provides several advantages. These advantages are further explained below. For example, sample sets may be combined such that all sample sets of such a group share the same coding parameters. The same coding parameters can be transmitted together with one of the sample sets in the group. By doing so, it is not necessary to transmit the coding parameters separately for each sample set of the group of sample sets, instead the coding parameters are transmitted only once for the whole group of sample sets. As a result, the rate of the side information for the transmission of the coding parameters is reduced, which may improve the overall coding efficiency. As an alternative approach, it is possible to transmit additional refinements for one or more coding parameters to one or more sample sets of the group of sample sets.Refinement can be applied either to all of the sample sets of a group, or only to the sample set that is transmitted.

[0085] The combining aspect, which will be further described below, also allows the encoder more freedom in creating the bitstream 22, since the combining approach significantly increases the options for partitioning the image sample array. This can improve the coding efficiency, since the encoder can choose from more options, such as to minimize a certain rate / distortion amount. There are several options for the encoder's operation. In a simple approach, the encoder first determines the optimal subdivision of the sample array. With brief reference to FIG. 1, the subdivision unit 28 can determine the optimal subdivision in a first step. Then, for each sample set, a check is made to see whether combining with another sample set or another group of sample sets reduces a certain rate / distortion amount. The prediction parameters associated with the group of combined sample sets can now be re-evaluated, such as by performing a new motion search or by using the prediction parameters already determined for the common sample set. Then, the sample sets or groups of sample sets that are candidates for combining are evaluated for the group of sample sets under consideration. In a detailed approach, a particular rate / distortion penalty is evaluated for a group of additional candidate sample sets.

[0086] The combining approach described below does not change the order of processing of the sample sets, i.e. the combining idea can be implemented in such a way that there is no increase in latency, i.e. each sample set can be decoded at the same time as without using the combining approach.

[0087] For example, the combining approach results in increased coding efficiency if the bit rate saved by reducing the number of coded prediction parameters is greater than the bit rate otherwise used in coding the merge information to indicate the combining to the decoding side, as described further below. The described syntax extension for combining also gives the encoder more freedom in choosing the partitioning of the image or plane group into blocks. In other words, the encoder is not restricted to first perform the subdivision and then check whether some of the resulting blocks have the same or similar set of prediction parameters. As a simple alternative, the encoder first makes the subdivision decision depending on the rate / distortion loss and then checks for each block whether combining with one of the neighboring blocks or associated determined groups of blocks reduces the rate / distortion loss. The prediction parameters associated with the new group of blocks can now be re-evaluated, such as by performing a new motion search or with the determined prediction parameters for the current block, and the neighboring blocks or groups of blocks can be evaluated for the new group of blocks. The combining information can be signaled on a block-by-block basis. Effectively, the combination can also be understood as an estimation of the prediction parameters of the current block, where the estimated prediction parameters are set equal to the prediction parameters of one of the neighboring blocks. Alternatively, for blocks within a group of blocks, the residuals may be transmitted.

[0088] Thus, the basic idea underlying the merging, which will be described further below, is to reduce the bit rate required for the transmission of prediction parameters or other coding parameters by merging adjacent blocks into groups of blocks. Each group of blocks is then associated with a unique set of coding parameters, such as prediction parameters or residual coding parameters. The merging information, if present, is transmitted in the bitstream in addition to the subdivision information. The advantage of the merging idea is the increased coding efficiency resulting from the reduced rate of information associated with the coding parameters. The merging process described here can also be extended to dimensions other than the spatial dimension. For example, groups of respective sets of exemplars or blocks present in several different video images can be merged into one group of blocks. The merging can also be applied to 4D compression and light field coding.

[0089] Thus, returning to the description of figures 1 to 8, regardless of the specific way in which the subdivision units 28, 104a each subdivide the image, the subdivision followed by the combination process is advantageous. In particular, the latter can also subdivide the image in a similar way as H.264, for example. That is, each image can be subdivided into a regular array of rectangular or square macroblocks of a given size, such as 16x16 luminance samples, a size that is signalled in the data stream. Each macroblock has specific coding parameters associated with it, including among others partitioning parameters and corresponding prediction parameters in the data stream. The partitioning parameters define for each macroblock a partitioning into a regular subgrid partitioned with a number of times 1, 2, 4 etc., which is the prediction granularity. Also, a partitioning for the residual and a corresponding granularity of the transformation of the residual are defined.

[0090] In any case, the combination provides the advantages outlined above, such as the reduction of the bit rate of the associated information in image and video coding. A particular set of samples, representing a square or rectangular block, or an area of ​​any shape, or any other set of samples, such as any simply connected region or sample, is usually connected with a particular set of coding parameters. For each set of samples, coding parameters are included in the bitstream, representing, for example, prediction parameters that specify how to predict the corresponding set of samples using the coded samples. The partitioning of the sample array of the image into sets of samples may be fixed by the syntax or may be conveyed by corresponding subdivision information in the bitstream. The coding parameters of the sets of samples may be transmitted in a predefined order, as specified by the syntax. The combination unit 30 may inform a group of sets of samples of a common set of samples or of a current block, such as a prediction block or a residual block, combined with one or more other sets of samples according to the function of the combination. Thus, the coding parameters of a group of sets of samples need only be transmitted once. In certain embodiments, if the current sample set is combined with an existing group of sample sets or sample sets for which coding parameters have already been transmitted, the coding parameters of the current sample set are not transmitted. Instead, the coding parameters of the current sample set are set equal to the coding parameters of the sample set or group of sample sets to which the current sample set is combined. As an alternative approach, additional refinements for one or more coding parameters can be transmitted for the current sample set. The refinements can apply either to all sample sets of the group or only to the sample set for which the refinements are transmitted.

[0091] In an embodiment, the set of all coded / decoded sets of samples for each set of samples such as the above-mentioned prediction block, the above-mentioned residual block, the above-mentioned leaf blocks of the multi-tree subdivision, etc. is called the "set of sets of causal samples". See for example Fig. 3C. All blocks shown in this figure are the result of a certain subdivision such as prediction subdivision, residual subdivision, or multi-tree subdivision, etc., and the coding / decoding order defined between these blocks is defined by the arrows 350. If a certain one of these blocks is considered as the current set of samples or the current simply connected region, then its set of causal sample sets consists of all blocks that precede the current block in the order 350. However, other subdivisions that do not use multi-tree subdivisions are also possible as long as the combination principle described below is concerned.

[0092] The set of samples available for combination with the current sample set is referred to below as the "set of candidate sample sets" and is always a subset of the "set of causal sample sets". The method of forming the subset can be signalled to the decoder or specified in the data stream or bitstream from the encoder to the decoder. When a particular current sample set is encoded / decoded and its set of candidate sample sets is not empty, the encoder signals in the data stream and the decoder derives from the data stream whether a common sample set is to be combined with one of the sample sets of this candidate sample set or not. If so, they are signalled together. Otherwise, the set of candidate sample sets is empty and no combination is available for this block.

[0093] There are various ways to determine a subset of the causal sample set that represents a set of candidate sample sets. For example, the determination of the candidate sample set may be based on a uniquely geometrically defined sample in the current sample set, such as a top-left image sample of a rectangular or square block. Starting from this uniquely geometrically defined sample, a non-zero number of specific samples is determined that represents direct spatial neighbors of this uniquely geometrically defined sample. For example, this non-zero number of specific samples includes the top neighbor and the left neighbor of the uniquely geometrically defined sample of the current sample set. The non-zero number of neighbors is at most two, 1 if one of the top or left neighbors is unavailable or outside the image, and zero if neither neighbors are both neighbors.

[0094] A set of candidate sample sets is then determined to encompass the set of samples that includes at least one of the non-zero number of adjacent samples. For example, see FIG. 9A. Let the current sample set under consideration for merging be block X, and its unique geometrically defined sample be the top left sample illustrated at 400. The top and left adjacent samples of sample 400 are shown at 402 and 404. The set of causal sample sets or causal blocks are highlighted with shading. Of these blocks, blocks A and B include one of the adjacent samples 402 and 404, and therefore these blocks form a set of candidate blocks or a set of candidate sample sets.

[0095] In another embodiment, the set of candidate sample sets determined for combining purposes may additionally or exclusively include sample sets that include a non-zero number of samples, such as 1 or 2, in the same spatial location but in different images, i.e., previously encoded / decoded images. For example, in addition to blocks A and B of FIG. 9A, a block of a previously encoded image that includes samples in the same location as sample 400 may be used. Note that only the top neighbor sample 404 or only the left neighbor sample 402 may be used to define the above non-zero number of neighboring samples. In general, the set of candidate sample sets may be derived from processed data in the current image or other images. This derivation may include spatial directional information, such as transform coefficients associated with a particular direction or tilt of the current image, and may also include temporal directional information, such as a representation of neighboring motion. The set of candidate sample sets may be derived from such data available at the receiver / decoder, and, if present, other data and associated information in the data stream.

[0096] Both the encoder-side combiner 30 and the decoder-side combiner 104b derive candidate sample sets in parallel. As mentioned above, they may both determine the set of candidate sample sets independently of each other, based on a predefined methodology communicated to both. Alternatively, the encoder may signal hint information in the bitstream, which places combiner 104b in a position to derive these candidate sample sets in the same way that encoder-side combiner 30 determined the set of candidate sample sets.

[0097] As will be described in detail below, the combiner 30 and the data stream inserter 18 cooperate to transmit one or more syntax elements for each sample set, which specify whether the sample set is to be combined with another sample set that is part of the group of previously combined sample sets, and the set of candidate sample sets to be used for the combination. The extractor 102 then extracts these syntax elements and informs the combiner 104b accordingly. In particular, in the specific embodiment described below, one or two syntax elements are transmitted to specify the combining information for a particular sample set. The first syntax element specifies whether the current sample set is to be combined with another sample set. The second syntax element is transmitted only if the first syntax element specifies that the current sample set is to be combined with another sample set, and specifies the set of candidate sample sets to be used for the combination. If the derived set of candidate sample sets is empty, the first syntax element may not be transmitted. In other words, the first syntax element may be transmitted only if the derived set of candidate sample sets is not empty. The second syntax element may be transmitted only if the derived set of candidate sample sets contains more than one sample set, since if the set of candidate sample sets contains only one sample set, no further selection is possible. Furthermore, if the set of candidate sample sets contains more than one sample set, but all sample sets of the candidate sample sets are associated with the same coding parameter, the second syntax element may not be transmitted. In other words, the second syntax element may be transmitted only if at least two sample sets of the derived candidate sample sets are associated with different coding parameters.

[0098] In the bitstream, the combining information for a sample set may be coded before the prediction parameters or other specific coding parameters associated with that sample set, which may be transmitted only if the combining information conveys that the current sample set is not combined with other sample sets.

[0099] The joint information for a particular sample set, i.e. block, etc., may be coded after the true subset of prediction parameters, or in a more general sense, the coding parameters associated with the respective sample set, have been transmitted. The subset of prediction / coding parameters may consist of one or more reference image indices, or one or more components of a vector of motion parameters, or a reference index and one or more components of a vector of motion parameters, etc. The transmitted subset of prediction or coding parameters may be used to derive a set of candidate sample sets from the set of provisional candidate sample sets derived above. For example, a difference magnitude or distance between the coded prediction and coding parameters of the current sample set and the corresponding prediction and coding parameters of a preliminary set of candidate sample sets based on a predefined distance criterion may be calculated. Then, only sample sets for which the calculated difference magnitude or distance is less than or equal to a predefined or derived threshold are included in the final, i.e. reduced, set of candidate sample sets. See for example FIG. 9A. Let the current sample set be block X. Let a subset of coding parameters related to this block have already been inserted in the data stream 22. For example, assume that block X is a predictive block and the true subset of coding parameters was a subset of prediction parameters for this block X, such as a subset of a set including image reference indices, and motion mapping information, such as motion vectors. If block X is a residual block, the subset of coding parameters is a subset of residual information, such as a map representing the locations of transform coefficients and significant transform coefficients in block X. Based on this information, both the data stream insertion unit 18 and the extraction unit 102 can use this information to determine subsets of blocks A and B, which in this particular embodiment form a preliminary set of candidate sample sets as mentioned above. In particular, since blocks A and B belong to the set of causal sample sets, their coding parameters are available to both the encoder and the decoder when the coding parameters of block X are encoded / decoded.Thus, using the comparison of the magnitude of the difference, any number of blocks of the preliminary sets of candidate sample sets A and B may be excluded. The resulting reduced set of candidate sample sets may then be used as described above, i.e. to determine whether, depending on the number of sample sets in the reduced set of candidate sample sets, a merge indicator representing the join is transmitted in the data stream or extracted from the data stream, and whether a second syntax element needs to be transmitted in the data stream or extracted from the data stream, which indicates which sample sets in the reduced set of candidate sample sets are to be the join partner blocks. That is, the determination of the combination for a given simply connected region or the transmission of the syntax element of the respective combination may be performed based on the number of simply connected regions which are in a predetermined relative position to the given simply connected region and which at the same time are associated with coding parameters which satisfy a predetermined relationship to the first subset of coding parameters for the given simply connected region, and the adoption or prediction with extraction of the prediction residual may be performed on the second subset of coding parameters for the given simply connected region, that is, only one of the number of simply connected regions which are in a predetermined relative position to the given simply connected region and which at the same time are associated with coding parameters which satisfy a predetermined relationship to the first subset of coding parameters for the given simply connected region or a subset of the identified coding parameters may be adopted from the second subset of the given simply connected region and used for the prediction of the second subset of the given simply connected region.

[0100] The threshold against which the distance is compared may be fixed and communicated to both the encoder and the decoder, or may be derived based on a calculated distance, such as the median difference value or other central tendency. In this case, it is inevitable that the reduced set of candidate sample sets will be a proper subset of the preliminary set of candidate sample sets. Alternatively, only those sample sets are selected from the preliminary set of candidate sample sets that have the smallest distance based on the distance size. Alternatively, a set of samples is selected from the preliminary set of candidate sample sets using the distance size. In the latter case, the merge information only needs to specify whether the current sample set is merged with a single candidate sample set or not.

[0101] Thus, a set of candidate blocks is formed or derived as described below with reference to Fig. 9A. Starting from the top left sample position 400 of the current block X in Fig. 9A, the positions of its left neighbor sample 402 and its top neighbor sample 404 are derived at the encoder and decoder side. Thus, the set of candidate blocks can have at most only two members, i.e. blocks that contain one of the two sample positions of the shaded set of causal blocks in Fig. 9A, which in the case of Fig. 9 are blocks B and A. Thus, the set of candidate blocks can only have as its members the two direct neighbors of the top left sample position of the current block. In another embodiment, the set of candidate blocks is formed by all blocks that have been coded before the current block and that contain one or more samples that represent direct spatial neighbors of any sample of the current block. The direct spatial neighbors may be limited to the immediate neighbors to the left, and / or the immediate neighbors to the top, and / or the immediate neighbors to the right, and / or the immediate neighbors to the bottom of any example of the current block. For example, see FIG. 9B, which shows another block subdivision. In this case, the candidate blocks include four blocks, namely blocks A, B, C, and D.

[0102] The set of candidate blocks may additionally or exclusively include blocks that contain one or more samples in the same position as any sample in the current block, but in a different image, i.e., an image that has already been coded / decoded.

[0103] The set of candidate blocks also represents a subset of the set of blocks mentioned above, determined by neighboring relations in spatial or temporal direction. The subset of candidate blocks may be fixed, signaled or derived. The derivation of the subset of candidate blocks may take into account decisions about other blocks in the image or other images. For example, blocks associated with the same coding parameters or coding parameters that are much more similar than other candidate blocks may not be included in the set of candidate blocks.

[0104] The following description of the embodiment applies when only a maximum of two blocks containing adjacent exemplars to the left and above the top-left exemplar of the current block are considered as candidates.

[0105] If the set of candidate blocks is not empty, a flag called merge_flag is signaled, which specifies whether the current block is to be combined with one of the candidate blocks. If merge_flag is 0 (indicating "false"), this block is not combined with one of the candidate blocks and all coding parameters are transmitted as usual. If merge_flag is 1 (indicating "true"), the following applies: If the set of candidate blocks contains only one block, this candidate block is used for combining. Otherwise, the set of candidate blocks contains two blocks. If the prediction parameters of these two blocks are identical, these prediction parameters are used for the current block. Otherwise (if the two blocks have different prediction parameters), a flag called merge_left_flag is signaled. If merge_left_flag is 1 (indicating "true"), the block containing the adjacent sample position to the left of the top-left sample position of the current block is selected from the set of candidate blocks. If merge_left_flag is 0 (indicating "false"), another (i.e., top-neighboring) block is selected from the set of candidate blocks, and the prediction parameters of the selected block are used for the current block.

[0106] To summarise the several embodiments described above with respect to merging, reference is made to FIG. 10 which illustrates the steps performed by the extractor 102 to extract merging information from the data stream 22 input to the input 116 .

[0107] The process begins with identifying 450 candidate blocks or sample sets for the current sample set or block. Since the coding parameters for the blocks are transmitted in a particular one-dimensional order in the data stream 22, Figure 10 describes the process of obtaining joint information for the currently scanned sample set or block.

[0108] As mentioned above, the identification and step 450 may include an identification from the set of decoded blocks, i.e. causal blocks, based on a neighboring relation point of view. For example, neighboring blocks containing certain neighboring samples, which are spatially or temporally neighboring with one or more geometrically predetermined samples of the current block X, may be nominated candidates. Furthermore, the identification step may include two stages: a first stage including the above identification to obtain a preliminary set of candidate blocks, i.e. an identification based on neighboring relations, and a second stage in which only blocks decoded from the data stream before step 450 are nominated candidates whose transmitted coding parameters satisfy a certain relation with the proper subset of coding parameters of the current block X.

[0109] The process then proceeds to step 452, where it is determined whether the number of candidate blocks is greater than zero. If so, in step 454, merge_flag is extracted from the data stream. The extraction step 454 may include entropy decoding. The context for the entropy decoding of merge_flag in step 454 may be determined based on syntax elements belonging to a set of candidate blocks or a preliminary set of candidate blocks, for example. In that case, the dependency on the syntax elements may be limited to the information whether a block belonging to the set was a candidate for merging or not. An estimate of the probability of the selected context may be adapted.

[0110] On the other hand, if in step 452 the number of candidate blocks is determined to be zero, the process of Figure 10 proceeds to step 456, where the coding parameters of the current block are extracted from the bitstream, or, in the particular case of the two stages described above, the remaining coding parameters are extracted, after which extraction unit 102 proceeds to process the next block in the order of block scanning, such as order 350 shown in Figure 3C.

[0111] Returning to step 454, after the extraction of step 454, the process continues with step 458, where it is checked whether the extracted merge_flag indicates whether a merge of the current block occurs or not. If not, the process continues with step 456 described above. If a merge does occur, the process continues with step 460, where it is checked whether the number of candidate blocks is 1 or not. If so, there is no need to indicate and transmit a particular one of the candidate blocks, so the process of FIG. 10 continues with step 462, where the merge partner of the current block is set to its only candidate block. Then, in step 464, the coding parameters of the merged partner block are used for the adaptation or prediction of the coding parameters or the remaining coding parameters of the current block. In the adaptation case, the missing coding parameters of the current block are simply copied from the merge partner block. In the other case, i.e. in the prediction case, step 464 may further involve the extraction of residual data from the data stream. This residual data relates to the residual of the prediction of the missing coding parameters of the current block. This residual data may then be combined with predictions of these missing coding parameters obtained from the blocks with which it is combined.

[0112] On the other hand, if it is determined in step 460 that the number of candidate blocks is greater than one, the process of Fig. 10 proceeds to step 466, where it is checked whether the coding parameters or the corresponding parts of the coding parameters, i.e. the parts related to the parts not yet transmitted in the data stream for the current block, are identical to each other. If so, then in step 468 these common coding parameters are set as a reference for combination. Alternatively, the candidate block is set as a partner for combination. Then, in step 464, each corresponding coding parameter is used for adaptation or prediction.

[0113] The combination partner may itself be the block for which the combination was signaled, in which case the adopted or predicted coding parameters of the combination partner are used in step 464.

[0114] On the other hand, if not, i.e., if the coding parameters are not identical, the process of FIG. 10 proceeds to step 470, where the syntax element merge_left_flag is further extracted from the data stream. A separate set of contexts may be used for entropy decoding of this flag. Also, the set of contexts used for entropy decoding of merge_left_flag may consist of only one context. After step 470, the candidate block suggested by merge_left_flag is set as the merge partner in step 472 and used for adaptation or prediction in step 464. After step 464, the extraction unit 102 proceeds to process the next block in the block order.

[0115] Of course, there are many alternatives. For example, instead of the separate syntax elements of merge_flag and merge_left_flag described above, a syntax element may be combined and transmitted in the data stream to signal the operation of the merge in the combined syntax element. Also, the above merge_left_flag may be transmitted in the data stream regardless of whether the two candidate blocks have the same prediction parameters, thereby reducing the computational overhead for performing the operation of FIG. 10.

[0116] As already shown in FIG. 9B, the set of candidate blocks may include three or more blocks. Also, the information of the combination, i.e., information conveying whether the block is combined or not (and if so, combined with the candidate block), may be conveyed in one or more syntax elements. One syntax element, such as the above-mentioned merge_flag, may specify whether the block is combined with one of the candidate blocks. The flag may be transmitted only if the set of candidate blocks is not empty. A second syntax element, such as the above-mentioned merge_left_flag, may convey which candidate block is used for the combination. Typically, a selection from two or more candidate blocks is indicated. The second syntax element may be conveyed only if the first syntax element conveys a signal indicating that the current block is combined with one of the candidate blocks. Furthermore, the second syntax element may be transmitted only if the set of candidate blocks includes more than one candidate block and / or if one of the candidate blocks has a different prediction parameter than the other candidate blocks. The syntax depends on the number of candidate blocks and / or on how different the prediction parameters associated with the candidate blocks are.

[0117] The syntax that conveys which of the candidate blocks is to be used may be set at the encoder and decoder side simultaneously and / or in parallel. For example, if there are three choices for the candidate block identified in step 450, the syntax is selected such that only these three choices are available and considered for entropy coding, e.g., in step 470. In other words, the syntax elements are selected such that the symbol alphabet has as many elements as there are candidate block choices. All other choices may be considered as zero possibilities, and the entropy coding / decoding may be adjusted at the encoder and decoder simultaneously.

[0118] Furthermore, as mentioned in step 464, the prediction parameters estimated as a result of the combining process may represent the complete set of prediction parameters associated with the current block, or may represent a subset of these prediction parameters, such as prediction parameters for one hypothesis of a block where multiple hypothesis predictions are performed.

[0119] As mentioned above, the syntax elements related to the merge information can be entropy coded using context modeling. The syntax elements may consist of the above-mentioned merge_flag and merge_left_flag (or similar syntax elements). In a specific example, one of three context models or contexts can be used for coding / decoding merge_flag, such as in step 454. The index of the context model to be used, merge_flag_ctx, may be derived as follows: if the set of candidate blocks contains two elements, the value of merge_flag_ctx is equal to the sum of the values ​​of merge_flag of the two candidate blocks. On the other hand, if the set of candidate blocks contains one element, the value of merge_flag_ctx may be equal to twice the value of merge_flag of this one candidate block. Since each merge_flag of the adjacent candidate blocks can be 1 or 0, three contexts are available for merge_flag. The merge_left_flag may be coded using only a single probability model.

[0120] Also, in other embodiments, a different context model may be used, for example, non-binary syntax elements may be mapped to a sequence of binary symbols called bins. A context model for some syntax elements or bins of syntax elements that define the binding information may be derived based on the transmitted syntax elements of adjacent blocks or the number of candidate blocks, etc., and other syntax elements or bins of syntax elements may be coded with a fixed context model.

[0121] With respect to the above description of combining blocks, the set of candidate blocks may be derived in a manner similar to any of the above-mentioned embodiments, with the following modifications: Restrict the candidate blocks to blocks using motion compensated prediction or inter-prediction, respectively. Only such blocks can be members of the set of candidate blocks. Merging information propagation and context modeling can be done as described above.

[0122] Returning to the combination of the above-mentioned embodiment of multi-tree subdivision with the currently described aspect of combining. When a quadtree-based subdivision structure is used to divide an image into square blocks of variable size, the syntax elements specifying the combining, e.g. merge_flag and merge_left_flag, are alternated with the prediction parameters transmitted for each leaf node of the quadtree structure. For example, consider again FIG. 9A. FIG. 9A shows an example of a quadtree-based image subdivision into prediction blocks of variable size. The top two blocks of the largest size are called treeblocks, i.e. prediction blocks of the largest possible size. The other blocks in this figure are obtained by subdividing the corresponding treeblock. The current block is indicated by an "X". All shaded blocks are encoded / decoded before the current block, thus forming a set of causal blocks. As detailed in the description of the derivation of the set of candidate blocks for one of the embodiments, only blocks containing samples that are directly adjacent (i.e., to the top or to the left) to the position of the top left sample of the current block can be elements of the set of candidate blocks. Thus, the current block can be combined either with block "A" or with block "B". If merge_flag is 0 (indicating "false"), the current block "X" is not combined with either of the two blocks. If blocks "A" and "B" have identical prediction parameters, no distinction needs to be made, since combining with either of the two blocks will give the same result. Therefore, merge_left_flag is not transmitted in this case. Otherwise, if blocks "A" and "B" have different prediction parameters, then if merge_left_flag is 1 (indicating "true"), blocks "X" and "B" are combined, whereas if merge_left_flag is 0 (indicating "false"), blocks "X" and "A" are combined. In another preferred embodiment, additional adjacent (already transmitted) blocks represent candidates for combining.

[0123] Another example is shown in Fig. 9B, where the current block "X" and the left neighbor block "B" are treeblocks, i.e. of the maximum allowed block size. The size of the top neighbor block "A" is one quarter of the size of the treeblock. Blocks that are members of the set of causal blocks are shaded. In one preferred embodiment, the current block "X" can only be combined with the two blocks "A" or "B" and not with any other top neighbor. In another preferred embodiment, additional neighboring (already transmitted) blocks represent candidates for combining.

[0124] Before proceeding to the description of the aspects of handling different sample arrangements of an image according to an embodiment of the present application, the above description of the aspects of the multi-tree subdivision and the combination on the other hand makes it clear that these aspects provide advantages that can be used independently of each other. That is to say, as already explained above, the combination of the multi-tree subdivision and the combination provides specific advantages. However, advantages can also be obtained in other ways. For example, the function of combination is embodied in the subdivision performed by the subdivision unit 30, 104a in response to the subdivision of macroblocks that is not based on a quad-tree or multi-tree subdivision, but on the regular partitioning of these macroblocks into smaller sections. On the other hand, the combination of the use of the multi-tree subdivision with the transmission in the bitstream of an indication of the maximum size of the treeblocks and the use of the multi-tree subdivision with the transmission of the corresponding coding parameters of the blocks using a depth-order traversal order provides advantages, regardless of whether the function of combination is used in parallel or not. In general, the advantages of combination can be understood intuitively when considering the following: The coding efficiency is improved if the syntax for coding the sample arrays is extended in a way that allows not only subdivision of a block but also the combination of two or more blocks obtained after subdivision. This results in a group of blocks that are coded with the same prediction parameters. The prediction parameters for such a group of blocks need to be coded only once. Also, regarding the combination of sample sets, the possible sample sets are rectangular or square blocks, and the combined sample sets are collections of rectangular or square blocks. The possible sample sets are also image regions of any shape, and the combined sample sets are collections of image regions of any shape.

[0125] In the following, the focus is on handling different sample arrays of images when there are multiple sample arrays per image. The aspects outlined below provide advantages regardless of the type of subdivision used, i.e. subdivision based on multi-tree subdivision or not, and regardless of whether merging is used or not. Before proceeding to a description of specific embodiments for handling different sample arrays of an image, a short introduction on handling different sample arrays per image is provided before the main subject of these embodiments.

[0126] The following will focus on coding parameters between blocks of different sample arrays of an image in image or video coding. In particular, the following will focus on how to adaptively predict coding parameters between different sample arrays of an image in environments including, for example, the encoders and decoders of Figs. 1 and 2, respectively, but also other image or video coding environments. As mentioned above, the sample arrays can represent sample arrays related to different color components or sample arrays that associate an image with additional information such as transparency data or depth maps. Sample arrays related to color components of an image are also referred to as color planes. The techniques described below, also referred to as inter-plane hiring / prediction, can be used in block-based image and video encoders and decoders, where the order in which blocks of the image's sample arrays are processed is arbitrary.

[0127] Typically, image and video coders are designed to code color images (either still images or images from a video sequence). A color image consists of multiple color planes that represent the sample arrays of different color components. Often, a color image is coded as a set of sample arrays consisting of a luma plane and two chroma planes that specify the chrominance components. In some applications, it is also common for the set of coded sample arrays to consist of three color planes that represent the sample arrays of the three primary colors red, green, and blue. In addition, color images may consist of four or more color planes for improved color representation. In addition, an image may have associated auxiliary sample arrays that specify additional information about the image. For example, such auxiliary sample arrays may specify the transparency of the associated color sample array (suitable for certain display purposes) or a depth map (suitable for multi-view rendering, such as 3D displays).

[0128] In conventional image and video coding standards (such as H.264), the color planes are usually coded jointly. In that case, certain coding parameters such as prediction mode, reference index, motion vectors of the macroblock and submacroblocks are used for all color components of one block. The luma plane can be considered as the primary color plane where certain coding parameters are specified in the bitstream, and the chroma plane can be considered as the secondary plane where the corresponding coding parameters are estimated from the primary luma plane. Each luma block is associated with two chroma blocks representing the same area in the image. Depending on the chroma sampling format used, the chroma sample array for one block is smaller than the luma sample array. For each macroblock consisting of one luma component and two chroma components, the same partitioning is performed and partitioning into smaller blocks is performed (if the macroblock is subdivided). For each block (which may be the macroblock itself or a subblock of a macroblock) consisting of one block of luma samples and two blocks of chroma samples, the same set of prediction parameters, such as reference indices and motion parameters, and possibly an intra prediction mode, are used. In certain profiles of conventional video coding standards (such as the 4:4:4 profile of H.264), it is also possible to code different color planes of an image independently. In that configuration, it is possible to select macroblock partitioning, prediction modes, reference indices and motion parameters separately for the color components of a macroblock or a subblock. In conventional coding standards, either all color planes are coded using the same specific set of coding parameters (such as subdivision information and prediction parameters) or all color planes are coded completely independently of each other.

[0129] If the color planes are coded jointly, one set of subdivision and prediction parameters must be used for all color components of a block. This ensures that the collateral information is small, but may result in a less efficient coding compared to the case where they are coded independently. This is because using different block decomposition and prediction parameters for different color components may result in a smaller amount of rate / distortion loss. For example, using different motion vectors or reference frames for the chroma components may significantly reduce the energy of the residual signal for the chroma components, which may result in a more efficient overall coding. If the color planes are coded independently, coding parameters such as block partitioning, reference indexes, and motion parameters can be selected separately for each color component to optimize the coding efficiency for each color component. However, redundancy cannot be exploited between color components. Multiple transmissions of certain coding parameters increase the collateral information rate (compared to the case where they are coded jointly), which may negatively impact the overall coding efficiency. Furthermore, support for auxiliary sample sequences in modern video coding standards (e.g., H.264) is limited to encoding the auxiliary sample sequence using a unique set of coding parameters.

[0130] Thus, in all the embodiments described so far, the planes of the image can be treated as described above. However, as also described above, the overall coding efficiency may be improved when coding multiple sample arrays (which may be associated with different color planes and / or auxiliary sample arrays) if a block-wise decision can be made, for example whether all sample arrays of a block are coded with the same coding parameters or whether different coding parameters are used. The basic idea of ​​the following inter-plane prediction is to allow such a block-wise tailored decision. The encoder can select, for example based on a rate / distortion criterion, whether all or some of the sample arrays of a particular block are coded using the same coding parameters or whether different coding parameters are used for different sample arrays. This selection can also be achieved by signaling whether for a particular block of the sample array a particular coding parameter is estimated from a previously coded block at the same location of a different sample array. Different sample arrays for an image can also be arranged in groups, which are called sample array groups or plane groups. Each plane group can contain one or more sample arrays of the image. Then, the blocks of sample arrays in a plane group share the same selected coding parameters, such as subdivision information, prediction mode, mode of coding of residuals, etc., while other coding parameters, such as transform coefficient levels, are transmitted separately for each sample array in a plane group. A plane group is coded as a primary plane group, i.e., no coding parameters are estimated or predicted from other plane groups. For each block of a secondary plane group, it can be selected on an ad-hoc basis whether a new set of selected coding parameters is transmitted or whether the selected coding parameters are estimated or predicted from the primary plane group or another secondary plane group. The decision of whether the selected coding parameters for a particular block are estimated or predicted is included in the bitstream.Inter-plane prediction allows more freedom in choosing the trade-off between the rate of side information and the quality of prediction compared to state-of-the-art coding of images composed of multiple sample arrays. The advantage is an increased coding efficiency compared to conventional coding of images composed of multiple sample arrays.

[0131] By employing / predicting in a plane, image or video coders such as the above embodiments may be extended to allow tailoring of whether the set of selected coding parameters for a block of a color sample array or an auxiliary sample array or a set of color sample arrays and / or auxiliary sample arrays is estimated or predicted from previously coded blocks of other sample arrays in the same image, or whether the set of selected coding parameters for a block is coded independently without reference to the co-located blocks of other sample arrays in the same image. The decision of whether the set of selected coding parameters for a block of a sample array or for multiple sample array blocks is estimated or predicted may be included in the bitstream. Different sample arrays associated with one image do not have to be of the same size.

[0132] As mentioned above, the sample arrays associated with an image (sample arrays can represent color components and / or auxiliary sample arrays) may be arranged into two or more so-called plane groups, with each plane group consisting of one or more sample arrays. The sample arrays included in a particular plane group do not have to be of the same size. This arrangement into plane groups also includes the case where each sample array is coded separately.

[0133] Specifically, in an embodiment, for each block of a plane group, whether coding parameters specifying how the block is predicted are estimated or predicted from previously coded blocks of the same image in different plane groups, or whether these coding parameters are coded separately for the block, are selected accordingly. The coding parameters specifying how the block is predicted include one or more of the following coding parameters: a block prediction mode specifying what prediction is used for the block (intra prediction, inter prediction using a single motion vector and reference image, inter prediction using two motion vectors and reference images, inter prediction using a high-order, i.e. non-translational motion model and a single reference image, inter prediction using multiple motion models and reference images), an intra prediction mode specifying how an intra prediction signal is generated, an identifier specifying the number of prediction signals to be combined for generating a final prediction signal for the block, a reference index specifying a reference image to be used for motion compensation prediction, motion parameters specifying how to generate a prediction signal using the reference image (such as a displacement vector or an affine motion parameter), and an identifier specifying how to filter the reference image for generating a motion compensation prediction signal. Generally, a block can be associated with only a subset of the above coding parameters. For example, if the block prediction mode specifies that intra prediction is performed on the block, the coding parameters of the block can additionally include an intra prediction mode, but coding parameters such as reference indexes and motion parameters that specify how to generate a signal for inter prediction are not specified. Alternatively, if the block prediction mode specifies inter prediction, the associated coding parameters can additionally include reference indexes and motion parameters, but the intra prediction mode is not specified.

[0134] One of the two or more plane groups may be coded or indicated in the bitstream as a primary plane group. For all blocks of this primary plane group, coding parameters that specify how to generate a prediction signal are transmitted without reference to other plane groups of the same image. The remaining plane groups are coded as secondary plane groups. For each block of the secondary plane group, one or more syntax elements are transmitted that inform whether coding parameters that specify how to predict the block are estimated or predicted from co-located blocks of other plane groups, or whether a new set of these coding parameters is transmitted for the block. One of the one or more syntax elements may be called an inter-plane prediction flag or an inter-plane prediction parameter. If a syntax element signals that no estimation or prediction of the corresponding coding parameters is performed, a new set of the corresponding coding parameters for the block is transmitted in the bitstream. If a syntax element signals that an estimation or prediction of the corresponding coding parameters is performed, a co-located block of a so-called reference plane group is identified. The assignment of a reference plane group to a block may be configured in several ways. In one embodiment, a particular reference plane group is assigned to each secondary plane group, and this assignment can be fixed or can be signaled in a high-level syntactic construct such as a set of parameters, an access unit header, a picture header, a slice header, etc.

[0135] In a second embodiment, the reference plane group assignment is coded in the bitstream and signaled by one or more syntax elements that are coded for the block and specify whether the selected coding parameters are estimated or predicted, or coded separately.

[0136] To explain the above mentioned possibilities in relation to inter-plane prediction and the detailed embodiment below, reference is made to FIG. 11. FIG. 11 illustrates an image 500 consisting of three sample arrays 502, 504, 506. In FIG. 11, only the lower part of the sample arrays 502-506 is shown for the sake of clarity. The sample arrays are shown to correspond to each other spatially. The sample arrays 502-506 overlap each other along a direction 508. Moreover, when the samples of the sample arrays 502-506 are projected along the direction 508, the samples of all these sample arrays 502-506 correspond exactly to each other in spatial position. In other words, the planes 502 and 506 are spread horizontally and vertically to accommodate each other's spatial resolution and correspond to each other.

[0137] In the embodiment, all sample arrays of an image belong to the same part of the spatial scene, and the vertical and horizontal resolutions may be different for each of the sample arrays 502-506. Furthermore, for the purpose of illustration, the sample arrays 502, 504 belong to one plane group 510, while the sample array 506 belongs to another plane group 512. FIG. 11 also illustrates a case where the spatial resolution along the horizontal axis of the sample array 504 is twice the horizontal resolution of the sample array 502. Furthermore, the sample array 504 is a primary array with respect to the sample array 502. The sample array 502 is a subordinate array with respect to the primary array 504. As mentioned above, in this case, the subdivision into blocks of the sample array 504 determined by the subdivision unit 30 of FIG. 1 is adopted by the subordinate array 502. 11, the vertical resolution of sample array 502 is half the vertical resolution of primary array 504, so each block is divided into two horizontally aligned blocks, which, when measured in units of the sample positions in sample array 502, again become quadratic blocks.

[0138] 11, the subdivision selected for the sample array 506 is different from the subdivision of the other plane groups 510. As mentioned above, the subdivision unit 30 may select the subdivision of the array of pixels 506 separately, independent of the subdivision for the plane group 510. Of course, the resolution of the sample array 506 may also be different from the resolution of the planes 502, 504 of the plane group 510.

[0139] Here, when encoding the individual sample arrays 502-506, the encoder 10 may start with encoding the first array 504 of the plane group 510, for example in the manner described above. The blocks shown in FIG. 11 may be, for example, prediction blocks as described above. Alternatively, the blocks are residual blocks or other blocks that define the granularity for the definition of certain coding parameters. Inter-plane prediction is not limited to the quadtree or multitree subdivisions illustrated in FIG. 11.

[0140] After transmission of the syntax elements for the primary array 504, the encoder 10 may decide to declare the primary array 504 as the reference plane for the lower plane 502. The encoder 10 and the extractor 30 may each signal this decision via the bitstream 22. Alternatively, the relevance may be made clear since the sample array 504 is the primary array of the plane group 510, and this information may be part of the bitstream 22. In either case, for each block in the sample array 502, the inserter 18, or another module of the encoder 10 in conjunction with the inserter 18, may decide to suppress transmission of the coding parameters of this block in the bitstream and instead use in the bitstream for that block the coding parameters of the block at the same location in the primary array 504 instead, or to send a signal indicating that the coding parameters of the block at the same location in the primary array 504 are to be used as a prediction of the coding parameters of the current block of the sample array 502, and transmit only the residual data for the current block of the sample array 502 in the bitstream. If the decision is not taken, the coding parameters are transmitted in the data stream as usual. The decision is signalled in the data stream 22 for each block. On the decoder side, the extractor 102 uses the inter-plane prediction information for each block to obtain the coding parameters of the respective block of the sample array 502 accordingly, i.e. by estimating the coding parameters of the co-located block of the primary array 504. Alternatively, if the inter-plane adoption / prediction information suggests an inter-plane adoption / prediction, this is done by extracting residual data for that block from the data stream and combining this with the prediction obtained from the coding parameters of the co-located block of the primary array 504. Alternatively, this is done by extracting the coding parameters of the current block of the sample array 502 as usual, without regard to the primary array 504.

[0141] As mentioned above, the location of the reference plane is not limited to the same plane group as the block currently being inter-predicted. Thus, as mentioned above, the plane group 510 may be the primary plane group or the reference plane group for the secondary plane group 512. In this case, the bitstream may include a syntax element for each block of the sample array 506 indicating whether the above-mentioned adoption / prediction of the coding parameters of the co-located macroblock in either the primary plane group or the reference plane group 502, 504 is performed. In the latter case, the coding parameters of the current block of the sample array 506 are transmitted as usual.

[0142] The subdivision and / or prediction parameters for planes within a plane group may be identical, i.e. they are only coded once for the plane group (all secondary planes of a plane group derive their subdivision information and / or prediction parameters from the primary plane in the same plane group) and contextual prediction or estimation of the subdivision information and / or prediction parameters is done between plane groups. The reference plane group may be a primary plane group or a secondary plane group.

[0143] Since the subdivision of the primary sample array 504 is spatially performed by the subsample array 502, except for the further partitioning of the adopted leaf blocks into quadratic blocks, it is easy to understand that blocks of different planes in a plane group are co-located. In case of inter-plane adoption / prediction between different plane groups, this co-location may be defined to allow more freedom between the subdivision of these plane groups. Given a reference plane group, the co-located blocks in the reference plane group are identified. The co-located blocks and the reference plane group can be derived by the following process: A particular sample 514 in a current block 516 of one of the sample arrays 506 of the secondary plane group 512 is selected. This may be the top left sample of the current block 516, as illustrated by 514 in FIG. 11. It may also be a sample in the current block 516 that is close to the center of the current block 516 or any other sample in the current block, and is geometrically uniquely defined. The location of this selected sample 515 in the sample arrays 502, 504 of the reference plane group 510 is calculated. The location of the sample 514 in the sample arrays 502, 504 is shown in Figure 11 at 518, 520, respectively. Which plane 502, 504 of the reference plane group 510 is actually used may be pre-specified or may be signaled in the bitstream. The sample in the corresponding sample array 502 or 504 of the reference plane group 510 that is closest to the location 518, 520, respectively, is identified and the block containing this sample is selected as the co-located block in the sample arrays 502, 504, respectively. In the case of Figure 11, these are blocks 522, 524, respectively. Alternative approaches for identifying co-located blocks in other planes are described below.

[0144] In an embodiment, the coding parameters specifying the prediction for the current block 516 are fully estimated using the corresponding prediction parameters of the co-located blocks 522 / 524 in the different plane groups 510 of the same image 500 without transmitting any additional side information. This estimation is done by simply copying the corresponding coding parameters or by adapting the coding parameters to take into account the difference between the current plane group 512 and the reference plane group 510. For example, the adaptation may involve a correction of the motion parameters (such as a correction of the displacement vectors) to take into account the phase difference between the luma and chroma sample arrays. Also, the adaptation may involve a modification of the precision of the motion parameters (such as a modification of the displacement vector precision) to take into account the different resolution of the luma and chroma sample arrays. In another embodiment, one or more estimated coding parameters for specifying the generation of the prediction signal are not directly used for the current block 516 but are used as a prediction of the corresponding coding parameters for the current block 516 and a refinement of these coding parameters for the current block 516 is transmitted in the bitstream 22. For example, the estimated motion parameters are not used directly, but the motion parameter differences (such as the difference in displacement vectors) are coded in the bitstream, specifying the deviation between the motion parameters used for the current block 516 and the estimated motion parameters. At the decoder side, the actual motion parameters used are obtained by combining the differences between the estimated and transmitted motion parameters.

[0145] In another embodiment, the subdivision of blocks such as the above-mentioned prediction subdivision tree blocks into prediction blocks (i.e. blocks of samples for which the same set of prediction parameters is used) is estimated or predicted from previously coded co-located blocks of different plane groups of the same image, i.e. bit sequences of Fig. 6A or Fig. 6b, as appropriate. In the embodiment, one of the two or more plane groups is coded as a primary plane group. For all blocks of this primary plane group, subdivision information is transmitted without reference to other plane groups of the same image. The remaining plane groups are coded as secondary plane groups. For blocks of the secondary plane groups, one or more syntax elements are transmitted, signaling whether subdivision information is estimated or predicted from co-located blocks of other plane groups or whether subdivision information is transmitted in the bitstream. One of the one or more syntax elements may be called an inter-plane prediction flag or an inter-plane prediction parameter. If the syntax element signals that no estimation or prediction of subdivision information is performed, then the subdivision information of the block is transmitted in the bitstream without reference to other plane groups of the same image. If a syntax element signals that subdivision information is estimated or predicted, then a so-called co-located block of the reference plane group is identified. The assignment of the reference plane group to the block can be configured in several ways. In one embodiment, a specific reference plane group is assigned to each of the secondary plane groups. This assignment can be fixed or signaled in a high-level syntax structure such as a set of parameters, an access unit header, a picture header, a slice header, etc. In a second embodiment, the assignment of the reference plane group is coded in the bitstream and signaled by one or more syntax elements. This syntax element is coded for the block and specifies whether the subdivision information is estimated or predicted or coded separately. The reference plane group can be the primary plane group or another secondary plane group.Given a reference plane group, co-located blocks in the reference plane group are identified. Co-located blocks are blocks in the reference plane group that correspond to the same image area as the current block or represent blocks in the reference plane group that share the largest portion of the image area with the current block. The co-located blocks can be partitioned into smaller prediction blocks.

[0146] In other embodiments, subdivision information for the current block, such as the quadtree-based subdivision information of Fig. 6A or Fig. 6b, is fully estimated using subdivision information of co-located blocks in different plane groups of the same image without transmitting additional accompanying information. As one particular example, if the co-located block is partitioned into two or four prediction blocks, the current block is also partitioned into two or four sub-blocks for prediction purposes. As another particular example, if the co-located block is partitioned into four sub-blocks and one of these sub-blocks is further partitioned into four smaller sub-blocks, the current block is also partitioned into four sub-blocks and one of these sub-blocks (corresponding to the sub-blocks of the co-located block that are further decomposed) is also partitioned into four smaller sub-blocks. In other preferred embodiments, the estimated subdivision information is not used directly for the current block, but is used as a prediction of the actual subdivision information for the current block and the corresponding refinement information is transmitted in the bitstream. For example, the subdivision information estimated from the co-located block may be further refined. For each sub-block that corresponds to a sub-block in a co-located block that is not partitioned into smaller blocks, a syntax element can be coded in the bitstream that specifies whether the sub-block is further decomposed in the current plane group. The transmission of such a syntax element can be conditioned on the size of the sub-block. Alternatively, a signal can be sent in the bitstream indicating that a sub-block that is further partitioned in the reference plane group is not partitioned into smaller blocks in the current plane group.

[0147] In another embodiment, both the subdivision of a block into predictive blocks and the coding parameters specifying how its sub-blocks are predicted are adaptively estimated or predicted from previously coded co-located blocks of different plane groups of the same image. In a preferred embodiment of the invention, one of the two or more plane groups is coded as a primary plane group. For all blocks of this primary plane group, subdivision information and prediction parameters are transmitted without reference to other plane groups of the same image. The remaining plane groups are coded as secondary plane groups. For blocks of the secondary plane groups, one or more syntax elements are transmitted that signal whether subdivision information and prediction parameters are estimated or predicted from co-located blocks of other plane groups or whether subdivision information and prediction parameters are transmitted in the bitstream. One of the one or more syntax elements may be called an inter-plane prediction flag or an inter-plane prediction parameter. If the syntax element signals that no estimation or prediction of subdivision information and prediction parameters is performed, the subdivision information of the block and the prediction parameters for the resulting sub-blocks are transmitted in the bitstream without reference to other plane groups of the same image. If a syntax element signals that the subdivision information and prediction parameters for a subblock are estimated or predicted, then a so-called co-located block of a reference plane group is identified. The assignment of a reference plane group to a block can be configured in several ways. In one embodiment, a specific reference plane group is assigned to each of the secondary plane groups. This assignment can be fixed or signaled in a high-level syntax structure such as a set of parameters, an access unit header, an image header, or a slice header. In a second embodiment, the assignment of a reference plane group is coded in the bitstream and signaled by one or more syntax elements. This syntax element is coded for the block and specifies whether the subdivision information and prediction parameters are estimated or predicted or coded separately.The reference plane group may be a primary plane group or another secondary plane group. Given a reference plane group, a co-located block in the reference plane group is identified. The co-located block may be a block in the reference plane group that corresponds to the same image area as the current block, or a block that represents a block in the reference plane group that shares the largest portion of the image area with the current block. The co-located block may be partitioned into smaller prediction blocks. In a preferred embodiment, the subdivision information for the current block and the prediction parameters for the resulting sub-blocks are fully estimated using the subdivision information of the co-located blocks in different plane groups of the same image and the prediction parameters of the corresponding sub-blocks, without transmitting additional ancillary information. As one particular example, if the co-located block is partitioned into two or four prediction blocks, the current block is also partitioned into two or four sub-blocks for prediction purposes, and the prediction parameters for the sub-blocks of the current block are derived as described above. As another example, if a co-located block is partitioned into four sub-blocks and one of these sub-blocks is further partitioned into four smaller sub-blocks, then the current block is also partitioned into four sub-blocks and one of these sub-blocks (corresponding to a sub-block of the further decomposed co-located block) is also partitioned into four smaller sub-blocks and prediction parameters for all sub-blocks that are not further partitioned are estimated as described above. In another preferred embodiment, the subdivision information is fully estimated based on the subdivision information of the co-located block in the reference plane group, but the prediction parameters estimated for a sub-block are only used as a prediction of the actual prediction parameters of the sub-block. The deviation between the actual prediction parameters and the estimated prediction parameters are coded in the bitstream. In another embodiment, the estimated subdivision information is used as a prediction for the actual subdivision information of the current block and the difference is transmitted in the bitstream (as described above) but the prediction parameters are fully estimated.In another embodiment, both the estimated subdivision information and the estimated prediction parameters are used as prediction and the difference between the actual subdivision information and prediction parameters and the estimated values ​​are transmitted in the bitstream.

[0148] In another embodiment, for blocks of a plane group, the mode of residual coding (e.g., transform type) is adaptively selected whether it is estimated or predicted from co-located coded blocks of different plane groups of the same image, or whether it is coded separately for the blocks. This embodiment is similar to the above-mentioned embodiment of adaptive estimation / prediction of prediction parameters.

[0149] In another embodiment, the subdivision of a block (such as a prediction block) into transform blocks (i.e. blocks of samples to which a 2D transform is applied) is adaptively estimated or predicted from co-located coded blocks of a different plane group of the same image. This embodiment is similar to the above-mentioned embodiment in which the estimation / prediction of the subdivision into prediction blocks is adaptive.

[0150] In another embodiment, the subdivision of blocks into transform blocks and the mode of coding of the residuals for the resulting transform blocks (e.g. transform type) are adaptively estimated or predicted from co-located coded blocks of different plane groups of the same image. This embodiment is similar to the above-mentioned embodiment of the subdivision into prediction blocks and the adaptive estimation / prediction of prediction parameters for the resulting prediction blocks.

[0151] In another embodiment, the subdivision of the block into prediction blocks, the associated prediction parameters, the subdivision information of the prediction block and the mode of coding of the residual for the transformation block are adaptively estimated or predicted from previously coded co-located blocks of a different plane group of the same image. This embodiment is a combination of the above-mentioned embodiments. It is also possible to estimate or predict only a part of the above-mentioned coding parameters.

[0152] Thus, as mentioned above, inter-plane adoption / prediction can improve coding efficiency, however this is also possible when other block subdivisions than multi-tree based subdivisions are used, and is independent of whether block merging is performed or not.

[0153] The embodiments outlined above regarding inter-plane matching / prediction can be applied to image and video encoders and decoders. The encoders and decoders divide the color planes of an image and, if present, the auxiliary sample arrays associated with the image into blocks and associate the blocks with coding parameters. For each block, a set of coding parameters may be included in the bitstream. For example, these coding parameters may be parameters describing how to predict or decode the block at the decoder side. For example, the coding parameters may represent the prediction mode of a macroblock or block, subdivision information, intra prediction mode, reference indices used for motion compensation prediction, motion parameters such as displacement vectors, mode of coding of residuals, transform coefficients, etc. The sample arrays associated with the images may be of different sizes.

[0154] Next, an extended signaling scheme for coding parameters in a tree-based partitioning scheme, such as that described above with reference to Figures 1 to 8, is described. The scheme described below can be combined with any one or combination of the above embodiments, but the effects and advantages of the extended signaling scheme, as well as the other schemes, namely, combination and inter-plane adoption / prediction, are described independently of the above embodiments. This scheme is referred to below as inheritance.

[0155] An improved coding scheme for encoding the ancillary information in a tree-based partitioning scheme, called inheritance, is described below, which generally provides the following advantages over conventional schemes for handling the coding parameters:

[0156] Typically, in conventional image and video coding, an image, or a set of certain sample arrays within an image, is decomposed into blocks and associated with certain coding parameters. Typically, an image is composed of several sample arrays. In addition, an image may be associated with additional auxiliary sample arrays that specify, for example, transparency information, depth maps, etc. The sample arrays of an image (including the auxiliary sample arrays) can be grouped into one or more so-called plane groups, each plane group being composed of one or more sample arrays. A plane group of an image can be coded independently or, if the image is associated with several plane groups, can be coded by prediction from other plane groups of the same image. Typically, each plane group is decomposed into blocks. The blocks (or the corresponding blocks of sample arrays) are predicted either by inter-picture prediction or by intra-picture prediction. The blocks can be of various sizes and are either square or rectangular. The partitioning of an image into blocks can be fixed by the syntax or can be signaled (at least in part) in the bitstream. Often syntax elements are transmitted that signal subdivision information for blocks of a given size. Such syntax elements allow to specify whether and how a block is to be subdivided into smaller blocks and associated with coding parameters for prediction purposes etc. For every sample of a block (or for the corresponding block of the sample array), the decoding of the associated coding parameters is specified in a certain way. In an example, all samples in a block are predicted using the same set of prediction parameters, such as reference indexes (identifying a reference image in the set of previously coded images), motion parameters (specifying the amount of motion of the block between the reference image and the current image), parameters for specifying the interpolation filter, intra prediction mode etc. The motion parameters can be represented by a displacement vector with horizontal and vertical components or by higher-order motion parameters, such as affine motion parameters consisting of six components. It is also possible to associate several sets of specific prediction parameters (such as reference indexes and motion parameters) to a single block.Then, for each set of these specific prediction parameters, a single intermediate prediction signal for the block (or the corresponding block of the sample array) is generated, and the final prediction signal is formed by a combination, including superposition, of the intermediate prediction signals. The corresponding weighting parameters, possibly with a constant offset (added to the weighted sum), can be fixed for either the image, the reference image or the set of reference images, or can be included in the set of prediction parameters for the corresponding block. The difference between the original block (or the corresponding block of the sample array) and the prediction signal, also called the residual signal, is usually transformed and quantized. Often, a two-dimensional transform is applied to the residual signal (or the corresponding sample array of the residual block). For transform coding, the block (or the corresponding block of the sample array) for which a specific set of prediction parameters is used can be further divided before applying the transform. The transform block can be the same size as the block used for prediction, or smaller. It is also possible that the transform block contains several blocks used for prediction. Each transform block can be of various sizes, resulting in a square or rectangular block. After transformation, the resulting transform coefficients are quantized to obtain the so-called transform coefficient levels. The transform coefficient levels and the prediction parameters and, if present, the subdivision information are entropy coded.

[0157] In some image and video coding standards, the syntax offers very limited possibilities for subdivision of an image (or a group of planes) into blocks. Usually, it is only possible to specify whether and (possibly how) a block of a given size can be subdivided into smaller blocks. For example, in H.264 the maximum block size is 16x16. A 16x16 block is also called a macroblock, and each image is partitioned into macroblocks in a first step. For each 16x16 macroblock, it can be signaled whether it is coded as a 16x16 block, two 16x8 blocks, two 8x16 blocks, or four 8x8 blocks. If a 16x16 block is subdivided into four 8x8 blocks, each of these 8x8 blocks can be coded as one 8x8 block, two 8x4 blocks, two 4x8 blocks, or four 4x4 blocks. In the current standards for image and video coding, the reduced possibilities for specifying the partitioning into blocks have the advantage that the rate of the side information for conveying the subdivision information can be kept small, but the disadvantage is that the bit rate required for the transmission of the prediction parameters of the blocks can be large, as explained below. The rate of the side information for conveying the prediction information usually accounts for a significant amount of the overall bit rate of the blocks. Therefore, if this side information is reduced, the coding efficiency can be improved, which can be achieved, for example, by using a larger block size. Images of real images and video sequences consist of objects of arbitrary shapes with specific properties. For example, such objects or object parts are characterized by a specific texture or a specific motion. Usually, the same set of prediction parameters is applied to such objects or object parts. However, the boundaries of the objects usually do not coincide with the boundaries of the blocks of the larger prediction blocks (for example, 16x16 macroblocks in H.264). Usually, the encoder makes the subdivision decision (among a limited number of possibilities) that minimizes the amount of loss of a certain rate / distortion.This may result in a large number of small blocks for an object of any shape. As these small blocks are each associated with a set of prediction parameters that need to be transmitted, the rate of the associated information may become a significant part of the overall bit rate. However, some of the small blocks still represent the same object or a part of an object region, so the prediction parameters of the resulting blocks are identical or very similar. It is believed that the coding efficiency will be improved if the syntax is extended in a way that allows not only subdivision of the blocks, but also sharing of coding parameters between the blocks obtained after subdivision. In a tree-based subdivision, the sharing of coding parameters can be realized for a given set of blocks by assigning the coding parameters or parts of them to one or more parent nodes in the tree-based hierarchy. As a result, the shared parameters or parts of them can be used to reduce the amount of associated information required to signal the actual selection of the coding parameters for the blocks obtained after subdivision. This can be done by omitting to signal the parameters for the subsequent blocks or by using the shared parameters for the prediction and / or context modeling of the parameters for the subsequent blocks.

[0158] The basic idea of ​​the inheritance scheme described below is to reduce the bit rate required for the transmission of coding parameters by sharing information according to a tree-based hierarchy of blocks. The shared information signal is transmitted in the bitstream (added to the subdivision information). The advantage of the inheritance scheme is an increase in coding efficiency due to the reduced information rate associated with the coding parameters.

[0159] In order to reduce the associated information rate, in the embodiment described below, the signaling of the respective coding parameters for a particular sample set, i.e. a simply connected region, of a subdivision of the multi-tree is transmitted in an efficient manner in the data stream. A particular sample set, i.e. a simply connected region, may represent a rectangular or square block, or an area of ​​any shape, or any other sample set. An inheritance scheme described below makes it unnecessary to include the coding parameters for each of these sample sets explicitly in full in the bitstream. The coding parameters may represent prediction parameters that specify how to predict the corresponding sample set using the already coded samples. The numerous possibilities and examples described above also apply here. As also indicated above and as will be further described below, for the following inheritance scheme, the tree-based partitioning of the sample array of the image into sample sets may be fixed by the syntax or may be conveyed by the corresponding subdivision information in the bitstream. As mentioned above, the coding parameters for the sample sets may be transmitted in a predefined order specified by the syntax.

[0160] According to the inheritance scheme, the decoder or the extractor 102 of the decoder is configured to derive information about the coding parameters of the individual simply connected regions or sample sets in a specific way. In particular, coding parameters or parts thereof, such as parameters used for prediction purposes, are shared between blocks according to a given tree-based partitioning scheme, with sharing groups according to a tree structure determined by the encoder or inserter 18. In a particular embodiment, the sharing of coding parameters for all child nodes of a given internal node of the partitioning tree is indicated using a specific sharing flag of a binary value. As an alternative approach, the coding parameter refinements can be transmitted for each node, such that the accumulated parameter refinements according to the tree-based block hierarchy can be applied to all sample sets of the block of a given leaf node. In another embodiment, parts of the coding parameters transmitted for the internal nodes according to the tree-based block hierarchy can be used for context-adaptive entropy encoding and entropy decoding of the coding parameters or parts thereof for the block of a given leaf node.

[0161] 12A and 12B show the basic idea of ​​inheritance in the specific case of using a quadtree-based partitioning. However, as shown many times above, other multi-tree subdivisions may be used. In FIG. 12A, a tree structure is shown, and in FIG. 12B, a spatial partitioning corresponding to the tree structure of FIG. 12A is shown. The partitioning shown is similar to that shown in FIG. 3A to FIG. 3C. In general, the inheritance scheme allows for the assignment of ancillary information to nodes of different non-leaf layers in the tree structure. Depending on the assignment of ancillary information to nodes of different layers in the tree, such as the internal nodes and the root node in the tree of FIG. 12A, different degrees of sharing of ancillary information can be achieved in the tree hierarchy of blocks shown in FIG. 12B. For example, if it is determined that all leaf nodes in the fourth layer, all of which in the case of FIG. 12A have the same parent node, share the ancillary information, this essentially means that the smallest blocks shown in FIG. 12B as 156a to 156d share this ancillary information, and it is not necessary to transmit the ancillary information for each of these small blocks 156a to 156d, i.e., four times. However, this is left as an option for the encoder. It is also possible to determine the entire area of ​​the hierarchical level 1 (second layer) of FIG. 12A, i.e., the upper right quarter of the tree block 150, including sub-blocks 154a, 154b, and 154d and the smaller sub-blocks 156a to 156d, as the area in which the coding parameters are shared. Thus, the area in which the ancillary information is shared increases. The next level of increase is to combine all sub-blocks in the first layer, i.e., sub-blocks 152a, 152c, and 152d, and the smaller blocks. In other words, in this case, the entire treeblock has associated information assigned to it, and all subblocks of this treeblock 150 share the associated information.

[0162] In the following description of inheritance, the following notation is used to describe the embodiment. a. The reconstructed sample of the current leaf node: r b. Reconstructed specimen of adjacent leaf: r' c. Predictor of the current leaf node: p d. Residual of the current leaf node: Res e. The reconstructed residual of the current leaf node: RecRes f. Scaling and inverse transformation: SIT g. Shared flag: f

[0163] As a first example of inheritance, we consider the signaling of intra prediction at internal nodes. Specifically, we consider how intra prediction modes are signaled at internal nodes for tree-based block partitioning for prediction purposes. Internal nodes (including the root node) can convey some of the associated information used by the corresponding child nodes by traversing the tree from the root node to the leaf nodes. Specifically, for internal nodes, a shared flag f is transmitted with the following meaning:

[0164] If f has a value of 1 ("true"), all child nodes of a given internal node share the same intra prediction mode. In addition to the sharing flag f with a value of 1, the internal node signals the parameters of the intra prediction mode used for all its child nodes. Therefore, all subsequent child nodes do not transmit any prediction mode information or sharing flag. When reconstructing all associated leaf nodes, the decoder applies the intra prediction mode from the corresponding internal node.

[0165] If the value of f is 0 ("false"), then the child nodes of the corresponding internal node do not share the same intra-prediction mode, and each child node that is an internal node transmits a separate sharing flag.

[0166] Figure 12C illustrates the signaling of intra prediction at the internal nodes mentioned above: the first layer internal nodes convey the shared flag and side information provided by the intra prediction mode information, while the child nodes do not transmit any side information.

[0167] As a second example of inheritance, we consider the refinement of inter prediction. In particular, we consider how to propagate ancillary information of inter prediction mode in intra mode of tree-based block partitioning for the purpose of refinement of motion parameters provided by motion vectors, etc. Internal nodes (including the root node) can propagate part of the ancillary information that is refined in the corresponding child node by traversing the tree from the root node to the leaf nodes. In particular, for internal nodes, a shared flag f is transmitted with the following meaning:

[0168] If f has the value 1 ("true"), all child nodes of a given internal node share the same motion vector reference. In addition to a sharing flag f with value 1, the internal node also transmits a motion vector and a reference index. All subsequent child nodes then do not transmit any further sharing flags, but may transmit a refinement of this inherited motion vector reference. Upon reconstruction of all associated leaf nodes, the decoder adds the motion vector refinement of a given leaf node to the inherited motion vector reference belonging to the corresponding internal parent node with a sharing flag f with value 1. This means that the motion vector refinement of a given leaf node is the difference between the actual motion vector applied at this leaf node for motion compensated prediction and the motion vector reference of the corresponding internal parent node.

[0169] If the value of f is 0 ("false"), then the child nodes of the corresponding internal node do not necessarily share the same inter prediction mode, and no refinement of motion parameters is performed in the child nodes by using motion parameters from the corresponding internal node. Each child node that is an internal node transmits a separate sharing flag.

[0170] Fig. 12D illustrates the above-mentioned refinement of motion parameters. The first-layer internal nodes transmit the shared flag and associated information. The leaf child nodes transmit only the refinement of motion parameters, and no associated information is transmitted by the second-layer internal child nodes, for example.

[0171] Reference is now made to FIG. 13. FIG. 13 is a flow chart illustrating a mode of operation of a decoder, such as the decoder of FIG. 2, in reconstructing an array of information samples representing a spatial example information signal from a data stream. The array of information samples is subdivided into leaf areas of different sizes by a subdivision of a multi-tree. As explained above, each leaf area is associated with one hierarchical level of a series of hierarchical levels of the subdivision of the multi-tree. For example, all blocks shown in FIG. 12B are leaf areas. For example, the leaf area 156c is associated with the fourth layer (or level 3) of the hierarchy. Each leaf area is associated with coding parameters. Examples of these coding parameters are explained above. The coding parameters are represented by a respective set of syntax elements for each leaf area. The respective syntax element type of each syntax element is one of a series of syntax element types. For example, such syntax element types are a prediction mode, a component of a motion vector, an indication of an intra prediction mode, etc. The decoder performs the following steps according to FIG. 13:

[0172] In step 550, the inheritance information is extracted from the data stream. In the case of Fig. 2, the extractor 102 performs step 550. The inheritance information indicates whether inheritance is used for the current arrangement of information samples. The following description makes clear that there are several possibilities for the inheritance information, such as a sharing flag f and the propagation of information in a multi-tree structure that is divided into primary and secondary parts.

[0173] The array of information samples may already be part of the image, such as a treeblock, i.e., treeblock 150 in Fig. 12B. Thus, the inheritance information indicates whether inheritance is used for a particular treeblock 150. Such inheritance information may be inserted in the data stream, for example, for all treeblocks of a prediction subdivision.

[0174] Furthermore, if it is indicated that inheritance is used, the information of inheritance indicates at least one area of ​​inheritance of the array of information samples. This area of ​​inheritance is composed of a set of leaf areas. It corresponds to one hierarchical level of the series of hierarchical levels of the subdivision of the multi-tree that is lower than each hierarchical level to which the set of leaf areas is associated. In other words, the information of inheritance indicates whether inheritance is used for the current array of samples, such as the tree block 150. If inheritance is used, at least one area of ​​inheritance or a sub-area of ​​this tree block 150 is indicated, in which the leaf areas share the coding parameters. Thus, the area of ​​inheritance does not have to be a leaf area. In the example of FIG. 12B, this area of ​​inheritance may be, for example, an area formed by the sub-blocks 156a to 156b. Alternatively, the area of ​​inheritance may be larger and may further include the sub-blocks 154a, 154b, and 154d. Furthermore, the area of ​​inheritance may be the tree block 150 itself, and all the leaf blocks share the coding parameters associated with the area of ​​inheritance.

[0175] Also, multiple regions of inheritance may be defined within each sample array or tree block 150. For example, if the bottom left sub-block 152c is also partitioned into smaller blocks, then sub-block 152c can also form a region of inheritance.

[0176] In step 552, the information of inheritance is checked whether inheritance is used. If inheritance is used, the process of FIG. 13 proceeds to step 554, where a subset of inheritance including at least one syntax element of a given syntax element type is extracted from the data stream for each region of inter-plane inheritance. In the following step 556, this subset of inheritance is copied to or used as a prediction of a corresponding subset of inheritance of syntax elements in a set of syntax elements representing coding parameters associated with a set of leaf regions constituting each at least one region of inheritance. In other words, for each region of inheritance indicated in the information of inheritance, the data stream constitutes a subset of inheritance of syntax elements. In further words, the inheritance is related to at least one specific syntax element type or category of syntax elements that can be used for inheritance. For example, syntax elements of a prediction mode or an inter-plane prediction mode or an intra-prediction mode may be subject to inheritance. For example, the subset of inheritance included in the data stream for the region of inheritance may include syntax elements of an inter-plane prediction mode. The inheritance subset may further include syntax elements whose type depends on the value of the above-mentioned fixed syntax element type associated with the inheritance scheme. For example, if the inter-plane prediction mode is a fixed component of the inheritance subset, syntax elements that define motion compensation, such as components of a motion vector, may or may not be included in the inheritance subset depending on the syntax. For example, if the upper right quarter of the tree block 150, i.e., subblock 152b, is the inheritance region, only the inter-plane prediction mode or the inter-plane prediction mode and the motion vector and the motion vector index may be indicated for this inheritance region.

[0177] All syntax elements contained in the inheritance subset are either copied to the corresponding coding parameters of the leaf blocks in the inheritance domain, i.e., leaf blocks 154a, 154b, 154d, and 156a to 156d, or used as their predictions. If predictions are used, the residuals are transmitted for each leaf block.

[0178] One possibility of transmitting the inheritance information for the tree block 150 is the transmission of the shared flag f described above. In this case, the extraction of the inheritance information in step 550 includes the following. In particular, the decoder is configured to extract the shared flag f from the data stream using the order of the hierarchical levels from the lower hierarchical level to the higher hierarchical level for the non-leaf area corresponding to any of the inheritance sets of at least one hierarchical level of the subdivision of the multi-tree, and to check whether the inheritance or the shared flag indicates inheritance. For example, the inheritance set of the hierarchical levels is formed by the hierarchy of the first to third layers in FIG. 12A. Thus, any of the nodes of the subtree structure located in any of the first to third layers, but not the leaf nodes, have an associated shared flag in the data stream. The decoder extracts these shared flags in the order of the traversal, such as depth order or width order, from the first to third layers. If one of the shared flags is 1, the decoder recognizes that the leaf block included in the corresponding inheritance area shares the subset of the inheritance that is subsequently extracted in step 554. There is no need to check the inheritance flags for the child nodes of the current node, in other words, the inheritance flags for these child nodes are not transmitted in the data stream, since it is clear that the domains of these nodes already belong to the domain of inheritance where a subset of the inheritance of syntax elements is shared.

[0179] The sharing flag f can be alternated with bits conveying subdivisions of the quadtree described above. For example, an alternating bit sequence containing both subdivision and sharing flags can be shown as follows:

number

[0180] Another way of defining the inheritance information indicating the inheritance domain is the use of two subdivisions defined in a subordinate manner to each other, as described above for the prediction and residual subdivisions, respectively. In general, the leaf blocks of a primary subdivision can form the inheritance domains that define the domains where the inheritance subsets of syntax elements are shared. At the same time, the blocks within these inheritance domains to which the inheritance subsets of syntax elements are copied or used as predictions are defined by the subordinate subdivisions.

[0181] For example, consider the residual tree as an extension of the prediction tree. Furthermore, consider the case where the prediction block can be further divided into smaller blocks for the purpose of residual coding. For each prediction block corresponding to a leaf node of the prediction-related quadtree, one or more sub-quadtrees determine the corresponding subdivision for residual coding.

[0182] In this case, instead of using the signaling of predictions at the internal nodes, the residual tree is interpreted as follows: a refinement of the prediction tree is also specified, in the sense of using a certain prediction mode (informed by the corresponding leaf node of the tree related to the prediction), and refined reference samples are also used. The following example illustrates this case.

[0183] For example, Fig. 14A and Fig. 14B show a quad-tree partitioning for intra prediction. For one particular leaf node of the primary subdivision, the adjacent reference samples are highlighted. Fig. 14B shows the residual quad-tree subdivision with refined reference samples for the same predicted leaf node. All sub-blocks shown in Fig. 14B share the same inter-plane prediction parameters included in the data stream for each leaf block highlighted in Fig. 14A. Thus, Fig. 14A shows an example of a conventional quad-tree partitioning for intra prediction, with reference samples shown for one particular leaf node. However, in a preferred embodiment, a separate intra prediction signal is calculated for each leaf node in the residual tree, for example by using the adjacent samples of the reconstructed leaf node in the residual tree, as shown in Fig. 14B shaded with grey stripes. Then, a reconstructed signal for a given residual leaf node is obtained in the usual way by adding the quantized residual signal to this prediction signal. This reconstructed signal is used as a reference signal for the subsequent prediction process. The decoding order for the prediction is the same as the decoding order for the residual.

[0184] As shown in FIG. 15, in the decoding process, for each residual leaf node, a prediction signal p is calculated using the reference sample r′ according to the actual intra prediction mode (as indicated at the leaf node of the quadtree associated with the prediction).

[0185] SIT processing,

number

number

[0186] The decoding order for prediction is the same as the residual decoding order shown in FIG.

[0187] Each of the residual leaf nodes is decoded as described in the previous paragraph. The reconstructed signal r is stored in a buffer, as shown in Figure 16. From this buffer, the reference sample r' is taken for the next prediction and decoding process.

[0188] With reference to Figures 1 to 16, specific embodiments have been described by combining the various aspects outlined above. Further, other embodiments of the present application will be described which focus on the specific aspects described above and which are embodiments which combine several of the above-mentioned embodiments.

[0189] In particular, the embodiment described above based on the configurations of figures 1 and 2 is mainly a combination of many aspects of the present application, which can be advantageously used in other applications and other coding fields. As reiterated in the above description, for example, multi-tree subdivision may be used without combining and / or inter-plane adoption / prediction and / or inheritance. For example, aspects such as transmitting the maximum block size, using a depth-ordered traversal order, adapting the context according to the hierarchical level of the respective subdivision flags, transmitting the maximum hierarchical level in the bitstream, all provide advantages without being mutually dependent, in order to reduce the bit rate of the associated information. This is also true when considering combination schemes. The advantages are obtained by combining regardless of the detailed way in which the image is subdivided into simply connected regions, and regardless of the presence of multiple sample arrays and the use of inter-plane adoption / prediction and / or inheritance. The same is true for the advantages obtained from inter-plane adoption / prediction and inheritance.

[0190] Therefore, the embodiments outlined below synthesize the above-mentioned embodiments with respect to aspects related to bonding, and since the following embodiments synthesize the above-mentioned embodiments, it can be considered that many of the details described above can be combined with the embodiments described below.

[0191] FIG. 17 shows a decoder according to an embodiment of the present application. The decoder of FIG. 17 comprises an extractor 600 and a reconstructor 602. The extractor 600 is configured to extract payload data from a data stream 604 for each of a number of simply connected regions into which an array of information samples representing a spatially sampled information signal is subdivided. As mentioned above, the simply connected regions into which the array of information samples is subdivided are square or rectangular in shape, resulting from a multi-tree subdivision. Furthermore, the embodiment specifically described for the subdivision of the sample array is merely a specific embodiment, and other subdivisions may be used. Some possibilities are shown in FIGS. 18A-18C. For example, FIG. 18A shows a subdivision of a sample array 606 into a regular two-dimensional array of non-overlapping adjacent tree blocks 608, some of which are further subdivided into sub-blocks 610 of various sizes according to the multi-tree structure. As mentioned above, FIG. 18A shows a quad-tree subdivision, but other numbers of child nodes of each parent node may also be partitioned. In the embodiment shown in Fig. 18B, the multi-tree subdivision is applied directly to the entire pixel array 606, thereby subdividing the sample array 606 into sub-blocks of various sizes. That is, the entire pixel array 606 is treated as a tree block. Fig. 18C shows another embodiment. In this embodiment, the sample array is structured as a regular two-dimensional array of mutually adjacent square or rectangular macroblocks. Each of these macroblocks 612 is individually associated with a partitioning information, according to which the macroblock 612 is either left unpartitioned or partitioned into a regular two-dimensional array of blocks of a size indicated by the partitioning information. As can be seen, all the subdivisions in Figs. 13A to 13C result in the subdivision of the sample array 606 into non-overlapping simply connected regions, for example in the embodiments of Figs. 18A to 18C. However, several alternatives are possible. For example, the blocks may overlap each other, but are limited to the condition that each block has a part that does not overlap any of its neighboring blocks.Alternatively, the neighboring blocks aligned with the current block in a given direction are restricted to overlap each representation of the block with at most one block, meaning that the left and right neighboring blocks overlap the current block so as to completely cover it, but do not overlap each other, and similarly for the vertical and diagonal neighboring blocks.

[0192] As described above with reference to Figures 1 to 16, the array of information samples does not necessarily represent a video image or a still image. The sample array 606 may represent a depth map or a transparency map of a particular scene. As already explained above, the payload data associated with each of the plurality of simply connected regions may include residual data in the spatial or transform domain, such as transform coefficients, and a significance map identifying the locations of significant transform coefficients in the transform block corresponding to the residual block. In general, the payload data extracted by the extractor 600 for each simply connected region from the data stream 604 is data in which the associated simply connected region is spatially described, either directly in the spatial or spectral domain, or as a residual relative to some prediction, for example.

[0193] The reconstruction unit 602 is configured to reconstruct the sequence of information samples from the payload data for the singly connected regions of the sequence of information samples by processing, for each singly connected region, the payload data for each singly connected region in a manner dictated by the coding parameters associated with each singly connected region. As explained above, the coding parameters may be prediction parameters, so that the singly connected regions illustrated in Figures 18A to 18C may correspond to the above-mentioned prediction blocks, i.e. blocks for which the data stream 604 specifies prediction details for the prediction of the individual singly connected regions. However, the coding parameters are not limited to prediction parameters. The coding parameters may indicate a transformation used to transform the payload data. Alternatively, they may define a filter used to reconstruct the individual simply connected regions during the reconstruction of the sequence of information samples.

[0194] The extractor 600 is configured to identify, for a given simply connected region, a simply connected region among the plurality of simply connected regions that is in a predetermined relative position to the given simply connected region. This step has been described in detail above with respect to step 450. That is, the identification may be performed depending on the predetermined relative position as well as a subset of the coding parameters associated with the given simply connected region. After the identification, the extractor 600 extracts a merging indicator for the given simply connected region from the data stream 604. If the number of simply connected regions in a predetermined relative position to the given simply connected region is greater than zero, this corresponds to the description of steps 452 and 454 above. If the merging indicator indicates the operation of merging the given block, the extractor 600 is configured to check whether the number of simply connected regions in a predetermined relative position to the given simply connected region is one or whether the number of simply connected regions in a predetermined relative position to the given simply connected region is greater than one but their coding parameters match each other. If one of these two cases is true, then the extractor 600 may employ the coding parameters or use them to predict the coding parameters of the given simply connected region or a remaining subset thereof, as described above in steps 458 to 468. If the latter check reveals that the number of simply connected regions in a given relative position to the given simply connected region is greater than one and that the coding parameters are different from one another, as described above with reference to figure 10, then further indications may be extracted by the extractor 600 from the data stream 604.

[0195] The latter check can be performed to suppress the transmission of further indicators of one or a subset of the candidate simply connected regions, thereby reducing the associated information overhead.

[0196] Fig. 19 shows a general structure of an encoder for generating a data stream that can be decoded by the decoder of Fig. 17. The encoder of Fig. 19 comprises a data generator 650 and an inserter 652. The data generator 650 is configured to encode the sequence of information samples into payload data in association with coding parameters associated with each simply connected region to indicate how the payload data for each simply connected region is reconstructed for each of the multiple interconnected regions into which the sequence of information samples is subdivided. The inserter 652 performs the same identification and validation as performed by the extractor 600 of the decoder of Fig. 12, but inserts instead of extracting merge indicators and suppresses the insertion of coding parameters into the data stream. Alternatively, instead of inserting all coding parameters into the data stream, it inserts the residuals of the respective predictions instead of the adoption / prediction as described above with reference to Figs. 12 and 10, respectively.

[0197] It should also be noted that the encoder structure of Fig. 19 is shown simply as a schematic, and in practice the determination of the payload data, the coding parameters and the merging index may be an iterative process, for example considering that if the coding parameters of adjacent simply connected regions are similar but not identical to each other, the merging index allows to completely suppress the coding parameters of one of the simply connected regions and to present only the residual instead of presenting all of these coding parameters, it may be determined by the iterative process that it is desirable to tolerate small differences between these coding parameters rather than signaling them to the decoder.

[0198] Fig. 20 shows another embodiment of a decoder. The decoder of Fig. 20 comprises a subdivision unit 700, a combination unit 702 and a reconstruction unit 704. The subdivision unit is configured to spatially sub-divide an array of samples representing a spatial sampling of a two-dimensional information signal into a plurality of non-overlapping simply connected regions of different sizes by recursive multiple partitioning depending on a subset of syntax elements contained in the data stream. This multiple partitioning may thus correspond to the embodiments outlined above with reference to Figs. 1 to 16 or Fig. 18A or 18B. The syntax elements contained in the data stream for indicating the subdivision may be defined as shown above with reference to Figs. 6A and 6b or in another way.

[0199] The combining unit 702 is configured to combine spatially adjacent simply connected regions of the plurality of simply connected regions according to a second subset of syntax elements in the data stream that are separated from the first subset to obtain an intermediate subdivision of the sample sequence into disjoint sets of simply connected regions whose union is a plurality of simply connected regions. In other words, the combining unit 702 combines the simply connected regions to assign in a unique manner to the combined groups of simply connected regions. The second subset of syntax elements indicating the information of the combination may be defined in the manner described above with reference to Fig. 19 and Fig. 10 or in another manner. That is, the encoder being able to indicate the subdivision by using a subset that is separated from the subset indicating the combination increases the freedom of the encoder to adapt the subdivision of the sample sequence to the actual sample sequence content, improving the coding efficiency. The reconstructing unit 704 is configured to reconstruct the sample sequence from the data stream using the intermediate subdivision. As indicated above, the reconstructing unit may utilize the intermediate subdivision by adopting / predicting the coding parameters of the combination partner for the current simply connected region. Alternatively, the reconstructor 704 may apply a transformation or prediction operation to the combined regions of the joined groups of simply connected regions.

[0200] Fig. 21 shows a possible encoder for generating a data stream that can be decoded by the decoder of Fig. 15. The encoder comprises a subdivision / combination stage 750 and a data stream generator 752. The subdivision / combination stage is arranged to determine an intermediate subdivision of the array of information samples representing a spatial sampling of a two-dimensional information signal, and two disjoint sets of simply connected regions whose union is a plurality of simply connected regions. The intermediate subdivision is defined by a first subset and a second subset of syntax elements. According to the first subset of syntax elements, the array of information samples is subdivided by recursive multiple partitioning into a plurality of non-overlapping simply connected regions of different sizes. According to a second subset of syntax elements separated from the first subset, spatially adjacent simply connected regions of the plurality of simply connected regions are combined to obtain the intermediate subdivision. The data stream generator 752 encodes the array of information samples into a data stream using the intermediate subdivision. Also, the subdivision / combination stage 750 inserts the first and second subsets into the data stream. As in FIG. 14, the process of determining the first and second subsets and syntax elements generated by the data stream generator 752 may also be an iterative process. For example, the subdivision / combination stage 750 may predetermine an optimal subdivision, and the data stream generator 752 may determine a corresponding optimal set of syntax elements for encoding the sample sequence using the sample subdivision in conjunction with the subdivision / combination stage, and then set the syntax elements described for the combination, so that the associated information overhead is reduced. Also, the encoding process does not have to end here. The subdivision / combination stage 750 may interface with the data stream generator 752 to set different subdivision and syntax element settings than those predeterminable by the data stream generator 752, in order to determine whether the rate / distortion ratio is improved by exploiting the beneficial properties of the combination.

[0201] As mentioned above, the embodiment described with reference to Figs. 17 to 21 is a synthesis of the embodiment described with reference to Figs. 1 to 16, so that the elements of Figs. 1 to 16 can be uniquely associated with the elements shown in Figs. 17 to 21. For example, the extractor 102 together with the subdivision unit 104a and the combination unit 104b perform the tasks performed by the extractor 600 of Fig. 17. The subdivision unit performs the management of the subdivision and the adjacency relations between the individual simply connected regions. The combination unit 104b then manages the combination of the simply connected regions into groups and identifies the correct coding parameters to be copied or used as prediction of the current simply connected region if the currently decoded combination information indicates a combination event. If entropy decoding is used to extract the data, the extractor 102 performs the extraction of the actual data from the data stream using the correct context. The remaining elements of Fig. 2 are examples of the reconstructor 602. Of course, the reconstructor 602 may be embodied in other forms than those shown in Fig. 2. For example, the reconstructor 602 may not use motion compensated prediction and / or intra prediction, as well as other possibilities. Also, as mentioned above, the simply connected regions referred to in relation to the description of Figure 17 may correspond to the prediction blocks mentioned above or any of the other subdivisions mentioned above, such as residual subdivision or filter subdivision, as already indicated above.

[0202] Comparing the encoder of Fig. 19 with the example of Fig. 1, a data generator 650 includes all elements except the data stream inserter 18, which corresponds to the inserter 652 of Fig. 19. For the data generator 650 too, other coding approaches than the hybrid coding approach shown in Fig. 1 can be used.

[0203] Comparing the decoder of Fig. 20 with the example shown in Fig. 2, the subdivision unit 104a and the combination unit 104b correspond to the subdivision unit 100 and the combination unit 102, respectively, of Fig. 20, and the elements 106 and 114 correspond to the reconstruction unit 704. The extraction unit 102 is commonly involved in the functions of all the elements shown in Fig. 20.

[0204] With respect to the encoder of FIG. 21, the subdivision / combination stage 750 corresponds to the subdivision unit 28 and the combination unit 30, and the data stream generation unit 752 includes all other elements shown in FIG.

[0205] Although some aspects have been described in terms of an apparatus, it will be apparent that these aspects also serve as descriptions of corresponding methods, with blocks or devices corresponding to means or features of the means. Similarly, aspects described in terms of means also serve as descriptions of corresponding blocks or items or functions of the corresponding apparatus. Some or all of the means may be implemented by (or using) a hardware apparatus, such as a microprocessor, a programmable computer, an electronic circuit, etc. In some embodiments, one or more of the most important means may be implemented by such an apparatus.

[0206] A signal that has been encoded / compressed according to the present invention may be stored on a digital storage medium or may be transmitted over a transmission medium, such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0207] The embodiments of the present invention can be implemented in hardware or software depending on the particular implementation requirements. For example, they can be implemented using digital storage media such as floppy disks, DVDs, Blu-ray Discs, CDs, ROMs, PROMs, EPROMs, EEPROMs, flash memories, etc., on which electronically readable control signals are stored and which are (or can be) used in conjunction with a programmable computer system to perform the respective methods. Thus, the digital storage media may be computer readable.

[0208] Some embodiments of the present invention include a data storage medium having electronically readable control signals stored thereon that can be used with a programmable computer system to perform one of the methods described herein.

[0209] In general, embodiments of the present invention may be implemented as a computer program product including program code that performs one of the methods when the computer program product is executed on a computer. The program code may be stored, for example, on a machine-readable storage medium. In another embodiment, the computer program for performing one of the methods described herein is included and stored on a machine readable storage medium.

[0210] In other words, an embodiment of the inventive method is, therefore, a computer program comprising a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0211] Therefore, a further embodiment of the inventive method is a data storage medium (or a digital storage medium, or a computer readable medium) having recorded thereon a computer program for performing one of the methods described herein.

[0212] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program performing one of the methods described herein, for example a data stream or a sequence of signals adapted to be transmitted over a data communications connection, such as the Internet.

[0213] Other embodiments comprise a processing means, such as a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein. Another embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0214] In some embodiments, a programmable logic device (such as a field programmable gate array) may be used to implement some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may work in conjunction with a microprocessor to perform one of the methods described herein. In general, it is preferred that the methods be performed by a hardware apparatus.

[0215] The above-described embodiments are merely examples for illustrating the basic nature of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be obvious to those skilled in the art. Therefore, the present invention shall be limited only by the scope of the corresponding patent claims, and not by the specific details shown in the description of the embodiments herein.

Claims

1. an extractor for extracting payload data from the data stream for each of a plurality of simply connected regions into which an array of information samples representing a spatially sampled information signal is subdivided; a reconstruction unit for reconstructing, for each singly connected region of the sequence of information samples from the payload data by processing, for the each singly connected region, payload data for the each singly connected region in a manner indicated by encoding parameters associated with the each singly connected region; Equipped with The extraction unit is for a given simply connected region, identifying a simply connected region among the plurality of simply connected regions that is in a given relative positional relationship with the given simply connected region; If the number of simply connected regions in a predetermined relative position to the given simply connected region is greater than zero, a merge indicator for the given simply connected region is extracted from the data stream. Extract from If the merge indicator indicates a merge operation for the given block, if the number of simply connected regions in a predetermined relative position to the given simply connected region is one, then the coding parameters of the simply connected region are adopted as the coding parameters for the given simply connected region, or the coding parameters for the given simply connected region are predicted from the coding parameters of the simply connected regions in a predetermined relative position to the given simply connected region, and a prediction residual for the given simply connected region is extracted from the data stream. decoder.

2. 2. A decoder according to claim 1, The extraction unit further comprises: if the number of simply connected regions in a predetermined relative position to the predetermined simply connected region is greater than one, checking whether the coding parameters associated with the simply connected regions are identical to one another; If they are identical, then the coding parameters of said simply connected region are taken as the coding parameters for said given simply connected region, or alternatively the coding parameters for said given simply connected region are predicted from the coding parameters of a number of simply connected regions which are in a predetermined relative position to said given simply connected region, and the residual of said prediction is extracted from the data stream. decoder.

3. A decoder according to claim 1 or 2, a subdivision unit for spatially subdividing the array of informative samples into the simply-connected regions using a multi-tree subdivision by performing multiple partitions recursively such that the simply-connected regions have different sizes; The extractor processes the simply connected regions in a depth-ordered traversal order. decoder.

4. 4. A decoder as claimed in claim 1, wherein the subdivision unit spatially divides the array of information samples into tree root regions of a maximum region size, and subdivides at least a subset of the tree root regions into smaller simply connected regions of different sizes by recursively partitioning the subset of the tree root regions multiple times in accordance with multi-tree subdivision information, wherein the tree root regions are not partitioned in accordance with the multi-tree subdivision information, and the smaller simply connected regions form a plurality of simply connected regions.

5. In a decoder described in any one of claims 1 to 4, the extraction unit, when identifying a simply connected region that has a predetermined relative positional relationship with the specified simply connected region, if there is an information sample adjacent to the information sample at the boundary of the specified simply connected region, identifies the simply connected region containing the information sample adjacent to the information sample at the boundary of the specified simply connected region.

6. A decoder according to any one of claims 1 to 5, The extraction unit is extracting from the data stream a first subset of coding parameters for the given simply connected region prior to said identifying; identifying a simply connected region that is in a predetermined relative position to the given simply connected region and that is associated with coding parameters that satisfy a predetermined relationship to the first subset of coding parameters for the given simply connected region; extracting and employing or predicting a residual of the prediction with a second subset of coding parameters for the given simply connected region that is disjoint from the first subset of coding parameters based on the number of simply connected regions that are in a predetermined relative positional relationship with the given simply connected region and that are associated with coding parameters that satisfy the predetermined relationship with the first subset of coding parameters for the given simply connected region; decoder.

7. 7. A decoder as claimed in claim 1, wherein, when coding parameters associated with a number of simply connected regions that are in a predetermined relative positional relationship with respect to the given simply connected region are not equal to each other, the extraction unit extracts reference adjacent identifiers from the data stream, identifies a proper subset of the number of simply connected regions that are in a predetermined relative positional relationship with respect to the given simply connected region, and adopts the coding parameters of this proper subset as coding parameters for the given simply connected region, or predicts the coding parameters of the given simply connected region from the proper subset and extracts a prediction residual from the data stream.

8. an extractor configured to extract payload data from the data stream for each of a plurality of simply connected regions into which an array of information samples representing a spatially sampled information signal is subdivided; a reconstruction unit configured to reconstruct, for each singly connected region of the sequence of information samples from the payload data by processing, for each singly connected region, payload data for the each singly connected region in a manner indicated by coding parameters associated with the each singly connected region; Equipped with The extraction unit is extracting from the data stream a first subset of coding parameters for a given simply connected region; for a given simply connected region, identifying a simply connected region among the plurality of simply connected regions that is in a given relative positional relationship with the given simply connected region; If the number of simply connected regions in a predetermined relative position to the given simply connected region is greater than zero, a merge indicator for the given simply connected region is extracted from the data stream. Extract from If the merging indicator indicates a merging operation for the given block, for each of a plurality of simply connected regions that are in a predetermined relative position to the given simply connected region, a distance between the first subset of coding parameters of the given simply connected region and a corresponding subset of coding parameters of each simply connected region that are in a predetermined relative position to the given simply connected region is calculated according to a predetermined distance criterion, and the corresponding subset of coding parameters of the simply connected region that has the smallest distance is adopted as a second subset of coding parameters for the given simply connected region that is disjoint from the first subset, or alternatively, the second subset of coding parameters for the given simply connected region is predicted from the corresponding subset of coding parameters of the simply connected region that has the smallest distance, and a prediction residual for the given simply connected region is extracted from the data stream. decoder.

9. 1. A decoder for decoding a data stream in which a two-dimensional information signal is encoded, comprising: a subdivision unit configured to spatially sub-divide, by recursive multiple partitioning based on a first subset of syntax elements included in the data stream, an array of information samples representing a spatial sampling of the two-dimensional information signal into a plurality of simply connected regions of different sizes; a combiner for combining spatially adjacent ones of the plurality of simply connected regions based on a second subset of syntax elements in the data stream, the second subset being separated from the first subset, to obtain an intermediate subdivision of the array of information samples into disjoint sets of simply connected regions whose union is the plurality of simply connected regions; a reconstruction unit for reconstructing said sequence of information samples from the data stream using said intermediate subdivision; A decoder comprising:

10. 9. A decoder as claimed in claim 8, wherein the subdivision unit spatially divides the array of information samples into tree root regions of a maximum region size, and subdivides at least a subset of the tree root regions into smaller simply connected regions of different sizes by recursively partitioning the subset of the tree root regions multiple times according to the first subset of syntax elements, the tree root regions not partitioned according to the multi-tree subdivision information, and the smaller simply connected regions form a plurality of simply connected regions.

11. 11. A decoder according to claim 10, wherein the subdivision unit divides the array of information samples into tree root regions such that the tree root regions are rectangular blocks of a size determined by the maximum region size and are regularly arranged to cover the array of information samples without gaps.

12. A decoder according to claim 10 or 11, The subdivision unit, when subdividing the subset of the tree root region, for each tree root region, determining whether the each tree root region is partitioned in the first subset of syntax elements; If each of said tree root regions is partitioned, partitioning each of said tree-root regions into regions of said first hierarchical level according to a partitioning rule associated with said first hierarchical level; recursively repeating the checking and partitioning for the regions of the first hierarchical level to obtain regions of a higher hierarchical level using partitioning rules associated with the higher hierarchical level; stopping said recursive iteration if no further partitioning occurs according to said first subset of syntax elements or if a maximum hierarchical level has been reached; wherein the regions of the subset of the tree root region that are not further partitioned according to said first subset of syntax elements represent said smaller simply connected regions and leaf regions of a multi-tree subdivision, respectively. decoder.

13. 13. The decoder of claim 12, wherein the maximum hierarchical level is indicated by the first subset of syntax elements.

14. 14. A decoder as claimed in claim 12 or 13, wherein the subdivision unit performs partitioning into sub-regions of equal size according to the partitioning rules associated with the first hierarchical level and higher hierarchical levels, such that the number of resulting sub-regions is common for all hierarchical levels.

15. 15. A decoder according to any one of claims 12 to 14, wherein the reconstruction unit extracts from the data stream first syntax elements associated with the leaf regions of the subset of tree root regions in depth-first traversal order.

16. A decoder as claimed in any one of claims 12 to 15, wherein the first subset of syntax elements includes a partitioning indication flag associated with each tree root region and with regions of the first hierarchical level and higher hierarchical levels that do not belong to the region of the highest hierarchical level, the partitioning indication flag indicating whether the associated tree root region and the regions of the first hierarchical level and higher hierarchical level are each partitioned.

17. A decoder according to any one of claims 9 to 16, The reconstruction unit, at a granularity corresponding to the intermediate subdivision, determining a prediction mode to be used from at least an intra prediction mode and an inter prediction mode; Transformation from the spectral domain to the spatial domain, - performing inter prediction and / or setting parameters of inter prediction; and performing intra prediction and / or setting parameters of intra prediction; Perform one or more of the following: decoder.

18. A decoder according to any one of claims 9 to 16, The reconstructor performs the reconstruction at a granularity based on the intermediate subdivision. determining a prediction mode to be used from at least an intra prediction mode and an inter prediction mode; Transformation from the spectral domain to the spatial domain, performing inter prediction and / or setting parameters of inter prediction; performing intra prediction and / or setting parameters of intra prediction; Extracting and setting encoding parameters; - extracting coding parameter predictions and extracting coding parameter residuals and setting coding parameters by combining said coding parameter predictions and said coding parameter residuals at a finer granularity determined by a subdivision of said small simply connected regions; Perform one or more of the following: decoder.

19. extracting payload data from the data stream for each of a plurality of simply connected regions into which the array of information samples representing the spatially sampled information signal is subdivided; reconstructing the sequence of information samples from the payload data for each singly connected region of the sequence of information samples by processing, for each singly connected region, payload data for the each singly connected region in a manner indicated by encoding parameters associated with the each singly connected region; In the extraction, for a given simply connected region, identifying a simply connected region among the plurality of simply connected regions that is in a given relative positional relationship with the given simply connected region; If the number of simply connected regions in a predetermined relative position to the given simply connected region is greater than zero, a merge indicator for the given simply connected region is extracted from the data stream. Extract from If the merge indicator indicates a merge operation for the given block, if the number of simply connected regions in a predetermined relative position to the given simply connected region is one, then the coding parameters of the simply connected region are adopted as the coding parameters for the given simply connected region, or the coding parameters for the given simply connected region are predicted from the coding parameters of the simply connected regions in a predetermined relative position to the given simply connected region, and a prediction residual for the given simply connected region is extracted from the data stream. Decryption method.

20. 1. A method for decoding a data stream in which a two-dimensional information signal is encoded, comprising the steps of: spatially subdividing the array of information samples representing a spatial sampling of the two-dimensional information signal into a plurality of simply connected regions of different sizes by recursive multiple partitioning based on a first subset of syntax elements included in the data stream; combining spatially adjacent ones of the plurality of simply connected regions based on a second subset of syntax elements in the data stream, the second subset being separated from the first subset, to obtain an intermediate subdivision of the array of information samples into disjoint sets of simply connected regions whose union is the plurality of simply connected regions; Reconstructing the array of information samples from the data stream using the intermediate subdivision. method.

21. 1. An encoder for encoding an array of information samples representing a spatially sampled information signal into payload data for each of a number of simply connected regions into which the array of information samples is subdivided, and encoding coding parameters associated with each of the simply connected regions to indicate how payload data for each of the simply connected regions is reconstructed from the payload data for each of the simply connected regions, comprising: for a given simply connected region, identifying a simply connected region among the plurality of simply connected regions that is in a given relative positional relationship with the given simply connected region; if the number of simply connected regions in a predetermined relative position to the given simply connected region is greater than zero, then add a merge indicator for the given simply connected region to the data stream. Insert, If the merge indicator indicates a merge operation for the given block, if the number of simply connected regions in a predetermined relative position to the given simply connected region is one, then do not insert the coding parameters of the given simply connected region into the data stream, or alternatively predict the coding parameters for the given simply connected region from the coding parameters of the simply connected regions in a predetermined relative position to the given simply connected region, and insert a residual of the prediction for the given simply connected region into the data stream. Encoder.

22. 22. An encoder as claimed in claim 21, further comprising: if the number of simply connected regions which are in a predetermined relative position to the given simply connected region is greater than one, the encoder checks whether the coding parameters associated with the simply connected regions are identical to each other, and if so, does not insert the coding parameters for the given simply connected region into the data stream; or, alternatively, predicts the coding parameters for the given simply connected region from the coding parameters of the number of simply connected regions which are in a predetermined relative position to the given simply connected region and inserts a residual of the prediction into the data stream.

23. 1. An encoder for generating a data stream in which a two-dimensional information signal is encoded, comprising: a subdivision / combination stage for determining a first subset of syntax elements defining a spatial subdivision, by recursive multiple partitioning, of the array of information samples representing a spatial sampling of said two-dimensional information signal into a plurality of simply connected regions of different sizes, and a second subset of syntax elements separated from said first subset defining a combination of spatially adjacent simply connected regions of said plurality of simply connected regions to obtain an intermediate subdivision of said array of information samples into disjoint sets of simply connected regions whose union is said plurality of simply connected regions; a data stream generator for encoding the sequence of information samples into a data stream using the intermediate subdivision and inserting the first and second subsets of syntax elements into the data stream; An encoder comprising:

24. 1. An encoder configured to encode an array of information samples representing a spatially sampled information signal into payload data for each of a plurality of simply connected regions into which the array of information samples is subdivided, and to encode coding parameters associated with each of the simply connected regions to indicate how payload data for each of the simply connected regions is reconstructed from the payload data for each of the simply connected regions, inserting a first subset of coding parameters for a given simply connected region into the data stream; identifying a simply connected region that is located in a predetermined relative positional relationship with respect to the predetermined simply connected region among the plurality of simply connected regions; if the number of simply connected regions in a predetermined relative position to the given simply connected region is greater than zero, then add a merge indicator for the given simply connected region to the data stream. Insert, If the merge indicator indicates a merge operation for the given block, calculating, for each of a plurality of simply connected regions at a predetermined relative position to the predetermined simply connected region, a distance between the first subset of coding parameters of the given simply connected region and a corresponding subset of coding parameters of each simply connected region at a predetermined relative position to the given simply connected region according to a predetermined distance criterion, not inserting a second subset of coding parameters for the given simply connected region which is disjoint from the first subset into the data stream, or predicting the second subset of coding parameters for the given simply connected region from the corresponding subset of coding parameters of the simply connected region which has the smallest distance, and inserting a residual of the prediction for the given simply connected region into the data stream. Encoder.

25. 1. A method of encoding an array of information samples representing a spatially sampled information signal into payload data for each of a number of simply connected regions into which the array of information samples is subdivided, and encoding coding parameters associated with each of the simply connected regions to indicate how payload data for each of the simply connected regions is reconstructed from the payload data for each of the simply connected regions, comprising: for a given simply connected region, identifying a simply connected region among the plurality of simply connected regions that is in a given relative positional relationship with the given simply connected region; if the number of simply connected regions in a predetermined relative position to the given simply connected region is greater than zero, then add a merge indicator for the given simply connected region to the data stream. Insert, If the merge indicator indicates a merge operation for the given block, if the number of simply connected regions in a predetermined relative position to the given simply connected region is one, then do not insert the coding parameters of the given simply connected region into the data stream, or alternatively predict the coding parameters for the given simply connected region from the coding parameters of the simply connected regions in a predetermined relative position to the given simply connected region, and insert a residual of the prediction for the given simply connected region into the data stream. A method for encoding information sample arrays.

26. 1. A method for generating a two-dimensional information signal encoded data stream, comprising the steps of: determining a first subset of syntax elements defining a spatial subdivision, by recursive multiple partitioning, of an array of information samples representing a spatial sampling of the two-dimensional information signal into a plurality of simply connected regions of different sizes, and a second subset of syntax elements separated from the first subset defining a combination of spatially adjacent simply connected regions of the plurality of simply connected regions to obtain an intermediate subdivision of the array of information samples into disjoint sets of simply connected regions whose union is the plurality of simply connected regions; Encoding the sequence of information samples into a data stream using the intermediate subdivision and inserting the first and second subsets of syntax elements into the data stream. Data stream generation method.

27. A computer readable digital storage medium having stored thereon a computer program comprising a program code which, when running on a computer, performs the method according to any one of claims 19, 20, 25 and 26.

28. a data stream encoding an array of information samples representing a spatially sampled information signal, the data stream including payload data for each of a number of simply connected regions into which the array of information samples is subdivided, and coding parameters associated with each simply connected region for indicating how the payload data for each simply connected region is reconstructed from the payload data for each simply connected region; moreover, a merging index for a given simply connected region, the number of simply connected regions being in a predetermined relative positional relationship with respect to the given simply connected region being greater than zero, among the plurality of simply connected regions; If the merging indicator for each given singly connected region indicates a merging operation and the number of simply connected regions in a predetermined relative position to each given singly connected region is one, then the coding parameters for each given singly connected region are not included in the data stream, or a prediction residual for each given singly connected region is included for reconstructing the coding parameters for each given singly connected region by predicting them from the coding parameters of the simply connected regions in a predetermined relative position to each given singly connected region. Data stream.

29. 29. A data stream as claimed in claim 28, wherein if the number of simply connected regions which are in a predetermined relative position to each of said predetermined singly connected regions is greater than one and the coding parameters associated with said simply connected regions are identical to each other, the data stream does not include coding parameters for each of said predetermined singly connected regions, or further includes prediction residuals for each of said predetermined singly connected regions for reconstructing the coding parameters for each of said predetermined singly connected regions by predicting them from the coding parameters of the number of simply connected regions which are in a predetermined relative position to each of said predetermined singly connected regions.

30. A data stream in which a two-dimensional information signal is encoded, a first subset of syntax elements defining a spatial subdivision, by recursive multiple partitioning, of an array of information samples representing a spatial sampling of said two-dimensional information signal into a plurality of simply connected regions of different sizes; a second subset of syntax elements separated from the first subset that defines a combination of spatially adjacent simply connected regions of the plurality of simply connected regions to obtain an intermediate subdivision of the sequence of information samples into disjoint sets of simply connected regions whose union is the plurality of simply connected regions; The sequence of information samples is encoded into the data stream based on the intermediate subdivision. Data stream.

Images