Combining specimen regions

Abstract

The present invention provides a method for reducing the amount of data used in reconstructing an array of samples from a data stream by encoding desired combinations or groupings of simply connected regions into which the array of information samples is subdivided. [Solution] The encoder 10 defines a predetermined relative positional relationship for a simply connected region, identifies a simply connected region among a plurality of simply connected regions that has a predetermined relative positional relationship to the predetermined simply connected region, spatially subdivides the region of samples representing a spatial sampling of a two-dimensional information signal into a plurality of simply connected regions of different sizes by recursively partitioning multiple times according to a first subset of syntax elements included in the data stream, combines spatially adjacent simply connected regions according to a second subset of syntax elements in the data stream that is separated from the first subset, and performs intermediate subdivision of the array of samples into disjoint sets of simply connected regions whose union is a plurality of simply connected regions.

Description

[Technical Field]

[0001] The present invention relates to a coding scheme for two-dimensionally sampled information signals such as moving or still images. [Background technology]

[0002] Typically, in image and video coding, an image, or a set of specific sample arrays within an image, is decomposed into blocks and associated with specific coding parameters. An image typically consists of multiple sample arrays. Furthermore, an image may be associated with additional auxiliary sample arrays that specify, for example, transparency information or depth maps. The sample arrays (including the auxiliary sample arrays) of an image can be grouped into one or more so-called plane groups, each consisting of one or more sample arrays. A plane group of an image can be coded independently or, if the image is associated with multiple plane groups, can be coded by prediction from other plane groups of the same image. Typically, each plane group is decomposed into blocks. Blocks (or corresponding blocks of sample arrays) are predicted using either inter-picture or intra-picture prediction. Blocks can be of various sizes and can be either square or rectangular. The partitioning of an image into blocks can be fixed by syntax or (at least in part) signaled within the bitstream. Often, syntax elements are transmitted that signal the subdivision of blocks of a given size. These syntax elements specify whether and how a block is to be subdivided into smaller blocks and associated with coding parameters, e.g., for prediction purposes. For every sample of a block (or the corresponding block of a sample array), the decoding of the associated coding parameters is specified in a specific way. For example, all samples in a block are predicted using the same set of prediction parameters, such as reference indices (identifying a reference picture among a set of previously coded pictures), motion parameters (specifying the amount of block motion between the reference picture and the current picture), and parameters for specifying interpolation filters, intra-prediction modes, etc. Motion parameters can be represented by displacement vectors with horizontal and vertical components, or by higher-order motion parameters, such as affine motion parameters with six components. It is also possible to associate multiple sets of specific prediction parameters (e.g., reference indices, motion parameters, etc.) with a single block.In this case, for each set of these specific prediction parameters, a single intermediate prediction signal for the block (or the corresponding block in the sample array) is generated, and the final prediction signal is formed by combining the intermediate prediction signals, including superposition. The corresponding weighting parameters, possibly with a constant offset (added to the weighted sum), can be fixed for either the image, the reference image, or the set of reference images, or can be included in the set of prediction parameters for the corresponding block. The difference between the original block (or the corresponding block in the sample array) and the prediction signal, also known as the residual signal, is typically transformed and quantized. A two-dimensional transform is often applied to the residual signal (or the corresponding sample array of the residual block). For transform coding, a block (or the corresponding block in the sample array) for which a specific set of prediction parameters is used can be further divided before applying the transform. The transform block can be the same size as the block used for prediction, or smaller. A transform block can also contain multiple blocks used for prediction. Each transform block can be of various sizes, resulting in a square or rectangular block. After transformation, the resulting transform coefficients are quantized to obtain so-called transform coefficient levels. The transform coefficient levels and prediction parameters, and subdivision information, if present, are entropy coded. Summary of the Invention [Problem to be solved by the invention]

[0003] In image and video coding standards, the syntax provides very limited possibilities for subdividing an image (or a group of planes) into blocks. It is usually possible to specify only whether and, possibly, how a block of a given size can be subdivided into smaller blocks. For example, in H.264, the maximum block size is 16x16. A 16x16 block is also called a macroblock, and each image is partitioned into macroblocks in the first step. For each 16x16 macroblock, it is possible to signal whether it will be coded as a 16x16 block, two 16x8 blocks, two 8x16 blocks, or four 8x8 blocks. If a 16x16 block is subdivided into four 8x8 blocks, each of these 8x8 blocks can be coded as one 8x8 block, two 8x4 blocks, two 4x8 blocks, or four 4x4 blocks. In current image and video coding standards, reducing the possibilities for specifying partitioning into blocks has the advantage of keeping the side information rate for signaling subdivision information small, but the disadvantage, as explained below, is that the bit rate required for transmitting prediction parameters for a block can be large. The side information rate for signaling prediction information usually accounts for a significant portion of the overall bit rate for a block. Therefore, reducing this side information can improve coding efficiency, which can be achieved, for example, by using larger block sizes. Images in real images and video sequences consist of objects of arbitrary shapes with specific properties. For example, these objects or object parts are characterized by a specific texture or a specific motion. The same set of prediction parameters is usually applied to these objects or object parts. However, object boundaries usually do not coincide with the boundaries of the possible blocks of a larger prediction block (e.g., a 16x16 macroblock in H.264). The encoder usually makes a subdivision decision (among a limited number of possibilities) that minimizes a certain rate / distortion loss.This can result in a large number of small blocks for an object of arbitrary shape. Each of these small blocks is associated with a set of prediction parameters that need to be transmitted, and the associated information rate can become a significant fraction of the overall bit rate. However, since some of the small blocks still represent the same object or a region of the object, the prediction parameters of the resulting large number of blocks will be identical or very similar.

[0004] That is, subdivision or tiling of an image into smaller parts or tiles or blocks significantly impacts the coding efficiency and coding complexity. As outlined above, the more subdivision of an image into smaller blocks, the more spatial granularity the coding parameter settings can be, and the better these coding parameters adapt to the image / video material. On the other hand, the more granular the coding parameter settings, the greater the burden of the amount of accompanying information needed to inform the decoder of the desired settings. It should also be noted that allowing the encoder to (further) spatially subdivide an image / video into blocks significantly increases the amount of possible coding parameter settings, and as a result, it generally becomes more difficult to find the best rate / distortion compromise for the coding parameter settings. [Means for solving the problem]

[0005] The present invention provides a coding scheme for encoding arrays of information samples representing spatially sampled two-dimensional information signals, such as, but not limited to, moving images and still images, that allows for an improved trade-off between encoding complexity and achievable rate / distortion ratio and / or allows for improved rate / distortion ratios to be achieved.

[0006] This object is achieved by a decoder, an encoder, a method, a computer program and a data stream according to the independent claims.

[0007] In an embodiment, desired combinations or groupings of simply connected regions resulting from the subdivision of the sequence of information samples are coded to reduce the amount of data. For this purpose, a predetermined relative positional relationship is defined for the simply connected regions. This allows for the identification of a simply connected region among multiple simply connected regions that is located at a predetermined relative position relative to the given simply connected region. That is, if the number is zero, no merge indicator for the given simply connected region may be present in the data stream. Furthermore, if the number of simply connected regions located at a predetermined relative position relative to the given simply connected region is one, the coding parameters of the simply connected region may be employed or used to predict the coding parameters for the given simply connected region, and no further syntax elements are required. Otherwise, if the number of simply connected regions located at a predetermined relative position relative to the given simply connected region is greater than one, the introduction of further syntax elements may be suppressed even if the coding parameters associated with these identified simply connected regions are identical to each other.

[0008] In an embodiment, if the coding parameters of adjacent singly connected regions are not equal to each other, a proper subset of the number of singly connected regions at a predetermined relative position to a given singly connected region may be identified by a reference neighbor identifier, and this proper subset is used when adopting the coding parameters or predicting the coding parameters of a given singly connected region.

[0009] In yet another embodiment, a region of samples representing a spatial sampling of a two-dimensional information signal is spatially sub-divided by multiple recursive partitioning into multiple simply connected regions of different sizes according to a first subset of syntax elements contained in the data stream. Subsequently, spatially adjacent simply connected regions are combined according to a second subset of syntax elements in the data stream that is separated from the first subset, resulting in an intermediate sub-division of the sequence of samples into disjoint sets of simply connected regions whose union is multiple simply connected regions. The intermediate sub-division is used to reconstruct the sequence of samples from the data stream. This reduces the importance of optimizing the sub-division, since overly dense sub-divisions can be corrected by subsequent merging. Furthermore, the combination of sub-division and merging allows for intermediate sub-divisions that would not be possible using multiple recursive partitioning alone, and the combination of sub-division and merging using a set of separated syntax elements improves the fit of the effective or intermediate sub-division to the actual content of the two-dimensional information signal. The additional overhead incurred by the additional subset of syntax elements to indicate the details of the merging is negligible compared to the benefits.

[0010] In an embodiment, an array of information samples representing a spatially sampled information signal is first spatially divided into tree root regions and then subdivided according to multi-tree subdivision information extracted from the data stream. By performing multiple recursive partitioning on subsets of the tree root regions, at least a subset of the tree root regions is subdivided into smaller simply connected regions of various sizes. To be able to find a desirable compromise between too fine and too coarse subdivision in terms of rate / distortion with a reasonable encoding complexity, the size of the largest region of the tree root region into which the array of information samples is spatially divided is included in the data stream and extracted from the data stream at the decoding side. Therefore, the decoder may include an extraction unit capable of extracting the maximum region size and multi-tree subdivision information from the data stream, a subdivision unit capable of spatially dividing an array of information samples representing a spatially sampled information signal into tree root regions of the maximum region size, subdividing according to the multi-tree subdivision information, and subdividing at least a subset of the tree root regions into smaller simply connected regions of various sizes by recursively partitioning the subsets of the tree root regions multiple times, and a reconstruction unit capable of reconstructing the array of information samples from the data stream using the subdivision into smaller simply connected regions.

[0011] In an embodiment, the data stream also includes the maximum hierarchical level at which a subset of the root region of a tree is subject to multiple recursive partitioning, making it easier to convey multi-tree subdivision information and requiring fewer bits to encode.

[0012] Furthermore, the reconstruction unit may be configured to perform one or more of the following, with a granularity depending on the intermediate subdivision: determining a prediction mode to use from at least intra prediction modes and inter prediction modes; converting from the spectral domain to the spatial domain; performing inter prediction and / or setting parameters for inter prediction; performing intra prediction and / or setting parameters for intra prediction.

[0013] Furthermore, the extractor may be configured to extract syntax elements associated with leaf regions of the partitioned treeblock from the data stream in a depth-first traversal order. In this manner, the extractor can more likely utilize statistics of syntax elements of adjacent coded leaf regions than if a width-first traversal order were used. In another embodiment, a further subdivision unit is used to subdivide at least a subset of the smaller singly-connected regions into even smaller singly-connected regions according to further multi-tree subdivision information. The first-stage subdivision may be used by the reconstructor to predict the domain of information samples, and the second-stage subdivision may be used by the reconstructor to retransform from the spectral domain to the spatial domain. Defining the residual subdivision at a lower level relative to the prediction subdivision reduces the overall bit consumption for coding the subdivision. Meanwhile, the limitations and freedom of the residual subdivision due to the lower definition have a slight negative impact on coding efficiency. This is because in most cases the parts of an image with similar motion compensation parameters are larger than the parts with similar spectral characteristics.

[0014] In yet another embodiment, a further maximum region size is included in the data stream, which defines the size of the tree-root subregions into which the smaller simply-connected regions are first divided before at least a subset of the tree-root subregions are further subdivided into smaller simply-connected regions according to the further multi-tree subdivision information. This allows for a unique setting of the maximum region size for prediction subdivision, while allowing for an improved rate / distortion trade-off through residual subdivision.

[0015] In yet another embodiment of the present invention, a first subset of syntax elements, separated from a second subset of syntax elements forming the multi-tree subdivision information, is included in the data stream. A decoder-side combiner can combine smaller, spatially adjacent simply connected regions of the multi-tree subdivisions according to the first subset of syntax elements to obtain an intermediate subdivision of the sample array. The reconstructor can be configured to reconstruct the sample array using the intermediate subdivision. This allows the encoder to perform subdivisions that are effective for the spatial distribution of the characteristics of the information sample array, making it easier to find an optimal rate / distortion compromise. For example, if the size of the largest region is large, the tree root region is likely to be large, making the multi-tree subdivision information complex. On the other hand, if the size of the largest region is small, it is likely that neighboring tree root regions are related to information content with similar characteristics and are also processed together. Combining eliminates the gap between the above two extreme cases, thereby enabling subdivision with a granularity close to the optimal level. From the encoder's point of view, the combined syntax element simplifies the encoding procedure or reduces the computational complexity, since if the encoder mistakenly uses too fine a subdivision, it corrects this mistake by subsequently setting the combined syntax element, with or without adaptation of a small fraction of the syntax elements that were set before the setting of the combined syntax element.

[0016] In yet another embodiment, the maximum region size and multi-tree subdivision information is used for residual subdivision rather than prediction subdivision.

[0017] When dealing with singly connected regions of a quadtree subdivision of an array of information samples representing a spatially sampled information signal, a depth-order traversal order is used in accordance with an embodiment, rather than a width-order traversal order. Using a depth-order traversal order increases the probability that each singly connected region has neighboring singly connected regions that have been traversed, and ensures that information about these neighboring singly connected regions is available when reconstructing each current singly connected region.

[0018] If the array of information samples is first partitioned into a regular array of tree root regions of zeroth-order hierarchical size, and then at least a subset of the tree root regions is subdivided into smaller simply connected regions of different sizes, the reconstructor may traverse the tree root regions using a zigzag scan, and for each partitioned tree root region, process the simply connected leaf regions in a depth-first traversal order, and then proceed to the next tree root region in a zigzag scan order. Furthermore, according to the depth-first traversal order, simply connected leaf regions at the same hierarchical level may be traversed in a zigzag scan order. In this way, a state in which the probability of having adjacent simply connected leaf regions is increased is maintained.

[0019] In an embodiment, flags associated with nodes of the multi-tree structure are arranged sequentially in depth-order traversal order, but the sequential encoding of flags uses a probability estimation context that is the same for flags associated with nodes of the multi-tree structure located at the same hierarchical level of the multi-tree structure, but different for nodes of the multi-tree structure located at different hierarchical levels of the multi-tree structure, thereby enabling an improved compromise between the amount of context provided and adaptation to the statistics of the actual symbols of the flags.

[0020] In an embodiment, the context of the probability estimation for a given flag used also depends on proceeding through the depth-first traversal order of the given flag, and on the fact that the given flag corresponds to a region of the root region of the tree that is in a predetermined relative position to the corresponding region. Similar to the idea underlying the described aspect, the use of a depth-first traversal order ensures with a high probability that the encoded flags will include flags that correspond to regions that are adjacent to the region corresponding to the given flag. This knowledge is exploited to improve the adaptability to the context used for the given flag.

[0021] The flags used to set the context for a given flag may correspond to regions located above and / or to the left of the region to which the given flag corresponds. Furthermore, the flags used to select a context may be restricted to flags that belong to the same hierarchical level as the node with which a given flag is associated.

[0022] In an embodiment, the coded signaling information includes an indication of the highest hierarchical level and a set of flags associated with nodes of the multi-tree structure other than the highest hierarchical level. Each flag specifies whether the associated node is an intermediate node or a child node. The set of flags is sequentially decoded from the data stream in a depth-order or breadth-order traversal order, skipping nodes at the highest hierarchical level and automatically designating the same leaf node. This results in a reduced coding rate.

[0023] In other embodiments, the highest hierarchical level may be indicated in the coded signaling of the multi-tree structure, thereby restricting the presence of flags to hierarchical levels other than the highest hierarchical level, thereby preventing further segmentation of blocks at the highest hierarchical level.

[0024] If a spatial multi-tree subdivision is part of a secondary subdivision of a leaf node and the root region of the tree of the primary multi-tree subdivision is not partitioned, the context used to encode the flags of the secondary subdivision may be selected such that the context is the same for flags associated with regions of the same size. Preferred embodiments of the present invention will now be described with reference to the following figures: [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a configuration diagram illustrating an encoder according to an embodiment of the present application; [Figure 2] FIG. 2 is a block diagram showing a decoder according to an embodiment of the present invention; [Figure 3A] 1 is a schematic diagram illustrating an example of a quadtree subdivision, showing the first hierarchical level. [Figure 3B] FIG. 1 is a schematic diagram illustrating an example of a quadtree subdivision, showing the second hierarchical level. [Figure 3C] FIG. 1 is a schematic diagram illustrating an example of a quadtree subdivision, showing the third hierarchical level. [Figure 4] FIG. 4 is a schematic diagram illustrating a tree structure of a subdivision of the quadtree of FIGS. 3A to 3C according to an embodiment. [Figure 5A] FIG. 3D is a schematic diagram illustrating a subdivision of the quadtree of FIGS. 3A to 3C. [Figure 5B] FIG. 4 is a schematic diagram showing the indexed tree structure of the individual leaf blocks of FIGS. 3A to 3C. [Figure 6A] 5A-5C are schematic diagrams illustrating sequences of binary strings or flags representing subdivisions of the tree structure of FIG. 4 and the quadtrees of FIGS. 3A-3C according to different embodiments. [Figure 6B] 5A-5C are schematic diagrams illustrating sequences of binary strings or flags representing subdivisions of the tree structure of FIG. 4 and the quadtrees of FIGS. 3A-3C according to different embodiments. [Figure 7] 10 is a flowchart illustrating steps performed by a data stream extraction unit according to an embodiment. [Figure 8]10 is a flowchart illustrating the functioning of a data stream extraction unit according to another embodiment; [Figure 9A] FIG. 1 is a schematic diagram illustrating a quadtree subdivision according to an embodiment, showing candidate blocks adjacent to a given block highlighted; [Figure 9B] FIG. 1 is a schematic diagram illustrating a quadtree subdivision according to an embodiment, showing candidate blocks adjacent to a given block highlighted; [Figure 10] 10 is a flowchart illustrating the functioning of a data stream extraction unit according to another embodiment; [Figure 11] 1 is a schematic diagram illustrating the construction of an image from planes and plane groups and illustrating encoding using inter-plane adaptation / prediction according to an embodiment. FIG. [Figure 12A] FIG. 2 is a schematic diagram showing a subtree structure to explain an inheritance scheme according to an embodiment. [Figure 12B] FIG. 2 is a schematic diagram illustrating subdivisions corresponding to subtree structures to explain an inheritance scheme according to an embodiment. [Figure 12C] FIG. 2 is a schematic diagram illustrating a subtree structure to explain the inheritance scheme of adoption and prediction according to an embodiment. [Figure 12D] FIG. 2 is a schematic diagram illustrating a subtree structure to explain the inheritance scheme of adoption and prediction according to an embodiment. [Figure 13] 10 is a flowchart illustrating steps performed by an encoder to implement an inheritance scheme according to an embodiment. [Figure 14] FIG. 14A is a diagram illustrating a first-order subdivision to illustrate the possibility of implementing an inheritance scheme in relation to inter-prediction according to an embodiment, and FIG. 14B is a diagram illustrating a lower-order subdivision to illustrate the possibility of implementing an inheritance scheme in relation to inter-prediction according to an embodiment. [Figure 15] FIG. 1 is a block diagram illustrating the process of decoding in relation to an inheritance scheme, according to an embodiment. [Figure 16]1 is a schematic diagram illustrating the order of traversal of sub-regions of a multi-tree subdivision for a sub-region subject to intra prediction according to an embodiment; FIG. [Figure 17] FIG. 2 is a configuration diagram showing a decoder according to an embodiment. [Figure 18A] 5A and 5B are schematic diagrams illustrating different subdivision possibilities in another embodiment; [Figure 18B] 5A and 5B are schematic diagrams illustrating different subdivision possibilities in another embodiment; [Figure 18C] 5A and 5B are schematic diagrams illustrating different subdivision possibilities in another embodiment; [Figure 19] FIG. 1 is a configuration diagram showing an encoder according to an embodiment. [Figure 20] FIG. 10 is a block diagram showing a decoder according to another embodiment of the present invention; [Figure 21] FIG. 10 is a configuration diagram showing an encoder according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] In the following description of the figures, elements that appear in multiple figures are designated by common reference numerals, and redundant descriptions of these elements are omitted. Rather, a description of an element that appears in one figure also applies to the other figures, if that element also appears in those figures, so long as the accompanying descriptions for those figures explain any differences regarding that element.

[0027] Furthermore, the following description begins with the encoder and decoder embodiments described in Figures 1 through 11. The embodiments described for these figures combine many aspects of the present application, but are also effective when implemented separately within a coding scheme. Therefore, for the following figures, embodiments utilizing the above aspects using each of these embodiments separately will be briefly described, and will represent different abstractions of the embodiments described for Figures 1 and 11.

[0028] FIG. 1 illustrates an encoder according to an embodiment of the present invention. The encoder 10 of FIG. 1 includes a prediction unit 12, a residual precoder 14, a residual reconstruction unit 16, a data stream insertion unit 18, and a block segmentation unit 20. The encoder 10 is configured to encode a temporally and spatially sampled information signal into a data stream 22. The temporally and spatially sampled information signal may be, for example, a video, i.e., a series of images. Each image represents an array of image samples. Other examples of temporally and spatial information signals include depth images captured by a time-of-light camera. It should be noted that the spatially sampled information signal may contain multiple arrays per frame or timestamp. For example, a color video may contain one array of luma samples per frame along with two arrays of chroma samples. The temporal sampling rates of different components of the information signal, i.e., luma and chroma, may differ, as may the spatial resolution. The video may also be accompanied by additional spatially sampled information, such as depth or transparency information. However, in the following description, in order to facilitate understanding of the main contents of the present application, we will first focus on the processing of one of these arrays, and then discuss the handling of multiple planes.

[0029] 1 is configured to create a data stream 22 such that the syntax elements of the data stream 22 describe images at a granularity lying between the entire image and individual image samples. To this end, the segmentation unit 20 is configured to subdivide each image 24 into singly connected regions 26 of various sizes. In the following, these regions will be referred to simply as blocks or subregions 26.

[0030] As will be explained in more detail below, the segmentation unit 20 uses multi-tree subdivision to subdivide the image 24 into blocks 26 of various sizes. More precisely, in the particular embodiment outlined below with respect to Figures 1 to 11, it mostly uses quad-tree subdivision. The segmentation unit 20 may also comprise an internally connected subdivision unit 28, as will be explained in more detail below. The subdivision unit 28 subdivides the image 24 into said blocks 26. The subdivision unit 28 is connected to a combiner unit 30. The combiner unit 30 can combine groups of these blocks 26 to obtain an effective subdivision or granularity lying between the non-subdivision of the image 24 and the subdivision defined by the subdivision unit 28.

[0031] 1, the predictor 12, the residual precoder 14, the residual reconstructor 16, and the data stream inserter 18 perform the image subdivision defined by the partitioner 20. For example, as described in more detail below, the predictor 12 uses the prediction subdivision defined by the partitioner 20 to: Set prediction parameters corresponding to each sub-region according to the selected prediction mode, and make a decision for each individual sub-region of the prediction subdivision as to whether the respective sub-region should be subjected to intra-picture prediction or inter-picture prediction.

[0032] The residual precoder 14 may then use the subdivision of the residual of the image 24 to encode the residual of the prediction of the image 24 provided by the predictor 12. When the residual reconstructor 16 reconstructs the residual from the syntax elements output by the residual precoder 14, the residual reconstructor 16 also performs the subdivision of the residual. The data stream inserter 18 may use the subdivision of the prediction and the residual to: determine the insertion order and adjacency of the syntax elements output by the residual precoder 14 and predictor 12 into the data stream 22, for example using entropy encoding, for insertion of the syntax elements.

[0033] As shown in FIG. 1, the encoder 10 includes an input 32 through which an original information signal is input. A subtractor 34, a residual precoder 14, and a data stream inserter 18 are serially connected, in that order, between the input 32 and the output of the data stream inserter 18, from which the encoded data stream 22 is output. The subtractor 34 and the residual precoder 14 are part of a prediction loop. The prediction loop is closed by a residual construction unit 16, an adder 36, and a predictor 12, which are serially connected in the following order between the output of the residual precoder 14 and the inverting input of the subtractor 34. The output of the predictor 12 is also connected to another input of the adder 36. The predictor 12 may also have another input directly connected to the input 32 and also connected to the output of the adder 36 via an optional in-loop filter 38. Furthermore, since the prediction unit 12 generates accompanying information during operation, an output of the prediction unit 12 is also connected to a data stream insertion unit 18. Similarly, the splitting unit 20 has an output connected to another input of the data stream insertion unit 18. Having described the structure of encoder 10, the mode of operation will now be described in more detail.

[0034] As explained above, the segmentation unit 20 determines for each image 24 how to sub-divide it into sub-regions 26. According to the sub-division of the image 24 used for prediction, the prediction unit 12 determines for each sub-region corresponding to this sub-division how to predict the respective sub-region. The prediction unit 12 outputs a prediction of the sub-region to an inverting input of the subtractor 34 and to another input of the adder 36, and outputs prediction information to the data stream insertion unit 18 that reflects how the prediction unit 12 obtained this prediction from previously encoded parts of the video.

[0035] At the output of the subtractor 34, the residual precoder 14 processes this prediction residual according to the residual subdivision defined by the divider 20 to obtain a prediction residual. As will be explained in more detail below with reference to Figures 3 to 10, the residual subdivision of the image 24 used by the residual precoder 14 may be related to the prediction subdivision used by the predictor 12, such that each prediction subregion is adopted as a residual subregion or is further subdivided into smaller residual subregions. There may also be completely independent prediction and residual subdivisions.

[0036] The residual precoder 14 performs a two-dimensional transformation from the spatial domain to the spectral domain on each residual sub-region. This is followed by or essentially involves quantization of the resulting transform coefficients of the resulting transform block, whereby distortion occurs as a result of quantization noise. For example, the data stream inserter 18 may losslessly encode syntax elements describing these transform coefficients into the data stream 22, e.g., using entropy encoding.

[0037] The residual reconstruction unit 16 then retransforms the transform coefficients into a residual signal using requantization with retransform. To obtain the prediction residual, the residual signal is combined in an adder 36 with the prediction used by the subtractor 34. As a result, at the output of the adder 36, a reconstructed part or sub-region of the current image is obtained. The predictor 12 may also directly use the reconstructed image sub-regions by intra-prediction, in order to predict a particular prediction sub-region by estimation from neighboring reconstructed prediction sub-regions. However, intra-prediction performed in the spectral domain by directly predicting the spectrum of the current sub-region from the spectra of neighboring sub-regions is also theoretically possible.

[0038] For inter-prediction, predictor 12 may use a filtered version of the encoded and reconstructed image by an optional in-loop filter 38. For example, in-loop filter 38 may comprise a deblocking filter and / or an adaptive filter having a transfer function that benefits from shaping the quantization noise as described above.

[0039] The predictor 12 selects prediction parameters that indicate how to perform a prediction of a particular prediction subregion compared to the original examples in the image 24. As explained in more detail below, the prediction parameters may include an indication of a prediction mode, such as intra-picture prediction or inter-picture prediction, for each prediction subregion. In the case of intra-picture prediction, the prediction parameters may include an indication of the angle along which the boundaries of the prediction subregion within which the intra-prediction is performed primarily extend. In the case of inter-picture prediction, the prediction parameters may include motion vectors, video indices, and / or higher-order motion transformation parameters. In the case of both intra-picture and / or inter-picture prediction, the prediction parameters may include optional filter information for filtering the reconstructed image examples on which the current prediction subregion is predicted.

[0040] As will be explained in more detail below, the subdivision defined by the partitioning unit 20 significantly affects the rate-distortion ratio achievable by the residual precoder 14, the predictor 12, and the data stream insertion unit 18. Finer subdivisions may result in better predictions obtained by the predictor 12, and may result in a smaller residual signal encoded by the residual precoder 14 that can be coded using fewer bits, but they require a significantly higher coding rate for the prediction parameters 40 output by the predictor 12 and inserted into the data stream 22. The opposite is true for coarser subdivisions. The above considerations also apply equally to residual subdivisions. Transforming an image with a finer granularity for individual transform blocks reduces the computational complexity of the transform and increases the spatial resolution of the resulting transform. That is, smaller residual subregions result in more consistent spectral distributions of the contents of the individual residual subregions. However, the spectral resolution decreases, and the ratio between significant coefficients and insignificant coefficients, i.e., coefficients quantized to zero, deteriorates. In other words, the granularity of the transformation must be locally adapted to the image content. Furthermore, apart from the positive effects of a finer granularity, a finer granularity usually increases the amount of accompanying information needed to indicate the selected subdivision to the decoder. As will be explained in more detail below, the following embodiments describe an encoder 10 that is able to adapt the subdivision very effectively to the content of the information signal to be encoded, and that is able to communicate the subdivision information to be used on the decoding side by instructing a data stream insertion unit 18 to insert subdivision information into the coded data stream 22.

[0041] However, before defining the subdivision of the divider 20 in detail, a decoder according to an embodiment of the present application will be described in detail with reference to FIG.

[0042] The decoder of Figure 2 is generally designated by the reference numeral 100 and comprises an extraction unit 102, a segmentation unit 104, a residual reconstruction unit 106, an adder 108, a prediction unit 110, an optional in-loop filter 112, and an optional post-filter 114. The extraction unit 102 receives an encoded data stream at an input 116 of the decoder 100 and extracts, from subdivision information 118 of the encoded data stream, prediction parameters 120 and residual data 122 that the extraction unit 102 outputs to the image segmentation unit 104, the prediction unit 110, and the residual reconstruction unit 106, respectively. An output of the residual reconstruction unit 106 is connected to a first input of the adder 108. Another input and an output of the adder 108 are connected to a prediction loop. The prediction loop includes an optional in-loop filter 112 and a prediction unit 110 connected in series in this order, with a bypass path directly from the output of the adder 108 to the prediction unit 110. 1 above, one for intra-picture prediction and the other for inter-picture prediction. Either the output of the adder 108 or the output of the in-loop filter 112 may be connected to an output 124 of the decoder 100, at which the reconstructed information signal is output, e.g., to a playback device. An optional post-filter 114 may be connected in the path to the output 124 in order to improve the visual impression of the reconstructed signal at the output 124.

[0043] Generally, the residual reconstruction unit 106, the adder 108, and the prediction unit 110 function similarly to the elements 16, 36, and 12 in FIG. 1 . In other words, they perform operations similar to those of the above-mentioned elements in FIG. 1 . To this end, the residual reconstruction unit 106 and the prediction unit 110 are controlled by the prediction parameters 120 and the subdivision defined by the image division unit 104, respectively, in accordance with the subdivision information 118 from the extraction unit 102. This aims to predict a small region of prediction in the same way as the prediction unit 12 did or decided to do, and to retransform the received transform coefficients at the same granularity as the residual precoder 14 performed the transformation. Next, the image division unit 104 reorganizes the subdivision selected by the division unit 20 in FIG. 1 in a synchronized manner by referring to the subdivision information 118. The extraction unit may then use the subdivision information to control data extraction, such as with regard to context selection, neighbor relationship determination, probability estimation, data stream syntax analysis, etc.

[0044] Several variations of the above-described embodiment are possible. Some of the subdivisions performed by the subdivision unit 28 and the combinations performed by the combination unit 30 are described in detail below. Others are described with reference to Figures 12 through 16. Where appropriate, all of these variations may be applied individually or in subsets to the descriptions provided above for Figures 1 and 2, respectively. For example, the division units 20 and 104 may not only determine prediction subdivisions and residual subdivisions for each image. Instead, the subdivisions of the optional in-loop filters 38 and 112 may be determined either independently or dependently on other subdivisions for prediction or residual coding. Furthermore, the subdivision determination or redivision by these elements need not be performed on a frame-by-frame basis. Instead, the redivision or subdivision determined for a particular frame may be reused or adopted for a certain number of subsequent frames before moving on to a new subdivision.

[0045] To provide further details regarding the division of an image into smaller regions, the following description will first focus on the subdivision performed by subdivision units 28 and 104a, followed by the combining process performed by combiner 30 and combiner 104b, and finally, the inter-plane matching / prediction.

[0046] When the subdivision unit 28, 104a divides an image, it can divide the image into a number of blocks, potentially of various sizes, for the purposes of encoding predictions and residuals for image or video data. As mentioned above, the image 24 may be used as one or more arrays of image samples. In the case of a YUV / YCbCr color space, for example, one array may represent the luma channel, and the other two arrays may represent the chroma channels. These arrays may have different dimensions. All arrays may be grouped into one or more plane groups, each consisting of one or more consecutive planes, such that each plane is included in only one plane group. For each plane group, the following applies: The first array of a particular plane group is called the primary array of this plane group. Any subsequent arrays are called subarrays. The division of the blocks of the primary arrays may be based on a quadtree approach. An optimal approach is described below. The division of the blocks of the subarrays may be derived based on the division of the primary arrays.

[0047] In the embodiment described below, the subdivision unit 28, 104a is configured to divide the primary array into a number of equally sized square blocks, so-called treeblocks, as follows: If a quadtree is used, the length of the treeblock boundary is typically a power of two, such as 16, 32, 64, etc. However, it should be noted that strictly speaking, other tree types can also be used, such as binary trees or trees with any number of leaves. Furthermore, the number of children of a tree can vary depending on the level of the tree and the signal it represents.

[0048] Additionally, as mentioned above, the array of samples may represent information other than a video sequence, such as a depth map or a light field. For simplicity, the following description focuses on a quadtree as a representative example of a multi-tree. A quadtree is a tree in which each internal node has exactly four children. Each treeblock constitutes a primary quadtree, and each leaf of the primary quadtree constitutes a sub-quadtree. The primary quadtree determines the subdivision of a given treeblock for prediction, and the sub-quadtree determines the subdivision of a given prediction block for residual coding.

[0049] The root node of a first-order quadtree corresponds to a complete treeblock. For example, Figure 3A shows treeblock 150. It is assumed that each image is divided into a regular grid of lines and columns, such as treeblock 150, such that the sample array is completely covered. However, it should be noted that not all of the block subdivisions described below require seamless subdivisions without overlapping. Instead, adjacent blocks may overlap each other, as long as no leaf block is a proper subdivision of an adjacent leaf block.

[0050] According to the quadtree structure of treeblock 150, each node can be further divided into four child nodes. For a first-order quadtree, this means that each treeblock 150 can be divided into four subblocks, each half the width and height of treeblock 150. In FIG. 3A, these subblocks are designated 152a through 152d. In a similar manner, each of these subblocks can be further divided into four smaller subblocks, each half the width and height of the original subblock. In FIG. 3d, an example of subblock 152c is shown, subdivided into four smaller subblocks 154a through 154d. In FIGS. 3A through 3C, an example of how treeblock 150 is divided is shown: treeblock 150 is first divided into four subblocks 152a through 152d. Then, the bottom-left subblock 152c is further divided into four smaller subblocks 154a through 154d. Finally, as shown in Figure 3C, the top right of these smaller sub-blocks, block 154b, is again divided into four blocks each having one-eighth the width and height of the original treeblock 150. These smaller blocks are designated 156a through 156d.

[0051] Figure 4 shows the tree structure underlying the quadtree-based partitioning examples shown in Figures 3A through 3D. The numbers next to the tree nodes are values ​​called subdivision flags, which will be explained in more detail later when discussing quadtree structure signaling. The root node of the quadtree is shown at the top of the figure ("Level 0"). The four branches of this root node at level 1 correspond to the four subblocks shown in Figure 3A. In Figure 3B, the third of these subblocks is further subdivided into four subblocks, so the third node at level 1 in Figure 4 also has four branches. In addition, there are four subordinate branches connecting to the second node at level 2 of the quadtree hierarchy, corresponding to the subdivision of the second (top right) child node in Figure 3C. The node at level 3 is not further subdivided.

[0052] Each leaf of the primary quadtree corresponds to a variable-sized block, for which individual prediction parameters can be specified (i.e., intra- or inter-prediction mode, motion parameters, etc.). These blocks are hereinafter referred to as prediction blocks. These leaf blocks are specifically illustrated in FIG. 3C. Briefly referring back to the descriptions of FIGS. 1 and 2, the partitioning unit 20 or subdivision unit 28 determines the subdivision of the quadtree described above. The subdivision units 152a-152d determine which of the treeblock 150, subblocks 152a-152d, smaller subblocks 154a-154d, etc., should be subdivided or further partitioned. This aims to achieve an optimal tradeoff between fine prediction subdivision and coarse prediction subdivision, as already indicated above. The predictor 12 then uses the defined prediction subdivision to determine the above-mentioned prediction parameters with a granularity corresponding to the prediction subdivision. Alternatively, a defined prediction subdivision is used for each prediction subregion represented by the blocks shown in FIG. 3C, for example.

[0053] The prediction block shown in Figure 3C can be further divided into smaller blocks for the purpose of residual coding. For each prediction block, i.e., for each leaf node of the primary quadtree, one or more sub-quadtrees determine corresponding subdivisions for residual coding. For example, if the size of the residual block is at most 16x16, a given 32x32 prediction block can be divided into four 16x16 blocks, each determined by a sub-quadtree for residual coding. Each 16x16 block in this example corresponds to a root node of a sub-quadtree.

[0054] Similar to the subdivision of a given treeblock into prediction blocks, each prediction block can be divided into multiple residual blocks by decomposing the sub-quadtree. Each leaf of the sub-quadtree corresponds to a residual block. The residual blocks can be assigned individual residual coding parameters (i.e., transform mode, transform coefficients, etc.) by the residual precoder 14, followed by the residual reconstruction unit 16, 106, which are controlled by the residual coding parameters.

[0055] In other words, the subdivision unit 28 may be configured to determine a prediction subdivision and a lower-level residual subdivision for each image or each group of images. This is done as follows: First, the image is divided into a regular array of tree blocks 150, and a subset of these tree blocks is recursively partitioned by quadtree subdivision to obtain a prediction subdivision into prediction blocks. If no partitioning is performed on each tree block, or on leaf blocks of the quadtree subdivision, the prediction block may be a tree block. Then, the subset of these prediction blocks is further subdivided in a similar manner as follows: If the prediction block is larger than the maximum size of the lower-level residual subdivision, first, each prediction block is divided into a regular array of subtree blocks. Then, the subset of these subtree blocks is subdivided according to the quadtree subdivision procedure to obtain a residual block. If no partitioning is performed on each prediction block into subtree blocks, the residual block may be a prediction block. It may also be a subtree block if no further division into smaller regions is performed for each subtree block or subdivision leaf block of the residual quadtree.

[0056] As outlined above, the subdivision selected for the primary array may be mapped to a sub-array. This is easy when the sub-array has the same dimensions as the primary array. However, if the dimensions of the primary array and the sub-array are different, special methods are required. In general, mapping a subdivision of the primary array to a sub-array with different dimensions is possible by spatial mapping, i.e., by spatially mapping the block boundaries of the subdivision of the primary array to the sub-array. In particular, for each sub-array, horizontal and vertical scaling factors may be used that determine the ratio of the dimensions of the primary array to the sub-array. The division of the sub-array into sub-blocks for prediction and residual coding may be determined using the treeblocks of the sub-array that result from applying the relative scaling factors to the primary quadtree and sub-quadtree of each aligned treeblock of the primary array. If the horizontal and vertical scaling factors are different (e.g., 4:2:2 chroma subsampling), the resulting prediction blocks and residual blocks of the sub-array will not be square. In this case, it is possible to predetermine or select whether a non-square residual block (for the entire sequence, one image of the sequence, or a single prediction or residual block) is to be divided into square blocks or not depending on the situation. For example, in the first case, if the mapped block is not square, the encoder and decoder accept the subdivision into square blocks. In the second case, the subdivision unit 28 signals the selection to the subdivision unit 104a via the data stream insertion unit 18 and the data stream 22. For example, in the case of 4:2:2 chroma subsampling, if the subarray is half the width and the same height as the primary array, the height of the residual block is twice its width. By dividing this block vertically, two square blocks are obtained again from one block.

[0057] As mentioned above, the subdivision unit 28 or the division unit 20, respectively, communicates the quadtree-based division to the subdivision unit 104a via the data stream 22. To do so, the subdivision unit 28 informs the data stream insertion unit 18 of the subdivision selected for the image 24. The data stream insertion unit then transmits to the decoding side the primary and secondary quadtree structures, i.e., the division of the image array into blocks of variable size for prediction or residual coding in the data stream or bitstream 22.

[0058] The minimum and maximum allowed block sizes are transmitted as side information and may vary from picture to picture. They can also be fixed in the encoder and decoder. These minimum and maximum block sizes may be different for predicted and residual blocks. To communicate the structure of the quadtree, the quadtree needs to be traversed, and for each node, it needs to be specified whether this particular node is a leaf node of the quadtree (i.e., the corresponding block is not further subdivided) or whether it branches into four child nodes (i.e., the corresponding block is divided into four sub-blocks of half size).

[0059] Within an image, information is propagated treeblock by treeblock in a raster scan order, such as left to right and top to bottom, as shown at 140 in Figure 5A. Other scan orders are also possible, such as bottom right to top left, or a checkerboard pattern. In a preferred embodiment, to propagate subdivision information, each treeblock, i.e., each quadtree, is traversed in depth order.

[0060] In a preferred embodiment, not only the subdivision information, i.e., the structure of the tree, but also the prediction data, i.e., the payload associated with the leaf nodes of the tree, are transmitted / processed in depth-order. This is done because depth-order traversal has significant advantages over width-order traversal. Figure 5B shows the structure of a quadtree with leaf nodes denoted a, b, ..., j. Figure 5A shows the resulting block partitioning. If the blocks / leaf nodes are traversed in width order, they are obtained in the order abjchidefg, while in depth order they are abc...ij. As can be seen from Figure 5A, in depth order, the left-neighboring block and the top-neighboring block are always transmitted / processed before the current block. In this way, motion vector prediction and context modeling can always use the parameters specified for the left- and top-neighboring blocks, resulting in improved coding efficiency. This would not be the case in width order, since, for example, block j is transmitted before blocks e, g, and i.

[0061] Finally, the signaling for each treeblock is recursively performed along the primary quadtree structure, so that for each node, a flag is transmitted specifying whether the corresponding block is to be divided into four subblocks. If the flag value is "1" (indicating "true"), this signaling process is repeated recursively for all four child nodes or subblocks in raster scan order (top left, top right, bottom left, bottom right) until a leaf node of the primary quadtree is reached. Note that leaf nodes are characterized by having a subdivision flag value of "0." If a node is at the lowest hierarchical level of the primary quadtree and corresponds to the smallest allowable size of a prediction block, the subdivision flag need not be transmitted. In the example of Figures 3A-3C, a "1" is initially transmitted, indicating that treeblock 150 is to be divided into four subblocks 152a-152d, as shown at 190 in Figure 6A. Next, in raster scan order 200, encoding of subdivision information for all four sub-blocks 152a-152d is performed recursively. For the first two sub-blocks 152a and 152b, a "0" is transmitted (see 202 in FIG. 6A) specifying that no subdivision will be performed. For the third sub-block 152c (bottom left), a "1" is transmitted (see 204 in FIG. 6A) specifying that this block will be sub-divided. This recursive approach then processes the four sub-blocks 154a-154d of this block. Here, a "0" is transmitted (206) for the first sub-block, and a "1" is transmitted (208) for the second (top right) sub-block. This results in processing the four blocks 156a-156d, which represent the smallest block size in FIG. 3C. If the smallest allowable block size in this example is reached, no further subdivision is possible, and no further data needs to be transmitted. Alternatively, "0000" is transmitted to specify that these blocks are not further divided, as shown in Figure 6A at 210. After this, "00" is transmitted for the bottom two blocks of Figure 3B (see 212 in Figure 6A), and finally, a "0" is transmitted for the bottom right block of Figure 3A (see 214).Thus, the complete binary sequence representing the quadtree structure looks like Figure 6A.

[0062] In this binary sequence shown in Figure 6A, different background shadings correspond to different levels in the quadtree-based subdivision hierarchy. Shading 216 represents level 0 (corresponding to a block size equal to the original treeblock size), shading 218 represents level 1 (corresponding to a block size equal to half the original treeblock size), shading 220 represents level 2 (corresponding to a block size equal to 1 / 4 the original treeblock size), and shading 222 represents level 3 (corresponding to a block size equal to 1 / 8 the original treeblock size). All subdivision flags at the same hierarchical level (corresponding to the same block size and the same representation in the illustrated binary sequence) may be entropy coded, for example by the insertion unit 18, using one and the same probability model. In the case of width-order traversal, the subdivision information is transmitted in a different order as shown in Figure 6b.

[0063] Similar to the subdivision of each treeblock for prediction purposes, the resulting division of each prediction block into residual blocks must also be transmitted in the bitstream. The maximum and minimum block sizes for residual coding may also be transmitted as side information, and may vary from image to image. The maximum and minimum block sizes for residual coding may also be fixed within the encoder and decoder. At each leaf node of the primary quadtree shown in FIG. 3C, the corresponding prediction block may be divided into residual blocks of the maximum allowable size. These blocks are the root nodes that constitute the structure of the lower quadtree for residual coding. For example, if the maximum size of the residual blocks of an image is 64x64 and the size of the prediction blocks is 32x32, the entire prediction block corresponds to one root node of the lower (residual) quadtree of size 32x32. On the other hand, if the maximum size of the residual blocks of an image is 16x16, the 32x32 prediction block is composed of four root nodes of the residual quadtree of size 16x16. Within each prediction block, for each root node, the sub-quadtree structure is propagated in raster scan order (left to right, top to bottom). As in the case of the primary (prediction) quadtree structure, for each node, a flag is coded specifying whether this particular node is split into four child nodes. If the value of this flag is "1", then this procedure is repeated recursively for all four corresponding child nodes and corresponding sub-blocks in raster scan order (top left, top right, bottom left, bottom right) until a leaf node of the sub-quadtree is reached. As in the case of the primary quadtree, no propagating is required for the nodes at the lowest hierarchical level of the sub-quadtree, since these nodes correspond to the smallest size blocks of the residual block and cannot be further divided.

[0064] For entropy coding, the subdivision flags of residual blocks belonging to residual blocks of the same block size may be encoded using one and the same probability model.

[0065] Thus, according to the example illustrated above with reference to Figures 3A to 6A, the subdivision unit 28 defined a primary subdivision for prediction purposes and subdivisions of blocks of different sizes of the primary subdivision for residual coding purposes. The data stream insertion unit 18 coded the primary subdivision for each tree block in zigzag scan order by transmitting a bit sequence constructed according to Figure 6A, as well as coding the maximum size of the primary block and the maximum hierarchical level of the primary subdivision. For each prediction block thus defined, associated prediction parameters are included in the data stream. Furthermore, for each prediction block that is less than or equal to the maximum size of the residual subdivision and for each block at the root of the residual tree divided from the prediction block that exceeds the maximum size defined for the residual block, similar information, i.e., the maximum size, the maximum hierarchical level, and the bit sequence as shown in Figure 6A, are coded. For each residual block thus defined, residual data is inserted into the data stream.

[0066] The extraction unit 102 extracts each bit sequence from the data stream at the input unit 116 and notifies the obtained subdivision information to the division unit 104. Furthermore, the data stream insertion unit 18 and the extraction unit 102 may use the above-mentioned order between the prediction blocks and the residual blocks to transmit further syntax elements, such as the residual data output by the residual precoder 14 and the prediction parameters output by the prediction unit 12. Using this order has the advantage that it may be possible to select a suitable context for encoding individual syntax elements of a particular block by utilizing already coded / decoded syntax elements of neighboring blocks. Furthermore, similar to the residual precoder 14 and the prediction unit 12, the residual reconstruction unit 106 and the precoder 110 may also process the individual prediction blocks and residual blocks in the order outlined above.

[0067] Figure 7 is a flow chart showing the steps performed by the extractor 102 to extract subdivision information from the data stream 22 when encoding is performed in the manner outlined above. In a first step, the extractor 102 divides the image 24 into tree-root blocks 150. This step is shown as step 300 in Figure 7. In step 300, the extractor 102 may extract the maximum size of the prediction blocks from the data stream 22. Alternatively, in step 300, the extractor 102 may extract the maximum hierarchical level from the data stream 22. Next, in step 302, the extractor 102 decodes flags or bits from the data stream. When step 302 is performed for the first time, the extractor 102 recognizes that each flag is the first flag in the bit sequence belonging to the first block 150 at the root of the tree in the tree root block traversal order 140. Because this flag is a flag at hierarchical level 0, the extractor 102 can determine the context using the context modeling associated with that hierarchical level 0 in step 302. For each context, a respective probability estimate may be made for entropy decoding of the associated flag. The context probability estimate may be context-specifically adapted to the statistics of the symbols in each context. For example, to determine a suitable context for decoding the hierarchical level 0 flag in step 302, the extractor 102 may select one of the following contexts from the set of contexts: The context is associated with that hierarchical level 0 depending on the flags of the neighboring treeblocks at hierarchical level 0, and further depending on the information contained in the bit strings defining the quadtree subdivisions of the treeblocks neighboring the currently processed treeblock, such as the top and left neighboring treeblocks.

[0068] In the next step, i.e., step 304, the extractor 102 checks whether the most recently decoded flag indicates a segmentation. If so, the extractor 102 performs segmentation of the current block (here, a treeblock) in step 306 or indicates this segmentation to the subdivision unit 104a. Then, in step 308, it checks whether the current hierarchical level is equal to the maximum hierarchical level minus one. For example, the extractor 102 may retain the maximum hierarchical level extracted from the data stream in step 300. If the current hierarchical level is not equal to the maximum hierarchical level minus one, the extractor 102 increments the current hierarchical level by one in step 310 and returns to step 302 to decode the next flag from the data stream. Here, since the flag decoded in step 302 belongs to a different hierarchical level, according to an embodiment, the extractor 102 may select one of another set of contexts belonging to the current hierarchical level. This selection may be based on the bit sequence of the subdivisions of the decoded neighboring treeblocks, such as in FIG. 6A.

[0069] If the flag is decoded and it is determined in step 304 that this flag does not indicate partitioning of the current block, the extractor 102 proceeds to step 312 to determine whether the current hierarchical level is 0. If so, the extractor 102 proceeds to step 314 to process the next tree root block in the traversal order 140, or stops the process of extracting subdivision information if there are no tree root blocks remaining to be processed.

[0070] It should be noted that the description of Figure 7 focuses solely on decoding the flag indicating subdivision for the prediction subdivision. In practice, step 314 may also include decoding other bins or syntax elements related to the current treeblock, etc. In any case, if there is a further next tree root block, extractor 102 proceeds from step 314 to step 302 to decode the next flag from the subdivision information, i.e., the first flag in the flag sequence for the new tree root block.

[0071] If step 312 determines that the hierarchical level is not 0, the process proceeds to step 316, where it is determined whether there are any further child nodes related to the current node. That is, when the extraction unit 102 performs the determination in step 316, it has already determined in step 312 that the current hierarchical level is a level other than 0. There is a parent node that belongs to the tree root block 150 or one of the smaller blocks 152a-152d, or even to the smaller blocks 152a-152d. The node in the tree structure to which the most recently decoded flag belongs has a parent node that is also common to the other three nodes in the current tree structure. The order of traversal of such child nodes that share a common parent node is illustrated in FIG. 3A as an example for the case of hierarchical level 0, denoted by reference numeral 200. Thus, in step 316, the extraction unit 102 determines whether all four child nodes have been traversed in the process of FIG. 7. If not, i.e., if there are further child nodes in addition to the current parent node, the process of FIG. 7 proceeds to step 318. The next child node in the current hierarchical level is traversed according to the zigzag traversal order 200, so that the corresponding sub-block represents the current block for process 7. A flag is then decoded from the data stream for the current block or current node in step 302. However, if there are no more child nodes of the current parent node in step 316, the process of Figure 7 proceeds to step 320, where the current hierarchical level is decremented by one, and the process proceeds to step 312.

[0072] The extraction unit 102 and the subdivision unit 104a cooperate to obtain the encoder-selected subdivision from the data stream by performing the steps shown in Figure 7. The process of Figure 7 summarizes the prediction subdivision in the above case. Figure 8, combined with the flowchart of Figure 7, shows how the extraction unit 102 and the subdivision unit 104a cooperate to obtain the residual subdivision from the data stream.

[0073] 8 specifically illustrates the steps performed by the extraction unit 102 and the subdivision unit 104a for each prediction block resulting from the prediction subdivision. These prediction blocks are traversed in a depth-first traversal order within each tree block 150 currently being scanned to traverse leaf blocks, as shown in FIG. 3C, according to the zigzag scan order 140 within the prediction subdivision tree block 150, as described above. The leaf blocks of the partitioned primary tree block are scanned according to the depth-first traversal order. The subblocks at a particular hierarchical level to which the common current node belongs are scanned in a zigzag scan order 200, scanning the subdivisions of each of these subblocks first, followed by the next subblock in the zigzag scan order 200. In the example of FIG. 3C, the resulting traversal order among the leaf nodes of treeblock 150 is shown at 350 .

[0074] For the currently scanned prediction block, the process of Figure 8 starts at step 400. In step 400, an internal parameter representing the current size of the current block is set equal to the size of hierarchical level 0 of the residual subdivision, i.e. the maximum block size of the residual subdivision. It should be noted that the maximum size of the residual block may be smaller than the minimum block size of the prediction subdivision, or may be larger than it. In other words, in embodiments, the encoder is free to choose between the above possibilities.

[0075] In the next step, step 402, a check is made to see if the size of the prediction block of the currently scanned block is greater than an internal parameter representing the current size. If it is greater than the internal parameter, the currently scanned prediction block is greater than the maximum size of the residual block. This prediction block is a leaf block of a prediction subdivision or a tree block of a prediction subdivision and is not further partitioned. In this case, the process of FIG. 8 proceeds to step 300 of FIG. 7. That is, the currently scanned prediction block is partitioned into residual tree root blocks, and the first flag in the flag sequence of the first residual tree block in this currently scanned prediction block is decoded, e.g., in step 302.

[0076] However, if the size of the currently scanned prediction block is less than or equal to the internal parameter representing the current size, the process of Figure 8 proceeds to step 404, where the size of the prediction block is checked to determine whether it is equal to the internal parameter representing the current size. If so, the splitting step 300 may be skipped and the process proceeds directly to step 302 of Figure 7.

[0077] However, if the size of the currently scanned prediction block is smaller than the internal parameter representing the current size, the process of FIG. 8 proceeds to step 406, where the hierarchical level is incremented by one and the current size is set to the size of the new hierarchical level, such as divided by two (in both axes in the case of quadtree subdivision). The check of step 404 is then performed again. The effect of the loop formed by steps 404 and 406 is that the hierarchical level always corresponds to the size of the corresponding block to be partitioned, regardless of whether each prediction block is smaller or larger than the maximum size of the residual block. Thus, when decoding the flag in step 302, context modeling is performed depending on both the hierarchical level to which the flag refers and the size of the block. Using different contexts for flags of different hierarchical levels or block sizes has the following advantages: The probability estimates closely match the actual probability distribution for the occurrence of flag values; and, at the same time, the number of contexts managed is relatively small. As a result, the overhead of context management is reduced and the contexts closely match the actual symbol statistics.

[0078] As mentioned above, there may be multiple sample arrays, and these sample arrays may be grouped into one or more plane groups. The input signal received at input unit 32 and encoded may be, for example, an image of a video sequence or a still image. Thus, an image may be provided in the form of one or more sample arrays. In the context of encoding an image of a video sequence or a still image, a sample array may refer to three color planes, such as red, green, and blue, or luma and chroma planes, such as in a YUV or YCbCr color representation. There may also be sample arrays representing alpha, i.e., transparency, and / or depth information for 3D video material. A number of such sample arrays may be grouped together in so-called plane groups. For example, luma (Y) may be one plane group with only one sample array, and chroma, such as CbCr, in another plane group with two sample arrays. In another example, YUV may be one plane group with three planes, and depth information for 3D video material in another plane group with only one sample array. For all plane groups, one primary quadtree structure may be coded in data stream 22 to represent the division into prediction blocks, and for each prediction block, a secondary quadtree structure may be coded that represents the division into residual blocks. Thus, following the first example above, where the luma components form one plane group and the chroma components form another plane group, there would be one quadtree structure for prediction blocks of the luma planes, one quadtree structure for residual blocks of the luma planes, one quadtree structure for prediction blocks of the chroma planes, and one quadtree structure for residual blocks of the chroma planes. However, in the second example above, there would be one quadtree structure for prediction blocks of both luma and chroma (YUV), one quadtree structure for residual blocks of both luma and chroma (YUV), one quadtree structure for prediction blocks of depth information for 3D video material, and one quadtree structure for residual blocks of depth information for 3D video material.

[0079] Furthermore, the above description describes how the input signal is divided into prediction blocks using a first-order quadtree structure, and these prediction blocks are further subdivided into residual blocks using a sub-quadtree structure. In another embodiment, the subdivision does not have to end at the sub-quadtree stage, i.e. the blocks resulting from the division using the sub-quadtree structure may be further subdivided using a third quadtree structure. This division may be performed with the aim of using further coding tools that facilitate the encoding of the residual signal.

[0080] The above description has focused on the subdivision performed by subdivision unit 28 and subdivision unit 104a, respectively. As noted above, the subdivision is defined by subdivision units 28, 104a, respectively, which controls the granularity at which the above modules of encoder 10 and decoder 100 perform their processing. However, in the embodiments described below, combiner unit 30 and combiner unit 104b follow subdivision units 228, 104a, respectively. However, combiner units 30, 104b are optional and may be omitted.

[0081] However, as will be explained in more detail below, the combining unit actually provides the encoder with the opportunity to combine some of the prediction blocks or residual blocks into groups or clusters, allowing other modules, or at least some of the other modules, to handle these groups of blocks together. For example, if signaling a group of prediction blocks with common parameters for all blocks belonging to this group offers better rate / distortion potential than signaling prediction parameters for all of these prediction blocks individually, the predictor 12 may sacrifice small deviations between the prediction parameters of some of the prediction blocks, as determined by the optimization using the subdivision unit 28, and instead use common prediction parameters for all of these prediction blocks. However, the process of obtaining a prediction based on these common prediction parameters in the predictors 12 and 110 themselves may be performed for each prediction block. However, it is also possible for the predictors 12 and 110 to perform a single prediction process for the entire group of prediction blocks.

[0082] As will be explained in more detail below, it is also possible that the grouping of prediction blocks is not solely for the purpose of using identical or common prediction parameters for a group of prediction blocks, but alternatively or additionally, this allows the encoder 10 to transmit one prediction parameter for this group together with the prediction residuals of the prediction blocks belonging to this group, thereby reducing the overhead of signaling the prediction parameters for this group. In the latter case, the combined process has less impact on the data stream insertion unit 18 than the decisions made by the residual precoder 14 and the prediction unit 12, as will be explained in more detail below. Strictly speaking, it should be noted that the above aspects also apply to other subdivisions, such as the above-mentioned residual subdivision and filter subdivision.

[0083] First, the combination of sets of samples, such as predicted blocks and residual blocks, as described above is motivated in a more general sense, i.e., not limited to the multi-tree subdivision described above, although the following description will focus on the combination of blocks resulting from the multi-tree subdivision in the above embodiment.

[0084] Generally speaking, combining syntax elements associated with specific sets of samples for the purpose of transmitting associated coding parameters can reduce the rate of side information in image and video coding applications. For example, the sample array of a signal to be encoded is typically partitioned into specific sets of samples or sets of samples. A specific set of samples may represent a rectangular or square block, or another set of samples, including regions of any shape, such as quadratic blocks. In the above-described embodiment, the simply connected regions were the prediction blocks and residual blocks resulting from the subdivision of the multi-tree. The subdivision of the sample array may be fixed by the syntax, or, as described above, the subdivision information may be signaled, at least in part, within the bitstream. To keep the rate of side information for transmitting subdivision information low, the syntax typically allows only a limited number of simple partitioning choices, such as subdivision of blocks into smaller blocks. The set of samples is associated with specific coding parameters, such as the prediction information or the mode of residual coding, as described above in more detail. For each sample set, individual coding parameters may be transmitted, specifying the coding of the prediction and / or the residual. To achieve improved coding efficiency, the combining aspect described below, i.e., combining two or more sample sets into so-called groups of sample sets, offers several advantages. These advantages are further described below. For example, sample sets may be combined such that all sample sets of such a group share the same coding parameters. The same coding parameters can be transmitted together with one of the sample sets in the group. This avoids the need to transmit coding parameters separately for each sample set of the group of sample sets; instead, the coding parameters are transmitted only once for the entire group of sample sets. This may result in a reduction in the side information rate for transmitting the coding parameters, potentially improving the overall coding efficiency. As an alternative approach, additional refinements of one or more coding parameters can be transmitted for one or more sample sets of the group of sample sets.Refinement can be applied either to all of the sample sets of a group, or only to the sample set that is transmitted.

[0085] The combining aspect, described further below, also provides the encoder with more flexibility in creating the bitstream 22, since the combining approach significantly increases the number of partitioning options for the image sample array. This can improve coding efficiency, as the encoder has more options to choose from, such as minimizing a particular rate / distortion budget. There are several possibilities for how the encoder can operate. In a simple approach, the encoder first determines the optimal subdivision of the sample array. With brief reference to FIG. 1 , the subdivision unit 28 can determine the optimal subdivision in an initial stage. Each sample set is then examined to see if combining with another sample set or group of sample sets reduces the particular rate / distortion budget. Prediction parameters associated with the combined group of sample sets can then be reevaluated, such as by performing a new motion search or using prediction parameters previously determined for the common sample set. Then, candidate sample sets or groups of sample sets for combining are evaluated for the group of sample sets under consideration. In a more detailed approach, a particular rate / distortion penalty is evaluated for a group of additional candidate sample sets.

[0086] It should be noted that the combining approach described below does not change the order of processing of the sample sets, i.e. the combining concept can be implemented in such a way that there is no increase in latency, i.e. each sample set can be decoded at the same time as without using the combining approach.

[0087] For example, if the bitrate savings achieved by reducing the number of coded prediction parameters is greater than the bitrate otherwise used to code merge information for indicating the combination to the decoding side, the combination approach, as described further below, results in improved coding efficiency. It should also be noted that the described syntax extension for combination allows the encoder more freedom in choosing the partitioning of pictures or plane groups into blocks. In other words, the encoder is not restricted to first performing subdivision and then checking whether some of the resulting blocks have the same or similar sets of prediction parameters. As a simple alternative, the encoder first determines the subdivision according to the rate / distortion loss, and then checks for each block whether combining with a neighboring block or one of the associated groups of determined blocks reduces the rate / distortion loss. The prediction parameters associated with the new group of blocks can then be reevaluated, such as by performing a new motion search or using the prediction parameters determined for the current block, and the neighboring block or group of blocks can be evaluated for the new group of blocks. The combination information can be signaled on a block-by-block basis. In effect, combining can also be understood as an estimation of the prediction parameters of the current block, where the estimated prediction parameters are set equal to the prediction parameters of one of the neighboring blocks. Alternatively, for blocks within a group of blocks, the residual may be transmitted.

[0088] Thus, the basic idea underlying the combining technique described further below is to reduce the bitrate required for transmitting prediction or other coding parameters by combining adjacent blocks into groups of blocks. Each group of blocks is then associated with a unique set of coding parameters, such as prediction or residual coding parameters. The combining information, if present, is transmitted in the bitstream in addition to the subdivision information. The advantage of the combining concept is the increased coding efficiency resulting from the reduced information rate associated with the coding parameters. It should be noted that the combining process described here can also be extended to dimensions other than the spatial dimension. For example, groups of respective sets of samples or blocks present in several different video images can be combined into one group of blocks. Combining can also be applied to 4D compression and light field coding.

[0089] 1 through 8, regardless of the specific manner in which the subdivision units 28 and 104a subdivide the image, the subdivision followed by the merging process is advantageous. Specifically, the latter can also subdivide images in a manner similar to H.264, for example. That is, each image can be subdivided into a regular array of rectangular or square macroblocks of a predetermined size, such as 16x16 luma samples, or a size signaled in the data stream. Each macroblock has associated coding parameters, including, among other things, partitioning parameters and corresponding prediction parameters in the data stream. The partitioning parameters define the partitioning of each macroblock into a regular subgrid of 1, 2, 4, etc., which corresponds to the prediction granularity. The partitioning for the residual and the corresponding granularity of the transform of the residual are also defined.

[0090] In either case, combining provides the advantages outlined above, such as reducing the bit rate of associated information in image and video coding. A particular set of samples, representing a square or rectangular block, or an area of ​​any shape, or any other set of samples, such as any simply connected region or sample, is typically associated with a particular set of coding parameters. For each set of samples, coding parameters are included in the bitstream, such as prediction parameters that specify how to predict the corresponding set of samples using previously coded samples. The partitioning of the image sample array into sets of samples may be fixed by syntax or conveyed by corresponding subdivision information in the bitstream. The coding parameters for the sets of samples may be transmitted in a predetermined order specified by the syntax. The combining unit 30 can communicate to a group of sets of samples about a common set of samples or a current block, such as a predicted block or residual block, combined with one or more other sets of samples according to the combining function. Therefore, the coding parameters for a group of sets of samples need only be transmitted once. In certain embodiments, if the current sample set is to be combined with an existing group of sample sets or sample sets for which coding parameters have already been transmitted, the coding parameters of the current sample set are not transmitted. Instead, the coding parameters of the current sample set are set equal to the coding parameters of the sample set or group of sample sets to which the current sample set is combined. As an alternative approach, additional refinements for one or more coding parameters can be transmitted for the current sample set. The refinements can apply either to all sample sets of the group or only to the sample set for which the refinement is transmitted.

[0091] In an embodiment, the set of all coded / decoded sample sets for each sample set, such as the above-mentioned prediction block, the above-mentioned residual block, or the above-mentioned leaf blocks of a multi-tree subdivision, is referred to as the "set of causal sample sets." See, for example, FIG. 3C. All blocks shown in this figure are the result of a specific subdivision, such as a prediction subdivision or a residual subdivision, or a multi-tree subdivision, and the coding / decoding order defined between these blocks is defined by arrow 350. If a specific one of these blocks is considered as the current sample set or the current simply connected region, then the set of causal sample sets consists of all blocks that precede the current block in the order 350. However, other subdivisions that do not use multi-tree subdivision are also possible, as long as the combination principle described below is followed.

[0092] The set of samples that can be used to combine with the current sample set is referred to below as the "set of candidate sample sets" and is always a subset of the "set of causal sample sets." The method for forming the subset can be signaled to the decoder or specified in the data stream or bitstream from the encoder to the decoder. When a particular current sample set is encoded / decoded and its set of candidate sample sets is not empty, the encoder signals in the data stream and the decoder derives from the data stream whether a common sample set is to be combined with one of the candidate sample sets. If so, they are signaled together. Otherwise, the set of candidate sample sets is empty and no combining is available for this block.

[0093] There are various ways to determine a subset of the set of causal sample sets that represents the set of candidate sample sets. For example, the determination of the candidate sample set may be based on a uniquely geometrically defined sample in the current sample set, such as the sample in the top-left image of a rectangular or square block. Starting from this uniquely geometrically defined sample, a non-zero number of specific samples is determined, which represents direct spatial neighbors of this uniquely geometrically defined sample. For example, this non-zero number of specific samples includes the top neighbor and the left neighbor of the uniquely geometrically defined sample in the current sample set. The non-zero number of neighbors is at most two, is one if one of the top or left neighbors is unavailable or outside the image, and is zero if neither neighbor exists.

[0094] A set of candidate sample sets is then determined to encompass sample sets that contain at least one non-zero number of the neighboring samples. See, for example, FIG. 9A. Let the current sample set under consideration for merging be block X, and its unique geometrically defined sample be the top-left sample, illustrated at 400. The neighboring samples above and to the left of sample 400 are shown at 402 and 404. The set of causal sample sets, or causal blocks, is highlighted by shading. Of these blocks, blocks A and B contain one of the neighboring samples 402 and 404, and therefore these blocks form a set of candidate block sets, or a set of candidate sample sets.

[0095] In another embodiment, the set of candidate sample sets determined for combining purposes may additionally or exclusively include sample sets containing a specific non-zero number of samples. The specific non-zero number of samples may be one or two, and may be in the same spatial location but in a different image, such as a previously coded / decoded image. For example, in addition to blocks A and B in FIG. 9A, a block from a previously coded image containing a sample in the same location as sample 400 may be used. Note that only the top neighbor 404 or only the left neighbor 402 may be used to define the above non-zero number of neighboring samples. In general, the set of candidate sample sets may be derived from previously processed data in the current image or another image. This derivation may include spatial information, such as transform coefficients associated with a specific direction or the tilt of the current image, and may also include temporal information, such as a representation of neighboring motion. The set of candidate sample sets may be derived from such data available to the receiver / decoder, and, if present, from other data and associated information in the data stream.

[0096] It should be noted that both the encoder-side combiner 30 and the decoder-side combiner 104b derive candidate sample sets in parallel. As noted above, they may independently determine the set of candidate sample sets based on a predetermined method communicated to both. Alternatively, the encoder may signal hints in the bitstream, which would position combiner 104b to derive these candidate sample sets in the same way that encoder-side combiner 30 determined the set of candidate sample sets.

[0097] As will be described in detail below, the combiner 30 and the data stream inserter 18 cooperate to transmit one or more syntax elements for each sample set. These syntax elements specify whether the sample set is to be combined with another sample set that is part of the group of previously combined sample sets, and the set of candidate sample sets to be used for the combination. The extractor 102 then extracts these syntax elements and notifies the combiner 104b accordingly. In particular, in the specific embodiment described below, one or two syntax elements are transmitted to specify the combining information for a particular sample set. The first syntax element specifies whether the current sample set is to be combined with another sample set. The second syntax element is transmitted only if the first syntax element specifies that the current sample set is to be combined with another sample set, and specifies the set of candidate sample sets to be used for the combination. If the derived set of candidate sample sets is empty, the first syntax element does not need to be transmitted. In other words, the first syntax element may be transmitted only if the derived set of candidate sample sets is not empty. The second syntax element may be transmitted only if the derived set of candidate sample sets contains multiple sample sets, since if the set of candidate sample sets contains only one sample set, no further selection is possible. Furthermore, if the set of candidate sample sets contains multiple sample sets, but all sample sets in the candidate sample sets are associated with the same coding parameter, the second syntax element may not be transmitted. In other words, the second syntax element may be transmitted only if at least two sample sets in the derived candidate sample sets are associated with different coding parameters.

[0098] In the bitstream, the combining information for a sample set may be coded before the prediction parameters or other specific coding parameters associated with that sample set, which may be transmitted only if the combining information signals that the current sample set will not be combined with other sample sets.

[0099] The joint information for a particular sample set, i.e., a block, may be coded after the proper subset of prediction parameters, or more generally, the coding parameters associated with each sample set, has been transmitted. The subset of prediction / coding parameters may consist of one or more reference image indices, or one or more components of a vector of motion parameters, or a reference indices and one or more components of a vector of motion parameters, etc. The transmitted subset of prediction or coding parameters can be used to derive a set of candidate sample sets from the larger set of provisional candidate sample sets derived above. For example, a difference magnitude or distance between the coded prediction and coding parameters of the current sample set and the corresponding prediction and coding parameters of a preliminary set of candidate sample sets can be calculated based on a predetermined distance criterion. Only sample sets whose calculated difference magnitude or distance is equal to or less than a predetermined or derived threshold are then included in the final, i.e., reduced, set of candidate sample sets. See, for example, FIG. 9A. Let the current sample set be block X. Assume that a subset of coding parameters associated with this block has already been inserted into the data stream 22. For example, assume that block X is a predictive block, and the proper subset of coding parameters is a subset of prediction parameters for this block X, such as a subset of a set including image reference indices, and motion mapping information, such as motion vectors. If block X is a residual block, the subset of coding parameters is a subset of residual information, such as a map representing the locations of transform coefficients and significant transform coefficients within block X. Based on this information, both the data stream insertion unit 18 and the extraction unit 102 can use this information to determine subsets of blocks A and B, which in this particular embodiment form the preliminary set of candidate sample sets mentioned above. In particular, since blocks A and B belong to the set of causal sample sets, their coding parameters are available to both the encoder and the decoder when the coding parameters of block X are currently being coded / decoded.Therefore, using the comparison of the magnitude of the difference, any number of blocks of the preliminary sets of candidate sample sets A and B may be used to exclude them. The resulting reduced set of candidate sample sets may then be used as described above, i.e., to determine whether a merge indicator representing the join is transmitted in or extracted from the data stream, and whether a second syntax element needs to be transmitted in or extracted from the data stream, depending on the number of sample sets in the reduced set of candidate sample sets. The second syntax element indicates which sample sets in the reduced set of candidate sample sets are to be the blocks to join. That is, the determination of the combination for a given singly connected region or the transmission of the syntax element of the respective combination may be based on the number of singly connected regions which are located in a predetermined relative position to the given singly connected region and which at the same time have associated coding parameters which satisfy a predetermined relationship to the first subset of coding parameters for the given singly connected region, and the adoption or prediction of the extraction of the prediction residual may be performed on the second subset of coding parameters for the given singly connected region, that is, only one of the number of singly connected regions which are located in a predetermined relative position to the given singly connected region and which at the same time have associated coding parameters which satisfy a predetermined relationship to the first subset of coding parameters for the given singly connected region or only a subset of the identified coding parameters may be adopted from the second subset of given singly connected regions and used for the prediction of the second subset of given singly connected regions, respectively.

[0100] The threshold against which the distance is compared may be fixed and communicated to both the encoder and decoder, or may be derived based on a calculated distance, such as the median difference value or other central tendency. In this case, it is inevitable that the reduced set of candidate sample sets will be a proper subset of the preliminary set of candidate sample sets. Alternatively, only those sample sets are selected from the preliminary set of candidate sample sets that have the smallest distance based on the distance magnitude. Alternatively, a single set of samples is selected from the preliminary set of candidate sample sets using the distance magnitude. In the latter case, the merge information only needs to specify whether the current sample set is merged with a single candidate sample set.

[0101] Thus, a set of candidate blocks is formed or derived as described below with reference to Figure 9A. Starting with the top-left sample location 400 of current block X in Figure 9A, the locations of its left neighboring sample 402 and its top neighboring sample 404 are derived at the encoder and decoder sides. Thus, the set of candidate blocks can have at most two members, namely, blocks that contain one of the two sample locations of the shaded set of causal blocks in Figure 9A, which in this case are blocks B and A. Thus, the set of candidate blocks can only have as its members the two direct neighbors of the top-left sample location of the current block. In another embodiment, the set of candidate blocks is formed by all blocks coded before the current block and that contain one or more samples representing direct spatial neighbors of any sample of the current block. The direct spatial neighbors may be limited to the immediate neighbors to the left, and / or the immediate neighbors above, and / or the immediate neighbors to the right, and / or the immediate neighbors below, of any sample of the current block. For example, see Figure 9B, which shows another block subdivision. In this case, the candidate block includes four blocks, namely, blocks A, B, C, and D.

[0102] The set of candidate blocks may additionally or exclusively include blocks containing one or more samples that are in the same position as any sample in the current block but are contained in a different image, i.e., an already coded / decoded image.

[0103] Alternatively, the set of candidate blocks may represent a subset of the set of blocks described above, determined by spatial or temporal neighbor relationships. The subset of candidate blocks may be fixed, signaled, or derived. The derivation of the subset of candidate blocks may take into account decisions about other blocks in the image or other images. For example, blocks associated with the same coding parameters or coding parameters that are much more similar than other candidate blocks may not be included in the set of candidate blocks.

[0104] The following description of the embodiment applies when only two blocks, at most, containing samples adjacent to the left and above the top-left sample of the current block, are considered as candidates.

[0105] If the set of candidate blocks is not empty, a flag called merge_flag is signaled which specifies whether the current block is combined with one of the candidate blocks. If merge_flag is equal to 0 (indicating "false"), this block is not combined with one of the candidate blocks and all coding parameters are transmitted as usual. If merge_flag is equal to 1 (indicating "true"), the following applies: If the set of candidate blocks contains only one block, this candidate block is used for combining. Otherwise, the set of candidate blocks contains two blocks. If the prediction parameters of these two blocks are identical, these prediction parameters are used for the current block. Otherwise (if the two blocks have different prediction parameters), a flag called merge_left_flag is signaled. If merge_left_flag is 1 (indicating "true"), the block containing the adjacent sample position to the left of the top-left sample position of the current block is selected from the set of candidate blocks. If merge_left_flag is 0 (indicating "false"), another (i.e., top-neighboring) block is selected from the set of candidate blocks, and the prediction parameters of the selected block are used for the current block.

[0106] To summarise the several embodiments of combining described above, reference is made to FIG. 10 which illustrates the steps performed by the extractor 102 to extract combining information from the data stream 22 received at the input 116 .

[0107] The process begins with identifying candidate blocks or sample sets for the current sample set or block at 450. Since the coding parameters for the blocks are transmitted in a particular one-dimensional order in the data stream 22, Figure 10 describes the process of obtaining joint information for the currently scanned sample set or block.

[0108] As mentioned above, the identification step 450 may involve identification from a set of decoded blocks, i.e., causal blocks, based on a neighboring relationship perspective. For example, neighboring blocks containing certain neighboring samples that are spatially or temporally adjacent to one or more geometrically predetermined samples of the current block X may be designated candidates. Furthermore, the identification step may include two stages: a first stage including the above-mentioned identification to obtain a preliminary set of candidate blocks, i.e., identification based on neighboring relationships, and a second stage in which only blocks decoded from the data stream before step 450 are designated candidates whose transmitted coding parameters satisfy a certain relationship with a proper subset of the coding parameters of the current block X.

[0109] Next, the process proceeds to step 452, where it is determined whether the number of candidate blocks is greater than zero. If so, in step 454, merge_flag is extracted from the data stream. The extraction step 454 may include entropy decoding. The context for the entropy decoding of merge_flag in step 454 may be determined based on, for example, the set of candidate blocks or syntax elements belonging to a preliminary set of candidate blocks, etc. In that case, the dependency on the syntax elements may be limited to information on whether a block belonging to the set was a candidate for merging. An estimate of the probability of the selected context may be adapted.

[0110] On the other hand, if step 452 determines that the number of candidate blocks is zero, the process of Figure 10 proceeds to step 456, where the coding parameters of the current block are extracted from the bitstream. Also, in the particular case of the two stages described above, the remaining coding parameters are extracted, after which extraction unit 102 proceeds to process the next block in the order of block scanning, such as order 350 shown in Figure 3C.

[0111] Returning to step 454, after the extraction of step 454, the process proceeds to step 458, where it is determined whether the extracted merge_flag indicates whether merging of the current block will occur. If merging will not occur, the process proceeds to step 456 described above. If merging will occur, the process proceeds to step 460, where it is determined whether the number of candidate blocks is one. If the number of candidate blocks is one, there is no need to indicate or transmit a specific candidate block among the candidate blocks, and the process of FIG. 10 proceeds to step 462, where the merge partner of the current block is set to that single candidate block. Then, in step 464, the coding parameters of the merged partner block are used to adapt or predict the coding parameters or remaining coding parameters of the current block. In the case of adaptation, the missing coding parameters of the current block are simply copied from the merge partner block. In the other case, i.e., in the case of prediction, step 464 may further involve extracting residual data from the data stream. This residual data relates to the residual of the prediction of the missing coding parameters of the current block. This residual data may then be combined with predictions of these missing coding parameters obtained from the blocks with which it is combined.

[0112] On the other hand, if it is determined in step 460 that the number of candidate blocks is greater than one, the process of Figure 10 proceeds to step 466, where it is determined whether the coding parameters or corresponding parts of the coding parameters, i.e., parts relating to parts not yet transmitted in the data stream for the current block, are identical to each other. If so, in step 468, these common coding parameters are set as the reference for combination. Alternatively, the candidate block is set as the partner for combination. Each corresponding coding parameter is then used for adaptation or prediction in step 464.

[0113] It should be noted that the combination partner may itself be the block for which the combination is signaled, in which case the adopted or predicted coding parameters of the combination partner are used in step 464.

[0114] On the other hand, if not, i.e., if the coding parameters are not identical, the process in Fig. 10 proceeds to step 470, where the syntax element merge_left_flag is further extracted from the data stream. A separate set of contexts may be used for entropy decoding of this flag. Also, the set of contexts used for entropy decoding of merge_left_flag may consist of only one context. After step 470, the candidate block suggested by merge_left_flag is set as the combining partner in step 472 and used for adaptation or prediction in step 464. After step 464, the extraction unit 102 proceeds to process the next block in block order.

[0115] Of course, there are many alternatives. For example, instead of the separate syntax elements of merge_flag and merge_left_flag described above, a combined syntax element may be transmitted in the data stream to signal the combining operation. Also, the above merge_left_flag may be transmitted in the data stream regardless of whether the two candidate blocks have the same prediction parameters, thereby reducing the computational overhead for performing the processing of FIG. 10.

[0116] As already shown in FIG. 9B and elsewhere, the set of candidate blocks may include three or more blocks. Furthermore, combining information, i.e., information indicating whether a block is combined (and, if combined, combined with a candidate block), may be conveyed by one or more syntax elements. A single syntax element, such as the above-mentioned merge_flag, may specify whether a block is combined with one of the candidate blocks. The flag may be transmitted only if the set of candidate blocks is not empty. A second syntax element, such as the above-mentioned merge_left_flag, may signal which candidate block is used for combining. Typically, a selection from two or more candidate blocks is indicated. The second syntax element may be conveyed only if the first syntax element signals that the current block is combined with one of the candidate blocks. Furthermore, the second syntax element may be transmitted only if the set of candidate blocks includes multiple candidate blocks and / or if one of the candidate blocks has different prediction parameters from the other candidate blocks. The syntax depends on the number of candidate blocks and / or the extent to which the prediction parameters associated with the candidate blocks differ.

[0117] The syntax signaling which candidate blocks are used may be configured simultaneously and / or in parallel at the encoder and decoder sides. For example, if there are three choices for the candidate blocks identified in step 450, the syntax is selected so that only these three choices are available and considered for entropy coding, e.g., in step 470. In other words, syntax elements are selected so that the symbol alphabet has as many elements as there are candidate block choices. All other choices may be considered zero possibilities, and entropy coding / decoding may be coordinated simultaneously at the encoder and decoder.

[0118] Furthermore, as noted in step 464, the prediction parameters estimated as a result of the combining process may represent the complete set of prediction parameters associated with the current block, or may represent a subset of these prediction parameters, such as prediction parameters for one hypothesis of a multi-hypothesis block.

[0119] As mentioned above, syntax elements related to merge information can be entropy coded using context modeling. The syntax elements may consist of the above-mentioned merge_flag and merge_left_flag (or similar syntax elements). In a specific example, one of three context models or contexts can be used to encode / decode merge_flag, such as in step 454. The index of the context model to be used, merge_flag_ctx, may be derived as follows: If the set of candidate blocks includes two elements, the value of merge_flag_ctx is equal to the sum of the merge_flag values ​​of the two candidate blocks. On the other hand, if the set of candidate blocks includes one element, the value of merge_flag_ctx may be equal to twice the value of the merge_flag of this one candidate block. Since each merge_flag of adjacent candidate blocks can be 1 or 0, three contexts are available for merge_flag. The merge_left_flag may be coded using only a single probability model.

[0120] In other embodiments, a different context model may be used. For example, non-binary syntax elements may be mapped to a sequence of binary symbols called bins. The context model for some syntax elements or bins of syntax elements that define the binding information may be derived based on the transmitted syntax elements of neighboring blocks or the number of candidate blocks, while other syntax elements or bins of syntax elements may be coded with a fixed context model.

[0121] With respect to the above description of combining blocks, the set of candidate blocks may be derived in a manner similar to any of the above-described embodiments, with the following modifications: Restrict candidate blocks to blocks that use motion compensated prediction or inter-prediction, respectively. Only such blocks can be members of the set of candidate blocks. Merging information propagation and context modeling can be performed as described above.

[0122] Returning now to the combination of the above-described multi-tree subdivision embodiment and the currently described combining aspect. When a quadtree-based subdivision structure is used to divide an image into square blocks of variable size, syntax elements specifying combining, e.g., merge_flag and merge_left_flag, are alternated with prediction parameters transmitted for each leaf node of the quadtree structure. For example, consider again FIG. 9A. FIG. 9A shows an example of a quadtree-based image subdivision into variable-sized prediction blocks. The top two blocks with the largest size are called treeblocks, i.e., prediction blocks of the largest possible size. The other blocks in this diagram are obtained by subdividing the corresponding treeblocks. The current block is indicated by an "X." All shaded blocks are encoded / decoded before the current block, thus forming a set of causal blocks. For one embodiment, as detailed in the description of the derivation of the set of candidate blocks, only blocks containing samples that are directly adjacent (i.e., above or to the left) to the location of the top-left sample of the current block can be elements of the set of candidate blocks. Thus, the current block can be combined with either block "A" or block "B". If merge_flag is equal to 0 (indicating "false"), the current block "X" is not combined with either of the two blocks. If blocks "A" and "B" have identical prediction parameters, combining with either of the two blocks will produce the same result, so no distinction needs to be made. Therefore, in this case, merge_left_flag is not transmitted. Otherwise, if blocks "A" and "B" have different prediction parameters, then if merge_left_flag is 1 (indicating "true"), blocks "X" and "B" are combined, while if merge_left_flag is 0 (indicating "false"), blocks "X" and "A" are combined. In another preferred embodiment, additional adjacent (already transmitted) blocks represent candidates for combining.

[0123] Figure 9B shows another example, where the current block "X" and its left neighbor "B" are treeblocks, i.e., they are of the maximum allowed block size. The size of the top neighbor "A" is one-quarter of the size of the treeblock. Blocks that are members of the set of causal blocks are shaded. In one preferred embodiment, the current block "X" can only be combined with two blocks "A" or "B", and not with any other top neighbors. In another preferred embodiment, additional neighboring (already transmitted) blocks represent candidates for combining.

[0124] Before proceeding to the description of the aspects of handling different sample arrangements of an image according to the present application, the above description of the aspects of performing multi-tree subdivision and combining while communicating information on the other hand makes it clear that these aspects provide advantages that can be utilized independently of each other. That is, as already explained above, the combination of multi-tree subdivision and combining provides specific advantages. However, advantages can also be obtained in other ways. For example, the combining function is embodied in the subdivision performed by the subdivision unit 30, 104a in response to the subdivision of macroblocks that regularly partitions these macroblocks into smaller sections, rather than based on quadtree or multi-tree subdivision. On the other hand, the combination of multi-tree subdivision that transmits an indication of the maximum size of treeblocks in the bitstream and the use of multi-tree subdivision that transmits the corresponding coding parameters of the blocks using a depth-first traversal order provides advantages, regardless of whether the combining functions are utilized in parallel. In general, the advantages of combining can be understood intuitively by considering the following: Coding efficiency is improved if the syntax for coding sample arrays is extended in a way that not only allows for subdivision of a block, but also allows for the combination of two or more blocks obtained after subdivision. This results in a group of blocks that are coded with the same prediction parameters. The prediction parameters for such a group of blocks need only be coded once. Also, regarding the combination of sample sets, it is reiterated that the sample sets considered are rectangular or square blocks, and the combined sample sets are collections of rectangular or square blocks. The sample sets considered can also be image regions of any shape, and the combined sample sets are collections of image regions of any shape.

[0125] The following discussion focuses on the handling of different sample arrays per image when there are multiple sample arrays per image. The aspects outlined in the following subsections provide benefits regardless of the type of subdivision used, i.e., whether subdivision is based on multi-tree subdivision or not, and whether merging is used or not. Before proceeding to a description of specific embodiments for handling different sample arrays per image, a short introduction to handling different sample arrays per image is provided before the main subject of these embodiments is described.

[0126] The following discussion focuses on coding parameters between blocks of different sample arrays of an image in image or video coding. In particular, the discussion focuses on adaptively predicting coding parameters between different sample arrays of an image in environments, including, for example, the encoders and decoders of FIGS. 1 and 2, respectively, as well as other image or video coding environments. As discussed above, the sample arrays can represent sample arrays associated with various color components or sample arrays that associate an image with additional information, such as transparency data or depth maps. Sample arrays associated with color components of an image are also referred to as color planes. The techniques described below, also referred to as inter-plane adoption / prediction, can be used in block-based image and video encoders and decoders, where the blocks of the image's sample arrays are processed in any order.

[0127] Image and video coders are typically designed to encode color images (either still images or images from a video sequence). A color image consists of multiple color planes, each representing a different color component. Color images are often encoded as a set of sample arrays consisting of a luma plane and two saturation planes, which specify the chrominance components. In some applications, it is common for the set of encoded sample arrays to consist of three color planes, representing the red, green, and blue primary color samples. Furthermore, color images may consist of four or more color planes for enhanced color representation. Additionally, images can have associated auxiliary sample arrays that specify additional information about the image. For example, these auxiliary sample arrays might specify the transparency of the associated color sample array (suitable for specific display purposes) or a depth map (suitable for multi-view rendering, such as 3D displays).

[0128] In traditional image and video coding standards (e.g., H.264), color planes are typically coded jointly. In this case, certain coding parameters, such as prediction modes, reference indices, and motion vectors for a macroblock and its sub-macroblocks, are used for all color components of a block. The luma plane can be considered the primary color plane, where certain coding parameters are specified in the bitstream, and the chroma plane can be considered a secondary plane, where the corresponding coding parameters are inferred from the primary luma plane. Each luma block is associated with two chroma blocks that represent the same region in the image. Depending on the chroma sampling format used, the chroma sample array for a block may be smaller than the luma sample array. For each macroblock consisting of one luma component and two chroma components, the same partitioning is performed, and the partitioning into smaller blocks (if the macroblock is subdivided) is performed. For each block (which may be the macroblock itself or a subblock of a macroblock), consisting of one block of luma samples and two blocks of chroma samples, the same set of prediction parameters, such as reference indices and motion parameters, and possibly an intra prediction mode, are used. Certain profiles of conventional video coding standards (e.g., the 4:4:4 profile of H.264) also allow for independent coding of different color planes of an image. In this configuration, macroblock partitioning, prediction modes, reference indices, and motion parameters can be selected separately for the color components of a macroblock or subblock. In conventional coding standards, either all color planes are coded using the same specific set of coding parameters (e.g., subdivision information and prediction parameters), or all color planes are coded completely independently of each other.

[0129] When color planes are coded jointly, one set of subdivision and prediction parameters must be used for all color components of a block. This ensures small side information, but may result in lower coding efficiency compared to when coded independently. This is because using different block decomposition and prediction parameters for different color components may result in smaller rate / distortion losses. For example, using different motion vectors or reference frames for the chroma components may significantly reduce the energy of the residual signal for the chroma components, increasing overall coding efficiency. When color planes are coded independently, coding parameters such as block partitioning, reference indices, and motion parameters can be selected separately for each color component to optimize coding efficiency for each color component. However, redundancy between color components cannot be exploited. Multiple transmission of certain coding parameters increases the side information rate (compared to when coded jointly), and this increase in side information rate may negatively impact overall coding efficiency. Furthermore, support for auxiliary sample sequences in modern video coding standards (such as H.264) is limited to encoding the auxiliary sample sequences using a unique set of coding parameters.

[0130] Thus, in all of the embodiments described so far, image planes can be treated as described above. However, as also described above, the overall coding efficiency of coding multiple sample arrays (which may be associated with different color planes and / or auxiliary sample arrays) may be improved if block-level decisions are possible, e.g., whether all sample arrays of a block are coded with the same coding parameters or whether different coding parameters are used. The basic idea behind the inter prediction described below is to enable such block-level, context-sensitive decisions. An encoder can select, based on, for example, rate / distortion criteria, whether all or some of the sample arrays of a particular block are coded using the same coding parameters or whether different coding parameters are used for different sample arrays. This selection can also be achieved by signaling, for a particular block of a sample array, whether a particular coding parameter is inferred from a previously coded block in the same location in a different sample array. Different sample arrays for an image can also be arranged in groups, which are called sample array groups or plane groups. Each plane group can contain one or more sample arrays of the image. Blocks of sample arrays within a plane group share the same selected coding parameters, such as subdivision information, prediction mode, and residual coding mode, while other coding parameters, such as transform coefficient levels, are transmitted separately for each sample array within a plane group. One plane group is coded as a primary plane group, i.e., no coding parameters are estimated or predicted from other plane groups. For each block of a secondary plane group, it is possible to select whether a new set of selected coding parameters is transmitted, or whether the selected coding parameters are estimated or predicted from the primary plane group or another secondary plane group, depending on the situation. The decision of whether the selected coding parameters for a particular block are estimated or predicted is included in the bitstream.Inter-prediction allows for more flexibility in the trade-off between the rate of side information and the quality of prediction compared to state-of-the-art coding of images consisting of multiple sample arrays. The advantage is improved coding efficiency compared to traditional coding of images consisting of multiple sample arrays.

[0131] Intra-plane adaptation / prediction may extend image or video coders such as those described above to allow for tailored selection of whether a set of selected coding parameters for a block of a color sample array, or an auxiliary sample array, or a set of color sample arrays and / or auxiliary sample arrays, is estimated or predicted from previously coded co-located blocks of other sample arrays in the same image, or whether the set of selected coding parameters for a block is coded independently without reference to co-located blocks of other sample arrays in the same image. The decision of whether a set of selected coding parameters for a block of a sample array, or for multiple sample array blocks, is estimated or predicted may be included in the bitstream. Different sample arrays associated with an image need not be the same size.

[0132] As mentioned above, the sample arrays associated with an image (sample arrays can represent color component and / or auxiliary sample arrays) may be arranged into two or more so-called plane groups, with each plane group consisting of one or more sample arrays. The sample arrays included in a particular plane group do not have to be the same size. This arrangement into plane groups also includes the case where each sample array is coded separately.

[0133] Specifically, in embodiments, for each block in a plane group, coding parameters specifying how the block is predicted may be estimated or predicted from previously coded blocks in different plane groups for the same image, or these coding parameters may be coded separately for the block, depending on the circumstances. The coding parameters specifying how the block is predicted may include one or more of the following: a block prediction mode specifying what type of prediction is used for the block (intra prediction, inter prediction using a single motion vector and reference image, inter prediction using two motion vectors and reference images, inter prediction using a higher-order, i.e., non-translational, motion model and a single reference image, inter prediction using multiple motion models and reference images); an intra prediction mode specifying how to generate an intra prediction signal; an identifier specifying how many prediction signals are combined to generate a final prediction signal for the block; a reference index specifying the reference image used for motion-compensated prediction; motion parameters (such as displacement vectors or affine motion parameters) specifying how to generate a prediction signal using the reference image; and an identifier specifying how to filter the reference image to generate the motion-compensated prediction signal. Generally, a block may be associated with only a subset of the above coding parameters. For example, if the block prediction mode specifies that intra prediction is to be performed on the block, the coding parameters of the block may additionally include the intra prediction mode, but coding parameters such as reference indices and motion parameters that specify how to generate the inter prediction signal are not specified. Alternatively, if the block prediction mode specifies inter prediction, the associated coding parameters may additionally include reference indices and motion parameters, but the intra prediction mode is not specified.

[0134] One of two or more plane groups may be coded or indicated in the bitstream as a primary plane group. For all blocks in this primary plane group, coding parameters specifying how to generate a prediction signal are transmitted without reference to other plane groups in the same image. The remaining plane groups are coded as secondary plane groups. For each block in the secondary plane group, one or more syntax elements are transmitted that signal whether coding parameters specifying how to predict the block are estimated or predicted from co-located blocks in other plane groups, or whether a new set of these coding parameters is transmitted for the block. One of the one or more syntax elements may be referred to as an inter-prediction flag or inter-prediction parameters. If a syntax element signals that no estimation or prediction of the corresponding coding parameters is performed, a new set of corresponding coding parameters for the block is transmitted in the bitstream. If a syntax element signals that estimation or prediction of the corresponding coding parameters is performed, a co-located block in a so-called reference plane group is identified. The assignment of reference plane groups to blocks can be configured in several ways. In one embodiment, a specific reference plane group is assigned to each secondary plane group, and this assignment can be fixed or signaled in a high-level syntactic construct such as a set of parameters, access unit header, picture header, slice header, etc.

[0135] In a second embodiment, the reference plane group assignment is coded in the bitstream and signaled by one or more syntax elements that are coded for the block and specify whether the selected coding parameters are estimated, predicted, or coded separately.

[0136] To explain the above possibilities in relation to inter-prediction and the detailed embodiments below, reference is made to FIG. 11. FIG. 11 illustrates an image 500 composed of three sample arrays 502, 504, and 506. For clarity, only the lower portions of the sample arrays 502-506 are shown in FIG. 11. The sample arrays are shown to spatially correspond to one another. The sample arrays 502-506 overlap one another along a direction 508. Furthermore, when the samples of the sample arrays 502-506 are projected along the direction 508, the samples of all of these sample arrays 502-506 are exactly aligned with one another in spatial position. In other words, the planes 502 and 506 extend horizontally and vertically to accommodate each other's spatial resolution and correspond to one another.

[0137] In this embodiment, all sample arrays of an image belong to the same portion of the spatial scene, and the vertical and horizontal resolutions of sample arrays 502-506 may differ. For purposes of illustration, sample arrays 502 and 504 belong to one plane group 510, while sample array 506 belongs to another plane group 512. FIG. 11 also illustrates a case in which the spatial resolution along the horizontal axis of sample array 504 is twice the horizontal resolution of sample array 502. Furthermore, sample array 504 is considered to be a primary array relative to sample array 502. Sample array 502 is considered to be a subordinate array relative to primary array 504. As previously mentioned, in this case, the subdivision of sample array 504 into blocks determined by subdivision unit 30 of FIG. 1 is adopted by subordinate array 502. 11, the vertical resolution of sample array 502 is half the vertical resolution of linear array 504, so each block is divided into two equal horizontal blocks, which, when measured in units of sample positions within sample array 502, again become quadratic blocks.

[0138] 11, the subdivision selected for the sample array 506 differs from the subdivision of the other plane groups 510. As previously mentioned, the subdivision unit 30 may select the subdivision of the pixel array 506 separately, independently of the subdivision for the plane group 510. Of course, the resolution of the sample array 506 may also differ from the resolution of the planes 502, 504 of the plane group 510.

[0139] Here, when encoding individual sample arrays 502-506, encoder 10 may start by encoding primary array 504 of plane group 510, e.g., in the manner described above. The blocks shown in FIG. 11 may be, e.g., prediction blocks as described above. Alternatively, the blocks may be residual blocks or other blocks that define the granularity for the definition of particular coding parameters. Inter-prediction is not limited to the quadtree or multitree subdivision illustrated in FIG. 11.

[0140] After transmitting the syntax elements for the primary array 504, the encoder 10 may decide to declare the primary array 504 as the reference plane for the lower plane 502. The encoder 10 and the extractor 30 may each signal this decision via the bitstream 22. Alternatively, the relevance of the sample array 504 may be made clear by the sample array 504 being the primary array of the plane group 510, and this information may be part of the bitstream 22. In either case, for each block in the sample array 502, the inserter 18, or another module of the encoder 10 operating in conjunction with the inserter 18, may decide to suppress transmission of the coding parameters of this block in the bitstream and instead use the coding parameters of the co-located block in the primary array 504 in the bitstream for that block, or signal that the coding parameters of the co-located block in the primary array 504 should be used as predictions for the coding parameters of the current block of the sample array 502, and transmit only the residual data for the current block of the sample array 502 in the bitstream. If the decision is not made, the coding parameters are transmitted in the data stream as usual. The decision is signaled in the data stream 22 for each block. At the decoder side, the extractor 102 uses the inter-prediction information for each block to obtain the coding parameters of the respective block of the sample array 502 accordingly, i.e., by estimating the coding parameters of the co-located block of the primary array 504. Alternatively, if the inter-plane adoption / prediction information indicates inter-plane adoption / prediction, this is done by extracting residual data for that block from the data stream and combining this residual data with the prediction obtained from the coding parameters of the co-located block of the primary array 504. Alternatively, this is done by extracting the coding parameters of the current block of the sample array 502 as usual, regardless of the primary array 504.

[0141] As mentioned above, the location of the reference plane is not limited to the same plane group as the block currently being inter-predicted. Therefore, as mentioned above, the plane group 510 may be the primary plane group or the reference plane group for the secondary plane group 512. In this case, the bitstream may include, for each block in the sample array 506, a syntax element indicating whether the coding parameters of the co-located macroblock in either the primary plane group or the reference plane group 502, 504 are to be adopted / predicted as described above. In the latter case, the coding parameters of the current block in the sample array 506 are transmitted as usual.

[0142] The subdivision and / or prediction parameters for planes within a plane group may be identical, i.e., they are only coded once for the plane group (all secondary planes of a plane group derive their subdivision information and / or prediction parameters from the primary planes in the same plane group), and contextual prediction or estimation of the subdivision information and / or prediction parameters occurs between plane groups. The reference plane group may be a primary plane group or a secondary plane group.

[0143] Except for the further partitioning described above to divide adopted leaf blocks into quadratic blocks, the subdivision of the primary sample array 504 is spatially performed by the lower sample array 502, so it is easy to understand that blocks of different planes within a plane group are co-located. In the case of inter-plane adoption / prediction between different plane groups, this co-location may be defined to allow for more flexibility between the subdivision of these plane groups. Given a reference plane group, the co-located blocks within the reference plane group are identified. The co-located blocks and reference plane group can be derived by the following process: A specific sample 514 within a current block 516 of one of the sample arrays 506 of the secondary plane group 512 is selected. This may be the top-left sample of the current block 516, as illustrated by 514 in FIG. 11 . It may also be a sample within the current block 516 that is closer to the center of the current block 516, or any other sample within the current block, as long as it is geometrically uniquely defined. The location of this selected sample 515 within the sample arrays 502, 504 of the reference plane group 510 is calculated. The location of sample 514 within sample arrays 502 and 504 is shown at 518 and 520, respectively, in Figure 11. Which plane 502 or 504 of reference plane group 510 is actually used may be pre-specified or signaled in the bitstream. The sample in the corresponding sample array 502 or 504 of reference plane group 510 that is closest to location 518 or 520, respectively, is identified, and the block containing this sample is selected as the co-located block within sample arrays 502 and 504, respectively. In the case of Figure 11, these are blocks 522 and 524, respectively. Alternative approaches for identifying co-located blocks in other planes are described below.

[0144] In an embodiment, the coding parameters specifying the prediction for the current block 516 are fully estimated using corresponding prediction parameters for the co-located blocks 522 / 524 in a different plane group 510 of the same image 500, without transmitting any additional side information. This estimation is performed by simply copying the corresponding coding parameters or by adapting the coding parameters to account for differences between the current plane group 512 and the reference plane group 510. For example, this adaptation may involve adding motion parameters (e.g., correcting displacement vectors) to account for phase differences between the luma and chroma sample arrays. This adaptation may also involve modifying the precision of the motion parameters (e.g., modifying displacement vector precision) to account for different resolutions of the luma and chroma sample arrays. In another embodiment, one or more estimated coding parameters for specifying the generation of the prediction signal are not directly used for the current block 516, but are used as predictions of corresponding coding parameters for the current block 516, and refinement of these coding parameters for the current block 516 is transmitted in the bitstream 22. For example, the estimated motion parameters are not used directly, but motion parameter differences (such as displacement vector differences) are coded in the bitstream, specifying the deviation between the motion parameters used for the current block 516 and the estimated motion parameters. At the decoder side, the actual used motion parameters are obtained by combining the differences between the estimated and transmitted motion parameters.

[0145] In another embodiment, the subdivision of blocks, such as the predictive subdivision treeblocks described above, into prediction blocks (i.e., blocks of examples for which the same set of prediction parameters is used) is estimated or predicted from previously coded co-located blocks in different plane groups of the same image, i.e., the bit sequences of Figures 6A or 6b, as appropriate. In this embodiment, one of two or more plane groups is coded as a primary plane group. For all blocks in this primary plane group, subdivision information is transmitted without reference to other plane groups of the same image. The remaining plane groups are coded as secondary plane groups. For blocks in the secondary plane groups, one or more syntax elements are transmitted that signal whether subdivision information is estimated or predicted from co-located blocks in other plane groups or whether subdivision information is transmitted in the bitstream. One of the one or more syntax elements may be referred to as an inter-prediction flag or inter-prediction parameter. If a syntax element signals that subdivision information is not estimated or predicted, then subdivision information for the block is transmitted in the bitstream without reference to other plane groups of the same image. If a syntax element signals that subdivision information is estimated or predicted, it identifies a co-located block in a so-called reference plane group. The assignment of reference plane groups to blocks can be configured in several ways. In one embodiment, a specific reference plane group is assigned to each secondary plane group. This assignment can be fixed or signaled in a high-level syntax construct such as a set of parameters, an access unit header, a picture header, or a slice header. In a second embodiment, the assignment of reference plane groups is coded in the bitstream and signaled by one or more syntax elements. This syntax element is coded for the block and specifies whether subdivision information is estimated, predicted, or coded separately. The reference plane group can be a primary plane group or another secondary plane group.Given a reference plane group, co-located blocks within the reference plane group are identified. A co-located block is a block within the reference plane group that corresponds to the same image area as the current block or represents a block within the reference plane group that shares the largest portion of image area with the current block. The co-located blocks can be partitioned into smaller prediction blocks.

[0146] In other embodiments, subdivision information for a current block, such as the quadtree-based subdivision information of FIG. 6A or 6b, is fully estimated using subdivision information of co-located blocks in different plane groups of the same image, without transmitting additional accompanying information. As one particular example, if the co-located block is partitioned into two or four prediction blocks, the current block is also partitioned into two or four sub-blocks for prediction purposes. As another particular example, if the co-located block is partitioned into four sub-blocks, and one of these sub-blocks is further partitioned into four smaller sub-blocks, the current block is also partitioned into four sub-blocks, and one of these sub-blocks (corresponding to the sub-blocks of the co-located block that are further decomposed) is also partitioned into four smaller sub-blocks. In other preferred embodiments, the estimated subdivision information is not used directly for the current block, but is used as a prediction of the actual subdivision information for the current block, and corresponding refinement information is transmitted in the bitstream. For example, the subdivision information estimated from the co-located block may be further refined. For each sub-block that corresponds to a sub-block in a co-located block that is not partitioned into smaller blocks, a syntax element can be coded in the bitstream that specifies whether the sub-block is further decomposed in the current plane group. The transmission of such a syntax element can be conditioned on the size of the sub-block. Alternatively, a signal can be sent in the bitstream indicating that a sub-block that is further partitioned in the reference plane group will not be partitioned into smaller blocks in the current plane group.

[0147] In other embodiments, both the subdivision of a block into predictive blocks and the coding parameters specifying how its sub-blocks are predicted are adaptively estimated or predicted from previously coded co-located blocks in different plane groups of the same image. In a preferred embodiment of the present invention, one of two or more plane groups is coded as a primary plane group. For all blocks in this primary plane group, subdivision information and prediction parameters are transmitted without reference to other plane groups of the same image. The remaining plane groups are coded as secondary plane groups. For blocks in the secondary plane groups, one or more syntax elements are transmitted that signal whether subdivision information and prediction parameters are estimated or predicted from co-located blocks in other plane groups or whether subdivision information and prediction parameters are transmitted in the bitstream. One of the one or more syntax elements may be referred to as an inter-prediction flag or inter-prediction parameters. If a syntax element signals that subdivision information and prediction parameters are not estimated or predicted, the subdivision information of the block and the prediction parameters for the resulting sub-blocks are transmitted in the bitstream without reference to other plane groups of the same image. If a syntax element signals that subdivision information and prediction parameters for a sub-block are estimated or predicted, then a co-located block in a so-called reference plane group is identified. The assignment of reference plane groups to blocks can be configured in several ways. In one embodiment, a specific reference plane group is assigned to each secondary plane group. This assignment can be fixed or signaled in a high-level syntax construct such as a parameter set, an access unit header, a picture header, or a slice header. In a second embodiment, the assignment of reference plane groups is coded in the bitstream and signaled by one or more syntax elements. This syntax element is coded for the block and specifies whether the subdivision information and prediction parameters are estimated or predicted, or coded separately.The reference plane group may be a primary plane group or another secondary plane group. Given a reference plane group, a co-located block within the reference plane group is identified. The co-located block may be a block within the reference plane group that corresponds to the same image region as the current block, or a block representing a block within the reference plane group that shares the largest portion of the image region with the current block. The co-located block may be partitioned into smaller prediction blocks. In a preferred embodiment, the subdivision information for the current block and the prediction parameters for the resulting sub-blocks are fully estimated using the subdivision information of co-located blocks within different plane groups of the same image and the prediction parameters of the corresponding sub-blocks, without transmitting any additional ancillary information. As one specific example, if the co-located block is partitioned into two or four prediction blocks, the current block is also partitioned into two or four sub-blocks for prediction purposes, and prediction parameters for the sub-blocks of the current block are derived as described above. As another example, if a co-located block is partitioned into four sub-blocks and one of these sub-blocks is further partitioned into four smaller sub-blocks, then the current block is also partitioned into four sub-blocks, one of these sub-blocks (corresponding to a sub-block of the further decomposed co-located block) is also partitioned into four smaller sub-blocks, and prediction parameters for all sub-blocks that are not further partitioned are estimated as described above. In another preferred embodiment, the subdivision information is fully estimated based on the subdivision information of the co-located block in the reference plane group, but the prediction parameters estimated for a sub-block are only used as a prediction of the sub-block's actual prediction parameters. The deviation between the actual prediction parameters and the estimated prediction parameters is coded in the bitstream. In another embodiment, the estimated subdivision information is used as a prediction for the current block's actual sub-division information and the difference is transmitted in the bitstream (as described above), but the prediction parameters are fully estimated.In another embodiment, both the estimated subdivision information and the estimated prediction parameters are used as predictions and the difference between the actual subdivision information and prediction parameters and the estimated values ​​is transmitted in the bitstream.

[0148] In another embodiment, for blocks of a plane group, the mode of residual coding (e.g., transform type) is adaptively selected whether it is estimated or predicted from co-located coded blocks of different plane groups of the same image, or whether the mode of residual coding is coded separately for the blocks. This embodiment is similar to the adaptive estimation / prediction of prediction parameters described above.

[0149] In another embodiment, the subdivision of a block (e.g., a prediction block) into transform blocks (i.e., blocks of samples to which a 2D transform is applied) is adaptively estimated or predicted from co-located coded blocks of a different plane group of the same image. This embodiment is similar to the above-described embodiment in which the estimation / prediction of the subdivision into prediction blocks is adaptive.

[0150] In another embodiment, the subdivision of a block into transform blocks and the mode of coding of the residuals for the resulting transform blocks (e.g., transform type) are adaptively estimated or predicted from co-located coded blocks of a different plane group of the same image. This embodiment is similar to the above-described embodiment of adaptive subdivision into prediction blocks and estimation / prediction of prediction parameters for the resulting prediction blocks.

[0151] In another embodiment, the subdivision of the block into prediction blocks, the associated prediction parameters, the subdivision information of the prediction block and the mode of coding of the residual for the transform block are adaptively estimated or predicted from previously coded co-located blocks of a different plane group of the same image. This embodiment is a combination of the above-mentioned embodiments. It is also possible to estimate or predict only some of the above-mentioned coding parameters.

[0152] Thus, as mentioned above, inter-plane adoption / prediction can potentially improve coding efficiency, however, this gain in coding efficiency from inter-plane adoption / prediction is also possible when multi-tree based subdivision and other block subdivisions are used, regardless of whether block merging is performed or not.

[0153] The embodiments outlined above regarding inter-plane matching / prediction are applicable to image and video encoders and decoders. The encoder and decoder divide the color planes of an image, and, if present, auxiliary sample arrays associated with the image, into blocks and associate these blocks with coding parameters. For each block, a set of coding parameters may be included in the bitstream. For example, these coding parameters may describe how the block is predicted or decoded at the decoder side. For example, the coding parameters may represent the prediction mode of a macroblock or block, subdivision information, intra-prediction mode, reference indices used for motion-compensated prediction, motion parameters such as displacement vectors, mode of residual coding, transform coefficients, etc. The sample arrays associated with an image may have different sizes.

[0154] Next, we will describe an extended signaling scheme for coding parameters in a tree-based partitioning scheme, such as that described above with reference to Figures 1 to 8. The scheme described below can be combined with any one or combination of the above embodiments, but the effects and advantages of the extended signaling scheme, as well as the other schemes, namely, join and inter-plane adoption / prediction, will be described independently of the above embodiments. This scheme will be referred to as inheritance hereinafter.

[0155] An improved coding scheme for encoding ancillary information in tree-based partitioning schemes, called inheritance, is described below, which generally provides the following advantages over conventional schemes for handling coding parameters:

[0156] Typically, in traditional image and video coding, an image, or a set of specific sample arrays within an image, is decomposed into blocks and associated with specific coding parameters. An image typically consists of multiple sample arrays. Furthermore, an image may be associated with additional auxiliary sample arrays that specify, for example, transparency information or depth maps. The sample arrays (including the auxiliary sample arrays) of an image can be grouped into one or more so-called plane groups, each consisting of one or more sample arrays. A plane group of an image can be coded independently or, if the image is associated with multiple plane groups, can be coded by prediction from other plane groups of the same image. Each plane group is typically decomposed into blocks. Blocks (or corresponding blocks of sample arrays) are predicted either by inter-picture prediction or intra-picture prediction. Blocks can be of various sizes and are either square or rectangular. The partitioning of an image into blocks can be fixed by syntax or (at least in part) signaled within the bitstream. Syntax elements that signal subdivision information for blocks of a given size are often transmitted. These syntax elements allow specifying whether and how a block is to be subdivided into smaller blocks and associated with coding parameters, e.g., for prediction purposes. For every sample of a block (or the corresponding block of a sample array), the decoding of the associated coding parameters is specified in a specific way. For example, all samples in a block are predicted using the same set of prediction parameters, such as reference indices (which identify a reference picture in a set of previously coded pictures), motion parameters (which specify the amount of block motion between the reference picture and the current picture), and parameters for specifying interpolation filters, intra-prediction modes, etc. Motion parameters can be represented by displacement vectors with horizontal and vertical components, or by higher-order motion parameters, such as affine motion parameters with six components. It is also possible to associate multiple sets of specific prediction parameters (e.g., reference indices and motion parameters) with a single block.In this case, for each set of these specific prediction parameters, a single intermediate prediction signal for the block (or the corresponding block in the sample array) is generated, and the final prediction signal is formed by combining the intermediate prediction signals, including superposition. The corresponding weighting parameters, possibly with a constant offset (added to the weighted sum), can be fixed for either the image, the reference image, or the set of reference images, or can be included in the set of prediction parameters for the corresponding block. The difference between the original block (or the corresponding block in the sample array) and the prediction signal, also known as the residual signal, is typically transformed and quantized. A two-dimensional transform is often applied to the residual signal (or the corresponding sample array of the residual block). For transform coding, a block (or the corresponding block in the sample array) for which a specific set of prediction parameters is used can be further divided before applying the transform. The transform block can be the same size as the block used for prediction, or smaller. A transform block can also contain multiple blocks used for prediction. Each transform block can be of various sizes, resulting in a square or rectangular block. After transformation, the resulting transform coefficients are quantized to obtain so-called transform coefficient levels. The transform coefficient levels and prediction parameters, and subdivision information, if present, are entropy coded.

[0157] In some image and video coding standards, the syntax provides very limited possibilities for subdividing an image (or a group of planes) into blocks. It is usually possible to specify only whether and, possibly, how a block of a given size can be subdivided into smaller blocks. For example, in H.264, the maximum block size is 16x16. The 16x16 blocks are also called macroblocks, and each image is partitioned into macroblocks in the first step. For each 16x16 macroblock, it is possible to signal whether it will be coded as a 16x16 block, two 16x8 blocks, two 8x16 blocks, or four 8x8 blocks. If a 16x16 block is subdivided into four 8x8 blocks, each of these 8x8 blocks can be coded as one 8x8 block, two 8x4 blocks, two 4x8 blocks, or four 4x4 blocks. In current image and video coding standards, reducing the possibilities for specifying partitioning into blocks has the advantage of keeping the side information rate for conveying subdivision information small, but the disadvantage, as explained below, is that the bit rate required for transmitting prediction parameters for a block can be large. The side information rate for conveying prediction information typically accounts for a significant portion of the overall bit rate for a block. Therefore, reducing this side information can improve coding efficiency, which can be achieved, for example, by using larger block sizes. Images in real-world images and video sequences consist of objects of arbitrary shapes with specific properties. For example, these objects or object parts are characterized by a specific texture or a specific motion. The same set of prediction parameters is usually applied to these objects or object parts. However, object boundaries usually do not coincide with the boundaries of larger prediction blocks (e.g., 16x16 macroblocks in H.264). The encoder typically makes subdivision decisions (among a limited set of possibilities) that minimize a certain rate / distortion loss.This can result in a large number of small blocks for an object of arbitrary shape. Because each small block is associated with a set of prediction parameters that need to be transmitted, the ancillary information rate can become a significant portion of the overall bit rate. However, because some of the small blocks still represent the same object or a region of an object, the prediction parameters of the resulting blocks will be identical or very similar. Coding efficiency would be improved if the syntax were extended in a way that not only allows for block subdivision but also allows for coding parameter sharing among the resulting blocks after subdivision. In tree-based subdivision, coding parameter sharing can be achieved for a given set of blocks by assigning coding parameters, or parts of them, to one or more parent nodes in a tree-based hierarchy. As a result, the shared parameters, or parts of them, can be used to reduce the ancillary information needed to inform the actual selection of coding parameters for the resulting blocks after subdivision. This can be achieved by omitting to signal parameters for subsequent blocks or by using shared parameters for parameter prediction and / or context modeling for subsequent blocks.

[0158] The basic idea of ​​the inheritance scheme described below is to reduce the bit rate required to transmit the coding parameters by sharing information according to a tree-based hierarchy of blocks. The shared information signal is transmitted in the bitstream (in addition to the subdivision information). The advantage of the inheritance scheme is an increase in coding efficiency due to the reduced information rate associated with the coding parameters.

[0159] To reduce the associated information rate, in the embodiments described below, the respective coding parameters for a particular sample set, i.e., a simply connected region, of a multi-tree subdivision are transmitted in the data stream in an efficient manner. A particular sample set, i.e., a simply connected region, can represent a rectangular or square block, or a region of any shape, or any other sample set. An inheritance scheme, described below, obviates the need to explicitly include the coding parameters for each of these sample sets in the bitstream. The coding parameters may also represent prediction parameters that specify how to predict the corresponding sample set using previously coded samples. The numerous possibilities and examples described above also apply here. As indicated above and further described below, with regard to the following inheritance scheme, the tree-based partitioning of the image sample array into sample sets may be fixed by the syntax or may be conveyed by the corresponding subdivision information in the bitstream. As mentioned above, the coding parameters for the sample sets may be transmitted in a predetermined order specified by the syntax.

[0160] According to the inheritance scheme, the decoder or the decoder's extractor 102 is configured to derive information about the coding parameters of individual singly connected regions or sample sets in a specific way. In particular, coding parameters or parts thereof, such as parameters used for prediction purposes, are shared between blocks according to a given tree-based partitioning scheme, with the sharing groups according to a tree structure determined by the encoder or inserter 18. In a specific embodiment, the sharing of coding parameters for all child nodes of a given internal node of the partitioning tree is indicated using a specific sharing flag with a binary value. As an alternative approach, coding parameter refinements can be transmitted for each node, such that cumulative parameter refinements according to the tree-based block hierarchy can be applied to all sample sets of the block of a given leaf node. In another embodiment, parts of the coding parameters transmitted for internal nodes according to the tree-based block hierarchy can be used for context-adaptive entropy coding and entropy decoding of the coding parameters or parts thereof for the block of a given leaf node.

[0161] Figures 12A and 12B illustrate the basic concept of inheritance in the specific case of using a quadtree-based partitioning. However, as noted above, other multi-tree subdivisions may be used. Figure 12A illustrates a tree structure, and Figure 12B illustrates a spatial partitioning corresponding to the tree structure of Figure 12A. The partitioning shown is similar to that shown in Figures 3A through 3C. In general, inheritance schemes allow for the assignment of associated information to nodes at different non-leaf levels within a tree structure. Depending on the assignment of associated information to nodes at different levels within the tree, such as the internal nodes and the root node within the tree of Figure 12A, varying degrees of sharing of associated information can be achieved within the tree hierarchy of blocks shown in Figure 12B. For example, if all leaf nodes in the fourth layer, all of which have the same parent node in the case of FIG. 12A, are determined to share ancillary information, this essentially means that the smallest blocks, designated 156a through 156d in FIG. 12B, share this ancillary information, eliminating the need to transmit the ancillary information for each of these small blocks 156a through 156d, i.e., four times. However, this remains an option for the encoder. Alternatively, the entire area of ​​hierarchical level 1 (second layer) in FIG. 12A, i.e., the upper right quarter of treeblock 150, including subblocks 154a, 154b, and 154d and the smaller subblocks 156a through 156d, could be determined as the area in which coding parameters are shared. Thus, the area in which ancillary information is shared increases. The next level of increase is to combine all subblocks in the first layer, i.e., subblocks 152a, 152c, and 152d, and the smaller blocks. In other words, in this case, the entire treeblock has associated information assigned to it, and all sub-blocks of this treeblock 150 share the associated information.

[0162] In the following description of inheritance, the following notation is used to describe the embodiment. a. The reconstructed sample of the current leaf node: r b. Reconstructed specimen of adjacent leaf: r' c. Predictor of the current leaf node: p d. Residual of the current leaf node: Res e. Reconstructed residual of the current leaf node: RecRes f. Scaling and inverse transformation: SIT g. Shared flag: f

[0163] As a first example of inheritance, we consider signaling intra prediction at internal nodes. Specifically, we consider how intra prediction modes are signaled at internal nodes for tree-based block partitioning for prediction purposes. Internal nodes (including the root node) can propagate some of the associated information used by corresponding child nodes by traversing the tree from the root node to the leaf nodes. Specifically, for internal nodes, a shared flag f is transmitted with the following meaning:

[0164] If f has a value of 1 ("true"), all child nodes of a given internal node share the same intra prediction mode. In addition to the sharing flag f having a value of 1, the internal node also propagates the parameters of the intra prediction mode used by all its child nodes. Therefore, all subsequent child nodes do not transmit any prediction mode information or sharing flag. When reconstructing all associated leaf nodes, the decoder applies the intra prediction mode from the corresponding internal node.

[0165] If the value of f is 0 ("false"), then the child nodes of the corresponding internal node do not share the same intra-prediction mode, and each child node that is an internal node transmits a separate sharing flag.

[0166] Figure 12C illustrates the signaling of intra prediction at the internal nodes mentioned above: the internal nodes in the first layer convey the shared flag and the side information provided by the intra prediction mode information, while the child nodes do not transmit any side information.

[0167] As a second example of inheritance, we will explain the refinement of inter prediction. Specifically, we will explain how to propagate ancillary information of inter prediction mode in intra mode of tree-based block partitioning for the purpose of refining motion parameters provided by motion vectors, etc. Internal nodes (including the root node) can propagate part of the ancillary information that is refined in the corresponding child node by traversing the tree from the root node to the leaf nodes. Specifically, for internal nodes, a shared flag f is transmitted with the following meaning:

[0168] If f has the value 1 ('true'), all child nodes of a given internal node share the same motion vector reference. In addition to a sharing flag f with value 1, the internal node also transmits a motion vector and a reference index. All subsequent child nodes then do not transmit any further sharing flags, but may transmit a refinement of this inherited motion vector reference. Upon reconstruction of all associated leaf nodes, the decoder adds the motion vector refinement of a given leaf node to the inherited motion vector reference belonging to the corresponding internal parent node with a sharing flag f with value 1. This means that the motion vector refinement of a given leaf node is the difference between the actual motion vector applied at this leaf node for motion compensated prediction and the motion vector reference of the corresponding internal parent node.

[0169] If the value of f is 0 ("false"), the child nodes of the corresponding internal node do not necessarily share the same inter-prediction mode, and no motion parameter refinement is performed at the child nodes by using motion parameters from the corresponding internal node. Each child node that is an internal node transmits a separate sharing flag.

[0170] 12D illustrates the above-mentioned motion parameter refinement. The internal nodes in the first layer transmit the shared flag and associated information. The leaf child nodes transmit only the motion parameter refinement, without associated information, as do the internal child nodes in the second layer, for example.

[0171] Reference is now made to Figure 13. Figure 13 is a flowchart illustrating a mode of operation of a decoder, such as the decoder of Figure 2, in reconstructing an array of information samples representing a spatial example information signal from a data stream. The array of information samples is subdivided into leaf areas of different sizes by a multi-tree subdivision. As explained above, each leaf area is associated with one hierarchical level of a series of hierarchical levels of the multi-tree subdivision. For example, all blocks shown in Figure 12B are leaf areas. For example, leaf area 156c is associated with layer 4 (or level 3) of the hierarchy. Each leaf area is associated with coding parameters. Examples of these coding parameters are explained above. The coding parameters are represented by a respective set of syntax elements for each leaf area. Each syntax element has a respective syntax element type of one of a series of syntax element types. For example, such syntax element types can be a prediction mode, a component of a motion vector, an indication of an intra-prediction mode, etc. The decoder performs the following steps according to Figure 13:

[0172] In step 550, inheritance information is extracted from the data stream. In the case of Fig. 2, the extractor 102 performs step 550. The inheritance information indicates whether inheritance is used for the current arrangement of information samples. The following description will reveal several possibilities for the inheritance information, such as a sharing flag f and information propagation for a multi-tree structure divided into primary and secondary parts.

[0173] The array of information samples may already be part of the image, such as a treeblock, i.e., treeblock 150 in Figure 12B. Thus, inheritance information indicates whether inheritance is used for a particular treeblock 150. Such inheritance information may be inserted into the data stream, for example, for all treeblocks of a prediction subdivision.

[0174] Furthermore, if inheritance is indicated, the inheritance information indicates at least one inheritance region of the information sample array. This inheritance region is composed of a set of leaf regions and corresponds to one hierarchical level of the series of multi-tree subdivisions located below each hierarchical level associated with the set of leaf regions. In other words, the inheritance information indicates whether inheritance is used for the current sample array, such as tree block 150. If inheritance is used, at least one inheritance region or sub-region of tree block 150 is indicated, within which leaf regions share coding parameters. Thus, the inheritance region does not have to be a leaf region. In the example of FIG. 12B, this inheritance region may be, for example, the region formed by sub-blocks 156a to 156b. Alternatively, the inheritance region may be larger and include sub-blocks 154a, 154b, and 154d. Furthermore, the inheritance region may be tree block 150 itself, with all leaf blocks sharing the coding parameters associated with the inheritance region.

[0175] It should also be noted that multiple regions of inheritance may be defined within each sample array or treeblock 150. For example, if the bottom-left subblock 152c is also partitioned into smaller blocks, then subblock 152c can also form a region of inheritance.

[0176] In step 552, the inheritance information is checked to determine whether inheritance is used. If inheritance is used, the process of FIG. 13 proceeds to step 554, where an inheritance subset including at least one syntax element of a predetermined syntax element type is extracted from the data stream for each inter-plane inheritance region. In the following step 556, this inheritance subset is copied to or used as a prediction of a corresponding inheritance subset of syntax elements in a set of syntax elements representing coding parameters associated with a set of leaf regions constituting each of the at least one inheritance region. In other words, for each inheritance region indicated in the inheritance information, the data stream constitutes an inheritance subset of syntax elements. In other words, the inheritance relates to at least one specific syntax element type or category that can be used for inheritance. For example, syntax elements of a prediction mode, an inter-prediction mode, or an intra-prediction mode may be subject to inheritance. For example, the inheritance subset included in the data stream for the inheritance region may include a syntax element of an inter-prediction mode. The inheritance subset may also include additional syntax elements, whose types depend on the value of the above-mentioned fixed syntax element type associated with the inheritance scheme. For example, if an inter-prediction mode is a fixed component of the inheritance subset, syntax elements defining motion compensation, such as components of a motion vector, may or may not be included in the inheritance subset, depending on the syntax. For example, if the upper right quarter of treeblock 150, i.e., subblock 152b, is the inheritance region, the inheritance region may indicate only the inter-prediction mode, or the inter-prediction mode, a motion vector, and a motion vector index.

[0177] All syntax elements contained in a subset of the inheritance are copied to or used as predictions for the corresponding coding parameters of the leaf blocks in that region of the inheritance, i.e., leaf blocks 154a, 154b, 154d, and 156a to 156d. If prediction is used, a residual is transmitted for each leaf block.

[0178] One possibility for transmitting inheritance information for tree block 150 is the transmission of the aforementioned sharing flag f. In this case, extracting the inheritance information in step 550 includes the following: In particular, the decoder is configured to extract sharing flags f from the data stream for non-leaf regions corresponding to any of the inheritance sets of at least one hierarchical level of the multi-tree subdivision, using the hierarchical level order from a lower hierarchical level to a higher hierarchical level, and to check whether inheritance is specified in each inheritance or sharing flag. For example, the inheritance set of hierarchical levels is formed by the first to third layers of the hierarchy in FIG. 12A. Therefore, any of the nodes of the subtree structure located in any of the first to third layers, rather than the leaf nodes, have an associated sharing flag in the data stream. The decoder extracts these sharing flags in a traversal order, such as depth order or width order, from the first to third layers. If one of the sharing flags is 1, the decoder recognizes that the leaf blocks included in the corresponding inheritance region share the inheritance subset subsequently extracted in step 554. There is no need to check the inheritance flags for the child nodes of the current node, in other words, the inheritance flags for these child nodes are not transmitted in the data stream, since it is clear that the domain of these nodes already belongs to the domain of inheritance with which an inheritance subset of syntax elements is shared.

[0179] The sharing flag f can be alternated with the above bits that convey the subdivision of the quadtree described above. For example, an alternating bit sequence containing both subdivision and sharing flags can be shown as follows:

number

[0180] Another way to define the inheritance information indicating the inheritance domain is to use two subdivisions defined in a subordinate manner to each other, as described above for the prediction and residual subdivisions, respectively. In general, the leaf blocks of a primary subdivision can form inheritance domains that define the domains where inheritance subsets of syntax elements are shared. At the same time, the blocks within these inheritance domains where inheritance subsets of syntax elements are copied or used as predictions are defined by subordinate subdivisions.

[0181] For example, consider the residual tree as an extension of the prediction tree. Furthermore, consider the case where the prediction block can be further divided into smaller blocks for the purpose of residual coding. For each prediction block corresponding to a leaf node of the prediction-related quadtree, one or more subordinate quadtrees determine the corresponding subdivision for residual coding.

[0182] In this case, instead of using the signaling of predictions at the internal nodes, the residual tree is interpreted as follows: a refinement of the prediction tree is also specified in the sense of using a certain prediction mode (signaled by the corresponding leaf node of the tree related to the prediction) and a refined reference example is also used. The following example illustrates this case.

[0183] For example, Figures 14A and 14B show quadtree partitioning for intra prediction. For one particular leaf node of the primary subdivision, neighboring reference samples are highlighted. Figure 14B shows the residual quadtree subdivision for the same predicted leaf node, along with refined reference samples. All sub-blocks shown in Figure 14B share the same inter-prediction parameters contained in the data stream for each leaf block highlighted in Figure 14A. Thus, Figure 14A shows an example of a conventional quadtree partitioning for intra prediction, with reference samples for one particular leaf node shown. However, in a preferred embodiment, a separate intra-prediction signal is calculated for each leaf node in the residual tree, e.g., by using neighboring samples of the reconstructed leaf node in the residual tree, as shown by the gray stripe shading in Figure 14B. The quantized residual signal is then added to this prediction signal to obtain a reconstructed signal for a given residual leaf node in the usual way. This reconstructed signal is used as a reference signal for subsequent prediction processes. The decoding order for prediction is the same as the decoding order for the residual.

[0184] As shown in Figure 15, during the decoding process, for each leaf node of the residual, a prediction signal p is calculated using the reference sample r' according to the actual intra prediction mode (as indicated by the leaf node of the quadtree associated with the prediction).

[0185] SIT processing,

number

number

[0186] The decoding order for prediction is the same as the decoding order for residuals shown in FIG.

[0187] Each of the residual leaf nodes is decoded as described in the previous paragraph. The reconstructed signal r is stored in a buffer, as shown in Figure 16. From this buffer, the reference sample r' is taken for the next prediction and decoding process.

[0188] With reference to Figures 1 to 16, specific embodiments have been described that combine the various aspects outlined above. However, other embodiments of the present application will now be described that focus on the specific aspects described above and combine several of the above-mentioned embodiments.

[0189] In particular, the embodiments described above based on the configurations of Figures 1 and 2 primarily combine many aspects of the present application, and can be used advantageously in other applications and other coding fields. As reiterated above, for example, multi-tree subdivision may be used without combining and / or inter-plane adoption / prediction and / or inheritance. For example, aspects such as transmitting a maximum block size, using a depth-ordered traversal order, adapting the context according to the hierarchical level of each subdivision flag, and transmitting a maximum hierarchical level in the bitstream, all provide advantages independent of one another, with the aim of reducing the bitrate of side information. The same is true when considering combining schemes. Benefits are obtained from combining regardless of the detailed manner in which an image is subdivided into simply connected regions, and regardless of the presence of multiple sample arrays and the use of inter-plane adoption / prediction and / or inheritance. The same is true for the benefits obtained from inter-plane adoption / prediction and inheritance.

[0190] Therefore, the embodiments outlined below synthesize the above-mentioned embodiments with respect to aspects related to coupling, and since the following embodiments synthesize the above-mentioned embodiments, it can be considered that many of the above-mentioned details can be combined with the embodiments described below.

[0191] FIG. 17 illustrates a decoder according to an embodiment of the present application. The decoder of FIG. 17 includes an extractor 600 and a reconstructor 602. The extractor 600 is configured to extract payload data from a data stream 604 for each of a plurality of singly-connected regions into which an array of information samples representing a spatially sampled information signal is subdivided. As described above, the singly-connected regions into which the array of information samples is subdivided may be square or rectangular in shape due to multi-tree subdivision. Furthermore, the specifically described embodiment of the subdivision of the sample array is merely a specific embodiment, and other subdivisions may be used. Some possibilities are illustrated in FIGS. 18A-18C. For example, FIG. 18A illustrates the subdivision of a sample array 606 into a regular two-dimensional array of non-overlapping, adjacent treeblocks 608, some of which are further subdivided into subblocks 610 of various sizes according to the multi-tree structure. As described above, while FIG. 18A illustrates a quadtree subdivision, partitioning each parent node into other numbers of child nodes is also possible. In the embodiment shown in Figure 18B, multi-tree subdivision is applied directly to the entire pixel array 606, thereby subdividing the sample array 606 into sub-blocks of various sizes. That is, the entire pixel array 606 is treated as a tree block. Figure 18C shows another embodiment. In this embodiment, the sample array is structured as a regular two-dimensional array of adjacent square or rectangular macroblocks 612. Each of these macroblocks 612 is individually associated with partitioning information, and according to this partitioning information, the macroblock 612 is either left unpartitioned or partitioned into a regular two-dimensional array of blocks of the size indicated by the partitioning information. As can be seen, all of the subdivisions in Figures 13A-13C subdivide the sample array 606 into non-overlapping, simply connected regions, such as those in the embodiments of Figures 18A-18C. However, several alternatives are possible. For example, blocks may overlap each other, provided that each block has a portion that does not overlap any of its neighboring blocks.Alternatively, the neighboring blocks arranged alongside the current block in a given direction are restricted to overlapping each sample of the block by at most one block, meaning that the left and right neighboring blocks overlap the current block so as to completely cover it, but do not overlap each other, and similarly for the vertical and diagonal neighboring blocks.

[0192] As discussed above with reference to Figures 1 through 16, the array of information samples does not necessarily represent a video image or a still image. The sample array 606 may also represent a depth map or a transparency map of a particular scene. As already explained above, the payload data associated with each of the plurality of simply connected regions may include residual data in the spatial or transform domain, such as transform coefficients, and a significance map identifying the locations of significant transform coefficients within the transform block corresponding to the residual block. In general, the payload data extracted by the extractor 600 for each simply connected region from the data stream 604 is data that describes the associated simply connected region spatially, either directly or as a residual relative to some prediction, in either the spatial or spectral domain.

[0193] The reconstructor 602 is configured to reconstruct the sequence of information samples from the payload data for the singly connected regions of the sequence of information samples by processing, for each singly connected region, the payload data for each singly connected region in a manner specified by the coding parameters associated with each singly connected region. As explained above, the coding parameters may be prediction parameters, so that the singly connected regions shown in Figures 18A-18B may correspond to the prediction blocks mentioned above, i.e., blocks for which the data stream 604 specifies prediction details for the prediction of individual singly connected regions. However, the coding parameters are not limited to prediction parameters. The coding parameters may indicate a transform used to transform the payload data. Alternatively, they may define a filter used to reconstruct the individual singly connected regions during the reconstruction of the sequence of information samples.

[0194] The extractor 600 is configured to identify, for a given singly connected region, a singly connected region among the plurality of singly connected regions that has a predetermined relative position relative to the given singly connected region. This step is described in detail above with reference to step 450. That is, the identification may be performed depending on a subset of the coding parameters associated with the given singly connected region in addition to the predetermined relative position. After the identification, the extractor 600 extracts a merge indicator for the given singly connected region from the data stream 604. If the number of singly connected regions that has a predetermined relative position relative to the given singly connected region is greater than zero, the above description of steps 452 and 454 applies. If the merge indicator indicates that the given block should be merged, the extractor 600 is configured to check whether the number of singly connected regions that has a predetermined relative position relative to the given singly connected region is one, or whether the number of singly connected regions that has a predetermined relative position relative to the given singly connected region is greater than one but their coding parameters match each other. If one of these two is the case, the extractor 600 may employ the coding parameters or use them to predict the coding parameters of the given singly connected region or a remaining subset thereof, as described above in steps 458 to 468. If the latter check reveals that the number of simply connected regions in a given relative position to the given singly connected region is greater than one and that the coding parameters are different from one another, as described above with reference to Figure 10, further indicators may be extracted from the data stream 604 by the extractor 600.

[0195] The latter check allows suppressing the transmission of further indicators of one or a subset of candidate simply connected regions, thereby reducing the associated information overhead.

[0196] FIG. 19 shows the general structure of an encoder for generating a data stream that can be decoded by the decoder of FIG. 17. The encoder of FIG. 19 includes a data generator 650 and an inserter 652. The data generator 650 is configured to encode the sequence of information samples into payload data, for each of a plurality of interconnected regions into which the sequence of information samples is subdivided, in association with coding parameters associated with each singly-connected region to indicate how the payload data for each singly-connected region is reconstructed. The inserter 652 performs the same identification and validation as the extractor 600 of the decoder of FIG. 12, but inserts rather than extracts merge indicators and suppresses the insertion of coding parameters into the data stream. Alternatively, rather than inserting all coding parameters into the data stream, the inserter inserts the residuals of the respective predictions instead of the adoption / prediction described above with reference to FIG. 12 and FIG. 10, respectively.

[0197] It should also be noted that the encoder structure in Figure 19 is a schematic representation, and in practice the determination of the payload data, coding parameters, and merging index may be an iterative process. For example, considering that if the coding parameters of adjacent singly connected regions are similar but not identical to each other, the merging index allows for the complete suppression of one coding parameter of the singly connected region, and instead of presenting all of these coding parameters, it may be determined that it is desirable to tolerate small differences between these coding parameters rather than signaling them to the decoder.

[0198] Figure 20 shows another embodiment of a decoder. The decoder of Figure 20 comprises a subdivision unit 700, a combination unit 702, and a reconstruction unit 704. The subdivision unit is configured to spatially subdivide an array of samples representing a spatial sampling of a two-dimensional information signal into a plurality of non-overlapping, simply connected regions of different sizes by recursive multiple partitioning depending on a subset of syntax elements contained in the data stream. This multiple partitioning may therefore correspond to the embodiments outlined above with reference to Figures 1 to 16 or Figure 18A or 18B. The syntax elements contained in the data stream for indicating the subdivision may be defined as shown above with reference to Figures 6A and 6b, or may be defined in another way.

[0199] The combining unit 702 is configured to combine spatially adjacent singly connected regions from the plurality of singly connected regions according to a second subset of syntax elements in the data stream that are separated from the first subset to obtain an intermediate subdivision of the sample array into disjoint sets of singly connected regions whose union is a plurality of singly connected regions. In other words, the combining unit 702 combines the singly connected regions and assigns them in a unique manner to the combined groups of singly connected regions. The second subset of syntax elements indicating the combination information may be defined in the manner described above with reference to Figures 19 and 10, or in other ways. That is, the encoder's ability to indicate the subdivision by using a subset that is separated from the subset indicating the combination increases the encoder's freedom to adapt the subdivision of the sample array to the actual sample array content, thereby improving coding efficiency. The reconstructing unit 704 is configured to reconstruct the sample array from the data stream using the intermediate subdivision. As indicated above, the reconstructing unit may utilize the intermediate subdivision by adopting / predicting the coding parameters of the combined partner for the current singly connected region. Alternatively, the reconstructor 704 may apply a transformation or prediction process to the combined regions of the connected groups of simply connected regions.

[0200] FIG. 21 shows a possible encoder for generating a data stream that can be decoded by the decoder of FIG. 15. The encoder includes a subdivision / combination stage 750 and a data stream generator 752. The subdivision / combination stage determines an intermediate subdivision of an array of information samples representing a spatial sampling of a two-dimensional information signal, and two disjoint sets of simply connected regions whose union is a plurality of simply connected regions. The intermediate subdivision is defined by a first subset and a second subset of syntax elements that subdivide the array of information samples into a plurality of non-overlapping simply connected regions of different sizes by recursive multiple partitioning. Spatially adjacent simply connected regions of the plurality of simply connected regions are combined to obtain the intermediate subdivision, defined according to the first subset of syntax elements and according to a second subset of syntax elements that is separate from the first subset. The data stream generator 752 encodes the array of information samples into a data stream using the intermediate subdivision. Also, the subdivision / combining stage 750 inserts the first and second subsets into the data stream. As in FIG. 14 , the process of determining the first and second subsets and syntax elements generated by the data stream generator 752 may also be an iterative process. For example, the subdivision / combining stage 750 may predetermine an optimal subdivision, and the data stream generator 752 may determine a corresponding optimal set of syntax elements for encoding the sample sequence using the sample subdivision in conjunction with the subdivision / combining stage, and then configure the syntax elements described for combining to reduce the associated information overhead. Also, the encoding process need not end here. The subdivision / combining stage 750 may interface with the data stream generator 752 to configure subdivision and syntax element settings other than those previously determined by the data stream generator 752 to determine whether exploiting the beneficial properties of combining can improve the rate-distortion ratio.

[0201] As mentioned above, the embodiments described with reference to Figures 17 to 21 are a synthesis of the embodiments described above with reference to Figures 1 to 16, and therefore the elements of Figures 1 to 16 can be uniquely associated with the elements shown in Figures 17 to 21. For example, the extractor 102, together with the subdivision unit 104a and the combination unit 104b, performs the tasks performed by the extractor 600 of Figure 17. The subdivision unit manages the subdivision and the adjacency relationships between the individual simply connected regions. The combination unit 104b then manages the combination of the simply connected regions into groups and, if the currently decoded combination information indicates a combination event, identifies the correct coding parameters to be copied or used as a prediction for the current simply connected region. If entropy decoding is used to extract the data, the extractor 102 performs the actual extraction of the data from the data stream using the correct context. The remaining elements in Figure 2 are examples of the reconstructor 602. Of course, the reconstructor 602 may be embodied in forms other than those shown in Figure 2. For example, the reconstructor 602 may not use motion compensated prediction and / or intra prediction, as well as other possibilities. Also, as mentioned above, the simply connected regions mentioned in connection with the description of Figure 17 correspond to the prediction blocks mentioned above or any of the other subdivisions mentioned above, such as residual subdivisions or filter subdivisions, as already indicated above.

[0202] Comparing the encoder of Figure 19 with the example of Figure 1, the data generator 650 includes all elements except the data stream inserter 18, which corresponds to the inserter 652 of Figure 19. For the data generator 650 as well, other coding approaches than the hybrid coding approach shown in Figure 1 can be used.

[0203] Comparing the decoder of Figure 20 with the example shown in Figure 2, subdivision unit 104a and combination unit 104b correspond to subdivision unit 100 and combination unit 102 in Figure 20, respectively, and elements 106 and 114 correspond to reconstruction unit 704. Extraction unit 102 is commonly involved in the functions of all elements shown in Figure 20.

[0204] 21, the subdivision / combination stage 750 corresponds to the subdivision unit 28 and the combination unit 30, and the data stream generation unit 752 includes all other elements shown in FIG.

[0205] Although some aspects have been described in terms of an apparatus, it is clear that these aspects also serve as descriptions of corresponding methods, with blocks or devices corresponding to means or features of means. Similarly, aspects described in terms of means also serve as descriptions of corresponding blocks or items or functions of the corresponding apparatus. Some or all of the means may be implemented by (or using) hardware apparatuses such as microprocessors, programmable computers, electronic circuits, etc. In some embodiments, one or more of the most important means may be implemented by such apparatuses.

[0206] Signals encoded / compressed according to the present invention can be stored on a digital storage medium or transmitted over a transmission medium, such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0207] Embodiments of the present invention can be implemented in hardware or software, depending on specific implementation requirements. For example, they can be implemented using digital storage media, such as floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs, EPROMs, EEPROMs, flash memories, etc., on which electronically readable control signals are stored and which are (or can be used with) a programmable computer system to perform the respective methods. Thus, the digital storage media may be computer-readable.

[0208] Some embodiments of the present invention include a data storage medium storing electronically readable control signals that can be used with a programmable computer system to perform one of the methods described herein.

[0209] In general, embodiments of the present invention may be implemented as a computer program product including program code that, when executed on a computer, performs one of the methods. The program code may, for example, be stored on a machine-readable storage medium. In another embodiment, the computer program for performing one of the methods described herein is included and stored on a machine-readable storage medium.

[0210] In other words, an embodiment of the inventive method is, therefore, a computer program comprising a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0211] Therefore, a further embodiment of the inventive method is a data storage medium (or digital storage medium or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein.

[0212] A further embodiment of the inventive methods is therefore a data stream or series of signals representing a computer program for performing one of the methods described herein, for example a data stream or series of signals adapted to be transmitted over a data communications connection such as the Internet.

[0213] Other embodiments comprise a processing means, such as a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein. Another embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0214] In some embodiments, a programmable logic device (such as a field programmable gate array) may be used to implement some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may interface with a microprocessor to perform one of the methods described herein. In general, it is desirable for the methods to be performed by a hardware apparatus.

[0215] The above-described embodiments are merely examples for illustrating the basic nature of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. Therefore, it is intended to be limited only by the scope of the corresponding patent claims, and not by the specific details shown in the description of the embodiments herein.

Claims

1. an extractor configured to extract payload data from the data stream for each of a plurality of partitioned blocks of an array of information samples representing spatially sampled information of a color plane of an image, the image including a plurality of color planes; a reconstruction unit configured to reconstruct the image from the payload data of each block of the array of information samples by processing the payload data for each block in a manner defined by coding parameters associated with the block; A decoder comprising: The extraction unit further configured to extract from the data stream, for a given block, combined merge syntax elements for the given block, each combined merge syntax element indicating whether a merge operation for the given block is applied or not, and if so, identifying for each combined merge syntax element a particular block of two possible blocks for merge operation, wherein the particular block is at a predetermined relative position to the given block; and further configured to: adopt the coding parameters of the particular block as the coding parameters of the given block for the plurality of color planes of the image if one or more of the combined merge syntax elements is a merge operation for the given block; and extract from the data stream the coding parameters of the given block for each of the plurality of color planes of the image if the combined merge syntax element does not indicate a merge operation. decoder.

2. 1. A decoder for decoding a data stream in which an image is encoded, comprising: a subdivision unit configured to spatially subdivide the array of information samples representing a spatially sampled version of the image into a plurality of simply connected regions of different sizes in dependence on a first subset of syntax elements included in the data stream by recursively partitioning multiple times; a merging unit configured to merge a given simply connected region into a spatially adjacent simply connected region depending on a second subset of syntax elements in the data stream, the second subset being separate from the first subset, the second subset of syntax elements including combined merge syntax elements, each indicating whether a given simply connected region merging operation is applied or not, and if so, identifying for each combined merge syntax element a particular spatially adjacent simply connected region of two possible spatially adjacent simply connected regions for merging, wherein the particular spatially adjacent simply connected region is at a predetermined relative position to the given simply connected region; a reconstruction unit configured to reconstruct the sequence of information samples from the data stream and to adopt or predict coding parameters of a given singly-connected region from the particular adjacent singly-connected regions into which the given singly-connected region is merged for a plurality of color planes of the image, and to extract coding parameters of unmerged singly-connected regions from the data stream for each of the plurality of color planes; A decoder comprising:

3. The decoder of claim 1 , wherein the data stream includes depth information.

4. 2. The decoder of claim 1, wherein the data stream includes a sequence of images, each image including one array of luma samples and two arrays of chroma samples, and wherein a scaling factor of a horizontal spatial resolution of the array of chroma samples relative to the array of luma samples is different from a scaling factor of a vertical spatial resolution of the array of chroma samples.

5. The decoder of claim 1 , wherein the decoder is configured to independently decode the multiple color planes of the image.

6. extracting payload data from the data stream for each of a plurality of partitioned blocks of an array of information samples representing spatially sampled information of a color plane of an image, the image including a plurality of color planes; reconstructing the image from the payload data for each block of the array of information samples by processing, for each block, the payload data in a manner specified by coding parameters associated with the block; 1. A method for decoding, comprising: The extracting step includes: extracting from the data stream, for a given block, combined merge syntax elements for the given block, each combined merge syntax element indicating whether a merge operation for the given block applies, and if so, identifying for each combined merge syntax element a particular block of two possible blocks for merge operation, wherein the particular block is in a predetermined relative position to the given block; adopting the coding parameters of the particular block as the coding parameters of the given block for the plurality of color planes of the image if one or more of the combined merge syntax elements indicates a merging operation for the given block, and extracting the coding parameters of the given block for each of the plurality of color planes of the image from the data stream if the combined merge syntax elements do not indicate a merging operation; A method comprising:

7. 1. A method for decoding an image-encoded data stream, comprising: spatially subdividing an array of information samples representing a spatially sampled version of the image into a plurality of simply connected regions of different sizes depending on a first subset of syntax elements contained in the data stream by recursively partitioning multiple times; merging predetermined simply connected regions into spatially adjacent simply connected regions depending on a second subset of syntax elements in the data stream, the second subset being separate from the first subset, the second subset of syntax elements including combined merge syntax elements, each indicating whether a predetermined simply connected region merging operation is applicable, and if so, identifying, for each combined merge syntax element, a particular spatially adjacent simply connected region of two possible spatially adjacent simply connected regions for merging, wherein the particular spatially adjacent simply connected regions are at a predetermined relative position to the predetermined simply connected regions; reconstructing the array of information samples from the data stream and employing or predicting coding parameters of a given singly-connected region from the particular spatially adjacent singly-connected regions into which the given singly-connected region is merged for a plurality of color planes of the image, and extracting coding parameters of unmerged singly-connected regions from the data stream for each of the plurality of color planes; A method comprising:

8. The method of claim 6 , wherein the data stream includes depth information.

9. 7. The method of claim 6, wherein the data stream encodes a sequence of images, each image including one array of luma samples and two arrays of chroma samples, and wherein a horizontal spatial resolution scaling factor of the array of chroma samples relative to the array of luma samples is different from a vertical spatial resolution scaling factor.

10. The method of claim 6 , wherein the multiple color planes of the image are decoded independently.

11. 1. An encoder configured to encode, for each of a plurality of blocks into which an array of information samples representing spatially sampled information of a color plane of an image is subdivided, said array of information samples and coding parameters associated with each of said blocks into payload data for said each block, thereby defining how to reconstruct the image from the payload data for each of said blocks, and further configured to insert into the data stream, for a given block, combined merge syntax elements for the given block, each of the combined merge syntax elements indicating whether a merge operation for the given block is applied, and if so, specifying for each combined merge syntax element a particular block of two possible blocks, wherein the particular block is positioned relative to the given block, and if one or more of the combined merge syntax elements indicate a merge operation for the given block, coding parameters of the particular block are taken as coding parameters of the given block for multiple color planes of the image, and if the combined merge syntax elements do not indicate a merge operation, inserting into the data stream coding parameters of the given block for each of the multiple color planes of the image. Encoder.

12. 1. An encoder for generating a data stream in which an image is encoded, comprising: a subdivision / merge stage configured to determine, by recursive multiple partitioning, a first subset of syntax elements defining a spatial subdivision of an array of information samples representing a spatially sampled image into a plurality of simply connected regions of different sizes, and a second subset of syntax elements dissociated from the first subset defining merging of a given simply connected region into spatially adjacent simply connected regions, the second subset of syntax elements including combined merge syntax elements each indicating whether a given simply connected region merging operation is to be applied, and if so, identifying for each combined merge syntax element a particular spatially adjacent singly connected region of two spatially adjacent simply connected regions for merging, the particular spatially adjacent singly connected regions being at a predetermined relative position to the given simply connected regions; a data stream generator configured to encode the sequence of information samples into the data stream and to insert the first and second subsets of syntax elements into the data stream; Equipped with coding parameters of the given simply-connected region are indicated by a second subset of syntax elements to be adopted or predicted from the particular spatially adjacent simply-connected regions into which the given simply-connected region is merged for multiple color planes of the image; a data stream inserter configured to insert, for each of the plurality of color planes, coding parameters of unmerged simply connected regions into the data stream; Encoder.

13. The encoder of claim 11 , wherein the data stream includes depth information.

14. 12. The encoder of claim 11, wherein the data stream encodes a sequence of images, each image comprising an array of luma samples and two arrays of chroma samples, and wherein a scaling factor for horizontal spatial resolution of the array of chroma samples relative to the array of luma samples is different from a scaling factor for vertical spatial resolution.

15. The encoder of claim 11 , wherein the encoder is configured to independently encode the multiple color planes of the image.

16. 1. A method for defining how to reconstruct an image from payload data for each of a plurality of blocks into which an array of information samples representing spatially sampled information of a color plane of an image is subdivided, the method comprising: encoding the array of information samples and coding parameters associated with each of the blocks into payload data; for a given block, inserting into the data stream combined merge syntax elements for the given block, each of the combined merge syntax elements indicating whether a merge operation for the given block is applied, and if so, for each combined merge syntax element, identifying a particular block of two possible blocks, the particular block being in a predetermined relative position to the given block, and if one or more of the combined merge syntax elements indicate a merge operation for the given block, the coding parameters of the particular block are taken as coding parameters for the given block for multiple color planes of the image, and if the combined merge syntax elements do not indicate a merge operation, inserting into the data stream coding parameters of the given block for each of the multiple color planes of the image. A method comprising:

17. 1. A method for generating an image-encoded data stream, comprising: determining a first subset of syntax elements defining a spatial subdivision of an array of information samples representing a spatially sampled image into a plurality of simply connected regions of different sizes by recursively partitioning multiple times, and a second subset of syntax elements dissociated from the first subset defining merging of a given simply connected region into spatially adjacent simply connected regions, the second subset of syntax elements including combined merge syntax elements each indicating whether a merging operation of a given simply connected region is to be applied, and if so, for each combined merge syntax element, identifying a particular spatially adjacent singly connected region of two spatially adjacent simply connected regions for merging, wherein the particular spatially adjacent singly connected regions are at a predetermined relative position to the given simply connected regions; encoding the sequence of information samples into the data stream and inserting the first and second subsets of syntax elements into the data stream; Including, coding parameters of the given simply-connected region are indicated by a second subset of syntax elements to be adopted or predicted from the particular spatially adjacent simply-connected regions into which the given simply-connected region is merged for multiple color planes of the image; a data stream inserter configured to insert, for each of the plurality of color planes, coding parameters of unmerged simply connected regions into the data stream; method.

18. The method of claim 16 , wherein the data stream includes depth information.

19. 17. The method of claim 16, wherein the data stream encodes a sequence of images, each image including an array of luma samples and two arrays of chroma samples, and wherein a horizontal spatial resolution scaling factor of the array of chroma samples relative to the array of luma samples is different from a vertical spatial resolution scaling factor.

20. The method of claim 16 , wherein the color planes of the image are coded independently.

21. 1. A method for decoding a data stream encoding a sequence of information samples, comprising:

20. A method comprising the steps of receiving and decoding said data stream, wherein said sequence of information samples has been coded according to the method of claim 16.

22. 1. A method for decoding a data stream encoding an image, comprising:

20. A method comprising receiving and decoding said data stream in which said image is encoded according to the method of claim 17.

23. 1. A method for decoding a data stream in which an array of information samples representing spatially sampled information of a color plane of an image is encoded, the method comprising: receiving and decoding the data stream, the data stream including payload data for each of a plurality of blocks into which the array of information samples is subdivided and coding parameters associated with each of the blocks, to reconstruct the image from the payload data for each of the blocks; the data stream further includes combined merge syntax elements for predetermined blocks, each of the combined merge syntax elements indicating whether a merging operation for the predetermined block is applied, and if so, specifying for each combined merge syntax element a particular block of two possible blocks, the particular block being in a predetermined relative position with respect to the predetermined block, and wherein coding parameters of the particular block are adopted as coding parameters of the predetermined block for multiple color planes of the image; for each given block for which the combined merge syntax element does not indicate a merge operation, encoding coding parameters for the given block for each of the plurality of color planes of the image; method.

24. 1. A method for decoding an image-encoded data stream, comprising: receiving and decoding the data stream, the data stream comprising: a first subset of syntax elements defining a spatial subdivision of an array of information samples representing a spatially sampled version of the image into a plurality of simply connected regions of different sizes by recursively partitioning multiple times; a second subset of syntax elements, separate from the first subset, that define merging of a given simply connected region into spatially adjacent simply connected regions, the second subset of syntax elements including combined merge syntax elements, each of which indicates whether a given simply connected region merging operation is applied, and if so, for each combined merge syntax element, identifies a particular spatially adjacent singly connected region of two spatially adjacent simply connected regions for merging, the particular spatially adjacent singly connected regions being at a predetermined relative position to the given simply connected regions; Including, the sequence of information samples is encoded into the data stream; coding parameters of the given simply connected region are indicated by a second subset of syntax elements to be adopted or predicted from the particular adjacent simply connected regions into which the given simply connected region is merged for multiple color planes of the image; the data stream includes, for each of the plurality of color planes, coding parameters of dismerged simply connected regions; method.

25. The method of claim 21 , wherein the data stream includes depth information.

26. 22. The method of claim 21, wherein the data stream encodes a sequence of images, each image including one array of luma samples and two arrays of chroma samples, and wherein a horizontal spatial resolution scaling factor of the array of chroma samples relative to the array of luma samples is different from a vertical spatial resolution scaling factor.

27. 22. The method of claim 21, wherein the multiple color planes of the image are decoded independently.

Images