Inheritance in sample array multi-tree subdivision

Abstract

To provide a coding system for coding an array of information samples each representing a spatially sampled information signal.SOLUTION: A method improves adjustment between encoding complexity and achievable rate / distortion ratio and enhances rate / distortion ratio, not only by using multi-tree subdivision in order to subdivide a continuous area, namely, a sample array into leaf regions but also by using intermediate regions in order to share coding parameters among the corresponding arrayed leaf blocks. By this method, coding procedures locally performed in tiles (leaf regions) can be individually associated with the coding parameters, and thereby there is no need to explicitly transmit separately all of the coding parameters for each leaf region. In other words, similarities can effectively be used by using the multi-tree subdivision.SELECTED DRAWING: Figure 12A

Description

[Technical Field]

[0001] The present invention relates to a coding scheme for an array of information samples representing a spatially sampled information signal, such as a moving or still image. [Background technology]

[0002] Typically, in image and video coding, an image, or a set of specific sample arrays within an image, is decomposed into blocks and associated with specific coding parameters. An image typically consists of multiple sample arrays. Furthermore, an image may be associated with additional auxiliary sample arrays that specify, for example, transparency information or depth maps. The sample arrays (including the auxiliary sample arrays) of an image can be grouped into one or more so-called plane groups, each consisting of one or more sample arrays. The plane groups of an image can be coded independently or, if the image is associated with multiple plane groups, can be coded by prediction from other plane groups of the same image. Typically, each plane group is decomposed into blocks. The blocks (or the corresponding blocks of sample arrays) are predicted either by inter-picture prediction or intra-picture prediction. Blocks can be of various sizes and can be either square or rectangular. The partitioning of an image into blocks can be fixed by syntax or (at least in part) signaled within the bitstream. Often, syntax elements are transmitted that signal the subdivision of blocks of a given size. These syntax elements specify whether and how a block is subdivided into smaller blocks and associated with coding parameters, e.g., for prediction purposes. For every sample of a block (or the corresponding block of a sample array), the decoding of the associated coding parameters is specified in a specific way. For example, all samples in a block are predicted using the same set of prediction parameters, such as reference indices (identifying a reference image in a set of previously coded images), motion parameters (specifying the amount of block motion between the reference image and the current image), and parameters for specifying interpolation filters, intra-prediction modes, etc. Motion parameters can be represented by displacement vectors with horizontal and vertical components, or by higher-order motion parameters, such as affine motion parameters with six components. It is also possible to associate multiple sets of specific prediction parameters (e.g., reference indices, motion parameters, etc.) with a single block.In this case, for each set of these specific prediction parameters, a single intermediate prediction signal for the block (or the corresponding block in the sample array) is generated, and the final prediction signal is formed by combining the intermediate prediction signals, including superposition. The corresponding weighting parameters, possibly with a constant offset (added to the weighted sum), can be fixed for either the image, the reference image, or the set of reference images, or can be included in the set of prediction parameters for the corresponding block. The difference between the original block (or the corresponding block in the sample array) and the prediction signal, also known as the residual signal, is typically transformed and quantized. A two-dimensional transform is often applied to the residual signal (or the corresponding sample array of the residual block). For transform coding, a block (or the corresponding block in the sample array) for which a specific set of prediction parameters is used can be further divided before applying the transform. The transform block can be the same size as the block used for prediction, or smaller. A transform block can also contain multiple blocks used for prediction. Each transform block can be of various sizes, resulting in a square or rectangular block. After transformation, the resulting transform coefficients are quantized to obtain so-called transform coefficient levels. The transform coefficient levels and prediction parameters, and subdivision information, if present, are entropy coded. Summary of the Invention [Problem to be solved by the invention]

[0003] In image and video coding standards, the syntax provides very limited possibilities for subdividing an image (or a group of planes) into blocks. It is usually possible to specify only whether and, possibly, how a block of a given size can be subdivided into smaller blocks. For example, in H.264, the maximum block size is 16x16. A 16x16 block is also called a macroblock, and each image is partitioned into macroblocks in the first step. For each 16x16 macroblock, it is possible to signal whether it will be coded as a 16x16 block, two 16x8 blocks, two 8x16 blocks, or four 8x8 blocks. If a 16x16 block is subdivided into four 8x8 blocks, each of these 8x8 blocks can be coded as one 8x8 block, two 8x4 blocks, two 4x8 blocks, or four 4x4 blocks. In current image and video coding standards, reducing the possibilities for specifying partitioning into blocks has the advantage of keeping the side information rate for signaling subdivision information small, but the disadvantage, as explained below, is that the bit rate required for transmitting prediction parameters for a block can be large. The side information rate for signaling prediction information usually accounts for a significant portion of the overall bit rate for a block. Therefore, reducing this side information can improve coding efficiency, which can be achieved, for example, by using larger block sizes. Images in real images and video sequences consist of objects of arbitrary shapes with specific properties. For example, these objects or object parts are characterized by a specific texture or a specific motion. The same set of prediction parameters is usually applied to these objects or object parts. However, object boundaries usually do not coincide with the boundaries of the possible blocks of a larger prediction block (e.g., a 16x16 macroblock in H.264). The encoder usually makes a subdivision decision (among a limited number of possibilities) that minimizes a certain rate / distortion loss.This can result in a large number of small blocks for an object of arbitrary shape. Each of these small blocks is associated with a set of prediction parameters that need to be transmitted, and the associated information rate can become a significant fraction of the overall bit rate. However, since some of the small blocks still represent the same object or a region of the object, the prediction parameters of the resulting large number of blocks will be identical or very similar.

[0004] That is, subdivision or tiling of an image into smaller parts or tiles or blocks significantly impacts the coding efficiency and coding complexity. As outlined above, the more subdivision of an image into smaller blocks, the more spatial granularity the coding parameter settings can be, and the better these coding parameters adapt to the image / video material. On the other hand, the more granular the coding parameter settings, the greater the burden of the amount of accompanying information needed to inform the decoder of the desired settings. It should also be noted that allowing the encoder to (further) spatially subdivide an image / video into blocks significantly increases the amount of possible coding parameter settings, and as a result, it generally becomes more difficult to find the best rate / distortion compromise for the coding parameter settings. [Means for solving the problem]

[0005] The present invention provides a coding scheme for encoding an array of information samples representing a spatially sampled information signal, such as, but not limited to, moving images or still images, which allows for an improved trade-off between encoding complexity and achievable rate / distortion ratio and / or allows for improved rate / distortion ratios to be achieved.

[0006] This object is achieved by a decoder according to claim 1, an encoder according to claim 10, a method according to claim 11, a computer program according to claim 12 and a data stream according to claim 13.

[0007] The idea underlying the present invention is as follows: not only is a multi-tree subdivision used to subdivide contiguous regions, i.e., sample arrays, into leaf regions, but intermediate regions are also used for sharing coding parameters between corresponding arrayed leaf blocks, thereby achieving a better compromise between encoding complexity and achievable rate / distortion ratio and / or improving the rate / distortion ratio. In this way, the coding steps performed locally in a tile (leaf region) can be individually associated with coding parameters, eliminating the need to explicitly transmit all coding parameters separately for each leaf region. In other words, the use of multi-tree subdivision allows for an effective exploitation of similarities.

[0008] In an embodiment, an array of information samples representing a spatially sampled information signal is first spatially divided into tree root regions and then subdivided according to multi-tree subdivision information extracted from the data stream. By performing multiple recursive partitioning on subsets of the tree root regions, at least a subset of the tree root regions is subdivided into smaller simply connected regions of various sizes. To be able to find a desirable compromise between too fine and too coarse subdivision in terms of rate / distortion with a reasonable encoding complexity, the size of the largest region of the tree root region into which the array of information samples is spatially divided is included in the data stream and extracted from the data stream at the decoding side. Therefore, the decoder may include an extraction unit capable of extracting the maximum region size and multi-tree subdivision information from the data stream, a subdivision unit capable of spatially dividing an array of information samples representing a spatially sampled information signal into tree root regions of the maximum region size, subdividing according to the multi-tree subdivision information, and subdividing at least a subset of the tree root regions into smaller simply connected regions of various sizes by recursively partitioning the subsets of the tree root regions multiple times, and a reconstruction unit capable of reconstructing the array of information samples from the data stream using the subdivision into smaller simply connected regions.

[0009] In an embodiment, the data stream also includes the maximum hierarchical level at which a subset of the root region of a tree is subject to multiple recursive partitioning, making it easier to convey multi-tree subdivision information and requiring fewer bits to encode.

[0010] Furthermore, the reconstruction unit may be configured to perform one or more of the following with a granularity depending on the intermediate subdivision: determining a prediction mode to use from at least intra prediction modes and inter prediction modes; converting from the spectral domain to the spatial domain; performing inter prediction and / or setting parameters for inter prediction; performing intra prediction and / or setting parameters for intra prediction.

[0011] Additionally, the extractor may be configured to extract syntax elements associated with leaf regions of the partitioned treeblock from the data stream in a depth-first traversal order, which allows the extractor to more likely utilize statistics of the syntax elements of adjacent encoded leaf regions than would be possible using a width-first traversal order.

[0012] In another embodiment, a further subdivision unit is used to subdivide at least a subset of the smaller simply connected regions into even smaller simply connected regions according to the further multi-tree subdivision information. The first stage subdivision may be used by the reconstruction unit to perform prediction of the domain of information samples, and the second stage subdivision may be used by the reconstruction unit to perform retransformation from the spectral domain to the spatial domain. Defining the residual subdivision at a lower level relative to the prediction subdivision reduces the overall bit consumption for coding the subdivision. On the other hand, the limitations and freedom of the lower definition of the residual subdivision have a slight negative impact on coding efficiency. This is because in most cases, parts of an image with similar motion compensation parameters are larger than parts with similar spectral characteristics.

[0013] In yet another embodiment, a further maximum region size is included in the data stream, which defines the size of the tree-root subregions into which the smaller simply-connected regions are first divided before at least a subset of the tree-root subregions are further subdivided into smaller simply-connected regions according to the further multi-tree subdivision information. This allows for a unique setting of the maximum region size for prediction subdivision, while allowing for an improved rate / distortion trade-off through residual subdivision.

[0014] In yet another embodiment of the present invention, a first subset of syntax elements, separated from a second subset of syntax elements forming the multi-tree subdivision information, is included in the data stream. A decoder-side combiner can combine smaller, spatially adjacent simply connected regions of the multi-tree subdivisions according to the first subset of syntax elements to obtain an intermediate subdivision of the sample array. The reconstructor can be configured to reconstruct the sample array using the intermediate subdivision. This allows the encoder to perform subdivisions that are effective for the spatial distribution of the characteristics of the information sample array, making it easier to find an optimal rate / distortion compromise. For example, if the size of the largest region is large, the tree root region is likely to be large, making the multi-tree subdivision information complex. On the other hand, if the size of the largest region is small, it is likely that neighboring tree root regions are related to information content with similar characteristics and are also processed together. Combining eliminates the gap between the above two extreme cases, thereby enabling subdivision with a granularity close to the optimal level. From the encoder's point of view, the combined syntax element simplifies the encoding procedure or reduces the computational complexity, since if the encoder mistakenly uses too fine a subdivision, it corrects this mistake by subsequently setting the combined syntax element, with or without adaptation of a small fraction of the syntax elements that were set before the setting of the combined syntax element.

[0015] In yet another embodiment, the maximum region size and multi-tree subdivision information is used for residual subdivision instead of prediction subdivision.

[0016] When dealing with singly connected regions of a quadtree subdivision of an array of information samples representing a spatially sampled information signal, a depth-order traversal order is used in accordance with an embodiment, rather than a width-order traversal order. Using a depth-order traversal order increases the probability that each singly connected region has neighboring singly connected regions that have been traversed, and ensures that information about these neighboring singly connected regions is available when reconstructing each current singly connected region.

[0017] If the array of information samples is first partitioned into a regular array of tree root regions of zeroth-order hierarchical size, and then at least a subset of the tree root regions is subdivided into smaller simply connected regions of different sizes, the reconstructor may traverse the tree root regions using a zigzag scan, and for each partitioned tree root region, process the simply connected leaf regions in a depth-first traversal order, and then proceed to the next tree root region in a zigzag scan order. Furthermore, according to the depth-first traversal order, simply connected leaf regions at the same hierarchical level may be traversed in a zigzag scan order. In this way, a state in which the probability of having adjacent simply connected leaf regions is increased is maintained.

[0018] In an embodiment, flags associated with nodes of the multi-tree structure are arranged sequentially in depth-order traversal order, but the sequential encoding of flags uses a probability estimation context that is the same for flags associated with nodes of the multi-tree structure located at the same hierarchical level of the multi-tree structure, but different for nodes of the multi-tree structure located at different hierarchical levels of the multi-tree structure, thereby enabling an improved compromise between the amount of context provided and adaptation to the statistics of the actual symbols of the flags.

[0019] In embodiments, different contexts of probability estimation for a given flag are used depending on flags corresponding to regions in the root region of the tree that precede the given flag in depth-first traversal order and that are in a predetermined relative position to the region to which the given flag corresponds. Similar to the idea underlying the described aspect, the use of a depth-first traversal order ensures with a high probability that the encoded flags will include flags corresponding to regions that are adjacent to the region to which the given flag corresponds. This knowledge is exploited to improve the adaptability of the given flag to the context to which it is used.

[0020] The flags used to set the context for a given flag may correspond to the region located above and / or to the left of the region to which the given flag corresponds. Furthermore, the flags used to select the context may be limited to flags that belong to the same hierarchical level as the node with which the given flag is associated.

[0021] In an embodiment, the coded signaling information includes an indication of the highest hierarchical level and a set of flags associated with nodes of the multi-tree structure other than the highest hierarchical level. Each flag specifies whether the associated node is an intermediate node or a child node. The set of flags is sequentially decoded from the data stream in a depth-order or breadth-order traversal order, skipping nodes at the highest hierarchical level and automatically designating the same leaf node. This results in a reduced coding rate.

[0022] In other embodiments, the highest hierarchical level may be indicated in the coded signaling of the multi-tree structure, thereby restricting the presence of flags to hierarchical levels other than the highest hierarchical level, thereby preventing further segmentation of blocks at the highest hierarchical level.

[0023] If a spatial multi-tree subdivision is part of a secondary subdivision of a leaf node and the root region of the tree of the primary multi-tree subdivision is not partitioned, the context used to encode the flags of the secondary subdivision may be selected such that the context is the same for flags associated with regions of the same size.

[0024] In another embodiment, desired combinations or groupings of singly connected regions resulting from the subdivision of the sequence of information samples are coded to reduce the amount of data. For this purpose, a predetermined relative positional relationship is defined for the singly connected regions. This allows for the identification of a singly connected region among multiple singly connected regions that is located at a predetermined relative position relative to the given singly connected region. That is, if the number is zero, no combination index for the given singly connected region may be present in the data stream. Furthermore, if the number of singly connected regions located at a predetermined relative position relative to the given singly connected region is one, the coding parameters of the singly connected region may be employed or used to predict the coding parameters for the given singly connected region, and no further syntax elements are required. Otherwise, if the number of singly connected regions located at a predetermined relative position relative to the given singly connected region is greater than one, the introduction of further syntax elements may be suppressed, even if the coding parameters associated with these identified singly connected regions are identical to each other.

[0025] In an embodiment, if the coding parameters of adjacent singly connected regions are not equal to each other, a proper subset of the number of singly connected regions at a predetermined relative position to a given singly connected region may be identified by a reference neighbor identifier, and this proper subset is used when adopting the coding parameters or predicting the coding parameters of a given singly connected region.

[0026] In yet another embodiment, a region of samples representing a spatial sampling of a two-dimensional information signal is spatially sub-divided by multiple recursive partitioning into multiple simply connected regions of different sizes according to a first subset of syntax elements contained in the data stream. Subsequently, spatially adjacent simply connected regions are combined according to a second subset of syntax elements in the data stream that is separated from the first subset, resulting in an intermediate sub-division of the sequence of samples into disjoint sets of simply connected regions whose union is multiple simply connected regions. The intermediate sub-division is used to reconstruct the sequence of samples from the data stream. This reduces the importance of optimizing the sub-division, since overly dense sub-divisions can be corrected by subsequent merging. Furthermore, the combination of sub-division and merging allows for intermediate sub-divisions that would not be possible using multiple recursive partitioning alone, and the combination of sub-division and merging using a set of separated syntax elements improves the fit of the effective or intermediate sub-division to the actual content of the two-dimensional information signal. The additional overhead incurred by the additional subset of syntax elements to indicate the details of the merging is negligible compared to the benefits.

[0027] Preferred embodiments of the present invention will now be described with reference to the following figures: [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a configuration diagram illustrating an encoder according to an embodiment of the present application; [Figure 2] FIG. 2 is a block diagram showing a decoder according to an embodiment of the present invention; [Figure 3A] 1 is a schematic diagram illustrating an example of a quadtree subdivision, showing the first hierarchical level. [Figure 3B] FIG. 1 is a schematic diagram illustrating an example of a quadtree subdivision, showing the second hierarchical level. [Figure 3C] FIG. 1 is a schematic diagram illustrating an example of a quadtree subdivision, showing the third hierarchical level. [Figure 4]FIG. 4 is a schematic diagram illustrating a tree structure of a subdivision of the quadtree of FIGS. 3A to 3C according to an embodiment. [Figure 5A] FIG. 3D is a schematic diagram illustrating a subdivision of the quadtree of FIGS. 3A to 3C. [Figure 5B] FIG. 4 is a schematic diagram showing the indexed tree structure of the individual leaf blocks of FIGS. 3A to 3C. [Figure 6A] 5A-5C are schematic diagrams illustrating sequences of binary strings or flags representing subdivisions of the tree structure of FIG. 4 and the quadtrees of FIGS. 3A-3C according to different embodiments. [Figure 6B] 5A-5C are schematic diagrams illustrating sequences of binary strings or flags representing subdivisions of the tree structure of FIG. 4 and the quadtrees of FIGS. 3A-3C according to different embodiments. [Figure 7] 10 is a flowchart illustrating steps performed by a data stream extraction unit according to an embodiment. [Figure 8] 10 is a flowchart illustrating the functioning of a data stream extraction unit according to another embodiment; [Figure 9A] FIG. 1 is a schematic diagram illustrating a quadtree subdivision according to an embodiment, showing candidate blocks adjacent to a given block highlighted; [Figure 9B] FIG. 1 is a schematic diagram illustrating a quadtree subdivision according to an embodiment, showing candidate blocks adjacent to a given block highlighted; [Figure 10] 10 is a flowchart illustrating the functioning of a data stream extraction unit according to another embodiment; [Figure 11] 1 is a schematic diagram illustrating the construction of an image from planes and plane groups and illustrating encoding using inter-plane adaptation / prediction according to an embodiment. FIG. [Figure 12A] FIG. 2 is a schematic diagram showing a subtree structure to explain an inheritance scheme according to an embodiment. [Figure 12B] FIG. 2 is a schematic diagram illustrating subdivisions corresponding to subtree structures to explain an inheritance scheme according to an embodiment. [Figure 12C]FIG. 2 is a schematic diagram illustrating a subtree structure to explain the inheritance scheme of adoption and prediction according to an embodiment. [Figure 12D] FIG. 2 is a schematic diagram illustrating a subtree structure to explain the inheritance scheme of adoption and prediction according to an embodiment. [Figure 13] 10 is a flowchart illustrating steps performed by an encoder to implement an inheritance scheme according to an embodiment. [Figure 14] (A) is a diagram showing a first-order subdivision to illustrate the possibility of implementing an inheritance scheme in relation to inter-plane prediction according to an embodiment, and (B) is a diagram showing a lower-order subdivision to illustrate the possibility of implementing an inheritance scheme in relation to inter-plane prediction according to an embodiment. [Figure 15] FIG. 1 is a block diagram illustrating the process of decoding in relation to an inheritance scheme, according to an embodiment. [Figure 16] 1 is a schematic diagram illustrating the order of traversal of sub-regions of a multi-tree subdivision for a sub-region subject to intra prediction according to an embodiment; FIG. [Figure 17A] 5A and 5B are schematic diagrams illustrating different subdivision possibilities in another embodiment; [Figure 17B] 5A and 5B are schematic diagrams illustrating different subdivision possibilities in another embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0029] In the following description of each figure, elements that appear in multiple figures are designated by common reference numerals, and redundant descriptions of these elements are omitted. In other words, a description of an element that appears in one figure also applies to other figures in which that element appears, to the extent that the description is relevant.

[0030] Furthermore, the following description begins with the encoder and decoder embodiments described in Figures 1 through 11. The embodiments described for these figures combine many aspects of the present application, but are also effective when implemented separately within a coding scheme. Therefore, for the following figures, embodiments utilizing the above aspects using each of these embodiments separately will be briefly described, and will represent different abstractions of the embodiments described for Figures 1 and 11.

[0031] FIG. 1 illustrates an encoder according to an embodiment of the present invention. The encoder 10 of FIG. 1 includes a prediction unit 12, a residual precoder 14, a residual reconstruction unit 16, a data stream insertion unit 18, and a block segmentation unit 20. The encoder 10 is configured to encode a temporally and spatially sampled information signal into a data stream 22. The temporally and spatially sampled information signal may be, for example, a video, i.e., a series of images. Each image represents an array of image samples. Other examples of temporally and spatial information signals include depth images captured by a time-of-light camera. It should be noted that the spatially sampled information signal may contain multiple arrays per frame or timestamp. For example, a color video may contain one array of luma samples per frame along with two arrays of chroma samples. The temporal sampling rates of different components of the information signal, i.e., luma and chroma, may differ, as may the spatial resolution. The video may also be accompanied by additional spatially sampled information, such as depth or transparency information. However, in the following description, in order to facilitate understanding of the main contents of the present application, we will first focus on the processing of one of these arrays, and then discuss the handling of multiple planes.

[0032] 1 is configured to create a data stream 22 such that the syntax elements of the data stream 22 describe images at a granularity lying between the entire image and individual image samples. To this end, the segmentation unit 20 is configured to subdivide each image 24 into singly connected regions 26 of various sizes. In the following, these regions will be referred to simply as blocks or subregions 26.

[0033] As will be explained in more detail below, the segmentation unit 20 uses multi-tree subdivision to subdivide the image 24 into blocks 26 of various sizes. More precisely, in the particular embodiment outlined below with respect to Figures 1 to 11, it mostly uses quad-tree subdivision. The segmentation unit 20 may also comprise an internally connected subdivision unit 28, as will be explained in more detail below. The subdivision unit 28 subdivides the image 24 into said blocks 26. The subdivision unit 28 is connected to a combiner unit 30. The combiner unit 30 can combine groups of these blocks 26 to obtain an effective subdivision or granularity lying between the non-subdivision of the image 24 and the subdivision defined by the subdivision unit 28.

[0034] 1, the predictor 12, the residual precoder 14, the residual reconstructor 16, and the data stream inserter 18 perform the subdivision of the image defined by the partitioner 20. For example, as described in more detail below, the predictor 12 uses the prediction subdivision defined by the partitioner 20 to: set prediction parameters corresponding to each subregion according to a selected prediction mode, and make a decision for each individual subregion of the prediction subdivision as to whether the subregion should be subjected to intra-picture or inter-picture prediction;

[0035] The residual precoder 14 may then use the subdivision of the residual of the image 24 to encode the residual of the prediction of the image 24 provided by the predictor 12. When the residual reconstructor 16 reconstructs the residual from the syntax elements output by the residual precoder 14, the residual reconstructor 16 also performs the subdivision of the residual. The data stream inserter 18 may use the subdivision of the prediction and the residual to: determine the insertion order and adjacency of the syntax elements output by the residual precoder 14 and predictor 12 into the data stream 22, for example using entropy encoding, for insertion of the syntax elements.

[0036] As shown in FIG. 1, the encoder 10 includes an input 32 through which an original information signal is input. A subtractor 34, a residual precoder 14, and a data stream inserter 18 are serially connected, in that order, between the input 32 and the output of the data stream inserter 18, from which the encoded data stream 22 is output. The subtractor 34 and the residual precoder 14 are part of a prediction loop. The prediction loop is closed by a residual construction unit 16, an adder 36, and a predictor 12, which are serially connected in the following order between the output of the residual precoder 14 and the inverting input of the subtractor 34. The output of the predictor 12 is also connected to another input of the adder 36. The predictor 12 may also have another input directly connected to the input 32 and connected to the output of the adder 36 via an optional in-loop filter 38. Furthermore, since the prediction unit 12 generates accompanying information during operation, an output of the prediction unit 12 is also connected to a data stream insertion unit 18. Similarly, the splitting unit 20 has an output connected to another input of the data stream insertion unit 18.

[0037] Having described the structure of encoder 10, the mode of operation will now be described in more detail.

[0038] As explained above, the segmentation unit 20 determines for each image 24 how to sub-divide it into sub-regions 26. According to the sub-division of the image 24 used for prediction, the prediction unit 12 determines for each sub-region corresponding to this sub-division how to predict the respective sub-region. The prediction unit 12 outputs a prediction of the sub-region to an inverting input of the subtractor 34 and to another input of the adder 36, and outputs prediction information to the data stream insertion unit 18 that reflects how the prediction unit 12 obtained this prediction from the encoded part of the video.

[0039] At the output of the subtractor 34, the residual precoder 14 processes this prediction residual according to the residual subdivision indicated by the divider 20 to obtain a prediction residual. As will be explained in more detail below with reference to Figures 3 to 10, the residual subdivision of the image 24 used by the residual precoder 14 may be related to the prediction subdivision used by the predictor 12, such that each prediction subregion is adopted as a residual subregion or is further subdivided into smaller residual subregions. There may also be completely independent prediction and residual subdivisions.

[0040] The residual precoder 14 performs a two-dimensional transformation from the spatial domain to the spectral domain on each residual sub-region. This is followed by or essentially involves quantization of the resulting transform coefficients of the resulting transform block, whereby distortion occurs as a result of quantization noise. For example, the data stream inserter 18 may losslessly encode syntax elements describing these transform coefficients into the data stream 22, e.g., using entropy encoding.

[0041] The residual reconstruction unit 16 then retransforms the transform coefficients into a residual signal using requantization with retransform. To obtain the prediction residual, the residual signal is combined in an adder 36 with the prediction used by the subtractor 34. As a result, at the output of the adder 36, a reconstructed part or sub-region of the current image is obtained. The predictor 12 may also directly use the reconstructed image sub-region for intra prediction, since it predicts a particular prediction sub-region by estimation from neighboring reconstructed prediction sub-regions. However, intra prediction performed in the spectral domain by directly predicting the spectrum of the current sub-region from the spectra of neighboring sub-regions is also theoretically possible.

[0042] For inter-plane prediction, predictor 12 may use a filtered version of the encoded and reconstructed image by an optional in-loop filter 38. For example, in-loop filter 38 may comprise a deblocking filter and / or an adaptive filter having a transfer function that benefits from shaping the quantization noise as described above.

[0043] The predictor 12 selects prediction parameters that indicate how to perform a prediction of a particular prediction subregion relative to the original examples in the image 24. As explained in more detail below, the prediction parameters may include an indication of a prediction mode, such as intra-picture prediction or inter-picture prediction, for each prediction subregion. In the case of intra-picture prediction, the prediction parameters may include an indication of the angle along which the boundaries of the prediction subregion within which intra-prediction is performed primarily extend. In the case of inter-picture prediction, the prediction parameters may include motion vectors, video indices, and / or higher-order motion transformation parameters. In the case of both intra-picture and / or inter-picture prediction, the prediction parameters may include optional filter information for filtering the reconstructed image examples on which the current prediction subregion is predicted.

[0044] As will be explained in more detail below, the subdivision defined by the partitioning unit 20 significantly affects the rate-distortion ratio achievable by the residual precoder 14, the predictor 12, and the data stream insertion unit 18. Finer subdivisions may result in better predictions obtained by the predictor 12, and may result in a smaller residual signal encoded by the residual precoder 14 that can be coded using fewer bits, but they require a significantly higher coding rate for the prediction parameters 40 output by the predictor 12 and inserted into the data stream 22. The opposite is true for coarser subdivisions. The above considerations also apply equally to residual subdivisions. Transforming an image with a finer granularity for individual transform blocks reduces the computational complexity of the transform and increases the spatial resolution of the resulting transform. That is, smaller residual subregions result in more consistent spectral distributions of the contents of the individual residual subregions. However, the spectral resolution decreases, and the ratio between significant coefficients and insignificant coefficients, i.e., coefficients quantized to zero, deteriorates. In other words, the granularity of the transformation must be locally adapted to the image content. Furthermore, apart from the positive effects of a finer granularity, a finer granularity usually increases the amount of accompanying information needed to indicate the selected subdivision to the decoder. As will be explained in more detail below, the following embodiments describe an encoder 10 that is able to adapt the subdivision very effectively to the content of the information signal to be encoded, and that is able to communicate the subdivision information to be used on the decoding side by instructing a data stream insertion unit 18 to insert subdivision information into the coded data stream 22.

[0045] However, before defining the subdivision of the divider 20 in detail, a decoder according to an embodiment of the present application will be described in detail with reference to FIG.

[0046] The decoder of Figure 2 is generally designated by the reference numeral 100 and comprises an extraction unit 102, a segmentation unit 104, a residual reconstruction unit 106, an adder 108, a prediction unit 110, an optional in-loop filter 112, and an optional post-filter 114. The extraction unit 102 receives an encoded data stream at an input 116 of the decoder 100 and extracts, from subdivision information 118 of the encoded data stream, prediction parameters 120 and residual data 122 that the extraction unit 102 outputs to the image segmentation unit 104, the prediction unit 110, and the residual reconstruction unit 106, respectively. An output of the residual reconstruction unit 106 is connected to a first input of the adder 108. Another input and an output of the adder 108 are connected to a prediction loop. The prediction loop includes an optional in-loop filter 112 and a prediction unit 110 connected in series in this order, with a bypass path directly from the output of the adder 108 to the prediction unit 110. 1 above, one for intra-picture prediction and the other for inter-picture prediction. Either the output of the adder 108 or the output of the in-loop filter 112 may be connected to an output 124 of the decoder 100, at which the reconstructed information signal is output, e.g., to a reproduction device. An optional post-filter 114 may be connected in the path to the output 124 in order to improve the visual impression of the reconstructed signal at the output 124.

[0047] Generally, the residual reconstruction unit 106, the adder 108, and the prediction unit 110 function similarly to the elements 16, 36, and 12 in FIG. 1 . In other words, they perform operations similar to those of the above-mentioned elements in FIG. 1 . To this end, the residual reconstruction unit 106 and the prediction unit 110 are controlled by the prediction parameters 120 and the subdivision indicated by the image division unit 104, respectively, in accordance with the subdivision information 118 from the extraction unit 102. This aims to predict a small region of prediction in the same way as the prediction unit 12 did or decided to do, and to retransform the received transform coefficients at the same granularity as the residual precoder 14 performed the transformation. Next, the image division unit 104 reorganizes the subdivision selected by the division unit 20 in FIG. 1 in a synchronized manner by referring to the subdivision information 118. The extraction unit may then use the subdivision information to control data extraction, such as context selection, adjacency determination, probability estimation, and data stream syntax analysis.

[0048] Several variations of the above-described embodiment are possible. Some of the subdivisions performed by the subdivision unit 28 and the combinations performed by the combination unit 30 are described in detail below. Others are described with reference to Figures 12 through 16. Where appropriate, all of these variations may be applied individually or in subsets to the descriptions provided above for Figures 1 and 2, respectively. For example, the division units 20 and 104 may not only determine prediction subdivisions and residual subdivisions for each image. Instead, the subdivisions of the optional in-loop filters 38 and 112 may be determined either independently or dependently on other subdivisions for prediction or residual coding. Furthermore, the subdivision determinations by these elements may not be performed on a frame-by-frame basis. Instead, the subdivisions determined for a particular frame may be reused or adopted for a certain number of subsequent frames before transitioning to a new subdivision.

[0049] To provide further details regarding the division of an image into smaller regions, the following description will first focus on the subdivision performed by subdivision units 28 and 104a, followed by the combining process performed by combiner 30 and combiner 104b, and finally, the inter-plane matching / prediction.

[0050] When the subdivision unit 28, 104a divides an image, it can divide the image into a number of blocks, potentially of various sizes, for the purposes of encoding predictions and residuals for image or video data. As described above, the image 24 may be used as one or more arrays of image sample values. In the case of a YUV / YCbCr color space, for example, one array may represent the luma channel and two other arrays may represent the chroma channels. These arrays may have different dimensions. All arrays may be grouped into one or more plane groups, each consisting of one or more consecutive planes, such that each plane is included in only one plane group. For each plane group, the following applies: The first array of a particular plane group is called the primary array of this plane group. Any subsequent arrays are called subarrays. The division of the blocks of the primary arrays may be based on a quadtree approach, as described below. The division of the blocks of the subarrays may be derived based on the division of the primary arrays.

[0051] In the embodiment described below, the subdivision unit 28, 104a is configured to divide the primary array into a number of equal-sized square blocks, so-called treeblocks, as follows: If a quadtree is used, the length of the treeblock boundary is typically a power of two, such as 16, 32, or 64. However, it should be noted that strictly speaking, other tree types can also be used, such as binary trees or trees with any number of leaves. Furthermore, the number of children of a tree can vary depending on the level of the tree and the signal it represents.

[0052] Additionally, as mentioned above, the array of samples may represent information other than a video sequence, such as a depth map or a light field. For simplicity, the following description focuses on a quadtree as a representative example of a multi-tree. A quadtree is a tree in which each internal node has exactly four children. Each treeblock constitutes a primary quadtree, and each leaf of the primary quadtree constitutes a sub-quadtree. The primary quadtree determines the subdivision of a given treeblock for prediction, and the sub-quadtree determines the subdivision of a given prediction block for residual coding.

[0053] The root node of a first-order quadtree corresponds to a complete treeblock. For example, Figure 3A shows treeblock 150. It is assumed that each image is divided into a regular grid of lines and columns, such as treeblock 150, such that the sample array is completely covered. However, it should be noted that not all of the block subdivisions described below require seamless subdivisions without overlapping. Instead, adjacent blocks may overlap each other, as long as no leaf block is a proper subdivision of an adjacent leaf block.

[0054] According to the quadtree structure of treeblock 150, each node can be further divided into four child nodes. For a first-order quadtree, this means that each treeblock 150 can be divided into four subblocks, each half the width and height of treeblock 150. In FIG. 3A, these subblocks are designated 152a through 152d. In a similar manner, each of these subblocks can be further divided into four smaller subblocks, each half the width and height of the original subblock. In FIG. 3d, an example of subblock 152c is shown, subdivided into four smaller subblocks 154a through 154d. In FIGS. 3A through 3C, an example of how treeblock 150 is divided is shown: treeblock 150 is first divided into four subblocks 152a through 152d. Then, the bottom-left subblock 152c is further divided into four smaller subblocks 154a through 154d. Finally, as shown in Figure 3C, the top right of these smaller sub-blocks, block 154b, is again divided into four blocks each having one-eighth the width and height of the original treeblock 150. These smaller blocks are designated 156a through 156d.

[0055] Figure 4 shows the tree structure underlying the quadtree-based partitioning examples shown in Figures 3A through 3D. The numbers next to the tree nodes are values ​​called subdivision flags, which will be explained in more detail later when discussing quadtree structure signaling. The root node of the quadtree is shown at the top of the figure ("Level 0"). The four branches of this root node at level 1 correspond to the four subblocks shown in Figure 3A. In Figure 3B, the third of these subblocks is further subdivided into four subblocks, so the third node at level 1 in Figure 4 also has four branches. In addition, there are four subordinate branches connecting to the second node at level 2 of the quadtree hierarchy, corresponding to the subdivision of the second (top right) child node in Figure 3C. The node at level 3 is not further subdivided.

[0056] Each leaf of the primary quadtree corresponds to a variable-sized block, for which individual prediction parameters can be specified (i.e., intra- or inter-plane prediction mode, motion parameters, etc.). These blocks are hereinafter referred to as prediction blocks. These leaf blocks are specifically illustrated in FIG. 3C. Briefly referring to the descriptions of FIGS. 1 and 2, the partitioning unit 20 or subdivision unit 28 determines the subdivision of the quadtree described above. The subdivision units 152a-152d determine which of the treeblock 150, subblocks 152a-152d, smaller subblocks 154a-154d, etc., should be subdivided or further partitioned. This aims to achieve an optimal trade-off between fine prediction subdivision and coarse prediction subdivision, as already indicated above. The predictor 12 then uses the indicated prediction subdivision to determine the above-mentioned prediction parameters with a granularity corresponding to the prediction subdivision. Alternatively, for each prediction subregion represented by a block as shown in FIG. 3C, a directed prediction subdivision is used.

[0057] The prediction block shown in Figure 3C can be further divided into smaller blocks for the purpose of residual coding. For each prediction block, i.e., for each leaf node of the primary quadtree, one or more sub-quadtrees determine corresponding subdivisions for residual coding. For example, if the size of the residual block is at most 16x16, a given 32x32 prediction block can be divided into four 16x16 blocks, each determined by a sub-quadtree for residual coding. Each 16x16 block in this example corresponds to a root node of a sub-quadtree.

[0058] Similar to the subdivision of a given treeblock into prediction blocks, each prediction block can be divided into multiple residual blocks by decomposing the sub-quadtree. Each leaf of the sub-quadtree corresponds to a residual block. The residual blocks can be assigned individual residual coding parameters (i.e., transform mode, transform coefficients, etc.) by the residual precoder 14, followed by the residual reconstruction unit 16, 106, which are controlled by the residual coding parameters.

[0059] In other words, the subdivision unit 28 may be configured to determine a prediction subdivision and a lower-level residual subdivision for each image or each group of images. This is done as follows: First, the image is divided into a regular array of tree blocks 150, and a subset of these tree blocks is recursively partitioned by quadtree subdivision to obtain a prediction subdivision into prediction blocks. If no partitioning is performed on each tree block, or on leaf blocks of the quadtree subdivision, the prediction block may be a tree block. Then, the subset of these prediction blocks is further subdivided in a similar manner as follows: If the prediction block is larger than the maximum size of the lower-level residual subdivision, first, each prediction block is divided into a regular array of subtree blocks. Then, the subset of these subtree blocks is subdivided according to the quadtree subdivision procedure to obtain a residual block. If no partitioning is performed on each prediction block into subtree blocks, the residual block may be a prediction block. It may also be a subtree block if no further division into smaller regions is performed for each subtree block or subdivision leaf block of the residual quadtree.

[0060] As outlined above, the subdivision selected for the primary array may be mapped to a sub-array. This is easy when the sub-array has the same dimensions as the primary array. However, if the dimensions of the primary array and the sub-array are different, special methods are required. In general, mapping a subdivision of the primary array to a sub-array with different dimensions is possible by spatial mapping, i.e., by spatially mapping the block boundaries of the subdivision of the primary array to the sub-array. In particular, for each sub-array, horizontal and vertical scaling factors may be used to determine the ratio of the dimensions of the primary array to the sub-array. The division of the sub-array into sub-blocks for prediction and residual coding may be determined using the treeblocks of the sub-array obtained by applying the relative scaling factors to the primary quadtree and sub-quadtree of each arranged treeblock of the primary array. If the horizontal and vertical scaling factors are different (e.g., 4:2:2 chroma subsampling), the resulting prediction blocks and residual blocks of the sub-array will not be square. In this case, it is possible to predetermine or select whether a non-square residual block (for the entire sequence, one image of the sequence, or a single prediction or residual block) is to be divided into square blocks or not depending on the situation. For example, in the first case, if the mapped block is not square, the encoder and decoder accept the subdivision into square blocks. In the second case, the subdivision unit 28 signals the selection to the subdivision unit 104a via the data stream insertion unit 18 and the data stream 22. For example, in the case of 4:2:2 chroma subsampling, if the subarray is half the width and the same height as the primary array, the height of the residual block is twice its width. By dividing this block vertically, two square blocks are obtained again from one block.

[0061] As mentioned above, the subdivision unit 28 or the division unit 20, respectively, communicates the quadtree-based division to the subdivision unit 104a via the data stream 22. To do so, the subdivision unit 28 informs the data stream insertion unit 18 of the subdivision selected for the image 24. The data stream insertion unit then transmits to the decoding side the primary and secondary quadtree structures, i.e., the division of the image array into blocks of variable size for prediction or residual coding in the data stream or bitstream 22.

[0062] The minimum and maximum allowed block sizes are transmitted as side information and may vary from picture to picture. They can also be fixed in the encoder and decoder. These minimum and maximum block sizes may be different for predicted and residual blocks. To communicate the structure of the quadtree, the quadtree needs to be traversed, and for each node, it needs to be specified whether this particular node is a leaf node of the quadtree (i.e., the corresponding block is not further subdivided) or whether it branches into four child nodes (i.e., the corresponding block is divided into four sub-blocks of half size).

[0063] Within an image, information is propagated treeblock by treeblock in a raster scan order, such as left to right and top to bottom, as shown at 140 in Figure 5A. Other scan orders are also possible, such as bottom right to top left, or a checkerboard pattern. In a preferred embodiment, to propagate subdivision information, each treeblock, i.e., each quadtree, is traversed in depth order.

[0064] In a preferred embodiment, not only the subdivision information, i.e., the structure of the tree, but also prediction data, i.e., the payload associated with the leaf nodes of the tree, are transmitted / processed in depth-order. This is done because depth-order traversal has significant advantages over width-order traversal. Figure 5B shows the structure of a quadtree with leaf nodes denoted a, b, ..., j. Figure 5A shows the resulting block division. If the blocks / leaf nodes are traversed in width order, they are obtained in the order abjchidefg, while in depth order they are abc...ij. As can be seen from Figure 5A, in depth order, the left-neighboring block and the top-neighboring block are always transmitted / processed before the current block. In this way, motion vector prediction and context modeling can always use the parameters specified for the left- and top-neighboring blocks, resulting in improved coding efficiency. This would not be the case in width order, since, for example, block j is transmitted before blocks e, g, and i.

[0065] Finally, for each treeblock, the signaling is recursively performed along the primary quadtree structure, so that for each node, a flag is transmitted specifying whether the corresponding block is to be divided into four subblocks. If the flag value is "1" (indicating "true"), this signaling process is repeated recursively for all four child nodes or subblocks in raster scan order (top left, top right, bottom left, bottom right) until a leaf node of the primary quadtree is reached. Leaf nodes are characterized by having a subdivision flag value of "0." If the node is at the lowest hierarchical level of the primary quadtree and corresponds to the smallest allowable size of a prediction block, the subdivision flag need not be transmitted. In the example of Figures 3A-3C, a "1" is initially transmitted, indicating that the treeblock 150 is to be divided into four subblocks 152a-152d, as shown at 190 in Figure 6A. Next, in raster scan order 200, encoding of subdivision information for all four sub-blocks 152a-152d is performed recursively. For the first two sub-blocks 152a and 152b, a "0" is transmitted (see 202 in FIG. 6A) specifying that no subdivision will be performed. For the third sub-block 152c (bottom left), a "1" is transmitted (see 204 in FIG. 6A) specifying that this block will be sub-divided. This recursive approach then processes the four sub-blocks 154a-154d of this block. Here, a "0" is transmitted (206) for the first sub-block, and a "1" is transmitted (208) for the second (top right) sub-block. This results in processing the four blocks 156a-156d, which represent the smallest block size in FIG. 3C. If the smallest allowable block size in this example is reached, no further subdivision is possible, and no further data needs to be transmitted. Alternatively, "0000" is transmitted to specify that these blocks are not further divided, as shown in Figure 6A at 210. After this, "00" is transmitted for the bottom two blocks of Figure 3B (see 212 in Figure 6A), and finally, a "0" is transmitted for the bottom right block of Figure 3A (see 214).Thus, the complete binary sequence representing the quadtree structure looks like Figure 6A.

[0066] In this binary sequence shown in Figure 6A, different background shadings correspond to different levels in the quadtree-based subdivision hierarchy. Shading 216 represents level 0 (corresponding to a block size equal to the original treeblock size), shading 218 represents level 1 (corresponding to a block size equal to half the original treeblock size), shading 220 represents level 2 (corresponding to a block size equal to 1 / 4 the original treeblock size), and shading 222 represents level 3 (corresponding to a block size equal to 1 / 8 the original treeblock size). All subdivision flags at the same hierarchical level (corresponding to the same block size and the same representation in the illustrated binary sequence) may be entropy coded, for example by the insertion unit 18, using one and the same probability model.

[0067] In the case of width-order traversal, the subdivision information is transmitted in a different order as shown in FIG. 6B.

[0068] Similar to the subdivision of each treeblock for prediction purposes, the resulting division of each prediction block into residual blocks must also be transmitted in the bitstream. The maximum and minimum block sizes for residual coding may also be transmitted as side information, and may vary from image to image. The maximum and minimum block sizes for residual coding may also be fixed within the encoder and decoder. At each leaf node of the primary quadtree shown in FIG. 3C, the corresponding prediction block may be divided into residual blocks of the maximum allowable size. These blocks are the root nodes that constitute the structure of the lower quadtree for residual coding. For example, if the maximum size of the residual blocks of an image is 64x64 and the size of the prediction blocks is 32x32, the entire prediction block corresponds to one root node of the lower (residual) quadtree of size 32x32. On the other hand, if the maximum size of the residual blocks of an image is 16x16, the 32x32 prediction block is composed of four root nodes of the residual quadtree of size 16x16. Within each prediction block, for each root node, the sub-quadtree structure is propagated in raster scan order (left to right, top to bottom). As in the case of the primary (prediction) quadtree structure, for each node, a flag is coded specifying whether this particular node is split into four child nodes. If the value of this flag is "1", then this procedure is repeated recursively for all four corresponding child nodes and corresponding sub-blocks in raster scan order (top left, top right, bottom left, bottom right) until a leaf node of the sub-quadtree is reached. As in the case of the primary quadtree, no propagating is required for the nodes at the lowest hierarchical level of the sub-quadtree, since these nodes correspond to the smallest size blocks of the residual block and cannot be further divided.

[0069] For entropy coding, the subdivision flags of residual blocks belonging to residual blocks of the same block size may be encoded using one and the same probability model.

[0070] Thus, according to the example illustrated above with reference to Figures 3A to 6A, the subdivision unit 28 defined a primary subdivision for prediction purposes and subdivisions of blocks of different sizes of the primary subdivision for residual coding purposes. The data stream insertion unit 18 coded the primary subdivision for each tree block in zigzag scan order by transmitting a bit sequence constructed according to Figure 6A, as well as coding the maximum size of the primary block and the maximum hierarchical level of the primary subdivision. For each prediction block thus defined, associated prediction parameters are included in the data stream. Furthermore, for each prediction block that is less than or equal to the maximum size of the residual subdivision and for each block at the root of the residual tree divided from the prediction block that exceeds the maximum size defined for the residual block, similar information, i.e., the maximum size, the maximum hierarchical level, and the bit sequence as shown in Figure 6A, are coded. For each residual block thus defined, residual data is inserted into the data stream.

[0071] The extraction unit 102 extracts each bit sequence from the data stream at the input unit 116 and notifies the obtained subdivision information to the division unit 104. Furthermore, the data stream insertion unit 18 and the extraction unit 102 may use the above-mentioned order between the prediction blocks and the residual blocks to transmit further syntax elements, such as the residual data output by the residual precoder 14 and the prediction parameters output by the prediction unit 12. Using this order has the advantage that it may be possible to select a suitable context for encoding individual syntax elements of a particular block by utilizing already coded / decoded syntax elements of neighboring blocks. Furthermore, similar to the residual precoder 14 and the prediction unit 12, the residual reconstruction unit 106 and the precoder 110 may also process the individual prediction blocks and residual blocks in the order outlined above.

[0072] Figure 7 is a flow chart showing the steps performed by the extractor 102 to extract subdivision information from the data stream 22 when encoding is performed in the manner outlined above. In a first step, the extractor 102 divides the image 24 into tree-root blocks 150. This step is shown as step 300 in Figure 7. In step 300, the extractor 102 may extract the maximum size of the prediction blocks from the data stream 22. Alternatively, in step 300, the extractor 102 may extract the maximum hierarchical level from the data stream 22.

[0073] Next, in step 302, the extractor 102 decodes flags or bits from the data stream. When step 302 is performed for the first time, the extractor 102 recognizes that each flag is the first flag in the bit sequence belonging to the first block 150 at the root of the tree in the tree root block traversal order 140. Because this flag is a flag at hierarchical level 0, the extractor 102 can determine the context using the context modeling associated with that hierarchical level 0 in step 302. For each context, a respective probability estimate may be made for entropy decoding of the associated flag. The context probability estimate may be context-specifically adapted to the statistics of the symbols in each context. For example, to determine a suitable context for decoding the hierarchical level 0 flag in step 302, the extractor 102 may select one of the following contexts from the set of contexts: The context is associated with that hierarchical level 0 depending on the flags of the neighboring treeblocks at hierarchical level 0, and further depending on the information contained in the bit strings defining the quadtree subdivisions of the treeblocks neighboring the currently processed treeblock, such as the top and left neighboring treeblocks.

[0074] In the next step, i.e., step 304, the extractor 102 checks whether the most recently decoded flag indicates a segmentation. If so, the extractor 102 performs segmentation of the current block (here, a treeblock) in step 306 or indicates this segmentation to the subdivision unit 104a. Then, in step 308, it checks whether the current hierarchical level is equal to the maximum hierarchical level minus one. For example, the extractor 102 may retain the maximum hierarchical level extracted from the data stream in step 300. If the current hierarchical level is not equal to the maximum hierarchical level minus one, the extractor 102 increments the current hierarchical level by one in step 310 and returns to step 302 to decode the next flag from the data stream. Here, since the flag decoded in step 302 belongs to a different hierarchical level, according to an embodiment, the extractor 102 may select one of another set of contexts belonging to the current hierarchical level. This selection may be based on the bit sequence of the subdivisions of the decoded neighboring treeblocks, such as in FIG. 6A.

[0075] If the flag is decoded and it is determined in step 304 that this flag does not indicate partitioning of the current block, the extractor 102 proceeds to step 312 to determine whether the current hierarchical level is 0. If so, the extractor 102 proceeds to step 314 to process the next tree root block in the traversal order 140, or stops the process of extracting subdivision information if there are no tree root blocks remaining to be processed.

[0076] It should be noted that the description of Figure 7 focuses solely on decoding the flag indicating subdivision for the prediction subdivision. In practice, step 314 may also include decoding other bins or syntax elements related to the current treeblock, etc. In any case, if there is a further next tree root block, extractor 102 proceeds from step 314 to step 302 to decode the next flag from the subdivision information, i.e., the first flag in the flag sequence for the new tree root block.

[0077] If step 312 determines that the hierarchical level is not 0, the process proceeds to step 316, where it is determined whether there are any further child nodes related to the current node. That is, when the extraction unit 102 performs the determination in step 316, it has already determined in step 312 that the current hierarchical level is a level other than 0. There is a parent node that belongs to the tree root block 150 or one of the smaller blocks 152a-152d, or even to the smaller blocks 152a-152d. The node in the tree structure to which the most recently decoded flag belongs has a parent node that is also common to the other three nodes in the current tree structure. The order of traversal of such child nodes that share a common parent node is illustrated in FIG. 3A as an example for the case of hierarchical level 0, denoted by reference numeral 200. Thus, in step 316, the extraction unit 102 determines whether all four child nodes have been traversed in the process of FIG. 7. If not, i.e., if there are further child nodes in addition to the current parent node, the process of FIG. 7 proceeds to step 318. The next child node in the current hierarchical level is traversed according to the zigzag traversal order 200, so that the corresponding sub-block represents the current block for process 7. A flag is then decoded from the data stream for the current block or current node in step 302. However, if there are no more child nodes of the current parent node in step 316, the process of Figure 7 proceeds to step 320, where the current hierarchical level is decremented by one, and the process proceeds to step 312.

[0078] The extraction unit 102 and the subdivision unit 104a work together to obtain the encoder-selected subdivision from the data stream by performing the steps shown in Figure 7. The process of Figure 7 summarizes the prediction subdivision for the above case. Figure 8, combined with the flowchart of Figure 7, shows how the extraction unit 102 and the subdivision unit 104a work together to obtain the residual subdivision from the data stream.

[0079] 8 specifically illustrates the steps performed by the extraction unit 102 and the subdivision unit 104a for each prediction block resulting from the prediction subdivision. These prediction blocks are traversed in a depth-first traversal order within each tree block 150 currently being scanned to traverse leaf blocks, as shown in FIG. 3C, according to the zigzag scan order 140 within the prediction subdivision tree block 150, as described above. The leaf blocks of the partitioned primary tree block are scanned according to the depth-first traversal order. The subblocks at a particular hierarchical level to which the common current node belongs are scanned in a zigzag scan order 200, scanning the subdivisions of each of these subblocks first, followed by the next subblock in the zigzag scan order 200.

[0080] In the example of FIG. 3C, the resulting traversal order among the leaf nodes of treeblock 150 is shown at 350 .

[0081] For the currently scanned prediction block, the process of Figure 8 starts at step 400. In step 400, an internal parameter representing the current size of the current block is set equal to the size of hierarchical level 0 of the residual subdivision, i.e. the maximum block size of the residual subdivision. It should be noted that the maximum size of the residual block may be smaller than the minimum block size of the prediction subdivision, or may be larger than it. In other words, in embodiments, the encoder is free to choose between the above possibilities.

[0082] In the next step, step 402, a check is made to see if the size of the prediction block of the currently scanned block is greater than an internal parameter representing the current size. If it is greater than the internal parameter, the currently scanned prediction block is greater than the maximum size of the residual block. This prediction block is a leaf block of a prediction subdivision or a tree block of a prediction subdivision and is not further partitioned. In this case, the process of FIG. 8 proceeds to step 300 of FIG. 7. That is, the currently scanned prediction block is partitioned into residual tree root blocks, and the first flag in the flag sequence of the first residual tree block in this currently scanned prediction block is decoded, e.g., in step 302.

[0083] However, if the size of the currently scanned prediction block is less than or equal to the internal parameter representing the current size, the process of Figure 8 proceeds to step 404, where the size of the prediction block is checked to determine whether it is equal to the internal parameter representing the current size. If so, the splitting step 300 may be skipped and the process proceeds directly to step 302 of Figure 7.

[0084] However, if the size of the currently scanned prediction block is smaller than the internal parameter representing the current size, the process of FIG. 8 proceeds to step 406, where the hierarchical level is incremented by one and the current size is set to the size of the new hierarchical level, such as divided by two (in both axes in the case of quadtree subdivision). The check of step 404 is then performed again. The effect of the loop formed by steps 404 and 406 is that the hierarchical level always corresponds to the size of the corresponding block to be partitioned, regardless of whether each prediction block is smaller or larger than the maximum size of the residual block. Thus, when decoding the flag in step 302, context modeling is performed depending on both the hierarchical level to which the flag refers and the size of the block. Using different contexts for flags of different hierarchical levels or block sizes has the following advantages: The probability estimates closely match the actual probability distribution for the occurrence of flag values; and, at the same time, the number of contexts managed is relatively small. As a result, the overhead of context management is reduced and the contexts closely match the actual symbol statistics.

[0085] As mentioned above, there may be multiple sample arrays, and these sample arrays may be grouped into one or more plane groups. The input signal received at input unit 32 and encoded may be, for example, an image of a video sequence or a still image. Thus, an image may be provided in the form of one or more sample arrays. In the context of encoding an image of a video sequence or a still image, a sample array may refer to three color planes, such as red, green, and blue, or luma and chroma planes, such as in a YUV or YCbCr color representation. There may also be sample arrays representing alpha, i.e., transparency, and / or depth information for 3D video material. A number of such sample arrays may be grouped together in so-called plane groups. For example, luma (Y) may be one plane group with only one sample array, and chroma, such as CbCr, in another plane group with two sample arrays. In another example, YUV may be one plane group with three planes, and depth information for 3D video material in another plane group with only one sample array. For all plane groups, one primary quadtree structure may be coded in data stream 22 to represent the division into prediction blocks, and for each prediction block, a secondary quadtree structure may be coded that represents the division into residual blocks. Thus, following the first example above, where the luma components form one plane group and the chroma components form another plane group, there would be one quadtree structure for prediction blocks of the luma planes, one quadtree structure for residual blocks of the luma planes, one quadtree structure for prediction blocks of the chroma planes, and one quadtree structure for residual blocks of the chroma planes. However, in the second example above, there would be one quadtree structure for prediction blocks of both luma and chroma (YUV), one quadtree structure for residual blocks of both luma and chroma (YUV), one quadtree structure for prediction blocks of depth information for 3D video material, and one quadtree structure for residual blocks of depth information for 3D video material.

[0086] Furthermore, the above description describes how the input signal is divided into prediction blocks using a first-order quadtree structure, and these prediction blocks are further subdivided into residual blocks using a sub-quadtree structure. In another embodiment, the subdivision does not have to end at the sub-quadtree stage, i.e. the blocks resulting from the division using the sub-quadtree structure may be further subdivided using a third quadtree structure. This division may be performed with the aim of using further coding tools that facilitate the encoding of the residual signal.

[0087] The above description has focused on the subdivision performed by subdivision unit 28 and subdivision unit 104a, respectively. As noted above, the subdivision is defined by subdivision units 28, 104a, respectively, which controls the granularity at which the above modules of encoder 10 and decoder 100 perform their processing. However, in the embodiments described below, combiner unit 30 and combiner unit 104b follow subdivision units 228, 104a, respectively. However, combiner units 30, 104b are optional and may be omitted.

[0088] However, as will be explained in more detail below, the combining unit actually provides the encoder with the opportunity to combine some of the prediction blocks or residual blocks into groups or clusters, allowing other modules, or at least some of the other modules, to handle these groups of blocks together. For example, if signaling a group of prediction blocks with common parameters for all blocks belonging to this group offers better rate / distortion potential than signaling prediction parameters for all of these prediction blocks individually, the predictor 12 may sacrifice small deviations between the prediction parameters of some of the prediction blocks, as determined by the optimization using the subdivision unit 28, and instead use common prediction parameters for all of these prediction blocks. However, the process of obtaining a prediction based on these common prediction parameters in the predictors 12 and 110 themselves may be performed for each prediction block. However, it is also possible for the predictors 12 and 110 to perform a single prediction process for the entire group of prediction blocks.

[0089] As will be explained in more detail below, it is also possible that the grouping of prediction blocks is not only aimed at using the same or common prediction parameters for a group of prediction blocks, but also, or in addition, allows the encoder 10 to transmit one prediction parameter for this group together with the prediction residuals of the prediction blocks belonging to this group, thereby reducing the overhead of signaling the prediction parameters for this group. The process of combining has less impact on the data stream insertion unit 18 than the decisions made by the residual precoder 14 and the prediction unit 12, as will be explained in more detail below. Strictly speaking, it should be noted that the above aspects also apply to other subdivisions, such as the above-mentioned residual subdivision and filter subdivision.

[0090] First, the combination of sets of samples, such as the predicted block and the residual block, can be realized in various ways, i.e., not limited to the multi-tree subdivision described above, but the following description will focus on the combination of blocks resulting from the multi-tree subdivision in the above embodiment.

[0091] In general, combining syntax elements associated with specific sets of samples for the purpose of transmitting associated coding parameters can reduce the side information rate in image and video coding. For example, the sample array of a signal to be encoded is typically partitioned into specific sets of samples. A specific set of samples may represent a rectangular or square block, or another set of samples, including regions of any shape, such as a triangle. In the above-described embodiment, the simply connected regions were the prediction blocks and residual blocks resulting from the subdivision of the multi-tree. The subdivision of the sample array may be fixed by the syntax, or, as described above, the subdivision information may be signaled, at least in part, within the bitstream. To keep the side information rate for transmitting subdivision information low, the syntax typically allows only a limited number of simple partitioning choices, such as subdivision of blocks into smaller blocks. The sample sets are associated with specific coding parameters, which specify the prediction information, the mode of residual coding, etc., as described above in more detail. For each sample set, individual coding parameters may be transmitted, specifying the coding of the prediction and / or the residual. To achieve improved coding efficiency, the combining aspect described below, i.e., combining two or more sample sets into so-called groups of sample sets, offers several advantages. These advantages are further described below. For example, sample sets may be combined such that all sample sets of such a group share the same coding parameters. The same coding parameters can be transmitted together with one of the sample sets in the group. This avoids the need to transmit coding parameters separately for each sample set of the group of sample sets; instead, the coding parameters are transmitted only once for the entire group of sample sets. This may result in a reduction in the side information rate for transmitting the coding parameters, potentially improving the overall coding efficiency. As an alternative approach, additional refinements of one or more coding parameters can be transmitted for one or more sample sets of the group of sample sets.Refinement can be applied either to all of the sample sets of a group, or only to the sample set that is transmitted.

[0092] The combining aspect, described further below, also provides the encoder with more flexibility in creating the bitstream 22, since the combining approach significantly increases the number of partitioning options for the image sample array. This can improve coding efficiency by allowing the encoder to select from a larger number of options, such as to minimize a particular rate / distortion budget. There are several options for the encoder's operation. In a simple approach, the encoder first determines the optimal subdivision of the sample array. With brief reference to FIG. 1 , the subdivision unit 28 can determine the optimal subdivision in the initial stage. Each sample set is then examined to see if combining with another sample set or group of sample sets reduces the particular rate / distortion budget. Prediction parameters associated with the combined group of sample sets can then be reevaluated, such as by performing a new motion search or using prediction parameters previously determined for the common sample set. Then, candidate sample sets or groups of sample sets for combining are evaluated for the group of sample sets under consideration. In a detailed approach, a particular rate / distortion penalty is evaluated for a group of additional candidate sample sets.

[0093] The combining approach described below does not change the order of processing of the sample sets, i.e., the combining concept can be implemented in such a way that there is no increase in latency, i.e., each sample set can be decoded at the same time as without the combining approach.

[0094] For example, if the bitrate savings from reducing the number of coded prediction parameters is greater than the bitrate otherwise used to code the combination information to indicate the combination to the decoder, the combination approach, as described further below, results in improved coding efficiency. The described syntax extension for combination also allows the encoder more freedom in choosing the partitioning of pictures or plane groups into blocks. In other words, the encoder is not restricted to first performing subdivision and then determining whether some of the resulting blocks have the same or similar sets of prediction parameters. As a simple alternative, the encoder can first determine the subdivision according to the rate / distortion loss, and then determine for each block whether combining with a neighboring block or one of the associated groups of previously determined blocks reduces the rate / distortion loss. The prediction parameters associated with the new group of blocks can then be reevaluated, such as by performing a new motion search or using the prediction parameters previously determined for the current block, and the neighboring block or group of blocks can be evaluated for the new group of blocks. The combination information can be signaled on a block-by-block basis. In effect, combining can also be understood as an estimation of the prediction parameters of the current block, where the estimated prediction parameters are set equal to the prediction parameters of one of the neighboring blocks. Alternatively, for blocks within a group of blocks, the residual may be transmitted.

[0095] Thus, the basic idea underlying the combining discussed further below is to reduce the bitrate required for transmitting prediction or other coding parameters by combining adjacent blocks into groups of blocks. Each group of blocks is then associated with a unique set of coding parameters, such as prediction or residual coding parameters. The combining information, if present, is transmitted in the bitstream in addition to the subdivision information. The benefit of the combining concept is improved coding efficiency as a result of the reduced information rate associated with the coding parameters. The combining process described here can also be extended to dimensions other than spatial. For example, groups of respective sets of samples or blocks present in several different video images can be combined into one group of blocks. Combining can also be applied to 4D compression and light field coding.

[0096] 1 through 8, regardless of the specific manner in which the subdivision units 28 and 104a subdivide the image, the subdivision followed by the merging process is advantageous. Specifically, the latter can also subdivide images in a manner similar to H.264, for example. That is, each image can be subdivided into a regular array of rectangular or square macroblocks of a predetermined size, such as 16x16 luminance samples, or a size signaled in the data stream. Each macroblock has associated coding parameters, including, among other things, partitioning parameters and corresponding prediction parameters in the data stream. The partitioning parameters define the partitioning of each macroblock into a regular subgrid of 1, 2, 4, etc., which corresponds to the prediction granularity. The partitioning for the residual and the corresponding granularity of the transformation of the residual are also defined.

[0097] In either case, combining provides the advantages outlined above, such as reducing the bit rate of associated information in image and video coding. A particular set of samples, representing a square or rectangular block, or an area of ​​any shape, or any other set of samples, such as any simply connected region or sample, is typically associated with a particular set of coding parameters. For each set of samples, coding parameters are included in the bitstream, such as prediction parameters that specify how to predict the corresponding set of samples using previously coded samples. The partitioning of the image sample array into sets of samples may be fixed by syntax or conveyed by corresponding subdivision information in the bitstream. The coding parameters for the sets of samples may be transmitted in a predetermined order specified by the syntax. The combining unit 30 can communicate to a group of sets of samples about a common set of samples or a current block, such as a predicted block or residual block, combined with one or more other sets of samples according to the combining function. Therefore, the coding parameters for a group of sets of samples need only be transmitted once. In certain embodiments, if the current sample set is to be combined with an existing group of sample sets or sample sets for which coding parameters have already been transmitted, the coding parameters of the current sample set are not transmitted. Instead, the coding parameters of the current sample set are set equal to the coding parameters of the sample set or group of sample sets to which the current sample set is combined. As an alternative approach, additional refinements for one or more coding parameters can be transmitted for the current sample set. The refinements can apply either to all sample sets of the group or only to the sample set for which the refinement is transmitted.

[0098] In an embodiment, the set of all coded / decoded sample sets for each sample set, such as the above-mentioned prediction block, the above-mentioned residual block, or the above-mentioned leaf blocks of a multi-tree subdivision, is referred to as the "set of causal sample sets." See, for example, FIG. 3C. All blocks shown in this figure are the result of a specific subdivision, such as a prediction subdivision or a residual subdivision, or a multi-tree subdivision, and the coding / decoding order defined between these blocks is defined by arrow 350. If a specific one of these blocks is considered as the current sample set or the current simply connected region, then the set of causal sample sets consists of all blocks that precede the current block in the order 350. However, other subdivisions that do not use multi-tree subdivision are also possible, as long as the combination principle described below is followed.

[0099] The set of samples that can be used to combine with the current sample set is referred to below as the "set of candidate sample sets" and is always a subset of the "set of causal sample sets." The method for forming the subset can be signaled to the decoder or specified in the data stream or bitstream from the encoder to the decoder. When a particular current sample set is encoded / decoded and its set of candidate sample sets is not empty, the encoder signals in the data stream and the decoder derives from the data stream whether a common sample set is to be combined with one of the candidate sample sets. If so, they are signaled together. Otherwise, the set of candidate sample sets is empty and no combining is available for this block.

[0100] There are various ways to determine a subset of the set of causal samples that represents the set of candidate sample sets. For example, the determination of the set of candidate samples may be based on a uniquely geometrically defined sample in the current sample set, such as a sample in the top left image of a rectangular or square block. Starting from this uniquely geometrically defined sample, a non-zero number of specific samples is determined. This means that for this uniquely geometrically defined sample, Represents direct spatial neighbors. For example, this non-zero number of specific exemplars includes the top and left neighbors of a uniquely geometrically defined exemplar in the current exemplar set. The non-zero number of neighbors is at most 2, 1 if one of the top or left neighbors is unavailable or outside the image, and 0 if neither neighbor is present.

[0101] A set of candidate sample sets is then determined to encompass sample sets that contain at least one non-zero number of the neighboring samples. See, for example, FIG. 9A. Let the sample set under consideration for merging be block X, and its unique geometrically defined sample be the top-left sample, illustrated at 400. The neighboring samples above and to the left of sample 400 are shown at 402 and 404. The set of causal sample sets, or causal blocks, is highlighted by shading. Of these blocks, blocks A and B contain one of the neighboring samples 402 and 404, and therefore these blocks form a set of candidate block sets, or a set of candidate sample sets.

[0102] In another embodiment, the set of candidate sample sets determined for combining purposes may additionally or exclusively include sample sets containing a specific non-zero number of samples. The specific non-zero number of samples may be one or two, and may be in the same spatial location but in a different image, i.e., a previously coded / decoded image. For example, in addition to blocks A and B in FIG. 9A, a block of a coded image containing a sample in the same location as sample 400 may be used. Note that only the top neighbor 404 or only the left neighbor 402 may be used to define the above non-zero number of neighboring samples. In general, the set of candidate sample sets may be derived from processed data in the current image or other images. This derivation may include spatial information, such as transform coefficients associated with a specific direction or the tilt of the current image, and may also include temporal information, such as a representation of neighboring motion. The set of candidate sample sets may be derived from such data available to the receiver / decoder, and, if present, from other data and associated information in the data stream.

[0103] Both the encoder-side combiner 30 and the decoder-side combiner 104b derive candidate sample sets in parallel. As noted above, they may independently determine the set of candidate sample sets based on a predetermined method communicated to both. Alternatively, the encoder may signal hints in the bitstream, which would position combiner 104b to derive these candidate sample sets in the same way that encoder-side combiner 30 determined the set of candidate sample sets.

[0104] As will be described in detail below, the combiner 30 and the data stream inserter 18 cooperate to transmit one or more syntax elements for each sample set. These syntax elements specify whether the sample set is to be combined with another sample set that is part of the group of previously combined sample sets, and the set of candidate sample sets to be used for the combination. The extractor 102 then extracts these syntax elements and notifies the combiner 104b accordingly. In particular, in the specific embodiment described below, one or two syntax elements are transmitted to specify the combining information for a particular sample set. The first syntax element specifies whether the current sample set is to be combined with another sample set. The second syntax element is transmitted only if the first syntax element specifies that the current sample set is to be combined with another sample set, and specifies the set of candidate sample sets to be used for the combination. If the derived set of candidate sample sets is empty, the first syntax element does not need to be transmitted. In other words, the first syntax element may be transmitted only if the derived set of candidate sample sets is not empty. The second syntax element may be transmitted only if the derived set of candidate sample sets contains multiple sample sets, since if the set of candidate sample sets contains only one sample set, no further selection is possible. Furthermore, if the set of candidate sample sets contains multiple sample sets, but all sample sets in the candidate sample sets are associated with the same coding parameter, the second syntax element may not be transmitted. In other words, the second syntax element may be transmitted only if at least two sample sets in the derived candidate sample sets are associated with different coding parameters.

[0105] In the bitstream, the combining information for a sample set may be coded before the prediction parameters or other specific coding parameters associated with that sample set, which may be transmitted only if the combining information signals that the current sample set will not be combined with other sample sets.

[0106] The joint information for a particular sample set, i.e., a block, may be coded after the proper subset of prediction parameters, or more generally, the coding parameters associated with each sample set, has been transmitted. The subset of prediction / coding parameters may consist of one or more reference image indices, or one or more components of a vector of motion parameters, or a combination of a reference indices and one or more components of a vector of motion parameters, etc. The transmitted subset of prediction or coding parameters can be used to derive a set of candidate sample sets from the set of provisional candidate sample sets derived above. For example, a difference magnitude or distance between the coded prediction and coding parameters of the current sample set and the corresponding prediction and coding parameters of a preliminary set of candidate sample sets can be calculated based on a predetermined distance criterion. Only sample sets whose calculated difference magnitude or distance is equal to or less than a predetermined or derived threshold are then included in the final, i.e., reduced, set of candidate sample sets. See, for example, FIG. 9A. Let the current sample set be block X. Assume that a subset of coding parameters associated with this block has already been inserted into the data stream 22. For example, assume that block X is a predictive block, and the proper subset of coding parameters is a subset of prediction parameters for this block X, such as a subset of a set including image reference indices, and motion mapping information, such as motion vectors. If block X is a residual block, the subset of coding parameters is a subset of residual information, such as a map representing the locations of transform coefficients and significant transform coefficients within block X. Based on this information, both the data stream insertion unit 18 and the extraction unit 102 can use this information to determine subsets of blocks A and B, which in this particular embodiment form the preliminary set of candidate sample sets mentioned above. In particular, since blocks A and B belong to the set of causal sample sets, their coding parameters can be used by both the encoder and the decoder when encoding / decoding the coding parameters of block X.Thus, using the comparison of the magnitude of the difference, any number of blocks of the preliminary sets of candidate sample sets A and B may be eliminated. The resulting reduced set of candidate sample sets may then be used as described above, i.e., to determine whether a join indicator representing the join is transmitted in or extracted from the data stream, depending on the number of sample sets in the reduced set of candidate sample sets, and whether a second syntax element needs to be transmitted in or extracted from the data stream, which second syntax element indicates which sample sets in the reduced set of candidate sample sets are to be the blocks to join with.

[0107] The threshold against which the distance is compared may be fixed and communicated to both the encoder and decoder, or may be derived based on a calculated distance, such as the median difference value or other central tendency. In this case, it is inevitable that the reduced set of candidate sample sets will be a proper subset of the preliminary set of candidate sample sets. Alternatively, only those sample sets are selected from the preliminary set of candidate sample sets that have the smallest distance based on the distance magnitude. Alternatively, a single set of samples is selected from the preliminary set of candidate sample sets using the distance magnitude. In the latter case, the merge information only needs to specify whether the current sample set is merged with a single candidate sample set.

[0108] Thus, a set of candidate blocks is formed or derived as described below with reference to Figure 9A. Starting with the top-left sample location 400 of current block X in Figure 9A, the locations of its left neighboring sample 402 and its top neighboring sample 404 are derived at the encoder and decoder sides. Thus, the set of candidate blocks can have at most two members, namely, blocks that contain one of the two sample locations of the shaded set of causal blocks in Figure 9A, which in this case are blocks B and A. Thus, the set of candidate blocks can only have as its members the two direct neighbors of the top-left sample location of the current block. In another embodiment, the set of candidate blocks is formed by all blocks coded before the current block and that contain one or more samples representing direct spatial neighbors of any sample of the current block. The direct spatial neighbors may be limited to the immediate neighbors to the left, and / or the immediate neighbors above, and / or the immediate neighbors to the right, and / or the immediate neighbors below, of any sample of the current block. For example, see Figure 9B, which shows another block subdivision. In this case, the candidate block includes four blocks, namely, blocks A, B, C, and D.

[0109] The set of candidate blocks may additionally or exclusively include blocks containing one or more samples that are in the same position as any sample in the current block but are contained in a different image, i.e., an already coded / decoded image.

[0110] Furthermore, the set of candidate blocks represents a subset of the set of blocks described above, determined by spatial or temporal neighbor relationships. The subset of candidate blocks may be fixed, propagated, or derived. The derivation of the subset of candidate blocks may take into account decisions about other blocks in the image or other images. For example, blocks associated with the same coding parameters or coding parameters that are much more similar than other candidate blocks may not be included in the set of candidate blocks.

[0111] The following description of the embodiment applies when only two blocks, at most, containing samples adjacent to the left and above the top-left sample of the current block, are considered as candidates.

[0112] If the set of candidate blocks is not empty, a flag called merge_flag is signaled, which specifies whether the current block is combined with one of the candidate blocks. If merge_flag is 0 (indicating "false"), this block is not combined with one of the candidate blocks and all coding parameters are transmitted as usual. If merge_flag is 1 (indicating "true"), the following applies: If the set of candidate blocks contains only one block, this candidate block is used for combining. Otherwise, the set of candidate blocks contains two blocks. If the prediction parameters of these two blocks are identical, these prediction parameters are used for the current block. Otherwise (if the two blocks have different prediction parameters), a flag called merge_left_flag is signaled. If merge_left_flag is 1 (indicating "true"), the block containing the adjacent sample position to the left of the top-left sample position of the current block is selected from the set of candidate blocks. If merge_left_flag is 0 (indicating "false"), another (i.e., top-neighboring) block is selected from the set of candidate blocks, and the prediction parameters of the selected block are used for the current block.

[0113] To summarise the several embodiments of combining described above, reference is made to FIG. 10 which illustrates the steps performed by the extractor 102 to extract combining information from the data stream 22 received at the input 116 .

[0114] The process begins with identifying candidate blocks or sample sets for the current sample set or block at 450. Since the coding parameters for the blocks are transmitted in a particular one-dimensional order in the data stream 22, Figure 10 describes the process of obtaining joint information for the currently scanned sample set or block.

[0115] As mentioned above, the identification step 450 may involve identification from a set of decoded blocks, i.e., causal blocks, based on a neighboring relationship perspective. For example, neighboring blocks containing certain neighboring samples that are spatially or temporally adjacent to one or more geometrically predetermined samples of the current block X may be designated candidates. Furthermore, the identification step may include two stages: a first stage including the above-mentioned identification to obtain a preliminary set of candidate blocks, i.e., identification based on neighboring relationships, and a second stage in which only blocks decoded from the data stream before step 450 are designated candidates whose transmitted coding parameters satisfy a certain relationship with a proper subset of the coding parameters of the current block X.

[0116] Next, the process proceeds to step 452, where it is determined whether the number of candidate blocks is greater than zero. If so, in step 454, merge_flag is extracted from the data stream. The extraction step 454 may include entropy decoding. The context for the entropy decoding of merge_flag in step 454 may be determined based on, for example, the set of candidate blocks or syntax elements belonging to a preliminary set of candidate blocks, etc. In that case, the dependency on the syntax elements may be limited to information on whether a block belonging to the set was a candidate for merging. An estimate of the probability of the selected context may be adapted.

[0117] On the other hand, if step 452 determines that the number of candidate blocks is zero, the process of Figure 10 proceeds to step 456, where the coding parameters of the current block are extracted from the bitstream. Also, in the particular case of the two stages described above, the remaining coding parameters are extracted, after which extraction unit 102 proceeds to process the next block in the order of block scanning, such as order 350 shown in Figure 3C.

[0118] Returning to step 454, after the extraction of step 454, the process proceeds to step 458, where it is determined whether the extracted merge_flag indicates whether merging of the current block will occur. If merging will not occur, the process proceeds to step 456 described above. If merging will occur, the process proceeds to step 460, where it is determined whether the number of candidate blocks is one. If the number of candidate blocks is one, there is no need to indicate or transmit a specific candidate block among the candidate blocks, and the process of FIG. 10 proceeds to step 462, where the merge partner of the current block is set to that single candidate block. Then, in step 464, the coding parameters of the merged partner block are used to adapt or predict the coding parameters or remaining coding parameters of the current block. In the case of adaptation, the missing coding parameters of the current block are simply copied from the merge partner block. In the other case, i.e., in the case of prediction, step 464 may further involve extracting residual data from the data stream. This residual data relates to the residual of the prediction of the missing coding parameters of the current block. This residual data may then be combined with predictions of these missing coding parameters obtained from the blocks with which it is combined.

[0119] On the other hand, if it is determined in step 460 that the number of candidate blocks is greater than one, the process of Figure 10 proceeds to step 466, where it is determined whether the coding parameters or corresponding parts of the coding parameters, i.e., parts relating to parts not yet transmitted in the data stream for the current block, are identical to each other. If so, in step 468, these common coding parameters are set as the reference for combination. Alternatively, the candidate block is set as the partner for combination. Each corresponding coding parameter is then used for adaptation or prediction in step 464.

[0120] The combining partner may itself be the block for which the combining is signaled, in which case the adopted or predicted coding parameters of the combining partner are used in step 464.

[0121] On the other hand, if not, i.e., if the coding parameters are not identical, the process in Fig. 10 proceeds to step 470, where the syntax element merge_left_flag is further extracted from the data stream. A separate set of contexts may be used for entropy decoding of this flag. Also, the set of contexts used for entropy decoding of merge_left_flag may consist of only one context. After step 470, the candidate block suggested by merge_left_flag is set as the combining partner in step 472 and used for adaptation or prediction in step 464. After step 464, the extraction unit 102 proceeds to process the next block in block order.

[0122] Of course, there are many alternatives. For example, instead of the separate syntax elements of merge_flag and merge_left_flag described above, a combined syntax element may be transmitted in the data stream to signal the combining operation. Also, the above merge_left_flag may be transmitted in the data stream regardless of whether the two candidate blocks have the same prediction parameters, thereby reducing the computational overhead for performing the processing of FIG. 10.

[0123] As already shown in FIG. 9B and elsewhere, the set of candidate blocks may include three or more blocks. Furthermore, combining information, i.e., information indicating whether a block is combined (and, if combined, combined with a candidate block), may be conveyed by one or more syntax elements. A single syntax element, such as the above-mentioned merge_flag, may specify whether a block is combined with one of the candidate blocks. The flag may be transmitted only if the set of candidate blocks is not empty. A second syntax element, such as the above-mentioned merge_left_flag, may signal which candidate block is used for combining. Typically, a selection from two or more candidate blocks is indicated. The second syntax element may be conveyed only if the first syntax element signals that the current block is combined with one of the candidate blocks. Furthermore, the second syntax element may be transmitted only if the set of candidate blocks includes multiple candidate blocks and / or if one of the candidate blocks has different prediction parameters from the other candidate blocks. The syntax depends on the number of candidate blocks and / or the extent to which the prediction parameters associated with the candidate blocks differ.

[0124] The syntax signaling which candidate blocks are used may be configured simultaneously and / or in parallel at the encoder and decoder sides. For example, if there are three choices for the candidate blocks identified in step 450, the syntax is selected so that only these three choices are available and considered for entropy coding, e.g., in step 470. In other words, syntax elements are selected so that the symbol alphabet has as many elements as there are candidate block choices. All other choices may be considered zero possibilities, and entropy coding / decoding may be coordinated simultaneously at the encoder and decoder.

[0125] Furthermore, as noted in step 464, the prediction parameters estimated as a result of the combining process may represent the complete set of prediction parameters associated with the current block, or may represent a subset of these prediction parameters, such as prediction parameters for one hypothesis of a multi-hypothesis block.

[0126] As mentioned above, syntax elements related to merge information can be entropy coded using context modeling. The syntax elements may consist of the above-mentioned merge_flag and merge_left_flag (or similar syntax elements). In a specific example, one of three context models or contexts can be used to encode / decode merge_flag, such as in step 454. The index of the context model to be used, merge_flag_ctx, may be derived as follows: If the set of candidate blocks includes two elements, the value of merge_flag_ctx is equal to the sum of the merge_flag values ​​of the two candidate blocks. On the other hand, if the set of candidate blocks includes one element, the value of merge_flag_ctx may be equal to twice the value of the merge_flag of this one candidate block. Since each merge_flag of adjacent candidate blocks can be 1 or 0, three contexts are available for merge_flag. The merge_left_flag may be coded using only a single probability model.

[0127] In other embodiments, a different context model may be used. For example, non-binary syntax elements may be mapped to a sequence of binary symbols called bins. The context model for some syntax elements or bins of syntax elements that define the binding information may be derived based on the transmitted syntax elements of neighboring blocks or the number of candidate blocks, while other syntax elements or bins of syntax elements may be coded with a fixed context model.

[0128] Regarding the above description of block combination, the set of candidate blocks may be derived in a manner similar to any of the above-mentioned embodiments, with the following modifications: Restrict candidate blocks to blocks that use motion compensated prediction or inter-prediction, respectively. Only such blocks can be members of the set of candidate blocks. Propagation of combination information and context modeling can be performed as described above.

[0129] Returning now to the combination of the above-described multi-tree subdivision embodiment and the currently described combining aspect. When a quadtree-based subdivision structure is used to divide an image into square blocks of variable size, syntax elements specifying combining, e.g., merge_flag and merge_left_flag, are alternated with prediction parameters transmitted for each leaf node of the quadtree structure. For example, consider again FIG. 9A. FIG. 9A shows an example of a quadtree-based image subdivision into variable-sized prediction blocks. The top two blocks with the largest size are called treeblocks, i.e., prediction blocks of the largest possible size. The other blocks in this diagram are obtained by subdividing the corresponding treeblocks. The current block is indicated by an "X." All shaded blocks are encoded / decoded before the current block, thus forming a set of causal blocks. For one embodiment, as detailed in the description of the derivation of the set of candidate blocks, only blocks containing samples that are directly adjacent (i.e., above or to the left) to the location of the top-left sample of the current block can be elements of the set of candidate blocks. Thus, the current block can be combined with either block "A" or block "B". If merge_flag is 0 (indicating "false"), the current block "X" is not combined with either of the two blocks. If blocks "A" and "B" have identical prediction parameters, combining with either of the two blocks will produce the same result, so no distinction needs to be made. Therefore, in this case, merge_left_flag is not transmitted. Otherwise, if blocks "A" and "B" have different prediction parameters, then if merge_left_flag is 1 (indicating "true"), blocks "X" and "B" are combined, while if merge_left_flag is 0 (indicating "false"), blocks "X" and "A" are combined. In another preferred embodiment, additional adjacent (already transmitted) blocks represent candidates for combining.

[0130] Figure 9B shows another example, where the current block "X" and its left neighbor "B" are treeblocks, i.e., they are of the maximum allowed block size. The size of the top neighbor block "A" is one-quarter of the size of the treeblock. Blocks that are members of the set of causal blocks are shaded. In one preferred embodiment, the current block "X" can only be combined with two blocks "A" or "B", and not with any other top neighboring blocks. In another preferred embodiment, additional neighboring (already transmitted) blocks represent candidates for combining.

[0131] Before proceeding to the description of the aspects of handling different sample arrangements of an image according to the present application, the above description of the aspects of performing multi-tree subdivision and combining while communicating information on the other hand makes it clear that these aspects provide advantages that can be utilized independently of each other. That is, as already explained above, the combination of multi-tree subdivision and combining provides specific advantages. However, advantages can also be obtained in other ways. For example, the combining function is embodied in the subdivision performed by the subdivision unit 30, 104a in response to the subdivision of macroblocks that regularly partitions these macroblocks into smaller sections, rather than based on quadtree or multi-tree subdivision. On the other hand, the combination of multi-tree subdivision that transmits an indication of the maximum size of treeblocks in the bitstream and the use of multi-tree subdivision that transmits the corresponding coding parameters of the blocks using a depth-first traversal order provides advantages, regardless of whether the combining functions are utilized in parallel. In general, the advantages of combining can be understood intuitively by considering the following: Coding efficiency is improved if the syntax for coding sample arrays is extended in a way that not only allows for subdivision of a block, but also allows for the combination of two or more blocks obtained after subdivision. This results in a group of blocks that are coded with the same prediction parameters. The prediction parameters for such a group of blocks need only be coded once. Also, regarding the combination of sample sets, it is reiterated that the sample sets considered are rectangular or square blocks, and the combined sample sets are collections of rectangular or square blocks. The sample sets considered can also be image regions of any shape, and the combined sample sets are collections of image regions of any shape.

[0132] The following discussion focuses on the handling of different sample arrays per image when there are multiple sample arrays per image. The aspects outlined below provide benefits regardless of the type of subdivision used, i.e., whether subdivision is based on multi-tree subdivision or not, and whether merging is used or not. Before proceeding to a description of specific embodiments for handling different sample arrays per image, a short introduction to handling different sample arrays per image is provided before the main subject of these embodiments is described.

[0133] The following discussion focuses on coding parameters between blocks of different sample arrays of an image in image or video coding. In particular, the discussion focuses on adaptively predicting coding parameters between different sample arrays of an image in environments, including, for example, the encoders and decoders of FIGS. 1 and 2, respectively, as well as other image or video coding environments. As discussed above, the sample arrays can represent sample arrays associated with various color components or sample arrays that associate an image with additional information, such as transparency data or depth maps. Sample arrays associated with color components of an image are also referred to as color planes. The techniques described below, also referred to as inter-plane adoption / prediction, can be used in block-based image and video encoders and decoders, where the blocks of the image's sample arrays are processed in any order.

[0134] Image and video coders are typically designed to encode color images (either still images or images from a video sequence). A color image consists of multiple color planes, each representing a different color component. Color images are often encoded as a set of sample arrays consisting of a luma plane and two saturation planes, which specify the chrominance components. In some applications, it is common for the set of encoded sample arrays to consist of three color planes, representing the red, green, and blue primary color samples. Furthermore, color images may consist of four or more color planes for enhanced color representation. Additionally, images can have associated auxiliary sample arrays that specify additional information about the image. For example, these auxiliary sample arrays might specify the transparency of the associated color sample array (suitable for specific display purposes) or a depth map (suitable for multi-view rendering, such as 3D displays).

[0135] In traditional image and video coding standards (e.g., H.264), color planes are typically coded jointly. In this case, certain coding parameters, such as prediction modes, reference indices, and motion vectors for a macroblock and its sub-macroblocks, are used for all color components of a block. The luma plane can be considered the primary color plane, where certain coding parameters are specified in the bitstream, and the chroma plane can be considered a secondary plane, where the corresponding coding parameters are inferred from the primary luma plane. Each luma block is associated with two chroma blocks that represent the same region in the image. Depending on the chroma sampling format used, the chroma sample array for a block may be smaller than the luma sample array. For each macroblock consisting of one luma component and two chroma components, the same partitioning is performed, and the partitioning into smaller blocks (if the macroblock is subdivided) is performed. For each block (which may be the macroblock itself or a subblock of a macroblock), consisting of one block of luma samples and two blocks of chroma samples, the same set of prediction parameters, such as reference indices and motion parameters, and possibly an intra prediction mode, are used. Certain profiles of conventional video coding standards (e.g., the 4:4:4 profile of H.264) also allow for independent coding of different color planes of an image. In this configuration, macroblock partitioning, prediction modes, reference indices, and motion parameters can be selected separately for the color components of a macroblock or subblock. In conventional coding standards, either all color planes are coded using the same specific set of coding parameters (e.g., subdivision information and prediction parameters), or all color planes are coded completely independently of each other.

[0136] When color planes are coded jointly, one set of subdivision and prediction parameters must be used for all color components of a block. This ensures small side information, but may result in lower coding efficiency compared to when coded independently. This is because using different block decomposition and prediction parameters for different color components may result in smaller rate / distortion losses. For example, using different motion vectors or reference frames for the saturation components may significantly reduce the energy of the residual signal for the saturation components, increasing overall coding efficiency. When color planes are coded independently, coding parameters such as block partitioning, reference indices, and motion parameters can be selected separately for each color component to optimize coding efficiency for each color component. However, redundancy between color components cannot be exploited. Multiple transmission of certain coding parameters increases the side information rate (compared to when coded jointly), and this increase in side information rate may negatively impact overall coding efficiency. Furthermore, support for auxiliary sample sequences in modern video coding standards (such as H.264) is limited to encoding the auxiliary sample sequences using a unique set of coding parameters.

[0137] Thus, in all of the embodiments described so far, image planes can be treated as described above. However, as also described above, the overall coding efficiency of coding multiple sample arrays (which may be associated with different color planes and / or auxiliary sample arrays) may be improved if block-level decisions are possible, e.g., whether all sample arrays of a block are coded with the same coding parameters or whether different coding parameters are used. The basic idea behind the inter-plane prediction described below is to enable such block-level, context-sensitive decisions. An encoder can select, based on, for example, rate / distortion criteria, whether all or some of the sample arrays of a particular block are coded using the same coding parameters or whether different coding parameters are used for different sample arrays. This selection can also be achieved by signaling, for a particular block of a sample array, whether a particular coding parameter is inferred from a previously coded block in the same location in a different sample array. Different sample arrays for an image can also be arranged in groups, which are called sample array groups or plane groups. Each plane group can contain one or more sample arrays of the image. Blocks of sample arrays within a plane group share the same selected coding parameters, such as subdivision information, prediction mode, and residual coding mode, while other coding parameters, such as transform coefficient levels, are transmitted separately for each sample array within a plane group. One plane group is coded as a primary plane group, i.e., no coding parameters are estimated or predicted from other plane groups. For each block of a secondary plane group, it is possible to select whether a new set of selected coding parameters is transmitted, or whether the selected coding parameters are estimated or predicted from the primary plane group or another secondary plane group, depending on the situation. The decision of whether the selected coding parameters for a particular block are estimated or predicted is included in the bitstream.Inter-plane prediction allows for more flexibility in the trade-off between the rate of side information and the quality of prediction compared to state-of-the-art coding of images consisting of multiple sample arrays. The advantage is improved coding efficiency compared to traditional coding of images consisting of multiple sample arrays.

[0138] Intra-plane adaptation / prediction may extend image or video coders such as those described above to allow for tailored selection of whether a set of selected coding parameters for a block of a color sample array, or an auxiliary sample array, or a set of color sample arrays and / or auxiliary sample arrays, is estimated or predicted from previously coded co-located blocks of other sample arrays in the same image, or whether the set of selected coding parameters for a block is coded independently without reference to co-located blocks of other sample arrays in the same image. The decision of whether a set of selected coding parameters for a block of a sample array, or for multiple sample array blocks, is estimated or predicted may be included in the bitstream. Different sample arrays associated with an image need not be the same size.

[0139] As mentioned above, the sample arrays associated with an image (sample arrays can represent color component and / or auxiliary sample arrays) may be arranged into two or more so-called plane groups, with each plane group consisting of one or more sample arrays. The sample arrays included in a particular plane group do not have to be the same size. This arrangement into plane groups also includes the case where each sample array is coded separately.

[0140] Specifically, in an embodiment, for each block in a plane group, coding parameters specifying how the block is predicted may be estimated or predicted from previously coded blocks in the same image in different plane groups, or these coding parameters may be coded separately for the block, depending on the circumstances. The coding parameters specifying how the block is predicted may include one or more of the following coding parameters: a block prediction mode specifying what type of prediction is used for the block (intra prediction, inter prediction using a single motion vector and reference picture, inter prediction using two motion vectors and reference pictures, inter prediction using a higher-order, i.e., non-translational, motion model and a single reference picture, inter prediction using multiple motion models and reference pictures), an intra prediction mode specifying how to generate an intra prediction signal, an identifier specifying the number of prediction signals to be combined to generate a final prediction signal for the block, a reference index specifying the reference picture to be used for motion-compensated prediction, motion parameters (such as displacement vectors or affine motion parameters) specifying how to generate a prediction signal using the reference picture, and an identifier specifying how to filter the reference picture to generate the motion-compensated prediction signal. Generally, a block can be associated with only a subset of the above coding parameters. For example, if the block prediction mode specifies that intra prediction is to be performed on the block, the coding parameters of the block may additionally include an intra prediction mode, but coding parameters such as reference indices and motion parameters that specify how to generate a signal for inter prediction are not specified. Alternatively, if the block prediction mode specifies inter prediction, the associated coding parameters may additionally include reference indices and motion parameters, but the intra prediction mode is not specified.

[0141] One of two or more plane groups may be coded or indicated in the bitstream as a primary plane group. For all blocks in this primary plane group, coding parameters specifying how to generate a prediction signal are transmitted without reference to other plane groups in the same image. The remaining plane groups are coded as secondary plane groups. For each block in the secondary plane group, one or more syntax elements are transmitted that signal whether coding parameters specifying how to predict the block are estimated or predicted from co-located blocks in other plane groups, or whether a new set of these coding parameters is transmitted for the block. One of the one or more syntax elements may be referred to as an inter-plane prediction flag or inter-plane prediction parameters. If a syntax element signals that no estimation or prediction of the corresponding coding parameters is performed, a new set of corresponding coding parameters for the block is transmitted in the bitstream. If a syntax element signals that estimation or prediction of the corresponding coding parameters is performed, a co-located block in a so-called reference plane group is identified. The assignment of reference plane groups to blocks can be configured in several ways. In one embodiment, a specific reference plane group is assigned to each secondary plane group, and this assignment can be fixed or signaled in a high-level syntactic construct such as a set of parameters, access unit header, picture header, slice header, etc.

[0142] In a second embodiment, the reference plane group assignment is coded in the bitstream. The coding parameters are coded for the block and signaled by one or more syntax elements that specify whether the selected coding parameters are estimated, predicted, or coded separately.

[0143] To explain the above-mentioned possibilities in relation to inter-plane prediction and the detailed embodiments below, reference is made to FIG. 11. FIG. 11 illustrates an image 500 composed of three sample arrays 502, 504, and 506. For clarity, only the lower portions of the sample arrays 502-506 are shown in FIG. 11. The sample arrays are shown to spatially correspond to one another. The sample arrays 502-506 overlap one another along a direction 508. Furthermore, when the samples of the sample arrays 502-506 are projected along the direction 508, the samples of all of these sample arrays 502-506 are exactly aligned with one another in spatial position. In other words, the planes 502 and 506 are expanded horizontally and vertically to accommodate each other's spatial resolution and correspond to one another.

[0144] In an embodiment, all sample arrays of an image belong to the same portion of the spatial scene, and the vertical and horizontal resolutions of sample arrays 502-506 may differ. Furthermore, for purposes of illustration, sample arrays 502 and 504 belong to one plane group 510, while sample array 506 belongs to another plane group 512. Also, FIG. 11 illustrates a case where the spatial resolution along the horizontal axis of sample array 504 is twice the horizontal resolution of sample array 502. Furthermore, sample array 504 is considered to be a primary array relative to sample array 502. Sample array 502 is considered to be a subordinate array relative to primary array 504. As previously mentioned, in this case, the subdivision of sample array 504 into blocks determined by subdivision unit 30 of FIG. 1 is adopted by subordinate array 502. 11, the vertical resolution of sample array 502 is half the vertical resolution of linear array 504, so each block is divided into two equal horizontal blocks, which, when measured in units of sample positions within sample array 502, again become quadratic blocks.

[0145] 11, the subdivision selected for the sample array 506 differs from the subdivision of the other plane groups 510. As previously mentioned, the subdivision unit 30 may select the subdivision of the pixel array 506 separately, independently of the subdivision for the plane group 510. Of course, the resolution of the sample array 506 may also differ from the resolution of the planes 502, 504 of the plane group 510.

[0146] Here, when encoding each individual sample array 502-506, the encoder 10 may start by encoding the primary array 504 of the plane group 510, e.g., in the manner described above. The blocks shown in Figure 11 may be, for example, the prediction blocks described above. Alternatively, the blocks may be residual blocks or other blocks that define the granularity for the definition of particular coding parameters. Inter-plane prediction is not limited to the quadtree or multitree subdivision illustrated in Figure 11.

[0147] After transmitting the syntax elements for the primary array 504, the encoder 10 may decide to declare the primary array 504 as the reference plane for the lower plane 502. The encoder 10 and the extractor 30 may each signal this decision via the bitstream 22. Alternatively, the relevance of the sample array 504 may be made clear by the sample array 504 being the primary array of the plane group 510, and this information may be part of the bitstream 22. In either case, for each block in the sample array 502, the inserter 18, or another module of the encoder 10 operating in conjunction with the inserter 18, may decide to suppress transmission of the coding parameters of this block in the bitstream and instead use the coding parameters of the co-located block in the primary array 504 in the bitstream for that block, or signal that the coding parameters of the co-located block in the primary array 504 should be used as predictions for the coding parameters of the current block of the sample array 502, and transmit only the residual data for the current block of the sample array 502 in the bitstream. If the decision is not made, the coding parameters are transmitted in the data stream as usual. The decision is signaled in the data stream 22 for each block. At the decoder side, the extractor 102 uses the inter prediction information for each block to obtain the coding parameters of the respective block of the sample array 502 accordingly, i.e., by estimating the coding parameters of the co-located block of the primary array 504. Alternatively, if the inter prediction information indicates inter prediction, this is done by extracting residual data for the block from the data stream and combining this residual data with the prediction obtained from the coding parameters of the co-located block of the primary array 504. Alternatively, this is done by extracting the coding parameters of the current block of the sample array 502 as usual, regardless of the primary array 504.

[0148] As mentioned above, the location of the reference plane is not limited to the same plane group as the block currently being inter-predicted. Therefore, as mentioned above, the plane group 510 may be the primary plane group or the reference plane group for the secondary plane group 512. In this case, the bitstream may include, for each block in the sample array 506, a syntax element indicating whether the coding parameters of the co-located macroblock in either the primary plane group or the reference plane group 502, 504 are to be adopted / predicted as described above. In the latter case, the coding parameters of the current block in the sample array 506 are transmitted as usual.

[0149] The subdivision and / or prediction parameters for planes within a plane group may be identical, i.e., they are only coded once for the plane group (all secondary planes of a plane group derive their subdivision information and / or prediction parameters from the primary planes in the same plane group), and contextual prediction or estimation of the subdivision information and / or prediction parameters occurs between plane groups.

[0150] The reference plane group may be a primary plane group or a secondary plane group.

[0151] Except for the further partitioning described above to divide adopted leaf blocks into quadratic blocks, the subdivision of the primary sample array 504 is spatially performed by the lower sample array 502, so it is easy to understand that blocks of different planes within a plane group are co-located. In the case of inter-plane adoption / prediction between different plane groups, this co-location may be defined to allow for more flexibility between the subdivision of these plane groups. Given a reference plane group, the co-located blocks within the reference plane group are identified. The co-located blocks and reference plane group can be derived by the following process: A specific sample 514 within a current block 516 of one of the sample arrays 506 of the secondary plane group 512 is selected. This may be the top-left sample of the current block 516, as illustrated by 514 in FIG. 11 . It may also be a sample within the current block 516 that is closer to the center of the current block 516, or any other sample within the current block, as long as it is geometrically uniquely defined. The location of this selected sample 515 within the sample arrays 502, 504 of the reference plane group 510 is calculated. The location of sample 514 within sample arrays 502 and 504 is shown at 518 and 520, respectively, in Figure 11. Which plane 502 or 504 of reference plane group 510 is actually used may be pre-specified or signaled in the bitstream. The sample in the corresponding sample array 502 or 504 of reference plane group 510 that is closest to location 518 or 520, respectively, is identified, and the block containing this sample is selected as the co-located block within sample arrays 502 and 504, respectively. In the case of Figure 11, these are blocks 522 and 524, respectively. Alternative approaches for identifying co-located blocks in other planes are described below.

[0152] In an embodiment, the coding parameters specifying the prediction for the current block 516 are fully estimated using corresponding prediction parameters of the co-located blocks 522 / 524 in a different plane group 510 of the same image 500, without transmitting any additional side information. This estimation is performed by simply copying the corresponding coding parameters or by adapting the coding parameters to account for differences between the current plane group 512 and the reference plane group 510. For example, this adaptation may involve correcting the motion parameters (e.g., correcting the displacement vectors) to account for phase differences between the luma and chroma sample arrays. This adaptation may also involve modifying the precision of the motion parameters (e.g., modifying the displacement vector precision) to account for different resolutions of the luma and chroma sample arrays. In another embodiment, one or more estimated coding parameters for specifying the generation of the prediction signal are not directly used for the current block 516, but are used as predictions of the corresponding coding parameters for the current block 516, and refinement of these coding parameters for the current block 516 is transmitted in the bitstream 22. For example, the estimated motion parameters are not used directly, but motion parameter differences (such as displacement vector differences) are coded in the bitstream, specifying the deviation between the motion parameters used for the current block 516 and the estimated motion parameters. At the decoder side, the actual motion parameters used are obtained by combining the differences between the estimated and transmitted motion parameters.

[0153] In another embodiment, the subdivision of blocks, such as the above-described predictive subdivision treeblocks, into prediction blocks (i.e., blocks of examples for which the same set of prediction parameters is used) is estimated or predicted from previously coded co-located blocks in different plane groups of the same image, i.e., the bit sequences of Figures 6A or 6B, as appropriate. In this embodiment, one of two or more plane groups is coded as a primary plane group. For all blocks in this primary plane group, subdivision information is transmitted without reference to other plane groups of the same image. The remaining plane groups are coded as secondary plane groups. For blocks in the secondary plane groups, one or more syntax elements are transmitted that signal whether subdivision information is estimated or predicted from co-located blocks in other plane groups or whether subdivision information is transmitted in the bitstream. One of the one or more syntax elements may be referred to as an inter-plane prediction flag or inter-plane prediction parameter. If a syntax element signals that subdivision information is not estimated or predicted, then the subdivision information for the block is transmitted in the bitstream without reference to other plane groups of the same image. If a syntax element signals that subdivision information is estimated or predicted, it identifies a co-located block in a so-called reference plane group. The assignment of reference plane groups to blocks can be configured in several ways. In one embodiment, a specific reference plane group is assigned to each secondary plane group. This assignment can be fixed or signaled in a high-level syntax construct such as a set of parameters, an access unit header, a picture header, or a slice header. In a second embodiment, the assignment of reference plane groups is coded in the bitstream and signaled by one or more syntax elements. This syntax element is coded for the block and specifies whether subdivision information is estimated, predicted, or coded separately. The reference plane group can be a primary plane group or another secondary plane group.Given a reference plane group, co-located blocks within the reference plane group are identified. A co-located block is a block within the reference plane group that corresponds to the same image area as the current block or represents a block within the reference plane group that shares the largest portion of image area with the current block. The co-located blocks can be partitioned into smaller prediction blocks.

[0154] In other embodiments, subdivision information for a current block, such as the quadtree-based subdivision information of FIG. 6A or 6B, is fully estimated using subdivision information of co-located blocks in different plane groups of the same image, without transmitting additional accompanying information. As one particular example, if the co-located block is partitioned into two or four prediction blocks, the current block is also partitioned into two or four sub-blocks for prediction purposes. As another particular example, if the co-located block is partitioned into four sub-blocks, and one of these sub-blocks is further partitioned into four smaller sub-blocks, the current block is also partitioned into four sub-blocks, and one of these sub-blocks (corresponding to the sub-blocks of the co-located block that are further decomposed) is also partitioned into four smaller sub-blocks. In other preferred embodiments, the estimated subdivision information is not used directly for the current block, but is used as a prediction of the actual subdivision information for the current block, and corresponding refinement information is transmitted in the bitstream. For example, the subdivision information estimated from the co-located block may be further refined. For each sub-block that corresponds to a sub-block in a co-located block that is not partitioned into smaller blocks, a syntax element can be coded in the bitstream that specifies whether the sub-block is further decomposed in the current plane group. The transmission of such a syntax element can be conditioned on the size of the sub-block. Alternatively, a signal can be sent in the bitstream indicating that a sub-block that is further partitioned in the reference plane group will not be partitioned into smaller blocks in the current plane group.

[0155] In other embodiments, both the subdivision of a block into prediction blocks and the coding parameters specifying how its sub-blocks are predicted are adaptively estimated or predicted from previously coded co-located blocks in different plane groups of the same image. In a preferred embodiment of the present invention, one of two or more plane groups is coded as a primary plane group. For all blocks in this primary plane group, subdivision information and prediction parameters are transmitted without reference to other plane groups of the same image. The remaining plane groups are coded as secondary plane groups. For blocks in the secondary plane groups, one or more syntax elements are transmitted that signal whether subdivision information and prediction parameters are estimated or predicted from co-located blocks in other plane groups or whether subdivision information and prediction parameters are transmitted in the bitstream. One of the one or more syntax elements may be referred to as an inter-plane prediction flag or inter-plane prediction parameters. If a syntax element signals that subdivision information and prediction parameters are not estimated or predicted, the subdivision information of the block and prediction parameters for the resulting sub-blocks are transmitted in the bitstream without reference to other plane groups of the same image. If a syntax element signals that subdivision information and prediction parameters for a sub-block are estimated or predicted, then a co-located block in a so-called reference plane group is identified. The assignment of reference plane groups to blocks can be configured in several ways. In one embodiment, a specific reference plane group is assigned to each secondary plane group. This assignment can be fixed or signaled in a high-level syntax construct such as a parameter set, an access unit header, a picture header, or a slice header. In a second embodiment, the assignment of reference plane groups is coded in the bitstream and signaled by one or more syntax elements. This syntax element is coded for the block and specifies whether the subdivision information and prediction parameters are estimated or predicted, or coded separately.The reference plane group may be a primary plane group or another secondary plane group. Given a reference plane group, a co-located block within the reference plane group is identified. The co-located block may be a block within the reference plane group that corresponds to the same image region as the current block, or a block representing a block within the reference plane group that shares the largest portion of the image region with the current block. The co-located block may be partitioned into smaller prediction blocks. In a preferred embodiment, the subdivision information for the current block and the prediction parameters for the resulting sub-blocks are fully estimated using the subdivision information of co-located blocks within different plane groups of the same image and the prediction parameters of the corresponding sub-blocks, without transmitting any additional ancillary information. As one specific example, if the co-located block is partitioned into two or four prediction blocks, the current block is also partitioned into two or four sub-blocks for prediction purposes, and prediction parameters for the sub-blocks of the current block are derived as described above. As another example, if a co-located block is partitioned into four sub-blocks and one of these sub-blocks is further partitioned into four smaller sub-blocks, then the current block is also partitioned into four sub-blocks, one of these sub-blocks (corresponding to a sub-block of the further decomposed co-located block) is also partitioned into four smaller sub-blocks, and prediction parameters for all sub-blocks that are not further partitioned are estimated as described above. In another preferred embodiment, the subdivision information is fully estimated based on the subdivision information of the co-located block in the reference plane group, but the prediction parameters estimated for a sub-block are only used as a prediction of the sub-block's actual prediction parameters. The deviation between the actual prediction parameters and the estimated prediction parameters is coded in the bitstream. In another embodiment, the estimated subdivision information is used as a prediction for the current block's actual sub-division information and the difference is transmitted in the bitstream (as described above), but the prediction parameters are fully estimated.In another embodiment, both the estimated subdivision information and the estimated prediction parameters are used as predictions and the difference between the actual subdivision information and prediction parameters and the estimated values ​​is transmitted in the bitstream.

[0156] In another embodiment, for blocks of a plane group, the mode of residual coding (e.g., transform type) is adaptively selected whether it is estimated or predicted from co-located coded blocks of different plane groups of the same image, or whether the mode of residual coding is coded separately for the blocks. This embodiment is similar to the adaptive estimation / prediction of prediction parameters described above.

[0157] In another embodiment, the subdivision of a block (e.g., a prediction block) into transform blocks (i.e., blocks of samples to which a 2D transform is applied) is adaptively estimated or predicted from co-located coded blocks of a different plane group of the same image. This embodiment is similar to the above-described embodiment in which the estimation / prediction of the subdivision into prediction blocks is adaptive.

[0158] In another embodiment, the subdivision of a block into transform blocks and the mode of coding of the residuals for the resulting transform blocks (e.g., transform type) are adaptively estimated or predicted from co-located coded blocks of a different plane group of the same image. This embodiment is similar to the above-described embodiment of adaptive subdivision into prediction blocks and estimation / prediction of prediction parameters for the resulting prediction blocks.

[0159] In another embodiment, the subdivision of the block into prediction blocks, the associated prediction parameters, the subdivision information of the prediction block and the mode of coding of the residual for the transform block are adaptively estimated or predicted from previously coded co-located blocks of a different plane group of the same image. This embodiment is a combination of the above-mentioned embodiments. It is also possible to estimate or predict only some of the above-mentioned coding parameters.

[0160] Thus, as mentioned above, inter-plane adoption / prediction can potentially improve coding efficiency, however, this is also possible when other block subdivisions than multi-tree based subdivisions are used, regardless of whether block merging is performed or not.

[0161] The embodiments outlined above regarding inter-plane matching / prediction are applicable to image and video encoders and decoders. The encoder and decoder divide the color planes of an image, and, if present, auxiliary sample arrays associated with the image, into blocks and associate these blocks with coding parameters. For each block, a set of coding parameters may be included in the bitstream. For example, these coding parameters may describe how the block is predicted or decoded at the decoder side. For example, the coding parameters may represent the prediction mode of a macroblock or block, subdivision information, intra-prediction mode, reference indices used for motion-compensated prediction, motion parameters such as displacement vectors, mode of coding of residuals, transform coefficients, etc. The sample arrays associated with an image may have different sizes.

[0162] Next, we will describe an extended signaling scheme for coding parameters in a tree-based partitioning scheme, such as that described above with reference to Figures 1 to 8. The scheme described below can be combined with any one or combination of the above embodiments, but the effects and advantages of the extended signaling scheme, as well as the other schemes, namely, join and inter-plane adoption / prediction, will be described independently of the above embodiments. This scheme will be referred to as inheritance hereinafter.

[0163] An improved coding scheme for encoding ancillary information in tree-based partitioning schemes, called inheritance, is described below, which generally provides the following advantages over conventional schemes for handling coding parameters:

[0164] Typically, in traditional image and video coding, an image, or a set of specific sample arrays within an image, is decomposed into blocks and associated with specific coding parameters. An image typically consists of multiple sample arrays. Furthermore, an image may be associated with additional auxiliary sample arrays that specify, for example, transparency information or depth maps. The sample arrays (including the auxiliary sample arrays) of an image can be grouped into one or more so-called plane groups, each consisting of one or more sample arrays. A plane group of an image can be coded independently or, if the image is associated with multiple plane groups, can be coded by prediction from other plane groups of the same image. Each plane group is typically decomposed into blocks. Blocks (or corresponding blocks of sample arrays) are predicted either by inter-picture prediction or intra-picture prediction. Blocks can be of various sizes and are either square or rectangular. The partitioning of an image into blocks can be fixed by syntax or (at least in part) signaled within the bitstream. Syntax elements that signal subdivision information for blocks of a given size are often transmitted. These syntax elements allow specifying whether and how a block is to be subdivided into smaller blocks and associated with coding parameters for prediction purposes, etc. For every sample of a block (or the corresponding block of a sample array), the decoding of the associated coding parameters is specified in a specific way. For example, all samples in a block are predicted using the same set of prediction parameters, such as reference indices (which identify a reference picture in a set of previously coded pictures), motion parameters (which specify the amount of block motion between the reference picture and the current picture), and parameters for specifying interpolation filters, intra-prediction modes, etc. Motion parameters can be represented by displacement vectors with horizontal and vertical components, or by higher-order motion parameters, such as affine motion parameters with six components. It is also possible to associate multiple sets of specific prediction parameters (such as reference indices and motion parameters) with a single block.In this case, for each set of these specific prediction parameters, a single intermediate prediction signal for the block (or the corresponding block in the sample array) is generated, and the final prediction signal is formed by combining the intermediate prediction signals, including superposition. The corresponding weighting parameters, possibly with a constant offset (added to the weighted sum), can be fixed for either the image, the reference image, or the set of reference images, or can be included in the set of prediction parameters for the corresponding block. The difference between the original block (or the corresponding block in the sample array) and the prediction signal, also known as the residual signal, is typically transformed and quantized. A two-dimensional transform is often applied to the residual signal (or the corresponding sample array of the residual block). For transform coding, a block (or the corresponding block in the sample array) for which a specific set of prediction parameters is used can be further divided before applying the transform. The transform block can be the same size as the block used for prediction, or smaller. A transform block can also contain multiple blocks used for prediction. Each transform block can be of various sizes, resulting in a square or rectangular block. After transformation, the resulting transform coefficients are quantized to obtain so-called transform coefficient levels. The transform coefficient levels and prediction parameters, and subdivision information, if present, are entropy coded.

[0165] In some image and video coding standards, the syntax provides very limited possibilities for subdividing an image (or a group of planes) into blocks. It is usually possible to specify only whether and, possibly, how a block of a given size can be subdivided into smaller blocks. For example, in H.264, the maximum block size is 16x16. The 16x16 blocks are also called macroblocks, and each image is partitioned into macroblocks in the first step. For each 16x16 macroblock, it is possible to signal whether it will be coded as a 16x16 block, two 16x8 blocks, two 8x16 blocks, or four 8x8 blocks. If a 16x16 block is subdivided into four 8x8 blocks, each of these 8x8 blocks can be coded as one 8x8 block, two 8x4 blocks, two 4x8 blocks, or four 4x4 blocks. In current image and video coding standards, reducing the possibilities for specifying partitioning into blocks has the advantage of keeping the side information rate for conveying subdivision information small, but the disadvantage, as explained below, is that the bit rate required for transmitting prediction parameters for a block can be large. The side information rate for conveying prediction information typically accounts for a significant portion of the overall bit rate for a block. Therefore, reducing this side information can improve coding efficiency, which can be achieved, for example, by using larger block sizes. Images in real-world images and video sequences consist of objects of arbitrary shapes with specific properties. For example, these objects or object parts are characterized by a specific texture or a specific motion. The same set of prediction parameters is usually applied to these objects or object parts. However, object boundaries usually do not coincide with the boundaries of larger prediction blocks (e.g., 16x16 macroblocks in H.264). The encoder typically makes subdivision decisions (among a limited set of possibilities) that minimize a certain rate / distortion loss.This can result in a large number of small blocks for an object of arbitrary shape. Because each small block is associated with a set of prediction parameters that need to be transmitted, the ancillary information rate can become a significant portion of the overall bit rate. However, because some of the small blocks still represent the same object or a region of an object, the prediction parameters of the resulting blocks will be identical or very similar. Coding efficiency would be improved if the syntax were extended in a way that not only allows for block subdivision but also allows for coding parameter sharing among the resulting blocks after subdivision. In tree-based subdivision, coding parameter sharing can be achieved for a given set of blocks by assigning coding parameters, or parts of them, to one or more parent nodes in a tree-based hierarchy. As a result, the shared parameters, or parts of them, can be used to reduce the ancillary information needed to inform the actual selection of coding parameters for the resulting blocks after subdivision. This can be achieved by omitting to signal parameters for subsequent blocks or by using shared parameters for parameter prediction and / or context modeling for subsequent blocks.

[0166] The basic idea of ​​the inheritance scheme described below is to reduce the bit rate required to transmit the coding parameters by sharing information according to a tree-based hierarchy of blocks. The shared information signal is transmitted in the bitstream (in addition to the subdivision information). The advantage of the inheritance scheme is an increase in coding efficiency due to the reduced information rate associated with the coding parameters.

[0167] To reduce the associated information rate, in the embodiments described below, the respective coding parameters for a particular sample set, i.e., a simply connected region, of a multi-tree subdivision are transmitted in the data stream in an efficient manner. A particular sample set, i.e., a simply connected region, may represent a rectangular or square block, or an area of ​​any shape, or any other sample set. The inheritance scheme described below eliminates the need to explicitly include the coding parameters for each of these sample sets in the bitstream. The coding parameters may also represent prediction parameters that specify how to predict the corresponding sample set using previously coded samples. The numerous possibilities and examples described above also apply here. As indicated above and further described below, with regard to the inheritance scheme described below, the tree-based partitioning of the image sample array into sample sets may be fixed by the syntax or may be conveyed by the corresponding subdivision information in the bitstream. As mentioned above, the coding parameters for the sample sets may be transmitted in a predetermined order specified by the syntax.

[0168] According to the inheritance scheme, the decoder or the decoder's extractor 102 is configured to derive information about the coding parameters of individual singly connected regions or sample sets in a specific way. In particular, coding parameters, or parts thereof, such as parameters used for prediction purposes, are shared between blocks according to a given tree-based partitioning scheme, with the sharing groups according to a tree structure determined by the encoder or inserter 18. In a specific embodiment, the sharing of coding parameters for all child nodes of a given internal node of the partitioning tree is indicated using a specific sharing flag with a binary value. As an alternative approach, coding parameter refinements can be transmitted for each node, such that cumulative parameter refinements according to the tree-based block hierarchy can be applied to all sample sets of the block of a given leaf node. In another embodiment, parts of the coding parameters transmitted for internal nodes according to the tree-based block hierarchy can be used for context-adaptive entropy encoding and decoding of the coding parameters, or parts thereof, for the block of a given leaf node.

[0169] Figures 12A and 12B illustrate the basic concept of inheritance in the specific case of using a quadtree-based partitioning. However, as noted above, other multi-tree subdivisions may be used. Figure 12A illustrates a tree structure, and Figure 12B illustrates a spatial partitioning corresponding to the tree structure of Figure 12A. The partitioning shown is similar to that shown in Figures 3A through 3C. In general, inheritance schemes allow for the assignment of associated information to nodes at different non-leaf levels within a tree structure. Depending on the assignment of associated information to nodes at different levels within the tree, such as the internal nodes and the root node within the tree of Figure 12A, varying degrees of sharing of associated information can be achieved within the tree hierarchy of blocks shown in Figure 12B. For example, if all leaf nodes in the fourth layer, all of which have the same parent node in the case of FIG. 12A, are determined to share ancillary information, this essentially means that the smallest blocks, designated 156a through 156d in FIG. 12B, share this ancillary information, eliminating the need to transmit the ancillary information for each of these small blocks 156a through 156d, i.e., four times. However, this remains an option for the encoder. Alternatively, the entire area of ​​hierarchical level 1 (second layer) in FIG. 12A, i.e., the upper right quarter of treeblock 150, including subblocks 154a, 154b, and 154d and the smaller subblocks 156a through 156d, could be determined as the area in which coding parameters are shared. Thus, the area in which ancillary information is shared increases. The next level of increase is to combine all subblocks in the first layer, i.e., subblocks 152a, 152c, and 152d, and the smaller blocks. In other words, in this case, the entire treeblock has associated information assigned to it, and all sub-blocks of this treeblock 150 share the associated information.

[0170] In the following description of inheritance, the following notation is used to describe the embodiment.

[0171] a. The reconstructed sample of the current leaf node: r b. Reconstructed specimen of adjacent leaf: r' c. Predictor of the current leaf node: p d. Residual of the current leaf node: Res e. Reconstructed residual of the current leaf node: RecRes f. Scaling and inverse transformation: SIT g. Shared flag: f

[0172] As a first example of inheritance, we consider signaling intra prediction at internal nodes. Specifically, we consider how intra prediction modes are signaled at internal nodes for tree-based block partitioning for prediction purposes. Internal nodes (including the root node) can propagate some of the associated information used by corresponding child nodes by traversing the tree from the root node to the leaf nodes. Specifically, for internal nodes, a shared flag f is transmitted with the following meaning:

[0173] If f has a value of 1 ("true"), all child nodes of a given internal node share the same intra prediction mode. In addition to the sharing flag f having a value of 1, the internal node also propagates the parameters of the intra prediction mode used by all its child nodes. Therefore, all subsequent child nodes do not transmit any prediction mode information or sharing flag. When reconstructing all associated leaf nodes, the decoder applies the intra prediction mode from the corresponding internal node.

[0174] If the value of f is 0 ("false"), then the child nodes of the corresponding internal node do not share the same intra-prediction mode, and each child node that is an internal node transmits a separate sharing flag.

[0175] Figure 12C illustrates the signaling of intra prediction at the internal nodes mentioned above: the internal nodes in the first layer convey the shared flag and the side information provided by the intra prediction mode information, while the child nodes do not transmit any side information.

[0176] As a second example of inheritance, we will explain the refinement of inter prediction. Specifically, we will explain how to propagate ancillary information of inter prediction mode in intra mode of tree-based block partitioning for the purpose of refining motion parameters provided by motion vectors, etc. Internal nodes (including the root node) can propagate part of the ancillary information that is refined in the corresponding child node by traversing the tree from the root node to the leaf nodes. Specifically, for internal nodes, a shared flag f is transmitted with the following meaning:

[0177] If f has the value 1 ('true'), all child nodes of a given internal node share the same motion vector reference. In addition to a sharing flag f with value 1, the internal node also transmits a motion vector and a reference index. All subsequent child nodes then do not transmit any further sharing flags, but may transmit a refinement of this inherited motion vector reference. Upon reconstruction of all associated leaf nodes, the decoder adds the motion vector refinement of a given leaf node to the inherited motion vector reference belonging to the corresponding internal parent node with a sharing flag f with value 1. This means that the motion vector refinement of a given leaf node is the difference between the actual motion vector applied at this leaf node for motion compensated prediction and the motion vector reference of the corresponding internal parent node.

[0178] If the value of f is 0 ("false"), the child nodes of the corresponding internal node do not necessarily share the same inter prediction mode, and no motion parameter refinement is performed at the child nodes by using the motion parameters from the corresponding internal node. Each child node that is an internal node transmits a separate sharing flag.

[0179] 12D illustrates the refinement of the motion parameters described above. The internal nodes in the first layer transmit the shared flag and associated information. The leaf child nodes transmit only the refinement of the motion parameters, without the associated information transmitted by the internal child nodes in the second layer, for example.

[0180] Reference is now made to Figure 13. Figure 13 is a flowchart illustrating a mode of operation of a decoder, such as the decoder of Figure 2, in reconstructing an array of information samples representing a spatial example information signal from a data stream. The array of information samples is subdivided into leaf areas of different sizes by a multi-tree subdivision. As explained above, each leaf area is associated with one hierarchical level of a series of hierarchical levels of the multi-tree subdivision. For example, all blocks shown in Figure 12B are leaf areas. For example, leaf area 156c is associated with layer 4 (or level 3) of the hierarchy. Each leaf area is associated with coding parameters. Examples of these coding parameters are explained above. The coding parameters are represented by a respective set of syntax elements for each leaf area. Each syntax element has a respective syntax element type of one of a series of syntax element types. For example, such syntax element types can be a prediction mode, a component of a motion vector, an indication of an intra-prediction mode, etc. The decoder performs the following steps according to Figure 13:

[0181] In step 550, inheritance information is extracted from the data stream. In the case of Fig. 2, the extractor 102 performs step 550. The inheritance information indicates whether inheritance is used for the current arrangement of information samples. The following description will reveal several possibilities for the inheritance information, such as a sharing flag f and information propagation for a multi-tree structure divided into primary and secondary parts.

[0182] The array of information samples may already be part of the image, such as a treeblock, i.e., treeblock 150 in Figure 12B. Thus, inheritance information indicates whether inheritance is used for a particular treeblock 150. Such inheritance information may be inserted into the data stream, for example, for all treeblocks of a prediction subdivision.

[0183] Furthermore, if inheritance is indicated, the inheritance information indicates at least one inheritance region of the information sample array. This inheritance region is composed of a set of leaf regions and corresponds to one hierarchical level of the series of multi-tree subdivisions located below each hierarchical level associated with the set of leaf regions. In other words, the inheritance information indicates whether inheritance is used for the current sample array, such as tree block 150. If inheritance is used, at least one inheritance region or sub-region of tree block 150 is indicated, within which leaf regions share coding parameters. Thus, the inheritance region does not have to be a leaf region. In the example of FIG. 12B, this inheritance region may be, for example, the region formed by sub-blocks 156a to 156b. Alternatively, the inheritance region may be larger and include sub-blocks 154a, 154b, and 154d. Furthermore, the inheritance region may be tree block 150 itself, with all leaf blocks sharing the coding parameters associated with the inheritance region.

[0184] Additionally, multiple regions of inheritance may be defined within each sample array or treeblock 150. For example, if the bottom-left subblock 152c is also partitioned into smaller blocks, then subblock 152c can also form a region of inheritance.

[0185] In step 552, the inheritance information is checked to determine whether inheritance is used. If inheritance is used, the process of FIG. 13 proceeds to step 554, where an inheritance subset including at least one syntax element of a predetermined syntax element type is extracted from the data stream for each inter-plane inheritance region. In the following step 556, this inheritance subset is copied to or used as a prediction of a corresponding inheritance subset of syntax elements in a set of syntax elements representing coding parameters associated with a set of leaf regions constituting each of the at least one inheritance region. In other words, for each inheritance region indicated in the inheritance information, the data stream constitutes an inheritance subset of syntax elements. In other words, the inheritance relates to at least one specific syntax element type or category that can be used for inheritance. For example, syntax elements of a prediction mode, an inter-plane prediction mode, or an intra-plane prediction mode may be subject to inheritance. For example, the inheritance subset included in the data stream for the inheritance region may include a syntax element of an inter-plane prediction mode. The inheritance subset may also include additional syntax elements, whose types depend on the value of the type of the fixed syntax element associated with the inheritance scheme. For example, if an inter-plane prediction mode is a fixed component of the inheritance subset, syntax elements defining motion compensation, such as components of a motion vector, may or may not be included in the inheritance subset, depending on the syntax. For example, if the upper right quarter of treeblock 150, i.e., subblock 152b, is the inheritance region, the inheritance region may indicate only the inter-plane prediction mode, or the inter-plane prediction mode, a motion vector, and a motion vector index.

[0186] All syntax elements contained in a subset of the inheritance are copied to or used as predictions for the corresponding coding parameters of the leaf blocks in that region of the inheritance, i.e., leaf blocks 154a, 154b, 154d, and 156a to 156d. If prediction is used, a residual is transmitted for each leaf block.

[0187] One possibility for transmitting inheritance information for tree block 150 is the transmission of the above-mentioned sharing flag f. In this case, extracting the inheritance information in step 550 includes the following: In particular, the decoder is configured to extract sharing flags f from the data stream for non-leaf regions corresponding to any of the inheritance sets of at least one hierarchical level of the multi-tree subdivision, using the hierarchical level order from lower to higher, and to check whether each inheritance or sharing flag indicates inheritance. For example, the inheritance set of hierarchical levels is formed by the first to third layers of FIG. 12A. Therefore, any of the nodes of the subtree structures located in any of the first to third layers, rather than leaf nodes, have an associated sharing flag in the data stream. The decoder extracts these sharing flags in the order from the first to third layers, in a traversal order such as depth order or width order. If one of the sharing flags is 1, the decoder recognizes that the leaf blocks included in the corresponding inheritance region share the inheritance subset subsequently extracted in step 554. There is no need to check the inheritance flags for the child nodes of the current node, in other words, the inheritance flags for these child nodes are not transmitted in the data stream, since it is clear that the domain of these nodes already belongs to the domain of inheritance with which an inheritance subset of syntax elements is shared.

[0188] The sharing flag f can be alternated with bits conveying subdivision of the quadtree described above. For example, an alternating bit sequence containing both subdivision and sharing flags can be shown as follows:

[0189]

number

[0190] This is the same subdivision information as shown in Figure 6A, but with the addition of two underlined sharing flags to indicate that all subblocks in the lower left quadrant of treeblock 150 in Figure 3C share coding parameters.

[0191] Another way to define the inheritance information indicating the inheritance domain is to use two subdivisions defined in a subordinate manner to each other, as described above for the prediction and residual subdivisions, respectively. In general, the leaf blocks of a primary subdivision can form inheritance domains that define the domains where inheritance subsets of syntax elements are shared. At the same time, the blocks within these inheritance domains where inheritance subsets of syntax elements are copied or used as predictions are defined by subordinate subdivisions.

[0192] For example, consider the residual tree as an extension of the prediction tree. Furthermore, consider the case where the prediction block can be further divided into smaller blocks for the purpose of residual coding. For each prediction block corresponding to a leaf node of the prediction-related quadtree, one or more subordinate quadtrees determine the corresponding subdivision for residual coding.

[0193] In this case, instead of using the signaling of predictions at the internal nodes, the residual tree is interpreted as follows: a refinement of the prediction tree is also specified in the sense of using a certain prediction mode (signaled by the corresponding leaf node of the tree related to the prediction) and a refined reference example is also used. The following example illustrates this case.

[0194] For example, Figures 14A and 14B show quadtree partitioning for intra-prediction. For one particular leaf node of the primary subdivision, neighboring reference samples are highlighted. Figure 14B shows the residual quadtree subdivision for the same predicted leaf node, along with refined reference samples. All sub-blocks shown in Figure 14B share the same intra-prediction parameters contained in the data stream for each leaf block highlighted in Figure 14A. Thus, Figure 14A shows an example of a conventional quadtree partitioning for intra-prediction, with reference samples shown for one particular leaf node. However, in a preferred embodiment, a separate intra-prediction signal is calculated for each leaf node in the residual tree, e.g., by using neighboring samples of the reconstructed leaf node in the residual tree, as shown by the gray stripe shading in Figure 14B. The quantized residual signal is then added to this prediction signal to obtain a reconstructed signal for a given residual leaf node in the usual way. This reconstructed signal is used as a reference signal for subsequent prediction processes. The decoding order for prediction is the same as the decoding order for the residual.

[0195] As shown in Figure 15, during the decoding process, for each leaf node of the residual, a prediction signal p is calculated using the reference sample r' according to the actual intra prediction mode (as indicated by the leaf node of the quadtree associated with the prediction).

[0196] SIT processing,

[0197]

number

[0198] After that, the reconstructed signal r is calculated and stored for processing to calculate the next prediction.

[0199]

number

[0200] The decoding order for prediction is the same as the decoding order for residuals shown in FIG.

[0201] Each of the residual leaf nodes is decoded as described in the previous paragraph. The reconstructed signal r is stored in a buffer, as shown in Figure 16. From this buffer, the reference sample r' is taken for the next prediction and decoding process.

[0202] While specific embodiments have been described with reference to Figures 1 through 16 by combining various aspects outlined above, other embodiments of the present application will now be described, which emphasize specific aspects described above and integrate some of the above-described embodiments. In particular, the embodiments described above based on the configurations of Figures 1 and 2 primarily combine many aspects of the present application and can be used in other applications and other coding fields with advantages. As reiterated above, for example, multi-tree subdivision may be used without combining and / or inter-plane adoption / prediction and / or inheritance. For example, aspects such as transmitting a maximum block size, using a depth-order traversal order, adapting context according to the hierarchical level of each subdivision flag, and transmitting a maximum hierarchical level in the bitstream, all provide advantages independently of each other, with the aim of reducing the bit rate of side information. This also applies when considering inter-plane implementation schemes. Benefits are obtained from inter-plane implementation regardless of the detailed manner in which an image is subdivided into simply connected regions, and benefits are obtained regardless of the combining scheme and / or inheritance. The same applies to the benefits obtained from combining and inheritance.

[0203] Therefore, taking the embodiments together, coding schemes using the inheritance scheme outlined above are not limited to hybrid coding environments. That is, reconstruction may occur without prediction. Inherited coding parameters may be related to other coding parameters, such as filter detail instructions. As noted above, the subdivision of the information sample array into simply connected regions results from quadtree subdivision and may be square or rectangular in shape. Furthermore, the embodiments specifically described for the subdivision of the sample array are merely specific, and other subdivisions may be used. Some possibilities are illustrated in Figures 17A and 17B. For example, Figure 17A illustrates the subdivision of a sample array 606 into a regular two-dimensional array of non-overlapping, adjacent treeblocks 608, some of which are further subdivided into subblocks 610 of various sizes according to a multi-tree structure. As noted above, while Figure 17A illustrates a quadtree subdivision, partitioning each parent node into other numbers of child nodes is also possible. In the embodiment shown in FIG. 17B, multi-tree subdivision is applied directly to the entire pixel array 606, thereby subdividing the sample array 606 into sub-blocks of various sizes. That is, the entire pixel array 606 is treated as a tree block. In both the subdivisions shown in FIGS. 17A and 17B, the sample array 606 is subdivided into simply connected regions without overlap, for example, according to the embodiments of FIGS. 17A and 17B. However, several alternatives are possible. For example, blocks may overlap each other, but each block may be restricted to having portions that do not overlap with any of its neighboring blocks. Alternatively, adjacent blocks aligned with the current block in a given direction may be restricted to having at most one block overlap each sample of the block. The latter condition means that the left and right neighboring blocks overlap the current block so that they completely cover it, but do not overlap each other, and similarly for vertical and diagonal neighboring blocks.

[0204] As discussed above with reference to Figures 1 to 16, the array of information samples does not necessarily represent a moving image or a still image, but may instead represent a depth map or transparency map of a particular scene.

[0205] The determination of the coding parameters and the inheritance information may be an iterative process, for example, where it is determined that it is desirable to tolerate small differences between the coding parameters of adjacent, previously rate-distortion-optimally determined, simply connected regions that belong to similar, previously rate-distortion-optimally determined parent regions, rather than signaling them to the decoder, considering that inheritance would completely suppress the explicit transmission of coding parameters for all of these simply connected regions, allowing only the presentation of residuals or the transmission of shared coding parameters instead of presenting all of these coding parameters.

[0206] Although some aspects have been described in terms of an apparatus, it is clear that these aspects also serve as descriptions of corresponding methods, with blocks or devices corresponding to means or features of means. Similarly, aspects described in terms of means also serve as descriptions of corresponding blocks or items or functions of the corresponding apparatus. Some or all of the means may be implemented by (or using) hardware apparatuses such as microprocessors, programmable computers, electronic circuits, etc. In some embodiments, one or more of the most important means may be implemented by such apparatuses.

[0207] Signals encoded / compressed according to the present invention can be stored on a digital storage medium or transmitted over a transmission medium, such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0208] Embodiments of the present invention may be implemented in hardware or software, depending on specific implementation requirements. For example, they may be implemented using digital storage media, such as floppy disks, DVDs, Blu-ray Discs, CDs, ROMs, PROMs, EPROMs, EEPROMs, flash memories, etc., on which electronically readable control signals are stored and which are (or can be used with) a programmable computer system to perform the respective methods. Thus, the digital storage media may be computer-readable.

[0209] Some embodiments of the present invention include a data storage medium storing electronically readable control signals that can be used with a programmable computer system to perform one of the methods described herein.

[0210] In general, embodiments of the present invention may be implemented as a computer program product including program code that, when executed on a computer, performs one of the methods. The program code may, for example, be stored on a machine-readable storage medium.

[0211] In another embodiment, the computer program for performing one of the methods described herein is included and stored on a machine-readable storage medium.

[0212] In other words, an embodiment of the inventive method is, therefore, a computer program comprising a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0213] Therefore, a further embodiment of the inventive method is a data storage medium (or digital storage medium or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein.

[0214] A further embodiment of the inventive methods is therefore a data stream or series of signals representing a computer program for performing one of the methods described herein, for example a data stream or series of signals adapted to be transmitted over a data communications connection such as the Internet.

[0215] Other embodiments comprise a processing means, such as a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0216] Another embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0217] In some embodiments, a programmable logic device (such as a field programmable gate array) may be used to implement some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may interface with a microprocessor to perform one of the methods described herein. In general, it is desirable for the methods to be performed by a hardware apparatus.

[0218] The above-described embodiments are merely examples for illustrating the basic nature of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. Therefore, it is intended to be limited only by the scope of the corresponding patent claims, and not by the specific details shown in the description of the embodiments herein.

Claims

1. 1. A decoder for reconstructing from a data stream an array of information samples representing a spatially sampled information signal subdivided into leaf areas of different sizes by a multi-tree subdivision, each leaf area being associated with one hierarchical level of a set of hierarchical levels of the multi-tree subdivision, each said leaf area being associated with coding parameters which are represented for each said leaf area by a respective set of syntax elements, each said syntax element being a respective syntax element type of a set of syntax element types; The decoder extracting inheritance information from the data stream, the inheritance information indicating whether inheritance is used or not, and if the use of inheritance is indicated, at least one inheritance area of ​​the array of information samples, which is composed of a set of leaf areas and corresponds to one hierarchical level of a series of hierarchical levels of the subdivision of the multi-tree, is lower than each hierarchical level with which the set of leaf areas is associated; Where the use of said inheritance is indicated, extracting from the data stream, for each inheritance region, a subset of inheritances that includes at least one syntax element of a predetermined syntax element type; For a subset of inheritance of a corresponding syntax element in the set of syntax elements representing coding parameters associated with each set of leaf areas constituting at least one inheritance area, the subset of inheritance is replicated or used as a prediction. decoder.

2. 2. The decoder of claim 1, For a region corresponding to one of the sets of inheritances of at least one hierarchical level of a subdivision of the multi-tree consisting of a corresponding set of sets of leaf regions, using the order of hierarchical levels from lower to higher hierarchical levels in which the array of information samples is recursively partitioned, extracting associated inheritance flags from the data stream and determining whether each of the inheritance flags indicates inheritance; For each region whose associated inheritance flag indicates inheritance: extracting from the data stream a subset of the inheritance that includes at least one syntax element of a predetermined syntax element type; for a subset of inheritance of a corresponding syntax element in the set of syntax elements representing coding parameters associated with a leaf region that constitutes a region for which the associated inheritance flag indicates inheritance, replicating the subset of inheritance or using the subset of inheritance as a prediction; When checking the inheritance flag using the hierarchy level order from the lower hierarchy level to the higher hierarchy level, suppressing extraction and checking of further inheritance flags for leaf regions that constitute regions whose associated inheritance flags indicate inheritance. decoder.

3. 3. The decoder of claim 2, wherein the decoder is a hybrid decoder.

4. 4. The decoder of claim 1, wherein the decoder decodes the residual of each leaf block.

5. 5. A decoder according to claim 1, wherein the decoder extracts from the data stream a corresponding inheritance subset of syntax elements in the set of syntax elements that represent coding parameters associated with leaf areas that do not belong to any of the sets of leaf areas that constitute any inheritance area indicated by the inheritance information.

6. 6. A decoder according to claim 1, wherein, if non-use of inheritance is indicated, the decoder extracts from the data stream a subset of the corresponding inheritance of the syntax elements in the set of syntax elements representing coding parameters associated with all leaf areas of the array of information samples.

7. 7. A decoder according to claim 1, wherein the decoder reconstructs the sequence of information samples using a prediction method that uses locally different prediction modes on a leaf area basis, and the type of the predetermined syntax element influences the prediction mode.

8. A decoder according to any one of claims 1 to 7, The at least one syntax element of the predetermined syntax element type is an intra-prediction mode syntax element, The decoder decoding, in residual decoding order, a residual signal for each leaf region constituting a region for which the associated inheritance flag indicates inheritance; By using adjacent samples of the reconstructed signal of the already reconstructed leaf region as reference signals, a separate intra prediction signal for each leaf region constituting a region for which the associated inheritance flag indicates inheritance is calculated in the order of decoding the residuals according to the intra prediction mode indicated by the intra prediction mode syntax element, and reconstructing each leaf region by adding the intra prediction signal and the residual signal. decoder.

9. 9. The decoder of claim 8, wherein the order of decoding the residuals is a depth-first traversal order.

10. 1. A method for reconstructing from a data stream an array of information samples representing a spatially sampled information signal subdivided by a multi-tree subdivision into leaf areas of different sizes, each leaf area being associated with a hierarchical level of a series of hierarchical levels of the multi-tree subdivision, each said leaf area being associated with coding parameters which are represented for each said leaf area by a respective set of syntax elements, each said syntax element being a respective syntax element type of a set of syntax element types; The method comprises: extracting inheritance information from the data stream, the inheritance information indicating whether inheritance is used or not, and if the use of inheritance is indicated, at least one inheritance area of ​​the array of information samples, which is composed of a set of leaf areas and corresponds to one hierarchical level of a series of hierarchical levels of the subdivision of the multi-tree, is lower than each hierarchical level with which the set of leaf areas is associated; Where the use of said inheritance is indicated, extracting from the data stream, for each inheritance region, a subset of inheritances that includes at least one syntax element of a predetermined syntax element type; For a subset of inheritance of a corresponding syntax element in the set of syntax elements representing coding parameters associated with each set of the leaf areas constituting at least one inheritance area, the subset of inheritance is replicated or used as a prediction. method.

11. 1. An encoder for encoding into a data stream an array of information samples representing a spatially sampled information signal subdivided into leaf areas of different sizes by a subdivision of a multi-tree, wherein each leaf area is associated with one hierarchical level of a series of hierarchical levels of the subdivision of the multi-tree, and each leaf area is associated with coding parameters, which are represented for each leaf area by a respective set of syntax elements, each syntax element being a respective syntax element type of a set of syntax element types; The encoder comprises: inserting inheritance information into the data stream, the inheritance information indicating whether inheritance is to be used or not, and if the use of inheritance is indicated, at least one inheritance area of ​​the array of information samples, which is composed of a set of leaf areas and corresponds to one hierarchical level of a series of hierarchical levels of the subdivision of the multi-tree, is lower than each hierarchical level with which the set of leaf areas is associated; Where the use of said inheritance is indicated, inserting into the data stream, for each inheritance region, an inheritance subset including at least one syntax element of a predetermined syntax element type from the data stream; Suppressing the encoding of an inheritance subset of corresponding syntax elements in the set of syntax elements representing coding parameters associated with each set of said leaf areas constituting at least one inheritance area, or using the inheritance subset inserted into the data stream as a prediction in encoding a residual of the inheritance subset of corresponding syntax elements in the set of syntax elements representing coding parameters associated with each set of said leaf areas constituting at least one inheritance area. Encoder.

12. 1. A method for encoding into a data stream an array of information samples representing a spatially sampled information signal subdivided into leaf areas of different sizes by a multi-tree subdivision, wherein each leaf area is associated with one hierarchical level of a set of hierarchical levels of the multi-tree subdivision, and each leaf area is associated with coding parameters, which are represented for each leaf area by a respective set of syntax elements, each syntax element being a respective syntax element type of a set of syntax element types; The method comprises: inserting inheritance information into the data stream, the inheritance information indicating whether inheritance is to be used or not, and if the use of inheritance is indicated, at least one inheritance area of ​​the array of information samples, which is made up of a set of leaf areas and corresponds to one hierarchical level of a series of hierarchical levels of the subdivision of the multi-tree, is lower than each hierarchical level with which the set of leaf areas is associated; Where the use of said inheritance is indicated, inserting into the data stream, for each inheritance region, an inheritance subset including at least one syntax element of a predetermined syntax element type from the data stream; Suppressing the encoding of an inheritance subset of corresponding syntax elements in the set of syntax elements representing coding parameters associated with each set of said leaf areas constituting at least one inheritance area, or using the inheritance subset inserted into the data stream as a prediction in encoding a residual of the inheritance subset of corresponding syntax elements in the set of syntax elements representing coding parameters associated with each set of said leaf areas constituting at least one inheritance area. method.

13. A computer-readable digital storage medium having stored thereon a computer program comprising program code which, when running on a computer, performs the method of any one of claims 10 and 12.

14. a data stream in which an array of information samples representing a spatially sampled information signal is coded into a data stream by a multi-tree subdivision into leaf areas of different sizes, each leaf area being associated with one hierarchical level of a series of hierarchical levels of the multi-tree subdivision, each leaf area being associated with coding parameters, the coding parameters being represented for each leaf area by a respective set of syntax elements, each syntax element being a respective syntax element type of a set of syntax element types; The data stream The information sample sequence includes inheritance information, the inheritance information indicating whether or not inheritance is to be used, and if the use of inheritance is indicated, at least one inheritance region of the information sample sequence corresponding to one hierarchical level of a series of hierarchical levels of the subdivision of the multi-tree is composed of a set of leaf regions, and is lower than each hierarchical level with which the set of leaf regions is associated; Where the use of said inheritance is indicated, an inheritance subset including at least one syntax element of a predetermined syntax element type from the data stream is included in the data stream for each inheritance region; does not include a subset of the inheritance of corresponding syntax elements in the set of syntax elements representing coding parameters associated with each set of the leaf areas constituting at least one inheritance area, or includes a residual of the subset of the inheritance of corresponding syntax elements in the set of syntax elements representing coding parameters associated with each set of the leaf areas constituting at least one inheritance area with respect to the subset of the inheritance inserted as a prediction; Data stream.

Images