Combining regions of the specimen

Abstract

The present invention provides a method for reducing the amount of data used in reconstructing an array of samples from a data stream by encoding desired combinations or groupings of simply connected regions into which the array of information samples is subdivided. [Solution] The encoder 10 defines a predetermined relative positional relationship for a simply connected region, identifies a simply connected region among a plurality of simply connected regions that has a predetermined relative positional relationship to the predetermined simply connected region, spatially subdivides the region of samples representing a spatial sampling of a two-dimensional information signal into a plurality of simply connected regions of different sizes by recursively partitioning multiple times according to a first subset of syntax elements included in the data stream, combines spatially adjacent simply connected regions according to a second subset of syntax elements in the data stream that is separated from the first subset, and performs intermediate subdivision of the array of samples into disjoint sets of simply connected regions whose union is a plurality of simply connected regions.

Description

Technical Field

[0001] The present invention relates to an encoding system for two-dimensionally sampled information signals such as moving images or still images.

Background Art

[0002] Typically, in image and video encoding, an image, or a specific set of sample sequences within an image, is broken down into blocks and associated with specific encoding parameters. An image typically consists of multiple sample sequences. Furthermore, an image may be associated with additional auxiliary sample sequences specifying, for example, transparency information or depth maps. The sample sequences (including auxiliary sample sequences) of an image can be grouped into one or more so-called planar groups, each consisting of one or more sample sequences. Planar groups of an image can be encoded independently, or, if the image is associated with multiple planar groups, they can be encoded by predictions from other planar groups of the same image. Typically, each planar group is broken down into blocks. Blocks (or corresponding blocks of sample sequences) are predicted by either interpicture prediction or intrapicture prediction. Blocks can be of various sizes, either square or rectangular. The division of an image into blocks can be fixed by syntax, or (at least partially) transmitted as a signal within the bitstream. Often, syntactic elements are transmitted that transmit signals for subdivisions of blocks of a given size. These syntactic elements allow a block to be subdivided into smaller blocks, specifying whether or not they are associated with coding parameters for prediction purposes, and how. For all samples in a block (or the corresponding block in a sample sequence), the decoding of the associated coding parameters is specified in a particular way. For example, predictions for all samples in a block are made using the same set of prediction parameters, such as a reference index (identifying the reference image within a set of already coded images), motion parameters (specifying the amount of block movement between the reference image and the current image), and parameters for specifying interpolation filters and intra-prediction modes. Motion parameters can be represented by displacement vectors with horizontal and vertical components, or higher-order motion parameters such as affine motion parameters consisting of six components. It is also possible to associate multiple sets of specific prediction parameters (such as reference indexes and motion parameters) with a single block.In this case, for each set of these specific prediction parameters, a single intermediate prediction signal is generated for the block (or the corresponding block in the sample sequence), and the final prediction signal is formed by combinations including superpositions of these intermediate prediction signals. The corresponding weighting parameters can be fixed for either the image, the reference image, or the set of reference images, or they can be included in the set of prediction parameters for the corresponding block, sometimes with a constant offset (added to the weighted sum). The difference between the original block (or the corresponding block in the sample sequence) and the prediction signal is also called the residual signal, and is usually transformed and quantized. Often, a two-dimensional transformation is applied to the residual signal (or the corresponding sample sequence of the residual block). For transform coding, the block (or the corresponding block in the sample sequence) for which a specific set of prediction parameters is used can be further subdivided before the transformation is applied. The transformed block is the same size as or smaller than the block used for prediction. Also, a transformed block can contain multiple blocks used for prediction. Each transformed block can be of various sizes, and can be a square or rectangular block. After the transformation, the resulting transformation coefficients are quantized to obtain so-called transformation coefficient levels. The conversion coefficient levels, prediction parameters, and subdivision information, if present, are entropy-encoded. [Overview of the Initiative] [Problems that the invention aims to solve]

[0003] In image and video encoding standards, the possibilities for subdividing an image (or planar group) into blocks, as provided by the syntax, are very limited. Typically, all that can be specified is whether a block of a given size can be subdivided into smaller blocks, and (if applicable, how). For example, in H.264, the maximum block size is 16x16. A 16x16 block is also called a macroblock, and each image is divided into macroblocks in the first step. For each 16x16 macroblock, it is possible to signal whether it will be encoded as a 16x16 block, two 16x8 blocks, two 8x16 blocks, or four 8x8 blocks. If a 16x16 block is subdivided into four 8x8 blocks, each of these 8x8 blocks can be encoded as one 8x8 block, two 8x4 blocks, two 4x8 blocks, or four 4x4 blocks. The latest standards for image and video encoding have the advantage of reducing the rate of associated information used to transmit (signal) subdivision information by minimizing the possibility of specifying block divisions. However, as explained below, there is a disadvantage that the bitrate required to transmit block prediction parameters can be high. Typically, the rate of information associated with transmitting prediction information accounts for a significant portion of the block's total bitrate. Therefore, reducing this associated information can potentially improve encoding efficiency, which can be achieved, for example, by using a larger block size. Images in actual image and video sequences are composed of objects of arbitrary shapes with specific properties. For example, such objects or parts of objects are characterized by unique textures or unique motions. Typically, the same set of prediction parameters is applied to such objects or parts of objects. However, typically, the boundaries of objects do not coincide with the boundaries of possible blocks in large prediction blocks (e.g., 16x16 macroblocks in H.264). Typically, the encoder makes subdivision decisions (from a limited set of possibilities) to minimize a certain rate / distortion loss.As a result, for any shape of object, a large number of small blocks can be generated. Each of these small blocks is associated with a set of predictive parameters that need to be transmitted, so the rate of the associated information can account for a significant portion of the total bitrate. However, since some of these small blocks still represent the same object or a region of an object, the predictive parameters of the resulting large number of blocks will be identical or very similar.

[0004] In short, subdividing or tiling an image into smaller parts, tiles, or blocks significantly impacts both the efficiency and complexity of encoding. As outlined above, the more finely an image is subdivided into smaller blocks, the more spatially detailed the encoding parameters can be set, improving the adaptability of these encoding parameters to the image / video material. On the other hand, the higher the granularity of the encoding parameter settings, the greater the burden of the accompanying information required to inform the decoder of the necessary settings. It should also be noted that allowing the encoder to spatially subdivide the image / video into (further) blocks significantly increases the number of possible encoding parameter settings, and as a result, it generally becomes more difficult to find the best rate / distortion compromise regarding encoding parameter settings. [Means for solving the problem]

[0005] The objective is to provide an encoding system for encoding an array of information samples representing a spatially sampled two-dimensional information signal. The array of information samples may include, but is not limited to, images from videos or still images. This encoding system enables improved balancing of encoding complexity and achievable rate / distortion ratio, and / or achieves an improvement in rate / distortion ratio.

[0006] This objective is achieved by the decoder, encoder, method, computer program, and data stream described in the independent claims of the patent claims.

[0007] In this embodiment, desirable combinations or groupings of single-connected regions formed by subdividing an array of information samples are encoded, thereby reducing the amount of data. To this end, a predetermined relative positional relationship is defined for the single-connected regions. This makes it possible to identify a single-connected region among multiple single-connected regions that have a predetermined relative positional relationship with that given single-connected region. That is, if the number is zero, there may be no merge index in the data stream for the given single-connected region. Furthermore, if the number of single-connected regions that have a predetermined relative positional relationship with that given single-connected region is 1, the encoding parameter of the single-connected region may be adopted or used to predict the encoding parameter for the given single-connected region, and no further syntactic elements are required. Otherwise, i.e., if the number of single-connected regions that have a predetermined relative positional relationship with that given single-connected region is greater than 1, the introduction of further syntactic elements may be suppressed, even if the encoding parameters associated with these identified single-connected regions are identical to each other.

[0008] In this embodiment, if the coding parameters of adjacent simply connected regions are not equal, a proper subset of the number of simply connected regions that have a predetermined relative positional relationship with a given simply connected region may be identified by a reference adjacency identifier. This proper subset is used when adopting coding parameters or when predicting coding parameters for a given simply connected region.

[0009] In yet another embodiment, depending on a first subset of syntactic elements contained in the data stream, the sample region representing the spatial sampling of the two-dimensional information signal is spatially subdivided by recursively performing multiple partitioning operations into multiple simply connected regions of different sizes. Subsequently, depending on a second subset of syntactic elements in the data stream separated from the first subset, spatially adjacent simply connected regions are combined to perform an intermediate subdivision of the sample sequence into disparate sets of simply connected regions whose union is multiple simply connected regions. This intermediate subdivision is used when reconstructing the sample sequence from the data stream. This reduces the importance of optimization for subdivision, as it can be corrected later by joining if the subdivision is too fine. Furthermore, the combination of subdivision and joining enables the realization of intermediate subdivisions that would not be possible with recursive partitioning operations alone, and by linking the subdivision and joining operations using the separated set of syntactic elements, the fit of the effective or intermediate subdivision to the actual content of the two-dimensional information signal can be improved. The additional overhead caused by the additional subset of syntactic elements to show the details of the joining operations is negligibly small compared to the benefits.

[0010] In this embodiment, an array of information samples representing a spatially sampled information signal is first spatially divided into the root region of a tree, and then further subdivided according to multi-tree subdivision information extracted from the data stream. By recursively subdividing a subset of the root region of the tree multiple times, at least a subset of the root region of the tree is subdivided into smaller, simply connected regions of varying sizes. To enable finding a desirable compromise between subdivisions that are too fine and too coarse in terms of rate / distortion at a reasonable encoding complexity, the size of the largest region of the root region of the tree into which the array of information samples is spatially divided is included in the data stream and extracted from the data stream on the decoding side. Therefore, the decoder may include an extraction unit that can extract the size of the largest region and multi-tree subdivision information from the data stream; a subdivision unit that can spatially divide an array of information samples representing spatially sampled information signals into the root region of a tree of the largest region size, subdivide it according to the multi-tree subdivision information, and recursively perform multiple partitioning operations on a subset of the root region of the tree, thereby subdividing at least a subset of the root region of the tree into smaller simply connected regions of various sizes; and a reconstruction unit that can reconstruct an array of information samples from the data stream using the subdivision into smaller simply connected regions.

[0011] Furthermore, in this embodiment, the data stream includes the highest hierarchical level to which a subset of the root region of the tree is subject to recursive multiple partitioning. This method facilitates the transmission of subdivision information of a multitree and reduces the number of bits required for encoding.

[0012] Furthermore, the reconstruction unit may be configured to perform at least one of the following actions at a granularity corresponding to the intermediate subdivision: determining which prediction mode to use from among the intra-prediction mode and inter-prediction mode; converting from the spectral domain to the spatial domain; performing inter-prediction and / or setting the parameters for inter-prediction; and performing intra-prediction and / or setting the parameters for intra-prediction.

[0013] Furthermore, the extraction unit may be configured to extract syntactic elements associated with leaf regions of segmented tree blocks from the data stream in a depth-order traverse order. This method allows the extraction unit to utilize statistics of the encoded syntactic elements of adjacent leaf regions with a higher probability than if a width-order traverse order were used. In another embodiment, a further subdivision unit is used to subdivide at least one subset of smaller simply connected regions into even smaller simply connected regions, according to further multi-tree subdivision information. The first stage of subdivision may be used by the reconstruction unit to predict the region of the information sample, and the second stage of subdivision may be used by the reconstruction unit to perform a retransformation from the spectral region to the spatial region. Defining the residual subdivision lower than the prediction subdivision reduces the bit consumption for encoding the entire subdivision. On the other hand, the restrictions and degrees of freedom on the residual subdivision due to the lower definition have a slight negative impact on encoding efficiency. This is because, in most cases, portions of the image with similar motion compensation parameters are larger than portions with similar spectral characteristics.

[0014] In yet another embodiment, the size of the largest region is included in the data stream, which defines the size of the tree root region formed when the smaller simply connected region is first subdivided before at least one subset of the tree root region is subdivided into even smaller simply connected regions according to further multi-tree subdivision information. This makes it possible to independently set the size of the largest region for prediction subdivision, while allowing for improvements in the rate / distortion compromise through residual subdivision.

[0015] In yet another embodiment of the present invention, a first subset of syntactic elements, separated from a second subset of syntactic elements forming the subdivision information of the multitree, is included in the data stream, and the decoding concatenation unit can combine smaller, spatially adjacent simply connected regions of the multitree subdivision to obtain intermediate subdivisions of the sample sequence, depending on the first subset of syntactic elements. The reconstruction unit may be configured to reconstruct the sample sequence using the intermediate subdivisions. This allows the encoder to perform effective subdivision with respect to the spatial distribution of the properties of the information sample sequence, making it easier to find the optimal compromise regarding rate / distortion. For example, if the size of the largest region is large, the tree root region is large, and therefore the multitree subdivision information is likely to be complex. On the other hand, if the size of the largest region is small, adjacent tree root regions are likely to relate to the content of information containing similar properties, and these tree root regions are also likely to be processed together. The concatenation eliminates the gap between these two extreme cases, resulting in subdivision with a near-optimal granularity. From the encoder's perspective, the concatenation syntactic elements simplify the encoding procedure or reduce the complexity of the calculations. This is because if the encoder mistakenly uses overly fine subdivisions, it can correct this error by subsequently setting concatenation syntactic elements. In this case, the adaptation of a small portion of the syntactic elements set before the concatenation syntactic elements may or may not be performed.

[0016] In yet another embodiment, the size of the largest region and the multi-tree subdivision information are used for residual subdivision rather than prediction subdivision.

[0017] When dealing with simply connected regions of a quadtree subdivision of an array of information samples representing spatially sampled information signals, a depth-ordered traverse order is used, rather than a width-ordered traverse order, according to the embodiment. By using a depth-ordered traverse order, the probability that each simply connected region has traversed adjacent simply connected regions increases, and information about these adjacent simply connected regions can be reliably used when reconstructing each current simply connected region.

[0018] If the sequence of information samples is first divided into a regular sequence of root regions of zero-order hierarchical trees, and then at least one subset of the root regions of the trees is subdivided into smaller simply connected regions of different sizes, the reconstruction unit may traverse the root regions of the trees using a zigzag scan, and for each of the subdivided root regions of the trees, process the simply connected leaf regions in a depth-order traverse order, and then proceed to the root region of the next tree in a zigzag scan order. Furthermore, simply connected leaf regions of the same hierarchical level may be traversed in a zigzag scan order according to the depth-order traverse order. In this way, a state with a high probability of having adjacent simply connected leaf regions is maintained.

[0019] In this embodiment, flags associated with nodes in a multitree structure are arranged sequentially in the order of depth-order traversal, but the sequential encoding of the flags uses the context of probability estimation. This is the same for flags associated with nodes in a multitree structure located within the same hierarchical level of the multitree structure, but different for nodes in a multitree structure located within different hierarchical levels of the multitree structure. As a result, improvements can be made in finding a compromise between the number of contexts provided and the adaptation of the flags to the actual symbol statistics.

[0020] In the embodiment, the context for probability estimation of a given flag used also depends on traversing the given flag in depth order and on the region of the root of the tree where the given flag is in a predetermined relative position to the corresponding region. Similar to the underlying idea of ​​the embodiment described, the use of depth order traversal ensures with a high probability that the encoded flag will include a flag corresponding to a region adjacent to the region corresponding to the given flag. This knowledge is used to improve adaptability to the context in which the given flag is used.

[0021] The flag used to set the context for a given flag may correspond to the area located above and / or to the left of the area to which the given flag corresponds. Furthermore, the flags used for context selection may be limited to flags belonging to the same hierarchical level as the node to which a given flag is associated.

[0022] In this embodiment, the encoded transmission information includes a representation of the highest hierarchical level and a set of flags associated with nodes in a multi-tree structure other than the highest hierarchical level. Each flag specifies whether the associated node is an intermediate node or a child node. The set of flags are sequentially decoded from the data stream in a traverse order, either in depth order or width order. During this process, nodes at the highest hierarchical level are skipped, and the same leaf node is automatically assigned. As a result, the encoding rate is reduced.

[0023] In other embodiments, the highest hierarchical level may be indicated by the encoded transmission information in a multi-tree structure. This makes it possible to limit the presence of flags to hierarchical levels other than the highest hierarchical level, thereby preventing further subdivision of the blocks at the highest hierarchical level.

[0024] The spatial multi-tree subdivision is part of the secondary subdivision of leaf nodes. When the root region of the tree of the primary multi-tree subdivision is not partitioned, the context used for encoding the flag of the secondary subdivision may be selected so that the context is the same for flags associated with regions of the same size. Preferred embodiments of the present invention will be described below with reference to the following drawings.

Brief Description of the Drawings

[0025] [Figure 1] It is a configuration diagram showing an encoder of an embodiment of the present application. [Figure 2] It is a configuration diagram showing a decoder of an embodiment of the present application. [Figure 3A] It is a schematic diagram showing an example of the quadtree subdivision, showing the first hierarchical level. [Figure 3B] It is a schematic diagram showing an example of the quadtree subdivision, showing the second hierarchical level. [Figure 3C] It is a schematic diagram showing an example of the quadtree subdivision, showing the third hierarchical level. [Figure 4] It is a schematic diagram showing the quadtree subdivision tree structure from FIGS. 3A to 3C according to an embodiment. [Figure 5A] It is a schematic diagram showing the quadtree subdivision from FIGS. 3A to 3C. [Figure 5B] It is a schematic diagram showing the tree structure with indexes of individual leaf blocks from FIGS. 3A to 3C. [Figure 6A] It is a schematic diagram showing a binary sequence or flag sequence representing the tree structure of FIG. 4 and the quadtree subdivision from FIGS. 3A to 3C according to different embodiments. [Figure 6B] It is a schematic diagram showing a binary sequence or flag sequence representing the tree structure of FIG. 4 and the quadtree subdivision from FIGS. 3A to 3C according to different embodiments. [Figure 7] It is a flowchart showing the steps performed by the data stream extraction unit of the embodiment. [Figure 8]This flowchart shows the functions of the data stream extraction unit in another embodiment. [Figure 9A] This is a schematic diagram illustrating a quadtree subdivision according to the embodiment, showing candidate blocks adjacent to a designated highlighted block. [Figure 9B] This is a schematic diagram illustrating a quadtree subdivision according to the embodiment, showing candidate blocks adjacent to a designated highlighted block. [Figure 10] This flowchart shows the functions of the data stream extraction unit in another embodiment. [Figure 11] This schematic diagram illustrates the composition of images from planes and plane groups according to the embodiment, and explains encoding using plane-to-plane fitting / prediction. [Figure 12A] This is a schematic diagram showing a subtree structure to illustrate the inheritance system according to the embodiment. [Figure 12B] This is a schematic diagram showing a subdivision corresponding to a subtree structure, in order to explain the inheritance system according to the embodiment. [Figure 12C] This schematic diagram shows a subtree structure to illustrate the inheritance system that performs adoption and prediction according to the embodiment. [Figure 12D] This schematic diagram shows a subtree structure to illustrate the inheritance system that performs adoption and prediction according to the embodiment. [Figure 13] This flowchart shows the steps performed by an encoder that implements an inheritance system according to an embodiment. [Figure 14] Figure 14A is a diagram showing a first-order subdivision to illustrate the possibility of implementing an inheritance system in relation to interpretation according to the embodiment, and Figure 14B is a diagram showing a lower subdivision to illustrate the possibility of implementing an inheritance system in relation to interpretation according to the embodiment. [Figure 15] This is a diagram illustrating the decoding process in relation to the inheritance system, according to the embodiment. [Figure 16]This is a schematic diagram showing the scanning order of sub-regions in a multi-tree subdivision with respect to a small region targeted for intra-prediction, according to the embodiment. [Figure 17] This is a configuration diagram showing a decoder according to an embodiment. [Figure 18A] This is a schematic diagram illustrating the possibility of different subdivisions in other embodiments. [Figure 18B] This is a schematic diagram illustrating the possibility of different subdivisions in other embodiments. [Figure 18C] This is a schematic diagram illustrating the possibility of different subdivisions in other embodiments. [Figure 19] This is a diagram showing the encoder according to the embodiment. [Figure 20] This is a configuration diagram showing a decoder in another embodiment. [Figure 21] This is a configuration diagram showing an encoder of another embodiment. [Modes for carrying out the invention]

[0026] In the following diagram descriptions, elements appearing in multiple diagrams are indicated by a common symbol, and redundant explanations of these elements are omitted. Rather, the explanation of an element appearing in one diagram applies to other diagrams where that element also appears, provided that the explanations accompanying those diagrams describe any differences in that element.

[0027] Furthermore, the following description will begin using the embodiments of encoders and decoders described in Figures 1 to 11. The embodiments described in these figures are combinations of many aspects of the present application, but they are also effective when implemented individually within an encoding system. Therefore, for the following figures, embodiments that utilize the above aspects by using each of these embodiments individually are briefly described and represent the abstract concepts of the embodiments described in Figures 1 and 11 in a different sense.

[0028] Figure 1 shows an encoder according to an embodiment of the present invention. The encoder 10 in Figure 1 comprises a prediction unit 12, a residual precoder 14, a residual reconstruction unit 16, a data stream insertion unit 18, and a block division unit 20. The encoder 10 is for encoding a temporally and spatially sampled information signal into a data stream 22. The temporally and spatially sampled information signal may be, for example, a video, i.e., a series of images. Each image represents an array of image samples. Other examples of temporally and spatially sampled information signals include, for example, depth images captured by a time-of-light camera. Furthermore, it should be noted that a spatially sampled information signal may contain multiple arrays per frame or timestamp. For example, in the case of a color video, each frame may contain an array of one luminance sample along with an array of two chroma samples. Also, the temporal sampling rates of different components of the information signal, namely luminance and chroma, may be different. The same applies to spatial resolution. In addition, the video may be further accompanied by spatially sampled information, such as depth or transparency information. However, in the following explanation, in order to make the main content of this application easier to understand, we will first focus on explaining one of these arrangements, and then explain how to handle multiple planes.

[0029] The encoder 10 in Figure 1 is configured to create a data stream 22 such that the syntactic elements of the data stream 22 describe the image at a granularity between the entire image and individual image samples. To this end, the splitting unit 20 is configured to subdivide each image 24 into simply connected regions 26 of varying sizes. Hereafter, these regions will simply be referred to as blocks or sub-regions 26.

[0030] As will be described in more detail below, the splitting unit 20 uses multitree subdivision to subdivide the image 24 into blocks 26 of varying sizes. More precisely, in the specific embodiments outlined below with respect to Figures 1 to 11, quadtree subdivision is used in most cases. The splitting unit 20 may also include an internally connected subdivision unit 28, as will be described in more detail below. The subdivision unit 28 subdivides the image 24 into the blocks 26. The subdivision unit 28 is connected to a joiner 30. The joiner 30 can join groups of these blocks 26 to obtain an effective subdivision or granularity that lies between the un-subdivision of the image 24 and the subdivision defined by the subdivision unit 28.

[0031] As shown by the dashed lines in Figure 1, the prediction unit 12, residual precoder 14, residual reconstruction unit 16, and data stream insertion unit 18 perform image subdivision as defined by the division unit 20. For example, as will be described in more detail below, the prediction unit 12 uses the prediction subdivision defined by the division unit 20 to: set prediction parameters corresponding to each subregion according to the selected prediction mode, and determine for each individual subregion of the prediction subdivision whether each subregion should be subject to intrapicture prediction or interpicture prediction.

[0032] Next, the residual precoder 14 may use the subdivision of the residuals of image 24 to encode the residuals of the prediction of image 24 provided by the prediction unit 12. When the residual reconstruction unit 16 reconstructs the residuals from the syntactic elements output by the residual precoder 14, the residual reconstruction unit 16 also performs the above-mentioned subdivision of the residuals. The data stream insertion unit 18 may use the above subdivision, i.e., the prediction and residual subdivision, to perform the following: For example, it may determine the insertion order and adjacency relationships of the syntactic elements in order to insert the syntactic elements output to the data stream 22 by the residual precoder 14 and the prediction unit 12 using entropy encoding or the like.

[0033] As shown in Figure 1, the encoder 10 includes an input section 32 to which the original information signal is input. The subtractor 34, residual precoder 14, and data stream insertion section 18 are connected in this order in series between the input section 32 and the output section of the data stream insertion section 18, which outputs the encoded data stream 22. The subtractor 34 and residual precoder 14 are part of the prediction loop. The prediction loop is closed by a residual builder 16, adder 36, and prediction section 12, which are connected in the following order in series between the output section of the residual precoder 14 and the inverted input section of the subtractor 34. The output section of the prediction section 12 is also connected to another input section of the adder 36. The prediction section 12 may also have an input section that is directly connected to the input section 32 and another input section that is also connected to the output section of the adder 36 via an optional in-loop filter 38. Furthermore, since the prediction unit 12 generates accompanying information during operation, the output of the prediction unit 12 is also connected to the data stream insertion unit 18. Similarly, the splitting unit 20 has an output that is connected to another input of the data stream insertion unit 18. We have described the structure of encoder 10, and below we will explain the operating modes in more detail.

[0034] As described above, the division unit 20 determines how to subdivide each image 24 into small regions 26. According to the subdivision of the image 24 used for prediction, the prediction unit 12 determines how to predict each small region corresponding to this subdivision. The prediction unit 12 outputs the prediction of the small region to the invert input of the subtractor 34 and the other input of the adder 36, and outputs prediction information to the data stream insertion unit 18 that reflects how the prediction unit 12 obtains this prediction from a previously encoded portion of the video.

[0035] At the output of the subtractor 34, the residual precoder 14 processes the residuals of the prediction according to the residual subdivision defined by the subdivision unit 20 to obtain the residuals of the prediction. As will be further described below with reference to Figures 3 to 10, the residual subdivision of the image 24 used by the residual precoder 14 may be related to the prediction subdivision used by the prediction unit 12, such that each prediction subregion is adopted as a residual subregion, or further subdivided into smaller residual subregions. Alternatively, the prediction and residual subdivisions may be completely independent.

[0036] The residual precoder 14 performs a transformation from the spatial domain to the spectral domain using a two-dimensional transformation for each residual sub-region. The resulting transformation block and the resulting transformation coefficients are subsequently quantized, or this is inherently involved. Distortion occurs as a result of quantization noise. For example, the data stream insertion unit 18 may perform lossless encoding of the syntactic elements describing the above transformation coefficients into the data stream 22, for example, using entropy encoding.

[0037] Next, the residual reconstruction unit 16 re-converts the conversion coefficients to a residual signal using requantization with re-conversion. To obtain the residual of the prediction, the residual signal is combined with the prediction used by the subtractor 34 in the adder 36. As a result, the reconstructed portion or subregion of the current image is obtained at the output of the adder 36. The prediction unit 12 may use the subregion of the reconstructed image directly by intra-prediction. This is to predict a subregion of a particular prediction by estimation from an adjacent reconstructed prediction subregion. However, intra-prediction performed within the spectral domain by directly predicting the spectrum of the current subregion from the spectra of adjacent subregions is also theoretically possible.

[0038] For interpretation, the prediction unit 12 may use an encoded and reconstructed image in a version filtered by an optional in-loop filter 38. For example, the in-loop filter 38 may have a transfer function that is effective in forming the quantization noise described above, and may include a deblocking filter and / or an adaptive filter.

[0039] The prediction unit 12 selects prediction parameters that indicate how to make predictions for a particular sub-region of prediction by comparing it with the original sample in image 24. As described in more detail below, the prediction parameters may include indications of prediction modes, such as intra-picture prediction or inter-picture prediction, for each sub-region of prediction. In the case of intra-picture prediction, the prediction parameters may include indications of the angle at which the boundary within the sub-region of prediction on which the intra-prediction is performed primarily extends. In the case of inter-picture prediction, the parameters may include motion vectors, video indices, and higher-order motion transformation parameters. In the case of both intra-picture prediction and / or inter-picture prediction, optional filter information may be included for filtering the reconstructed image sample on which the prediction of the current sub-region of prediction is based.

[0040] As will be explained in more detail below, the aforementioned subdivision defined by the division unit 20 significantly affects the rate / distortion rate that can be maximally achieved by the residual precoder 14, the prediction unit 12, and the data stream insertion unit 18. A finer subdivision may improve the predictions obtained by the prediction unit 12, and the residual signals encoded by the residual precoder 14 may become smaller and be encoded with fewer bits, but the encoding rate required for the prediction parameters 40 output by the prediction unit 12 and inserted into the data stream 22 will be very large. The opposite is true for coarser subdivisions. Furthermore, the same considerations apply to the subdivision of residuals. Transforming the image with a finer granularity for individual transformation blocks reduces the complexity of the transformation calculations and increases the spatial resolution of the resulting transformation. That is, smaller residual subregions increase the consistency of the spectral distribution of the contents of individual residual subregions. However, spectral resolution decreases, and the ratio between significant coefficients and non-significant, i.e., coefficients quantized to zero, deteriorates. In other words, the granularity of the transformation needs to be locally adapted to the content of the image. Furthermore, apart from the positive effects of finer granularity, finer granularity usually increases the amount of accompanying information needed to indicate the selected subdivision to the decoder. As will be described in more detail below, the embodiments described later describe an encoder 10 that can adapt very effectively to the subdivision of the content of the information signal to be encoded and can transmit the subdivision information used on the decoding side by instructing the data stream insertion unit 18 to insert the subdivision information into the encoded data stream 22. Details are shown below.

[0041] However, before defining the subdivision of the division section 20 in detail, a decoder according to the embodiment of the present application will be described in detail with reference to Figure 2.

[0042] The decoder in Figure 2, denoted by symbol 100, comprises an extraction unit 102, a splitting unit 104, a residual reconstruction unit 106, an adder 108, a prediction unit 110, an optional in-loop filter 112, and an optional post-filter 114. The extraction unit 102 receives the encoded data stream at the input unit 116 of the decoder 100 and extracts prediction parameters 120 and residual data 122 from the subdivision information 118 of the encoded data stream, which the extraction unit 102 outputs to the image splitting unit 104, the prediction unit 110, and the residual reconstruction unit 106, respectively. The output of the residual reconstruction unit 106 is connected to the first input of the adder 108. Another input and output of the adder 108 are connected to the prediction loop. The prediction loop has an optional in-loop filter 112 and the prediction unit 110 connected in series in that order, and there is a direct bypass path from the output of the adder 108 to the prediction unit 110. This is similar to the connection between the adder 36 and the prediction unit 12 in Figure 1 described above, that is, one is for intra-picture prediction and the other is for inter-picture prediction. Either the output of the adder 108 or the output of the in-loop filter 112 may be connected to the output 124 of the decoder 100, which outputs the reconstructed information signal to a playback device or the like. An optional post-filter 114 may be connected to the path to the output 124 in order to improve the image quality of the visual impression of the reconstructed signal at the output 124.

[0043] Generally, the residual reconstruction unit 106, the adder 108, and the prediction unit 110 function similarly to elements 16, 36, and 12 in Figure 1. In other words, they operate similarly to the above elements in Figure 1. To this end, the residual reconstruction unit 106 and the prediction unit 110 are controlled, respectively, by prediction parameters 120 and subdivisions defined by the image segmentation unit 104, according to subdivision information 118 from the extraction unit 102. This aims to predict small regions of prediction in the same way that the prediction unit 12 has done or decided to do, and to re-transform the received transformation coefficients at the same granularity as the transformation performed by the residual precoder 14. Next, the image segmentation unit 104 rearranges the subdivisions selected by the segmentation unit 20 in Figure 1 in a synchronized manner by referring to the subdivision information 118. Subsequently, the extraction unit may use the subdivision information to control data extraction, such as context selection, adjacency determination, probability estimation, and parsing of the data stream syntax.

[0044] Regarding the above embodiment, several variations are conceivable. Some of these variations, specifically the subdivision performed by the subdivision unit 28 and the joining performed by the joining unit 30, will be described in detail below. Other variations will be explained using Figures 12 to 16. Unless there is a problem, all of these modified embodiments may be applied individually or as subsets to the above-described explanations for Figures 1 and 2, respectively. For example, the subdivision units 20 and 104 do not have to determine only the subdivision of predictions and residuals for each image. Instead, they may determine the subdivision of each of the optional in-loop filters 38 and 112, either independently of or dependent on other subdivisions for coding predictions or residuals. Furthermore, the determination or re-subdivision by these elements does not have to be performed frame by frame. Instead, the re-subdivision or subdivision determined in a particular frame may be reused or adopted for a certain number of subsequent frames, after which a new subdivision may be initiated.

[0045] In order to further explain the division of images into smaller regions, the following explanation will first focus on the division performed by the subdivision units 28 and 104a. Next, the joining process performed by the joining units 30 and 104b will be explained. Finally, the fitting / prediction between planes will be explained.

[0046] When the subdivision units 28 and 104a divide an image, the image can be divided into numerous blocks for the purpose of encoding predictions and residuals for image or video data, and the blocks may be of various sizes. As described above, image 24 may be used as one or more arrays of sample values ​​for the image. In the YUV / YCbCr color space, for example, the first array may represent the luminance channels and the other two arrays may represent the chrominance channels. These arrays may have different dimensions. All arrays may be grouped into one or more planar groups, each planar group consisting of one or more consecutive planes such that each plane is included in only one planar group. For each planar group, the following applies: The first array of a particular planar group is called the primary array of that planar group. Any subsequent arrays are lower-order arrays. The division of blocks of primary arrays may be based on a quadtree approach. The optimal approach will be discussed later. The division of blocks of lower-order arrays may be derived based on the division of primary arrays.

[0047] In the embodiments described later, the subdivision units 28 and 104a are configured to divide a linear array into a number of equal-sized square blocks, so-called tree blocks, as follows: When a quadtree is used, the length of the tree block boundary is generally a power of 2, such as 16, 32, or 64. However, it should be noted that strictly speaking, other types of trees can also be used, such as binary trees or trees with any number of leaves. Furthermore, the number of children in a tree can vary depending on the level of the tree and the signal the tree represents.

[0048] In addition to the above, as mentioned above, the sample sequence may also represent information other than the video sequence, such as depth maps or light field of view. For the sake of brevity, the following explanation will focus on quadtrees as a representative example of multitrees. A quadtree is a tree in which each internal node has exactly four children. Each tree block constitutes a first-order quadtree, and each leaf of the first-order quadtree constitutes a lower-order quadtree. The first-order quadtree determines the subdivision of a given tree block for prediction, and the lower-order quadtree determines the subdivision of a given prediction block for residual coding.

[0049] The root node of a primary quadtree corresponds to a complete tree block. For example, Figure 3A represents tree block 150. Each image is assumed to be divided into a regular grid of lines and columns, like tree block 150, so that, for example, the sequence of samples is covered without gaps. However, note that for all of the block subdivisions shown below, seamless subdivision without overlap is not required. Rather, adjacent blocks may overlap each other as long as there are no leaf blocks that are proper subdivisions of adjacent leaf blocks.

[0050] Following the quadtree structure of tree block 150, each node can be further divided into four child nodes. This means that, in the case of a primary quadtree, each tree block 150 can be divided into four subblocks, each half the width and height of tree block 150. In Figure 3A, these subblocks are indicated by the labels 152a to 152d. In a similar manner, each of these subblocks can be further divided into four smaller subblocks, each half the width and height of the original subblock. In Figure 3d, an example of subblock 152c is shown, which has been subdivided into four smaller subblocks 154a to 154d. Figures 3A to 3C illustrate how tree block 150 can be divided as follows: Tree block 150 is first divided into four subblocks 152a to 152d. Then the lower left subblock 152c is further divided into four smaller subblocks 154a to 154d. Finally, as shown in Figure 3C, the upper right block 154b of these smaller subblocks is further divided into four blocks, each having 1 / 8 the width and height of the original tree block 150. These even smaller blocks are shown as 156a through 156d.

[0051] Figure 4 shows the basic tree structure of the quadtree-based partitioning example shown in Figures 3A to 3d. The numbers next to the nodes in the tree are values ​​called subdivision flags, which will be explained in detail later when describing the information transmission in the quadtree structure. The root node of the quadtree is shown at the top of the figure ("level 0"). The four branches at level 1 of this root node correspond to the four subblocks shown in Figure 3A. In Figure 3B, the third of these subblocks is further subdivided into four more subblocks, so the third node at level 1 in Figure 4 also has four branches. Furthermore, corresponding to the subdivision of the second child node (upper right) in Figure 3C, there are four lower branches connected to the second node at level 2 of the quadtree hierarchy. Nodes at level 3 are not subdivided any further.

[0052] Each leaf of the primary quadtree corresponds to a variable-size block, for which individual prediction parameters can be specified (i.e., intra-prediction mode or inter-prediction mode, motion parameters, etc.). Hereafter, these blocks will be referred to as prediction blocks. Specifically, these leaf blocks are shown in Figure 3C. Briefly referring to the explanations in Figures 1 and 2, the partitioning unit 20 or subdivision unit 28 makes the determination of the subdivision of the quadtree described above. Subdivision units 152a to 152d determine which of the tree block 150, subblocks 152a to 152d, and smaller subblocks 154a to 154d will be subdivided or further divided. As already shown above, this aims to find the optimal trade-off between fine prediction and subdivision and coarse prediction subdivision. Next, the prediction unit 12 uses the defined prediction subdivision to determine the prediction parameters described above at a granularity corresponding to the prediction subdivision. Alternatively, a defined subdivision of the prediction can be used for each of the prediction subregions represented by the blocks shown in Figure 3C, etc.

[0053] The prediction block shown in Figure 3C can be further divided into smaller blocks for the purpose of encoding the residuals. For each prediction block, that is, for each leaf node of the primary quadtree, the corresponding subdivision is determined by one or more lower-level quadtrees for encoding the residuals. For example, if the size of the residual block is up to 16x16, a given 32x32 prediction block can be divided into four 16x16 blocks, each determined by a lower-level quadtree for encoding the residuals. In this example, each 16x16 block corresponds to the root node of the lower-level quadtree.

[0054] Similar to the explanation for subdividing a given tree block into prediction blocks, each prediction block can be divided into numerous residual blocks by decomposing the lower quadtree. Each leaf of the lower quadtree corresponds to a residual block. A residual precoder 14, followed by residual reconstruction units 16 and 106, controlled by residual encoding parameters, can specify individual residual encoding parameters for each residual block (i.e., transformation mode, transformation coefficients, etc.).

[0055] In other words, the subdivision unit 28 may be configured to determine the subdivision of predictions and the subdivision of lower residuals for each image or each group of images. This is done as follows: First, the image is divided into a regular sequence of tree blocks 150, and subsets of these tree blocks are recursively partitioned by quadtree subdivision to obtain prediction subdivisions as prediction blocks. If partitioning is not performed in each tree block or leaf block of the quadtree subdivision, the prediction block may be a tree block. Subsequently, subsets of these prediction blocks are further subdivided in a similar manner as follows: If a prediction block is larger than the maximum size of the lower residual subdivision, first each prediction block is divided into a regular sequence of subtree blocks. Subsequently, subsets of these subtree blocks are subdivided according to the quadtree subdivision procedure to obtain residual blocks. If partitioning into subtree blocks is not performed in each prediction block, the residual block may be a prediction block. Furthermore, if no further subdivision into smaller regions is performed within each subtree block, or within the leaf blocks of the residual quadtree subdivision, then it may also be a subtree block.

[0056] As outlined above, a subdivision selected for a first-order array may be mapped to a lower-order array. This is straightforward when considering lower-order arrays with the same dimensions as the first-order array. However, if the dimensions of the first-order array and the lower-order array are different, a special method is required. Generally, mapping a subdivision of a first-order array to a lower-order array of different dimensions is possible by spatial mapping, i.e., spatially mapping the block boundaries of the first-order array subdivision to the lower-order array. In particular, for each lower-order array, horizontal and vertical scaling factors may be used to determine the ratio of the dimensions of the first-order array to the lower-order array. The subdivision from the lower-order array to subblocks for coding predictions and residuals may be determined using the tree blocks of the lower-order array obtained by applying relative scaling factors to the first-order quadtree and the lower-order quadtree of each arranged tree block of the first-order array. If the horizontal and vertical scaling factors are different (e.g., undersampling of 4:2:2 saturation), the resulting prediction and residual blocks of the lower-order array will not be square. In this case, it is possible to pre-determine or select, depending on the situation, whether a non-square residual block is subdivided into square blocks (for the entire sequence, a single image from the sequence, or a single prediction block or residual block). For example, in the first case, if the mapped block is not square, the encoder and decoder agree to subdivide it into square blocks. In the second case, the subdivision unit 28 sends a selection signal to the subdivision unit 104a via the data stream insertion unit 18 and the data stream 22. For example, in the case of 4:2:2 saturation subsampling, if the width of the lower array is half that of the first array and the height is the same, the height of the residual block is twice its width. By subdividing this block vertically, two square blocks are again obtained from one block.

[0057] As described above, the subdivision unit 28 or the subdivision unit 20 each transmits information about the quadtree-based subdivision to the subdivision unit 104a via the data stream 22. To this end, the subdivision unit 28 notifies the data stream insertion unit 18 of the selected subdivision for the image 24. Subsequently, the data stream insertion unit transmits to the decoding side the structure of the primary and secondary quadtrees, i.e., the subdivision of the image array into variable-sized blocks for encoding predictions or residuals within the data stream or bitstream 22.

[0058] The minimum and maximum allowable block sizes are transmitted as accompanying information and may vary from image to image. Alternatively, the minimum and maximum allowable block sizes can be fixed within the encoder and decoder. These minimum and maximum block sizes may differ between the prediction block and the residual block. To transmit the structure of the quadtree, it must be traversed, and for each node, it must be specified whether this particular node is a leaf node of the quadtree (i.e., the corresponding block is not further subdivided) or branches into four child nodes (i.e., the corresponding block is divided into four sub-blocks of half its size).

[0059] Information transfer within a single image is performed tree block by tree block in a raster scanning order, such as from left to right and from top to bottom, as shown in Figure 5A, 140. This scanning order may also be different, such as from bottom right to top left or in a checkerboard pattern. In a preferred embodiment, each tree block, i.e., each quadtree, is traversed in depth order to transfer subdivision information.

[0060] In a preferred embodiment, not only the subdivision information, i.e., the tree structure, but also the payload associated with the tree's leaf nodes, such as prediction data, is transmitted / processed in depth order. This is because depth-order traversal offers significant advantages over width-order traversal. Figure 5B shows the structure of a quadtree with leaf nodes labeled a, b, ..., j. Figure 5A shows the resulting block division. If the blocks / leaf nodes are traversed in width order, they are obtained in the order abjchidefg. In contrast, in depth order, they are abc...ij. As can be seen from Figure 5A, in depth order, the left-adjacent and top-adjacent blocks are always transmitted / processed before the current block. Thus, in motion vector prediction and context modeling, parameters specified for the left- and top-adjacent blocks can always be used to improve encoding efficiency. This is not the case in width order, for example, block j is transmitted before blocks e, g, and i.

[0061] Information is recursively transmitted for each tree block along the structure of the primary quadtree, so that for each node, a flag is transmitted specifying whether the corresponding block should be divided into four subblocks. If the value of this flag is "1" (indicating "true"), this information transmission process is repeated recursively for all four child nodes, i.e., subblocks, in the order of the raster scan (top left, top right, bottom left, bottom right), until it reaches the leaf node of the primary quadtree. It should be noted that leaf nodes are characterized by having a subdivision flag with a value of "0". If a node is at the lowest hierarchical level of the primary quadtree and corresponds to the smallest acceptable size of the predicted block, the subdivision flag does not need to be transmitted. In the example in Figures 3A to 3C, as shown at 190 in Figure 6A, "1" is transmitted first, which specifies that tree block 150 should be divided into four subblocks 152a to 152d. Next, in raster scan sequence 200, the encoding of subdivision information for all four subblocks 152a to 152d is performed recursively. For the first two subblocks 152a and 152b, a "0" is sent to indicate that subdivision will not be performed (see 202 in Figure 6A). For the third subblock 152c (bottom left), a "1" is sent to indicate that this block will be subdivision (see 204 in Figure 6A). This recursively processes the four subblocks 154a to 154d of this block. Here, a "0" is sent for the first subblock (206), and a "1" is sent for the second subblock (top right) (208). This processes the four blocks 156a to 156d, which make up the smallest block size in Figure 3C. Once the smallest allowable block size in this example is reached, further subdivision is not possible, so there is no need to transmit any more data. Alternatively, as shown in 210 in Figure 6A, "0000" is transmitted to indicate that these blocks will not be further divided. After this, "00" is transmitted for the two lower blocks in Figure 3B (see 212 in Figure 6A), and finally, "0" is transmitted for the lower right block in Figure 3A (see 214).Thus, the complete binary sequence representing the structure of the quadtree is as shown in Figure 6A.

[0062] In the binary sequence shown in Figure 6A, the different background shadings correspond to different levels of the quadtree-based subdivision hierarchy. Shading 216 represents level 0 (corresponding to a block size equal to the original tree block size), shading 218 represents level 1 (corresponding to a block size equal to half the original tree block size), shading 220 represents level 2 (corresponding to a block size equal to 1 / 4 of the original tree block size), and shading 222 represents level 3 (corresponding to a block size equal to 1 / 8 of the original tree block size). All subdivision flags at the same hierarchy level (corresponding to the same block size and the same representation in the exemplified binary sequence) may be entropy encoded using a single probabilistic model, for example, by the insertion 18. In the case of traversing in width order, the subdivision information is transmitted in a different order, as shown in Figure 6b.

[0063] Similar to the subdivision of each tree block for prediction purposes, the resulting subdivision from each prediction block to residual blocks also needs to be transmitted within the bitstream. Additionally, there may be maximum and minimum block sizes for the residual encoding transmitted as accompanying information, and these may vary from image to image. Furthermore, the maximum and minimum block sizes for residual encoding can be fixed within the encoder and decoder. At each leaf node of a first-order quadtree, as shown in Figure 3C, the corresponding prediction block may be subdivided into residual blocks of the maximum allowable size. These blocks are root nodes that constitute the lower-level quadtree structure for residual encoding. For example, if the maximum size of the residual block for an image is 64x64 and the size of the prediction block is 32x32, then the entire prediction block corresponds to one root node of a 32x32 lower-level (residual) quadtree. On the other hand, if the maximum size of the residual block for an image is 16x16, then the 32x32 prediction block consists of four 16x16 root nodes of residual quadtrees. Within each prediction block, information about the lower-level quadtree structure is transmitted for each root node in the order of raster scanning (left to right, top to bottom). As with the primary (prediction) quadtree structure, a flag is encoded for each node, specifying whether or not this particular node should be divided into four child nodes. If the value of this flag is "1", this procedure is recursively repeated for all of the corresponding four child nodes and their corresponding subblocks in the order of raster scanning (top left, top right, bottom left, bottom right) until the leaf node of the lower-level quadtree is reached. As with the primary quadtree, there is no need to transmit information about the nodes at the lowest hierarchical level of the lower-level quadtree, because these nodes correspond to the smallest block size of the residual block and cannot be divided further.

[0064] For entropy coding, the subdivision flags of residual blocks belonging to residual blocks of the same block size may be encoded using the same probabilistic model.

[0065] Thus, based on Figures 3A to 6A and following the example shown above, the subdivision unit 28 defined a primary subdivision for prediction purposes and lower subdivisions of blocks of different sizes from the primary subdivision for the purpose of encoding residuals. The data stream insertion unit 18 encoded the primary subdivision for each tree block by transmitting information in a zigzag scan order, using a bit sequence constructed according to Figure 6A, and also encoded the maximum size of the primary block and the maximum hierarchy level of the primary subdivision. For each prediction block defined in this way, the associated prediction parameters are included in the data stream. Furthermore, for each prediction block that was less than or equal to the maximum size of the residual subdivision, and for each block at the root of the residual tree that was divided from the prediction block and exceeded the maximum size defined for the residual block, similar information, namely the maximum size, maximum hierarchy level, and the encoding of a bit sequence as shown in Figure 6A, was performed. For each residual block defined in this way, the residual data is inserted into the data stream.

[0066] The extraction unit 102 extracts each bit sequence from the data stream at the input unit 116 and notifies the division unit 104 of the acquired subdivision information. Furthermore, the data stream insertion unit 18 and the extraction unit 102 may further transmit syntactic elements such as residual data output by the residual precoder 14 and prediction parameters output by the prediction unit 12, using the above order between the prediction block and the residual block. Using this order has the advantage that it may be possible to select a context suitable for encoding individual syntactic elements of a particular block by utilizing the encoded / decoded syntactic elements of adjacent blocks. Furthermore, similar to the residual precoder 14 and the prediction unit 12, the residual reconstruction unit 106 and the precoder 110 may also process individual prediction blocks and residual blocks in the order outlined above.

[0067] Figure 7 is a flowchart showing the steps taken by the extraction unit 102 to extract subdivision information from the data stream 22 when encoding is performed in the manner outlined above. In the first step, the extraction unit 102 divides the image 24 into root blocks 150 of the tree. This step is shown as step 300 in Figure 7. In step 300, the extraction unit 102 may extract the maximum size of the predicted blocks from the data stream 22. Alternatively, in step 300, the extraction unit 102 may extract the maximum hierarchical level from the data stream 22. Next, in step 302, the extraction unit 102 decodes a flag or bit from the data stream. If step 302 is performed for the first time, the extraction unit 102 recognizes that each flag is the first flag in a bit sequence belonging to the first block 150 of the root of the tree in the order 140 of scanning the root block of the tree. Since this flag is a hierarchical level 0 flag, the extraction unit 102 can determine the context in step 302 using the context modeling associated with that hierarchical level 0. For each context, an estimate of the probability of entropy decoding of the associated flag may be made. The probability estimate of the context may be applied context-specific to the symbol statistics of each context. For example, to determine a suitable context for decoding a hierarchical level 0 flag in step 302, the extraction unit 102 may select one of the following contexts from the set of contexts: The context is associated with hierarchy level 0 depending on the flags of the adjacent tree block's hierarchy level 0, or further, depending on the information contained in the bit string that defines the quadtree subdivision of the tree blocks adjacent to the currently processed tree block, such as the upper and left adjacent tree blocks.

[0068] In the next step, namely step 304, the extraction unit 102 checks whether the most recently decoded flag suggests partitioning. If so, in step 306, the extraction unit 102 partitions the current block (in this case, a tree block) or presents this partitioning to the subdivision unit 104a. Then, in step 308, it checks whether the current hierarchy level is equal to the maximum hierarchy level minus 1. For example, the extraction unit 102 may retain the maximum hierarchy level extracted from the data stream in step 300. If the current hierarchy level is not equal to the maximum hierarchy level minus 1, the extraction unit 102 increments the current hierarchy level by 1 in step 310 and returns to step 302 to decode the next flag from the data stream. Here, since the flag decoded in step 302 belongs to a different hierarchy level, according to the embodiment, the extraction unit 102 may select one of a set of other contexts belonging to the current hierarchy level. This selection may be based on the bit sequence of the decoded adjacent tree blocks, which is a subdivision as shown in Figure 6A.

[0069] If the flag is decoded and it is confirmed in step 304 that the flag does not indicate the current block's segmentation, the extraction unit 102 proceeds to step 312 to check whether the current hierarchy level is 0. If it is 0, the extraction unit 102 processes the root block of the next tree in scan order 140 in step 314. Alternatively, if there are no root blocks of the tree to be processed, the extraction of subdivision information is stopped.

[0070] It should be noted that the explanation in Figure 7 focuses only on the decoding of the flag indicating the subdivision of the prediction. In reality, step 314 may also include the decoding of other bins or syntactic elements related to the current tree block, etc. In any case, if there is a further root block of the next tree, the extraction unit 102 proceeds from step 314 to step 302 and decodes the next flag from the subdivision information, i.e., the first flag in the flag sequence relating to the root block of the new tree.

[0071] If step 312 determines that the hierarchy level is not 0, the process proceeds to step 316 to check whether there are any further child nodes related to the current node. In other words, when the extraction unit 102 performs the check in step 316, it has already been confirmed in step 312 that the current hierarchy level is a level other than 0. There is a parent node belonging to block 150, the root of the tree, or one of the smaller blocks 152a to 152d, or even smaller blocks 152a to 152d. The node in the tree structure to which the most recently decoded flag belongs has a parent node, and this is common to the other three nodes in the current tree structure. The order of scanning such child nodes that have a common parent node is illustrated in Figure 3A, with the case of hierarchy level 0 indicated by code 200. Thus, in step 316, the extraction unit 102 checks whether all four of these child nodes have been scanned in the process shown in Figure 7. If not, i.e., if there are further child nodes along with the current parent node, the process in Figure 7 proceeds to step 318. The next child node within the current hierarchy level is scanned according to the zigzag scan sequence 200. This ensures that the corresponding subblock represents the current block of process 7. Then, in step 302, the flag is decoded from the data stream relating to the current block or current node. However, if there are no more child nodes of the current parent node in step 316, the process in Figure 7 proceeds to step 320, the current hierarchy level is reduced by one, and the process proceeds to step 312.

[0072] The extraction unit 102 and the subdivision unit 104a cooperate to acquire the subdivision selected on the encoder side from the data stream by performing the steps shown in Figure 7. The process in Figure 7 summarizes the subdivision of the prediction in the case described above. Figure 8, combined with the flowchart in Figure 7, shows how the extraction unit 102 and the subdivision unit 104a cooperate to acquire the residual subdivision from the data stream.

[0073] Figure 8 shows, in particular, the steps performed by the extraction unit 102 and the subdivision unit 104a for each prediction block obtained as a result of prediction subdivision. These prediction blocks are traversed in depth order, following the zigzag scan sequence 140 within the prediction subdivision tree block 150, as shown in Figure 3C, for example, in order of depth order traversal, within each tree block 150 currently being scanned to traverse the leaf blocks. The leaf blocks of the segmented primary tree block are scanned according to the depth order traversal sequence. Subblocks of a particular hierarchical level to which the common current node belongs are scanned in zigzag scan sequence 200, first scanning the subdivisions of each of these subblocks, and then proceeding to the next subblock in this zigzag scan sequence 200. In the example in Figure 3C, the resulting scan sequence between the leaf nodes of tree block 150 is indicated by reference numeral 350.

[0074] For the prediction block currently being scanned, the process in Figure 8 begins at step 400. In step 400, an internal parameter representing the current size of the current block is set to equal the size of the residual subdivision hierarchy level 0, i.e., the maximum block size of the residual subdivision. It should be noted that the maximum size of the residual block may be smaller than or larger than the minimum block size of the prediction subdivision. In other words, in this embodiment, the encoder can be freely selected from the above possibilities.

[0075] In the next step, step 402, it is checked whether the size of the prediction block of the currently scanned block is greater than the internal parameter representing the current size. If it is greater than the internal parameter, the currently scanned prediction block is larger than the maximum size of the residual block. This prediction block is a leaf block of the prediction subdivision or a tree block of the prediction subdivision and is not further subdivided. In this case, the process in Figure 8 proceeds to step 300 in Figure 7. That is, the currently scanned prediction block is divided into blocks of the residual tree root, and the first flag of the flag sequence of the first residual tree block within this currently scanned prediction block is decoded in step 302, etc.

[0076] However, if the size of the prediction block currently being scanned is less than or equal to the internal parameter representing the current size, the process in Figure 8 proceeds to step 404, where the size of the prediction block is checked and it is determined whether or not it is equal to the internal parameter representing the current size. If they are equal, the splitting step 300 may be skipped, and the process proceeds directly to step 302 in Figure 7.

[0077] However, if the size of the prediction block currently being scanned is smaller than the internal parameter representing the current size, the process in Figure 8 proceeds to step 406, where the hierarchy level is increased by one, and the current size is set to the size of the new hierarchy level, such as the value obtained by dividing the current size by 2 (in both axes in the case of quadtree subdivision). Then, the check in step 404 is performed again. The effect of the loop formed in steps 404 and 406 is that the hierarchy level always corresponds to the size of the corresponding block being partitioned, regardless of whether each prediction block is less than or greater than the maximum size of the residual block. Thus, when decoding the flag in step 302, context modeling is performed depending on both the hierarchy level and the block size that the flag refers to. Using different contexts for flags of different hierarchy levels or block sizes has the following advantages: The probability estimation of the occurrence of the flag value fits the actual probability distribution well. On the other hand, the number of contexts to be managed is relatively small. As a result, the overhead of managing contexts is reduced, and the fit of the context to the actual symbol statistics is improved.

[0078] As described above, there may be multiple sample sequences, and these sample sequences may be grouped into one or more planar groups. The input signal input to the input unit 32 and encoded may be, for example, one image from a video sequence, or a still image. Thus, the image may be provided in the form of one or more sample sequences. In the context of encoding images from a video sequence or still images, the sample sequences may refer to three color planes such as red, green, and blue, or planes of luminance and saturation such as YUV or YCbCr color representations. There may also be a sample sequence representing alpha, i.e., transparency, and / or depth information for 3D video material. A number of such sample sequences may be grouped together as so-called planar groups. For example, luminance (Y) may be in one planar group with only one sample sequence, and saturation such as CbCr may be in another planar group with two sample sequences. In another example, YUV may be in one planar group with three matrices, and depth information for 3D video material may be in another planar group with only one sample sequence. For all planar groups, a first-order quadtree structure may be encoded within the data stream 22 to represent the partition into prediction blocks, and for each prediction block, a second-order quadtree structure may be encoded to represent the partition into residual blocks. Thus, following the first example above, where the luminance elements form one planar group and the saturation elements form other planar groups, there is one quadtree structure for the prediction blocks of the luminance plane, one quadtree structure for the residual blocks of the luminance plane, one quadtree structure for the prediction blocks of the saturation plane, and one quadtree structure for the residual blocks of the saturation plane. However, in the second example above, there is one quadtree structure for the prediction blocks of both luminance and saturation (YUV), one quadtree structure for the residual blocks of both luminance and saturation (YUV), one quadtree structure for the prediction blocks of depth information for 3D video material, and one quadtree structure for the residual blocks of depth information for 3D video material.

[0079] Furthermore, the above description described a method in which an input signal is divided into prediction blocks using a first-order quadtree structure, and these prediction blocks are further subdivided into residual blocks using a lower-order quadtree structure. In another embodiment, the subdivision does not have to end at the lower-order quadtree stage. That is, the blocks obtained from the division using the lower-order quadtree structure may be further subdivided using a third-order quadtree structure. This division may be performed with the aim of further using encoding tools to facilitate the encoding of the residual signal.

[0080] The above description focuses on the subdivision performed by the subdivision section 28 and the subdivision section 104a, respectively. As described above, the subdivision is defined by the subdivision sections 28 and 104a, respectively, and controls the granularity of processing performed by the above modules of the encoder 10 and decoder 100. However, in the embodiment described below, the coupling section 30 and the coupling section 104b follow the subdivision sections 228 and 104a, respectively. However, the coupling sections 30 and 104b are optional and may not be present.

[0081] However, in practice, as will be explained in more detail below, the coupling unit provides the encoder with the opportunity to group or cluster some of the prediction blocks or residual blocks, so that other modules, or at least some of the other modules, can handle these groups of blocks together. For example, if the transmission of information about grouping prediction blocks with common parameters for all blocks belonging to this group is more promising in terms of rate / distortion than the transmission of prediction parameters individually for all of these prediction blocks, the prediction unit 12 may sacrifice small deviations between the prediction parameters of some prediction blocks and instead use prediction parameters common to all of these prediction blocks, as determined by optimization using the subdivision of the subdivision unit 28. However, the process of obtaining predictions based on these common prediction parameters may be performed by the prediction units 12 and 110 themselves with respect to the prediction blocks. However, it is also possible for the prediction units 12 and 110 to perform the prediction process on the entire group of prediction blocks at once.

[0082] As will be explained in more detail below, the following is also possible: The grouping of prediction blocks is not solely for the purpose of using the same or common prediction parameters for a group of prediction blocks, or, furthermore, this allows the encoder 10 to transmit one prediction parameter for this group along with the residuals of the predictions of the prediction blocks belonging to this group, thereby reducing the overhead of information transmission for the prediction parameters of this group. In the latter case, the impact of the coupling process to the data stream insertion unit 18 is less than that of the determination by the residual precoder 14 and the prediction unit 12. However, details will be explained below. Strictly speaking, it should be noted that the above embodiments also apply to other subdivisions such as the residual subdivision and filter subdivision described above.

[0083] First, the joining of sample sets such as prediction blocks and residual blocks, as described above, is motivated in a more general sense, that is, it is not limited to the subdivision of the multitree described above. However, in the following description, we will focus on the joining of blocks obtained as a result of the subdivision of the multitree in the above embodiment.

[0084] Generally speaking, combining syntactic elements associated with a specific set of samples for the purpose of transmitting associated coding parameters can reduce the rate of accompanying information in image and video encoding applications. For example, the sample sequence of a signal to be encoded is typically partitioned into a specific set of samples or a set of samples. A specific set of samples may represent a rectangular or square block, or it may represent another set of samples containing regions of any shape, such as a quadratic block. In the embodiments described above, the simply connected regions were the prediction and residual blocks obtained as a result of subdividing a multitree. The subdivision of the sample sequence may be fixed by syntax, and, as described above, the subdivision information may be transmitted, at least partially, within the bitstream. To keep the rate of accompanying information for transmitting the subdivision information low, the syntax usually allows only a limited number of choices that result in simple partitioning, such as subdividing blocks into smaller blocks. The set of samples is associated with a specific coding parameter, which specifies the mode of coding for prediction information and residuals, etc. Further details regarding this are described above. For each set of samples, individual coding parameters specifying the coding of the predictions and / or residuals may be transmitted. To improve coding efficiency, several advantages can be obtained by combining two or more sets of samples into so-called groups of sample sets, as described below. These advantages are further explained below. For example, sets of samples may be combined such that all sets of samples in such a group share the same coding parameters. The same coding parameters can be transmitted together with one of the sets of samples in the group. In this way, it is not necessary to transmit the coding parameters individually to each set of samples in the group of sample sets, but instead the coding parameters are transmitted only once to the entire group of sample sets. As a result, the rate of accompanying information for the transmission of coding parameters is reduced, and the overall coding efficiency may be improved. Alternatively, it is possible to transmit additional refinements for one or more coding parameters to one or more sets of samples in a group of sample sets.Refinement can be applied either to the entire set of samples in the group, or only to the set of samples being transmitted.

[0085] Furthermore, the encoder has increased flexibility in creating the bitstream 22 depending on the combination of samples, which will be described further below. This is because the combination approach significantly increases the options for segmenting the image sample sequence. The encoder can improve encoding efficiency because it can choose from more options, such as minimizing a specific amount of rate / distortion. There are several possible ways in which the encoder can operate. In a simple approach, the encoder first determines the optimal subdivision of the sample sequence. Referring briefly to Figure 1, the subdivision unit 28 can determine the optimal subdivision in the initial stage. Then, for each set of samples, it is checked whether a specific amount of rate / distortion loss is reduced by combining it with another set of samples, or another group of sets of samples. Here, the predictive parameters associated with the group of combined sample sets can be re-evaluated by performing a search for new motion or by predictive parameters that have been determined for a common set of samples. Then, candidate sets of samples, or groups of sets of samples, for combination are evaluated for the group of samples being considered. A more detailed approach evaluates the loss at a specific rate / distortion for a group of additional candidate samples.

[0086] It should be noted that the joining approach described below does not change the order in which the sample sets are processed. In other words, the idea of ​​joining can be implemented in a way that does not increase latency, that is, so that each set of samples can be decoded at the same time as if the joining approach were not used.

[0087] For example, if the bitrate reduction achieved by decreasing the number of encoded predictive parameters is greater than the bitrate separately used for encoding merge information to indicate the join to the decoding side, the encoding efficiency will be further improved as a result of the join approach described later. It should also be noted that the extension of the syntax for joins described increases the encoder's flexibility in selecting how to partition an image or planar group into blocks. In other words, the encoder is not restricted to first performing subdivision and then checking whether any of the resulting blocks have the same or similar sets of predictive parameters. A simpler alternative is for the encoder to first determine subdivision based on the rate / distortion loss, and then for each block, check whether the rate / distortion loss will be reduced by joining with an adjacent block or one of the associated determined groups of blocks. Here, the predictive parameters associated with a new group of blocks can be re-evaluated by performing a new motion search or by determining the predictive parameters for the current block, and adjacent blocks or groups of blocks can be evaluated for the new group of blocks. The join information can be transmitted block by block. In effect, the coupling can also be understood as an estimation of the prediction parameter of the current block, where the estimated prediction parameter is set to be equal to one prediction parameter of an adjacent block. Alternatively, residuals may be transmitted for blocks within a group of blocks.

[0088] Thus, the fundamental idea underlying the concatenation described later is to reduce the bitrate required to transmit prediction parameters or other coding parameters by combining adjacent blocks into groups of blocks. In this case, each group of blocks is associated with a unique set of coding parameters, such as prediction parameters or residual coding parameters. If concatenation information exists, it is transmitted appended to the subdivision information within the bitstream. The advantage of the concatenation concept is the improved coding efficiency resulting from the reduced rate of information associated with the coding parameters. It should be noted that the concatenation process described here can be extended to dimensions other than spatial dimensions. For example, groups of samples or blocks present in images from several different videos can be combined into a single group of blocks. Concatenation can also be applied to 4D compression and optical field coding.

[0089] Therefore, returning to the explanation in Figures 1 to 8, regardless of the specific method by which the subdivision units 28 and 104a subdivide the image, there are advantages to the subsequent joining process. Specifically, the latter can subdivide images in a similar manner to H.264, etc. That is, each image can be subdivided into a regular arrangement of rectangular or square macroblocks of a predetermined size, such as 16x16 luminance samples or the size of information transmitted within the data stream. Each macroblock has specific encoding parameters associated with it, and in particular, the data stream contains segmentation parameters and corresponding prediction parameters. The segmentation parameters define the segmentation of each macroblock into a regular subgrid, which is segmented a number of times such as 1, 2, 4, which is the granularity of the prediction. The granularity of segmentation for residuals and the corresponding transformation of residuals are also defined.

[0090] In any case, the advantages outlined above, such as the reduction of the bitrate of the accompanying information in the encoding of images and videos, are obtained by the coupling. A particular set of samples, representing a square or rectangular block, a region of any shape, or any other set of samples, such as any simply connected region or sample, is typically coupled to a particular set of coding parameters. For each set of samples, the coding parameters are included in the bitstream, and the coding parameters represent prediction parameters, etc., that specify how to predict the corresponding set of samples using the encoded samples. The segmentation from the sample sequence of an image to a set of samples may be fixed by syntax or may be conveyed by the corresponding subdivision information in the bitstream. The coding parameters of a set of samples may be transmitted in a predetermined order specified by syntax. The coupling unit 30 can transmit information to a group of sample sets about a common set of samples, or the current block, such as a prediction block or residual block, coupled with one or more other sets of samples, according to the function of coupling. Therefore, the coding parameters of a group of sample sets need to be transmitted only once. In certain embodiments, if the current set of samples is joined to a set of samples whose encoded parameters have already been transmitted, or to an existing group of samples, the encoded parameters of the current set of samples are not transmitted. Instead, the encoded parameters of the current set of samples are set to be equal to the encoded parameters of the set of samples or group of samples to which the current set of samples is joined. Alternatively, additional refinements for one or more encoded parameters can be transmitted for the current set of samples. The refinements can be applied to all of the sample sets in the group, or only to the sample sets to which the refinements are transmitted.

[0091] In this embodiment, for each set of samples, such as the prediction blocks, residual blocks, and leaf blocks of the multi-tree subdivision described above, the set of all encoded / decoded sample sets is referred to as the "set of causal sample sets." For example, see Figure 3C. All the blocks shown in this figure are the results of a specific subdivision, such as a prediction subdivision or residual subdivision, or a multi-tree subdivision, and the encoding / decoding order defined between these blocks is defined by arrow 350. If we consider a particular block among these blocks as the current sample set or the current simply connected region, then the set of causal sample sets for that block consists of all blocks that precede the current block in order 350. However, other subdivisions that do not use multi-tree subdivision are also possible, as far as the joining principles described below are concerned.

[0092] The set of samples that can be used to combine with the current set of samples is referred to below as the "set of candidate sets of samples" and is always a subset of the "set of causal sets of samples". The way in which the subset is formed can be communicated to the decoder or specified within the data stream or bitstream from the encoder to the decoder. When a particular set of current samples is encoded / decoded and the set of candidate sets of samples is not empty, the encoder communicates within the data stream whether a common set of samples can be combined with one of the sets of samples in this set of candidate samples, and the decoder derives from the data stream whether or not they can be combined. If they can be combined, they are communicated together. Otherwise, the set of candidate sets of samples is empty, and combining cannot be used in this block.

[0093] There are various ways to determine a subset of the set of causal samples that represents a set of candidate samples. For example, the determination of a set of candidate samples may be based on a uniquely geometrically defined sample in the current set of samples, such as the top-left image sample of a rectangular or square block. Starting from this uniquely geometrically defined sample, a non-zero specific number of samples is determined. This represents the directly spatially adjacent samples to this uniquely geometrically defined sample. For example, this non-zero specific number of samples includes those adjacent to the top and left of the uniquely geometrically defined sample in the current set of samples. The non-zero number of adjacent samples is at most 2, 1 if one of the top or left adjacent samples is unavailable or lies outside the image, and zero if neither has any adjacent samples.

[0094] Then, a set of candidate sample sets is determined to include a set of samples that contain at least one non-zero adjacent sample from the above set of adjacent samples. See, for example, Figure 9A. Let the current set of samples being considered for joining be block X, and its uniquely geometrically defined sample be the top-left sample exemplified by 400. The samples adjacent to the top and left of sample 400 are shown as 402 and 404. Sets of causal sample sets, or sets of causal blocks, are highlighted with shading. Of these blocks, blocks A and B contain one of the adjacent samples 402 and 404, so these blocks form a set of candidate blocks, or a set of candidate sample sets.

[0095] In another embodiment, the set of candidate sample sets determined for the purpose of joining may additionally or exclusively include a set of samples containing a specific non-zero number of samples. The specific non-zero number of samples is 1 or 2, and these samples are spatially in the same location but contained in different images, i.e., previously encoded / decoded images. For example, in addition to blocks A and B in Figure 9A, a block of previously encoded images containing a sample in the same location as sample 400 can be used. Note that the above non-zero number of adjacent samples can be defined using only the upper adjacent sample 404 or only the left adjacent sample 402. In general, the set of candidate sample sets may be derived from previously processed data in the current image or other images. This derivation may include spatial directional information such as transformation coefficients associated with a particular direction or the tilt of the current image, and may also include temporal directional information such as representations of adjacent motion. The set of candidate sample sets may be derived from such data available to the receiver / decoder, and, if present, from other data and associated information in the data stream.

[0096] It should be noted that candidate sample sets are derived in parallel at both the encoder-side coupling unit 30 and the decoder-side coupling unit 104b. As described above, both may independently determine sets of candidate sample sets based on a predetermined method notified to both. Alternatively, the encoder may transmit hint information within the bitstream. This places the coupling unit 104b in a position to derive these sets of candidate sample sets in the same way that the encoder-side coupling unit 30 determined the sets of candidate sample sets.

[0097] As will be explained in detail below, the joiner 30 and the data stream insertion unit 18 work together to transmit one or more syntactic elements for each set of samples. These syntactic elements specify whether the set of samples is to be joined with another set of samples that is part of a group of already joined sets of samples, and specify a set of candidate sets of samples to be used for the join. Subsequently, the extraction unit 102 extracts these syntactic elements and notifies the joiner 104b accordingly. In particular, in the specific embodiments described later, one or two syntactic elements are transmitted to specify the join information for a particular set of samples. The first syntactic element specifies whether the current set of samples is to be joined with another set of samples. The second syntactic element is transmitted only if the first syntactic element specifies that the current set of samples is to be joined with another set of samples, and specifies a set of candidate sets of samples to be used for the join. If the derived set of candidate sets of samples is empty, the first syntactic element does not need to be transmitted. In other words, the first syntactic element may be transmitted only if the derived set of candidate sample sets is not empty. The second syntactic element may be transmitted only if the derived set of candidate sample sets contains multiple sample sets, because if the set of candidate sample sets contains only one sample set, no further selection is possible. Furthermore, if the set of candidate sample sets contains multiple sample sets, but all of the sample sets within the candidate sample sets are associated with the same encoding parameter, the second syntactic element does not need to be transmitted. In other words, the second syntactic element may be transmitted only if at least two of the sample sets within the derived set of candidate sample sets are associated with different encoding parameters.

[0098] Within the bitstream, information about the combination of a set of samples may be encoded before the prediction parameters or other specific encoding parameters associated with that set of samples. The prediction parameters or encoding parameters may only be transmitted if the combination information communicates that the current set of samples will not be combined with any other set of samples.

[0099] Information about the combination of a particular set of samples, i.e., a block, may be encoded after the transmission of a true subset of the prediction parameters, or more generally, the encoding parameters associated with each set of samples. A subset of prediction / encoding parameters may consist of one or more reference image indices, or components of one or more motion parameter vectors, or a reference index and components of one or more motion parameter vectors, etc. A transmitted subset of prediction or encoding parameters can be used to derive a set of candidate sample sets from the larger set of provisional candidate sample sets derived above. For example, the magnitude of the difference or gap can be calculated between the encoded prediction and encoding parameters of the current sample set and the corresponding prediction and encoding parameters of the preliminary set of candidate sample sets, based on a predetermined gap criterion. Only sample sets whose calculated magnitude of the difference or gap is less than or equal to a predetermined threshold or derived threshold are included in the final, i.e., reduced set of candidate sample sets. See, for example, Figure 9A. Let the current sample set be block X. Suppose a subset of encoding parameters associated with this block has already been inserted into data stream 22. For example, suppose block X is a prediction block, and the true subset of the coding parameters is a subset of the prediction parameters for this block X, such as a subset of the set containing the image reference index, and motion mapping information, such as motion vectors. If block X is a residual block, the subset of the coding parameters is a subset of residual information, such as transformation coefficients within block X and a map representing the locations of significant transformation coefficients. Based on this information, both the data stream insertion unit 18 and the extraction unit 102 can use this information to determine subsets of blocks A and B that, in this particular embodiment, form a preliminary set of the candidate sample set described above. In particular, since blocks A and B belong to a set of causal sample sets, their coding parameters are available to both the encoder and decoder as the coding parameters of block X are currently being coded / decoded.Therefore, using the above comparison of the magnitude of the difference, any number of blocks in the preliminary sets of candidate sample sets A and B may be excluded. The resulting reduced set of the candidate sample sets may then be used as described above, that is, to determine whether, depending on the number of sample sets in the reduced set of the candidate sample sets, a merge metric representing the join is transmitted within the data stream or extracted from the data stream, and whether a second syntactic element needs to be transmitted within the data stream or extracted from the data stream. The second syntactic element represents which sample set in the reduced set of the candidate sample sets will be the block to join. In other words, the determination of a combination for a given single-connected region, or the transmission of syntactic elements of each combination, may be based on the number of single-connected regions that are in a predetermined relative position to the given single-connected region and simultaneously have coding parameters associated with a first subset of coding parameters for the given single-connected region that satisfy a predetermined relationship, and the adoption or prediction for extracting the residual of the prediction may be performed on a second subset of coding parameters for the given single-connected region. In other words, only one of the number of single-connected regions that are in a predetermined relative position to the given single-connected region and simultaneously have coding parameters associated with a first subset of coding parameters for the given single-connected region that satisfy a predetermined relationship, or a subset of a specified coding parameter, may be adopted from the second subset of the given single-connected region and used, respectively, for the prediction of the second subset of the given single-connected region.

[0100] The thresholds used to compare the gaps may be fixed and communicated to both the encoder and decoder, or they may be derived based on calculated gaps, such as the median of the difference values ​​or other central tendencies. In this case, the reduced set of candidate samples is inevitably a proper subset of the preset set of candidate samples. Alternatively, only those sets of samples are selected from the preset set of candidate samples that minimize the gap based on the gap magnitude. Alternatively, one set of samples is selected from the preset set of candidate samples using the gap magnitude described above. In the latter case, the only information that needs to be specified in the join is whether the current set of samples is joined with a single set of candidate samples.

[0101] Therefore, the set of candidate blocks is formed or derived as described below with reference to Figure 9A. Starting from the position 400 of the top-left sample of the current block X in Figure 9A, the position of the adjacent sample 402 to its left, and the position of the adjacent sample 404 above it are derived by the encoder and decoder. Thus, the set of candidate blocks can have at most two elements, namely, blocks containing one of the two sample positions from the shaded set of causal blocks in Figure 9A, which in Figure 9 would be blocks B and A. Thus, the set of candidate blocks can have as its elements only the two blocks directly adjacent to the top-left sample position of the current block. In other embodiments, the set of candidate blocks is formed by all blocks containing one or more samples that were encoded before the current block and represent any sample in the current block that is directly spatially adjacent. Direct spatial adjacency may be limited to direct adjacency to the left, and / or to the top, and / or to the right, and / or to the bottom, of any sample in the current block. See, for example, Figure 9B, which shows the subdivision of another block. In this case, the candidate block contains four blocks, namely blocks A, B, C, and D.

[0102] Additionally, the set of candidate blocks may additionally or exclusively include blocks containing one or more samples that are in the same position as any sample in the current block but are contained in a different image, i.e., an encoded / decoded image.

[0103] Alternatively, the set of candidate blocks represents a subset of the aforementioned set of blocks, determined by adjacency relationships in the spatial or temporal direction. The subset of candidate blocks may be fixed, transmitted, or derived. In deriving a subset of candidate blocks, decisions about the image or other blocks in the image may be taken into consideration. For example, blocks associated with the same coding parameters, or coding parameters that are more similar than other candidate blocks, may not be included in the set of candidate blocks.

[0104] The following description of the embodiments applies when, at most, only two blocks are considered as candidates, including the samples adjacent to the left and top of the top-left sample of the current block.

[0105] If the set of candidate blocks is not empty, a flag called merge_flag is propagated that specifies whether the current block will be merged with any of the candidate blocks. If merge_flag is equal to 0 (indicating "false"), this block is not merged with any of the candidate blocks, and all encoding parameters are transmitted as usual. If merge_flag is equal to 1 (indicating "true"), the following applies: If the set of candidate blocks contains only one block, this candidate block is used for the merge. Otherwise, the set of candidate blocks contains two blocks. If the prediction parameters of these two blocks are identical, these prediction parameters are used for the current block. Otherwise (if the two blocks have different prediction parameters), a flag called merge_left_flag is propagated. If merge_left_flag is 1 (indicating "true"), the block containing the sample position adjacent to the left of the top-left sample position of the current block is selected from the set of candidate blocks. If merge_left_flag is 0 (indicating "false"), another block (i.e., an adjacent block at the top) is selected from the set of candidate blocks. The prediction parameters of the selected block are used for the current block.

[0106] To summarize the several embodiments of the coupling described above, refer to Figure 10, which shows the steps taken by the extraction unit 102 to extract coupling information from the data stream 22 input to the input unit 116.

[0107] Processing begins at 450, which identifies candidate blocks or sets of samples for the current set or block of samples. Since the encoding parameters for the blocks are transmitted in a specific one-dimensional order within the data stream 22, Figure 10 illustrates the process of obtaining join information for the currently scanned set or block of samples.

[0108] As described above, the identification and step 450 may include identification from the set of decoded blocks, i.e., causal blocks, based on the perspective of adjacency. For example, adjacent blocks containing specific adjacent samples that are spatially or temporally adjacent to one or more geometrically predetermined samples of the current block X may be designated candidates. Furthermore, the identification step may include two stages: a first stage which includes the above identification, i.e., identification based on adjacency, to obtain a preliminary set of candidate blocks; and a second stage in which only blocks decoded from the data stream before step 450 become designated candidates whose transmitted coding parameters satisfy a specific relationship with a proper subset of the coding parameters of the current block X.

[0109] Next, the process proceeds to step 452, where it is determined whether the number of candidate blocks is greater than zero. If it is, the merge_flag is extracted from the data stream in step 454. Step 454 of the extraction may include entropy decoding. The context for the entropy decoding of the merge_flag in step 454 may be determined, for example, based on syntactic elements belonging to a set of candidate blocks or a pre-set such as candidate blocks. In this case, the degree of dependence on syntactic elements may be limited to information about whether the blocks belonging to the set in question were subject to merge. An estimation of the probability of the selected context may be applied.

[0110] On the other hand, if the number of candidate blocks is determined to be zero in step 452, the process in Figure 10 proceeds to step 456, where the encoding parameters of the current block are extracted from the bitstream. In the specific cases described above, the remaining encoding parameters are extracted, and then the extraction unit 102 proceeds to process the next block in the order of block scanning, such as the order 350 shown in Figure 3C.

[0111] Returning to step 454, the process proceeds to step 458 after the extraction in step 454, where it is checked whether the extracted merge_flag indicates whether a merge of the current block will occur. If a merge will not occur, the process proceeds to step 456 above. If a merge will occur, the process proceeds to step 460, where it is checked whether the number of candidate blocks is 1. If the number of candidate blocks is 1, there is no need to indicate and transmit a specific candidate block from among the candidate blocks, so the process in Figure 10 proceeds to step 462, where the merge partner of the current block is set to that single candidate block. Then, in step 464, the coding parameters of the merged partner block are used to adapt or predict those coding parameters, or the remaining coding parameters of the current block. In the case of adaptation, the missing coding parameters of the current block are simply copied from the merged partner block. In the other case, namely prediction, step 464 may further extract residual data from the data stream. This residual data relates to the residuals of the prediction of the missing coding parameters of the current block. Furthermore, this residual data may be combined with predictions of these missing coding parameters obtained from the block being joined.

[0112] On the other hand, if step 460 determines that the number of candidate blocks is greater than 1, the process in Figure 10 proceeds to step 466, where it is checked whether the coding parameters or the corresponding parts of the coding parameters, i.e., the parts related to the portion of the data stream for the current block that has not yet been transmitted, are identical to each other. If they are identical, in step 468, these common coding parameters are set as the reference for joining. Alternatively, the candidate blocks are set as the joining partners. Then, in step 464, the respective coding parameters are used for adaptation or prediction.

[0113] It should be noted that the coupling partner itself may be the block from which the coupling was performed. In this case, the coding parameters adopted or obtained by prediction of the coupling partner are used in step 464.

[0114] On the other hand, if this is not the case, i.e., if the coding parameters are not identical, the process in Figure 10 proceeds to step 470, in which the syntactic element merge_left_flag is further extracted from the data stream. A separate set of contexts may be used for the entropy decoding of this flag. Alternatively, the set of contexts used for the entropy decoding of merge_left_flag may consist of only one context. After step 470, the candidate block suggested by merge_left_flag is set as the merge partner in step 472 and used for adaptation or prediction in step 464. After step 464, the extraction unit 102 proceeds to process the next block in block order.

[0115] Naturally, many other forms exist. For example, instead of the separate syntactic elements merge_flag and merge_left_flag mentioned above, the syntactic elements may be combined and transmitted within the data stream, and the combined syntactic elements may convey information about the merge process. Also, the merge_left_flag mentioned above may be transmitted within the data stream regardless of whether the two candidate blocks have the same prediction parameters, thereby reducing the computational overhead for performing the process shown in Figure 10.

[0116] As already shown in Figure 9B, the set of candidate blocks may contain three or more blocks. Furthermore, information about joining, i.e., whether a block is joined (and if so, with a candidate block), may be transmitted by one or more syntactic elements. Whether a block is joined with any of the candidate blocks can be specified by a single syntactic element, such as the merge_flag mentioned above. The flag may only be transmitted if the set of candidate blocks is not empty. A second syntactic element, such as the merge_left_flag mentioned above, may transmit information about which candidate block is used for joining. Typically, this indicates a selection from two or more candidate blocks. The second syntactic element may only be transmitted if the first syntactic element transmits a signal indicating that the current block is joined with one of the candidate blocks. Additionally, the second syntactic element may only be transmitted if the set of candidate blocks contains multiple candidate blocks, and / or if any of the candidate blocks have different prediction parameters than the other candidate blocks. The syntax will depend on the number of candidate blocks and / or how different the prediction parameters associated with the candidate blocks are.

[0117] The syntax that communicates which of the candidate blocks will be used may be set simultaneously and / or in parallel on the encoder and decoder sides. For example, if there are three choices for the candidate block identified in step 450, the syntax is selected so that only these three choices are available and considered for entropy coding, for example, in step 470. In other words, the syntax elements are selected so that the symbol alphabet has as many elements as there are choices for the candidate block. The possibility of all other choices can be considered zero, and entropy coding / decoding may be coordinated simultaneously in the encoder and decoder.

[0118] Furthermore, as mentioned in step 464, the prediction parameters estimated as a result of the join process may represent the complete set of prediction parameters associated with the current block. Alternatively, they may represent a subset of these prediction parameters, such as the prediction parameters for one hypothesis in a block where predictions are made using multiple hypotheses.

[0119] As mentioned above, syntactic elements related to the join information can be entropy-encoded using context modeling. Syntactic elements may consist of merge_flag and merge_left_flag (or similar syntactic elements) as described above. In specific examples, such as in step 454, one of three context models or contexts can be used to encode / decode merge_flag. The index merge_flag_ctx of the context model used may be derived as follows: If the set of candidate blocks contains two elements, the value of merge_flag_ctx is equal to the sum of the merge_flag values ​​of the two candidate blocks. On the other hand, if the set of candidate blocks contains one element, the value of merge_flag_ctx may be equal to twice the value of the merge_flag of this one candidate block. Since each merge_flag of adjacent candidate blocks is either 1 or 0, three contexts can be used for merge_flag. merge_left_flag may be encoded using only a single probabilistic model.

[0120] In another embodiment, a different context model may be used. For example, non-binary syntactic elements may be mapped to a set of binary symbols called a bin. A context model for some syntactic elements or bins of syntactic elements that define the binding information may be derived based on the number of transmitted syntactic elements in adjacent blocks, or the number of candidate blocks, and other syntactic elements or bins of syntactic elements may be encoded in a fixed context model.

[0121] With regard to the description of block joining described above, the set of candidate blocks may be derived in a manner that modifies any of the embodiments described above, as follows: the candidate blocks are restricted to blocks that use motion-compensated prediction or mutual prediction, respectively. Only such blocks can be elements of the set of candidate blocks. The transfer of merge information and context modeling can be performed as described above.

[0122] Returning to the combination of the multi-tree subdivision embodiment described above and the join configuration now being discussed, when dividing an image into variable-sized square blocks using a quadtree-based subdivision structure, syntactic elements specifying joins, such as merge_flag and merge_left_flag, are treated alternately with prediction parameters transmitted for each leaf node of the quadtree structure. For example, consider Figure 9A again. Figure 9A shows an example of quadtree-based image subdivision into variable-sized prediction blocks. The top two blocks of the largest size are called tree blocks; that is, the maximum possible size prediction blocks. The other blocks in this figure are obtained by subdividing the corresponding tree blocks. The current block is indicated by "X". All shaded blocks are encoded / decoded before the current block, so that a set of causal blocks is formed. As detailed in the description of the derivation of the set of candidate blocks for one embodiment, only blocks containing samples directly adjacent (i.e., above or to the left) to the position of the top-left sample of the current block can be elements of the set of candidate blocks. Therefore, the current block can be merged with either block "A" or block "B". If merge_flag is equal to 0 (indicating "false"), the current block "X" will not be merged with either of the two blocks. If blocks "A" and "B" have the same prediction parameters, merging with either block will result in the same outcome, so there is no need to distinguish between them. Therefore, in this case, merge_left_flag is not transmitted. However, if blocks "A" and "B" have different prediction parameters, if merge_left_flag is 1 (indicating "true"), blocks "X" and "B" will be merged, while if merge_left_flag is 0 (indicating "false"), blocks "X" and "A" will be merged. In another preferred embodiment, additional adjacent (transmitted) blocks represent candidates for merging.

[0123] Figure 9B shows another example. Here, the current block "X" and the block "B" adjacent to its left are tree blocks, i.e., they represent the maximum allowable block size. The size of the block "A" adjacent to the top is one-quarter the size of the tree block. Blocks that are elements of a set of causal blocks are shaded. In one preferred embodiment, the current block "X" can only be joined with two blocks "A" or "B", and cannot be joined with any other blocks adjacent to it at the top. In another preferred embodiment, additional adjacent (transmitted) blocks represent candidates for joining.

[0124] Before proceeding to the description of the embodiments of the present application that handle different sample sequences within a single image, the above description of the embodiments that perform information transmission by subdividing a multitree while simultaneously performing joining has made it clear that these embodiments offer the advantage of being usable independently of each other. In other words, as already explained above, the combination of multitree subdivision and joining provides concrete advantages. However, advantages can also be obtained in other ways. For example, the joining function is embodied in subdivision performed by subdivision units 30, 104a, corresponding to macroblock subdivision that regularly divides these macroblocks into smaller units, rather than based on quadtree or multitree subdivision. On the other hand, the combination of multitree subdivision that transmits an indication of the maximum size of a tree block within a bitstream and multitree subdivision that transmits the corresponding encoding parameters of a block using a depth-ordered traverse sequence provides advantages regardless of whether the joining function is used in parallel or not. In general, the advantages of joining can be intuitively understood by considering the following. The coding efficiency of a sample sequence is improved when the syntax for coding the sample sequence is extended in a way that not only allows for the subdivision of blocks but also allows for the joining of two or more blocks obtained after subdivision. As a result, a group of blocks is obtained that are coded with the same prediction parameters. The prediction parameters for such a group of blocks need to be coded only once. Again, regarding the joining of sample sets, a possible sample set is a set of rectangular or square blocks, and the joined sample set is a collection of rectangular or square blocks. A possible sample set can also be an image region of any shape, and the joined sample set is a collection of image regions of any shape.

[0125] The following section focuses on how to handle different sample sequences in an image when each image has multiple sample sequences. The embodiments outlined in the following sub-descriptions offer advantages regardless of the type of subdivision used, i.e., whether or not subdivision is based on multi-tree subdivision, and whether or not joining is used. Before describing specific embodiments of how to handle different sample sequences within a single image, a brief introduction to the handling of different sample sequences per image will be provided before discussing the main topic of these embodiments.

[0126] The following section focuses on the coding parameters between different sample array blocks within a single image during image or video encoding. In particular, it emphasizes how to predict coding parameters between different sample arrays within a single image in various environments, including not only the encoders and decoders shown in Figures 1 and 2, but also other image or video encoding environments. As mentioned above, a sample array can represent a sample array associated with various color components, or a sample array that associates an image with additional information such as transparency data or depth maps. A sample array associated with the color components of an image is also called a color plane. The technique described below, also known as inter-plane adoption / prediction, can be used in image and video encoders and decoders that utilize blocks. In this case, the order in which the blocks of the image sample array are processed is arbitrary.

[0127] Generally, image and video coders are designed for encoding color images (either still images or images in a sequence of videos). A color image consists of multiple color planes representing sample sequences of different color components. Often, a color image is a luminance plane. and Two saturation planes and Encoded as a set of sample sequences composed of two chroma planes is color The difference component is specified. In some applications, it is encoded. Ta It is also common for the set of sample sequences to consist of three color planes representing the sample sequences of the three primary colors: red, green, and blue. Furthermore, to improve color representation, color images ga 4 It may also consist of more than one color plane. Furthermore, an image can be associated with an auxiliary sample sequence that specifies additional information about the image. For example, such an auxiliary sample sequence may be a sample sequence that specifies the transparency of the associated color sample sequence (suitable for specific display purposes), or a depth sample sequence. degree This includes sample arrays that specify the top (suitable for multi-view rendering such as 3D display).

[0128] In conventional image and video encoding standards (such as H.264), color planes are typically encoded together. In this case, specific encoding parameters such as the prediction mode, reference index, and motion vector of a macroblock and its lower macroblocks are used for all color components of a single block. The luminance plane can be thought of as a primary color plane where specific encoding parameters are specified within the bitstream, and the chroma plane can be thought of as a secondary plane where corresponding encoding parameters are inferred from the primary luminance plane. Each luminance block is associated with two chroma blocks representing the same region in the image. Depending on the chroma sampling format used, the chroma sample array may be smaller than the luminance sample array for a single block. The same subdivision is performed for each macroblock consisting of one luminance component and two chroma components, resulting in further subdivision into smaller blocks (if the macroblock is subdivided). Each block (which may be the macroblock itself or a subblock of a macroblock) consists of one block of luminance samples and two blocks of saturation samples, and the same set of prediction parameters, such as reference index and motion parameters, is used, and in some cases, an intra-prediction mode is used. Certain profiles of conventional video encoding standards (such as the H.264 4:4:4 profile) also allow for the independent encoding of different color planes within a single image. In this configuration, the macroblock segmentation, prediction mode, reference index, and motion parameters can be selected separately for the color components of a macroblock or subblock. Conventional encoding standards either encode all color planes using the same specific set of encoding parameters (such as subdivision information and prediction parameters), or encode all color planes completely independently of each other.

[0129] When a color plane is encoded collectively, all color components of a block must use a single set of subdivision and prediction parameters. This ensures that accompanying information is kept to a minimum, but it may reduce encoding efficiency compared to independent encoding. This is because using different block decomposition and prediction parameters for different color components can reduce rate / distortion loss. For example, using different motion vectors or reference frames for saturation components can significantly reduce the residual signal energy for saturation components, improving overall encoding efficiency. When a color plane is encoded independently, encoding parameters such as block subdivision, reference index, and motion parameters can be selected separately for each color component to optimize encoding efficiency for each component. However, redundancy cannot be utilized between color components. Multiple transmissions of specific encoding parameters increase the rate of accompanying information (compared to collectively encoding), and this increase in the rate of accompanying information can negatively impact overall encoding efficiency. Furthermore, support for auxiliary sample sequences in the latest video encoding standards (such as H.264) is limited to cases where the auxiliary sample sequence is encoded using a custom set of encoding parameters.

[0130] Therefore, in all embodiments described so far, the planes of the image can be treated as described above. However, as also described above, if block-level decisions are possible, for example, if it is possible to determine whether all sample sequences in a block are encoded with the same encoding parameters or whether different encoding parameters are used, the overall encoding efficiency when encoding multiple sample sequences (which may be related to different color planes and / or auxiliary sample sequences) may be improved. The basic idea of ​​interpretation below is to enable such block-level decisions. The encoder can, for example, select whether all or part of the sample sequences in a particular block are encoded using the same encoding parameters or whether different encoding parameters are used for different sample sequences, based on a rate / distortion criterion. This selection can also be achieved by communicating whether a particular encoding parameter can be estimated for a particular block of a sample sequence from an encoded block at the same location in a different sample sequence. Different sample sequences for a single image can also be grouped together, and these are called sample sequence groups or plane groups. Each plane group can contain one or more sample sequences of an image. Blocks of sample sequences within a planar group share the same selected coding parameters, such as subdivision information, prediction mode, and residual coding mode. Other coding parameters, such as transformation coefficient levels, are transmitted separately for each sample sequence within the planar group. A single planar group is coded as a primary planar group; that is, no coding parameters are inferred or predicted from other planar groups. For each block of a secondary planar group, it is possible to choose, depending on the situation, whether a new set of selected coding parameters is transmitted, or whether the selected coding parameters are inferred or predicted from a primary planar group or another secondary planar group. The decision of whether the selected coding parameters for a particular block are inferred or predicted is included in the bitstream.Interpretation offers greater flexibility in choosing the trade-off between the rate of associated information and the quality of prediction compared to modern encoding of images composed of multiple sample sequences. The advantage lies in improved encoding efficiency compared to conventional encoding of images composed of multiple sample sequences.

[0131] By adopting / predicting within a plane, the image or video coder in the above embodiment may be extended as follows: For a color sample sequence, or an auxiliary sample sequence, or a block of a set of color sample sequences and / or auxiliary sample sequences, it may be extended to allow for a situational selection of whether the set of encoding parameters to be selected is estimated or predicted from an encoded block at the same location in another sample sequence within the same image, or whether the set of encoding parameters to be selected for a block is encoded independently without referring to a block at the same location in another sample sequence within the same image. The decision of whether the set of encoding parameters to be selected is estimated or predicted for one block of sample sequences, or for multiple blocks of sample sequences, may be included in the bitstream. Different sample sequences associated with a single image do not need to be the same size.

[0132] As described above, the sample sequences associated with an image (which may represent color components and / or auxiliary sample sequences) may be arranged in two or more so-called planar groups, each consisting of one or more sample sequences. The sample sequences included in a particular planar group do not need to be the same size. Such arrangements into planar groups also include cases where each sample sequence is encoded separately.

[0133] Specifically, in the embodiment, for each block in a planar group, it is selected on a case-by-case basis whether the coding parameters specifying the prediction method for the block are estimated or predicted from coded blocks at the same location in different planar groups for the same image, or whether these coding parameters are coded separately for each block. The coding parameters specifying the prediction method for a block include one or more of the following coding parameters: a block prediction mode specifying what prediction is used for the block (intra-prediction, inter-prediction using a single motion vector and a reference image, inter-prediction using two motion vectors and a reference image, inter-prediction using a higher-order, i.e., non-translational motion model and a single reference image, inter-prediction using multiple motion models and reference images); an intra-prediction mode specifying how the intra-prediction signal is generated; an identifier specifying the number of prediction signals to be combined for generating the final prediction signal for the block; a reference index specifying the reference image used for motion-compensated prediction; motion parameters (such as displacement vectors and affine motion parameters) specifying how the prediction signal is generated using the reference image; and an identifier specifying how the reference image is filtered for generating the motion-compensated prediction signal. Generally, a block can only be associated with a subset of the above coding parameters. For example, in block prediction mode, if intra-prediction is specified for a block, the block's coding parameters may include the intra-prediction mode as an additional parameter, but coding parameters such as the reference index and motion parameters that specify how the inter-prediction signal is generated are not specified. Alternatively, in block prediction mode, if inter-prediction is specified, the associated coding parameters may include the reference index and motion parameters as additional parameters, but the intra-prediction mode is not specified.

[0134] One of two or more planar groups may be encoded or indicated in the bitstream as a primary planar group. For all blocks in this primary planar group, encoding parameters specifying how to generate prediction signals are transmitted without referring to other planar groups in the same image. The remaining planar groups are encoded as secondary planar groups. For each block in the secondary planar group, one or more syntactic elements are transmitted that convey whether encoding parameters specifying how to predict the block are estimated or predicted from a block in the same location in another planar group, or whether a new set of these encoding parameters is transmitted for the block. One of the one or more syntactic elements may be called an inter-prediction flag or inter-prediction parameter. If a signal is transmitted by the syntactic element indicating that the corresponding encoding parameters are not estimated or predicted, a new set of the corresponding encoding parameters for the block is transmitted in the bitstream. If a signal is transmitted by the syntactic element indicating that the corresponding encoding parameters are estimated or predicted, the block in the same location in the so-called reference planar group is identified. The assignment of a reference planar group to a block can be constructed in several ways. In one embodiment, a specific group of reference planes is assigned to each of the secondary plane groups. This assignment can be fixed, or it can be communicated through a high-level syntactic structure such as a set of parameters, an access unit header, an image header, or a slice header.

[0135] In the second embodiment, the assignment of a reference plane group is encoded within a bitstream and signaled by one or more syntactic elements. These syntactic elements are encoded for a block and specify whether the encoding parameters to be selected are inferred or predicted, or whether they are encoded separately.

[0136] To illustrate the above possibilities in relation to interpretation and the detailed embodiments described below, refer to Figure 11. Figure 11 illustrates image 500, which consists of three sample sequences 502, 504, and 506. In Figure 11, for clarity, only the lower portions of sample sequences 502–506 are shown. The sample sequences are shown to correspond to each other spatially. Sample sequences 502–506 overlap each other along direction 508. Furthermore, when the samples of sample sequences 502–506 are projected along direction 508, all of these samples of sample sequences 502–506 precisely coincide in spatial position with each other. In other words, planes 502 and 506 extend horizontally and vertically, adapting to each other's spatial resolution and corresponding to each other.

[0137] In this embodiment, all sample sequences of the image belong to the same portion of the spatial scene, and the vertical and horizontal resolutions may differ for each of the sample sequences 502 to 506. Furthermore, for illustrative purposes, sample sequences 502 and 504 are assumed to belong to one planar group 510, while sample sequence 506 belongs to another planar group 512. Figure 11 also illustrates the case where the spatial resolution along the horizontal axis of sample sequence 504 is twice the horizontal resolution of sample sequence 502. Furthermore, sample sequence 504 is assumed to be a first-order sequence relative to sample sequence 502. Sample sequence 502 is a lower-order sequence relative to the first-order sequence 504. As mentioned above, in this case, the subdivision of sample sequence 504 into blocks determined by the subdivision section 30 in Figure 1 is adopted by the lower-order sequence 502. In the example in Figure 11, since the vertical resolution of sample sequence 502 is half that of the first-order sequence 504, each block is equally divided into two horizontally aligned blocks. When these blocks are measured in terms of the position of the sample within sample sequence 502, the division results in blocks that are again quadratic.

[0138] As illustrated in Figure 11, the subdivision selected for sample array 506 is different from the subdivision of the other planar groups 510. As mentioned above, the subdivision unit 30 may select the subdivision of pixel array 506 separately, regardless of the subdivision of planar groups 510. Naturally, the resolution of sample array 506 may also be different from the resolution of planes 502 and 504 of planar group 510.

[0139] Here, when encoding individual sample sequences 502-506, the encoder 10 may begin by encoding the first sequence 504 of the planar group 510, for example, in the manner described above. The blocks shown in Figure 11 may be, for example, the prediction blocks described above. Alternatively, the blocks may be residual blocks, or other blocks that define the granularity for defining specific encoding parameters. Interpretation is not limited to the quadtree or multitree subdivision illustrated in Figure 11.

[0140] After transmitting syntactic elements for the primary array 504, the encoder 10 may decide to declare that the primary array 504 is the reference plane of the lower plane 502. The encoder 10 and the extraction unit 30 may each transmit this decision via the bitstream 22. Alternatively, since the sample array 504 is a primary array of the plane group 510, its relationship may be clear, and this information may be part of the bitstream 22. In any case, for each block in the sample array 502, the insertion unit 18, or another module of the encoder 10 working in conjunction with the insertion unit 18, may decide to: suppress the transmission of the encoding parameters for this block in the bitstream and instead use the encoding parameters of the block at the same location in the primary array 504 in the bitstream for that block, or send a signal indicating that the encoding parameters of the block at the same location in the primary array 504 will be used as a prediction for the encoding parameters of the current block in the sample array 502, and transmit only the residual data for the current block in the sample array 502 in the bitstream. If no decision is made, the coding parameters are transmitted normally within the data stream. For each block, the decision is communicated within the data stream 22. On the decoder side, the extraction unit 102 uses the interprediction information for each block to obtain the coding parameters for each block in the sample sequence 502 accordingly. That is, it does this by estimating the coding parameters for the block at the same location in the first-order sequence 504. Alternatively, if interplanar adoption / prediction information suggests interplanar adoption / prediction, it does this by extracting residual data for that block from the data stream and combining this residual data with the prediction obtained from the coding parameters for the block at the same location in the first-order sequence 504. Alternatively, it does this by extracting the coding parameters for the current block in the sample sequence 502 as usual, regardless of the first-order sequence 504.

[0141] As mentioned above, the location of the reference plane is not limited to the same plane group as the block currently subject to interpretation. Therefore, as stated above, plane group 510 may be the primary plane group or the reference plane group for the secondary plane group 512. In this case, the bitstream may include a syntactic element indicating whether to adopt / predict the above-mentioned encoding parameters of the macroblock at the same location in either plane 502 or 504 of the primary plane group or the reference plane group 510 for each block of the sample array 506. In the latter case, the encoding parameters of the current block of the sample array 506 are transmitted as usual.

[0142] The subdivision and / or prediction parameters for planes within a plane group may be identical. That is, they are encoded only once for the plane group (all secondary planes in a plane group estimate subdivision information and / or prediction parameters from primary planes within the same plane group), and predictions or estimations tailored to the subdivision information and / or prediction parameters are performed across plane groups. A reference plane group can be either a primary plane group or a secondary plane group.

[0143] Except for the further subdivision described above, which turns the adopted leaf blocks into quadratic blocks, the subdivision of the first-order sample array 504 is performed spatially by the lower-order sample array 502, making it easy to understand that blocks of different planes within a planar group are in the same location. In the case of adoption / prediction between planes between different planar groups, this same location may be defined in such a way that it increases the degrees of freedom between the subdivisions of these planar groups. Given a reference planar group, blocks that are in the same location within the reference planar group are identified. Blocks and reference planar groups that are in the same location can be derived by the following process: A specific sample 514 is selected within one of the current blocks 516 in the sample array 506 of the quadratic planar group 512. This may be the top-left sample of the current block 516, as illustrated in 514 in Figure 11. It may also be a sample within the current block 516 closer to the center, or any other sample within the current block, and is geometrically uniquely defined. The location of this selected sample 515 in the sample arrays 502 and 504 of the reference planar group 510 is calculated. The positions of sample 514 within sample sequences 502 and 504 are shown at 518 and 520, respectively, in Figure 11. Which of planes 502 or 504 of the reference plane group 510 is actually used may be predetermined or communicated within the bitstream. The samples in the corresponding sample sequences 502 or 504 of the reference plane group 510 that are closest to positions 518 and 520 are identified, and the blocks containing these samples are selected as the blocks in the same location within sample sequences 502 and 504, respectively. In Figure 11, these are blocks 522 and 524, respectively. Another approach to identifying blocks in the same location on other planes will be discussed later.

[0144] In one embodiment, without transmitting any additional accompanying information, coding parameters specifying the prediction for the current block 516 are fully estimated using the corresponding prediction parameters of blocks 522 / 524 located in the same place within different planar groups 510 of the same image 500. This estimation is performed by simply copying the corresponding coding parameters or by adapting the coding parameters to account for the difference between the current planar group 512 and the reference planar group 510. For example, in this adaptation, motion parameters may be added (e.g., correction of the displacement vector) to account for the phase difference between the luminance and chrominance sample arrays. Alternatively, the accuracy of the motion parameters may be modified (e.g., modification of the displacement vector accuracy) to account for the different resolutions of the luminance and chrominance sample arrays. In another embodiment, one or more estimated coding parameters specifying the generation of the prediction signal are not used directly for the current block 516 but are used as predictions for the corresponding coding parameters for the current block 516, and refinements of these coding parameters for the current block 516 are transmitted within the bitstream 22. For example, the estimated motion parameters are not used directly, but the difference in motion parameters (such as the difference in displacement vectors) that specifies the deviation between the motion parameters used in the current block 516 and the estimated motion parameters is encoded in the bitstream. On the decoder side, the actual motion parameters used are obtained by combining the difference between the estimated motion parameters and the transmitted motion parameters.

[0145] In another embodiment, the subdivision of blocks such as the prediction subdivision tree blocks described above into prediction blocks (i.e., blocks of samples where the same set of prediction parameters is used) is estimated or predicted, depending on the context, from encoded blocks at the same location in different planar groups of the same image, i.e., from the bit sequences in Figure 6A or Figure 6b. In this embodiment, one of two or more planar groups is encoded as a primary planar group. For all blocks in this primary planar group, subdivision information is transmitted without referring to other planar groups of the same image. The remaining planar groups are encoded as secondary planar groups. For blocks in the secondary planar groups, one or more syntactic elements are transmitted that indicate whether subdivision information is estimated or predicted from blocks at the same location in other planar groups, or whether subdivision information is transmitted in the bitstream. One or more syntactic elements may be called an inter-prediction flag or inter-prediction parameter. If the syntactic elements indicate that no subdivision information is estimated or predicted, the subdivision information for the block is transmitted in the bitstream without referring to other planar groups of the same image. If syntactic elements convey information indicating that subdivision information is estimated or predicted, the blocks in the same location within the so-called reference plane group are identified. The assignment of reference plane groups to blocks can be constructed in several ways. In one embodiment, a particular reference plane group is assigned to each of the secondary plane groups. This assignment can be fixed or conveyed through a high-level syntactic structure such as a set of parameters, an access unit header, an image header, or a slice header. In a second embodiment, the assignment of reference plane groups is encoded within the bitstream, with information conveyed by one or more syntactic elements. These syntactic elements are encoded for the block and specify whether subdivision information is estimated or predicted, or whether it is encoded separately. The reference plane group may be a primary plane group or another secondary plane group.Given a reference plane group, blocks at the same location within that group are identified. These same-location blocks represent blocks within the reference plane group that correspond to the same image region as the current block, or blocks within the reference plane group that share the largest portion of the image region with the current block. These same-location blocks can be subdivided into smaller prediction blocks.

[0146] In other embodiments, subdivision information for the current block, such as subdivision information based on a quadtree in Figure 6A or Figure 6b, is entirely estimated using subdivision information for blocks at the same location in different planar groups of the same image, without transmitting any additional accompanying information. As one particular example, if a block at the same location is subdivided into two or four predicted blocks, the current block is also subdivided into two or four subblocks for prediction purposes. As another particular example, if a block at the same location is subdivided into four subblocks, and one of these subblocks is further subdivided into four smaller subblocks, the current block is also subdivided into four subblocks, and one of these subblocks (corresponding to a subblock of the block at the same location that is further subdivided) is also subdivided into four smaller subblocks. In other preferred embodiments, the estimated subdivision information is not used directly for the current block but is used as a prediction of the actual subdivision information for the current block, and the corresponding refinement information is transmitted in the bitstream. For example, the subdivision information estimated from a block at the same location may be further refined. For each subblock corresponding to a subblock within the same block that is not subdivided into smaller blocks, a syntactic element can be encoded in the bitstream. This specifies whether the subblock is further subdivided in the current planar group. The transmission of such syntactic elements can be conditional on the size of the subblock. Alternatively, a signal can be sent in the bitstream indicating that a subblock that is further subdivided in the reference planar group is not subdivided into smaller blocks in the current planar group.

[0147] In other embodiments, both the subdivision of a block into prediction blocks and the coding parameters specifying how to predict those subblocks are, in context, estimated or predicted from coded blocks at the same location in different planar groups for the same image. In a preferred embodiment of the present invention, one of two or more planar groups is coded as a primary planar group. For all blocks in this primary planar group, subdivision information and prediction parameters are transmitted without referring to other planar groups of the same image. The remaining planar groups are coded as secondary planar groups. For blocks in the secondary planar groups, one or more syntactic elements are transmitted that inform whether the subdivision information and prediction parameters are estimated or predicted from blocks at the same location in other planar groups, or whether the subdivision information and prediction parameters are transmitted in the bitstream. One of the one or more syntactic elements may be called an inter-prediction flag or inter-prediction parameter. If the syntactic element informs that no estimation or prediction of subdivision information and prediction parameters is performed, the subdivision information of the block and the prediction parameters for the resulting subblocks are transmitted in the bitstream without referring to other planar groups of the same image. If the syntactic elements communicate that subdivision information and predictive parameters for subblocks are estimated or predicted, the blocks in the same location within the so-called reference plane group are identified. The assignment of reference plane groups to blocks can be configured in several ways. In one embodiment, a specific reference plane group is assigned to each of the secondary plane groups. This assignment can be fixed or communicated through a high-level syntactic structure such as a parameter set, access unit header, image header, or slice header. In a second embodiment, the assignment of reference plane groups is encoded within the bitstream and communicated by one or more syntactic elements. These syntactic elements are encoded for the block and specify whether subdivision information and predictive parameters are estimated or predicted, or whether they are encoded separately.The reference plane group may be a primary plane group or another secondary plane group. Given a reference plane group, blocks at the same location within the reference plane group are identified. Blocks at the same location may be blocks in the reference plane group corresponding to the same image region as the current block, or blocks representing blocks in the reference plane group that share the largest portion of the image region with the current block. Blocks at the same location can be subdivided into smaller prediction blocks. In a preferred embodiment, the subdivision information for the current block and the prediction parameters for the resulting subblocks are entirely estimated using the subdivision information for blocks at the same location in different plane groups of the same image and the prediction parameters for the corresponding subblocks, without transmitting any additional accompanying information. As one particular example, if blocks at the same location are subdivided into two or four prediction blocks, the current block is also subdivided into two or four subblocks for prediction purposes, and the prediction parameters for the subblocks of the current block are derived as described above. As another example, if a block in the same location is divided into four subblocks, and one of these subblocks is further divided into four smaller subblocks, then the current block is also divided into four subblocks, and one of these subblocks (corresponding to a subblock of the block in the same location that is further divided) is also divided into four smaller subblocks, and the predictive parameters for all subblocks that are not further divided are estimated as described above. In another preferred embodiment, the subdivision information is fully estimated based on the subdivision information of the block in the same location within the reference plane group, but the predicted parameters estimated for the subblocks are used only as predictions for the actual predicted parameters of the subblocks. The deviation between the actual predicted parameters and the estimated predicted parameters is encoded in the bitstream. In another embodiment, the estimated subdivision information is used as a prediction for the actual subdivision information of the current block, and the difference is transmitted in the bitstream (as described above), but the predicted parameters are fully estimated.In another embodiment, both the estimated subdivision information and the estimated prediction parameters are used as predictions, and the difference between the actual subdivision information and prediction parameters and the estimated values ​​is transmitted within the bitstream.

[0148] In another embodiment, for a block of planar groups, the mode of residual coding (such as the transformation type) is selected on a case-by-case basis whether it is estimated or predicted from an encoded block at the same location in a different planar group of the same image, or whether the mode of residual coding is coded separately for each block. This embodiment is similar to the above embodiment in which the estimation / prediction of prediction parameters is performed on a case-by-case basis.

[0149] In another embodiment, the subdivision of blocks (such as prediction blocks) into transformation blocks (i.e., blocks of samples to which a two-dimensional transformation is applied) is contextually estimated or predicted from encoded blocks located at the same location in different planar groups of the same image. This embodiment is similar to the above embodiment in which the estimation / prediction of the subdivision into prediction blocks is contextually performed.

[0150] In another embodiment, the subdivision of blocks into transform blocks, and the mode of encoding the residuals for the resulting transform blocks (such as the transform type), are inferred or predicted from encoded blocks located at the same location in different planar groups of the same image. This embodiment is similar to the above embodiment in which the subdivision into predict blocks and the estimation / prediction of predict parameters for the resulting predict blocks are performed in a contextual manner.

[0151] In another embodiment, the subdivision of blocks into prediction blocks, associated prediction parameters, subdivision information of prediction blocks, and the mode of encoding residuals for transform blocks are contextually estimated or predicted from encoded blocks located at the same location in different planar groups of the same image. This embodiment is a combination of the embodiments described above. It is also possible to estimate or predict only some of the encoding parameters described above.

[0152] Therefore, as mentioned above, inter-plane adoption / prediction can potentially improve coding efficiency. However, even when multi-tree based subdivision and other types of block subdivision are used, improvement in coding efficiency gain through inter-plane adoption / prediction is possible, regardless of whether or not block merging occurs.

[0153] The embodiments outlined above regarding plane-to-plane fitting / prediction can be applied to image and video encoders and decoders. The encoder and decoder divide the color plane of the image, and any auxiliary sample arrays associated with the image, into blocks, and associate these blocks with coding parameters. For each block, a set of coding parameters may be contained within the bitstream. For example, these coding parameters may describe how the block is predicted or decoded on the decoder side. For example, coding parameters may represent macroblock or block prediction modes, subdivision information, intra-prediction modes, reference indices used for motion compensation prediction, motion parameters such as displacement vectors, residual coding modes, transformation coefficients, etc. The sample arrays associated with the image may each have different sizes.

[0154] Next, we will describe an extended information transfer system for coding parameters in a tree-based compartmentalization system, as described above, with reference to Figures 1 through 8, for example. The system described below can be combined with any one or a combination of the embodiments described above, but the effects and benefits of the extended information transfer system, as well as other systems, namely, combination and interplanar adoption / prediction, will be described independently of the embodiments described above. This system will be referred to as inheritance below.

[0155] An improved encoding scheme for the encoding of associated information in a tree-based partitioning system is called inheritance, and is described below. This generally provides the following advantages compared to conventional systems for handling encoding parameters.

[0156] Typically, in conventional image and video encoding, an image, or a specific set of sample sequences within an image, is broken down into blocks and associated with specific encoding parameters. An image typically consists of multiple sample sequences. Furthermore, an image may be associated with additional auxiliary sample sequences specifying, for example, transparency information or depth maps. The sample sequences (including auxiliary sample sequences) of an image can be grouped into one or more so-called planar groups, each consisting of one or more sample sequences. Planar groups of an image can be encoded independently, or, if the image is associated with multiple planar groups, they can be encoded by predictions from other planar groups of the same image. Typically, each planar group is broken down into blocks. Blocks (or corresponding blocks of sample sequences) are predicted either by predictions between images or within images. Blocks can be of various sizes, either square or rectangular. The division of an image into blocks can be fixed by syntax, or (at least partially) transmitted within the bitstream. Often, syntactic elements are transmitted that convey subdivision information for blocks of a given size. These syntactic elements allow a block to be subdivided into smaller blocks, specifying whether or not they are associated with coding parameters for prediction purposes, and how. For all samples in a block (or the corresponding block in a sample sequence), the decoding of the associated coding parameters is specified in a particular way. In the example, predictions are made for all samples in a block using the same set of prediction parameters, such as a reference index (identifying the reference image within a set of coded images), motion parameters (specifying the amount of block movement between the reference image and the current image), and parameters for specifying interpolation filters and intra-prediction modes. Motion parameters can be represented by displacement vectors with horizontal and vertical components, or higher-order motion parameters such as affine motion parameters consisting of six components. It is also possible to associate multiple sets of specific prediction parameters (such as reference indexes and motion parameters) with a single block.In this case, for each set of these specific prediction parameters, a single intermediate prediction signal is generated for the block (or the corresponding block in the sample sequence), and the final prediction signal is formed by combinations including superpositions of these intermediate prediction signals. The corresponding weighting parameters can be fixed for either the image, the reference image, or a set of reference images, or they can be included in the set of prediction parameters for the corresponding block, sometimes with a constant offset (added to the weighted sum). The difference between the original block (or the corresponding block in the sample sequence) and the prediction signal is also called the residual signal, and is usually transformed and quantized. Often, a two-dimensional transformation is applied to the residual signal (or the corresponding sample sequence of the residual block). For transform coding, the block (or the corresponding block in the sample sequence) for which a specific set of prediction parameters is used can be further subdivided before the transformation is applied. The transformed block is the same size as or smaller than the block used for prediction. Also, a transformed block can contain multiple blocks used for prediction. Each transformed block can be of various sizes, and can be a square or rectangular block. After the transformation, the resulting transformation coefficients are quantized to obtain so-called transformation coefficient levels. The conversion coefficient levels, prediction parameters, and subdivision information, if present, are entropy-encoded.

[0157] In some image and video encoding standards, the possibilities for subdividing an image (or planar group) into blocks, as provided by the syntax, are very limited. Typically, all that can be specified is whether a block of a given size can be subdivided into smaller blocks, and (if applicable, how). For example, in H.264, the maximum block size is 16x16. A 16x16 block is also called a macroblock, and each image is divided into macroblocks in the first step. For each 16x16 macroblock, it can be communicated whether it will be encoded as a 16x16 block, two 16x8 blocks, two 8x16 blocks, or four 8x8 blocks. If a 16x16 block is subdivided into four 8x8 blocks, each of these 8x8 blocks can be encoded as one 8x8 block, two 8x4 blocks, two 4x8 blocks, or four 4x4 blocks. The latest standards for image and video encoding have the advantage of reducing the rate of accompanying information used to transmit subdivision information by minimizing the possibility of specifying block divisions. However, as explained below, there is a disadvantage that the bitrate required to transmit block prediction parameters can be high. Typically, the rate of accompanying prediction information used to transmit prediction information accounts for a significant portion of the block's total bitrate. Therefore, reducing this accompanying information can potentially improve encoding efficiency, which can be achieved, for example, by using a larger block size. Images in actual image and video sequences are composed of objects of arbitrary shapes with specific properties. For example, such objects or parts of objects are characterized by unique textures or unique motions. Typically, the same set of prediction parameters is applied to such objects or parts of objects. However, typically, the boundaries of objects do not coincide with the boundaries of large prediction blocks (e.g., 16x16 macroblocks in H.264). Typically, the encoder makes subdivision decisions (from a limited set of possibilities) to minimize a certain rate / distortion loss.As a result, for objects of arbitrary shape, a large number of small blocks may be generated. Since each of these small blocks is associated with a set of predictive parameters that need to be transmitted, the rate of associated information can account for a significant portion of the total bitrate. However, since some of these small blocks still represent the same object or a region of an object, the predictive parameters of the resulting large number of blocks will be identical or very similar. It is believed that encoding efficiency will improve if the syntax is extended in a way that not only allows for the subdivision of blocks but also allows for the sharing of encoded parameters among the resulting blocks. In tree-based subdivision, the sharing of encoded parameters for a given set of blocks can be achieved by assigning the encoded parameters or a portion of them to one or more parent nodes in the tree-based hierarchy. As a result, the associated information required to transmit information about the actual selection of encoded parameters for the resulting blocks can be reduced using the shared parameters or a portion of them. This reduction can be achieved by omitting the transmission of parameter information for subsequent blocks or by using the shared parameters for parameter prediction and / or contextual modeling for subsequent blocks.

[0158] The basic idea behind the inheritance scheme described below is to reduce the bitrate required to transmit encoding parameters by sharing information according to a tree-based block hierarchy. The shared information signals are transmitted within the bitstream (added to the subdivision information). The advantage of the inheritance scheme is improved encoding efficiency due to the reduced rate of information associated with the encoding parameters.

[0159] To reduce the rate of accompanying information, in the embodiments described below, the encoding parameters for each specific set of samples in a multi-tree subdivision, i.e., a simply connected region, are transmitted in an efficient manner within the data stream. A specific set of samples, i.e., a simply connected region, can represent a rectangular or square block, a region of any shape, or any other set of samples. The inheritance scheme described below eliminates the need to explicitly include the encoding parameters for each of these sets of samples entirely within the bitstream. The encoding parameters may represent prediction parameters that specify how to predict the corresponding set of samples using the encoded samples. The numerous possibilities and examples described above also apply here. As shown above, and further described below, with respect to the inheritance schemes below, the tree-based subdivision of the image sample array into sets of samples may be fixed by syntax or transmitted by the corresponding subdivision information within the bitstream. As described above, the encoding parameters for sets of samples may be transmitted in a predetermined order specified by syntax.

[0160] In accordance with the inheritance scheme, the decoder or the decoder's extractor 102 is configured to derive information about the coding parameters of individual simply connected regions or sets of samples in a specific manner. In particular, coding parameters or parts thereof, such as parameters used for prediction purposes, are shared between blocks following a predetermined tree-based partitioning scheme, along with shared groups following a tree structure determined by the encoder or insertor 18. In certain embodiments, the sharing of coding parameters for all child nodes of a given internal node in the partitioning tree is indicated using a specific binary sharing flag. Alternatively, the refinement of coding parameters can be transmitted for each node so that the refinement of parameters accumulated according to the tree-based block hierarchy can be applied to all sets of samples in a given leaf node block. In another embodiment, parts of the coding parameters transmitted for internal nodes according to the tree-based block hierarchy can be used for context-adapted entropy coding and entropy decoding of coding parameters or parts thereof for a given leaf node block.

[0161] Figures 12A and 12B illustrate the basic concept of inheritance in a specific case using a quadtree-based partitioning. However, as shown several times above, other multi-tree subdivisions may also be used. Figure 12A shows the tree structure, and Figure 12B shows the spatial partitioning corresponding to the tree structure in Figure 12A. The partitioning shown here is similar to that shown in Figures 3A to 3C. In general, the inheritance scheme makes it possible to assign associated information to nodes in different non-leaf layers within the tree structure. Depending on the assignment of associated information to nodes in different layers within the tree, such as internal nodes and the root node in the tree in Figure 12A, sharing of associated information can be achieved to varying degrees within the block tree hierarchy shown in Figure 12B. For example, in the case of Figure 12A, if it is determined that all leaf nodes in the fourth layer, all having the same parent node, share the associated information, then in effect, the smallest blocks shown as 156a to 156d in Figure 12B share this associated information, eliminating the need to transmit the associated information four times for each of these small blocks 156a to 156d. However, this remains an encoder option. It is also possible to determine the entire area of ​​hierarchy level 1 (second layer) in Figure 12A, i.e., the upper right quarter of tree block 150, including subblocks 154a, 154b, and 154d, and the even smaller subblocks 156a to 156d, as the area where the encoding parameters are shared. Thus, the area where the associated information is shared increases. The next level of increase is to combine all the subblocks of the first layer, i.e., subblocks 152a, 152c, and 152d, with the smaller blocks mentioned above. In other words, in this case, the entire tree block has the associated information that is assigned to it, and all the subblocks of this tree block 150 share that associated information.

[0162] In the following explanation of inheritance, the following notation will be used in the description of embodiments. a. Reconstructed specimen of the current leaf node: r b. Reconstructed specimen of adjacent leaf: r' c. Predictors of the current leaf node: p d. Residuals of the current leaf node: Res e. Reconstructed residuals of the current leaf node: RecRes f. Scaling and inverse transform: SIT g. Shared flag: f

[0163] As the first example of inheritance, we will describe the signaling of intra-prediction at internal nodes. Specifically, we will describe the method of signaling the intra-prediction mode at internal nodes for tree-based block partitioning for prediction purposes. Internal nodes (including the root node) can transmit some of the accompanying information used by their corresponding child nodes by traversing the tree from the root node to the leaf nodes. Specifically, for internal nodes, a shared flag f is transmitted in the following sense:

[0164] If the value of f is 1 ("true"), all child nodes of a given internal node share the same intra-prediction mode. In addition to the shared flag f with a value of 1, the internal node transmits information about the parameters of the intra-prediction mode used by all child nodes. Therefore, all subsequent child nodes do not transmit any information about the prediction mode or the shared flag. When rebuilding all related leaf nodes, the decoder applies the intra-prediction mode from the corresponding internal node.

[0165] If the value of f is 0 ("false"), the child nodes of the corresponding internal node do not share the same intra-prediction mode, and each child node that is an internal node transmits a separate sharing flag.

[0166] Figure 12C illustrates the signaling of intra-prediction at the internal nodes described above. The internal nodes of the first layer transmit the shared flag and the accompanying information provided by the intra-prediction mode information, while the child nodes do not transmit any accompanying information.

[0167] As a second example of inheritance, we will describe the refinement of interprediction. Specifically, we will describe how information associated with the interprediction mode is transmitted in the internal mode of tree-based block partitioning, aimed at refining motion parameters provided by motion vectors, etc. Internal nodes (including the root node) can transmit some of the information refined in their corresponding child nodes by traversing the tree from the root node to the leaf nodes. Specifically, for internal nodes, the shared flag f is transmitted in the following sense:

[0168] If the value of f is 1 ("true"), all child nodes of a given internal node share the same motion vector reference. In addition to the shared flag f with a value of 1, the internal node also transmits the motion vector and the reference index. Therefore, all subsequent child nodes do not transmit any further shared flags, but may transmit a refinement of this inherited motion vector reference. When rebuilding all related leaf nodes, the decoder adds the refinement of the motion vector of a given leaf node to the inherited motion vector reference belonging to the corresponding internal parent node that has the shared flag f with a value of 1. This means that the refinement of the motion vector of a given leaf node is the difference between the actual motion vector applied for motion compensation prediction at this leaf node and the motion vector reference of the corresponding internal parent node.

[0169] If the value of f is 0 ("false"), the child nodes of the corresponding internal node do not necessarily share the same interprediction mode, and motion parameter refinement is not performed in the child nodes by using motion parameters from the corresponding internal node. Each child node that is an internal node transmits a separate shared flag.

[0170] Figure 12D illustrates the refinement of motion parameters described above. The internal nodes of the first layer transmit shared flags and associated information. The leaf nodes, which are child nodes, transmit only the refinement of motion parameters and no associated information, such as that transmitted by the internal child nodes of the second layer.

[0171] Figure 13 is described below. Figure 13 is a flowchart illustrating the mode of operation of a decoder, such as the decoder in Figure 2, in the reconstruction of an array of information samples representing spatial example information signals from a data stream. The array of information samples is subdivided into leaf regions of different sizes by multi-tree subdivision. As described above, each leaf region is associated with one of the hierarchy levels of the multi-tree subdivision. For example, all blocks shown in Figure 12B are leaf regions. For example, leaf region 156c is associated with the fourth layer (or level 3) of the hierarchy. Each leaf region is associated with coding parameters. Examples of these coding parameters are described above. For each leaf region, the coding parameters are represented by a set of syntactic elements. The type of each syntactic element is one of a set of syntactic element types. For example, such syntactic element types include prediction mode, motion vector components, and intra-prediction mode indications. The decoder performs the following steps according to Figure 13.

[0172] In step 550, inheritance information is extracted from the data stream. In the case of Figure 2, the extraction unit 102 performs step 550. The inheritance information indicates whether or not inheritance is used in the current sequence of the information sample. The following description reveals that there are several possibilities for inheritance information, such as the shared flag f and information transfer in a multi-tree structure that is divided into primary and secondary parts.

[0173] The information sample array may already be part of an image, such as a tree block, for example, tree block 150 in Figure 12B. Therefore, the inheritance information indicates whether or not inheritance is used for a particular tree block 150. Such inheritance information may be inserted into the data stream for all tree blocks of the prediction subdivision, for example.

[0174] Furthermore, if it is indicated that inheritance is to be used, the inheritance information indicates at least one inheritance region of the sequence of information samples. This inheritance region consists of a set of leaf regions and corresponds to one of the set of hierarchical levels of the multitree subdivision that lies below each hierarchical level to which the set of leaf regions is associated. In other words, the inheritance information indicates whether inheritance is used for the current sequence of samples, such as tree block 150. If inheritance is used, at least one inheritance region or subregion of this tree block 150 is indicated, in which the leaf regions share coding parameters. Thus, the inheritance region does not have to be a leaf region. In the example in Figure 12B, this inheritance region may be, for example, the region formed by subblocks 156a to 156b. Alternatively, the inheritance region may be larger and may also include subblocks 154a, 154b, and 154d. Furthermore, the inheritance region may be tree block 150 itself, and all leaf blocks may share the coding parameters associated with that inheritance region.

[0175] It should also be noted that multiple inheritance regions may be defined within each of a single sample sequence or tree block 150. For example, suppose the lower left subblock 152c is also divided into smaller blocks. In this case, subblock 152c can also form an inheritance region.

[0176] In step 552, the inheritance information is checked to determine whether inheritance is used. If inheritance is used, the process in Figure 13 proceeds to step 554, where a subset of inheritances containing at least one syntactic element of a given syntactic element type is extracted from the data stream for each interplanar inheritance region. In the subsequent step 556, this subset of inheritances is copied to the subset of inheritances of the corresponding syntactic elements in the set of syntactic elements representing the coding parameters associated with the set of leaf regions that constitute each at least one inheritance region, or is used as its prediction. In other words, for each inheritance region indicated in the inheritance information, the data stream constitutes a subset of inheritances of syntactic elements. In other words, inheritance is associated with at least one specific type or category of syntactic elements that can be used for inheritance. For example, syntactic elements of prediction mode, inter-prediction mode, or intra-prediction mode may be subject to inheritance. For example, the subset of inheritances contained in the data stream for an inheritance region may contain syntactic elements of inter-prediction mode. Furthermore, a subset of inheritance may include syntactic elements whose type depends on the value of the type of the fixed syntactic element associated with the inheritance scheme. For example, if the interpretation mode is a fixed component of the subset of inheritance, syntactic elements that define motion compensation, such as components of motion vectors, may or may not be included in the subset of inheritance depending on the syntax. For example, if the upper right quarter of tree block 150, i.e., subblock 152b, is the inheritance region, then for this inheritance region, we may have only the interpretation mode, or the interpretation mode and motion vectors and the index of the motion vectors.

[0177] All syntactic elements contained in a subset of inheritance are copied to the corresponding coding parameters of the leaf blocks in that inheritance domain, i.e., leaf blocks 154a, 154b, 154d, and 156a through 156d, or used as predictions. If predictions are used, residuals are transmitted for each individual leaf block.

[0178] One possibility for transmitting inheritance information about tree block 150 is the transmission of the shared flag f described above. In this case, the extraction of inheritance information in step 550 includes the following: In particular, the decoder is configured to extract the shared flag f from the data stream for non-leaf regions corresponding to any of the sets of inheritances at at least one hierarchical level of the subdivision of the multitree, using the hierarchical level order from lower to higher hierarchical levels, and to check whether inheritance is defined by each inheritance flag or shared flag. For example, the sets of inheritances at hierarchical levels are formed by the hierarchy of layers 1 through 3 in Figure 12A. Thus, any node in the subtree structure located in any of layers 1 through 3, rather than a leaf node, has the associated shared flag in the data stream. The decoder extracts these shared flags in the order of layers 1 through 3, in a traverse order such as depth order or width order. If one of the shared flags is 1, the decoder recognizes that the leaf blocks contained in the region of the corresponding inheritance share the subset of inheritance that will subsequently be extracted in step 554. It is no longer necessary to check the inheritance flag for the child nodes of the current node. In other words, the inheritance flag for these child nodes is not transmitted in the data stream. This is because it is clear that the domain of these nodes already belongs to the domain of inheritance, which shares a subset of the inheritance of syntactic elements.

[0179] The shared flag f can be treated alternately with the bits described above that transmit information about the quadtree subdivision. For example, a bit sequence that includes both the subdivision flag and the shared flag and treats them alternately is shown as follows:

number

[0180] Another way to define inheritance information that indicates the domain of inheritance is to use two subdivisions defined in a mutually subordinate manner, as described above for predictive and residual subdivisions. In general, leaf blocks of a first-order subdivision can form a domain of inheritance that defines a region in which a subset of inherited syntactic elements is shared. At the same time, blocks within these domains of inheritance, in which a subset of inherited syntactic elements is copied or used as a predictive, are defined by lower-order subdivisions.

[0181] For example, consider the residual tree as an extension of the prediction tree. Furthermore, consider the case where the prediction blocks can be further subdivided into smaller blocks for the purpose of encoding the residuals. For each prediction block corresponding to a leaf node in the quadtree associated with the prediction, one or more lower-level quadtrees determine the corresponding subdivision for encoding the residuals.

[0182] In this case, instead of using signaling of predictions in the internal nodes, the residual tree is interpreted as follows: Refinement of the prediction tree is also specified (information is conveyed by the corresponding leaf nodes of the tree related to the prediction), in the sense of using a constant prediction mode, and the refined reference sample is also used. The following example illustrates this case.

[0183] For example, Figures 14A and 14B show a quadtree partitioning for intra-prediction. For a specific leaf node in the first-order subdivision, the adjacent reference sample is highlighted. Figure 14B shows a quadtree subdivision of residuals for the same predictive leaf node, along with the refined reference sample. All subblocks shown in Figure 14B share the same intra-prediction parameters contained within the data stream for each leaf block highlighted in Figure 14A. Thus, Figure 14A shows an example of a conventional quadtree partitioning for intra-prediction, with a reference sample for a specific leaf node. However, in a preferred embodiment, a separate intra-prediction signal is computed for each leaf node in the residual tree by using adjacent samples of the reconstructed leaf nodes in the residual tree, for example, as shaded with gray stripes in Figure 14B. The reconstructed signal for a given residual leaf node is then obtained in the usual manner by adding the quantized residual signal to this predictive signal. This reconstructed signal is used as a reference signal for subsequent prediction processing. The decoding order for prediction is the same as the decoding order for residuals.

[0184] As shown in Figure 15, in the decoding process, for each leaf node of the residual, the predicted signal p is calculated using the reference sample r' (as shown in the leaf node of the quadtree associated with the prediction) according to the actual intra-prediction mode.

[0185] SIT processing,

number

number

[0186] The decoding order for prediction is the same as the decoding order for residuals shown in Figure 16.

[0187] Each leaf node of the residual is decoded as described in the previous paragraph. The reconstructed signal r is stored in a buffer, as shown in Figure 16. For the next prediction and decoding process, the reference sample r' is taken from this buffer.

[0188] Referring to Figures 1 to 16, specific embodiments have been described by combining the various embodiments outlined above. Furthermore, other embodiments of the present application will be described, which place emphasis on the specific embodiments described above and are embodiments that combine some of the embodiments described above.

[0189] In particular, the embodiments described above, based on the configurations of Figures 1 and 2, are primarily combinations of numerous aspects of the present application and offer advantages when used in other applications or other coding fields. As repeatedly stated above, for example, multi-tree subdivision may be used without concatenation and / or inter-plane adoption / prediction and / or inheritance. For example, aspects such as transmission of the maximum block size, use of depth-ordered traversal order, adaptation of context according to the hierarchical level of each subdivision flag, and transmission of the maximum hierarchical level within the bitstream, all offer advantages independently of each other, with the aim of reducing the bitrate of associated information. This is also true when considering concatenation schemes. Benefits are obtained from concatenation regardless of the detailed method by which the image is subdivided into simply connected regions, and also regardless of the presence of multiple sample sequences or the use of inter-plane adoption / prediction and / or inheritance. The same applies to the benefits obtained from inter-plane adoption / prediction and inheritance.

[0190] Therefore, the embodiments outlined below combine the above embodiments with respect to aspects related to coupling. Since the following embodiments combine the above embodiments, many of the details described above can be considered to be combined with the embodiments described below.

[0191] Figure 17 shows a decoder according to an embodiment of the present invention. The decoder in Figure 17 comprises an extraction unit 600 and a reconstruction unit 602. The extraction unit 600 is configured to extract payload data from a data stream 604 for each of a plurality of simply connected regions formed by subdividing an array of information samples representing spatially sampled information signals. As described above, the simply connected regions formed by subdividing the array of information samples are produced by the subdivision of a multitree and take the shape of a square or rectangle. Furthermore, the embodiments specifically described for the subdivision of the sample array are merely specific embodiments, and other subdivisions may be used. Some possibilities are shown in Figures 18A to 18C. For example, Figure 18A shows the subdivision of a sample array 606. It is subdivided into a regular two-dimensional array of non-overlapping, adjacent tree blocks 608, some of which are subdivided into subblocks 610 of various sizes according to a multitree structure. As described above, Figure 18A shows the subdivision of a quadtree, but it is also possible to subdivide the parent node into a number of other child nodes. In the embodiment shown in Figure 18B, the sample array 606 is subdivided into subblocks of various sizes by directly applying multi-tree subdivision to the entire pixel array 606. In other words, the entire pixel array 606 is treated as a tree block. Figure 18C shows another embodiment. In this embodiment, the sample array has a structure of a regular two-dimensional arrangement of adjacent square or rectangular macroblocks. Each of these macroblocks 612 is individually associated with partitioning information, and according to this partitioning information, the macroblock 612 is either left unpartitioned or partitioned into a regular two-dimensional arrangement of blocks of the sizes indicated by the partitioning information. As can be seen from the above, in all the subdivisions from Figures 13A to 13C, the sample array 606 is subdivided into non-overlapping single-connected regions, for example, in the embodiments of Figures 18A to 18C. However, several other methods are also possible. For example, the blocks may overlap with each other, but only if each block has a portion that does not overlap with any of its adjacent blocks.Alternatively, the state is limited to a condition where at most one adjacent block, positioned alongside the current block in a given direction, overlaps each sample of the block. The latter condition means that adjacent blocks to the left and right completely cover the current block, but do not overlap each other, and the same applies to adjacent blocks in the vertical and diagonal directions.

[0192] As described above with reference to Figures 1 to 16, the sequence of information samples does not necessarily represent images from a video or still images. Sampling sequence 606 may represent a depth map or transparency map of a particular scene. As already explained above, the payload data associated with each of the multiple single-connected regions may include residual data in the spatial or transformation domain, such as transformation coefficients, and significance maps that identify the locations of significant transformation coefficients within the transformation blocks corresponding to the residual blocks. In general, the payload data extracted by the extraction unit 600 for each single-connected region from the data stream 604 is data that spatially describes the associated single-connected region, either directly in the spatial or spectral domain, or as residuals to some prediction.

[0193] The reconstruction unit 602 is configured to reconstruct the sequence of information samples from the payload data for each single-connected region of the sequence of information samples by processing the payload data for each single-connected region in a manner defined by the coding parameters associated with each single-connected region. As described above, the coding parameters may also be prediction parameters, so the single-connected regions shown in Figures 18A to 18B may correspond to the prediction blocks described above, i.e., blocks in which the data stream 604 specifies the details of the prediction for the prediction of individual single-connected regions. However, the coding parameters are not limited to prediction parameters. The coding parameters can indicate the transformations used to transform the payload data, or they can define filters used to reconstruct individual single-connected regions when reconstructing the sequence of information samples.

[0194] The extraction unit 600 is configured to identify single-connected regions that have a predetermined relative positional relationship with a given single-connected region from among multiple single-connected regions. Details of this step are described above in relation to step 450. In other words, the identification may be performed according to a subset of the coding parameters associated with the given single-connected region, in addition to the predetermined relative positional relationship. After the identification, the extraction unit 600 extracts a merge index for the given single-connected region from the data stream 604. If the number of single-connected regions that have a predetermined relative positional relationship with the given single-connected region is greater than zero, this corresponds to the description in steps 452 and 454 above. If the merge index suggests a process of combining a predetermined block, the extraction unit 600 is configured to check whether the number of single-connected regions that have a predetermined relative positional relationship with the given single-connected region is 1, or whether the number of single-connected regions that have a predetermined relative positional relationship with the given single-connected region is greater than 1, but their coding parameters match each other. If either of these two conditions is met, the extraction unit either adopts the coding parameters as described above in steps 458 to 468, or uses them to predict the coding parameters of a given single-connected region or the remaining subset thereof. If, as described above with reference to Figure 10, the latter confirmation reveals that the number of single-connected regions in a predetermined relative position to a given single-connected region is greater than one and that their coding parameters are different from each other, the extraction unit 600 may further extract indices from the data stream 604.

[0195] By performing the latter verification, the transmission of further indicators that represent one or a subset of the candidate simply connected region can be suppressed, thereby reducing the overhead of associated information.

[0196] Figure 19 shows a general structure of an encoder that generates a data stream that can be decoded by the decoder in Figure 17. The encoder in Figure 19 comprises a data generation unit 650 and an insertion unit 652. The data generation unit 650 is configured to encode the sequence of information samples into payload data, in association with encoding parameters associated with each simply connected region to indicate how the payload data for each simply connected region is reconstructed, for each of the multiple interconnected regions formed by subdividing the sequence of information samples. The insertion unit 652 performs identification and verification in the same way as the extraction unit 600 of the decoder in Figure 12, but performs insertion instead of extraction of merge indicators, thereby suppressing the insertion of encoding parameters into the data stream. Alternatively, instead of inserting all encoding parameters into the data stream, it inserts the residuals of each prediction instead of the adoption / prediction described above, referring to Figures 12 and 10, respectively.

[0197] Furthermore, it should be noted that the encoder structure in Figure 19 is schematic, and in practice, the determination of payload data, coding parameters, and merge index may be an iterative process. For example, considering that when the coding parameters of adjacent simply connected regions are similar but do not match, the merge index can completely suppress one coding parameter of the simply connected region, and instead of presenting all of these coding parameters, it may be determined through iterative processing that it is preferable to tolerate slight differences between these coding parameters rather than conveying such differences to the decoder.

[0198] Figure 20 shows another embodiment of the decoder. The decoder in Figure 20 comprises a subdivision unit 700, a coupling unit 702, and a reconstruction unit 704. The subdivision unit is configured to spatially subdivide an array of samples representing the spatial sampling of a two-dimensional information signal into multiple non-overlapping simply connected regions of different sizes by recursive subdivision, depending on a subset of syntactic elements contained in the data stream. Thus, this subdivision may correspond to the embodiments outlined above with reference to Figures 1 to 16 or Figure 18A or Figure 18B. The syntactic elements contained in the data stream for indicating subdivision may be defined as shown above with reference to Figures 6A and 6b, or may be defined in other ways.

[0199] The joiner 702 is configured to obtain an intermediate subdivision of the sample sequence into disparate sets of simply connected regions, where the union is a set of simply connected regions, by combining spatially adjacent simply connected regions, depending on a second subset of syntactic elements in the data stream separated from the first subset. In other words, the joiner 702 combines the simply connected regions and assigns them to the join group of simply connected regions in a unique manner. The second subset of syntactic elements indicating the join information may be defined in the manner described above with reference to Figures 19 and 10, or in other ways. That is, the encoder's ability to indicate subdivision by using a subset separated from the subset indicating the join increases the encoder's degree of freedom in adapting the sample sequence subdivision to the actual contents of the sample sequence, thereby improving encoding efficiency. The reconstruction 704 is configured to reconstruct the sample sequence from the data stream using the intermediate subdivision. As shown above, the reconstruction 704 may utilize the intermediate subdivision by adopting / predicting the coding parameters of the joining partner for the current simply connected region. Alternatively, the reconstruction unit 704 may apply the transformation or prediction process to a combined region of a group of connected single-connected regions.

[0200] Figure 21 shows a possible encoder that generates a data stream that can be decoded by the decoder in Figure 15. This encoder comprises a subdivision / combination stage 750 and a data stream generation unit 752. In the subdivision / combination stage, an intermediate subdivision of the array of information samples representing the spatial sampling of a two-dimensional information signal, and two disjoint sets of simply connected regions whose union is a plurality of simply connected regions are determined. This intermediate subdivision is defined by a first subset and a second subset of syntactic elements that subdivide the array of information samples into multiple non-overlapping simply connected regions of different sizes by a recursive multiple partitioning. By defining according to the first subset of syntactic elements, and according to the second subset of syntactic elements separated from the first subset, spatially adjacent simply connected regions among the plurality of simply connected regions are combined to obtain the intermediate subdivision. The data stream generation unit 752 uses the intermediate subdivision to encode the array of information samples into a data stream. Furthermore, in the subdivision / combination stage 750, the first subset and the second subset are inserted into the data stream. As in the case of Figure 14, the process of determining the first and second subsets and syntactic elements generated by the data stream generation unit 752 may also be an iterative process. For example, the optimal subdivision may be determined in advance in the subdivision / combination stage 750, and the data stream generation unit 752 may determine the corresponding optimal set of syntactic elements for encoding the sample sequence using the sample subdivision along with the subdivision / combination stage, and then set the syntactic elements describing the combination so as to reduce the overhead of the associated information. Also, the encoding process does not have to end here. The subdivision / combination stage 750 may be linked with the data stream generation unit 752 to make settings different from the optimal settings of subdivision and syntactic elements determined in advance by the data stream generation unit 752 in order to determine whether the rate / distortion rate will improve by utilizing the beneficial properties of the combination.

[0201] As described above, the embodiments described with reference to Figures 17 to 21 are a synthesis of the embodiments described with reference to Figures 1 to 16, so that the elements of Figures 1 to 16 can be uniquely related to the elements shown in Figures 17 to 21. For example, the extraction unit 102, together with the subdivision unit 104a and the join unit 104b, performs the tasks performed by the extraction unit 600 in Figure 17. The subdivision unit performs subdivision and manages the adjacency relationships between individual simply connected regions. Subsequently, the join unit 104b manages the joining of simply connected regions to a group and, if the join event is indicated by the information of the join currently being decoded, identifies the correct coding parameters to be copied or used as predictions for the current simply connected region. If entropy decoding is used for data extraction, the extraction unit 102 extracts the actual data from the data stream using the correct context. The remaining elements in Figure 2 are an example of the reconstruction unit 602. Of course, the reconstruction unit 602 may be embodied in a form other than that shown in Figure 2. For example, the reconstruction unit 602 does not need to use motion compensation prediction and / or intra-prediction. The same applies to other possibilities. Furthermore, as mentioned above, the simply connected region referred to in relation to the explanation of Figure 17 corresponds to either the prediction block described above, or one of the other subdivisions described above, such as the subdivision of residuals or the subdivision of filters, as already shown above.

[0202] Comparing the encoder in Figure 19 with the example in Figure 1, the data generation unit 650 includes all elements except the data stream insertion unit 18, which corresponds to the insertion unit 652 in Figure 19. The data generation unit 650 can also use a different coding approach than the hybrid coding approach shown in Figure 1.

[0203] Comparing the decoder in Figure 20 with the example shown in Figure 2, the subdivision section 104a and the coupling section 104b correspond to the subdivision section 100 and the coupling section 102 in Figure 20, respectively, and elements 106 and 114 correspond to the reconstruction section 704. The extraction section 102 is involved in the function of all elements shown in Figure 20 in common.

[0204] Regarding the encoder in Figure 21, the subdivision / combination stage 750 corresponds to the subdivision unit 28 and the combination unit 30, and the data stream generation unit 752 includes all the other elements shown in Figure 10.

[0205] While several embodiments of the apparatus have been described, it is clear that these embodiments also describe the corresponding methods, and blocks or devices correspond to means or features of means. Similarly, embodiments described of means also describe the corresponding blocks or items or the functions of the corresponding apparatus. Some or all of the means may be performed by (or using) hardware devices such as microprocessors, programmable computers, or electronic circuits. In some embodiments, one or more of the most important means may be performed by such devices.

[0206] Signals encoded / compressed according to this invention can be stored in a digital storage medium. Alternatively, they can be transmitted via a transmission medium such as a wireless transmission medium or a wired transmission medium such as the internet.

[0207] Embodiments of the present invention can be implemented in hardware or software, depending on specific implementation requirements. For example, they can be implemented using digital storage media such as floppy disks, DVDs, Blu-ray®, CDs, ROMs, PROMs, EPROMs, EEPROMs, and flash memory, which store electronically readable control signals and are used (or can be used) with a programmable computer system to perform the respective methods. Therefore, the digital storage media may be computer-readable.

[0208] Some embodiments of the present invention include a data storage medium for storing electronically readable control signals. The data storage medium can be used in conjunction with a programmable computer system, and one of the methods described herein is performed.

[0209] Generally, embodiments of the present invention can be implemented as a computer program product including program code, and when the computer program product is executed on a computer, one of the methods is executed by the program code. The program code may be stored, for example, on a machine-readable storage medium. In other embodiments, a computer program that performs one of the methods described herein is included and stored in a machine-readable storage medium.

[0210] Therefore, in other words, an embodiment of the method of the present invention is a computer program that, when executed on a computer, includes program code that performs one of the methods described herein.

[0211] Therefore, another embodiment of the method of the present invention is a data storage medium (or digital storage medium or computer-readable medium) on which a computer program performing one of the methods described herein is recorded.

[0212] Therefore, another embodiment of the method of the present invention is a data stream or a series of signals representing a computer program that performs one of the methods described herein. For example, the data stream or a series of signals may be configured to be transmitted over a data communication connection, such as the Internet.

[0213] Other embodiments include means for processing configured to perform or applied to perform one of the methods described herein, such as a computer or program-controllable logical device. Another embodiment comprises a computer on which a computer program that performs one of the methods described herein is installed.

[0214] In some embodiments, programmable logic devices (such as field-programmable gate arrays) may be used to implement some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may work in conjunction with a microprocessor to perform one of the methods described herein. Generally, it is preferable that the methods be performed by hardware devices.

[0215] The embodiments described above are merely examples to illustrate the basic nature of the present invention. Modifications and variations of the arrangements and details described herein will be obvious to those skilled in the art. Therefore, the invention is limited only by the scope of the claims of the corresponding patent and not by the specific details shown herein.

Claims

1. An extraction unit configured to extract payload data from a data stream for each of several blocks formed by partitioning an array of information samples representing spatially sampled information of the color plane of an image, wherein the image includes multiple color planes. A reconstruction unit configured to reconstruct the image from the payload data by processing the payload data of each block in a manner defined by the encoding parameters associated with each block of the information sample sequence, and A decoder equipped with, The extraction unit is For a given block, the system is configured to extract a combined merge syntax element for the given block from the data stream, where each of the combined merge syntax elements indicates whether or not a merge process is applied to the given block, and if so, for each combined merge syntax element, a specific block out of two possible blocks is identified for the merge process, and the specific block is in a predetermined relative position to the given block. If one or more of the combined merge syntax elements indicate a merge operation of a predetermined block, the encoding parameters of the specific block are adopted as the encoding parameters of the predetermined block for the multiple color planes of the image; if the combined merge syntax elements do not indicate a merge operation, the encoding parameters of the predetermined block are further extracted from the data stream for each of the multiple color planes of the image. decoder.

2. A decoder for decoding a data stream in which an image is encoded, A subdivision unit configured to spatially subdivide an array of information samples representing spatially sampled images by recursively subdividing them multiple times into multiple simply connected regions of different sizes, depending on a first subset of syntactic elements included in the data stream, A merge unit configured to merge a predetermined single-connected region into a spatially adjacent single-connected region, depending on a second subset of syntactic elements in the data stream, which is separated from the first subset, wherein the second subset of syntactic elements includes merged merge syntactic elements, each merged merge syntactic element of the second subset of syntactic elements indicates whether a merge operation of a predetermined single-connected region is applicable, and if applicable, for each merged merge syntactic element, a specific spatially adjacent single-connected region out of two possible spatially adjacent single-connected regions is identified for the merge operation, and the specific spatially adjacent single-connected region is in a predetermined relative positional relationship with respect to the predetermined single-connected region. A reconstruction unit is configured to reconstruct the sequence of information samples from the data stream, to adopt or predict the coding parameters of a predetermined single-connected region from a specific spatially adjacent single-connected region where the predetermined single-connected region is merged for multiple color planes of the image, and to extract the coding parameters of the unmerged single-connected region from the data stream for each of the multiple color planes. A decoder equipped with a decoder.

3. The decoder according to claim 1, wherein the data stream includes depth information.

4. The decoder according to claim 1, wherein the data stream includes a sequence of images, each image including one array of luminance samples and two arrays of chrominance samples, as an array of information samples representing spatially sampled information of the color plane of the images, wherein the scaling factor of the horizontal spatial resolution of the array of chrominance samples relative to the array of luminance samples is different from the scaling factor of the vertical spatial resolution.

5. The decoder according to claim 1, wherein the decoder is configured to independently decode the plurality of color planes of the image.

6. A step of extracting payload data from a data stream for each of several blocks formed by partitioning an array of information samples representing spatially sampled information of the color plane of an image, wherein the image includes multiple color planes. The steps include: reconstructing the image from the payload data of each block by processing the payload data for each block of the information sample sequence in a manner defined by the encoding parameters associated with each block; A method for decryption, including, The extraction step described above is: A step of extracting combined merge syntax elements for a predetermined block from the data stream, wherein each of the combined merge syntax elements indicates whether or not a merge process is applied to the predetermined block, and if applicable, for each combined merge syntax element, a specific block out of two possible blocks is identified for the merge process, and the specific block is in a predetermined relative position to the predetermined block. If one or more of the combined merge syntax elements suggest a merge operation of a predetermined block, the coding parameters of the specific block are adopted as the coding parameters of the predetermined block for the multiple color planes of the image; if the combined merge syntax elements do not indicate a merge operation, the coding parameters of the predetermined block are extracted from the data stream for each of the multiple color planes of the image. Methods that include...

7. A method for decoding a data stream in which an image is encoded, The steps include: spatially subdividing an array of information samples representing spatially sampled images by recursively dividing them multiple times into multiple simply connected regions of different sizes, depending on a first subset of syntactic elements included in the data stream; A step of merging a predetermined single-connected region into a spatially adjacent single-connected region, depending on a second subset of syntactic elements in the data stream, which is separated from the first subset, wherein the second subset of syntactic elements includes merged merge syntactic elements, each merged merge syntactic element of the second subset of syntactic elements indicates whether a merge operation of the predetermined single-connected region is applicable, and if applicable, for each merged merge syntactic element, a specific spatially adjacent single-connected region out of two possible spatially adjacent single-connected regions is identified for the merge operation, and the specific spatially adjacent single-connected region is in a predetermined relative positional relationship with respect to the predetermined single-connected region. The steps include: reconstructing the sequence of information samples from the data stream; adopting or predicting the coding parameters of a predetermined single-connected region from a specific spatially adjacent single-connected region where the predetermined single-connected region is merged for multiple color planes of the image; and extracting the coding parameters of unmerged single-connected regions from the data stream for each of the multiple color planes. Methods that include...

8. The method according to claim 6, wherein the data stream includes depth information.

9. The method according to claim 6, wherein the data stream encodes a sequence of images, each image comprising one array of luminance samples and two arrays of chrominance samples, as an array of information samples representing spatially sampled information of the color plane of the image, and the scaling factor of the horizontal spatial resolution of the array of chrominance samples relative to the array of luminance samples is different from the scaling factor of the vertical spatial resolution.

10. The method according to claim 6, wherein the plurality of color planes of the aforementioned image are decoded independently.

11. An encoder configured to define a method for reconstructing the image from the payload data of each block, wherein for each of the multiple blocks formed by subdividing an array of information samples representing spatially sampled information of the color plane of an image, the array of information samples and the encoding parameters associated with each block are encoded into payload data, and the image is reconstructed from the payload data of each block. For a predetermined block, the system is configured to insert a combined merge syntax element for the predetermined block into the data stream, each of the combined merge syntax elements indicating whether or not a merge process is applied to the predetermined block, and if so, for each combined merge syntax element, it identifies a specific block out of two possible blocks, the specific block is in a predetermined relative position to the predetermined block, and if one or more of the combined merge syntax elements suggest a merge process for the predetermined block, the encoding parameters of the specific block are adopted as the encoding parameters of the predetermined block for multiple color planes of the image, and if the combined merge syntax elements do not indicate a merge process, the encoding parameters of the predetermined block are inserted into the data stream for each of the multiple color planes of the image. Encoder.

12. An encoder for generating a data stream in which an image is encoded, A subdivision / merge stage configured to determine a first subset of syntactic elements that define spatially subdividing an array of information samples representing spatially sampled images into multiple simply connected regions of different sizes by recursively subdividing it multiple times, and a second subset of syntactic elements separated from the first subset that defines merging a predetermined simply connected region into a spatially adjacent simply connected region, wherein each merged merge syntactic element in the second subset of syntactic elements indicates whether or not a merge process for a predetermined simply connected region is applied, and if it is applied, for each merged merge syntactic element, a specific spatially adjacent simply connected region among two spatially adjacent simply connected regions is identified for the merge process, and the specific spatially adjacent simply connected region is in a predetermined relative positional relationship with respect to the predetermined simply connected region, A data stream generation unit configured to encode the sequence of information samples into the data stream and to insert a first subset and a second subset of the syntactic elements into the data stream. Equipped with, The coding parameters for the predetermined single-connected region are indicated by a second subset of the syntactic elements to be adopted or predicted from the specific spatially adjacent single-connected regions where the predetermined single-connected region is merged with respect to multiple color planes of the image. The data stream insertion unit is configured to insert the encoding parameters of unmerged single-connected regions into the data stream for each of the plurality of color planes. Encoder.

13. The encoder according to claim 11, wherein the data stream includes depth information.

14. The encoder according to claim 11, wherein the data stream encodes a sequence of images, each image comprising one array of luminance samples and two arrays of chrominance samples, as an array of information samples representing spatially sampled information of the color plane of the image, and the scaling factor of the horizontal spatial resolution of the array of chrominance samples relative to the array of luminance samples is different from the scaling factor of the vertical spatial resolution.

15. The encoder according to claim 11, wherein the encoder is configured to independently encode the plurality of color planes of the image.

16. A method for defining a method for reconstructing an image from the payload data of each block, wherein for each of the multiple blocks formed by subdividing an array of information samples representing spatially sampled information of the color plane of an image, the array of information samples and the encoding parameters associated with each block are encoded into payload data, the method being defined A step of inserting a combined merge syntax element for a predetermined block into a data stream, wherein each of the combined merge syntax elements indicates whether or not a merge process is applied to the predetermined block, and if applicable, for each combined merge syntax element, a specific block out of two possible blocks is identified, the specific block is in a predetermined relative position to the predetermined block, and if one or more of the combined merge syntax elements suggest a merge process for the predetermined block, the encoding parameter of the specific block is adopted as the encoding parameter of the predetermined block for a plurality of color planes of the image, and if the combined merge syntax elements do not indicate a merge process, the encoding parameter of the predetermined block is inserted into the data stream for each of the plurality of color planes of the image. Methods that include...

17. A method for generating a data stream in which an image is encoded, A step of determining a first subset of syntactic elements that defines spatially subdividing an array of information samples representing spatially sampled images into multiple simply connected regions of different sizes by recursively segmenting multiple times, and a second subset of syntactic elements separated from the first subset that defines merging predetermined simply connected regions into spatially adjacent simply connected regions, wherein the second subset of syntactic elements includes merged syntactic elements, each merged merge syntactic element of the second subset of syntactic elements indicates whether a merge operation of predetermined simply connected regions is applied, and if it is applied, for each merged merge syntactic element, a specific spatially adjacent simply connected region out of two spatially adjacent simply connected regions is identified for the merge operation, and the specific spatially adjacent simply connected region is in a predetermined relative positional relationship with respect to the predetermined simply connected region. The steps include encoding the sequence of information samples into the data stream, and inserting a first subset and a second subset of the syntactic elements into the data stream. Includes, The coding parameters for the predetermined single-connected region are indicated by a second subset of the syntactic elements to be adopted or predicted for multiple color planes of the image from the specific spatially adjacent single-connected regions to which the predetermined single-connected region is merged, The data stream insertion unit is configured to insert the encoding parameters of unmerged single-connected regions into the data stream for each of the plurality of color planes. method.

18. The method according to claim 16, wherein the data stream includes depth information.

19. The method according to claim 16, wherein the data stream encodes a sequence of images, each image comprising one array of luminance samples and two arrays of chrominance samples, as an array of information samples representing spatially sampled information of the color plane of the image, and the scaling factor of the horizontal spatial resolution of the array of chrominance samples relative to the array of luminance samples is different from the scaling factor of the vertical spatial resolution.

20. The method according to claim 16, wherein the plurality of color planes of the aforementioned image are encoded independently.

21. A method for decoding a data stream in which an array of information samples representing spatially sampled information of the color plane of an image is encoded, The method includes receiving and decoding a data stream that includes payload data for each of a plurality of blocks formed by subdividing the sequence of information samples, and encoding parameters associated with each block, and defining a method for reconstructing the image from the payload data for each block. The data stream further includes combined merge syntax elements for a predetermined block, each of which indicates whether a merge process is applied to the predetermined block, and if so, for each combined merge syntax element, a specific block is identified from two possible blocks, the specific block is in a predetermined relative position to the predetermined block, and for multiple color planes of the image, the encoding parameters of the specific block are adopted as the encoding parameters of the predetermined block. For each predetermined block in the combined merge syntax element that does not indicate a merge process, the encoding parameters of each predetermined block are encoded in the data stream for each of the multiple color planes of the image. method.

22. A method for decoding a data stream in which an image is encoded, The step includes receiving and decoding the data stream, the data stream is A first subset of syntactic elements that defines spatially subdividing an array of information samples representing spatially sampled images into multiple simply connected regions of different sizes by recursively subdividing it multiple times, A second subset of syntactic elements, separated from the first subset, defines merging a predetermined single-connected region into a spatially adjacent single-connected region, wherein the second subset of syntactic elements includes merged merge syntactic elements, each merged merge syntactic element of the second subset of syntactic elements indicates whether or not the merge operation of the predetermined single-connected region is applicable, and if applicable, for each merged merge syntactic element, a specific spatially adjacent single-connected region out of two spatially adjacent single-connected regions is identified for the merge operation, and the specific spatially adjacent single-connected region is in a predetermined relative positional relationship with the predetermined single-connected region, and Includes, The sequence of the aforementioned information samples is encoded into the data stream, The coding parameters for the predetermined single-connected region are indicated by a second subset of the syntactic elements to be adopted or predicted for multiple color planes of the image from the specific spatially adjacent single-connected regions to which the predetermined single-connected region is merged, The data stream includes, for each of the plurality of color planes, encoding parameters for unmerged single-connected regions. method.

Images