Improvement in boundary forced partition

Abstract

To provide a method and device of partitioning an image or an image slice.SOLUTION: In particular, an image or image slice is split into an integer number of coding tree units in both vertical and horizontal directions. The remaining boundary portion is then at least partly forced-split so that the deepest partitioning level is a result of binary partitioning, i.e. includes a coding unit which is longer on one side than on the other side. The longer side is parallel or concurrent to the image or image slice boundary.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] FIELD This disclosure relates to image and / or video encoding and decoding, and in particular to dividing an image into coding units. [Background technology]

[0002] Current block-based hybrid video codecs employ predictive coding. Pictures of a video sequence are sub-divided into blocks of pixels, and then these blocks are coded. Instead of coding the blocks pixel by pixel, the whole block is predicted using already coded pixels that are spatially or temporally close to the block. The encoder further processes only the difference between the block and the prediction of that block. The further processing generally involves transforming the block pixels into coefficients in the transform domain. The coefficients may then be further compressed (e.g., by means of quantization) and further compacted (e.g., by entropy coding) to form a bitstream. The bitstream may further include any signaling information that enables a decoder to decode the coded video. For example, the signaling may include settings regarding encoder settings such as the size of the input picture, the frame rate, a quantization step indication, the prediction applied to the blocks of the picture, etc.

[0003] The difference between a block and a prediction of that block is known as the residual of the block. More specifically, each pixel of the block has a residual that is the difference between the intensity level of that pixel and the predicted intensity level of that pixel. The intensity level of a pixel is called the pixel value or the value of the pixel. The residuals of all pixels of a block are collectively called the residual of the block. In other words, a block has a residual that is a set or matrix of the residuals of all pixels of the block. The residual is then transformed, quantized, and coded together with the signaling information. The coding may include various forms of fixed and variable length coding, including arithmetic coding or other types of entropy coding.

[0004] In block-based hybrid video coding, each picture is partitioned into blocks of samples, and multiple blocks in a picture are aggregated to form slices as independently decodable entities. A block to which prediction and / or transformation is applied is called a coding unit (CU) or coding block (CB). Coding units may have different sizes.

[0005] For example, in High Efficiency Video Coding (HEVC, also known as H.265), a video frame is subdivided into coding tree units (CTUs, also called coding tree blocks, CTBs). CTBs are disjoint square blocks of the same size, e.g., 64 x 64 samples. Each CTB serves as the root of a coding tree, a quadtree structure that partitions blocks. CTBs may be further subdivided into coding blocks along the coding tree structure. For the coding blocks, a type of prediction is determined. The coding blocks may be further divided into smaller transform blocks, where transformation and quantization are applied.

[0006] More details on partitioning in HEVC can be found in V. Sze et al. (eds.), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014, Chapter 3.2.

[0007] Furthermore, WO 2016 / 090568 shows a binary tree structure for partitioning a unit into multiple smaller units using a binary tree structure in addition to a quad tree. Thus, first, the root unit is partitioned into a quad tree structure, and then the leaf nodes of the quad tree are further partitioned by the binary tree structure. Summary of the Invention [Means for solving the problem]

[0008] In the HEVC standard, CTUs and CUs that are on slice or picture boundaries are forced to split using quadtree partitioning until the bottom right sample of the leaf node falls within the slice or picture boundary. The forced quadtree partitioning does not need to be signaled in the bitstream as long as the partitioning rules are fixed (predefined). The purpose of the forced partitioning is to allow encoders and decoders to process pictures and slices that have horizontal and / or vertical sizes that are not integer multiples of the respective horizontal and / or vertical sizes of the CTUs.

[0009] In HEVC and WO 2016 / 090568, the forced partitioning process hierarchically splits CTUs and CUs at frame boundaries by a quadtree approach without rate-distortion (RC) optimization until the entire current CU falls within a slice or picture boundary. These forced partitions do not need to be signaled in the bitstream. However, splitting into very small coding units may introduce additional complexity and signaling requirements related to separate further processing (encoding, decoding) of the small coding units.

[0010] According to the present disclosure, forced partitioning applies binary tree partitioning at least at the deepest partitioning hierarchical level, or at least at a partitioning level other than the deepest partitioning hierarchical level where no coding unit remains completely within a boundary portion.

[0011] Such a partitioning provides greater efficiency as it results in a smaller number of coding units at the deepest hierarchical levels.

[0012] According to an aspect of the invention, an apparatus is provided for dividing an image portion into coding units, the apparatus including processing circuitry that, during operation, subdivides the image portion into an integer number of coding tree units having a predetermined size and boundary portions having a size less than the predetermined size in at least a first direction, either horizontally or vertically, and hierarchically partitions the boundary portions into coding units including a deepest coding unit having different sizes in respective horizontal and vertical directions, the deepest coding unit being smaller in the first direction.

[0013] The advantage of this approach is that it avoids too many small CUs on the picture borders, which can also result in more efficient coding of the border parts.

[0014] According to an example, the processing circuitry is configured to partition the boundary portion by binary tree partitioning at least at the deepest partitioning level.

[0015] In addition to or as an alternative to binary tree partitioning, the processing circuitry may also be configured to partition the boundary portion by quadtree partitioning.

[0016] According to one example, during operation, the processing circuit partitions the boundary portion by quad-tree partitioning at the N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchical levels of the partitioning, and partitions the boundary portion by binary tree partitioning at the remaining hierarchical levels of the partitioning.

[0017] This approach offers greater flexibility in construction and further ensures that the boundaries are coded efficiently by avoiding a large number of small square boundary CUs.

[0018] In one implementation, during operation, the processing circuitry segments the boundary portion until the deepest level coding unit remains entirely within the boundary portion.

[0019] In one implementation, during operation the processing circuitry partitions the boundary portion horizontally and vertically using a binary tree partition.

[0020] Additionally or alternatively, during operation the processing circuitry partitions corner portions of the boundary portions at each hierarchical level by quadtree partitioning.

[0021] As an alternative to mixing quad-tree and binary tree partitioning for the CTUs of the boundary portion, the processing circuitry may be configured to partition the boundary portion by binary tree partitioning at all levels of partitioning.

[0022] According to an aspect of the invention, an apparatus for encoding images of a video sequence is provided, the apparatus including an apparatus for dividing an image portion into coding units according to any of the above-mentioned examples and implementations, an image coding unit configured to code the coding units, and a bitstream forming unit configured to generate a bitstream including the encoded coding units and partitioning information indicating how the coding tree units are partitioned.

[0023] For example, the encoder further includes a rate-distortion optimization unit configured to determine a hierarchical partitioning of the boundary portions by rate-distortion optimization for any of the levels of partitioning apart from the deepest level.

[0024] Another aspect of the present invention includes an apparatus for decoding images of a video sequence, the apparatus including: an apparatus for determining a division of a decoded image portion into coding units according to any of the above-mentioned examples and implementations; a bitstream parser for parsing a bitstream including the coded coding units and partitioning information indicating how the coding tree units are partitioned and based on the determined partitioning of the image portion; and an image decoding unit for decoding the coded coding units.

[0025] According to an exemplary implementation, the division of the boundary portion into coding units is a predefined division, and no partitioning information is related to the division of the boundary portion.

[0026] The processing circuit of the device for decoding and / or the device for encoding may be configured to partition the boundary portion by quad-tree partitioning at the N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchical levels of partitioning, and partition the boundary portion by binary tree partitioning at the remaining hierarchical levels of partitioning. Furthermore, in an exemplary implementation, the partitioning information includes N or N is derived based on a type of prediction applied to encode the image to be processed. N may be derived based on the size of the boundary portion having a size less than a predetermined size in either the horizontal or vertical direction.

[0027] According to an aspect of the present invention, there is provided a method for dividing an image portion into coding units, the method comprising the steps of: subdividing the image portion into an integer number of coding tree units having a predetermined size and boundary portions having a size in at least a first direction, either horizontally or vertically, less than the predetermined size; and hierarchically partitioning the boundary portions into coding units including a deepest coding unit having different sizes in respective horizontal and vertical directions, the deepest coding unit being smaller in the first direction.

[0028] In an exemplary embodiment, the partitioning of the boundary portion further includes the steps of starting from a current partitioning level, which is the shallowest partitioning level; dividing the boundary portion into coding units of the current partitioning level in sequence until a boundary coding unit of the current partitioning level fits completely within the boundary portion; and as soon as a coding unit of the current partitioning level does not fit within the boundary portion, increasing the partitioning level and further partitioning said coding unit, where quadtree partitioning is used for the N shallowest partitioning levels and binary tree partitioning is used in other cases, where N is an integer greater than or equal to zero and less than the number of hierarchical levels of partitioning.

[0029] According to another aspect of the present invention, there is provided a method for encoding images of a video sequence, the method comprising the steps of dividing an image portion into coding units according to any of the methods specified above, encoding the coding units, and generating a bitstream comprising the encoded coding units and partitioning information indicating how the coding tree units are partitioned.

[0030] According to another aspect of the invention, a method for decoding an image of a video sequence, comprising the steps of determining a division of a decoded image portion into coding units according to any of the methods specified above, analysing a bitstream containing partitioning information based on the determined division of the image portion, indicating how the coded coding units and coding tree units are partitioned, and decoding the coded coding units.

[0031] According to an example, the partitioning step partitions the boundary portion by binary tree partitioning at least at the deepest partitioning level.

[0032] In addition to or as an alternative to binary tree partitioning, the partitioning step may also partition the boundary portion by quad-tree partitioning.

[0033] According to one example, the partitioning step includes partitioning the boundary portion by quad-tree partitioning at the N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchical levels of the partitioning, and partitioning the boundary portion by binary tree partitioning at the remaining hierarchical levels of the partitioning.

[0034] In one implementation, the partitioning step partitions the boundary portion until the deepest level coding unit remains entirely within the boundary portion.

[0035] In one implementation, during operation the processing circuitry partitions the boundary portion horizontally and vertically using a binary tree partition.

[0036] Additionally or alternatively, the processing step partitions, at each hierarchical level, corners of the boundary portion by quadtree partitioning.

[0037] As an alternative to mixing quad-tree and binary tree partitioning for the CTUs of the boundary portion, the partitioning step may partition the boundary portion by binary tree partitioning at all partitioning levels.

[0038] According to another aspect of the invention, a computer-readable medium is provided that stores instructions that, when executed by a processor, cause the processor to perform a method according to any of the above aspects and examples.

[0039] In the following, exemplary embodiments are explained in more detail with reference to the accompanying figures and drawings. [Brief description of the drawings]

[0040] [Figure 1] 1 is a block diagram illustrating an example structure of a video encoder. [Diagram 2] 2 is a block diagram illustrating an exemplary structure of a video decoder. [Diagram 3] FIG. 2 is a schematic diagram illustrating an example of a quadtree partitioning adopted by HEVC; [Figure 4] FIG. 1 is a schematic diagram showing an example of quad-tree / binary-tree division. [Diagram 5] FIG. 13 is a schematic diagram showing forced quadtree division of a boundary portion. [Figure 6] FIG. 2 is a block diagram illustrating an example of a device that may act as a splitting unit of an encoder or decoder. [Figure 7] FIG. 13 is a schematic diagram showing division of a boundary portion. [Figure 8] FIG. 1 is a schematic diagram illustrating an example of forced binary tree splitting of vertical boundaries. [Figure 9] 1A-1C are schematic diagrams showing examples of partitioning on vertical boundaries, horizontal boundaries, and combined boundaries. [Figure 10] 11 is a flow diagram illustrating an example partitioning syntax determination. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] The present invention relates to the division of an image into smaller units for further processing. Such a division may be advantageously used in the encoding and decoding of still or video images. In the following, exemplary video coders and decoders capable of implementing the division according to the present disclosure are described.

[0042] FIG. 1 shows an encoder 100 including an input for receiving an input block of a frame or picture of a video stream and an output for providing an encoded video bitstream. The term "frame" in this disclosure is used as a synonym for picture. However, it is noted that this disclosure is also applicable to fields when interlacing is applied. Generally, a picture includes m by n pixels. These correspond to image samples and may each include one or more color components. For simplicity, the following description refers to pixels meaning samples of luminance. However, it is noted that the division technique of this disclosure may be applied to any color component, including chrominance or components of a color space such as RGB. On the other hand, it may be beneficial to perform the division on only one component and apply the determined division to further (or all) remaining components.

[0043] The encoder 100 is configured to apply segmentation, prediction, transformation, quantization, and entropy coding to the video stream.

[0044] In the division unit 110, the input video frame is further divided before encoding. The blocks to be coded are not necessarily of the same size. One picture may contain blocks of different sizes, and the block rasters of different pictures of a video sequence may also be different. In particular, each video image (picture) is first sub-divided into CTUs of the same fixed size. The CTU size may be, for example, fixed by a standard and predefined. In HEVC, a size of 64 x 64 is used. However, the present disclosure is not limited to the standardized fixed size. It may be advantageous to provide a CTU size that may be set in the encoder and provided as a signaling parameter in the bitstream. For example, different CTU sizes may be beneficial for each different picture size and / or type of content. The CTU size may be signaled at any signaling level, for example, it may be common to the entire video sequence or a part thereof (i.e., multiple pictures), or it may be individual for each picture. Correspondingly, the CTU size may be signaled, for example, in a picture parameter set, PPS, known from current codecs (H.264 / AVC, H.265 / HEVC), or in a sequence parameter set, SPS, or in a video parameter set, VPS, or in a similar parameter set. Alternatively, the CTU size may be specified in a slice header or at any other level. The CTU size may take values ​​different from 64 x 64. The CTU size may, for example, be 128 x 128 samples in size. In general, to perform a hierarchical division by a binary tree of a quadtree, it may be beneficial to provide a CTU size that is a power of 2, i.e., in the format of 2^n, where n is an integer greater than 2.

[0045] The partitioning of a picture into CTUs and the partitioning of CTUs into CUs from V. Sze et al. (eds.), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014, is shown in Fig. 3. The partitioning follows a quadtree structure to accommodate various local features. On the left side, Fig. 3 shows a CTU partitioned hierarchically according to the quadtree structure on the right side. In particular, the coding tree defines a syntax that specifies the subdivision of the CTU into CUs. As with a CTU, a CU consists of a square block of samples and syntax associated with these sample blocks. Thus, the partitioning is performed hierarchically starting from a CTU (hierarchical depth 0) that may or may not be subdivided into four (quadtree) CUs of hierarchical depth 1. In Figure 3, the CTU is partitioned into CUs 8 and 16 at the first hierarchical depth (level) that are not further partitioned and therefore form leaves of the quadtree, and two further CUs that are further partitioned into CUs at hierarchical depth 2 (depth 2 CUs). In particular, the top left depth 1 CU is further sub-partitioned into depth 2 CUs 1, 2, 7 that form leaves of the quadtree, and another CU that is further partitioned into depth 3 CUs 3, 4, 5, and 6 that are all leaves. Similarly, the bottom left depth 1 CU is further partitioned into depth 2 CUs 13, 14, and 15 that are also leaves of the quadtree, and the remaining CUs that are further partitioned into level 3 CUs 9, 10, 11, and 12 that are all leaves and therefore are not further partitioned.

[0046] An example syntax for HEVC quadtree partitioning is shown in Table 1 below.

[0047] [Table 1]

[0048] In particular, at the CTU level, a flag named split_cu_flag is included in the bitstream to indicate whether the complete CTU forms a CU or whether the CTU is split into four equally sized blocks corresponding to square sample blocks. If a CTU is split, for each of the resulting blocks, another split_cu_flag is transmitted specifying whether the block represents a CU or whether the block is further split into four equally sized blocks. This hierarchical subdivision continues until none of the resulting blocks are further subdivided. A minimum size of a CU is signaled in the sequence parameter set, which can range from 8 x 8 luma samples to the size of the CTU. When the minimum CU size is reached in the hierarchical subdivision process, no split flag is transmitted for the corresponding blocks, and instead it is inferred that these blocks will not be further split. In a typical HEVC encoder setting, the maximum range of supported CU sizes is utilized, so that CUs ranging from 8 x 8 samples to 64 x 64 samples can be used. The CUs in a CTU are coded in a depth-first order. This coding order is also called z-scanning. It ensures that for each CU, except for CUs at the top or left boundary of the slice, all samples above the CU and to the left of the CU have already been coded, and therefore the corresponding samples can be used for intra prediction and the associated coding parameters can be used to predict the coding parameters of the current CU.

[0049] In other words, split_cu_flag[x0][y0] specifies whether the coding unit is split into coding units with half horizontal and vertical size. The array indexes x0, y0 specify the position (x0, y0) of the top-left luma sample of the considered coding block relative to the top-left luma sample of the picture. When split_cu_flag[x0][y0] is not present, the following applies at the decoder: - If log2CbSize (a parameter that specifies the coding block size) is greater than MinCbLog2SizeY (a parameter that specifies the minimum configurable coding unit size), the value of split_cu_flag[ x0 ][ y0 ] is inferred to be equal to 1. - Otherwise (log2CbSize is equal to MinCbLog2SizeY), the value of split_cu_flag[ x0 ][ y0 ] is inferred to be equal to 0.

[0050] The array CtDepth[ x ][ y ] specifies the depth of the coding tree for the luma coding block covering position ( x, y ). When split_cu_flag[ x0 ][ y0 ] is equal to 0, CtDepth[ x ][ y ] is inferred to be equal to cqtDepth for x = x0..x0 + nCbS - 1 and y = y0..y0 + nCbS - 1.

[0051] Fig. 4 shows mixed quadtree and binary tree partitioning. Quadtree partitioning is indicated by solid lines, while binary tree partitioning is indicated by dashed lines. The label 1 or 0 of the node representing the coding unit that is further divided by the binary tree indicates whether the binary partitioning is applied vertically or horizontally, respectively.

[0052] The horizontal and vertical size of a video picture in luma samples must be an integer multiple of the smallest CU size in luma samples signaled in the sequence parameter set, but need not be an integer multiple of the CTU size. If the horizontal or vertical size of a video picture does not represent an integer multiple of the CTU size, it is inferred that the CTUs at the boundaries are split until the resulting block boundaries coincide with the picture boundaries. For this forced split, no split flag is signaled, but the resulting block may be further split using the quadtree syntax described above. CUs that are outside the picture area are not coded.

[0053] This division is shown in Figure 5. In particular, Figure 5 shows a frame boundary, where the 56 lines (128 samples long) above the frame boundary are the slice or image boundary portion. The portion of the CTU below the frame boundary may belong to another slice, or may not exist at all, for example if the frame boundary is the bottom boundary of the picture. As can be seen, a forced quadtree division is applied to the 128 x 56 samples.

[0054] The subdivision of the chroma CTB is always aligned with the respective luma CTB subdivision in HEVC. It is noted that the present disclosure may treat the chroma components in the same way, but is not limited to it. There may also be independent division of different color components.

[0055] After performing image segmentation in the segmentation unit 110, transformation, quantization, and entropy coding are performed by the transformation unit 130, the quantization unit 140, and the entropy coding unit 150, respectively, to generate an encoded video bitstream as output.

[0056] A video stream may include multiple frames. For example, a block of a first frame of the video stream is intra-coded by the intra prediction unit 190. An intra frame is coded using only information from that frame, and therefore, the intra frame may be decoded independently of other frames. Thus, the intra frame may provide an entry point of the bitstream, for example, for random access. Blocks of other frames of the video stream may be inter-coded by the inter prediction unit 195, and each block of the inter-coded frame is predicted from a block of another frame (reference frame), for example, a previously coded frame. The mode selection unit 180 is configured to select whether a block of a frame should be intra-predicted or inter-predicted, i.e., whether the block is processed by the intra prediction unit 190 or the inter prediction unit 195. The mode selection unit 180 also controls an intra parameter of the inter prediction. To enable refreshing of image information, an inter-coded frame may include not only an inter-coded block, but also one or more intra-coded blocks. In contrast, an intra-frame contains only intra-coded blocks, and no inter-coded blocks. Intra-frames may be inserted into a video sequence (e.g., regularly, i.e., every time after a certain number of inter-frames) to provide an entry point for decoding, i.e., a point at which a decoder can begin decoding without using information from previous frames.

[0057] The intra prediction unit 190 is a block prediction unit. To perform spatial or temporal prediction, the coded block may be further processed by the inverse quantization unit 145 and the inverse transform unit 135. After the block reconstruction by the reconstructor 125, the loop filtering unit 160 may be applied to further improve the quality of the decoded image. The reconstructor 125 adds the decoded residual to the predictor to obtain a reconstructed block. The filtered block then forms a reference frame that is then stored in the frame buffer 170. Such a decoding loop (decoder) on the encoder side provides the advantage of generating a reference frame that is the same as the reference picture reconstructed on the decoder side. Therefore, the encoder and decoder sides operate in a corresponding manner. The term "reconstruction" in this specification refers to obtaining a reconstructed block by adding a decoded residual block to a prediction block.

[0058] The inter prediction unit 195 receives as input a block of a current frame or picture to be inter-coded and one or more reference frames or pictures from the frame buffer 170. Motion estimation and motion compensation are performed by the inter prediction unit 195. Motion estimation is used, for example, to obtain a motion vector and a reference frame based on a cost function. Motion compensation is then performed by translating a reference block of a reference frame to the current frame, that is, describing a current block of the current frame by a motion vector. The inter prediction unit 195 selects a prediction block (i.e., a predictor) for the current block from a set of candidate blocks (i.e., candidate predictors) in one or more reference frames such that the prediction block minimizes the cost function. In other words, the candidate block with the smallest cost function is used as the prediction block for the current block.

[0059] For example, the cost function may be a measure of the difference between the current block and the candidate block, i.e., a measure of the residual of the current block relative to the candidate block. For example, the cost function may be the sum of absolute differences (SAD) between every pixel (sample) of the current block and every pixel of the candidate block in the candidate reference picture. However, in general, any similarity metric, such as the mean square error (MSE) or the structural similarity metric (SSIM), may be employed.

[0060] However, the cost function may also be the number of bits required to code such an inter block and / or the distortion resulting from such coding. Thus, a rate-distortion optimization procedure may be used to determine coding parameters such as the selection of motion vectors and / or whether and in what settings inter or intra prediction should be used for a block in general.

[0061] The intra prediction unit 190 receives as input a block of the current frame or picture to be intra coded and one or more reference samples from already reconstructed areas of the current frame. Intra prediction then describes the pixels of the current block of the current frame by a function of the reference samples of the current frame. The intra prediction unit 190 outputs a prediction block for the current block, said prediction block advantageously minimizing the difference between the current block to be coded and its prediction block, i.e. said prediction block minimizes the residual block. The minimization of the residual block can be based, for example, on a rate-distortion optimization procedure. In particular, the prediction block is obtained as a directional interpolation of the reference samples. The direction may be determined by rate-distortion optimization and / or by calculating the similarity measure mentioned above in connection with inter prediction.

[0062] The difference between the current block and its prediction, i.e., the residual block, is then transformed by the transform unit 130. The transform coefficients are quantized by the quantization unit 140 and entropy coded by the entropy coding unit 150. The coded video bitstream so generated includes intra-coded and inter-coded blocks and corresponding signaling (such as mode indication, motion vector indication, and / or intra prediction direction). The transform unit 130 may apply a linear transform, such as a discrete Fourier transform (DFT) or a discrete cosine transform (DCT). Such a transform into the spatial frequency domain brings the advantage that the resulting coefficients generally have higher values ​​at lower frequencies. Thus, after effective coefficient scanning (such as zigzag) and quantization, the resulting sequence of values ​​generally has some larger values ​​at the beginning and end of the run of zeros. This allows for more efficient coding. The quantization unit 140 performs lossy compression by reducing the resolution of the coefficient values. The entropy coding unit 150 then assigns binary codewords to the coefficient values. The codewords are written into a bitstream, called the coded bitstream. The entropy coder also encodes signaling information (not shown in FIG. 1), which may include encoding the split flag syntax shown above.

[0063] 2 shows an example of a video decoder 200. The video decoder 200 includes, among other things, a reference picture buffer 270 and an intra prediction unit 290, which is a block prediction unit. The reference picture buffer 270 is configured to store at least one reference frame reconstructed from the encoded video bitstream of the encoded video bitstream. The intra prediction unit 290 is configured to generate a prediction block, which is an estimate of a block to be decoded. The intra prediction unit 290 is configured to generate this prediction based on reference samples obtained from the reference picture buffer 270.

[0064] Decoder 200 is configured to decode the encoded video bitstream generated by video encoder 100, but preferably both decoder 200 and encoder 100 generate identical predictions for each encoded / decoded block. The features of reference picture buffer 270 and intra prediction unit 290 are similar to those of reference picture buffer 170 and intra prediction unit 190 of FIG.

[0065] The video decoder 200 includes further units that are also present in the video encoder 100, such as, for example, an inverse quantization unit 240, an inverse transform unit 230, and a loop filtering unit 260, which correspond respectively to the inverse quantization unit 140, the inverse transform unit 150, and the loop filtering unit 160 of the video coder 100.

[0066] The bitstream parsing, entropy decoding, and partitioning unit 250 is configured to parse and decode the received coded video bitstream to obtain quantized residual transform coefficients and signaling information. The quantized residual transform coefficients are provided to the inverse quantization unit 240 and the inverse transform unit 230 to generate a residual block. The residual block is added to the prediction block in the reconstructor 225, and the resulting sum is provided to the loop filtering unit 260 to obtain a decoded video block. The frames of the decoded video are stored in the reference picture buffer 270 and can serve as reference frames for inter prediction. The signaling information parsed and decoded from the bitstream may generally include control information related to the partitioning of the frames. To further correctly parse and decode the image, the control information is used to restore the partitioning of the image into coding units to correctly assign subsequent decoded data to the respective coding units.

[0067] In general, the intra prediction units 190 and 290 in Figures 1 and 2 can use reference samples from already coded regions to generate a prediction signal for a block that needs to be coded or decoded.

[0068] The bitstream parsing, entropy decoding, and splitting unit 250 receives the encoded bitstream as its input. First, the bitstream may be parsed, i.e., signaling parameters and residuals are extracted from the bitstream. The syntax and semantics of the bitstream may be defined by a standard so that encoders and decoders may work in an interoperable manner.

[0069] This disclosure presents a new approach to boundary forced partitioning in block-based hybrid video coding. In particular, boundary forced partitioning is performed, at least in part, by using a binary tree (BT) structure. According to the inventors' observation, since most boundary blocks may use the same prediction mode, forced quad tree (QT) partitioning for frame boundary CTUs is not always necessary. Instead of forced QT partitioning, in one embodiment, forced horizontal BT for bottom boundary parts and / or forced vertical BT for right boundary parts are introduced (assuming that upper and lower neighboring parts are already reconstructed in the processing order of left-to-right and top-to-bottom CTUs).

[0070] In particular, as shown in Fig. 6, according to an embodiment, an apparatus 600 (which may be implemented in units 110, 250 described above) for dividing an image portion into coding units (610) includes a processing circuit that, during operation, subdivides the image portion into an integer number of coding tree units having a predetermined size and a boundary portion having a size less than the predetermined size in at least a first direction, which may be either a horizontal direction or a vertical direction. Further, the processing circuit, during operation, hierarchically partitions (630) the boundary portion into coding units, including a deepest coding unit having different sizes in respective horizontal and vertical directions, the deepest coding unit being smaller in the first direction.

[0071] The CTUs obtained by frame partitioning 610 may be further hierarchically partitioned 620. This partitioning may be performed in any manner, for example as shown in and described above with reference to Figures 3 and 4.

[0072] 6 shows the internal structure of the circuit of the device 600. This circuit may be any kind of hardware and software with functional units 610, 620 and 630 for respective division of the frame into CTUs, partitioning of the CTUs and partitioning of the boundary portions. These units may be implemented, for example, in a single processor. However, the invention is not limited to such application and these units may also be implemented by separate hardware parts.

[0073] The border portions may be on vertical boundaries only (if the vertical picture size is not an integer multiple of the vertical CTU size), or on horizontal boundaries only (if the horizontal picture size is not an integer multiple of the horizontal CTU size), or on both vertical and horizontal boundaries (if the horizontal and vertical picture sizes are not integer multiples of the respective horizontal and vertical CTU sizes).

[0074] A normal picture to be coded has a size that is a multiple of the CTU. FIG. 7 visualizes an example of a picture boundary 700 below and a corresponding boundary portion 710 (filled). The portion 750 shows the remaining portion of the image that has a size that is an integer multiple of the CTU in the vertical and horizontal directions. In particular, the vertical size of the CTU is denoted as 770V, while the horizontal size of the CTU is denoted as 770H. As can be seen in FIG. 7, the boundary portion in this example is an integer multiple of the CTU size 770H in the horizontal direction. However, in the vertical direction, the boundary portion 710 has a size that is the remainder of the vertical picture size divided by the vertical CTU size 770V. The portion 720 is only virtual and shows the difference between the height of the boundary portion and the CTU size. It is noted that in this implementation, the CTU is a square, and therefore the sizes 770H and 770V are the same. However, the disclosure is not limited thereto, and the vertical and horizontal sizes of the CTU may be different.

[0075] To encode (and correspondingly decode) the boundary portion, the boundary portion 710 in Figure 7 is divided into incomplete CTUs, i.e., portions having a horizontal size 770H of the CTU and a vertical size 770V that is smaller than the CTU size. These incomplete CTUs are further forced to be partitioned as shown in Figure 8.

[0076] FIG. 8 shows an example where the boundary part is at the horizontal boundary, and therefore the division is performed vertically. In particular, on the left side of FIG. 7, a boundary part is shown that corresponds to the width of the CTU (in this example, the width is 128 samples), but with a height (only 56 samples) lower than the height of the CTU. The vertical boundary is forced to be partitioned by a binary tree division. In particular, a depth 1 unit with 64 samples does not fit the high boundary part of 56 samples, so the largest unit size that fits the length of 56 samples is a depth 2 unit of size 32 samples. The remaining 24 samples are further divided into a depth 3 unit of size 16 samples and a remainder of 8 samples (corresponding to the size of a depth 4 CU), which reaches the picture boundary, and therefore no further division is necessary. In this example, no division is applied to the horizontal boundary. In summary, a boundary unit having the size of a CTU in one direction and a smaller size in another direction is partitioned by a binary tree partition, i.e., by hierarchically partitioning the boundary unit into two parts, one having the size of the next shallowest CU that fits the smaller size, and the remaining part being further partitioned until the deepest CU reaches the frame boundary. This partitioning of the CTU part of size 128 x 56 here results in three boundary coding units with respective sizes of 128 x 32, 128 x 16, and 128 x 8.

[0077] In summary, according to one implementation, the processing circuitry is configured to partition the boundary portions at all levels of partitioning by binary tree partitioning.

[0078] The right side of Figure 8 shows an example of partitioning where first a quad-tree partitioning is applied at hierarchical depth 1 (hierarchical depth 0 corresponding to the CT size), resulting in two CUs of size 64 x 56, which are then further bi-tree partitioned into respective pairs of CUs of size 64 x 32, 64 x 16, and 64 x 8.

[0079] The example of FIG. 8 shows the partitioning of the bottom boundary. However, the present disclosure is equally applicable to the partitioning of any other boundary, such as the right boundary. If the subdivision of the picture into CTUs is performed from left to right and from top to bottom, then the boundary portion is at the bottom and / or right side of the picture. However, the partitioning may also be performed from bottom to top and from right to left, in which case the top and left boundaries need to be partitioned, which is possible in the same way as described with reference to FIG. 8. Similarly, the above example shows the partitioning of such an incomplete CTU starting from the top of the CTU. However, it is also possible to partition the incomplete CTU generally from the picture (frame) boundary upwards.

[0080] As shown on the right side of FIG. 8, the incomplete CTU partition may be a mixed QT and BT partition. The processing circuit may be configured to partition the boundary portion by binary tree partitioning at least at the deepest partitioning level. This provides the advantage that larger boundaries that overlap frame boundaries do not need to be partitioned into the smallest units. Too many and too small CUs on frame boundaries may result in more coding complexity and larger bitstreams. In this embodiment, it is noted that the remaining depth hierarchy may be determined in any manner and may not be forced in a predefined manner, as long as the deepest CU is larger on one side than the other (i.e., resulting from binary tree partitioning). Rather, the partitioning may be done by RD optimization, with corresponding control information that allows the decoder to restore the same partitioning. The partitioning may also be partially forced and partially available for RD optimization.

[0081] In other words, the processing circuitry may be configured to partition the boundary portion also by quadtree partitioning. Thus, there may be a predefined mandatory QT or BT partition for each depth, either defined by the standard or configurable by control information included in the bitstream.

[0082] For example, the forced BT partitioning is performed only after a certain number of forced QT levels. This may be achieved in that the processing circuitry is configured to partition the boundary portion by quad-tree partitioning at the N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchical levels of partitioning. The processing circuitry is further configured to partition the boundary portion by binary tree partitioning at the remaining hierarchical levels of partitioning. Referring to FIG. 8, there is no QT partitioning performed on the left side.

[0083] Thus, the control information in the bitstream may include a QT / BT indicator that specifies whether QT is applied and, if so, how many levels are QT partitioned. Correspondingly, such a QT / BT indicator may take multiple values, such as values ​​0 and N corresponding to no QT partitioning (as shown on the left in FIG. 8). Such signaling is shorter than the signaling for each CU at each level, regardless of whether QT partitioning is applied, BT partitioning is applied, or no partitioning is applied. However, the present disclosure may use any kind of signaling.

[0084] Alternatively, or in combination with control signaling, the level of forced QT can be adaptively selected depending on the time ID or slice type. This approach provides the advantage that no explicit signaling is required (or a smaller amount of explicit signaling is required). Furthermore, the forced partitioning may be adaptively set in relation to at least the type of prediction.

[0085] HEVC specifies a temporal identifier (ID) that indicates a level in the hierarchical temporal prediction structure (dependency between inter-predicted frames). The type of slices can be, for example, intra-predicted slices, inter-predicted slices that apply prediction based on only one reference picture, inter-predicted slices that apply bidirectional prediction, etc.

[0086] For example, if the temporal identifier is 0 (meaning 1 picture) or if the slice type is an intra-predicted slice, the partitioning needs to be more accurate than for higher temporal IDs (inter-predicted pictures) or for P / B slices (inter-predicted slices).

[0087] The above parameters, time ID or slice type, are merely exemplary. Generally, further and / or different coding parameters may be applied. Furthermore, instead of slice type, prediction modes of CUs in neighboring CTUs may be used to determine the partitioning of boundary parts based on CTUs (CTUs with incomplete boundaries).

[0088] On the other hand, signaling the partitioning may provide more freedom to choose an appropriate partition so that the overall coding efficiency is increased. For example, the partitioning can be adjusted by optimizing a cost function (e.g., by RD optimization) and signaled in the bitstream to indicate how many levels of forced QT are used for the decoder (i.e., signaling whether QT is used at all or signaling N as described above).

[0089] Alternatively, boundary (incomplete) CTUs are first partitioned by RD optimization up to the penultimate hierarchical level (depth) or up to the signaled hierarchical level M. Then, the remaining deepest level blocks at the frame boundaries use forced BT partitioning or a combination of forced QT and forced BT partitioning.

[0090] FIG. 9 shows three examples of at least partially enforced boundary divisions.

[0091] When a CTU / CU is at the bottom boundary, horizontal forced BT partitioning may be used without iterative RD optimization (by splitting the vertical boundary) and without signaling a partitioning flag. The forced horizontal BT partitioning is terminated when the bottom right sample of the leaf node is within the bottom boundary of the slice / picture. After the forced horizontal BT partitioning, the forced partitioned CU from the boundary CTU may be further split based on the RD cost at the encoder, and a further partitioning flag may be signaled from the encoder to the decoder. Part (a) of FIG. 9 shows an example of a bottom boundary of 128 x 56 samples partitioned by forced horizontal BT (i.e., splitting the vertical boundary).

[0092] Similarly, for the CTU / CU at the right boundary, vertical forced BT partitioning is used without checking RD cost and without signaling split flags, possibly repeatedly, until the bottom right sample of the leaf node falls within the right boundary of the slice / picture. After forced vertical BT partitioning (for horizontal boundary), the forcedly partitioned CU from the boundary CTU may be further split by RD optimization at the encoder, and further split flags are signaled from the encoder to the decoder. Part (b) of Figure 8 shows an example of a right boundary with a length of 56 x 128 samples partitioned by forced horizontal BT.

[0093] According to another example, for a CTU / CU at the bottom right corner boundary, first, the forced hierarchical QT partitioning is used repeatedly without any signaling. When the bottom right sample of the current CU is at the bottom or right boundary, further forced horizontal or vertical BT partitioning is performed repeatedly until the bottom right sample of the leaf node is within the slice / picture boundary. Otherwise, when the bottom right sample of the current CU is further at the corner boundary, further forced QT partitioning is used until the bottom right sample of the leaf node is within the slice / picture frame. The forced partitioned CU obtained by partitioning from the boundary CTU may be further partitioned by RD optimization in the encoder, and further partition flags may be signaled from the encoder to the decoder. Part (c) of FIG. 9 shows an example of a bottom right corner boundary of 56 x 56 samples partitioned by forced QT and BT.

[0094] In particular, part (c) of FIG. 9 shows an incomplete corner boundary CTU of 56 x 56 samples. The first depth level of 64 x 64 samples after QT division is not within the boundary. The second depth level of 32 x 32 samples after QT is within the boundary but leaves a horizontal corner (lower left CU of 32 x 24 samples), a vertical corner (upper right CU of 24 x 32 samples), and another corner (lower right CU of 24 x 24 samples) to be further divided.

[0095] In this example, the horizontally bounding CUs and the vertically bounding CUs are further bitree partitioned such that the deepest level units have the longer side along the picture / slice boundary. The corner parts may be further partitioned in the same way as described above for the higher level corner parts, i.e., by quadtree partitioning.

[0096] In other words, during operation, the processing circuitry may partition the boundary portion horizontally and vertically by binary tree partitioning, especially when the boundary portion extends along both picture / slice boundaries, vertical boundaries and horizontal boundaries, and includes corner portions.

[0097] For the corner case, the processing circuitry operates by partitioning the corner portions of the boundary at each hierarchical level using a quadtree partition. For example, in part (c) of FIG. 9, for a corner with a size of 56 samples on both sides, CU 960 at the corner boundary, four levels of forced QT are required (a QT partition of the 128 x 128 CTU resulting in four 64 x 64 CUs with the top-left CU containing a 56 x 56 corner, a QT partition of the top-left CU of the 64 x 64 CU resulting in four 32 x 32 CUs with the bottom-right CU containing a 24 x 24 corner, a QT partition of the bottom-right CU of the 32 x 32 resulting in four 16 x 16 CUs with the bottom-right of those 16 x 16 CUs containing an 8 x 8 corner, and a final step of QT partitioning into the smallest size partitions where the top-left partition is in the picture while the other partitions are not).

[0098] On the other hand, for the bottom border CU 950, two levels of forced QT followed by two levels of forced BT are required (QT split of the 128x128 CTU, QT split of the top-left 64x64 CU, bipartition of the 32x32 CU with its 32x24 border part into two 32x16 CUs on the bottom border, and a final step of BT split of the lower 32x16 CU into two 32x8 CUs with only the top CU inside the picture border).

[0099] For corner CUs 960, the level N of the set QT is not taken into account and the corner CU is forced to be split by QT at all hierarchical levels. In particular, in the case of corners, if the bottom right sample of a leaf node (incomplete CU, i.e., partially inside the boundary part) is outside both the right and bottom boundaries, this current incomplete CU is recognized as bCorner (FIG. 10, step 1050, "Yes"). If bCorner is applicable to the current CU, the current CU is further split using forced QT. The partitioning ends when the resulting leaf node is completely inside the boundary part or if the corner part falls into the bottom or right boundary case (bBottom, bRight, i.e., the bottom right boundary of the current CU is outside the boundary part, i.e., only either the bottom or right boundary of the picture or slice).

[0100] In this example, when N=0 or 1 or 2, the forced QT splitting scheme is the same, since CU 960 always needs forced QT and CU 950 is split by QT at the first two levels anyway. This is because the CU at depth 0 (CTU), the CU at depth 1 (64 x 64 samples) are still corner CUs. Only after splitting the 64 x 64 CUs, there are 4 CUs at depth 2, one of which is not a border CU at all (top left), two are bottom border CUs and right border CUs, and only one is a corner CU.

[0101] If N=3, then one more forced QT is required, and then the forced splitting scheme is changed for CU 950 such that the bottom left 32 x 32 CU is further QT split instead of BT split. The splitting of CU 960 remains the same.

[0102] According to the present disclosure, a combination of forced QT and forced BT may be used as already briefly discussed above. Particularly for some complex texture or motion on the boundary, detailed segmentation may be beneficial for image quality. Therefore, a combined forced segmentation by QT and BT is also provided. In particular, in one combined implementation of forced QT and forced BT, forced BT segmentation is applied only after a certain number N of forced QT levels. Under other settings, if QT segmentation is allowed after BT segmentation, forced QT segmentation may be performed after forced BT.

[0103] In the combined implementation of forced BT and forced QT, there are several ways to define the number of levels of forced QT. When predefined levels of forced QT are used, first, all boundary CTUs are forced to be partitioned by the same QTBT structure, and further partitioning may be done based on RD optimization. The right side of Figure 8 shows an example of partitioning of one level of QT and three levels of BT of the lower boundary of 128 x 56 samples in an incomplete 128 x 128 CTU.

[0104] The level of enforced QT may be adaptively selected by encoder parameters such as the temporal ID or the type of slice. Certain slice types and lower temporal IDs may require more detailed partitioning. Furthermore, the level of enforced QT may be adjusted by rate-distortion optimization and signaled in the bitstream to indicate how many levels of enforced QT are used. Signaling of the number of levels at the CTU level may be used to indicate how many levels of enforced QT were used for the CTU. In addition, signaling of the number of levels in the slice header may be used to indicate how many levels of enforced QT were used for the current slice. Signaling of the number of levels in the SPS may be used to indicate how many levels of enforced QT were used for the coded video sequence. Signaling of the number of levels in the PPS may be used to indicate how many levels of enforced QT were used for the coded picture.

[0105] The number of levels of enforced QT may be determined based on any other approach. For example, the entropy of the original samples in the slice / frame / border region may be used. The entropy is calculated by summing over i the terms p_i * log2(p_i), where p_i is the probability of the intensity value I of the samples in the particular region. The symbol "*" means multiplication and "log2" means logarithm to the base 2. The higher the entropy, the more likely it is that structures are present in the slice / frame / border region, and therefore a smaller CU may be appropriate. However, these are only examples, and in principle the invention is not limited to any particular way in which the number of levels of enforced QT is determined. Any other measure, such as the variance of the presence of edges, may be used.

[0106] In operation, the processing circuitry segments the boundary portion until the deepest level coding unit remains within the boundary portion.

[0107] According to the present disclosure, a method for dividing an image portion into coding units is provided. The method includes subdividing the image portion into an integer number of coding tree units having a predetermined size and boundary portions having a size less than the predetermined size in at least a first direction, either horizontally or vertically. A hierarchical partitioning of the boundary portions into coding units is then performed, including a deepest coding unit having a different size in each horizontal and vertical direction, the deepest coding unit being smaller in the first direction.

[0108] In one implementation, the partitioning of the boundary portion further includes the steps of starting from a current partitioning level, which is the shallowest partitioning level; dividing the boundary portion into coding units of the current partitioning level in order, as long as the coding units of the current partitioning level do not reach the boundary portion; and, as soon as the coding units of the current partitioning level do not fit into the boundary portion, increasing the partitioning level and further partitioning the coding units, in which quad-tree partitioning is used for the N shallowest partitioning levels and binary-tree partitioning is used in other cases.

[0109] Sequential partitioning means that partitioning is performed from the shallowest level to the deepest level. In particular, a forced partitioning is performed at each hierarchical level for boundary coding units (boundary CUs), i.e., such CUs contain samples inside (within the boundary part) of the picture / slice boundary and parts outside the picture / slice boundary. Here, the term CU refers to a CU that has a size resulting from partitioning, but contains fewer samples than given by its size, i.e., is incomplete because it is partly outside the picture / slice in which it is partitioned.

[0110] Therefore, a forced split is performed to partition the boundary portion roughly from the largest CU to the smallest CU so that the entire boundary portion is partitioned. As already discussed above, partitioning information is not required for the forced split, and the partitioning information may be predefined. If there are several levels of QT to be forced, N may be signaled or predefined. Predefined N means that both the encoder and the decoder have knowledge of such N, which may be defined by a standard, for example.

[0111] However, after performing the forced split, the CUs of the forced split boundary part may be further partitioned. This is performed based on rate-distortion optimization and may be further signaled in the bitstream accordingly. In particular, for CUs in the boundary part that are larger than the minimum CU size, a split flag or multiple split flags may be signaled indicating whether and how such CUs are further partitioned.

[0112] It is noted that the partitioning information included in the bitstream may include a QT split flag and / or a BT split flag. QT and BT splitting may also be mixed as long as which of the two types of splitting is used is signaled or derivable. At the decoder side, after the forced splitting, further splitting is then performed based on the decoded partitioning information such as qt_split_cu_flag and bt_split_cu_flag. Figure 10 shows a detailed example implementation of the boundary partitioning and, in particular, the implementation of forced BT and one combination of forced QT and forced BT. The variables adopted in the figure have the following meanings: - bBottom indicates whether the bottom-right sample of the current CTU or CU is outside the bottom boundary. - bRight indicates whether the bottom right sample of the current CTU or CU is outside the right boundary. - bCorner indicates whether the bottom right sample of the current CTU or CU is outside the bottom right corner boundary. - FQT_LV indicates the level of forced QT before the forced BT is performed (corresponding to N above). - uiDepth indicates the hierarchical level at which the current partition is being partitioned.

[0113] As mentioned above, the level of enforced QT may be predefined and adaptively selected based on encoder parameters such as slice type / temporal ID or based on RD cost.

[0114] In particular, in step 1010, partitioning of a boundary CTU (level 0) begins. If in step 1020 the CTU is a CTU on the bottom or right boundary (but not a corner boundary) and if uiDepth is less than or equal to FQT_LV-1, the CTU is partitioned in step 1040 by forced BT. Otherwise, forced QT partitioning step 1060 is applied. Partitioning continues by further partitioning boundary CUs. For example, in the example of FIG. 8(b), FQT_LV of the CTU (level 1) is equal to 1. Since uiDepth=0 is equal to FQT_LV-1=0 in step 1030, the incomplete CTU is forced to be partitioned by QT in step 1060, resulting in two CUs (top-left and top-right) that need to be further partitioned. Each of these CUs is further partitioned iteratively. The top-left CU (level 1) is still incomplete and is therefore a boundary CU. As the bottom boundary CU in step 1020, since uiDepth=1 is greater than FQT_LV-1=0 in step 1030, a forced BT is applied to the top-left CU in step 1040. The top-right CU is partitioned in the same way. In step 1040, the boundary (incomplete) CU (64 x 24) is further partitioned in level 2 by BT. The remaining 64 x 8 CU in level 3 reaches the picture boundary, thus resulting in "No" in steps 1020 and 1050. The partition of size 8 as the smallest partition is not further partitioned in step 1070, but rather results in "yes" in step 1080, which stops the partitioning in step 1090. The second 64 x 24 CU is partitioned in the same way.

[0115] On the other hand, the 56x56 CTU in part (c) of Figure 9 is a corner CTU. In step 1020 (level 0), the CTU proceeds to step 1050 and is partitioned by QT as a corner CU in step 1060. In level 1, the top-left 56x56 CU is treated in the same manner, resulting in three border CUs (bottom-left, top-right, and corner). Assuming FQT_LV=0, i.e., no forced QT split for bBottom and bRight CUs, the bottom-left and top-right CUs are split by forced BT in step 1040 (after "yes" in step 1020 and "no" in step 1030). The corner 24x24 CU (level 2) passes from step 1020 to step 1050 and is partitioned by QT in step 1060, again resulting in three bordering CUs with sizes 16x8, 8x8, and 8x16, which do not need to be further divided, passing through "no" step 1020, "no" step 1050, step 1070, stopping condition 1080 and end 1090. The stopping condition may be, for example, a minimum size equal to 8 in this example.

[0116] Below, Table 2 shows an example syntax that allows signaling the QT split flag for normal (non-border) CUs. In addition, the BT split flag may be signaled for CUs other than border CUs. The coding_quadtree syntax is iterative, i.e., the partitioning is hierarchical. Partitioning is performed if the following conditions are true: x0 + cuWidth <= pic_width_in_luma_samples and y0 + cuHeight <= pic_height_in_luma_samples

[0117] In other words, as long as the bottom right corner of the current CU (i.e., the CU currently being processed by the partition) is within the picture, the CTU / CU is partitioned by a quadtree or a binary tree. It is noted that this syntax is only exemplary, and the present disclosure may also be applied to other cases, such as using only QT, or only BT, or a selectable further partition type for CTUs that are entirely within the picture / slice. The values ​​cuMinBTSiz, cuMaxBTSize, and cuMaxBTDepth adopted in the syntax are predefined values.

[0118] In the above condition, x0 and y0 specify the location of the top-left corner sample of the current CU, cuWidth and cuHeight specify the CU size in the horizontal and vertical directions, and parameters pic_width_in_luma_samples and pic_width_in_luma_samples specify the picture size in the horizontal and vertical directions. It is noted that in this example, the entire picture is partitioned. However, the same approach is applicable to parts of a picture that can be decoded independently, such as slices, tiles, or any other part of a picture that has a boundary.

[0119] Similar to the HEVC standard, when qt_split_cu_flag[x0][y0] is not present (boundary case), the following applies (at the encoder and decoder): - If cuWidth (CU width of the current level) is greater than MinQTSize (minimum QT size, here 8) and cuHeight (CU height of the current level) is greater than MinQTSize, the value of qt_split_cu_flag[x0][y0] is inferred to be equal to 1. In other words, the border is forced to be split. - Otherwise (log2CbSize is equal to MinCbLog2SizeY), the value of qt_split_cu_flag[x0][y0] is inferred to be equal to 0. In other words, if the current CU has the minimum allowed size MinCbLog2SizeY, the decoder assumes that the CU will not be split any further.

[0120] This is similar to the case in Table 1 above.

[0121] [Table 2A] [Table 2B]

[0122] As can be seen, bt_split_cu_flag may be signaled. This flag is signaled when BT split of a CTU is configured for a CTU / CU that is not on a boundary (see condition if(! (bBBoundary || bRBoundary))). In the boundary case, a forced split does not require an additional flag since the value of the flag is derived. In particular, if bt_split_cu_flag[x0][y0] is not present (boundary case), the following applies: - if bBBoundary (meaning the current CU / CTU is at the bottom boundary) is true and at the same time cuWidth is larger than MinBTSize or cuHeight is larger than MinBTSize (the current CU / CTU has no minimum size and therefore could in principle be further split), the value of bt_split_cu_flag[x0][y0] is inferred to be equal to 1 (meaning a further bisection split is performed, i.e. a binary tree split that would result in the longer side of the split CU being parallel / overlapping the bottom, i.e. horizontal boundary). - bRBoudary is true (meaning the current CU / CTU is on the right border) while cuWidth is greater than MinBTSize or cuHeight is greater than MinBTSize and the value of bt_split_cu_flag[x0][y0] is inferred to be equal to 2 (meaning a further bisection split is performed, i.e. a binary tree split that will result in the longer side of the split CU being parallel / overlapping the right, i.e. vertical border). - Otherwise, the value of bt_split_cu_flag[x0][y0] is inferred to be equal to 0, meaning that no further bisection will be performed.

[0123] Being on the boundary means that there are no CUs between the current CU and the boundary that are below the hierarchical level of the current partition, in other words, the current CU belongs to the boundary part at the current hierarchical level (corresponding to bBBoundary being true or bRBoundary being true). If the current CU is not a boundary CU, then BT partitioning is used if QT is not forced (cqtDepth > FQT_LV-1) and the BT flag is included in the bitstream.

[0124] Exemplary values ​​for an implementation of JEM are cuMinBTSize for Inter frames having a value of 4, cuMAXBTSize for Inter frames being 128, and cuMAXBTDepth being 3. However, this disclosure is not limited to these sizes and is applicable to any selected values.

[0125] It is noted that the above syntax is only exemplary and assumes knowledge of a variable FQT_LV, which indicates the number of forced QT division levels. If the value of FQT_LV is zero, then there are no forced QT levels and the entire division of the forced boundaries is performed by binary tree division. The syntax and semantics may be implemented differently without departing from this disclosure.

[0126] Another solution based on the JEM software (JEM5 and 6) could be to specify the following semantics: When qt_split_cu_flag[x0][y0] is not present (boundary case), the value of qt_split_cu_flag[x0][y0] is inferred to be equal to 1, which means further QT split in case of corner CUs or forced QT (if cqtDepth <= FQT_LV-1).

[0127] If bt_split_cu_flag[x0][y0] is not present (boundary case), the following applies: - If bBBoundary is true, the value of bt_split_cu_flag[x0][y0] is inferred to be equal to 1, meaning that the bottom border CU is split vertically by a bisection, i.e. to form two horizontal CUs (i.e. a CU that is longer horizontally than vertically). - If bRBoundary is true, the value of bt_split_cu_flag[x0][y0] is inferred to be equal to 2, meaning that the right-bordering CU is split horizontally by a bisection, i.e., to form two vertical CUs (i.e., a CU that is longer vertically than horizontally).

[0128] The above example shows a forced split using a parameter that defines the number of levels of forced QT before BT splitting is applied. Semantic rules define the split that is applied when the split flag is not present, i.e., at the boundary. However, this disclosure is not limited to such an approach.

[0129] Alternatively, first, the (incomplete) CTUs at the boundaries can be partitioned by RD optimization, and then the remaining blocks at the frame boundaries can use forced BT partitioning or a combination of forced QT and forced BT partitioning. Partitioning by RB optimization implies that a split flag is signaled in the bitstream for the corresponding CU. Partitioning by RD may be specified as QT (i.e., in the standard or by signaling common to a sequence or picture of pictures) or BT, so that the choice between BT and QT does not have to be signaled for each CU. Meanwhile, the split flag may be extended to indicate not only whether a given CU is split or not, but also whether the CU is split by QT or by BT.

[0130] At the decoder side, a forced partitioning is performed for the boundary CTU / CU. For forced BT partitioning, forced horizontal BT (vertical division to form horizontal partition) is used for the CTU / CU at the bottom boundary until the bottom right sample of the leaf node falls within the bottom boundary of the slice / picture. Forced vertical BT is used for the CTU / CU at the right boundary until the bottom right sample of the leaf node falls within the right boundary of the slice / picture. The bottom right corner CTU / CU is partitioned using forced QT until the bottom right sample of the leaf node falls within the slice / picture boundary or the leaf node falls in the right / bottom boundary case. As already explained above with reference to the syntax, further partitioning is performed based on the decoded partition flag, which may be a QT or BT partition flag depending on whether there is a number of levels of forced QT partitioning defined. The elements of the syntax are generated by the encoder and processed by the decoder as explained above.

[0131] For a combination of forced QT and forced BT, the number of forced QT levels for a boundary CTU (corresponding to the parameter FQT_LV) may be adaptively selected based on the temporal ID / slice type or decoded from the bitstream. The decoded forced QT levels may be extracted from the slice header, SPS, or PPS. A boundary CTU / CU is subjected to forced BT partitioning only after forced QT partitioning. The forced partitioning is terminated when the bottom right sample of the leaf node is within the boundary. Further partitioning is performed based on the decoded partitioning flags.

[0132] In other words, according to the first implementation, in the decoder, if the CTU is on a slice / picture boundary, the binary tree partitioning is performed without a binary tree partitioning indicator (flag) and / or a binary tree partitioning type indicator (indication of whether BT or QT should be performed). The binary partitioning is terminated when the bottom right sample of the leaf node is within the slice / picture boundary (i.e., does not cross the picture boundary).

[0133] Therefore, instead of forced QT boundary partitioning, forced BT for the CTU of the boundary is introduced. A forced horizontal BT for the bottom boundary and a forced vertical BT for the right boundary are implemented. For the bottom right corner boundary, forced QT is used until the current CU no longer needs to be forced partitioned (because the minimum size has been reached) or until the current CU is at the bottom or right boundary.

[0134] In another implementation of the present disclosure, a combination of forced QT and forced BT partitioning is also possible. Thus, forced BT partitioning is performed only after a certain number of forced QT levels. For the combination of forced QT and forced BT partitioning, the level of forced QT can be adaptively selected by time ID or slice type. Alternatively, the level of forced QT can be adjusted by cost and included in the bitstream to indicate how many levels of forced QT are used for the decoder.

[0135] First, even more flexibility is achieved when the boundary CTUs are partitioned by RD optimization and only the remaining blocks at the frame boundaries (i.e., incomplete CUs that are partially in the picture and partially outside the picture) can use the proposed forced BT partitioning or a combination of forced QT and forced BT partitioning as described above.

[0136] The partitioning described so far may be employed in still image or video coding, in particular in an encoder and a decoder, as shown in Figures 1 and 2 as partitioning units 110 and 250, respectively.

[0137] Therefore, the present disclosure also provides an apparatus for encoding images of a video sequence, the apparatus comprising: an apparatus (splitting unit) 110 for splitting an image portion according to any of the above-mentioned examples into coding units; image coding units 120-145, 160-195 configured to code the coding units; and a bitstream formation unit 150 configured to generate a bitstream comprising the coded coding units and partitioning information indicating how the coding tree units are partitioned.

[0138] The encoder may further include a rate-distortion optimization unit (not shown in FIG. 1) configured to determine hierarchical partitioning of boundary portions by rate-distortion optimization for any of the partitioning levels apart from the deepest level. The RD optimization unit may perform encoding using different encoder settings in relation to the partitioning and select the setting that results in the smallest cost function. The cost function may be the smallest rate for a given level of distortion or similarity between the prediction and the original block to be encoded, etc.

[0139] Further provided is an apparatus for decoding images of a video sequence, comprising an apparatus 250 for determining a division of an image portion to be decoded into coding units as described above, a bitstream parser 250 for parsing the coded coding units and a bitstream including division information based on the determined division of the image portions, indicating how the coding tree units are divided, and image decoding units 225-240, 260-295 for decoding the coded coding units.

[0140] It is noted that splitting and parsing are performed iteratively together, where parsing includes separating necessary control information from the bitstream for the current CU, and splitting uses parsed information and / or semantic knowledge to derive the location, size, and data of the CU. In particular, the parser may send a split flag, based on which the splitter either further splits the current CU or maintains the size of the CU. Depending on the split flag, the parser also either further parses the partition information for the CU at further depth, or parses the partition information for other CUs at the same depth, or parses the decoded data for the CU. Furthermore, the splitter is configured to partition the boundary portion as described above, even without further bitstream signaling, according to the mandatory partitioning rules.

[0141] In other words, the encoder and / or decoder may both adopt the division of the boundary part into coding units as a predefined division. Then, the partitioning information in the bitstream is not related to the division of the boundary part at all. This is the case when the boundary is divided in a predefined manner, which may further include deriving the division based on other encoder settings such as picture type / time ID / slice type, etc.

[0142] Alternatively, the encoder and / or decoder may be further configured to partition the boundary portion by quad-tree partitioning at the N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchical levels of partitioning, and partition the boundary portion by binary tree partitioning at the remaining hierarchical levels of partitioning, and the partitioning information includes an indication of N. The partitioning information may further include a partition flag to indicate whether and / or with what type of partitioning the CTU / CU of the boundary portion should be partitioned (which may include binary tree or quad tree and / or other options). The partition flag may be signaled for all partitions (CUs) or only for partitions that are not at boundaries at their hierarchical levels.

[0143] The encoder and / or decoder may be implemented (embedded) in a chip or by software, but may also be implemented in multiple hardware and software components.

[0144] In the above, mainly two types of partitioning are considered: quad-tree partitioning and binary-tree partitioning. However, the cost-based or pre-defined boundary constrained partitioning approach provided by the present disclosure is suitable not only for constrained QT or constrained BT but also for other constrained partitioning structures.

[0145] Above, we have mainly described the case where the deepest CU in the forced boundary division is a rectangular and not a square CU, ie a CU with one side longer than the other.

[0146] In general, however, the boundary portion may be partitioned by binary tree partitioning at least at one of the partitioning levels where the boundary coding units do not remain completely within the boundary portion. In other words, according to the present disclosure, an apparatus for partitioning an image portion into coding units may include a processing circuit that performs, during operation, sub-partitioning the image portion into an integer number of coding tree units having a predetermined size and a boundary portion having a size less than the predetermined size in at least a first direction, which is either horizontal or vertical, and hierarchically partitioning the boundary portion into coding units including coding units having different sizes in respective horizontal and vertical directions, where the coding units are partially within the boundary portion and partially outside the image portion. In particular, binary tree partitioning may be used for any boundary CU.

[0147] It is noted that the boundary portion shown in Fig. 7 includes several CTUs. The above exemplary embodiments and implementations have been described with respect to partitioning one CTU. This is because it may be advantageous and lead to better results in terms of coding efficiency and / or complexity even if only one of the (incomplete) CTUs in the boundary portion is partitioned as described above. However, two or more CTUs of the boundary portion may be partitioned as described above. The partitioning may be performed one CTU at a time in sequence. In some implementations, it may be beneficial to perform partitioning in parallel for different CTUs. Parallel processing may reduce the encoding / decoding delay if employed by the encoder / decoder, respectively.

[0148] In other words, it may be advantageous to apply the above-mentioned division to all CTUs in the boundary portion. While the disclosure is not so limited, the division may be applied to one or more boundary CTUs. The remaining CTUs may be divided by any other division, such as, for example, only QT.

[0149] Corresponding methods for partitioning are also provided. Corresponding encoding and decoding apparatus and methods employing such partitioning are further provided.

[0150] The present disclosure may be implemented in an apparatus. Such an apparatus may be a combination of software and hardware. For example, the intra-prediction and deblocking filter may be executed by a general-purpose processor, or a digital signal processor (DSP), or a chip such as a field programmable gate array (FPGA). However, the present invention is not limited to implementation on programmable hardware. It may be implemented in an application specific integrated circuit (ASIC) or by a combination of the above hardware components.

[0151] The determination of the intra prediction and deblocking filter may also be implemented by program instructions stored in a computer-readable medium, which, when executed by a computer, performs the steps of the method as described above. The computer-readable medium can be any medium on which a program is stored, such as, for example, a DVD, a CD, a USB (flash) drive, a hard disk, a server storage available via a network, etc.

[0152] The encoder and / or decoder may be implemented in various devices including a television set, a set-top box, a PC, a tablet, a smartphone, etc. The encoder and / or decoder may be software, an app that implements the steps of the method.

[0153] In summary, the present disclosure relates to partitioning an image or image slice. In particular, an image or image slice is partitioned into an integer number of coding tree units in both vertical and horizontal directions. The remaining boundary portion is then forced to be at least partially partitioned such that the deepest partition level is the result of a bipartite partition, that is, one side contains longer coding units than the other side. The longer side is parallel to or overlaps the image or image slice boundary. [Explanation of symbols]

[0154] 100 Encoder 110 Split Unit 120 Image Encoding Unit 125 Reconstructor 130 Conversion Unit 135 Reverse conversion unit 140 quantization units 145 Inverse Quantization Unit 150 entropy coding units 160 Loop Filtering Unit 170 Frame Buffer 180 Mode Selection Unit 190 intra prediction units 195 Inter Prediction Units 200 Video Decoder 225 Reconstructor 230 Reverse conversion unit 240 Inverse Quantization Unit 250 Bitstream Analysis, Entropy Decoding, and Splitting Unit 260 Loop Filtering Unit 270 Reference Picture Buffer 290 Intra Prediction Units 295 Image Decoding Unit 600 equipment 610 Functional Unit 620 Functional Unit 630 Functional Unit 700 bottom picture border 710 Boundary part 720 parts Vertical size of 770V CTU Horizontal size of 770H CTU 950 lower boundary CU 960 CU of the corner boundary

Claims

1. 1. An apparatus for dividing an image portion into coding units, the apparatus comprising, during operation: subdividing the image portion into an integer number of coding tree units having a predetermined size and a boundary portion having a size in at least a first direction, either horizontally or vertically, that is less than the predetermined size; hierarchically partitioning the boundary portion into coding units having different sizes in respective horizontal and vertical directions, including a deepest coding unit, the deepest coding unit being smaller in the first direction. The apparatus includes processing circuitry for performing the steps of:

2. The apparatus of claim 1 , wherein the processing circuitry is configured to partition the boundary portion by binary tree partitioning at least at a deepest level of partitioning.

3. 3. The apparatus of claim 1 or 2, wherein the processing circuitry is configured to partition the boundary portion also by quadtree partitioning.

4. The processing circuitry comprises: partitioning the boundary portion by quadtree partitioning at N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchical levels of partitioning; and The apparatus of claim 3, configured to perform partitioning of the boundary portions by binary tree partitioning at remaining hierarchical levels of partitioning.

5. 5. An apparatus according to claim 1, wherein the processing circuitry, in operation, segments the boundary portion until the deepest level coding unit remains entirely within the boundary portion.

6. 6. Apparatus according to any one of claims 1 to 5, wherein the processing circuitry, during operation, partitions the boundary portion horizontally and vertically by a binary tree partition.

7. 7. Apparatus according to any one of claims 1 to 6, wherein the processing circuitry, during operation, partitions corner portions of the boundary portions at each hierarchical level by quadtree decomposition.

8. 3. The apparatus of claim 1 or 2, wherein the processing circuitry is configured to partition the boundary portions by binary tree partitioning at all levels of partitioning.

9. 1. An apparatus for encoding images of a video sequence, comprising: The device for dividing an image portion into coding units according to any one of claims 1 to 8, an image coding unit configured to code the coding unit; and a bitstream forming unit configured to generate a bitstream including the encoded coding units and partition information indicating how the coding tree units are partitioned.

10. 10. The encoder of claim 9, further comprising a rate-distortion optimization unit configured to determine a hierarchical partitioning of the boundary portions by rate-distortion optimization for any level of partitioning apart from the deepest level.

11. 1. An apparatus for decoding images of a video sequence, comprising: The device for determining a division into coding units of an image portion to be decoded according to any one of claims 1 to 8, a bitstream parser for parsing a bitstream including partitioning information based on the determined partitioning of the image portion, the bitstream indicating how the coded coding units and the coding tree units are partitioned; and an image decoding unit for decoding the encoded coding unit.

12. 12. The apparatus according to claim 9, wherein the division of the boundary portion into coding units is a predefined division, and the partitioning information is not related to the division of the boundary portion.

13. the processing circuitry is configured to partition the boundary portion at N shallowest levels by quadtree partitioning, where N is an integer greater than or equal to zero and less than a number of hierarchical levels of partitioning, and to partition the boundary portion at remaining hierarchical levels of partitioning by binary tree partitioning; 12. The apparatus of claim 9, wherein the partitioning information comprises N or N is derived based on a type of prediction applied to decode the image being processed.

14. 1. A method for dividing an image portion into coding units, comprising the steps of: subdividing the image portion into an integer number of coding tree units having a predetermined size and a boundary portion having a size in at least a first direction, either horizontally or vertically, less than the predetermined size; hierarchically partitioning the boundary portion into coding units having different sizes in respective horizontal and vertical directions, including a deepest coding unit, the deepest coding unit being smaller in the first direction.

15. The step of dividing the boundary portion includes: starting at a current partitioning level, which is the shallowest partitioning level; dividing said boundary portion into coding units of said current partitioning level in turn until a coding unit of said current partitioning level is entirely contained within said boundary portion; 15. The method of claim 14, further comprising: increasing the partitioning level and further partitioning the coding unit as soon as the coding unit of the current partitioning level does not fit into the boundary portion, wherein quadtree partitioning is used for the N shallowest partitioning levels and binary tree partitioning is used otherwise, where N is an integer greater than or equal to zero and less than a number of hierarchical levels of partitioning.

16. 1. A method for encoding images of a video sequence, comprising the steps of: Dividing the image portion according to claim 14 or 15 into coding units; encoding the coding unit; generating a bitstream including the encoded coding units and partition information indicating how the coding tree units are partitioned.

17. 1. A method for decoding images of a video sequence, comprising the steps of: determining a division of the image portion to be decoded into coding units according to claim 14 or 15; analyzing the bitstream including partitioning information based on the determined division of the image portion, the partitioning information indicating how the coded coding units and the coding tree units are partitioned; and decoding the encoded coding unit.

18. A computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 14 or 17.

Images