Multi-type tree depth extension for picture boundary processing

Abstract

To provide an apparatus and a method for dividing an image into coding units.SOLUTION: An image is divided into hierarchically partitioned coding tree units (CTUs). Hierarchical partitioning includes multi-type partitioning, such as binary tree or quad tree partitioning. For CTUs that are entirely within the image and CTUs on the boundary, respective multi-type partition depths are selected. The present disclosure provides the flexibility of multi-type partitioning in the boundary portion of the image.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] TECHNICAL FIELD This disclosure relates to the field of video processing, and in particular to the topic commonly referred to as hybrid video coding and compression. [Background technology]

[0002] The Versatile Video Coding (VVC) next-generation standard is the latest collaborative video project between the standards organizations ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG), which work together in a partnership known as the Joint Video Exploration Team (JVET).

[0003] Current block-based hybrid video codecs use predictive coding. Pictures of a video sequence are subdivided into blocks of pixels, and then these blocks are coded. Instead of coding the block pixel by pixel, the entire block is predicted using already encoded pixels that are spatially or temporally neighboring to the block. The encoder further processes only the difference between the block and its prediction. Further processing usually includes transforming the block pixels into coefficients in a transform domain. The coefficients can then be further compressed (e.g., by quantization) and further compacted (e.g., by entropy coding) to form a bitstream. The bitstream may further include any signaling information that enables a decoder to decode the encoded video. For example, the signaling may include settings regarding encoder settings such as the size of the input picture, the frame rate, a quantization step indication, or the prediction applied to the blocks of the picture.

[0004] The difference between a block and its prediction is known as the residual of the block. More specifically, each pixel of the block has a residual that is the difference between the intensity level of that pixel and its predicted intensity level. The intensity level of a pixel is called the pixel value or the value of the pixel. The residuals of all pixels of a block are collectively called the residual of the block. In other words, a block has a residual that is a set or matrix of the residuals of all pixels of the block. The residual is then transformed, quantized, and coded along with the signaling information. The coding may include various forms of fixed-length and variable-length coding, including arithmetic coding or other entropy coding types.

[0005] In block-based hybrid video coding, each picture is partitioned into blocks of samples, and multiple blocks within a picture are aggregated to form a slice as an independently decodable entity. A block to which prediction and / or transformation is applied is called a coding unit (CU) or coding block (CB). Coding units may have different sizes.

[0006] For example, in High Efficiency Video Coding (HEVC, also known as H.265), a video frame is subdivided into coding tree units (CTUs, also called coding tree blocks, CTBs). The CTBs are disjoint square blocks of the same size, e.g., 64x64 samples. Each CTB serves as the root of a block partitioning quadtree structure, the coding tree. The CTBs may be further subdivided into coding blocks along the coding tree structure. For the coding blocks, a prediction type is determined. The coding blocks may be further divided into smaller transform blocks, where transformation and quantization are applied.

[0007] More details on partitioning in HEVC can be found in V. Sze et al. (Ed.), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014, Chapter 3.2.

[0008] In addition, WO 2016 / 090568 shows a binary tree structure for partitioning a unit into multiple smaller units using a quadtree and binary tree structure. Thus, the root unit is first partitioned by the quadtree structure, and then the leaf nodes of the quadtree are further partitioned by the binary tree structure. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] International Publication No. 2016 / 090568 [Non-patent literature]

[0010] [Non-Patent Document 1] V. Sze et al (Ed.), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014, Chapter 3.2 Summary of the Invention [Means for solving the problem]

[0011] Embodiments of the invention are defined by the features of the independent claims, further preferred embodiments of the embodiments are defined by the features of the dependent claims.

[0012] According to a general aspect, the disclosure provides an apparatus for partitioning an image into coding units, the apparatus including a processing circuit configured to: subdivide an image into coding tree units CTUs including non-border CTUs having a predetermined size in a horizontal and vertical direction and boundary CTUs having a portion in the image bounded by a horizontal or vertical image boundary, the portion having a size smaller than the predetermined size in a direction perpendicular to the image boundary, hierarchically partition the non-border CTUs and the boundary CTUs into respective coding units, the hierarchical partitioning of the non-border CTUs including a multi-type partitioning with a non-border multi-type partitioning maximum depth, the multi-type partitioning having a partitioning direction either vertically or horizontally, and the hierarchical partitioning of the boundary CTUs including a multi-type partitioning with a boundary multi-type partitioning maximum depth.

[0013] This offers the advantage of increased flexibility in boundary partitioning.

[0014] In a further embodiment of the device, the boundary multi-type partition maximum depth is at least the sum of an adaptive boundary multi-type partition depth and a predefined multi-type partition depth, where the adaptive boundary multi-type partition depth is a depth of a multi-type partition whose partition direction is in the direction of the image boundary.

[0015] This provides for adaptive determination of partition depth when using multi-type partitioning for boundary coding tree units or partition blocks.

[0016] For example, the predefined multitype partition depth is equal to the unbounded multitype partition maximum depth.

[0017] This allows for reuse of the unbounded multitype partition maximum depth.

[0018] In a further embodiment of the apparatus, the sum further includes a function of a ratio of sizes of a border partition block of the border CTU in a direction of the image boundary and in a direction perpendicular to the image boundary, the border partition block being a block of adaptive border multi-type partition depth.

[0019] This further increases the maximum depth of multi-type boundary partitioning, thereby providing greater partitioning flexibility.

[0020] For example, the function is the binary logarithm.

[0021] This is beneficial since it allows for a practical implementation.

[0022] In some further embodiments, a boundary multi-type partition maximum depth is predefined.

[0023] This facilitates reducing the computational cost in determining the hierarchical partitioning.

[0024] For example, the hierarchical partitioning of the boundary CTU further includes a quadtree partitioning.

[0025] This allows for flexible selection between different modes.

[0026] In a further embodiment of the apparatus, the bounded multi-type partition maximum depth is greater than or equal to the non-bounded multi-type partition maximum depth.

[0027] This achieves an increased maximum boundary partitioning depth possible.

[0028] There is further provided an apparatus for encoding images of a video sequence, comprising an apparatus for dividing images into coding units according to any of the above examples and embodiments, the apparatus further comprising an image coding unit configured to encode the coding units, and a bitstream forming unit configured to generate a bitstream including the encoded coding units and partitioning information indicating how the coding tree units have been partitioned.

[0029] In a further embodiment, the apparatus for encoding an image includes an apparatus for partitioning the image, wherein a boundary multi-type partitioning maximum depth is predefined, and the bitstream further includes an encoded sequence parameter set including the boundary multi-type partitioning maximum depth.

[0030] There is further provided a method for decoding images of a video sequence, comprising: a bitstream parser for parsing a bitstream including encoded coding units; an apparatus for determining a partition of an image according to any of the above examples and embodiments; and an image decoding unit for decoding the encoded coding units based on the determined partition of the image.

[0031] In a further embodiment, the apparatus for decoding an image includes an apparatus for determining a partition of the image, wherein a boundary multi-type partitioning maximum depth is predefined, and the bitstream further includes an encoded sequence parameter set including the boundary multi-type partitioning maximum depth, and the apparatus for determining the partition of the image is further configured to obtain a second multi-type partitioning maximum depth from the sequence parameter set.

[0032] According to another general aspect, a method is provided for partitioning an image into coding units, the method including the steps of subdividing an image into coding tree units CTUs including non-border CTUs having a predetermined size in horizontal and vertical directions and boundary CTUs having a portion in the image bounded by a horizontal or vertical image boundary, the portion having a size smaller than the predetermined size in a direction perpendicular to the image boundary, and hierarchically partitioning the non-border CTUs and the boundary CTUs into respective coding units, the hierarchical partitioning of the non-border CTUs including a multi-type partition with a non-border multi-type partition maximum depth, where the partition direction is either vertical or horizontal, and the hierarchical partitioning of the boundary CTUs including a multi-type partition with a boundary multi-type partition maximum depth.

[0033] In a further embodiment of the method, the boundary multi-type partition maximum depth is at least the sum of the adaptive boundary multi-type partition depth and the predefined multi-type partition depth, where the adaptive boundary multi-type partition depth is the depth of the multi-type partition whose partition direction is in the direction of the image boundary.

[0034] For example, the predefined multitype partition depth is equal to the unbounded multitype partition maximum depth.

[0035] In a further embodiment of the method, the sum further includes a function of a ratio of sizes of a border partition block of the border CTU in the direction of the image boundary and in the direction perpendicular to the image boundary, the border partition block being a block of adaptive border multi-type partition depth.

[0036] For example, the function is the binary logarithm.

[0037] In another embodiment, a boundary multi-type partition maximum depth is predefined.

[0038] In a further embodiment, the hierarchical partitioning of the boundary CTU further comprises a quadtree partitioning.

[0039] For example, the bounded multitype partition maximum depth is greater than or equal to the non-bounded multitype partition maximum depth.

[0040] There is further provided a method for encoding images of a video sequence, the method comprising a step of dividing the images into coding units according to any of the above embodiments, an image coding step of encoding the coding units, and a bitstream formation step of generating a bitstream including the encoded coding units and partitioning information indicating how the coding tree units have been partitioned.

[0041] In a further embodiment, the method for encoding an image includes a method for partitioning the image, wherein a boundary multi-type partitioning maximum depth is predefined, and the bitstream further includes an encoded sequence parameter set including the boundary multi-type partitioning maximum depth.

[0042] There is further provided a method for decoding an image of a video sequence, the method comprising the steps of analysing a bitstream including encoded coding units, determining a partition of the image according to any of the above embodiments, and an image decoding step of decoding the encoded coding units based on the determined partition of the image.

[0043] In a further embodiment, the method for decoding an image includes a method for determining a partition of the image, wherein a boundary multi-type partitioning maximum depth is predefined, and the bitstream further includes an encoded sequence parameter set including the boundary multi-type partitioning maximum depth, and the method for determining a partition of the image further includes a step of obtaining a second multi-type partitioning maximum depth from the sequence parameter set.

[0044] In a further aspect, the present disclosure provides a computer-readable medium having stored thereon instructions that, when executed by a processing circuit, cause the processing circuit to perform a method for dividing an image into coding units, a method for encoding an image of a video sequence, or a method for decoding an image of a video sequence according to any of the above embodiments.

[0045] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims.

[0046] The following embodiments of the invention are explained in more detail with reference to the accompanying figures and drawings. [Brief description of the drawings]

[0047] [Figure 1] 1 is a block diagram illustrating an exemplary structure of a video encoder configured to implement embodiments of the present invention. [Diagram 2] 2 is a block diagram illustrating an exemplary structure of a video decoder configured to implement embodiments of the present invention. [Diagram 3] FIG. 2 is a schematic diagram illustrating an example of quadtree partitioning used by HEVC. [Figure 4] FIG. 2 illustrates several modes of partitioning a coding unit. [Diagram 5] FIG. 1 is a schematic diagram illustrating an example of quadtree / binary tree partitioning. [Figure 6] FIG. 13 is a schematic diagram showing forced quadtree division of a boundary portion. [Figure 7] FIG. 13 is a schematic diagram showing a binary tree division of a boundary portion. [Figure 8] 1 is a block diagram of an apparatus for dividing an image into coding units; [Figure 9] FIG. 13 is a schematic diagram showing division of a boundary portion. [Figure 10] FIG. 1 illustrates an example of boundary partitioning using binary tree partitioning. [Figure 11]FIG. 13 illustrates a further example of boundary partitioning using binary tree partitioning. [Figure 12] FIG. 13 shows a comparison of boundary segmentation using quadtree and binary tree partitioning. [Figure 13] 1 is a flow chart illustrating an implementation of multitype tree maximum depth expansion. [Figure 14] 1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present invention. [Figure 15] 1 is a schematic diagram of a video coding device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0048] The present invention relates to dividing (i.e., partitioning) an image into smaller units for further processing. Such division may be advantageously used in coding and decoding of still or video images. In the following, exemplary video coders and decoders that may implement the division according to the present disclosure are described.

[0049] FIG. 1 shows an encoder 100 with an input for receiving an input block of a frame or picture of a video stream and an output for providing an encoded video bitstream. The term "frame" in this disclosure is used as a synonym for picture. However, it should be noted that the disclosure is also applicable to fields where interlacing is applied. In general, a picture includes m×n pixels. These correspond to image samples, each of which may include one or more color components. For simplicity, the following description refers to pixels meaning samples of luminance. However, it should be noted that the division approach of the disclosure may be applied to any color component, including chrominance, or to components of a color space such as RGB. On the other hand, it may be beneficial to perform the division on only one component and apply the determined division to more (or all) remaining components.

[0050] The encoder 100 is configured to apply partitioning, prediction, transformation, quantization, and entropy coding to the video stream.

[0051] In the division unit 110, the input video frame is further divided before coding. The blocks to be coded do not necessarily have the same size. One picture may contain blocks of different sizes, and the block rasters of different pictures of a video sequence may also be different. In particular, each video image (picture) is first subdivided into CTUs of the same fixed size. The CTU size may be fixed and predefined, for example in a standard. In HEVC, a size of 64x64 is used. However, the present disclosure is not limited to a standard fixed size. It may be preferable to provide a CTU size that can be set in the encoder and provided as a signaling parameter in the bitstream. For example, different CTU sizes may be beneficial for each different picture size and / or content type. The CTU size may be signaled at any signaling level, for example, it may be common for the entire video sequence or a part thereof (i.e., multiple pictures) or for each individual picture. Correspondingly, it may be signaled, for example, in the picture parameter set PPS or in the sequence parameter set SPS or in the video parameter set VPS or in similar parameter sets known from current codecs (H.264 / AVC, H.265 / HEVC). Alternatively, it may be specified in the slice header or at any other level. The CTU size may take values ​​different from 64x64. It may for example have a sample size of 128x128. In general, to perform a hierarchical partitioning by a binary or quadtree, it is necessary to use a 2-dimensional matrix with n being a power of 2, i.e., n is an integer greater than 2. n It may be beneficial to provide a CTU size that is in the format:

[0052] The partitioning of a picture into CTUs and the partitioning of CTUs into CUs is shown in Figure 3 from V. Sze et al. (Ed.), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014. The partitioning follows a quadtree structure to accommodate different local characteristics. On the left side, Figure 3 shows a CTU that is hierarchically partitioned according to the quadtree structure on the right side. In particular, the coding tree defines a syntax that specifies the subdivision of a CTU into CUs. As a CTU, a CU consists of a square block of samples and a syntax associated with these sample blocks. Thus, the partitioning is performed hierarchically starting from a CTU (hierarchical depth 0) that can be, but does not have to be, subdivided into four (in the quadtree) CUs of hierarchical depth 1. In Figure 3, the CTU is partitioned into CUs 8 and 16 at the first hierarchical depth (level), which are not further split and therefore form leaves of the quadtree, and two further CUs that are further partitioned into CUs at hierarchical depth 2 (depth 2 CUs). In particular, the depth 1 CU at the top left is further subdivided into CUs 1, 2, and 7 at depth 2, which form quadtree leaves, and one more CU that is further partitioned into CUs 3, 4, 5, and 6 at depth 3, which are all leaves. Similarly, the depth 1 CU at the bottom left is further partitioned into CUs 13, 14, and 15 at depth 2, which are also leaves of the quadtree, and the remaining CUs that are further partitioned into CUs 9, 10, 11, and 12 at level 3, which are all leaves and therefore are not further partitioned.

[0053] An example syntax for quadtree partitioning in HEVC is shown below in Table 1.

[0054] [Table 1]

[0055] In particular, at the CTU level, a flag named split_cu_flag is included in the bitstream, which indicates whether the complete CTU forms a CU or whether it is split into four equally sized blocks corresponding to a square sample block. If a CTU is split, for each resulting block, another split_cu_flag is sent specifying whether the block represents a CU or whether it is further split into four equally sized blocks. This hierarchical subdivision continues until the resulting block is not further subdivided. The minimum size of a CU is signaled in the sequence parameter set, which can range from 8x8 luma samples up to the size of the CTU. When the minimum CU size is reached in the process of hierarchical subdivision, no split flag is sent for the corresponding block, and instead it is inferred that these blocks will not be further split. In a typical HEVC encoder setting, the maximum range of supported CU sizes is utilized, so that CUs ranging from 8x8 to 64x64 samples may be used. CUs inside a CTU are coded in a depth-first order. This coding order is also called z-scan. It ensures that for each CU except those located at the top or left boundary of a slice, all samples above the CU and to the left of the CU have already been coded, so that the corresponding samples can be used for intra prediction and the associated coding parameters can be used to predict the coding parameters of the current CU.

[0056] In other words, split_cu_flag[x0][y0] specifies whether the coding unit is split into coding units with half the size in the horizontal and vertical directions. The array index x0,y0 specifies the position (x0,y0) of the top-left luma sample of the coding block in question relative to the top-left luma sample of the picture. If split_cu_flag[x0][y0] is not present, the following applies to the decoder: If log2CbSize (a parameter that specifies the coding block size) is greater than MinCbLog2SizeY (a parameter that specifies the minimum configurable coding unit size), the value of split_cu_flag[x0][y0] is inferred to be equal to 1. Otherwise (log2CbSize is equal to MinCbLog2SizeY), the value of split_cu_flag[x0][y0] is inferred to be equal to 0.

[0057] The array CtDepth[x][y] specifies the coding tree depth of the luma coding block covering position (x,y). If split_cu_flag[x0][y0] is equal to 0, then CtDepth[x][y] is inferred to be equal to cqtDepth for x=x0..x0+nCbS-1 and y=y0..y0+nCbS-1.

[0058] In VVC (Versatile Video Coding), segmentation structures including quaternary tree (QT) segmentation and multi-type (bisection / trisection / asymmetric bisection type) tree (BT / TT / ABT) segmentation structures replace the concept of multiple partitioning unit types. In other words, the new segmentation structures remove the separation of the concepts of CU (coding unit), PU (prediction unit), and TU (transform unit), except as required for CUs with sizes too large for the maximum transform length, and support greater flexibility in terms of CU partition shapes [JVET-J1002]. Figure 4 shows the partitioning modes currently used in VTM (VVC Test Model). Part (a) of Figure 4 shows a CTU or CU where no further partitioning is applied (no partitioning). Part (b) shows a quaternary tree (also commonly called "quaternary tree") partitioning mode where a CTU or CTU is partitioned both vertically and horizontally. Parts (c) and (d) show vertical and horizontal binary tree partitioning modes, respectively. Further, parts (e) and (f) show a vertical and horizontal ternary tree division, where it can be seen that there are two blocks of size ¼ and one block of size ½.

[0059] The following parameters are relevant for hierarchical compartmentalization of CTUs: CTU size: The size of the root node of the quaternary tree. MinQTSize: The minimum allowed quaternary tree leaf node size. MaxBTTSize: The maximum allowed binary and ternary tree root node size. MaxBTTDepth: The maximum allowed binary and ternary tree depth. MinBTTSize: The minimum allowed binary and ternary tree leaf node size. MinCUSize: The minimum allowed CU size. is defined and specified by the syntax elements of the Sequence Parameter Set (SPS) of the BT / TT / QT coding tree scheme.

[0060] Fig. 5 shows a mixed quad-tree and binary tree partitioning. Quad-tree partitioning is shown with solid lines, while binary tree partitioning is shown with dashed lines. The labels 1 or 0 on the nodes, which represent coding units that are further divided by the binary tree, indicate whether the binary partitioning is done vertically or horizontally, respectively.

[0061] The horizontal and vertical size of a video picture in luma samples must be an integer multiple of the smallest CU size in luma samples, as signaled in the sequence parameter set, but it does not have to be an integer multiple of the CTU size. If the horizontal or vertical size of a video picture does not represent an integer multiple of the CTU size, the boundary CTUs are inferred to be split until the resulting block boundaries coincide with the picture boundaries. For this forced split, no split flag is signaled, but the resulting block may be further split using the quadtree syntax described above. CUs that are outside the picture area are not coded.

[0062] This partitioning is shown in Figure 6, which is an illustration of a CTU (128x128) constrained QT partition of the lower boundary of an HD sequence (1920x1080). In particular, Figure 6 shows a frame boundary, above which 56 lines (128 samples long) are slice or image boundary portions. The portion of the CTU below the frame boundary may belong to another slice, or may not exist at all, for example if the frame boundary is the bottom boundary of the picture. As can be seen, a constrained quadtree partitioning is applied to 128x56 samples.

[0063] The subdivision of the chroma CTB is always aligned with the respective luma CTB subdivision in HEVC. Note that the present disclosure may process the chroma components in the same way, but is not limited thereto. There may also be an independent division of different color components.

[0064] Returning to FIG. 1, after image segmentation is performed by segmentation unit 110, transformation, quantization, and entropy coding are performed by transform unit 130, quantization unit 140, and entropy encoding unit 150, respectively, to generate the output as an encoded video bitstream.

[0065] A video stream may include multiple frames. For example, a block of a first frame of the video stream is intra-coded by the intra prediction unit 190. An intra frame is coded using information from only that frame, and thus it can be decoded independently from other frames. Thus, an intra frame may provide an entry point into the bitstream, for example for random access. Blocks of other frames of the video stream may be inter-coded by the inter prediction unit 195, and each block of an inter-coded frame is predicted from a block in another frame (reference frame), for example a previously coded frame. The mode selection unit 180 is configured to select whether a block of a frame is intra-predicted or inter-predicted, i.e., whether it is processed by the intra prediction unit 190 or the inter prediction unit 195. The mode selection unit 180 also controls an intra parameter of the inter prediction. To enable refreshing of image information, an inter-coded frame may include not only an inter-coded block, but also one or more intra-coded blocks. In contrast, intra-frames contain only intra-coded blocks, and no inter-coded blocks. Intra-frames may be inserted into a video sequence (e.g., periodically, i.e., after every certain number of inter-frames) to provide an entry point for decoding, i.e., a point at which a decoder may begin decoding without using information from previous frames.

[0066] The intra prediction unit 190 is a block prediction unit. To perform spatial or temporal prediction, the coded block may be further processed by the inverse quantization unit 145 and the inverse transform unit 135. After the reconstruction of the block by the reconstructor 125, the loop filtering unit 160 may be applied to further improve the quality of the decoded image. The reconstructor 125 adds the decoded residual to the predictor to obtain a reconstructed block. The filtered block then forms a reference frame, which is then stored in the frame buffer 170. Such a decoding loop (decoder) on the encoder side provides the advantage of generating the same reference frame as the reference picture reconstructed on the decoder side. Therefore, the encoder side and the decoder side operate in a corresponding manner. The term "reconstruction" here refers to obtaining a reconstructed block by adding the decoded residual block to the predictive block.

[0067] The inter prediction unit 195 receives as input a block of the current frame or picture to be inter-coded and one or several reference frames or pictures from the frame buffer 170. Motion estimation and motion compensation are performed by the inter prediction unit 195. Motion estimation is used to obtain a motion vector and a reference frame, for example based on a cost function. Motion compensation then describes the current block of the current frame in terms of the translation of a reference block of the reference frame to the current frame, i.e., by a motion vector. The inter prediction unit 195 selects a prediction block (i.e., a predictor) of the current block from a set of candidate blocks (i.e., candidate predictors) in one or several reference frames such that the prediction block minimizes the cost function. In other words, the candidate block with the smallest cost function is used as the prediction block of the current block.

[0068] For example, the cost function may be a measure of the difference between the current block and the candidate block, i.e., a measure of the residual of the current block relative to the candidate block. For example, the cost function may be the sum of absolute differences (SAD) of all pixels (samples) of the current block and all pixels of the candidate block in the candidate reference picture. However, in general, a similarity metric such as the mean squared error (MSE) or the structural similarity metric (SSIM) may be used.

[0069] However, the cost function may also be the number of bits required to code such inter-blocks and / or the distortion resulting from such coding. Thus, a rate-distortion optimization procedure may be used to determine the encoding parameters, such as the selection of motion vectors and / or in general, whether to use inter- or intra-prediction for a block and its settings.

[0070] The intra prediction unit 190 receives as input a block of the current frame or picture to be intra-coded and one or several reference samples from already reconstructed areas of the current frame. Intra prediction then describes the pixels of the current block of the current frame in terms of a function of the reference samples of the current frame. The intra prediction unit 190 outputs a prediction block of the current block, said prediction block preferably minimizing the difference between the current block to be coded and its prediction block, i.e. it minimizes the residual block. The minimization of the residual block may for example be based on a rate-distortion optimization procedure. In particular, the prediction block is obtained as a directional interpolation of the reference samples. The direction may be determined by rate-distortion optimization and / or by calculating a similarity measure as mentioned above in connection with inter prediction.

[0071] The difference between the current block and its prediction, i.e., the residual block, is then transformed by the transform unit 130. The transform coefficients are quantized by the quantization unit 140 and entropy coded by the entropy encoding unit 150. The encoded video bitstream thus produced includes intra- and inter-coding blocks and corresponding signaling (such as a mode indication, an indication of the motion vector, and / or an intra-prediction direction). The transform unit 130 may apply a linear transform, such as a discrete Fourier transform (DFT) or a discrete cosine transform (DCT). Such a transform into the spatial frequency domain provides the advantage that the resulting coefficients typically have higher values ​​at lower frequencies. Thus, after an effective coefficient scan (such as zigzag) and quantization, the resulting sequence of values ​​typically has some large values ​​at the beginning and ends with a run of zeros. This allows for even more efficient coding. The quantization unit 140 performs lossy compression by reducing the coefficient values ​​in resolution. The entropy coding unit 150 then assigns binary codewords to the coefficient values. The codewords are written into a bitstream, called the encoded bitstream. The entropy coder also codes signaling information (not shown in FIG. 1), which may include coding according to the split flag syntax shown above.

[0072] 2 shows an example of a video decoder 200. The video decoder 200 comprises, in particular, a reference picture buffer 270 and an intra-prediction unit 290, which is a block prediction unit. The reference picture buffer 270 is configured to store at least one reference frame reconstructed from an encoded video bitstream of an encoded video bitstream. The intra-prediction unit 290 is configured to generate a prediction block, which is an estimate of a block to be decoded. The intra-prediction unit 290 is configured to generate this prediction based on reference samples obtained from the reference picture buffer 270.

[0073] Decoder 200 is configured to decode the encoded video bitstream generated by video encoder 100, and preferably both decoder 200 and encoder 100 generate identical predictions for each block being encoded / decoded. The features of reference picture buffer 270 and intra prediction unit 290 are similar to those of reference picture buffer 170 and intra prediction unit 190 of FIG.

[0074] The video decoder 200 includes further units also present in the video encoder 100, such as an inverse quantization unit 240, an inverse transform unit 230, and a loop filtering unit 260, which correspond to the inverse quantization unit 140, the inverse transform unit 150, and the loop filtering unit 160 of the video encoder 100, respectively.

[0075] The bitstream parsing, entropy decoding, and partitioning unit 250 is configured to parse and decode the received encoded video bitstream to obtain quantized residual transform coefficients and signaling information. The quantized residual transform coefficients are provided to the inverse quantization unit 240 and the inverse transform unit 230 to generate a residual block. The residual block is added to the prediction block in the reconstructor 225, and the resulting sum is provided to the loop filtering unit 260 to obtain a decoded video block. The frames of the decoded video may be stored in the reference picture buffer 270 and serve as reference frames for inter prediction. The signaling information parsed and decoded from the bitstream may generally include control information related to frame partitioning. To further correctly parse and decode the image, the control information is used to restore the division of the image into coding units and correctly assign the next decoded data to the respective coding units.

[0076] In general, the intra prediction units 190 and 290 in Figures 1 and 2 may use reference samples from already encoded regions to generate a prediction signal for a block that needs to be encoded or decoded.

[0077] The bitstream parsing, entropy decoding, and splitting unit 250 receives an encoded bitstream as its input. The bitstream may be first parsed, i.e., signaling parameters and residuals are extracted from the bitstream. The syntax and semantics of the bitstream may be defined in a standard so that encoders and decoders may operate in an interoperable manner.

[0078] In the HEVC standard, a coding tree unit (CTU) or coding unit (CU) located on a slice / picture boundary is split using forced quadtree partitioning (QT) until the bottom right sample of the leaf node is located within the slice / picture boundary. The forced QT partitioning does not need to be signaled in the bitstream. The purpose of the forced partitioning is to enable encoding / decoding of boundary CTUs / CUs by the encoder / decoder. In other words, it is agreed at the encoding and decoding sides that the QT partitioning will be used without requiring further signaling of the partitioning mode.

[0079] In both the patent [WO2016090568] and VTM-1.0 on the QTBT structure, the boundary CTU / CU forced partitioning process is inherited from HEVC. It is that CTUs / CUs located on frame boundaries, especially CTUs / CUs through which boundaries pass such that parts of the CU / CU are outside the picture / frame (in this disclosure, such CTUs / CUs are also referred to as "boundary CTUs" and "boundary CUs", respectively), are first forcedly partitioned by the quadtree (QT) structure without rate-distortion (RD) optimization until the entire current CU is located inside the slice / picture boundary. These forced partitions do not need to be signaled in the bitstream. Further partitioning is optionally achieved based on RD optimization. Figure 6 shows an example of one forced partition of a lower boundary CTU (128x128) of an HD (1920x1080 pixels) sequence with forced QT.

[0080] Since only the QT partition structure is used in boundary partitioning, the limit for multi-type trees (BT / TT / ABT) in VTM is given by MaxBTTDepth in SPS. Therefore, for partitioning of boundary CTUs (i.e., CTUs that have parts on both sides of the boundary as shown in Figure 6), the limit for the total hierarchical depth (TotalDepthLim), i.e., the maximum hierarchical depth, follows from Equation (1) below. TotalDepthLim=QTDepth+MaxBttDepth(1)

[0081] In equation (1), MaxBttDepth is the allowed maximum binary and ternary tree depth specified in the SPS. QTDepth is the hierarchical depth of the partitioned block resulting from quadtree partitioning. That is, at each partitioning step of the hierarchical partitioning where quadtree division is applied, the value of QTDepth is increased by 1. For example, QTDepth may correspond to the parameter cqtDepth in the syntax example of Table 1. However, although the limit of QTDepth is incremented at each QT partitioning step, the hierarchical depth is finally limited by a certain allowed minimum CU size (e.g., corresponding to the SPS parameter minCUSize).

[0082] Picture boundaries can be processed using BT, TT, or ABT (Asymmetric Binary Tree). Both forced and adaptive methods can be used. If MTT (Multi-Type Tree) such as BT / TT is used for boundary partitioning instead of QT partitioning, the limit MaxBTTDepth of BTT (Binary and Ternary Tree) from SPS can be easily exceeded. Figure 7 is an example of BT used in CTU of the lower boundary of HD sequence. If BT starts to be used from CTU level to leaf nodes (coding units) that are completely inside the boundary, a depth of 4 is used for BT partitioning, which exceeds MaxBTTDepth from SPS (in VTM-1.0, VVC Test Model version 1.0, MaxBTTDepth is set to 3).

[0083] The objective of this disclosure is to improve the flexibility of boundary part partitioning and provide a picture boundary processing perspective. The approach of this disclosure is to define a boundary partition depth limit in addition to the prior art MaxBTTdepth from the SPS used for partitioning. For this purpose, a depth value is set specifically for boundary partitioning (which may be called, for example, ExtdMaxBTTDepth) rather than MaxBttDepth from equation (1). ExtdMaxBTTDepth may be predefined and fixed in the SPS and signaled in the SPS bitstream, or it may be calculated. Thus, when a multi-type tree (MTT) partition (such as BT, TT, or ABT) is used in picture boundary processing (forced or adaptive), the maximum MTT depth limit may be extended in both the decoder and the encoder.

[0084] In the following aspects and embodiments of the present disclosure, possible methods for obtaining and calculating ExtdMaxBTTDepth are described.

[0085] According to a general aspect, an apparatus 800 (which may be implemented in the units 110, 250 of the encoder or decoder described above) for partitioning an image into coding units is provided, as shown in FIG. 8. The apparatus includes a processing circuit configured to subdivide 810 an image into coding tree units CTUs. The CTUs include non-boundary CTUs having a predetermined size in the horizontal and vertical directions, and boundary CTUs having a portion in the image bounded by a horizontal or vertical image boundary, the portion having a size smaller than the predetermined size in the direction perpendicular to the image boundary. The circuit is further configured to partition the non-boundary CTUs and the boundary CTUs into respective coding units. Therein, the hierarchical partitioning of the non-boundary CTUs includes a multi-type partitioning with a non-boundary multi-type partitioning maximum depth, the multi-type partitioning being a partitioning in which the partitioning direction is either vertical or horizontal. Furthermore, the hierarchical partitioning 820 of the boundary CTUs includes a multi-type partitioning with a boundary multi-type partitioning maximum depth.

[0086] Thus, the circuitry of the device 800 for dividing an image into coding units is configured to partition non-boundary CTUs into coding units using a first partitioning depth limit of an MTT partitioning step included in the partitioning, i.e., a non-boundary multi-type partitioning maximum depth. The circuitry is further configured to partition 820 boundary CTUs into coding units using a second partitioning depth limit of an MTT partitioning included in the partitioning, i.e., a boundary multi-type partitioning maximum depth. In other words, according to the present disclosure, the circuitry of the device 800 uses respective MTT partitioning maximum depths (MTT depth limits) for non-boundary CTUs and boundary CTUs during operation.

[0087] The CTUs obtained by frame segmentation 810 may be further hierarchically segmented, including hierarchical segmentation of boundary CTUs 820. This segmentation may be performed, for example, in the manner shown and described above with reference to Figures 3 to 5.

[0088] 8 shows the internal structure of the circuit of the device 800. This circuit can be any kind of hardware and software having functional units 810 and 820 for the respective division (subdivision) of an image (or frame) into CTUs, for the segmentation of CTUs including non-boundary CTUs, and in particular of boundary CTUs. These units can be implemented, for example, on a single processor. However, the invention is not limited to such application and these units can also be implemented by separate hardware components.

[0089] In this disclosure, the term "border CTU" is used to denote a coding tree unit that is divided by an image boundary into a part of an image that is partitioned (within the image boundary) and a part that is not located inside the image (i.e., it is located beyond the image boundary). A border CTU exists when the size of the image to be coded is not an (integer) multiple of the CTU size in at least one direction.

[0090] FIG. 9 visualizes an example of a bottom picture boundary 900 and a corresponding boundary portion 910 (shaded) that includes a boundary CTU. The portion 950 shows the remaining portion of the image that has a size that is an integer multiple of the CTU in the vertical and horizontal directions. Furthermore, the vertical size of the CTU is shown as 970V, and the horizontal size of the CTU is shown as 970H. As can be seen in FIG. 9, the boundary portion in this example is an integer multiple of the CTU size 970H in the horizontal direction. However, in the vertical direction, the boundary portion 910 has a size that is the remaining size after the division of the vertical picture size by the vertical CTU size 970V. The portion 920 is only imaginary and shows the difference between the height of the boundary portion and the CTU size. It should be noted that in this embodiment, the CTU is a square, and thus the sizes 970H and 970V are the same. However, the disclosure is not limited thereto, and the vertical and horizontal sizes of the CTU may be different.

[0091] To encode (and correspondingly decode) the boundary portion, the boundary portion 910 in FIG. 9 is divided into incomplete CTUs, i.e., CTU portions having a horizontal size of CTU 970H and a vertical size smaller than CTU size 970V. These incomplete CTUs correspond to the "boundary CTUs" of this application, where "incomplete CTUs" correspond to portions of CTUs in an image. Although horizontal boundary portions are shown in FIG. 9, vertical boundary portions may be present in addition to or instead of the horizontal boundary portions. In particular, when the image size (i.e., the image width and height) in both the vertical and horizontal directions is not a multiple of the CTU size, there are further incomplete CTUs that are horizontally and vertically bounded CTUs, where the vertical and horizontal sizes of the CTU portions in the image are smaller than the size of the complete CTUs.

[0092] In contrast to a "boundary CTU," the term "non-boundary CTU" is used to indicate a CTU that fits completely into a frame or picture, i.e., a CTU inside portion 950 in the example shown in FIG. 9. That is, non-boundary CTUs are located inside an image that is encoded / decoded at its full size in the vertical and horizontal directions. However, non-boundary CTUs may be adjacent to at least one image boundary, or they may be further CTUs between the non-boundary CTU and the image boundary.

[0093] It should be further noted that non-boundary CTUs and boundary CTUs do not constitute different types of CTUs that are coded or specified differently from one another. The difference between non-boundary CTUs and boundary CTUs is whether they are located on a boundary (i.e., a boundary that passes through them) or inside a boundary that bounds an image.

[0094] Furthermore, whether a CTU is a boundary CTU is determined, for example, by comparing the position of the CTU (particularly, a suitable pixel position within the CTU) with the position of the boundary (or the vertical / horizontal image size of the sample). In the code, the CTU has a predefined fixed size, as described above, e.g., 128×128 or 64×64 as in the case of HEVC. The picture is divided into CTUs without overlaps. As a suitable pixel position, the bottom right corner may be selected, because it allows the CTU to be determined to be a boundary CTU with respect to each of the vertical and horizontal picture boundaries when the image is processed from left to right and from top to bottom (in general, which pixel position is optimal depends on the processing direction). The encoder / decoder checks the pixel at the bottom right corner of the CTU and compares it with the vertical and horizontal sizes of the image. If the bottom right pixel is located inside the picture boundary, the CTU is a non-boundary CTU, otherwise it is a boundary CTU. This method of determining whether a partitioned block lies on a boundary not only applies to CTUs, but may also be used for CUs and any partitioned blocks that result from splitting a CTU or any block in the partitioning hierarchy.

[0095] Furthermore, other methods of determining whether a CTU is a border CTU may be used. For example, as mentioned above, dividing the image size (width / height) by the CTU size may determine the number of CTUs and whether there are border CTUs vertically and horizontally. The CTUs may be indexed, and if there are border CTUs, the last k CTUs may be determined to be bottom border CTUs, or every kth CTU may correspond to a right border CTU (k is the number of CTUs per row (including border CTUs), i.e., the upper limit of the ratio of image width to CTU size).

[0096] The hierarchical depth of hierarchical partitioning is described above with respect to Figure 3. Thus, the partition depth corresponds to the number of partitioning steps performed to obtain a particular partitioning level, starting with the CTU having partitioning depth 0, which corresponds to level 0 (or layer 0), and going up to the depth of the coding unit.

[0097] For a non-bounded CTU, the hierarchical depth of the coding unit is limited such that it cannot exceed the maximum hierarchical depth according to a setting such as the setting of the SPS. For example, in the setting, different limits may be set for the partitioning depth of each of the quadtree partition and the multi-type partition. For example, the partitioning of the CTU includes both QT partition and MTT partition (such as BT, TT, and ABT partition) at different levels or different partial blocks at the same level, since the depth limit may be set separately for the QT partitioning step and the MTT partition. For example, the non-bounded MTT maximum depth may correspond to the parameter MaxBTTDepth of the current VTM, or it may be redefined and named, for example, MaxMTTDepth, and applied to all MTT partition types such as BT, TT, etc. Thus, the depth limit, i.e., the maximum overall depth of the CTU partition, may be the sum of the respective depth limits of the QT partition and the MTT (e.g., BTT).

[0098] On the other hand, for a boundary CTU, the present disclosure provides a depth constraint given by equation (2) below: TotalBPDepthLim=QTDepth+ExtdMaxBTTDepth(2)

[0099] As can be seen from equation (2), the depth limit of the boundary CTU, also referred to as the boundary partition depth in this disclosure, is generally composed of the depth of the QT division and the depth of the MTT division (which is not limited to bisection and trisection, and the name ExtdMaxBttDepth should be considered merely as an example). The QT depth may be the same QT depth used in equation (1) in the case of a forced QT division. However, equation (2) differs from equation (1) by a boundary multitype partition maximum depth ExtdMaxBTTDepth (extended bisection / trisection maximum depth), which may alternatively be referred to, for example, as "ExtdMaxMTTDepth".

[0100] Corresponding to the device 800, a method for partitioning an image into coding units is provided as a further aspect of the present disclosure. The method includes subdividing an image into coding tree units (CTUs). In the CTUs, non-border CTUs having a predetermined size in the horizontal and vertical directions and boundary CTUs having a portion in the image bounded by a horizontal or vertical image boundary, the portion having a size smaller than the predetermined size in the direction perpendicular to the image boundary. The method further includes hierarchically partitioning the non-border CTUs and the boundary CTUs into respective coding units. In the hierarchical partitioning of the non-border CTUs includes a multi-type partitioning with a non-border multi-type partitioning maximum depth, the multi-type partitioning being a partitioning in which the partitioning direction is either vertical or horizontal. The hierarchical partitioning of the boundary CTUs includes a multi-type partitioning with a boundary multi-type partitioning maximum depth.

[0101] In the following, several example embodiments of the present disclosure are described that further demonstrate both the apparatus 800 and the corresponding method for dividing an image into coding units.

[0102] EMBODIMENT 1 As explained above, HEVC or VTM-1.0 uses forced QT for boundary CTU partitioning, so there is no need to define a separate BTT depth limit for boundary partitioning. In HEVC or VTM-1.0, the parameter MaxBttDepth from equation (1) applies to both boundary and non-boundary cases.

[0103] However, since CE1 starts to include BT or TT as a boundary segment, the restriction of MaxBTTDepth of the boundary according to formula (1) may not be inappropriate for the reasons explained above in relation to SubCE2. In embodiment 1, a new MaxBTTDepth is defined that is used only for the boundary segment when the MTT segment structure is included in the boundary segment.

[0104] This disclosure provides a new limit definition of boundary partition MTT depth in hybrid video coding. As mentioned, this boundary multi-type partition maximum depth can be calculated or predefined. This embodiment 1 provides some exemplary methods of calculating boundary multi-type partition maximum depth.

[0105] In embodiment 1, the boundary multi-type partition maximum depth is at least the sum of the adaptive boundary multi-type partition depth and the predefined multi-type partition depth. The adaptive boundary multi-type partition depth is the depth of the multi-type partition whose partition direction is along the image boundary.

[0106] Therefore, in order to maintain the flexibility of boundary block partitioning, it is proposed to perform fair processing of the overall depth restriction of blocks located in boundary CTUs, where fairness is oriented towards the MTT partitioning of boundary CTUs compared to the QT partitioning in JEM picture boundary processing as defined by equation (1).

[0107] For this purpose, the boundary MTT partition maximum depth used in equation (2) is redefined as follows: ExtdMaxBTTDepth=BTTBPDepth+MaxBTTDepth(3)

[0108] Therein, the boundary MTT partition maximum depth, denoted as ExtdMaxBTTDepth, can be derived from equation (1) or (2). Thus, the overall partition depth limit of the boundary CTU is redefined as follows: TotalBPDepthLim=QTDepth+(BTTBPDepth+MaxBTTDepth)(4)

[0109] In formulas (3) and (4), the adaptive boundary multi-type partition depth of the present embodiment 1, BTTBPDepth, is determined at each layer, so that ExtdMaxBTTDepth also varies according to the partition layer, i.e., the layer of the coding block of each partition step and the layer of the resulting CU that is finally coded and decoded. For example, the adaptive boundary multi-type partition depth can be selected to be the same for different MTT partition modes, such as BT / TT / ABT boundary partitioning. The adaptive boundary multi-type partition depth is redetermined (i.e., increased) at each partition layer, regardless of whether BT, TT, ABT, or some other MTT partitioning type is performed at this layer. As in the case of QT partitioning, the absolute depth limit may result from a predetermined minimum CU size, for example, the SPS parameter minCUSize mentioned above. For the BT partitioning step, the depth is increased by one at each partition block resulting from the step. For TT partitioning, according to VTM-1.0, the depth is increased by 1 for the resulting 1 / 2 size blocks and by 2 for 1 / 4 size blocks to establish the relationship between the depth and size of the coding units and to ensure compatibility between BT and TT partitioning.

[0110] As mentioned above, the adaptive multi-type partition depth is a depth whose partition direction is in the direction of the image boundary. This means that the partition line that divides the block of the corresponding layer into two, three or more blocks of a subsequent deeper layer has the same direction, i.e., is parallel, to the image boundary in question, which passes through the boundary CTU to be partitioned. However, partitioning / partitioning steps with partitioning directions perpendicular to the picture boundary in question do not contribute to the value of the adaptive multi-type partition depth. In particular, if the picture boundary is a horizontal boundary, only steps in the horizontal partitioning direction are counted by the adaptive multi-type partition depth. Instead, if the picture boundary is a vertical boundary, only steps whose partitioning direction is vertical are counted. This can be seen in the partitioning examples shown in Figures 10 and 11.

[0111] However, the present disclosure is also applicable to boundary CUs and boundary partition blocks in a partitioning hierarchy of a boundary CTU that is located on both a horizontal boundary (e.g., bottom boundary) and a vertical boundary (e.g., right boundary), e.g., a bottom-right CTU. In this case, the steps MTT steps in both partitioning directions are counted by the BTTBPDepth term in equation (3) or by separate respective variables.

[0112] On the other hand, as can be understood from equations (3) and (4), the non-adaptive term in the boundary multi-type partition maximum depth (i.e., the predefined multi-type partition depth) may be equal to the non-boundary multi-type partition maximum depth, for example. However, the present disclosure is not limited thereto. For example, if an additional predefined multi-type boundary partition depth parameter is to be defined, it may be beneficial to set this parameter to a smaller value (e.g., to save one or more bits).

[0113] FIG. 10 shows a first example of boundary partitioning using binary tree partitioning, in particular an example of bottom boundary partitioning using BT. The starting point shown on the left side of the upper half of the figure shows the boundary CTU and the bottom picture boundary (thick solid line) that passes through the boundary CTU. Furthermore, in the upper right corner of the CTU, there is a marked block object obtained by image partitioning. The right side of the upper half of the figure shows the final partitioning (i.e., partitioning pattern) in which the boundary CTU is partitioned. However, in this first example and the example shown in FIG. 10, only the partitioning step of MTT (in particular BTT) is considered. According to equation (4), QT partitioning may be performed in a further step preceding the MTT partitioning step. Thus, the starting point does not necessarily have to be a CTU, but may be, for example, a square coding block (with equal vertical and horizontal sizes) of depth greater than 0 resulting from the QT partitioning.

[0114] In the lower half of Fig. 10, the partitioning steps 1 to 3 of partitioning the boundary CTU from the starting point to the final partitioning are shown. In it, the thick solid lines are the picture boundaries, the BTT boundaries and CTU partitions are shown as solid lines (inside the boundary) or dashed lines (outside the boundary), and the dotted lines (in step 3) are the "normal" partitions of the BTT (i.e., the partitions that are vertical are not boundary partitions whose partitioning direction is in the direction of the image boundary). If MaxBTTDepth (limit 3 in VTM) was used according to equation (1), the object block in the upper right corner of the CTU cannot be partitioned further since there are three MTT partitioning steps, and therefore CurrentBTTDepht(3)>=MaxBTTDepth(3) (CurrentBTTDepth is the depth of BTT partitioning at the current step, corresponding to the number of steps in which BTT partitioning is performed).

[0115] According to embodiment 1, in step 1, there is one layer of (BTT) boundary partitioning (i.e., partitioning whose split direction is the direction of the boundary). Therefore, BTTBPDepth=1. Therefore, from equation (3), we obtain ExtdMaxBTTdepth=BTTBPDepth+MaxBTTDepth=1+3=4 (MaxBTTDepth is set to 3 in SPS). Since the maximum BTT depth layer (boundary multi-type partition maximum depth) in step 1 is 4 and the block has already been partitioned once, the partitioned block can be MTT partitioned three more times according to ExtdMaxBTTdepth set in step 1 according to the boundary multi-type partition maximum depth.

[0116] In step 2, there are two layers of boundary partitioning obtained by the BP (boundary partitioning) performed so far, and BTTBPDepth=2. Therefore, from equation (3), we obtain ExtdMaxBTTdepth=BTTBPDepth+MaxBTTDepth=2+3=5 (MaxBTTDepth is set to 3 in SPS). Since the new maximum BTT depth layer (boundary multi-type partitioning maximum depth) in step 2 is 5 (calculated from the starting point (a)), and the block has already been partitioned twice, the partitioned block can be partitioned three more times according to the boundary multi-type partitioning maximum depth. It can be seen that the boundary partitioning (i.e., the division direction is horizontal and in the same direction as the image boundary direction), BTTBPDepth, and therefore ExtdMaxBttDepth, has been increased by 1 from step 1 to step 2.

[0117] In step 3, a further BT partitioning is performed. However, this last partitioning has a partitioning direction that is not in the direction of the image boundary, in particular, the partitioning direction of step 3 is perpendicular to the image boundary. For this reason, step 3 does not contribute to the value of BPPBPDepth. Therefore, there are still two layers of BTT BP, and BTTBPDepth=2. As follows, ExtdMaxBTTdepth=BTTBPDepth+MaxBTTDepth=2+3=5 (MaxBTTDepth is set to 3 in SPS). As explained, in adaptive multi-type partitioning depth, only the depth (number of steps / layers) of the multi-type partitioning whose partitioning direction is in the direction of the image boundary contributes. However, the boundary multi-type partitioning maximum depth is a limit of MTT boundary partitioning. Therefore, the maximum depth layer of step 3 remains 5 (calculated from the starting point), and the partitioned block can be BTT partitioned two more times, since the block has already been partitioned three times (here, all partitions are considered, not just BP).

[0118] A second example of MTT partition maximum depth calculation is shown in Fig. 11. In this figure, similar to Fig. 10, the thick solid line is the picture boundary, and the solid line (inside) or dashed line (outside) is the CTU boundary or BTT boundary partition. There is a marked object (i.e., the target coding block obtained by MTT partition) located inside the boundary CTU. To obtain the final partition pattern in the upper right part of the figure, four partitioning steps need to be performed, as shown in steps 1 to 4 of the figure.

[0119] If MaxBTTDepth is used according to Equation 1 and limited to 3 as in the case of VTM, the marked target block (at MTT depth 4) cannot be obtained and cannot be further partitioned because CurrentBTTDepht(4)>=MaxBTTDepth(3).

[0120] In step 1 of Figure 11, there is one layer of BTT BP, and BTTBPDepth = 1. Therefore, ExtdMaxBTTdepth = BTTBPDepth + MaxBTTDepth = 1 + 3 = 4 (MaxBTTDepth is set to 3 in SPS). The maximum BTT depth layer in step 1 is 4, and the block has already been partitioned once, so the partitioned block can be BTT partitioned three more times.

[0121] In step 2 of Figure 11, there are two layers of BTT BP, and BTTBPDepth = 2. Therefore, ExtdMaxBTTdepth = BTTBPDepth + MaxBTTDepth = 2 + 3 = 5. The maximum BTT depth layer in step 2 is 5, and the block has already been partitioned once, so the partitioned block can be BTT partitioned three more times.

[0122] In step 3 of Figure 11, there are three layers of BTT BP, and BTTBPDepth = 3. Therefore, ExtdMaxBTTdepth = BTTBPDepth + MaxBTTDepth = 3 + 3 = 6. The maximum BTT depth layer in step 2 is 6, and the block has already been partitioned once, so the partitioned block can be BTT partitioned three more times.

[0123] Furthermore, in step 4, there are 4 layers of BTT partitioning: ExtdMaxBTTdepth=BTTBPDepth+MaxBTTDepth=4+3=7. Since the maximum depth layer in step 4 is 7 (calculated from starting point (a)), and the block has already been partitioned 4 times, the partitioned block can be partitioned at most 3 more times.

[0124] In the boundary multi-type partition maximum depth, all partitions are considered and limited, not just BPs whose partition direction is in the direction of the boundary. However, in all of the steps in the example shown in Figure 11, multi-type partitions are performed whose partition direction is in the direction of the image boundary. Therefore, in each of these steps, the value of the adaptive boundary multi-type partition depth (BTTBPDepth in this example) is increased by 1.

[0125] By adapting the depth limit of boundary multi-type partitioning to the number of partitioning steps whose partitioning direction is in the same direction as the boundary, the present disclosure facilitates a partitioning pattern of coding units that more practically approximates the remaining portions of CTUs near image boundaries. In particular, the number of coding units of boundary CTUs / portions may be reduced, in contrast to using only QT partitioning.

[0126] EMBODIMENT 2 According to the embodiment 1 described above, the boundary multi-type partition maximum depth (for example, given as ExtdMaxBTTDepth) is at least the sum of the adaptive boundary multi-type partition depth and the predefined multi-type partition depth. Here, according to the embodiment 2, this sum further includes a function of the ratio of the size of the boundary partition block of the boundary CTU in the direction of the image boundary and the direction perpendicular to the image boundary. In it, the boundary partition block is a block of the adaptive boundary multi-type partition depth.

[0127] In this disclosure, the term "partition block" refers to a CTU or block resulting from splitting a lower level block / unit. Thus, a partition block is not limited to a CTU at the top of a hierarchical partitioning or coding unit that is the final result of the hierarchical partitioning to be encoded / decoded, but also includes blocks obtained in intermediate partitioning steps, such as steps 1 and 2 in FIG. 10 or steps 1 to 3 in FIG. 11. Furthermore, a "border partition block" is a partition block that is located at an image boundary (an image boundary through a block) in the same way that a border CTU is located at the boundary.

[0128] If the boundary in question is a horizontal image boundary, then the size ratio referred to above is the ratio of the horizontal size (width) to the vertical size (height) of the divided boundary segment block, whereas if the image boundary is vertical, then the ratio is the height divided by the width of the boundary segment block.

[0129] For example, the ratio function may be a binary logarithm log2 ratio. In embodiment 1, the total depth limit is regarded as the criterion for defining the BTT limit of the boundary block. Compared with the calculation of the MTT BP depth limit using formula 1, embodiment 2 considers the reachable size of the coding unit as a fairness criterion (i.e., the fairness between the QT division and the MTT division), in particular. Therefore, the boundary multi-type partition maximum depth is extended according to the following formula (5). ExtdMaxBTTDepth=log2Ratio+BTTBPDepth+MaxBTTDepth(5) where Ratio indicates the ratio of width to height (width / height if the border is horizontal, or height / width if the border is vertical).

[0130] An example of boundary partitioning according to embodiment 2 is shown in Fig. 12. Part (a) of the figure shows the boundary partitioning of VTM-1.0 with forced QT, while part (b) of Fig. 1 is the boundary partitioning with BT. As in Fig. 10 and Fig. 11, solid lines (inside the boundary). Dashed lines show CTU and boundary partitioning (split direction is boundary direction), and dotted lines show partitioning where split direction is not the same as boundary direction. In part (a) of Fig. 12, the top-left marked block of the CTU can be further partitioned with three additional levels of BT, as MaxBTTDepth from SPS is set to 3 in the configuration of VTM-1.0, and no BT partitioning has been performed so far. In part (b), the top-left marked block of the CTU can also be partitioned with three more levels of BT (or other MTT types), and thus the ExtdMaxBTTDepht of the marked block is obtained from equation (5) as follows: ExtdMaxBTTDepth=log2(boundary partition block ratio(4))+BTTBPdepth(2)+MaxBTTDepth(3)=7

[0131] For the current embodiment 2 shown in part (b) of Figure 12, according to ExtendedMaxBTTDepth indicating the boundary multi-type partition maximum depth, the marked block can potentially be divided into three further levels of BT (ExtdMaxBTTDepht(7)-CurrentBTDepht(4)) = 3.

[0132] Here, it is assumed that the marked block is obtained by two BT partition steps whose partition direction is the boundary direction, followed by two BT steps whose partition direction is perpendicular to the boundary partition, and the resulting current BT depth is equal to 4. However, the present disclosure is not limited to MTT partitions that are BTT partitions. For example, a partition mode in which four partitioned blocks are obtained in a single step in which a partitioned block is partitioned three or more times in the same partition direction may also be used (such a partition mode in which four partitioned blocks are obtained is sometimes called SplitInto4, which is different from QT partitions, although four partitioned blocks are also obtained in one step). In the example of FIG. 12, if the last two BT partition steps are replaced by one SplitInto4 partition step, the resulting current MTT depth is 3 instead of 4.

[0133] Furthermore, in MTT, the counted hierarchical depth can be determined with respect to the resulting block size of the smallest partition. For example, in the current TT implementation of VTM-1.0, the block is divided into 1 / 4, 1 / 2 and 1 / 4 subblocks / partition blocks. In the first and third partition blocks, the depth is counted twice in VTM-1.0 (and the ABT step can be counted twice in the smaller of the resulting partition blocks) to ensure that the depth is equally valid in both BT and TT. Thus, to ensure compatibility with BT partitioning, a single SplitInto4 step can also be counted as two steps. In this way, the resulting block size of the partition block / coding unit can be understood from the depth. However, in some other possible ways, each step can be counted once, if it is guaranteed that the respective partitioning type is known, for example from some variables.

[0134] Note that in equation (5), especially when BT partitioning is applied (as shown in part (b) of FIG. 12), the binary logarithm of the right-side boundary partition block ratio (first term) may be the same as the MTT BP depth (second term).

[0135] It should be further noted that the present disclosure is not limited to the function of the boundary partition block being a binary logarithm. For example, the quantification itself (the identity function) may be used, or, for example, the ratio may be multiplied by a constant or some adaptive parameter.

[0136] EMBODIMENT 3 In the above-described embodiments 1 and 2, how the boundary multi-type partition maximum depth can be calculated is described. Meanwhile, in the present embodiment 3, the boundary multi-type partition maximum depth is predefined (i.e., it is fixed). The predetermined value can be provided as the term BTTBPDepth in formula (4), or it can replace MaxBttDepth in a formula such as formula (1) for the boundary CTU.

[0137] For example, such a fixed boundary multi-type partition maximum depth may be obtained from bitstream signaling. For example, it may be signaled within a set of control parameters common to one or more video frames, such as PPS, SPS, VPS in HEVC. It may also be signaled once for the entire video sequence. The signaling may include further coding of the parameters, such as differential coding, predictive coding, entropy coding, or any other embedding into the bitstream.

[0138] However, as described, the present disclosure, including embodiments 1 to 3 above, is not limited to a boundary multi-type partition maximum depth that is fixed and signaled in the bitstream. It may also be derived, for example, based on one or more parameters conveyed in the bitstream (such as MaxBTTDepth in the SPS) and / or internal variables such as BTTBPDepth or QTDepth mentioned above. However, the derivation may also be based on a standard or signaled predefined relationship.

[0139] Although the description of some examples (including Equations 1 through 5) refers to BTT partitioning and the variables are named accordingly (e.g., BTTBPDepth, MaxBTTDepth), the present disclosure is also applicable to ABT (Asymmetric Bisection Partitioning) partitioning (wherein a partitioning block is partitioned into two or more partition blocks having different sizes, such as ¼ and ¾, in one partitioning step) or other types of MTT partitioning, such as SplitInto4Mode mentioned in connection with part (b) of FIG. 12. Thus, the variables may be named differently, e.g., BTTBPDepth, MaxBTTDepth, etc. Thus, the present invention facilitates providing flexibility for picture boundary processing using alternative multi-type partitioning modes in addition to QT partitioning, regardless of the particular MTT partitioning mode used.

[0140] However, in some embodiments, in addition to the MTT partition, the hierarchical partition of the bounded CTU (and the hierarchical partition of the non-bounded CTU) may further include a QT partition. This can be understood from equation (4), which defines the overall bounded partition maximum depth as the sum of the bounded multi-type partition maximum depth (term in brackets) and the QT partition depth (internal variable QTDepth). For example, as in VTM-1.0, the quaternary partition may be performed before the MTT partition. That is, a QT partition step is performed followed by an MTT partition step. However, the present disclosure is not limited thereto, and it is also applicable to configurations in which the QT partition is performed after the MTT partition. Thus, as mentioned, the starting point in FIG. 10 and FIG. 11 does not have to be a CTU, but may be a partition block obtained from one or more previous partitioning steps.

[0141] Furthermore, according to the present disclosure, the boundary multi-type partition maximum depth may be equal to or greater than the non-boundary multi-type partition maximum depth. For example, if no multi-type partitioning is performed in the boundary CTU, with the partitioning direction being the direction of the boundary, the boundary multi-type partition maximum depth may be equal to the non-boundary multi-type partition maximum depth. In this case, there is no partitioning step in which the internal variable BTTBPDepth (or MTTBPDepth) is incremented. Nevertheless, the apparatus for partitioning an image into coding units performs the determination of the non-boundary multi-type partition depth and the boundary multi-type partition maximum depth in different ways, as described.

[0142] However, in embodiment 3, in which the boundary multi-type partition maximum depth is predetermined, its predetermined value must be greater than the non-boundary multi-type partition maximum depth to allow greater flexibility in MTT partitioning in boundary CTUs.

[0143] A flow chart of the BT / TT (or MTT in general) maximum depth extension of the present disclosure is shown in Figure 13. The condition "Is there a CTU on the boundary?" indicates whether the current CTU / CU is located inside a CTU located on the boundary. If this condition is false (N), the regular MaxBTTDepth from the SPS is used as the non-boundary multi-type partition depth to limit the possible BT / TT (or other MTT) maximum depth. Otherwise, MaxBttdepth is extended and determined according to any of the embodiments of the present disclosure.

[0144] As mentioned, the apparatus 800 for splitting an image may be incorporated into a video encoding apparatus 100 or a decoding apparatus 200 as shown in Figures 1 and 2. Thus, the present invention further provides an apparatus 100 for encoding an image of a video sequence, i.e., an encoding apparatus. The encoding apparatus 100 comprises the apparatus 800 for splitting an image according to any of the embodiments described in this disclosure, an image coding unit for encoding a coding unit, and a bitstream forming unit configured to generate a bitstream including the encoded coding unit and partitioning information indicating how the coding tree units have been partitioned.

[0145] In particular, the partition information may include information indicating which partition type mode is applied for each CTU or for a partitioned block of an intermediate step, among QT and some MTT partition modes. For example, for each partition of a partitioned block or CTU, a parameter indicating the partition mode may be included in the bitstream. Alternatively, a particular partition mode may be forced to be used.

[0146] It should be noted that the partitioning modes of boundary and non-boundary CTUs may be different. For example, as illustrated above, the partitioning mode may be signaled on a CTU basis (and / or on a partitioned block basis) for both boundary and non-boundary CTUs. Alternatively, the partitioning mode of all CTUs of a particular boundary (vertical or horizontal) may be the same and set in the signaling associated with one or more video pictures.

[0147] However, the partitioning mode of a boundary CTU may be determined in a different manner than the partitioning mode of a non-boundary CTU, for example, for boundary CTUs (and partition blocks) a pre-determined partitioning mode may be enforced, i.e. the standard may define a fixed partitioning mode, or an algorithm for determining the partitioning mode of a boundary CTU, for example based on its position within the boundaries and / or size and / or partitioning modes of adjacent CTUs.

[0148] Correspondingly, the present disclosure further provides a method for encoding images of a video sequence, the method comprising the steps of a method for partitioning an image according to any of the embodiments described in the present disclosure, an image coding step for encoding coding units, and a bitstream formation step for generating a bitstream including the encoded coding units and partitioning information indicating how the coding tree units have been partitioned.

[0149] Also provided is an apparatus 200 and a method for decoding an image of a video sequence. When incorporated in the decoding apparatus 200, the apparatus 800 is used to determine a division of an image (to be decoded) into coding units. The decoding apparatus 200 includes a bitstream parser for parsing a bitstream including the encoded coding units, an apparatus 800 for determining a division of an image according to any of the embodiments, and an image decoding unit for decoding the encoded coding units based on the determined division of the image.

[0150] Correspondingly, a method for decoding an image includes a step of analyzing a bitstream including encoded coding units, a step of determining a division of the image according to any of the embodiments of the present disclosure, and an image decoding step of decoding the encoded coding units based on the determined division of the image.

[0151] At the decoder side, the maximum BT / TT depth is extended with the boundary CTU / CU, i.e., the boundary multi-type partition maximum depth is used for the decoded boundary CTU. The extension may be simply analyzed from the SPS, or may be derived based on certain conditions. Based on the first embodiment, the maximum BTT depth may be extended with the BTT boundary partition depth. Based on the second embodiment, the maximum BTT depth may be extended with the ratio of the boundary partition block and the BTT boundary partition depth.

[0152] In the encoding and decoding of the video image by the encoder / decoder and the respective encoding / decoding methods, when the partition (or partition determination) is performed according to embodiment 3, the boundary multi-type partition maximum depth is predefined. In this case, the encoded / decoded bitstream may further include an encoding sequence parameter set including the boundary multi-type partition maximum depth. On the decoder side, the device 800 for determining the partition of the image may then be further configured to obtain a second multi-type partition maximum depth from the sequence parameter set.

[0153] FIG. 14 is a conceptual or schematic block diagram illustrating one embodiment of a coding system 300, e.g., a picture coding system 300, comprising a source device 310 configured to provide encoded data 330, e.g., an encoded picture 330, to a destination device 320, e.g., for decoding the encoded data 330.

[0154] The source device 310 comprises an encoder 100 or encoding unit 100, and may additionally, i.e. optionally, comprise a picture source 312, a pre-processing unit 314, e.g. a picture pre-processing unit 314, and a communication interface or unit 318.

[0155] The picture source 312 may include or be any kind of picture capture device, e.g., for capturing real-world pictures, and / or any kind of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any kind of device for obtaining and / or providing real-world pictures, computer-animated pictures (e.g., screen content, virtual reality (VR) pictures), and / or combinations thereof (e.g., augmented reality (AR) pictures). In the following, all these kinds of pictures, as well as any other kind of pictures, will be referred to as "pictures" or "images" unless otherwise specified, and the previous discussion of the term "picture," which encompasses "video pictures" and "still pictures," still applies, unless explicitly specified otherwise.

[0156] A (digital) picture is or can be considered as a two-dimensional array or matrix of samples with intensity values. The samples in the array may also be called pixels (short form of picture element) or pels. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the picture. For color representation, three color components are usually used, i.e. a picture can be represented by or can contain three sample arrays. In an RBG format or color space, a picture comprises an array of corresponding red, green, and blue samples. However, in video coding, each pixel is usually represented in a luminance / chrominance format or color space, for example YCbCr, which includes a luminance component denoted by Y (although in some cases L may be used instead) and two chrominance components denoted by Cb and Cr. The luminance (or luma for short) component Y represents the brightness or gray level intensity (e.g., in the case of a grayscale picture), and the two chrominance (or chroma for short) components Cb and Cr represent the chromaticity or color information components. Thus, a picture in YCbCr format contains a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in RGB format may be converted or transformed into YCbCr format, and vice versa, a process also known as color conversion or color transformation. If the picture is monochrome, the picture may contain only a luminance sample array.

[0157] The picture source 312 may be, for example, a camera for capturing a picture, a memory that contains or stores a previously captured or generated picture, for example a picture memory, and / or any kind of interface (internal or external) for acquiring or receiving a picture. The camera may be, for example, a local or integrated camera integrated in the source device, and the memory may be, for example, a local or integrated memory integrated in the source device. The interface may be, for example, an external interface for receiving a picture from an external video source, for example an external picture capturing device such as a camera, an external memory, or an external picture generating device, for example an external computer graphic processor, computer, or server. The interface may be any kind of interface, for example a wired or wireless interface, an optical interface, conforming to a proprietary or standard interface protocol. The interface for acquiring the picture data 313 may be the same interface as the communication interface 318 or a part of it. The communication interface may be any interface, such as Ethernet, WLAN, Bluetooth, LTE, or a wired or non-wired interface, such as a satellite or optical interface. The transmission may be peer-to-peer or broadcast or multicast.

[0158] To distinguish from pre-processing unit 314 and the processing performed by pre-processing unit 314 , pictures or picture data 313 may also be referred to as raw pictures or raw picture data 313 .

[0159] Pre-processing unit 314 is configured to receive the (raw) picture data 313 and perform pre-processing on the picture data 313 to obtain a pre-processed picture 315 or pre-processed picture data 315. The pre-processing performed by pre-processing unit 314 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal.

[0160] The encoder 100 is configured to receive pre-processed picture data 315 and to provide encoded picture data (further details are described, for example, with reference to FIG. 1).

[0161] The communications interface 318 of the source device 310 may be configured to receive the encoded picture data and transmit it directly to another device, such as the destination device 320 or any other device, for storage or direct reconstruction, or to process the encoded picture data before storing the encoded data 330 and / or before transmitting the encoded data 330 to another device, such as the destination device 320 or any other device, for decoding or storage, respectively.

[0162] The destination device 320 comprises a decoder 200 or a decoding unit 200, and may additionally, i.e. optionally, comprise a communication interface or unit 322, a post-processing unit 326, and a display device 328.

[0163] The communications interface 322 of the destination device 320 is configured to receive the encoded picture data or encoded data 330, for example directly from the source device 310 or from any other source, for example a memory, for example a memory of encoded picture data.

[0164] The communication interface 318 and the communication interface 322 may be configured to transmit and receive the encoded picture data or the encoded data 330, respectively, via a direct communication link between the source device 310 and the destination device 320, e.g., a direct wired or wireless connection, or via any type of network, e.g., a wired (such as based on optical, power line, coaxial, or any other medium) or a wireless network, or any combination thereof, or any type of private network and public network, or any type of combination thereof.

[0165] The communications interface 318 may be configured, for example, to package the encoded picture data into an appropriate format, e.g., packets, for transmission over a communications link or network, and may further include data loss protection and data loss recovery.

[0166] Communications interface 322, which forms a counterpart of communications interface 318, may be configured to depackage encoded data 330, for example to obtain encoded picture data, and may further be configured to perform data loss protection and data loss recovery, including, for example, error concealment.

[0167] Both communication interface 318 and communication interface 322 may be configured as unidirectional communication interfaces, as indicated by the arrow in encoded picture data 330 of FIG. 14 pointing from the source device 310 to the destination device 320, or as bidirectional communication interfaces, and may be configured to, e.g., send and receive messages, e.g., set up connections, confirm and / or retransmit lost or delayed data, including picture data, and exchange any other information related to the communication link and / or data transmission, e.g., encoded picture data transmission.

[0168] The decoder 200 is configured to receive encoded picture data and provide decoded picture data or decoded pictures (further details are described, for example, with reference to FIG. 2).

[0169] The post-processor 326 of the destination device 320 is configured to post-process the decoded picture data, e.g., a decoded picture, to obtain post-processed picture data 327, e.g., a post-processed picture 327. The post-processing performed by the post-processing unit 326 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing, e.g., to prepare the decoded picture data for display, e.g., by a display device 328.

[0170] A display device 328 of destination device 320 is configured to receive the post-processed picture data 327 for displaying the picture, for example to a user or viewer. The display device 328 may be or comprise any type of display for presenting the reconstructed picture, for example an integrated or external display or monitor. The display may comprise, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or any type of other display including beamer, hologram, or 3D / VR glasses.

[0171] 14 illustrates the source device 310 and the destination device 320 as separate devices, an embodiment of the devices may also include both or both functionality, the source device 310 or corresponding functionality and the destination device 320 or corresponding functionality. In such an embodiment, the source device 310 or corresponding functionality and the destination device 320 or corresponding functionality may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0172] As will be clear to a person skilled in the art based on the description, the presence and (exact) division of functions of different units or functions within the source device 310 and / or destination device 320 as shown in FIG. 14 may vary depending on the actual device and application.

[0173] Thus, the source device 310 and the destination device 320 as shown in FIG. 14 are merely exemplary embodiments of the present invention, and embodiments of the present invention are not limited to those shown in FIG.

[0174] The source device 310 and the destination device 320 may include any of a wide range of devices, including any type of handheld or fixed device, such as, for example, a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video gaming console, a video streaming device, or a broadcast receiver device, and may use no operating system at all or any type of operating system.

[0175] 15 is a schematic diagram of a video coding device 1000 according to an embodiment of the present disclosure. The video coding device 1000 is suitable for implementing the disclosed embodiments as described herein, for example as an encoder or decoder. The video coding device 1000 comprises an input port 1010 and a receiver unit (Rx) 1020 for receiving data, a processor, logic unit, or central processing unit (CPU) 1030 for processing data, a transmitter unit (Tx) 1040 and an output port 1050 for transmitting data, and a memory 1060 for storing data. The video coding device 1000 may also comprise optical-electrical (OE) and electrical-optical (EO) components coupled to the input port 1010, the receiver unit 1020, the transmitter unit 1040, and the output port 1050 for outputting or inputting optical or electrical signals.

[0176] The present disclosure may be implemented in an apparatus. Such an apparatus may be a combination of software and hardware. For example, intra-prediction and deblocking filtering may be performed by a general-purpose processor, or a digital signal processor (DSP), or a chip such as a field programmable gate array (FPGA). However, the present invention is not limited to implementation on programmable hardware. It may be implemented on an application specific integrated circuit (ASIC) or by a combination of the hardware components mentioned above.

[0177] The present disclosure further provides a computer readable medium having stored thereon instructions that, when executed by a processing circuit, cause the processing circuit to perform any of the disclosed methods for segmenting, encoding, and decoding an image. The computer readable medium may be any medium having a program stored thereon, such as a DVD, a CD, a USB (flash) drive, a hard disk, or a server storage available via a network.

[0178] The encoder and / or decoder may be implemented in various devices including a TV set, a set-top box, a PC, a tablet or a smartphone etc. It may be a software, an app that implements the method steps.

[0179] In summary, the present disclosure provides an apparatus and method for partitioning an image into coding units. The image is partitioned into hierarchically partitioned coding tree units (CTUs). The hierarchical partitioning includes multi-type partitioning, such as binary tree or quad tree partitioning. For CTUs that are entirely within the image and CTUs on the boundary, respective multi-type partition depths are selected. The present disclosure provides the flexibility of multi-type partitioning in the boundary part of the image. [Explanation of symbols]

[0180] 100 Encoder, video encoding device 110 Split Unit 120 Residual Blocks 125 Reconstructor 130 Conversion Unit 135 Reverse conversion unit 140 quantization units 145 Inverse Quantization Unit 150 Entropy Encoding Units, Entropy Coding Units 160 Loop Filtering Unit 170 Frame Buffer, Reference Picture Buffer 180 Mode Selection Unit 190 intra prediction units 195 Inter Prediction Units 200 Video decoder, decoding device 225 Reconstructor 230 Reverse conversion unit 240 Inverse Quantization Unit 250 Bitstream parsing, entropy decoding and splitting unit 260 Loop Filtering Unit 270 Reference Picture Buffer 290 Intra Prediction Units 295 Inter Prediction Units 300 Picture Coding System 310 Source Device 312 Picture Source 313 Raw Picture Data 314 Preprocessor, Picture Preprocessing Unit 315 Preprocessed Picture Data 318 Communication interface, communication unit 320 Destination Device 322 Communication interface, communication unit 326 Post-processor, Post-processing unit 327 Post-Processed Picture 328 Display Device 330 Encoded Data 800 Apparatus for dividing an image into coding units 810 Image Segmentation, Frame Segmentation, Functional Units 820 Boundary Segmentation 900 Bottom Picture Border 910 Boundary part 920 parts 950 pieces 1000 Video Coding Devices 1010 Input port 1020 Receiver Unit 1030 Processor 1040 Transmitter Unit 1050 Output Port 1060 Memory 1070 Coding Module

Claims

1. 1. An apparatus for dividing an image into coding units, the apparatus comprising a processing circuit, the processing circuit comprising: subdividing the image into coding tree units CTUs including non-border CTUs having a horizontal and vertical size and boundary CTUs having a portion in the image bounded by a horizontal or vertical image border, the portion having a size smaller than the size in a direction perpendicular to the image border; hierarchically partitioning the non-border CTUs and the border CTUs into respective coding units; the hierarchical partitioning of the non-boundary CTU includes a multi-type partition having a maximum non-boundary multi-type partition depth, the partition direction being either the vertical direction or the horizontal direction; the hierarchical partitioning of the boundary CTU includes a multi-type partition having a boundary multi-type partition maximum depth; 23. An apparatus for segmenting an image, comprising:

2. 2. The apparatus for segmenting an image of claim 1, wherein the boundary multi-type partition maximum depth is at least the sum of an adaptive boundary multi-type partition depth and a predefined multi-type partition depth, the adaptive boundary multi-type partition depth being a depth of a multi-type partition whose partition direction is in the direction of the image boundary.

3. The apparatus for segmenting an image according to claim 2 , wherein the predefined multi-type partition depth is equal to the non-boundary multi-type partition maximum depth.

4. The apparatus for dividing an image, as described in claim 2 or 3, wherein the sum further includes a function of a ratio of sizes of a boundary partition block of the boundary CTU in the direction of the image boundary and in the direction perpendicular to the image boundary, the boundary partition block being a block of the adaptive boundary multi-type partition depth.

5. 5. The apparatus for segmenting an image according to claim 4, wherein the function is a binary logarithm.

6. The apparatus for segmenting an image according to claim 1 , wherein the boundary multi-type partition maximum depth is predefined.

7. The apparatus for segmenting an image according to claim 1 , wherein the hierarchical partitioning of the boundary CTUs further comprises a quadtree partitioning.

8. 8. An apparatus for segmenting an image according to claim 1, wherein the bounded multi-type partition maximum depth is greater than or equal to the non-bounded multi-type partition maximum depth.

9. An apparatus for segmenting an image according to any one of claims 1 to 8, an image coding unit configured to encode the coding unit; a bitstream forming unit configured to generate a bitstream including the encoded coding units and partitioning information indicating how the coding tree units are partitioned; 23. An apparatus for encoding images of a video sequence, comprising:

10. The apparatus for segmenting an image is an apparatus according to claim 6, the bitstream further comprises an encoded sequence parameter set including a boundary multi-type partitioning maximum depth; An apparatus for encoding images of a video sequence according to claim 9.

11. a bitstream parser for parsing a bitstream including the encoded coding units; An apparatus for determining a segmentation of an image according to any one of claims 1 to 8, an image decoding unit for decoding the encoded coding unit based on the determined partitioning of the image; 23. An apparatus for decoding images of a video sequence, comprising:

12. The device for determining a segmentation of an image is an device according to claim 6, the bitstream further comprises an encoded sequence parameter set including a boundary multi-type partitioning maximum depth; The apparatus for determining a segmentation of an image is further configured to obtain a second multi-type segmentation maximum depth from the sequence parameter set.

12. Apparatus for decoding an image according to claim 11.

13. 1. A method for dividing an image into coding units, the method comprising the steps of: subdividing the image into coding tree units CTUs including non-border CTUs having a horizontal and vertical size and boundary CTUs having a portion in the image bounded by a horizontal or vertical image border, the portion having a size smaller than the size in a direction perpendicular to the image border; hierarchically partitioning the non-border CTUs and the boundary CTUs into respective coding units, the hierarchical partitioning of the non-boundary CTU includes a multi-type partition having a maximum non-boundary multi-type partition depth, the partition direction being either the vertical direction or the horizontal direction; the hierarchical partitioning of the boundary CTU includes a multi-type partition having a boundary multi-type partition maximum depth; A method comprising:

14. 1. A method for encoding images of a video sequence, the method comprising the steps of: Dividing an image into coding units according to claim 13; an image coding step of encoding the coding unit; generating a bitstream including partitioning information indicating how the encoded coding units and the coding tree units are partitioned; A method comprising:

15. 1. A method for decoding images of a video sequence, the method comprising the steps of: Parsing a bitstream including encoded coding units; said step of determining a segmentation of the image according to claim 13; an image decoding step of decoding the encoded coding units based on the determined partitioning of the image; A method comprising:

16. A computer readable medium having stored thereon instructions which, when executed by a processing circuit, cause the processing circuit to perform the method of any one of claims 13 to 15.

Images