Improvement of mandatory boundary demarcation

Abstract

This invention relates to the division of an image or image slice. [Solution] In particular, the image or image slice is divided into an integer number of coding tree units both vertically and horizontally. The deepest level of division is then the result of the binary division, that is, the remaining boundary portion is forced to be divided at least partially such that one side contains longer coding units than the other side. The longer side is either parallel to the boundary of the image or image slice or overlaps with the boundary of the image or image slice.

Description

Technical Field

[0001] The present disclosure relates to image and / or video encoding and decoding, and more particularly to dividing an image into encoding units.

Background Art

[0002] Current block-based hybrid video coders employ predictive coding. Pictures in a video sequence are sub-divided into pixel blocks, which are then encoded. Instead of encoding the block pixel by pixel, the entire block is predicted using spatially or temporally nearby already-encoded pixels of the block. The encoder further processes only the difference between the block and its prediction. Further processing generally includes conversion of the block pixels to coefficients in a transform domain. Then, to form a bitstream, the coefficients may be further compressed (e.g., by means of quantization) and made more compact (e.g., by entropy coding). The bitstream may further include any signaling information that enables the decoder to decode the encoded video. For example, the signaling may include settings regarding encoder settings such as the size of the input picture, the frame rate, quantization step indication, the prediction applied to the blocks of the picture, etc.

[0003] The difference between a block and its prediction is known as the block residual. More specifically, each pixel in a block has a residual, which is the difference between the pixel's intensity level and its predicted intensity level. The pixel's intensity level is called the pixel value or pixel value. The residuals of all pixels in a block are collectively called the block residuals. In other words, a block has residuals, which are a set or matrix of residuals of all pixels in the block. The residuals are then transformed, quantized, and encoded together with the signaling information. The encoding may include various forms of fixed and variable-length coding, including arithmetic coding or other types of entropy coding.

[0004] In block-based hybrid video coding, each picture is divided into blocks of samples, and multiple blocks within a picture are aggregated to form slices of independently decodeable entities. The blocks to which prediction and / or transformation is applied are called coded units (CUs) or coded blocks (CBs). Coded units may have different sizes.

[0005] For example, in High Efficiency Video Coding (HEVC, also known as H.265), a video frame is subdivided into coding tree units (coding tree blocks, also called CTBs or CTUs). A CTB is a square block of the same size, for example, 64 x 64 samples, with no common parts. Each CTB acts as the root of the coding tree, a quadtree structure that divides the block. A CTB can be further subdivided into coding blocks along the coding tree structure. With respect to a coding block, the type of prediction is determined. A coding block may be further subdivided into smaller transformation blocks to which transformations and quantization are applied.

[0006] Further details regarding the classification in HEVC can be found in V. Sze et al. (eds.), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014, Chapter 3.2.

[0007] Furthermore, International Publication No. 2016 / 090568 demonstrates a binary tree structure for dividing a unit into multiple smaller units using a binary tree structure in addition to a quadtree. Thus, first, the root unit is divided into a quadtree structure, and then the leaf nodes of the quadtree are further divided by a binary tree structure. [Overview of the Initiative] [Means for solving the problem]

[0008] In the HEVC standard, CTUs and CUs at the boundaries of a slice or picture are forced to split using quadtree partitioning until the bottom-right sample of the leaf node enters the slice or picture boundary. This forced quadtree partitioning does not need to be signaled in the bitstream, as long as the partitioning rules are defined (predefined). The purpose of forced partitioning is to allow the encoder and decoder to process pictures and slices whose horizontal and / or vertical sizes are not integer multiples of the horizontal and / or vertical sizes of each CTU.

[0009] In HEVC and International Publication No. 2016 / 090568, the forced partitioning process hierarchically partitions CTUs and CUs at frame boundaries using a quadtree technique without rate-distortion (RC) optimization until the entire current CU falls within the boundaries of a slice or picture. These forced partitions do not need to be signaled within the bitstream. However, partitioning into very small encoded units may introduce further complexity and signaling requirements related to further processing (encoding, decoding) of the small encoded units.

[0010] According to this disclosure, the forced partitioning applies binary tree partitioning at least at the deepest partitioning hierarchy level, or at least at partitioning levels other than the deepest partitioning hierarchy level, such that the coding units do not remain entirely within the boundary portion.

[0011] Such a division results in fewer coding units at the deepest hierarchical levels, thus offering greater efficiency.

[0012] According to an aspect of the invention, there is a device for dividing an image portion into encoding units, the device includes a processing circuit that, during operation, subdivides the image portion into an integer number of encoding tree units having a predetermined size and boundary portions having a size less than a predetermined size in at least a first direction, either horizontal or vertical, and hierarchically divides the boundary portions into encoding units, each containing the deepest encoding unit having different sizes in the horizontal and vertical directions, wherein the deepest encoding unit is smaller in the first direction.

[0013] The advantage of this method is that it avoids too many small CUs on the picture boundaries, which can also lead to more efficient encoding of the boundary areas.

[0014] For example, the processing circuit is configured to partition the boundary portion by binary tree partitioning at least at the deepest level of partitioning.

[0015] In addition to, or as an alternative to, binary tree partitioning, the processing circuit may be configured to partition the boundary portion by quadtree partitioning as well.

[0016] For example, the processing circuit, while operating, divides the boundary portion by quadtree partitioning at the N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchy levels of the partitioning, and then divides the boundary portion by binary tree partitioning at the remaining hierarchy levels of the partitioning.

[0017] This method offers greater flexibility in configuration and further ensures that boundaries are efficiently encoded by avoiding CUs (Censor Units) of numerous small square boundaries.

[0018] In one implementation, the processing circuit demarcates the boundary region during operation until the deepest level of coding units remain entirely within the boundary region.

[0019] In one implementation, the processing circuit divides the boundary area horizontally and vertically using binary tree partitioning during operation.

[0020] Additionally or alternatively, the processing circuit, during operation, divides the corner portions of the boundary areas at each hierarchical level by a quadtree partition.

[0021] As an alternative to mixing quadruple and binary tree partitioning for boundary CTUs, the processing circuit may be configured to partition the boundary areas by binary tree partitioning at all levels of partitioning.

[0022] According to aspects of the invention, an apparatus for encoding images of a video sequence is provided, the apparatus comprising: an apparatus for dividing an image portion into encoding units according to any of the above examples and implementations; an image encoding unit configured to encode the encoding units; and a bitstream forming unit configured to generate a bitstream containing the encoded encoding units and partitioning information indicating how the encoding tree units are partitioned.

[0023] For example, the encoder further includes a rate-distortion optimization unit configured to determine that the boundary portion is hierarchically partitioned by rate-distortion optimization for any level of partitioning, except for the deepest level.

[0024] Another aspect of the present invention is an apparatus for decoding images of a video sequence, comprising: an apparatus for determining the division of the image portion to be decoded into coding units according to any of the above examples and implementations; a bitstream parser for analyzing a bitstream containing division information based on the determined division of the image portion, showing how the encoded coding units and coding tree units are divided; and an image decoding unit for decoding the encoded coding units.

[0025] In the exemplary implementation, the partitioning of the boundary portion into encoded units is a predefined partition, and the partitioning information is not related to the partitioning of the boundary portion.

[0026] The processing circuitry of the decoding and / or encoding device may be configured to partition the boundary portion by quadtree partitioning at the N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchy levels of partitioning, and to partition the boundary portion by binary tree partitioning at the remaining hierarchy levels of partitioning. Furthermore, in exemplary implementations, the partitioning information includes N, or N is derived based on the type of prediction applied to encode the image being processed. N may be derived based on the size of the boundary portion having a size less than a predetermined size in either the horizontal or vertical direction.

[0027] According to an aspect of the present invention, a method for dividing an image portion into coding units, the method comprising: subdividing the image portion into an integer number of coding tree units having a predetermined size and a boundary portion having a size less than the predetermined size in at least a first direction which is either horizontal or vertical; hierarchically classifying the boundary portion into coding units including the deepest coding units having different sizes in their respective horizontal and vertical directions, wherein the deepest coding units are smaller in the first direction.

[0028] In an exemplary embodiment, the classification of the boundary portion further comprises: starting from the current classification level which is the shallowest classification level; successively dividing the boundary portion into the coding units of the current classification level until the coding units of the boundary at the current classification level are completely within the boundary portion; immediately increasing the classification level and further classifying the coding units when the coding units of the current classification level do not fit within the boundary portion, wherein a quadtree classification is used for N of the shallowest classification levels and a binary tree classification is used in other cases, and N is an integer greater than or equal to zero and less than the number of classification hierarchy levels.

[0029] According to another aspect of the present invention, a method for encoding an image of a video sequence, the method comprising: dividing an image portion according to any of the methods described above into coding units; encoding the coding units; and generating a bitstream including the encoded coding units and classification information indicating how the coding tree units are classified.

[0030] A method for decoding images of a video sequence, comprising the steps of: determining the division of the image portion to be decoded into coding units according to any of the methods described above; analyzing a bitstream containing division information based on the determined division of the image portion, showing how the encoded coding units and coding tree units are divided; and decoding the encoded coding units.

[0031] For example, the partitioning step involves partitioning the boundary portion by binary tree splitting at least at the deepest level of partitioning.

[0032] In addition to, or as an alternative to, binary tree partitioning, the partitioning step may also involve partitioning the boundary portion by quadtree partitioning.

[0033] For example, the partitioning step involves partitioning the boundary portion by quadtree partitioning at the N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchy levels of partitioning, and then partitioning the boundary portion by binary tree partitioning at the remaining hierarchy levels of partitioning.

[0034] In one implementation, the demarcation step demarcates the boundary until the deepest level of coding units remain entirely within the boundary.

[0035] In one implementation, the processing circuit divides the boundary area horizontally and vertically using binary tree partitioning during operation.

[0036] Additionally or alternatively, the processing step involves dividing the corners of the boundary portion at each hierarchical level by a quadtree partition.

[0037] As an alternative to mixing quadruple and binary tree partitioning for boundary CTUs, the partitioning step may involve partitioning the boundary portion by binary tree partitioning at all levels of partitioning.

[0038] According to another aspect of the present invention, a computer-readable medium is provided for storing instructions that cause a processor to perform a method according to any of the above aspects and examples when executed by the processor.

[0039] Exemplary embodiments are described in more detail below with reference to the attached figures and drawings. [Brief explanation of the drawing]

[0040] [Figure 1] Block diagram illustrating the structure of a video encoder. [Figure 2] Block diagram illustrating the structure of a video decoder. [Figure 3] This is a schematic diagram illustrating an example of the quad tree partitioning used by HEVC. [Figure 4] This is a schematic diagram illustrating an example of the distinction between quad trees and binary trees. [Figure 5] This is a schematic diagram showing a forced quadtree split at the boundary. [Figure 6] This is a block diagram showing an example of a device that may operate as a split unit of an encoder or decoder. [Figure 7] This is a schematic diagram showing the division of the boundary area. [Figure 8] This is a schematic diagram illustrating an example of a forced binary tree partitioning along a vertical boundary. [Figure 9(a)] This is a schematic diagram illustrating an example of a division along a vertical boundary. [Figure 9(b)] This is a schematic diagram illustrating an example of a division along a horizontal boundary. [Figure 9(c)] This is a schematic diagram showing an example of a division along a combined boundary. [Figure 10] This flowchart illustrates the syntax for determining exemplary classifications. [Modes for carrying out the invention]

[0041] The present invention relates to the division of an image into smaller units for further processing. Such division may be advantageously used in the encoding and decoding of still or video images. Exemplary video coders and decoders capable of performing the divisions according to this disclosure are described below.

[0042] Figure 1 shows an encoder 100 including an input for receiving an input block of frames or pictures of a video stream and an output for providing an encoded video bitstream. The term “frame” in this disclosure is used as a synonym for “picture.” However, it should be noted that this disclosure is also applicable to fields where interlacing is applied. Generally, a picture contains m x n pixels. These correspond to image samples and may each contain one or more color components. For brevity, the following description refers to pixels meaning luminance samples. However, it should be noted that the partitioning techniques of this disclosure can be applied to any color component, including chrominance or components of a color space such as RGB. On the other hand, it may be beneficial to perform partitioning with respect to just one component and then apply the determined partitioning to further (or all) of the remaining components.

[0043] The encoder 100 is configured to apply segmentation, prediction, transformation, quantization, and entropy coding to the video stream.

[0044] In the splitting unit 110, the input video frame is further divided before encoding. The blocks to be encoded are not necessarily the same size. A single picture may contain blocks of different sizes, and the block rasters of different pictures in a video sequence may also be different. In particular, each video image (picture) is first subdivided into CTUs of the same fixed size. The CTU size may be predetermined, for example, by a standard. In HEVC, a size of 64 x 64 is used. However, this disclosure is not limited to a standardized fixed size. It may be advantageous to provide a CTU size that is set in the encoder and provided as a signaling parameter in the bitstream. For example, different CTU sizes may be beneficial for each different picture size and / or type of content. The CTU size may be signaled at any signaling level, for example, it may be common to the entire video sequence or a part thereof (i.e., multiple pictures), or it may be individual for each picture. Accordingly, the CTU size may be signaled, for example, in the picture parameter set (PPS), known from the current codec (H.264 / AVC, H.265 / HEVC), or in the sequence parameter set (SPS), or in the video parameter set (VPS), or in a similar parameter set. Alternatively, the CTU size may be specified in the slice header or at any other level. The CTU size may take values ​​other than 64 x 64. The CTU size may be, for example, the size of 128 x 128 samples. In general, it may be beneficial to provide a CTU size in the format of 2^n, where n is an integer greater than 2, in order to perform a hierarchical partition of a quadtree by a binary tree.

[0045] Figure 3 shows the partitioning of pictures into CTUs and CTUs into CUs from V. Sze et al. (eds.), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014. The partitioning follows a quadtree structure to accommodate various local features. On the left, Figure 3 shows the CTUs hierarchically partitioned according to the quadtree structure on the right. In particular, the coding tree defines the syntax that dictates the subpartitioning of CTUs into CUs. Similar to CTUs, CUs consist of square blocks of samples and syntax associated with these sample blocks. Thus, the partitioning is performed hierarchically, starting with a CTU (depth 0 of the hierarchy) which may or may not be subpartitioned into four (quadtree) CUs at a depth of 1 of the hierarchy. In Figure 3, the CTU is not further divided and is therefore divided into CUs 8 and 16 at the first hierarchy depth (level) that form leaves of the quadtree, and two further CUs that are further divided into CUs at hierarchy depth 2 (depth 2 CUs). In particular, the top left depth 1 CU is further divided into depth 2 CUs 1, 2, and 7 that form leaves of the quadtree, and another CU that is further divided into depth 3 CUs 3, 4, 5, and 6, all of which are leaves. Similarly, the bottom left depth 1 CU is further divided into depth 2 CUs 13, 14, and 15, which are also leaves of the quadtree, and the remaining CUs that are further divided into level 3 CUs 9, 10, 11, and 12, all of which are leaves and are therefore not further divided.

[0046] Table 1 below shows an example syntax for quadtree partitioning in HEVC.

[0047] [Table 1]

[0048] Specifically, at the CTU level, a flag named split_cu_flag is included in the bitstream, indicating whether a complete CTU forms a CU or whether the CTU is split into four equally sized blocks corresponding to square sample blocks. If the CTU is split, for each of the resulting blocks, another split_cu_flag is sent, specifying whether the block represents a CU or whether the block is further split into four equally sized blocks. This hierarchical sub-splitting continues until none of the resulting blocks can be further sub-split. The minimum size of a CU is signaled within the sequence parameter set, and this minimum size can be greater than or equal to 8 x 8 samples and less than or equal to the size of the CTU. When the minimum CU size is reached in the hierarchical sub-splitting process, no split flag is sent for the corresponding blocks; instead, it is assumed that these blocks will not be further split. In a typical HEVC encoder configuration, the maximum range of supported CU sizes is utilized so that CUs ranging from 8 x 8 samples to 64 x 64 samples can be used. CUs within a CTU are encoded in depth-first order. This coding order is also called a z-scan. This ensures that, with respect to each CU, all samples above and to the left of the CU have already been coded, except for CUs at the top or left boundary of the slice, and therefore the corresponding samples can be used for intra-prediction, and the associated coding parameters can be used to predict the coding parameters of the current CU.

[0049] In other words, split_cu_flag[ x0 ][ y0 ] specifies whether the coding unit is split into coding units having half the horizontal and vertical size. Array indices x0, y0 specify the position (x0, y0) of the top-left luma sample of the coding block being considered relative to the top-left luma sample of the picture. When split_cu_flag[ x0 ][ y0 ] does not exist, the following applies in the decoder: - If log2CbSize (a parameter specifying the encoded block size) is greater than MinCbLog2SizeY (a parameter specifying the minimum configurable encoded unit size), the value of split_cu_flag[x0][y0] is presumed to be equal to 1. - Otherwise (log2CbSize is equal to MinCbLog2SizeY), the value of split_cu_flag[x0][y0] is presumed to be equal to 0.

[0050] The array CtDepth[x][y] specifies the depth of the coding tree with respect to the Luma coding block covering position (x, y). When split_cu_flag[x0][y0] is equal to 0, CtDepth[x][y] is presumed to be equal to cqtDepth with respect to x = x0..x0 + nCbS - 1 and y = y0..y0 + nCbS - 1.

[0051] Figure 4 shows the divisions of a mixed quadtree and binary tree. The quadtree divisions are shown by solid lines, while the binary tree divisions are shown by dashed lines. The labels 1 or 0 on the nodes representing coding units further divided by the binary tree indicate whether the binary division is applied vertically or horizontally, respectively.

[0052] The horizontal and vertical dimensions of the video picture in a luma sample must be integer multiples of the minimum CU size in the luma sample, as transmitted within the sequence parameter set, but they do not need to be integer multiples of the CTU size. If the horizontal or vertical dimensions of the video picture do not represent integer multiples of the CTU size, it is assumed that the CTU of the boundary will be split until the boundary of the resulting block coincides with the boundary of the picture. No splitting flag is transmitted for this forced splitting, but the resulting block may be further split using the quadtree syntax described above. CUs outside the picture region are not encoded.

[0053] This partition is shown in Figure 5. Specifically, Figure 5 shows the frame boundary, and the 56 lines (length of 128 samples) above the frame boundary are the slice or image boundary portion. The portion of the CTU below the frame boundary may belong to another slice, or it may not exist at all, for example, if the frame boundary is the bottom boundary of the picture. As can be seen, the forced quadtree partition is applied to 128 x 56 samples.

[0054] In HEVC, the subdivisions of the saturation CTB are always aligned with the subdivisions of the respective luma CTBs. It should be noted that this disclosure may, but is not limited to, treating the saturation components in the same way. Independent subdivisions of different color components are also possible.

[0055] After image splitting is performed in the splitting unit 110, transformation, quantization, and entropy coding are performed by the transformation unit 130, the quantization unit 140, and the entropy coding unit 150, respectively, to generate an encoded video bitstream as output.

[0056] A video stream may contain multiple frames. For example, a block of the first frame of the video stream is intra-encoded by the intra-prediction unit 190. The intra-frame is encoded using only the information from that frame, and therefore can be decoded independently of the other frames. Thus, the intra-frame can provide an entry point to the bitstream, for example, for random access. Other blocks of frames in the video stream may be inter-encoded by the inter-prediction unit 195, and each block of the inter-encoded frame is predicted from another frame (reference frame), for example, a block of a previously encoded frame. The mode selection unit 180 is configured to select whether a block of a frame should be intra-predicted or inter-predicted, that is, whether the block is processed by the intra-prediction unit 190 or by the inter-prediction unit 195. The mode selection unit 180 also controls the intra-parameters of the inter-prediction. To enable the refreshing of image information, an inter-encoded frame may contain not only inter-encoded blocks but also one or more intra-encoded blocks. In contrast, an intraframe contains only intra-encoded blocks and not inter-encoded blocks. Intraframes may be inserted into a video sequence (for example, regularly, i.e., after every certain number of interframes) to provide entry points for decoding, i.e., points from which the decoder can begin decoding without using information from previous frames.

[0057] The intra-prediction unit 190 is a block prediction unit. To perform spatial or temporal prediction, the encoded blocks may be further processed by the inverse quantization unit 145 and the inverse transform unit 135. After block reconstruction by the reconstructor 125, the loop filtering unit 160 may be applied to further improve the quality of the decoded image. The reconstructor 125 adds the decoded residuals to the predictors to obtain the reconstructed blocks. The filtered blocks then form a reference frame, which is then stored in the frame buffer 170. Such a decoding loop (decoder) on the encoder side has the advantage of producing a reference frame that is the same as the reference picture reconstructed on the decoder side. Thus, the encoder and decoder sides operate in corresponding ways. The term “reconstruction” as used herein refers to obtaining the reconstructed blocks by adding the decoded residual blocks to the prediction blocks.

[0058] The interprediction unit 195 receives as input a block of the current frame or picture to be intercoded, and one or more reference frames or pictures from the frame buffer 170. Motion estimation and motion compensation are performed by the interprediction unit 195. Motion estimation is used, for example, to obtain a motion vector and reference frames based on a cost function. Motion compensation then describes the current block of the current frame by translating the reference block of the reference frame to the current frame, i.e., by the motion vector. The interprediction unit 195 selects a prediction block (i.e., predictor) for the current block from a set of candidate blocks (i.e., candidate predictors) in one or more reference frames such that the prediction block minimizes the cost function. In other words, the candidate block that minimizes the cost function is used as the prediction block for the current block.

[0059] For example, the cost function may be a measure of the difference between the current block and the candidate block, i.e., a measure of the residual of the current block relative to the candidate block. For example, the cost function may be the sum of absolute differences (SAD) between all pixels (samples) of the current block and all pixels of the candidate block in the candidate reference picture. However, generally, any similarity metric such as mean squared error (MSE) or structural similarity metric (SSIM) may be employed.

[0060] However, the cost function may also be the number of bits required to encode such interblocks and / or the resulting distortion from such encoding. Thus, a rate-distortion optimization procedure may be used to determine the choice of motion vector, as well as encoding parameters such as whether to use inter-prediction or intra-prediction with respect to blocks in general and in what settings.

[0061] The intra-prediction unit 190 takes as input a block of the current frame or picture to be intra-encoded, and one or more reference samples from an already reconstructed region of the current frame. The intra-prediction then describes the pixels of the current block of the current frame by a function of the reference samples of the current frame. The intra-prediction unit 190 outputs a predicted block with respect to the current block, which advantageously minimizes the difference between the current block to be encoded and its predicted block, i.e., the predicted block minimizes the residual block. Minimization of the residual block can be based, for example, on a rate-distortion optimization procedure. In particular, the predicted block is obtained as a directional interpolation of the reference samples. The direction may be determined by rate-distortion optimization and / or by computing the measure of similarity described above in relation to the intra-prediction.

[0062] Next, the difference between the current block and its prediction, i.e., the residual block, is transformed by the transformation unit 130. The transformation coefficients are quantized by the quantization unit 140 and entropy coded by the entropy coding unit 150. The thus generated coded video bitstream includes intracoded and intercoded blocks, as well as corresponding signaling (such as mode indication, motion vector indication, and / or intra-prediction direction). The transformation unit 130 may apply a linear transformation such as the Discrete Fourier Transform (DFT) or the Discrete Cosine Transform (DCT). Such a transformation to the spatial frequency domain has the advantage that the resulting coefficients generally have higher values ​​at lower frequencies. Thus, after effective coefficient scanning and quantization (such as zigzag), the resulting sequence of values ​​generally has some larger values ​​at the beginning and end of a sequence of zeros. This allows for even more efficient coding. The quantization unit 140 performs lossy compression by reducing the resolution of the coefficient values. Next, the entropy coding unit 150 assigns a binary codeword to the coefficient value. The codeword is written to a bitstream called the coded bitstream. The entropy coder also encodes signaling information (not shown in Figure 1), which may include encoding of the partition flag syntax shown above.

[0063] Figure 2 shows an example of a video decoder 200. The video decoder 200 includes, in particular, a reference picture buffer 270 and an intra-prediction unit 290, which is a block prediction unit. The reference picture buffer 270 is configured to store at least one reference frame reconstructed from the encoded video bitstream of the encoded video bitstream. The intra-prediction unit 290 is configured to generate a prediction block, which is an estimate of the block to be decoded. The intra-prediction unit 290 is configured to generate this prediction based on a reference sample taken from the reference picture buffer 270.

[0064] The decoder 200 is configured to decode the encoded video bitstream generated by the video encoder 100, but it is preferable that both the decoder 200 and the encoder 100 produce identical predictions for each block being encoded / decoded. The features of the reference picture buffer 270 and intra prediction unit 290 are similar to those of the reference picture buffer 170 and intra prediction unit 190 in Figure 1.

[0065] The video decoder 200 includes additional units that are also present in the video encoder 100, such as an inverse quantization unit 240, an inverse transform unit 230, and a loop filtering unit 260, which correspond to the inverse quantization unit 140, an inverse transform unit 150, and a loop filtering unit 160 of the video decoder 100, respectively.

[0066] The bitstream analysis, entropy decoding, and partitioning unit 250 is configured to analyze and decode the received encoded video bitstream to obtain quantized residual transformation coefficients and signaling information. The quantized residual transformation coefficients are supplied to the inverse quantization unit 240 and the inverse transformation unit 230 to generate residual blocks. The residual blocks are added to the prediction blocks in the reconstructor 225, and the resulting sum is supplied to the loop filtering unit 260 to obtain decoded video blocks. The decoded video frames are stored in the reference picture buffer 270 and can serve as reference frames for interpretation. The signaling information analyzed and decoded from the bitstream may generally include control information related to frame partitioning. To further analyze and decode the image correctly, the control information is used to reconstruct the partitioning of the image to the encoding units in order to correctly assign the subsequently decoded data to their respective encoding units.

[0067] In general, the intra-prediction units 190 and 290 in Figures 1 and 2 can use reference samples from already encoded regions to generate prediction signals for blocks that need to be encoded or decoded.

[0068] The bitstream analysis, entropy decoding, and splitting unit 250 receives an encoded bitstream as its input. First, the bitstream may be analyzed, i.e., the signaling parameters and residuals are extracted from the bitstream. The syntax and semantics of the bitstream may be defined by a standard so that the encoder and decoder can work in an interoperable manner.

[0069] This disclosure presents a novel method for coerced boundary demarcation in block-based hybrid video coding. In particular, the coerced boundary demarcation is performed, at least in part, by using a binary tree (BT) structure. In the inventors' view, since most boundary blocks may use the same prediction mode, coerced quadtree (QT) demarcation of frame boundary CTUs is not always necessary. Instead of coerced QT demarcation, in one embodiment, a coerced horizontal BT for the lower boundary portion and / or a coerced vertical BT for the right boundary portion are introduced (assuming the upper and lower adjacent portions have already been reconstructed in the order of processing CTUs from left to right and top to bottom).

[0070] In particular, as shown in Figure 6, according to the embodiment, there is an apparatus 600 (which may be implemented in the above-mentioned units 110, 250) for dividing an image portion into encoding units (610), which includes a processing circuit that, during operation, divides the image portion into an integer number of encoding tree units having a predetermined size and boundary portions having a size less than a predetermined size in at least one first direction, which is either horizontal or vertical. Furthermore, during operation, the processing circuit hierarchically divides the boundary portions into encoding units (630) that include the deepest encoding units having different sizes in the respective horizontal and vertical directions, with the deepest encoding units being smaller in the first direction.

[0071] The CTUs obtained by the frame division 610 may be further divided hierarchically (620). This division may be carried out in any way, for example, as shown in Figures 3 and 4 and described above with reference to those figures.

[0072] Figure 6 shows the internal structure of the circuit of device 600. This circuit may be any kind of hardware and software having functional units 610, 620, and 630 for dividing the frame into CTUs, partitioning the CTUs, and partitioning the boundary portions. These units may be implemented, for example, in a single processor. However, the invention is not limited to such applications, and these units may also be implemented by separate hardware parts.

[0073] Boundaries may exist only on vertical boundaries (if the size of a vertical picture is not an integer multiple of the vertical CTU size), only on horizontal boundaries (if the size of a horizontal picture is not an integer multiple of the horizontal CTU size), or on both vertical and horizontal boundaries (if the sizes of horizontal and vertical pictures are not integer multiples of their respective horizontal and vertical CTU sizes).

[0074] The typical picture to be encoded has a size that is a multiple of the CTU. Figure 7 visualizes an example of the boundary 700 and the corresponding (filled) boundary portion 710 of the picture below. Portion 750 shows the rest of the image, which has sizes that are integer multiples of the CTU in the vertical and horizontal directions. In particular, the vertical size of the CTU is denoted as 770V, while the horizontal size of the CTU is denoted as 770H. As can be seen in Figure 7, the boundary portion in this example is an integer multiple of the CTU size 770H in the horizontal direction. However, vertically, boundary portion 710 has a size that is the remainder obtained by dividing the vertical picture size by the vertical CTU size 770V. Portion 720 is merely hypothetical and shows the difference between the height of the boundary portion and the CTU size. It should be noted that in this implementation, the CTU is square, and therefore sizes 770H and 770V are the same. However, the disclosure is not limited thereto, and the vertical and horizontal sizes of the CTU may be different.

[0075] To encode (and correspondingly decode) the boundary portion, the boundary portion 710 in Figure 7 is divided into incomplete CTUs, i.e., portions having a horizontal size of 770H and a vertical size of 770V smaller than the CTU size. These incomplete CTUs are further forcibly subdivided as shown in Figure 8.

[0076] Figure 8 shows an example where the boundary is at a horizontal boundary, and therefore the partitioning is performed vertically. In particular, the left side of Figure 7 shows a boundary corresponding to the width of the CTU (in this example, its width is 128 samples) but a height lower than the height of the CTU (only 56 samples). The vertical boundary is forced to partition by binary tree partitioning. Specifically, a unit with a depth of 1, having 64 samples, does not fit the high boundary of 56 samples, so the largest unit size that fits the length of 56 samples is a unit with a depth of 2, having a size of 32 samples. The remaining 24 samples are further partitioned into a unit with a depth of 3, having a size of 16 samples and the remaining portion of 8 samples (corresponding to the size of a 4 CU), and the remaining portion reaches the boundary of the picture, and therefore no further partitioning is necessary. In this example, partitioning is not applied to the horizontal boundary. In summary, a boundary unit having a CTU size in one direction and a smaller size in the other direction is partitioned by a binary tree, that is, by hierarchically dividing the boundary unit into two parts, one having the next shallowest CU size to fit the smaller size, and the remaining part being further divided until the deepest CU reaches the frame boundary. This partitioning of a 128 x 56 CTU portion results in three boundary encoding units having sizes of 128 x 32, 128 x 16, and 128 x 8, respectively.

[0077] In summary, according to one implementation, the processing circuit is configured to partition the boundary parts at all levels of the partitioning using binary tree partitioning.

[0078] On the right side of Figure 8, an example of partitioning is shown where a quadtree partition is first applied at a hierarchy depth of 1 (hierarchy depth 0 corresponding to the CT size), resulting in two CUs of size 64 x 56, which are then further divided into pairs of CUs of sizes 64 x 32, 64 x 16, and 64 x 8.

[0079] The example in Figure 8 illustrates the division of the lower boundary. However, this disclosure is equally applicable to the division of any other boundary, such as the right boundary. When the subdivision of a picture into a CTU is performed from left to right and from top to bottom, the boundary portion is then located below and / or to the right of the picture. However, the division may also be performed from bottom to top and from right to left, in which case the top and left boundaries must be divided, which is possible in the same way as illustrated with reference to Figure 8. Similarly, the example above illustrates the division of such a CTU starting from the top of the incomplete CTU. However, generally speaking, it is also possible to divide an incomplete CTU upwards from the boundary of the picture (frame).

[0080] As shown on the right side of Figure 8, an incomplete CTU partition may be a mixed QT and BT partition. The processing circuit may be configured to partition the boundary portion by binary tree partitioning at least at the deepest partitioning level. This has the advantage that larger boundaries overlapping the frame boundary do not need to be partitioned into the smallest units. Too many or too few CUs on the frame boundary can result in greater encoding complexity and larger bitstreams. In this embodiment, it is noted that as long as the deepest CU is larger on one side than on the other (i.e., obtained as a result of binary tree partitioning), the hierarchy of the remaining depths may be determined in an arbitrary way and do not have to be forced in a predefined way. Rather, the partitioning may be performed by RD optimization with corresponding control information that allows the decoder to reconstruct the same partitioning. The partitioning may also be partially forced and partially available for RD optimization.

[0081] In other words, the processing circuit can also be configured to partition the boundary parts by quadtree partitioning. Therefore, there may be predefined mandatory QT or BT partitions for each depth, which can be configured by the standard or by control information contained in the bitstream.

[0082] For example, a forced BT partitioning is performed only after a certain number of forced QT levels. This may be achieved by configuring the processing circuit to partition the boundary portion by quadtree partitioning at the N shallowest levels, where N is a non-negative integer less than the number of hierarchy levels of partitioning. The processing circuit is further configured to partition the boundary portion by binary tree partitioning at the remaining hierarchy levels of partitioning. Referring to Figure 8, the left side shows no QT partitions being performed.

[0083] Therefore, the control information in the bitstream may include a QT / BT indicator that specifies whether QT is applied and, if so, how many levels are QT partitioned. Correspondingly, such a QT / BT indicator may take multiple values, such as values ​​corresponding to 0 and N when QT partitioning is not applied (as shown on the left in Figure 8). Such signaling is shorter than the signaling for each CU at each level, regardless of whether QT partitioning is applied, BT partitioning is applied, or no partitioning is applied. However, the present disclosure may use any kind of signaling.

[0084] Alternatively, or in combination with control signaling, the level of forced QT can be adaptively selected based on the time ID or slice type. This approach has the advantage of requiring no explicit signaling (or less explicit signaling). Furthermore, the forced segmentation may be set adaptively, at least in relation to the type of prediction.

[0085] HEVC defines time identifiers (IDs) that indicate levels within a hierarchical structure of temporal prediction (dependencies between inter-predicted frames). Slice types can include, for example, intra-predicted slices, inter-predicted slices that apply predictions based on only one reference picture, and inter-predicted slices that apply bidirectional predictions.

[0086] For example, if the time identifier is 0 (meaning 1 picture), or if the slice type is an intra-predicted slice, the classification needs to be more accurate with respect to higher time IDs (inter-predicted pictures) or P / B slices (inter-predicted slices).

[0087] The parameters, time IDs, or slice types described above are merely illustrative. In general, further and / or different encoding parameters may be applied. Furthermore, instead of slice types, the prediction mode of CUs within adjacent CTUs may be used to determine the demarcation of boundary portions based on CTUs (CTUs of incomplete boundaries).

[0088] On the other hand, signaling partitions may provide a greater degree of freedom in selecting appropriate partitions to improve overall coding efficiency. For example, partitions may be adjusted by optimizing the cost function (e.g., by RD optimization) and signaled within the bitstream to indicate how many levels of forced QT are used for the decoder (i.e., signaling whether QT is used at all, as described above, or signaling N).

[0089] Alternatively, the (incomplete) CTU at the boundary is first partitioned by RD optimization down to the second-to-last hierarchical level (depth) or to the signaled hierarchical level M. Then, the remaining deepest level blocks at the frame boundary use forced BT partitioning or a combination of forced QT and forced BT partitioning.

[0090] Figure 9 shows three examples of boundary demarcation that is at least partially enforced.

[0091] When a CTU / CU is at the lower boundary, forced horizontal BT partitioning may be used without repeated RD optimization and signaling of partition flags (by partitioning the vertical boundary). Forced horizontal BT partitioning is terminated when the lower right sample of a leaf node is within the lower boundary of the slice / picture. After forced horizontal BT partitioning, the forced partitioned CU from the boundary CTU may be further partitioned in the encoder based on the RD cost, and further partition flags may be signaled from the encoder to the decoder. The portion of Figure 9(a) shows an example of a lower boundary of 128 x 56 samples partitioned by forced horizontal BT (i.e., partitioning the vertical boundary).

[0092] Similarly, with respect to the CTU / CU at the right boundary, a forced vertical BT partitioning is used, presumably without repeated RD cost checks and without signaling of partition flags, until the lower right sample of the leaf node enters the right boundary of the slice / picture. After the forced vertical BT partitioning (of the horizontal boundary), the forced partitioned CU from the boundary CTU may be further partitioned by RD optimization in the encoder, and further partition flags are signaled from the encoder to the decoder. The portion of Figure 8(b) shows an example of a right boundary of length 56 x 128 samples partitioned by forced horizontal BT.

[0093] In another example, for a CTU / CU at the bottom-right corner boundary, a forced hierarchical QT partitioning is first used without any repeated signaling. When the bottom-right sample of the current CU is at the bottom or right boundary, further forced horizontal or vertical BT partitioning is repeated until the bottom-right sample of the leaf node falls within the slice / picture boundary. If, instead, the bottom-right sample of the current CU is further at a corner boundary, further forced QT partitioning is used until the bottom-right sample of the leaf node falls within the slice / picture frame. The forced partitioned CU obtained by partitioning from the boundary CTU may be further partitioned by RD optimization in the encoder, and a further partitioning flag may be signaled from the encoder to the decoder. Part 9(c) of Figure 9 shows an example of a bottom-right corner boundary of 56 x 56 samples partitioned by forced QT and BT.

[0094] In particular, section 9(c) of Figure 9 shows the CTU of the incomplete corner boundary of the 56 x 56 sample. The first depth level of the 64 x 64 sample after QT splitting is not within the boundary. The 32 x 32 sample at the second depth level after QT is within the boundary, but leaves behind a horizontal corner portion (lower left CU of the 32 x 24 sample), a vertical corner portion (upper right CU of the 24 x 32 sample), and another corner portion (lower right CU of the 24 x 24 sample) that will be further split.

[0095] In this example, the CUs of the horizontal and vertical boundaries are further binary-tree partitioned such that the deepest level units have the longer side along the picture / slice boundary. The corner sections may be further partitioned in the same way as described above for the higher level corner sections, i.e., by quadtree partitioning.

[0096] In other words, the processing circuit may, during operation, divide the boundary portion horizontally and vertically by binary tree partitioning. This is especially true when the boundary portion extends along the boundaries of both picture / slice, the vertical boundary, and the horizontal boundary, and includes corner portions.

[0097] In the case of corners, the processing circuit divides the corner portion of the boundary at each hierarchical level during operation by quadruple tree partitioning. For example, in part 9(c) of Figure 9, with respect to the corner boundary CU 960, where both sides have a size of 56 samples, four levels of forced QT are required (QT division of a 128 x 128 CTU resulting in four 64 x 64 CUs with the top left CU containing a 56 x 56 corner, QT division of a 64 x 64 CU resulting in four 32 x 32 CUs with the bottom right CU containing a 24 x 24 corner, QT division of a 32 x 32 CU resulting in four 16 x 16 CUs, with the bottom right of those 16 x 16 CUs containing an 8 x 8 corner, and the final step of QT division of the bottom right 16 x 16 incomplete CU into four 8 x 8 CUs, i.e., the smallest size partition where the top left partition is in the picture and the other partitions are not).

[0098] On the other hand, for the lower boundary CU 950, two levels of forced QT and two subsequent levels of forced BT are required (QT partitioning of the 128 x 128 CTU, upper left QT partitioning of the 64 x 64 CU, bipartition of the 32 x 32 CU into two 32 x 16 CUs on the lower boundary of the 32 x 24 boundary portion, and the final step of BT partitioning the lower 32 x 16 CU into two 32 x 8 CUs where only the upper CU is within the picture boundary).

[0099] Regarding corner CU 960, the set QT level N is not taken into consideration, and corner CUs are forced to split by QT at all hierarchical levels. In particular, for corners, if the bottom-right sample of the leaf node (an incomplete CU, i.e., partially within the boundary portion) is outside both the right and bottom boundaries, this current incomplete CU is recognized as a bCorner (Figure 10, step 1050, "Yes"). If a bCorner applies to the current CU, the current CU is further split using a forced QT. The splitting ends when the resulting leaf node is entirely within the boundary portion, or when the corner portion is at the bottom or right boundary (bBottom, bRight, i.e., the bottom-right boundary of the current CU is outside only the boundary portion, i.e., the bottom or right boundary of the picture or slice).

[0100] In this example, the forced QT partitioning scheme is the same for N=0, 1, or 2, because CU 960 always requires forced QT, and CU 950 is inevitably partitioned by QT at the first two levels. This is because the CU at depth 0 (CTU) and the CU at depth 1 (64 x 64 samples) are also corner CUs. Only after partitioning the 64 x 64 CUs, at depth 2, are four CUs, one of which is not a boundary CU at all (top left), two of which are bottom boundary CUs and right boundary CUs, and only one is a corner CU.

[0101] If N=3, another forced QT is required, and then the forced partitioning scheme is changed with respect to CU 950 so that the 32 x 32 CU in the lower left is further QT partitioned instead of BT partitioned. The partitioning of CU 960 remains the same.

[0102] According to this disclosure, a combination of forced QT and forced BT may be used as already briefly considered above. In particular, detailed demarcation may be beneficial for image quality with respect to some complex textures or motions on the boundary. Therefore, a combined forced demarcation by QT and BT is also provided. In particular, in an implementation of one combination of forced QT and forced BT, forced BT demarcation is applied only after a certain number N levels of forced QT. Under other settings, if QT demarcation is allowed after BT demarcation, forced QT demarcation may be performed after forced BT.

[0103] In combined implementations of forced BT and forced QT, there are several ways to define the number of forced QT levels. When predefined levels of forced QT are used, first, all boundary CTUs are forced to partition by the same QTBT structure, and further partitioning may be performed based on RD optimization. The right side of Figure 8 shows an example of one level of QT and three levels of BT partitioning at the lower boundary of a 128 x 56 sample in an incomplete 128 x 128 CTU.

[0104] The level of forced QT may be adaptively selected based on encoder parameters such as time ID or slice type. Certain slice types and lower time IDs may require more detailed subdivisions. Furthermore, the level of forced QT may be adjusted by rate-distortion optimization and signaled within the bitstream to indicate how many levels of forced QT are used. Signaling of the number of levels in the CTU level may be used to indicate how many levels of forced QT were used for the CTU. In addition, signaling of the number of levels in the slice header may be used to indicate how many levels of forced QT were used for the current slice. Signaling of the number of levels in the SPS may be used to indicate how many levels of forced QT were used for the encoded video sequence. Signaling of the number of levels in the PPS may be used to indicate how many levels of forced QT were used for the encoded picture.

[0105] The number of forced QT levels may be determined based on any other method. For example, the entropy of the original sample within a slice / frame / boundary region may be used. Entropy is calculated by summing the terms p_i * log2(p_i) with respect to i, where p_i is the probability of the intensity value I of the sample within a particular region. The symbol "*" means multiplication, and "log2" means base 2 logarithm. The higher the entropy, the greater the likelihood that a structure exists within the slice / frame / boundary region, and therefore a smaller CU may be appropriate. However, these are merely examples, and in principle, the present invention is not limited to any particular method by which the number of forced QT levels is determined. Any other measure, such as the variance of edge presence, may be used.

[0106] The processing circuit divides the boundary region during operation until the deepest level of encoding units remains within the boundary region.

[0107] The present disclosure provides a method for dividing an image portion into coding units. The method includes the step of subdividing the image portion into an integer number of coding tree units having a predetermined size and boundary portions having a size less than the predetermined size in at least one first direction, which is either horizontal or vertical. A hierarchical division of the boundary portions into coding units, each containing the deepest coding unit having different sizes in the horizontal and vertical directions, respectively, wherein the deepest coding unit is smaller in the first direction.

[0108] In one implementation, boundary partitioning includes the steps of: starting from the current partitioning level, which is the shallowest partitioning level; sequentially dividing the boundary into coding units of the current partitioning level, unless the coding units of the current partitioning level reach the boundary; and immediately if the coding units of the current partitioning level do not fit into the boundary, raising the partitioning level and further partitioning the coding units, wherein quadtree partitioning is used for the N shallowest partitioning levels and binary tree partitioning is used in other cases.

[0109] Sequential partitioning means that the partitioning is performed from the shallowest level to the deepest level. In particular, forced partitioning is performed at each hierarchical level with respect to the encoding units at the boundary (boundary CUs), that is, such CUs include samples within the boundary of the picture / slice (within the boundary portion) and portions outside the boundary of the picture / slice. Here, the term CU refers to a CU that has the size resulting from the partitioning, but is incomplete because it contains fewer samples than its size would suggest, i.e., because it is partially outside the picture / slice being partitioned.

[0110] Therefore, a forced partition is performed to roughly divide the boundary portion from the largest CU to the smallest CU so that the entire boundary portion is partitioned. As already discussed above, partitioning information is not required for forced partitioning, and the partitioning information can be predefined. If there are several QT levels to be forced, N may be signaled or predefined. A predefined N means that both the encoder and decoder have knowledge of such an N, which may be defined, for example, by a standard.

[0111] However, after performing a forced partition, the CUs in the forcibly partitioned boundary portion may be further partitioned. This may be done based on rate-distortion optimization and therefore further signaled within the bitstream. In particular, with respect to CUs in the boundary portion that are larger than the minimum CU size, one or more partition flags may be signaled indicating whether such CUs are further partitioned and how such CUs are further partitioned.

[0112] It should be noted that the splitting information included in the bitstream may include QT splitting flags and / or BT splitting flags. QT and BT splitting may also be mixed, as long as it is signalable or derivable which of the two types of splitting is used. On the decoder side, after the forced splitting, further splitting is performed based on the decoded splitting information such as qt_split_cu_flag and bt_split_cu_flag. Figure 10 shows a detailed exemplary implementation of boundary splitting, and in particular, one combination of forced BT and forced QT and forced BT. The variables used in the figure have the following meanings: - bBottom indicates whether the current CTU or CU's lower right sample is outside the lower boundary. - bRight indicates whether the current CTU or CU sample in the lower right is outside the right boundary. - bCorner indicates whether the current CTU or CU's lower right sample is outside the lower right corner boundary. - FQT_LV indicates the level of forced QT before forced BT is performed (corresponding to N above). - uiDepth indicates the current hierarchical level.

[0113] As mentioned above, the mandatory QT level can be predefined and adaptively selected based on encoder parameters such as slice type / time ID or RD cost.

[0114] In particular, in step 1010, the demarcation of boundary CTUs (level 0) begins. In step 1020, if the CTU is on the lower or right boundary (but not the corner boundary) and uiDepth is less than or equal to FQT_LV-1, the CTU is demarcated in step 1040 by a forced BT. Otherwise, step 1060 of forced QT demarcation is applied. Demarcation continues by further demarcating boundary CUs. For example, in the example in Figure 8(b), the FQT_LV of the CTU (level 1) is equal to 1. In step 1030, uiDepth=0 is equal to FQT_LV-1=0, so the incomplete CTU is forced to demarcate by the QT in step 1060, resulting in two CUs (top left and top right) that need to be further demarcated. Each of these CUs is further demarcated repeatedly. The top left CU (level 1) is still incomplete and is therefore a boundary CU. As the lower boundary CU in step 1020, since uiDepth=1 is greater than FQT_LV-1=0 in step 1030, a forced BT is applied to the upper left CU in step 1040. The upper right CU is partitioned in the same way. In step 1040, the boundary portion (incomplete) CU (64 x 24) is further partitioned at level 2 by BT. The remaining 64 x 8 CU at level 3 reaches the picture boundary and therefore results in "No" in steps 1020 and 1050. The partition of size 8 as the smallest partition is not further partitioned in step 1070, but rather results in "yes" in step 1080 and stops partitioning in step 1090. The second 64 x 24 CU is partitioned in the same way.

[0115] On the other hand, the 56 x 56 CTU in part 9(c) of Figure 9 is a corner CTU. In step 1020 (level 0), the CTU moves to step 1050 and is partitioned by QT as a corner CU in step 1060. In level 1, the top-left 56 x 56 CU is treated similarly, resulting in three boundary CUs (bottom-left, top-right, and corner). Assuming FQT_LV=0, i.e., no forced QT partitioning with respect to the bBottom and bRight CUs, the bottom-left and top-right CUs are partitioned by forced BT in step 1040 (after "yes" in step 1020 and "no" in step 1030). The 24 x 24 corner CU (level 2) moves from step 1020 to step 1050, is partitioned by QT in step 1060, and again generates three boundary CUs with sizes 16 x 8, 8 x 8, and 8 x 16, these three boundary CUs do not need to be further partitioned and move through steps 1020 ("no"), 1050 ("no"), and 1070 to the stopping condition 1080 and end 1090. The stopping condition may be, for example, the minimum size equal to 8 in this example.

[0116] Table 2 below shows exemplary syntax that allows signaling of QT partitioning flags for ordinary (non-boundary) CUs. In addition, BT partitioning flags may be signaled for CUs other than boundary CUs. The syntax of coding_quadtree is iterative, meaning partitioning is hierarchical. Partitioning is performed if the following conditions are true: x0 + cuWidth <= pic_width_in_luma_samples and y0 + cuHeight <= pic_height_in_luma_samples

[0117] In other words, a CTU / CU is partitioned by a quadtree or binary tree as long as the bottom-right corner of the current CU (i.e., the CU currently being processed by the partition) is inside a picture. It should be noted that this syntax is illustrative only and that this disclosure may also apply to other cases, such as using a selectable further type of partitioning for CTUs that are QT only, or BT only, or entirely inside a picture / slice. The values ​​cuMinBTSiz, cuMaxBTSize, and cuMaxBTDepth adopted in the syntax are predefined values.

[0118] In the above conditions, x0 and y0 specify the position of the sample at the top-left corner of the current CU, cuWidth and cuHeight specify the horizontal and vertical CU sizes, and the parameters pic_width_in_luma_samples and pic_width_in_luma_samples specify the horizontal and vertical picture sizes. Note that in this example, the entire picture is partitioned. However, the same technique can be applied to independently decodeable parts of the picture, such as slices, tiles, or any other parts of the picture with boundaries.

[0119] Similar to the HEVC standard, when qt_split_cu_flag[x0][y0] does not exist (in the case of a boundary), the following applies (in the encoder and decoder): - If cuWidth (the width of the CU at the current level) is greater than MinQTSize (the minimum QT size, 8 in this case) and cuHeight (the height of the CU at the current level) is greater than MinQTSize, then the value of qt_split_cu_flag[x0][y0] is presumed to be equal to 1. In other words, the boundary is forced to split. - Otherwise (log2CbSize is equal to MinCbLog2SizeY), the value of qt_split_cu_flag[x0][y0] is presumed to be equal to 0. In other words, if the current CU has the smallest allowed size MinCbLog2SizeY, the decoder assumes that the CU will not be split any further.

[0120] This is the same as in the case of Table 1 above.

[0121] [Table 2A] [Table 2B]

[0122] As can be seen, bt_split_cu_flag may be signaled. This flag is signaled when a BT split of a CTU is configured for a CTU / CU that is not on a boundary (see condition if(! (bBBoundary || bRBoundary))). In the case of a boundary, a forced split does not require an additional flag as the value of the flag is derived. In particular, if bt_split_cu_flag[x0][y0] does not exist (in the case of a boundary), the following applies: - If bBBoundary (meaning the current CU / CTU is on the lower boundary) is true, and at the same time cuWidth is greater than MinBTSize or cuHeight is greater than MinBTSize (meaning the current CU / CTU has no minimum size and can therefore be further split in principle), then the value of bt_split_cu_flag[x0][y0] is presumed to be equal to 1 (meaning a further binary split is performed, i.e., a binary tree split in which the longer side of the split CU is on the lower, i.e., parallel to / overlapping with the horizontal boundary). - bRBoudary (meaning the current CU / CTU is on the right boundary) is true, while cuWidth is greater than MinBTSize or cuHeight is greater than MinBTSize, and the value of bt_split_cu_flag[x0][y0] is inferred to be equal to 2 (meaning a further binary split is performed, i.e., a binary tree split in which the longer side of the split CU is on the right, i.e., parallel / overlapping with the vertical boundary). - Otherwise, the value of bt_split_cu_flag[x0][y0] is presumed to be equal to 0, meaning that no further bipartite split will be performed.

[0123] Being on a boundary means that there are no CUs below the current division's hierarchy level between the current CU and the boundary; in other words, the current CU belongs to the boundary portion at the current hierarchy level (corresponding to bBBoundary being true or bRBoundary being true). If the current CU is not a boundary CU, QT is not enforced (cqtDepth > FQT_LV-1), and the BT flag is present in the bitstream, then BT partitioning is used.

[0124] Exemplary values ​​for JEM implementations are cuMinBTSize for an interframe with a value of 4, cuMAXBTSize for an interframe with a value of 128, and cuMAXBTDepth with a value of 3. However, this disclosure is not limited to these sizes and is applicable to any selected value.

[0125] It should be noted that the syntax above is illustrative only and assumes knowledge of a variable FQT_LV indicating the number of mandatory QT partition levels. If the value of FQT_LV is zero, then there are no mandatory QT levels, and the entire partition of the mandatory boundary is performed by binary tree partitioning. The syntax and semantics may be implemented differently without departing from this disclosure.

[0126] Another solution based on JEM software (JEM5 and 6) could be to define the following semantics: When qt_split_cu_flag[x0][y0] does not exist (in the case of a boundary), the value of qt_split_cu_flag[x0][y0] is presumed to be equal to 1, which means a further QT split or forced QT (if cqtDepth <= FQT_LV-1) in the case of corner CUs.

[0127] If bt_split_cu_flag[x0][y0] does not exist (in the case of a boundary), the following applies: - If bBBoundary is true, we infer that the value of bt_split_cu_flag[x0][y0] is equal to 1, which means that the lower boundary CU is split vertically by a bipartite division, i.e., to form two horizontal CUs (i.e., a CU that is longer horizontally than vertically). - If bRBoundary is true, we infer that the value of bt_split_cu_flag[x0][y0] is equal to 2, which means that the CU on the right boundary is split horizontally by a bipartite division, i.e., to form two vertical CUs (i.e., a CU that is longer vertically than horizontally).

[0128] The example above illustrates a forced partition using a parameter that defines the number of forced QT levels before the BT partition is applied. The semantic rule defines the partition that is fitted at the boundary, i.e., when no partition flag exists. However, this disclosure is not limited to such methods.

[0129] Alternatively, the (incomplete) CTUs at the boundary may first be partitioned by RD optimization, and then the remaining blocks at the frame boundary may use forced BT partitioning or a combination of forced QT and forced BT partitioning. Partitioning by RB optimization suggests that a partitioning flag is signaled in the bitstream with respect to the corresponding CU. Partitioning by RD may be specified as either QT (i.e., by the standard or by a sequence of pictures or common signaling for pictures) or BT, so that the choice between BT and QT does not have to be signaled for each CU. On the other hand, the partitioning flag may be extended to indicate not only whether a given CU is partitioned, but also whether that CU is partitioned by QT or by BT.

[0130] On the decoder side, forced partitioning is performed on boundary CTU / CUs. With regard to forced BT partitioning, forced horizontal BT (vertical partitioning to form horizontal partitions) is used on CTU / CUs at the lower boundary until the bottom right sample of the leaf node falls within the lower boundary of the slice / picture. Forced vertical BT is used on CTU / CUs at the right boundary until the bottom right sample of the leaf node falls within the right boundary of the slice / picture. CTU / CUs at the bottom right corners are partitioned using forced QT until the bottom right sample of the leaf node falls within the boundary of the slice / picture or if the leaf node falls within the right / bottom boundary. Further partitioning is performed based on a decoded partitioning flag, which may be a QT or BT partitioning flag, depending on whether there is a defined number of levels of forced QT partitioning. The elements of the syntax are generated by the encoder and processed by the decoder as described above.

[0131] Regarding the combination of forced QT and forced BT, the number of forced QT levels for boundary CTU (corresponding to the parameter FQT_LV) may be adaptively selected based on the time ID / slice type or decoded from the bitstream. The decoded forced QT levels may be extracted from the slice header, SPS, or PPS. Boundary CTU / CU is subjected to forced BT partitioning only after forced QT partitioning. Forced partitioning is terminated when the lower right sample of a leaf node is within the boundary. Further partitioning is performed based on the decoded partitioning flag.

[0132] In other words, according to the first implementation, in the decoder, if the CTU is on a slice / picture boundary, binary tree partitioning is performed without an indicator (flag) of binary tree partitioning and / or an indicator of the type of binary tree partitioning (indication of whether BT or QT should be performed). Binary partitioning is terminated when the leaf node's lower right sample is inside the slice / picture boundary (i.e., not crossing the picture boundary).

[0133] Therefore, instead of mandatory QT boundary demarcation, mandatory BT is introduced for the boundary CTU. Mandatory horizontal BT for the lower boundary and mandatory vertical BT for the right boundary are implemented. For the lower right corner boundary, mandatory QT is used until the current CU no longer needs to be forcibly demarcated (because the minimum size has been reached) or until the current CU is on the lower or right boundary.

[0134] In other implementations of this disclosure, combinations of forced QT and forced BT partitioning are also possible. Thus, forced BT partitioning is performed only after a certain number of forced QT levels. With respect to combinations of forced QT and forced BT partitioning, the levels of forced QT may be adaptively selected by time ID or slice type. Alternatively, the levels of forced QT may be adjusted by cost and included in the bitstream to indicate how many levels of forced QT are used for the decoder.

[0135] Firstly, greater flexibility is achieved when boundary CTUs are partitioned by RD optimization, and only the remaining blocks at the frame boundary (i.e., incomplete CUs that are partially inside and partially outside the picture) can use the proposed forced BT partitioning or a combination of forced QT and forced BT partitioning, as described above.

[0136] The divisions described so far can be used in still image or video encoding. In particular, the divisions can be used in the encoder and decoder shown in division unit 110 and division unit 250, respectively, in Figures 1 and 2.

[0137] Accordingly, the present disclosure also provides an apparatus for encoding images of a video sequence, comprising: an apparatus (splitting unit) 110 for dividing an image portion relating to any of the above examples into encoding units; image encoding units 120-145, 160-195 configured to encode the encoding units; and a bitstream forming unit 150 configured to generate a bitstream containing the encoded encoding units and partitioning information indicating how the encoding tree units are partitioned.

[0138] The encoder may further include a rate-distortion optimization unit (not shown in Figure 1) configured to determine that the boundary portion is hierarchically partitioned by rate-distortion optimization for any level of partitioning, apart from the deepest level. The RD optimization unit may perform coding using different encoder settings in relation to the partitioning and select the setting that yields the smallest cost function. The cost function may be the smallest rate for a given level of distortion or similarity between the prediction and the original block being coded.

[0139] Furthermore, a device is provided for decoding images of a video sequence, comprising: a device 250 for determining the division of the image portion to be decoded into encoding units as described above; a bitstream parser 250 for analyzing a bitstream containing division information based on the determined division of the image portion, showing how the encoded encoding units and encoding tree units are divided; and image decoding units 225-240, 260-295 for decoding the encoded encoding units.

[0140] It should be noted that splitting and parsing are performed repeatedly together. Parsing involves separating the necessary control information from the bitstream relating to the current CU, and splitting uses the parsed information and / or semantic knowledge to derive the CU's location, size, and data. In particular, the parser may send a splitting flag, on which the splitter either further splits the current CU or maintains the size of that CU. Depending on the splitting flag, the parser also either further parses the splitting information relating to CUs of greater depth, or parses the splitting information relating to other CUs of the same depth, or parses the data to be decoded relating to the CU. Furthermore, the splitter is configured to split the boundary portions as described above, even without further bitstream signaling, according to enforced splitting rules.

[0141] In other words, both the encoder and / or decoder may adopt the division of the boundary portion into encoded units as a predefined division. The division information within the bitstream is then completely irrelevant to the division of the boundary portion. This is true when the boundary is divided in a predefined manner, which may further include deriving the division based on other encoder settings such as picture type / time ID / slice type.

[0142] Alternatively, the encoder and / or decoder may be further configured to partition the boundary portion by quadtree partitioning at N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchy levels of partitioning, and to partition the boundary portion by binary tree partitioning at the remaining hierarchy levels of partitioning, with the partitioning information including an indication of N. The partitioning information may further include partitioning flags to indicate whether and / or by what type of partitioning the boundary portion CTU / CU should be partitioned (which may include binary trees or quadtrees and / or other options). The partitioning flags may be signaled for all partitions (CUs), or only for partitions that are not on the boundary at their hierarchy levels.

[0143] The encoder and / or decoder may be implemented (built into) a chip or by software. However, the encoder and / or decoder may also be implemented in multiple hardware and software components.

[0144] The above primarily considered two types of partitioning: quadtree partitioning and binary tree partitioning. However, the cost-based or predefined boundary-enforced partitioning methods provided in this disclosure are suitable not only for enforced QT or enforced BT but also for other enforced partitioning structures.

[0145] The above primarily described the case where the deepest CU in a forced division of a boundary is a rectangular CU, not a square CU, that is, a CU where one side is longer than the other.

[0146] However, generally speaking, boundary portions may be partitioned by binary tree partitioning to at least one level of partitioning in which no encoding units of the boundary remain entirely within the boundary portion. In other words, according to the present disclosure, an apparatus for partitioning an image portion into encoding units may include, in operation, a processing circuit that subdivides the image portion into an integer number of encoding tree units of a predetermined size and boundary portions having a size less than a predetermined size in at least a first direction, either horizontally or vertically, and partitions the boundary portions hierarchically into encoding units, each containing encoding units of different sizes in the horizontal and vertical directions, such that the encoding units are partially within the boundary portion and partially outside the image portion. In particular, binary tree partitioning may be used for CUs of any boundary.

[0147] It should be noted that the boundary region shown in Figure 7 contains several CTUs. The exemplary embodiments and implementations above have been described with respect to the partitioning of one CTU. This is because even if only one of the (incomplete) CTUs in the boundary region is partitioned as described above, it is advantageous and may lead to better results in terms of coding efficiency and / or complexity. However, two or more CTUs in the boundary region may be partitioned as described above. The partitioning may be performed sequentially, one CTU at a time. In some implementations, it may be beneficial to perform partitioning in parallel with respect to different CTUs. Parallel processing may reduce coding / decoding delays if employed by the encoder / decoder respectively.

[0148] In other words, it may be advantageous to apply the above-described classification to all CTUs in the boundary portion. However, the disclosure is not limited thereto, and the classification may be applied to CTUs in one or more boundaries. The remaining CTUs may be divided by any other classification, such as QTs only.

[0149] Corresponding methods for partitioning are also provided. Corresponding encoding and decoding devices and methods employing such partitioning are further provided.

[0150] This disclosure may be implemented in a device. Such a device may be a combination of software and hardware. For example, the intra-prediction and deblocking filters may be performed by a chip such as a general-purpose processor, a digital signal processor (DSP), or a field-programmable gate array (FPGA). However, the present invention is not limited to implementation on programmable hardware. It may be implemented on an application-specific integrated circuit (ASIC) or by a combination of the hardware components described above.

[0151] The determination of intra-prediction and deblocking filters may also be implemented by program instructions stored on a computer-readable medium, which, when executed by a computer, perform the steps of the method as described above. The computer-readable medium can be any medium on which the program is stored, such as DVDs, CDs, USB (flash) drives, hard disks, or server storage accessible via a network.

[0152] Encoders and / or decoders may be implemented in a variety of devices, including television sets, set-top boxes, PCs, tablets, and smartphones. Encoders and / or decoders may also be software, apps that implement the steps of the method.

[0153] In summary, this disclosure relates to the decomposition of an image or image slice. Specifically, the image or image slice is divided into an integer number of coding tree units both vertically and horizontally. The deepest level of decomposition is then the result of a binary decomposition, that is, the remaining boundary is forced to be divided at least partially such that one side contains longer coding units than the other side. The longer side is either parallel to the boundary of the image or image slice or overlaps with the boundary of the image or image slice. [Explanation of Symbols]

[0154] 100 encoders 110 division units 120 Image Encoding Units 125 Reconstructor 130 Conversion Unit 135 Inverse Conversion Unit 140 Quantization Units 145 Inverse Quantization Unit 150 Entropy Coding Units 160 Loop Filtering Units 170 frame buffer 180 Mode Selection Unit 190 Intra Prediction Units 195 Interpretation Units 200 video decoders 225 Reconstructor 230 Reverse Conversion Unit 240 Inverse Quantization Units 250 bitstream analysis, entropy decoding, and partitioning units 260 Loop Filtering Unit 270 Reference Picture Buffer 290 Intra Prediction Units 295 Image Decoding Unit 600 equipment 610 Functional Unit 620 Functional Units 630 Functional Units 700 Boundary of the picture below 710 Boundary part 720 parts Vertical dimensions of 770V CTU Horizontal dimensions of the 770H CTU 950 Lower boundary CU CU of a 960-square boundary

Claims

1. A device for dividing an image portion into encoding units, during operation, The image portion is subdivided into an integer number of encoding tree units having a predetermined size, and boundary portions having a size less than the predetermined size in at least one first direction, either horizontally or vertically. The boundary portion is hierarchically divided into coding units, each containing the deepest coding unit having different sizes in the horizontal and vertical directions, wherein the deepest coding unit is smaller in the first direction. A device including a processing circuit that performs the following actions.

2. The apparatus according to claim 1, wherein the processing circuit is configured to partition the boundary portion by binary tree partitioning at least at the deepest level of partitioning.

3. The apparatus according to claim 1 or 2, wherein the processing circuit is configured to also partition the boundary portion by a quadtree.

4. The processing circuit described above The process involves partitioning the boundary portion by a quadtree at the N shallowest levels, where N is a non-negative integer and less than the number of hierarchy levels of the partitioning, and The apparatus according to claim 3, configured to divide the boundary portion by binary tree partitioning at the remaining hierarchical levels of the division.

5. The apparatus according to any one of claims 1 to 4, wherein the processing circuit divides the boundary portion until the deepest level encoding unit remains completely within the boundary portion during operation.

6. The apparatus according to any one of claims 1 to 5, wherein the processing circuit divides the boundary portion horizontally and vertically by binary tree partitioning during operation.

7. The apparatus according to any one of claims 1 to 6, wherein the processing circuit, during operation, divides the corner portion of the boundary portion by a quadruple tree at each hierarchical level.

8. The apparatus according to claim 1 or 2, wherein the processing circuit is configured to partition the boundary portion by binary tree partitioning at all levels of partitioning.

9. A device for encoding images in a video sequence, The apparatus for dividing an image portion into encoding units according to any one of claims 1 to 8, An image encoding unit configured to encode the aforementioned encoding unit, An apparatus comprising: an encoded encoding unit and a bitstream forming unit configured to generate a bitstream including partitioning information indicating how the encoded tree unit is partitioned.

10. The encoder according to claim 9, further comprising a rate-distortion optimization unit configured to determine that, with respect to any level of division except the deepest level, the boundary portion is hierarchically divided by rate-distortion optimization.

11. A device for decoding images in a video sequence, The apparatus for determining the division of an image portion to be decoded into encoding units according to any one of claims 1 to 8, A bitstream parser for analyzing a bitstream that shows how the encoded encoding units and the encoding tree units are partitioned, and which includes partitioning information based on the determined partitioning of the image portion, An apparatus comprising an image decoding unit for decoding the encoded encoding unit.

12. The apparatus according to any one of claims 9 to 11, wherein the division of the boundary portion into encoding units is a predefined division, and the division information is not related to the division of the boundary portion.

13. The processing circuit is configured to partition the boundary portion by quadtree partitioning at the N shallowest levels, where N is an integer greater than or equal to zero and less than the number of hierarchy levels of partitioning, and to partition the boundary portion by binary tree partitioning at the remaining hierarchy levels of partitioning. The apparatus according to any one of claims 9 to 11, wherein the classification information includes N, or N is derived based on the type of prediction applied to decode the image being processed.

14. A method for dividing an image portion into encoding units, The step of dividing the image portion into an integer number of encoding tree units having a predetermined size and boundary portions having a size less than the predetermined size in at least a first direction, which is either horizontal or vertical, A method comprising the step of hierarchically dividing the boundary portion into coding units, each containing the deepest coding unit having different sizes in the horizontal and vertical directions, wherein the deepest coding unit is smaller in the first direction.

15. The step of dividing the boundary portion is, The first step is to start from the current level of classification, which is the shallowest level of classification, The steps include: dividing the boundary portion into coding units of the current division level sequentially until the coding units of the current division level are completely contained within the boundary portion; The method according to claim 14, further comprising the step of raising the level of the partition and further partitioning the coding unit as soon as the coding unit of the current partition level does not fit into the boundary portion, wherein a quadtree partition is used for the N shallowest partition levels and a binary tree partition is used in other cases, where N is an integer greater than or equal to zero and less than the number of hierarchical levels of partitions.

16. A method for encoding images in a video sequence, The steps of dividing the image portion according to claim 14 or 15 into encoding units, The steps include encoding the aforementioned encoding unit, A method comprising the steps of generating a bitstream that includes the encoded encoding units and partitioning information indicating how the encoding tree units are partitioned.

17. A method for decoding images in a video sequence, A step of determining the division of the image portion to be decoded into encoding units according to claim 14 or 15, The steps include: showing how the encoded encoding units and the encoding tree units are partitioned, and analyzing a bitstream containing partitioning information based on the determined partitioning of the image portion; A method comprising the step of decoding the encoded encoding unit.

18. A computer-readable medium for storing instructions that cause the processor to perform the method described in claim 14 or 17 when executed by the processor.

Images