Method and apparatus for dividing image into coding units

Abstract

The invention provides an apparatus and a method for dividing an image into coding units. According to the method, an image is divided into coding tree units (CTUs for short), and the coding tree units are subjected to hierarchical division. Hierarchical partitioning includes multi-type partitioning, such as binary tree or quadtree partitioning. Corresponding multi-type division depths are selected for the CTUs completely located in the image and the CTUs located at the boundary. The invention provides multi-type division of image boundary portions, which can enhance flexibility.

Description

[0001] This application is a divisional application. The application number of the original application is 201980012698.X, and the original application date is May 29, 2019. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present invention relates to the field of video processing and, more particularly, to the subject generally referred to as hybrid video coding and compression. Background Art

[0003] The next generation standard of Versatile Video Coding (VVC) is the latest joint video project of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The two standardization organizations work together in a partnership called the Joint Video Exploration Team (JVET).

[0004] The current block-based hybrid video codec uses predictive coding. The images of the video sequence are divided into blocks of pixels, which are then encoded. Instead of encoding the blocks pixel by pixel, the blocks are predicted using the encoded pixels that are spatially or temporally adjacent to the entire block. The encoder further processes only the difference between the block and its predicted block. The further processing typically includes transforming the block pixels into coefficients in the transform domain. The coefficients can then be further compressed (e.g., by quantization) and recompressed (e.g., by entropy coding) to form a bitstream. The bitstream may also include any signaling information that enables the decoder to decode the encoded video. For example, the signaling may include settings related to the encoder settings, such as the size of the input image, the frame rate, the quantization step indication, the prediction applied to the image block, etc.

[0005] The difference between a block and its predicted block is called the residual of the block. More specifically, each pixel of the block has a residual, which is the difference between the intensity level of the pixel and its predicted intensity level. The intensity level of a pixel is called a pixel value. The residuals of all pixels of a block are collectively referred to as the residual of the block. In other words, a block has a residual, which is a set or matrix of the residuals of all pixels of the block. The residuals are then transformed, quantized and encoded together with the signaling information. The encoding can include various forms of fixed length encoding and variable length encoding, including arithmetic coding or other entropy coding types.

[0006] In block-based hybrid video coding, each image is divided into blocks of pixels (samples), and multiple blocks in an image are aggregated to form slices that are independently decodable entities. The block that is predicted and / or transformed is called a coding unit (CU) or coding block (CB). The size of the coding unit can be different.

[0007] For example, in High-Efficiency Video Coding (HEVC, also known as H.265), video frames are divided into coding tree units (CTU, also known as coding tree blocks (CTB)). CTBs are blocks of the same size but not intersecting, such as 64x64 pixels. Each CTB serves as the root of a block partition quadtree structure (i.e., coding tree). Along the coding tree structure, the CTB can be further divided into coding blocks, and the prediction type of the coding block is determined. The coding block can be further divided into smaller transform blocks, which can be transformed and quantized.

[0008] For details on the partitioning in HEVC, see V. Sze et al., ed., High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer-Verlag, 2014, Chapter 3.2.

[0009] In addition, WO 2016 / 090568 also shows a binary tree structure, which divides a unit into multiple smaller units by using a quadtree plus a binary tree structure. Therefore, the root unit is first divided into a quadtree structure, and then the quadtree leaf nodes are divided using a binary tree structure. Summary of the invention

[0010] Embodiments of the invention are defined by the features of the independent claims, and further advantageous implementations of the embodiments are defined by the features of the dependent claims.

[0011] According to a general aspect, the present invention provides an apparatus for dividing an image into coding units, the apparatus comprising a processing circuit. The apparatus is used to divide the image into a plurality of coding tree units (CTUs), the plurality of coding tree units comprising a non-boundary CTU and a boundary CTU, wherein the non-boundary CTU has a predetermined size in the horizontal and vertical directions, a portion of the boundary CTU falls within an image defined by a horizontal image boundary or a vertical image boundary, and a portion of the boundary CTU located within the image has a size smaller than the predetermined size in a direction perpendicular to the horizontal image boundary or the vertical image boundary; hierarchically dividing the non-boundary CTU and the boundary CTU into respective coding units (CUs), wherein the hierarchical division of the non-boundary CTU comprises a multi-type division, and a division depth of the multi-type division of the hierarchical division of the non-boundary CTU does not exceed a maximum non-boundary multi-type division depth; the multi-type division refers to a division in which the division direction is a vertical or horizontal direction, the hierarchical division of the boundary CTU comprises a multi-type division, and a division depth of the multi-type division of the hierarchical division of the boundary CTU does not exceed a maximum boundary multi-type division depth.

[0012] This allows for greater flexibility in boundary delineation.

[0013] Further, in one implementation of the device, the maximum boundary multi-type division depth is at least the sum of the adaptive boundary multi-type division depth and the predefined multi-type division depth, wherein the adaptive boundary multi-type division depth is the multi-type division depth with the horizontal image boundary direction or the vertical image boundary as the division direction.

[0014] This facilitates adaptive determination of the partition depth when multiple types of partitions are used in boundary coding tree units or partition blocks.

[0015] For example, the predefined multi-type partition depth is equal to the maximum non-boundary multi-type partition depth.

[0016] This allows the maximum non-bounded multi-type partition depth to be reused.

[0017] Further, in one implementation of the device, the sum also includes a function of the ratio of the size in the image boundary direction to the size in the image boundary direction of a boundary partition block perpendicular to the boundary CTU, and the boundary partition block is a block of the adaptive boundary multi-type partition depth.

[0018] This further increases the maximum depth of multi-type boundary partitioning, thereby enhancing partitioning flexibility.

[0019] For example, the function is the binary logarithm.

[0020] This is beneficial because it provides a practical implementation.

[0021] Further, in some embodiments, the maximum boundary multi-type partitioning depth is predefined.

[0022] This helps reduce the computational cost of determining the hierarchy partitioning.

[0023] For example, the hierarchical partitioning of the boundary CTU further includes quadtree partitioning.

[0024] This allows flexibility in choosing from different modes.

[0025] Furthermore, in an implementation of the device, the maximum boundary multi-type partitioning depth is greater than or equal to the maximum non-boundary multi-type partitioning depth.

[0026] This can enhance the maximum possible boundary partition depth.

[0027] The present invention also provides a device for encoding an image of a video sequence, the device comprising a device for dividing an image into coding units according to any one of the above examples and embodiments. The device also comprises an image encoding unit for encoding the coding unit, and a bit stream forming unit for generating a bit stream, the bit stream comprising the encoded coding unit and division information indicating how to divide the coding tree unit.

[0028] Further, in an implementation, the image encoding device includes the image division device, wherein the maximum boundary multi-type division depth is predefined; the bit stream also includes a coding sequence parameter set, and the coding sequence parameter set includes the maximum boundary multi-type division depth.

[0029] In addition, the present invention also provides a device for decoding a picture of a video sequence, the device comprising a bitstream parser for parsing a bitstream, the bitstream parser comprising an encoded coding unit; a picture division determination device according to any one of the above examples and embodiments; and a picture decoding unit, for decoding the encoded coding unit based on the determined picture division.

[0030] Further, in one implementation, the image decoding device includes the image division determination device, wherein the maximum boundary multi-type division depth is predefined; the bit stream also includes a coding sequence parameter set, and the coding sequence parameter set includes the maximum boundary multi-type division depth; the image division determination device is also used to obtain the second maximum multi-type division depth from the sequence parameter set.

[0031] According to another general aspect, the present invention provides a method for dividing an image into coding units. The method includes dividing the image into a plurality of coding tree units CTU, the plurality of coding tree units including a non-border CTU and a border CTU, wherein the non-border CTU has a predetermined size in the horizontal and vertical directions, a portion of the border CTU falls within an image defined by a horizontal image boundary or a vertical image boundary, and a portion of the border CTU located within the image has a size smaller than the predetermined size in a direction perpendicular to the horizontal image boundary or the vertical image boundary; hierarchically dividing the non-border CTU and the border CTU into respective coding units (CUs), wherein the hierarchical division of the non-border CTU is a multi-type division limited by a maximum non-border multi-type division depth, the multi-type division refers to a division in which the division direction is a vertical or horizontal direction, and the hierarchical division of the border CTU is a multi-type division limited by a maximum border multi-type division depth.

[0032] Furthermore, in one implementation of the method, the maximum boundary multi-type division depth is at least the sum of the adaptive boundary multi-type division depth and the predefined multi-type division depth, wherein the adaptive boundary multi-type division depth is the multi-type division depth with the image boundary direction as the division direction.

[0033] For example, the predefined multi-type partition depth is equal to the maximum non-boundary multi-type partition depth.

[0034] Further, in one implementation of the method, the sum also includes a function of the ratio of the size in the image boundary direction to the size in the image boundary direction of a boundary partition block perpendicular to the boundary CTU, and the boundary partition block is a block of the adaptive boundary multi-type partition depth.

[0035] For example, the function is the binary logarithm.

[0036] In another embodiment, the maximum boundary multi-type partitioning depth is predefined.

[0037] Furthermore, in an implementation manner, the hierarchical division of the boundary CTU further includes quadtree division.

[0038] For example, the maximum boundary multi-type partition depth is greater than or equal to the maximum non-boundary multi-type partition depth.

[0039] The present invention also provides a method for encoding an image of a video sequence, the method comprising a step of dividing the image into coding units according to any of the above embodiments; and a bit stream forming step of generating a bit stream including the encoded coding units and division information indicating how to divide the coding tree units.

[0040] Further, in an implementation, the image encoding method includes the image division method, wherein the maximum boundary multi-type division depth is predefined; the bit stream also includes a coding sequence parameter set, and the coding sequence parameter set includes the maximum boundary multi-type division depth.

[0041] The present invention also provides a method for decoding an image of a video sequence, the method comprising the steps of parsing a bit stream including encoded coding units; a step of determining image division according to any of the above-mentioned embodiments; and an image decoding step of decoding the encoded coding units based on the determined image division.

[0042] Further, in one implementation, the image decoding method includes the image division determination method, wherein the maximum boundary multi-type division depth is predefined; the bit stream also includes a coding sequence parameter set, and the coding sequence parameter set includes the maximum boundary multi-type division depth; the image division determination device is also used to obtain the second maximum multi-type division depth from the sequence parameter set.

[0043] On the other hand, the present invention provides a computer-readable medium storing instructions, which, when executed by a processing circuit, causes the processing circuit to perform a method for dividing an image into coding units, a method for encoding an image of a video sequence, or a method for decoding an image of a video sequence according to any of the above embodiments.

[0044] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages are apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The embodiments of the present invention are described in more detail below with reference to the accompanying drawings, wherein:

[0046] Figure 1 is a block diagram of an exemplary structure for implementing a video encoder according to an embodiment of the present invention;

[0047] Figure 2 is a block diagram of an exemplary structure for implementing a video decoder according to an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of an example of quadtree partitioning adopted by HEVC;

[0049] Figure 4 Schematic diagram of several coding unit division modes;

[0050] Figure 5 A schematic diagram of a quadtree / binary tree partitioning example;

[0051] Figure 6 Schematic diagram of forced quadtree partitioning of the boundary part;

[0052] Figure 7 Schematic diagram of binary tree partitioning of the boundary part;

[0053] Figure 8 A block diagram of an apparatus for dividing an image into coding units;

[0054] Fig. 9 Schematic diagram of the boundary division;

[0055] Fig.10 This is an example diagram of boundary partitioning using binary tree partitioning;

[0056] Fig.11 This is another example diagram of using binary tree partitioning to perform boundary partitioning;

[0057] Fig.12 A comparison diagram of boundary partitioning using quadtree partitioning and binary tree partitioning;

[0058] Fig.13 A flowchart of the implementation method of the maximum multi-type tree depth expansion;

[0059] Fig.14 A block diagram of an example of a video encoding system for implementing an embodiment of the present invention;

[0060] Fig.15 A schematic diagram of a video encoding device. DETAILED DESCRIPTION

[0061] The present invention relates to dividing (i.e. partitioning) an image into smaller units for further processing. This partitioning facilitates encoding and decoding of still images or video images. An exemplary video encoder and decoder are described below that can implement the partitioning according to the present invention.

[0062] Figure 1 An encoder 100 is shown, which includes an input terminal for receiving an input block of a frame or image of a video stream, and an output terminal for providing an encoded video bitstream. The term "frame" in the present invention is a synonym for image. However, it should be noted that the present invention is also applicable to fields when interlaced scanning is used. Typically, an image includes m×n pixels. These pixels correspond to image pixels, and each pixel may include one or more color components. For simplicity, the pixels mentioned in the following description are used to represent brightness pixels. However, it should be noted that the division method described in the present invention can be applied to any color component including chrominance or components of color spaces such as RGB. On the other hand, it may be beneficial to divide only one component and apply the determined division method to more (or all) remaining components.

[0063] The encoder 100 is used to divide, predict, transform, quantize and entropy encode a video stream.

[0064] In the partitioning unit 110, the input video frame is further divided before encoding. The sizes of the blocks to be encoded are not necessarily the same. An image may include blocks of different sizes, and the block grids of different images in a video sequence may also be different. In particular, each video image (picture) is first divided into CTUs of the same fixed size. The CTU size may be fixed and predefined (for example, in the standard). The CTU size adopted by HEVC is 64×64. However, the present invention is not limited to standard and fixed sizes. Providing a CTU size that can be set at the encoder, which can be used as a signaling parameter in the bitstream, may be advantageous. For example, different CTU sizes can be used for different image sizes and / or content types. The CTU size can be indicated by any level of signaling, for example, the CTU size can be common to the entire video sequence or a portion thereof (i.e., multiple images) or a single image. Accordingly, for example, the CTU size may also be indicated in a picture parameter set PPS, a sequence parameter set SPS or a video parameter set VPS known to the current codec (H.264 / AVC, H.265 / HEVC), or a similar parameter set. Alternatively, the CTU size may be specified in a slice header or at any other level. The CTU size may have values ​​other than 64×64. For example, it may be 128×128 pixels. In general, in order to perform hierarchical partitioning by a binary tree or a quadtree, it may be beneficial to provide a CTU size that is a power of 2, i.e., 2 n , where n is an integer greater than 2.

[0065] Figure 3 The illustrated partitioning of images into CTUs and CTUs into CUs is selected from High Efficiency Video Coding (HEVC): Algorithms and Architectures, edited by V. Sze et al., Springer-Verlag, 2014. The partitioning follows a quadtree structure to accommodate various local characteristics. Figure 3 The left side of the diagram shows the CTUs hierarchically divided according to the quadtree structure on the right. In particular, the coding tree defines a syntax that specifies the division of CTUs into CUs. Similar to a CTU, a CU consists of square blocks of pixels and syntax associated with these blocks of pixels. Therefore, starting with the CTU (hierarchical depth 0), the CTU can be divided into four CUs with a hierarchical depth of 1 (in the quadtree), or it may not be divided further. Figure 3, the CTU is divided into CU8 and 16 of the first hierarchical depth (level), and another two CUs, wherein the CU8 and 16 are no longer divided, thereby forming leaves of the quadtree, and the other two CUs are further divided into CUs with a hierarchical depth of 2 (CUs of depth -2). Specifically, the CU of depth -1 on the upper left side is further divided into CU1, 2 and 7 of depth -2, which form leaves of the quadtree, and another CU. The other CU is further divided into CU3, 4, 5 and 6 of depth -3, all of which are leaves. Similarly, the CU of depth -1 on the lower left side is further divided into CU13, 14 and 15 of depth -2 (also leaves of the quadtree), and one remaining CU. The remaining CU is further divided into CU9, 10, 11 and 12 of level -3, all of which are leaves, and therefore no longer divided.

[0066] An exemplary syntax of quadtree partitioning in HEVC is shown in Table 1.

[0067] Table 1: CTU partition flag syntax

[0068]

[0069]

[0070] In particular, at the CTU level, the bitstream contains a flag named split_cu_flag, which indicates whether the complete CTU forms a CU or is divided into four equal-sized blocks corresponding to square pixel blocks. If the CTU is divided, another split_cu_flag is transmitted for each resulting block, specifying whether the block represents a CU or is further divided into four equal-sized blocks. This hierarchical division continues until the resulting block is no longer divided. The minimum CU size is indicated in the sequence parameter set, which can range from 8×8 luminance pixels to the size of the CTU (including 8×8 luminance pixels and the size of the CTU). During the hierarchical division process, when the minimum CU size is reached, the division flag is not transmitted to the corresponding block; instead, it is inferred that these blocks are no longer divided further. Typical HEVC encoder settings utilize the maximum range of supported CU sizes, so that CUs ranging from 8×8 to 64×64 pixels can be used. CUs within a CTU are encoded in depth-first order. This coding order is also called z-scan. It ensures that for each CU, except for the pixels at the top or left boundary of the slice, all pixels above the CU and left for the CU are coded, so that the corresponding pixels can be used for intra prediction and the associated coding parameters can be used to predict the coding parameters of the current CU.

[0071] In other words, split_cu_flag[x0][y0] specifies whether the coding unit is split into coding units of half the horizontal and vertical size. The array index x0, y0 specifies the position (x0, y0) of the upper left luminance pixel of the coding block relative to the upper left luminance pixel of the image. When split_cu_flag[x0][y0] does not exist, the decoder processes as follows:

[0072] If log2CbSize (a parameter specifying the coding block size) is greater than MinCbLog2SizeY (a parameter specifying the minimum configurable coding unit size), the value of split_cu_flag[x0][y0] is inferred to be equal to 1.

[0073] Otherwise (log2CbSize is equal to MinCbLog2SizeY), it is inferred that the value of split_cu_flag[x0][y0] is equal to 0.

[0074] The array CtDepth[x][y] specifies the coding tree depth of the luma coding block including position (x, y). When split_cu_flag[x0][y0] is equal to 0, CtDepth[x][y] is inferred to be equal to cqtDepth from x=x0..x0+nCbS−1 and y=y0..y0+nCbS−1.

[0075] In VVC (Versatile Video Coding), quad tree (QT) partitioning and multi-type (binary / ternary / asymmetric binary) tree (BT / TT / ABT) partitioning structures replace the concept of multiple partition unit types. In other words, the new partitioning structure does not distinguish between CU (coding unit), PU (prediction unit) and TU (transform unit) concepts (unless the size of the CU is too large for the maximum transform length), and supports more flexible CU partition shapes [JVET-J1002]. Figure 4 The partitioning mode used in the current VTM (VVC test model) is shown. Figure 4 Part (a) shows a CTU or CU that is not further divided (not divided). Part (b) shows a quadtree (also commonly referred to as a "quadtree") partitioning mode, in which the CTU or CU is divided both vertically and horizontally. Parts (c) and (d) show binary tree partitioning modes in the vertical and horizontal directions, respectively. In addition, parts (e) and (f) show ternary tree partitioning in the vertical and horizontal directions. It can be seen that in the ternary tree partitioning, there are two blocks of size 1 / 4 and one block of size 1 / 2.

[0076] For the BT / TT / QT coding tree scheme, the parameters related to the hierarchical division of CTU are defined and specified by the syntax elements in the sequence parameter set (SPS), as follows:

[0077] –CTU size: the root node size of the quadtree

[0078] –MinQTSize: Minimum allowed quadtree leaf node size

[0079] –MaxBTTSize: Maximum allowed root node size for binary and ternary trees

[0080] –MaxBTTDepth: Maximum allowed binary and ternary tree depth

[0081] –MinBTTSize: Minimum allowed binary and ternary tree leaf node size

[0082] –MinCUSize: Minimum allowed CU size.

[0083] Figure 5 A hybrid quadtree and binary tree partition is shown. Quadtree partitions are represented by solid lines and binary tree partitions are represented by dashed lines. The labels 1 or 0 on the nodes indicate whether binary partitions are applied vertically or horizontally, respectively, where the nodes represent coding units that are further to be binary-tree partitioned.

[0084] In luminance pixels, the horizontal and vertical sizes of the video image must be integer multiples of the minimum CU size transmitted in the sequence parameter set, but not necessarily integer multiples of the CTU size. If the horizontal or vertical size of the video image is not an integer multiple of the CTU size, the CTU at the inferred boundary will be divided until the boundary of the resulting block coincides with the image boundary. For this forced division, the division flag is not sent, but the resulting block can be further divided using the above quadtree syntax. CUs located outside the image area are not encoded.

[0085] This division is Figure 6 As shown, Figure 6 The bottom boundary CTU (128×128) of an HD sequence (1920×1080) is shown to enforce QT partitioning. In particular, Figure 6 A frame boundary is shown with 56 lines (128 pixels long) on ​​the frame boundary that are part of the boundary of a slice or picture. The CTU portion below the frame boundary may belong to another slice or may not exist at all, for example, if the frame boundary is the bottom picture boundary. It can be seen that the forced quadtree partitioning is applied to 128×56 pixels.

[0086] In HEVC, the partitioning of chroma CTBs is always aligned with the partitioning of the corresponding luma CTBs. It should be noted that the present invention can process chroma components in the same way, but the present invention is not limited to this. There may also be independent partitioning of different color components.

[0087] Back to Figure 1 After the division unit 110 performs image division, the transformation unit 130, the quantization unit 140 and the entropy coding unit 150 respectively perform transformation, quantization and entropy coding to generate a coded video bit stream as an output.

[0088] The video stream may include multiple frames. For example, blocks of the first frame of the video stream are intra-coded by the intra-frame prediction unit 190. Intra-frames are coded using only information from the frame and can therefore be decoded independently of other frames. Therefore, intra-frames can provide entry points (e.g., for random access) in the bitstream. Blocks of other frames of the video stream can be inter-coded by the inter-frame prediction unit 195: each block of the inter-coded frame is predicted based on blocks in another frame (reference frame, e.g., a previously coded frame). The mode selection unit 180 is used to select blocks of a frame for intra-frame prediction or inter-frame prediction, i.e., blocks of the frame are processed by the intra-frame prediction unit 190 or by the inter-frame prediction unit 195. The mode selection unit 180 also controls intra-frame or inter-frame prediction parameters. In order to be able to refresh image information, an inter-frame coded frame may include not only an inter-frame coded block, but also one or more intra-frame coded blocks. In contrast, an intra-frame contains only intra-frame coded blocks and does not contain inter-frame coded blocks. Intra frames may be inserted into a video sequence (eg regularly, ie each time after a certain number of inter frames) in order to provide an entry point for decoding, ie a point where the decoder can start decoding without using information from previous frames.

[0089] The intra prediction unit 190 is a block prediction unit. To perform spatial prediction or temporal prediction, the coding block may be further processed by an inverse quantization unit 145 and an inverse transform unit 135. After the reconstructor 125 reconstructs the block, a loop filter unit 160 may be used to further improve the quality of the decoded image. The reconstructor 125 adds the decoded residual to the predictor to obtain the reconstructed block. The filter block then forms a reference frame and is stored in a frame buffer 170. This decoding loop (decoder) on the encoder side has the advantage of generating the same reference frame as the reference image reconstructed on the decoder side. Therefore, the encoder side and the decoder side operate in a corresponding manner. The term "reconstruction" here refers to obtaining a reconstructed block by adding a decoded residual block to a prediction block.

[0090] The inter-frame prediction unit 195 includes as input the current frame or image to be inter-coded and one or more reference frames or images from the frame buffer 170. The inter-frame prediction unit 195 performs motion estimation and motion compensation. Motion estimation is used to obtain motion vectors and reference frames (e.g., based on a cost function). Motion compensation then describes the current block of the current frame based on the translation of the reference block of the reference frame to the current frame (i.e., by motion vectors). The inter-frame prediction unit 195 selects a prediction block (i.e., a predictor) for the current block from a set of candidate blocks (i.e., candidate predictors) in one or more reference frames so that the prediction block minimizes the cost function. In other words, the candidate block with the smallest cost function will be used as the prediction block for the current block.

[0091] For example, the cost function may be a measure of the difference between the current block and the candidate block, i.e., a measure of the residual of the current block relative to the candidate block. For example, the cost function may be the sum of absolute differences (SAD) between all pixels (pixel points) of the current block and all pixels of the candidate block in the candidate reference image. However, any similarity metric may generally be used, such as mean square error (MSE) or structural similarity metric (SSIM).

[0092] However, the cost function may also be the number of bits necessary to encode such inter blocks and / or the distortion caused by such encoding. Thus, a rate-distortion optimization process may be used to decide on motion vector selection and / or generally on coding parameters, such as whether to use inter prediction or intra prediction for a block, and which settings to use.

[0093] The intra-frame prediction unit 190 receives as input a block of a current frame or image to be intra-coded and one or more reference pixels of a reconstructed area of ​​the current frame. Then, the intra-frame prediction 190 describes the pixels of the current block of the current frame according to a function of the reference pixels of the current frame. The inter-frame prediction unit 190 outputs a prediction block of the current block, wherein the prediction block advantageously minimizes the difference between the current block to be coded and its prediction block, i.e., minimizes the residual block. For example, the residual block can be minimized based on a rate-distortion optimization process. In particular, the prediction block is obtained as a directional interpolation of the reference pixel. The direction can be determined by rate-distortion optimization and / or by calculating the above-mentioned similarity measure related to inter-frame prediction.

[0094] The difference between the current block and its predicted block, i.e. the residual block, is transformed by the transform unit 130. The transform coefficients are quantized by the quantization unit 140 and entropy encoded by the entropy coding unit 150. The coded video bitstream generated thereby includes intra-frame coded blocks and inter-frame coded blocks and corresponding signaling (e.g., mode indication, indication of the motion vector and / or intra-frame prediction direction). The transform unit 130 may perform a linear transform, such as a discrete Fourier transform (DFT) or a discrete cosine transform (DCT). The advantage of this spatial frequency domain transform is that the resulting coefficients usually have higher values ​​in lower frequencies. Therefore, after the significant coefficient scan (e.g., Z scan) and quantization, the resulting sequence of values ​​usually has some larger values ​​at the beginning and ends with a string of zeros. This further improves the coding efficiency. The quantization unit 140 performs lossy compression by reducing the resolution of the coefficient values. Then, the entropy coding unit 150 assigns binary codewords to the coefficient values. The codewords are written into a bitstream called a coded bitstream. The entropy encoder also performs signaling information ( Figure 1 The byte sequence number (not shown) may be encoded according to the partition flag syntax shown above.

[0095] Figure 2 is an example of a video decoder 200. The video decoder 200 includes, in particular, a reference image buffer 270 and an intra prediction unit 290 as a block prediction unit. The reference image buffer 270 is used to store at least one reference frame reconstructed from a coded video bitstream. The intra prediction unit 290 is used to generate a prediction block, which is an estimate of a block to be decoded. The intra prediction unit 290 is used to generate the prediction block based on reference pixels obtained from the reference image buffer 270.

[0096] The decoder 200 is used to decode the encoded video bitstream generated by the video encoder 100. Preferably, the decoder 200 and the encoder 100 generate the same prediction block for the corresponding block to be encoded / decoded. The characteristics of the reference picture buffer 270 and the intra-frame prediction unit 290 are similar to those of the decoder 200 and the encoder 100. Figure 1 The features of the reference picture buffer 170 and the intra prediction unit 190 are similar.

[0097] The video decoder 200 includes other units that also exist in the video encoder 100, for example, an inverse quantization unit 240, an inverse transform unit 230 and a loop filter unit 260, which correspond to the inverse quantization unit 140, the inverse transform unit 150, and the loop filter unit 160 of the video encoder 100, respectively.

[0098] The bitstream parsing, entropy decoding and partitioning unit 250 is used to parse and decode the received coded video bitstream to obtain quantized residual transform coefficients and signaling information. The quantized residual transform coefficients are fed to the inverse quantization unit 240 and the inverse transform unit 230 to generate residual blocks. The residual block is added to the prediction block of the reconstructor 225, and the resulting sum is sent to the loop filter unit 260 to obtain a decoded video block. The frame of the decoded video can be stored in the reference image buffer 270 and used as a reference frame for inter-frame prediction. The signaling information parsed and decoded from the bitstream may generally include control information related to frame partitioning. In order to further correctly parse and decode the image, the control information is used to restore the divided image to the coding unit so that the following decoded data can be correctly allocated to the corresponding coding unit.

[0099] generally, Figure 1 and Figure 2 The intra prediction units 190 and 290 may generate a prediction signal of a block to be encoded or decoded using reference pixels in the encoded region.

[0100] The bitstream parsing, entropy decoding and partitioning unit 250 receives the encoded bitstream as its input. The bitstream may be parsed first, ie, signaling parameters and residuals are extracted from the bitstream. The syntax and semantics of the bitstream may be defined by a standard so that the encoder and decoder can interoperate.

[0101] In the HEVC standard, a coding tree unit (CTU) or coding unit (CU) located on a slice / image boundary is divided using a forced quad tree (QT) partition until the lower right pixel of the leaf node is within the slice / image boundary. The forced QT partition does not need to be indicated in the bitstream. The purpose of the forced partition is to enable the codec to encode and decode the boundary CTU / CU. In other words, the encoding side and the decoding side agree to use QT partitioning without further signaling the partitioning mode.

[0102] Whether it is the patent of the QTBT structure [WO2016090568] or VTM-1.0., the boundary CTU / CU forced partitioning process follows HEVC. This means that the CTU / CU located on the frame boundary, especially the CTU / CU partially located outside the image / frame due to the boundary crossing (in the present invention, such CTU / CU is also referred to as "boundary CTU" and "boundary CU" respectively), is first forced to be divided by the quadtree (quadtree, QT for short) structure without rate-distortion (RD for short) optimization until the entire current CU is within the slice / image boundary. These forced partitions do not need to be indicated in the bitstream. Further partitioning can be achieved based on RD optimization. Figure 6 This is an example of forced partitioning of the bottom boundary CTU (128×128) of an HD (1920×1080 pixel) sequence by QT.

[0103] Since the boundary partition only uses the QT partition structure, the limitation of the multi-type tree (BT / TT / ABT) in VTM can be seen from the MaxBTTDepth in SPS. Therefore, for the boundary CTU (i.e., the CTU partially located on both sides of the boundary, such as Figure 6 The limit of the total level depth (TotalDepthLim), that is, the maximum level depth, is calculated using the following equation (1):

[0104] TotalDepthLim=QTDepth+MaxBttDepth (1)

[0105] In equation (1), MaxBttDepth is the maximum binary and ternary tree depth allowed as specified in the SPS. QTDepth is the hierarchical depth of the partition block of the quadtree partition. That is, for each partition step in the hierarchical partitioning applying the quadtree partitioning, the value of QTDepth is increased by one. For example, QTDepth may correspond to the parameter cqtDepth in the syntax example of Table 1. However, despite the increase in the QTDepth limit in each QT partitioning step, the hierarchical depth is ultimately limited by a predetermined minimum allowed CU size (e.g., corresponding to the SPS parameter minCUSize).

[0106] Image boundaries can be processed using BT, TT, or ABT (asymmetric binary tree). Both forced and adaptive methods can be used. If MTT (multi-type tree) such as BT / TT is used for boundary partitioning instead of QT partitioning, it is easy to exceed the limit of SPSMaxBTTDepth for BTT (binary tree and ternary tree). Figure 7This is an example of BT used in the bottom boundary CTU of an HD sequence. If BT is used starting from the CTU level until the leaf nodes (coding units) that are completely within the boundary, the BT partition depth is 4, which exceeds the MaxBTTDepth of SPS (MaxBTTDepth is set to 3 in VTM-1.0 (VVC Test Model Version 1.0)).

[0107] The object of the present invention is to improve the flexibility of dividing the boundary part and provide image boundary processing. In addition to the MaxBTTdepth of the SPS used for division in the prior art, a method of the present invention also defines a boundary division depth limit. To this end, a depth value (for example, it can be called ExtdMaxBTTDepth) is set specifically for boundary division, instead of using the MaxBttDepth in equation (1). ExtdMaxBTTDepth can be predefined and fixed in the SPS and indicated in the SPS bitstream, or it can also be calculated. Therefore, if multi-type tree (MTT) division (such as BT, TT or ABT) is used in the image boundary (forced or adaptive) processing, the maximum MTT depth limit can be extended in both the decoder and the encoder.

[0108] The following aspects and embodiments of the present invention describe possible ways to obtain and calculate ExtdMaxBTTDepth.

[0109] According to the general aspects, Figure 8 A device 800 for dividing an image into coding units is provided (which can be implemented in the encoder unit 110 or the decoder unit 250 described above). The device includes a processing circuit for dividing 810 the image into coding tree units CTU. The CTU includes a non-border CTU having a predetermined size in the horizontal and vertical directions, and a border CTU that partially falls in an image delimited by a horizontal or vertical image boundary, wherein the size of the portion is less than a predefined size perpendicular to the image boundary direction. The circuit is also used to divide the non-border CTU and the border CTU into individual coding units. Wherein, the hierarchical partitioning of the non-border CTU includes multi-type partitioning, and the partitioning depth of the multi-type partitioning of the hierarchical partitioning of the non-border CTU does not exceed the maximum non-border multi-type partitioning depth; and the multi-type partitioning refers to the partitioning direction being the vertical direction or the horizontal direction. In addition, the hierarchical partitioning 820 of the border CTU includes multi-type partitioning, and the partitioning depth of the multi-type partitioning of the hierarchical partitioning of the border CTU does not exceed the maximum border multi-type partitioning depth.

[0110] Therefore, the circuit of the apparatus 800 for dividing an image into coding units is used to divide the non-border CTU into coding units using a first division depth limit (i.e., the maximum non-border multi-type division depth), wherein the first division depth limit is used for the MTT division included in the division. The circuit is also used to divide the border CTU 820 into coding units using a second division depth limit (i.e., the maximum border multi-type division depth), wherein the second division depth limit is used for the MTT division included in the division. In other words, according to the present invention, the circuit of the apparatus 800 uses the corresponding maximum MTT division depth (MTT depth limit) for non-border CTUs and border CTUs in operation.

[0111] The CTU obtained by frame division 810 may be further divided into layers, including a layer division 820 for a boundary CTU. This division may be performed in any manner, such as Figures 3 to 5 The divisions shown and described above in conjunction with them.

[0112] Figure 8 The internal structure of the circuit of the device 800 is shown. The circuit can be any type of hardware and software having functional units 810 and 820, and the functional units 810 and 820 are respectively used to divide (divide) the image (or frame) into CTUs, and divide the CTUs, wherein the CTUs include the non-boundary CTUs and especially include the boundary CTUs. These units can be implemented on a single processor. However, the present invention is not limited to such applications, and these units can also be implemented by separate hardware components.

[0113] In the present invention, the term "boundary CTU" is used to refer to a coding tree unit that is divided by a picture boundary into a part that falls within the picture to be divided (within the picture boundary) and a part that does not fall within the picture (i.e., is located outside the picture boundary). A boundary CTU exists if the size of the picture to be encoded is not an (integer) multiple of the CTU size in at least one direction.

[0114] Fig. 9 An example of a bottom image boundary 900 and a corresponding boundary portion 910 (shaded) including a boundary CTU is shown. Portion 950 shows the remainder of the image having a size in the vertical and horizontal directions that is an integer multiple of a CTU.

[0115] In addition, the vertical size of the CTU is represented as 970V, and the horizontal size of the CTU is represented as 970H. Fig. 9As shown, the boundary portion in this example is an integer multiple of the CTU size 970H in the horizontal direction. However, in the vertical direction, the size of the boundary portion 910 is the size of the remainder of the vertical image size divided by the vertical CTU size 970V. Portion 920 is only virtual and shows the difference between the height of the boundary portion and the CTU size. It should be noted that in the current implementation, the CTU is square, so 970H and 970V are the same size. However, the present invention is not limited to this, and the vertical and horizontal sizes of the CTU can be different.

[0116] To encode (and accordingly decode) the boundary portion, Fig. 9 The boundary portion 910 in the image is divided into incomplete CTUs, that is, CTU portions whose horizontal size is CTU 970H and whose vertical size is smaller than CTU 970V. These incomplete CTUs correspond to "boundary CTUs" in this application, and "incomplete CTUs" correspond to CTU portions within the image. Fig. 9 Horizontal boundary portions are shown, but there may be vertical boundary portions in addition to or instead of horizontal boundary portions. In particular, if the picture size (i.e., the width and height of the picture) is not a multiple of the CTU size in both the vertical and horizontal directions, there is also an incomplete CTU, which is both a horizontal boundary CTU and a vertical boundary CTU, wherein the vertical and horizontal sizes of the CTU portion within the picture are smaller than the size of the complete CTU.

[0117] In contrast to "boundary CTU", the term "non-boundary CTU" is used to refer to any CTU that is completely within a frame or picture, i.e. Fig. 9 In the example shown, the CTUs within the portion 950. That is, the non-border CTUs are located within the image to be encoded / decoded and have full size in the vertical and horizontal directions. However, the non-border CTUs may be adjacent to at least one image boundary, or they may also be other CTUs between the non-border CTU and any image boundary.

[0118] It should also be noted that non-border CTUs and border CTUs do not constitute different kinds of CTUs that differ from each other in the way they are coded or specified. Non-border CTUs and border CTUs differ in whether they are located on a boundary (ie, crossing their boundaries) or within the boundaries that delimit a picture.

[0119] In addition, whether a CTU is a boundary CTU can be determined by comparing the position of the CTU (particularly the appropriate pixel position in the CTU) with the position of the boundary (or the vertical / horizontal image size in pixels). During the encoding and decoding process, the CTU has a fixed predefined size, such as 128×128 or 64×64 in the above-mentioned HEVC. The image is divided into non-overlapping CTUs. The lower right corner can be selected as a suitable pixel position because if the image is processed from left to right and from top to bottom (usually, which pixel position is most appropriate depends on the processing direction), the lower right corner allows the CTU to be determined as a boundary CTU relative to each vertical and horizontal image boundary. The codec checks the pixel in the lower right corner of the CTU and compares it with the vertical and horizontal size of the image. If the lower right pixel is within the image boundary, the CTU is a non-boundary CTU, otherwise it is a boundary CTU. This way of determining whether a partition block is on the boundary is not only applied to CTUs, but also to CUs and any partition blocks generated by partitioning CTUs or certain blocks in the partition hierarchy.

[0120] In addition, other methods may be used to determine whether a CTU is a boundary CTU. For example, as described above, the image size (width / height) divided by the CTU size may determine the number of CTUs and whether there are boundary CTUs in the vertical and horizontal directions. The CTUs may be indexed, and if there are boundary CTUs, the last k CTUs may be determined as bottom boundary CTUs, or every kth CTU may correspond to the right boundary CTU of each row (k is the number of CTUs (including boundary CTUs), i.e., the upper limit of the ratio of the image width to the CTU size).

[0121] Combination of the above Figure 3 The hierarchical depth in hierarchical partitioning is explained. Thus, the partition depth corresponds to the number of partitioning steps performed to obtain a certain partition level, starting from a CTU with a partition depth of 0 corresponding to level 0 (or layer 0) to a CTU with a coding unit depth.

[0122] For non-boundary CTUs, the hierarchical depth of the coding unit is limited because it cannot exceed the maximum hierarchical depth required by the setting (e.g., the setting in the SPS). For example, in the setting, different limits can be set for the corresponding partition depths of quadtree partitioning and multi-type partitioning. For example, in different levels or different partial blocks at the same level, since depth limits can be set separately for QT partitioning and MTT partitioning, the partitioning of the CTU includes QT partitioning and MTT partitioning (such as BT, TT, and ABT partitioning). For example, the maximum non-boundary MTT depth may correspond to the parameter MaxBTTDepth of the current VTM, or may be redefined and named, such as MaxMTTDepth, and applied to all MTT partition types, such as BT and TT, etc. Therefore, the depth limit, that is, the maximum total depth of the CTU partitioning can be the sum of the depth limits for QT partitioning and MTT (e.g., BTT) partitioning.

[0123] On the other hand, for the boundary CTU, the present invention provides a depth limit as shown in the following equation (2):

[0124] TotalDepthLim=QTDepth+ExtdMaxBTTDepth (2)

[0125] As can be seen from equation (2), the depth limit of the boundary CTU (also referred to as the boundary partition depth in the present invention) is generally composed of the QT partition depth and the MTT partition depth (not limited to binary tree and ternary tree partition, the name ExtdMaxBttDepth is only considered as an example). For the case of mandatory QT partition, the QT depth can be the same as the QT depth used in equation (1). However, equation (2) differs from equation (1) in that the maximum boundary multi-type partition depth ExtdMaxBTTDepth (extended maximum binary tree / ternary tree depth) can also be referred to as, for example, "ExtdMaxMTTDepth".

[0126] Corresponding to the device 800, another aspect of the present invention provides a method for dividing an image into coding units. The method includes the step of dividing the image into a plurality of coding tree units CTU. The CTU includes a non-border CTU having a predetermined size in the horizontal and vertical directions, and a border CTU that partially falls in an image defined by a horizontal or vertical image boundary, wherein the size of the portion is smaller than the predefined size in a direction perpendicular to the image boundary. The method also includes the step of hierarchically dividing the non-border CTU and the border CTU into individual coding units. The hierarchical division of the non-border CTU includes multi-type division, and the division depth of the multi-type division of the hierarchical division of the non-border CTU does not exceed the maximum non-border multi-type division depth; the multi-type division refers to a division in which the division direction is a vertical direction or a horizontal direction. The hierarchical division of the border CTU includes multi-type division, and the division depth of the multi-type division of the hierarchical division of the border CTU does not exceed the maximum border multi-type division depth.

[0127] Several exemplary embodiments of the present invention will be described below, which further specify an apparatus 800 for dividing an image into coding units and a corresponding method.

[0128] Embodiment 1

[0129] As described above, since HEVC or VTM-1.0 uses mandatory QT in the partitioning of boundary CTUs, there is no need to define a separate BTT depth limit for boundary partitioning. In HEVC or VTM-1.0, the parameter MaxBttDepth in equation (1) applies to both boundary CTUs and non-border CTUs.

[0130] However, since CE 1 starts to use BT or TT as the boundary partition, it may not be appropriate to limit the MaxBTTDepth of the boundary according to equation (1) for the reasons explained above in conjunction with SubCE2. In embodiment 1, a new MaxBTTDepth is defined, which is only used for boundary partitioning when the MTT partitioning structure is involved in the boundary partitioning.

[0131] The present invention provides a new definition of the limit of the boundary partition MTT depth in hybrid video coding. As mentioned above, this maximum boundary multi-type partition depth can be obtained by calculation or pre-definition. This embodiment 1 provides several exemplary methods for calculating the maximum boundary multi-type partition depth.

[0132] In the first embodiment, the maximum boundary multi-type division depth is at least the sum of the adaptive boundary multi-type division depth and the predefined multi-type division depth. The adaptive boundary multi-type division depth refers to the multi-type division depth with the image boundary direction as the division direction.

[0133] Therefore, in order to maintain the flexibility of boundary block partitioning, a reasonable handling of the total depth limit of blocks located in the boundary CTU is proposed, where, compared with the QT partition in the JEM image boundary defined by equation (1), the rationality lies in directly performing MTT partitioning on the boundary CTU.

[0134] To this end, the maximum boundary MTT partition depth used in equation (2) is redefined as

[0135] ExtdMaxBTTDepth=BTTBPDepth+MaxBTTDepth (3).

[0136] Among them, the maximum boundary MTT partition depth represented by ExtdMaxBTTDepth is brought into equation (1) or (2). Therefore, the total partition depth limit of the boundary CTU is redefined as

[0137] TotalBPDepthLim=QTDepth+(BTTBPDepth+MaxBTTDepth) (4)

[0138] In equations (3) and (4), BTTBPDepth (i.e., the adaptive boundary multi-type partitioning depth of the first embodiment) is determined in each layer, and therefore also varies with the partitioning layer (i.e., the layer of the coding block in each partitioning step and the layer including the CU finally encoded and decoded obtained by the partitioning). For example, the same adaptive boundary multi-type partitioning depth can be selected for different MTT partitioning modes such as BT / TT / ABT boundary partitioning. At each partitioning layer, regardless of whether BT, TT, ABT or other types of MTT partitioning are performed on the layer, the adaptive boundary multi-type partitioning depth is re-determined (i.e., increased).

[0139] As in the case of QT partitioning, the absolute depth limit can be derived from a predetermined minimum CU size, such as the SPS parameter minCUSize mentioned above. For a BT partitioning step, the depth of each partitioned block resulting from this step is increased by 1. For a TT partitioning, the depth of the resulting 1 / 2-sized block is increased by 1 and the depth of the resulting 1 / 4-sized block is increased by 2, according to VTM-1.0, to establish the relationship between depth and coding unit size and to ensure compatibility between BT and TT partitioning.

[0140] As mentioned above, the adaptive multi-type division depth refers to the depth with the image boundary direction as the division direction. This means that the division line that divides the block on the corresponding layer into two, three or more blocks in the subsequent deeper layer has the same direction (i.e., parallel) as the image boundary under consideration that passes through the boundary CTU to be divided. However, the division / division steps whose division direction is perpendicular to the image boundary under consideration are not helpful for the value of the adaptive multi-type division depth. In particular, if the image boundary is a horizontal boundary, the adaptive multi-type division depth only calculates the steps in the horizontal division direction. Otherwise, if the image boundary is a vertical boundary, only the steps in the vertical division direction are calculated. This is in Fig.10 and Fig.11 This can be seen in the partitioning example shown.

[0141] However, the present invention is also applicable to boundary CUs and boundary partition blocks within the partition hierarchy of the boundary CTU, which are both located at the horizontal (e.g., bottom) boundary and the vertical (e.g., right) boundary, such as the CTU at the lower right corner. In this case, the MTT steps for the two partition directions will be calculated by the term BTTBPDepth in equation (3) or a separate corresponding variable.

[0142] On the other hand, it can be seen from equations (3) and (4) that, for example, the non-adaptive term in the maximum boundary multi-type partition depth (i.e., the predefined multi-type partition depth) may be equal to the maximum non-boundary multi-type partition depth. However, the present invention is not limited thereto. For example, when it is necessary to define an additional predefined multi-type boundary partition depth parameter, it may be beneficial to set this parameter to a smaller value (e.g., to save one or more bits).

[0143] Fig.10 This is a first example of using binary tree partitioning for boundary partitioning, and in particular, an example of using BT for bottom boundary partitioning. The starting point shown on the left side of the upper half of the figure shows a boundary CTU and a bottom image boundary (thick solid line) passing through the boundary CTU. In addition, there is a marker block object obtained by image partitioning at the upper right corner of the CTU. The right side of the upper half of the figure shows the final partitioning (i.e., the partitioning type) of the boundary CTU. However, in Fig.10 In the example shown, only the MTT (especially BTT) partitioning step is considered. According to equation (4), QT partitioning can be performed in other steps before the MTT partitioning step. Therefore, the starting point is not necessarily a CTU, but can also be a square coding block (equal vertical and horizontal size) with a depth greater than 0 generated by QT partitioning, etc.

[0144] Fig.10The lower half shows steps 1 to 3 of dividing the boundary CTU from the starting point to the final division. Among them, the thick solid line is the image boundary, the BTT boundary and the CTU are divided into solid lines (inside the boundary) or dashed lines (outside the boundary), and the dotted dashed line (step 3) is the normal BTT division (that is, the division in the vertical direction, not the boundary division with the image boundary direction as the division direction). If MaxBTTDepth is used according to equation (1) (the limit in VTM is 3), the object block in the upper right corner of the CTU cannot be further divided because there are 3 MTT division steps, so CurrentBTTDepht(3)>=MaxBTTDepth(3) (CurrentBTTDepth is the BTT division depth corresponding to the number of BTT division steps performed in the current step).

[0145] According to the first embodiment, there is a layer (BTT) of boundary division in step 1 (i.e., the division direction is the boundary direction). Therefore, BTTBPDepth∈=1. Therefore, according to equation (3), it can be concluded that ExtdMaxBTTdepth=BTTBPDepth+MaxBTTDepth=1+3=4 (in SPS, MaxBTTDepth is set to 3). The maximum BTT depth layer (maximum boundary multi-type division depth) in step 1 is 4. Since the block has been divided once, the divided block can be further divided into three MTT layers according to the maximum boundary multi-type division depth according to the ExtdMaxBTTdepth set in step 1.

[0146] In step 2, so far, the boundary partitioning obtained by BP (boundary partitioning) has two layers, BTTBPDepth = 2. Therefore, according to equation (3), it can be concluded that ExtdMaxBTTdepth = BTTBPDepth + MaxBTTDepth = 2 + 3 = 5 (in SPS, MaxBTTDepth is set to 3). The new maximum BTT depth layer (maximum boundary multi-type partitioning depth) in step 2 is 5 (calculated from the starting point (a)). Since the block has been divided twice, the divided block can be divided 3 more times according to the maximum boundary multi-type partitioning depth. It can be seen that due to the boundary partitioning (that is, the division direction is horizontal, which is the same as the image boundary direction), from step 1 to step 2, BTTBPDepth and the corresponding ExtdMaxBttDepth increase by 1.

[0147] Step 3 performs BT division again. However, the division direction of this final division is not the direction of the image boundary. In particular, the division direction in step 3 is perpendicular to the image boundary. Therefore, step 3 is not beneficial to the BPPBPDepth value. It can be seen that the BTT BP still has 2 layers, BTTBPDepth=2. Among them, ExtdMaxBTTdepth=BTTBPDepth+MaxBTTDepth=2+3=5 (in SPS, MaxBTTDepth is configured as 3). As mentioned above, in the adaptive multi-type division depth, only the depth (number of steps / layers) of the multi-type division with the image boundary direction as the division direction is useful. However, the maximum boundary multi-type division depth is a limitation for any MTT boundary division. Therefore, the maximum depth layer in step 3 is still 5 (calculated from the starting point). Since the block has been divided 3 times (all divisions are considered here, not just BP), the divided block can be subjected to 2 more BTT divisions.

[0148] Fig.11 The second example of maximum MTT partition depth calculation. The thick solid line in the figure is the image boundary, the solid line (inside) or the dotted line (outside) is the CTU boundary or BTT boundary division, and Fig.10 The same. There is a marked object in the boundary CTU (i.e., the target coding block obtained by MTT partitioning). In order to obtain the final partition type in the upper right part of the figure, four partitioning steps need to be performed, as shown in steps 1 to 4 in the figure.

[0149] If MaxBTTDepth is used according to Equation 1 and is limited to 3 in VTM, since CurrentBTTDepht(4)>=MaxBTTDepth(3), the marked target block (MTT depth is 4) cannot be obtained and cannot be further divided.

[0150] exist Fig.11 In step 1, BTT BP has one layer, BTTBPDepth = 1. Therefore, ExtdMaxBTTdepth = BTTBPDepth + MaxBTTDepth = 1 + 3 = 4 (in SPS, MaxBTTDepth is configured as 3). The maximum BTT depth layer in step 1 is 4. Since the block has been divided once, the divided block can be divided into 3 more BTT layers.

[0151] exist Fig.11 In step 2, BTT BP has two layers, BTTBPDepth = 2. Therefore, ExtdMaxBTTdepth = BTTBPDepth + MaxBTTDepth = 2 + 3 = 5. The maximum BTT depth layer in step 2 is 5. Since the block has been divided once, the divided block can be divided into 3 more BTT layers.

[0152] exist Fig.11 In step 3, BTT BP has 3 layers, BTTBPDepth = 3. Therefore, ExtdMaxBTTdepth = BTTBPDepth + MaxBTTDepth = 3 + 3 = 6. The maximum BTT depth layer in step 2 is 6. Since the block has been divided once, the divided block can be divided into 3 more BTT layers.

[0153] In addition, in step 4, the BTT is divided into 4 layers. ExtdMaxBTTdepth=BTTBPDepth+MaxBTTDepth=4+3=7. The maximum depth layer in step 4 is 7 (calculated from the starting point (a)). Since the block has been divided 4 times, the divided block can be divided at most 3 times.

[0154] In the maximum boundary multi-type partition depth, all partitions are considered and restricted, not just the BP with the boundary direction as the partition direction. However, Fig.11 All steps of the illustrated example perform multi-type partitioning with the image boundary direction as the partitioning direction. Therefore, in each step, the value of the adaptive boundary multi-type partitioning depth (BTTBPDepth in this example) increases by 1.

[0155] By limiting the depth of the boundary multi-type partitioning to the number of partitioning steps with the same partitioning direction as the boundary direction, the present invention makes the partitioning type of the coding unit more practical, wherein the coding unit is similar to the remaining part of the CTU near the boundary of the picture. In particular, the number of coding units in the boundary CTU / part can be reduced compared to using only QT partitioning.

[0156] Embodiment 2

[0157] According to the explanation of the above-mentioned embodiment 1, the maximum boundary multi-type partition depth (for example, the given ExtdMaxBTTDepth) is at least the sum of the adaptive boundary multi-type partition depth and the predefined multi-type partition depth. Now, according to the present embodiment 2, the sum also includes: a function of the ratio of the size of the boundary partition block of the boundary CTU in the direction of the image boundary and the direction perpendicular to the image boundary. The boundary partition block is a block of the adaptive boundary multi-type partition depth.

[0158] In the present invention, the term "partition block" refers to a CTU or block obtained by dividing a lower-level block / unit. Therefore, the partition block is not limited to the CTU at the top of the hierarchical partition or the coding unit as the final result of the hierarchical partition to be encoded / decoded, but also includes the CTU at the intermediate partition step (e.g., Fig.10 Steps 1 and 2 in , or Fig.11In addition, a "boundary partition block" is a partition block located on the image boundary (a picture boundary that passes through the block), just as a boundary CTU is located on the boundary.

[0159] If the boundary under consideration is a horizontal image boundary, the size ratio is the ratio of the horizontal size (width) of the boundary partition block divided by the vertical size (height). On the other hand, if the image boundary is vertical, the ratio is the height of the boundary partition block divided by the width.

[0160] For example, the function of the ratio can be a binary logarithm log2 Ratio. In the first embodiment, the total depth limit is regarded as a criterion for defining the boundary block BTT limit. Compared with the calculation of the MTT BP depth limit using equation 1, the second embodiment particularly takes the reachable size of the coding unit as a fairness criterion (i.e., fairness between QT and MTT partitions). Therefore, the maximum boundary multi-type partition depth is expanded according to the following equation (5).

[0161] ExtdMaxBTTDepth=log2Ratio+BTTBPDepth+MaxBTTDepth, (5)

[0162] Here, Ratio represents the width-to-height ratio (width / height if the border is horizontal, or height / width if the border is vertical).

[0163] Fig.12 This is an example of boundary division according to the second embodiment. Fig.12 Part (a) shows the boundary partitioning of VTM-1.0 using forced QT, while Fig.12 Part (b) is the boundary demarcation using BT. Fig.10 and 11 As shown in the figure, solid represents within the boundary. Dashed lines represent CTU and boundary divisions (division direction is the boundary direction), and dotted dashed lines represent divisions with different division directions from the boundary direction. Fig.12 In part (a), the marker block at the top left corner of the CTU can be further divided into 3 additional BT levels because in the configuration of VTM-1.0, the MaxBTTDepth of the SPS is set to 3 and no BT division has been performed so far. In part (b), the marker block at the top left corner of the CTU can also be divided into another 3 BT (or other MTT type) levels, so the ExtdMaxBTTDepht of the marker block is obtained from equation (5) as follows:

[0164] ExtdMaxBTTDepth = log2(boundary partition block ratio (4)) + BTTBPdepth (2) + MaxBTTDepth (3) = 7

[0165] exist Fig.12 In the case of the current embodiment 2 shown in part (b), according to ExtendedMaxBTTDepth representing the maximum boundary multi-type division depth, the marker block can be further divided into three BT levels: (ExtdMaxBTTDepht(7)-CurrentBTDepht(4))=3.

[0166] Here, it is assumed that the marker block has been obtained through two BT division steps with the boundary direction as the division direction, and then two BT division steps with the division direction perpendicular to the boundary division, so the current BT depth obtained is equal to 4. However, the MTT division of the present invention is not limited to BTT division. For example, in a single step of dividing a partition block three or more times in the same division direction, a partitioning pattern can also be used to obtain four partition blocks (this partitioning pattern for obtaining four partition blocks is sometimes called SplitInto4, which is different from the QT division that can obtain four partition blocks in the same step). Fig.12 In the example of , if the last two BT partitioning steps are replaced by a SplitInto4 partitioning step, the resulting current MTT depth is 3 instead of 4.

[0167] In addition, in MTT, the calculated hierarchical depth can be determined based on the block size obtained by the minimum result partitioning. For example, in the current TT implementation in VTM-1.0, the block is divided into 1 / 4, 1 / 2 and 1 / 4 sub-blocks / partitioned blocks. For the first and third partitioned blocks, the depth is calculated twice in VTM-1.0 to ensure that the depth is equally valid for BT and TT (and the ABT step can also be calculated twice for the smaller of the resulting partitioned blocks). Therefore, to ensure compatibility with BT partitioning, a single SplitInto4 step can also be counted as two steps. In this way, the block size of the resulting partitioned block / coding unit can be seen from the depth. However, in some other possible methods, if it is guaranteed (for example, from a variable) that the corresponding partition type is known, each step can be calculated once.

[0168] It should be noted that in equation (5), the binary logarithm of the right boundary partitioning block ratio (first term) may be the same as the MTT BP depth (second term), especially if BT partitioning is applied (e.g. Fig.12 (as shown in part (b)).

[0169] It should also be noted that the function of dividing the boundary into blocks in the present invention is not limited to the binary logarithm. For example, the ratio itself (identity function) can be used, or the ratio can be multiplied by a constant or some adaptive parameters.

[0170] Embodiment 3

[0171] The above-mentioned embodiments 1 and 2 have described how to calculate the maximum boundary multi-type partition depth. On the other hand, in embodiment 3, the maximum boundary multi-type partition depth is predefined (i.e., fixed). The predetermined value may be provided as the term BTTBPDepth in equation (4), or it may replace MaxBttDepth in the equation for boundary CTU (such as equation (1)).

[0172] For example, such a fixed maximum boundary multi-type partition depth can be obtained from signaling in the bitstream. For example, it can be indicated in a set of control parameters common to one or more video frames (such as PPS, SPS and VPS in HEVC). It can also be indicated for an entire video sequence at one time. The signaling may include further encoding of the parameters, such as differential encoding, predictive encoding, entropy encoding or any other encoding embedded in the bitstream.

[0173] However, as described above, the present invention including the above-mentioned embodiments 1 to 3 is not limited to the maximum boundary multi-type partition depth fixed and signaled in the bitstream. For example, it can also be derived based on one or more parameters transmitted in the bitstream (such as MaxBTTDepth in SPS) and / or based on internal variables such as BTTBPDepth or QTDepth. However, the derivation can also be predefined or indicative according to the standard.

[0174] Although the description of some examples (including equations 1 to 5) involves BTT partitioning and names the variables accordingly (e.g., BTTBPDepth and MaxBTTDepth), the present invention is also applicable to other types of MTT partitioning, such as ABT (asymmetric binary tree partitioning) partitioning (dividing a partition block into two or more partition blocks of different sizes, such as 1 / 4 and 3 / 4 in one partitioning step), or combining Fig.12 (b) The SplitInto4Mode mentioned in part. Therefore, the variable can have different names, such as BTTBPDepth, MaxBTTDepth, etc. Therefore, the present invention facilitates image boundary processing using multiple types of partitioning modes other than QT partitioning, and enhances flexibility without considering the specific MTT partitioning mode used.

[0175] However, in some embodiments, in addition to MTT partitioning, the hierarchical partitioning of boundary CTUs (and the hierarchical partitioning of non-boundary CTUs) may also include QT partitioning. This can be seen from equation (4), which defines the total maximum boundary partitioning depth as the sum of the maximum boundary multi-type partitioning depth (the term in brackets) and the QT partitioning depth (internal variable QTDepth). For example, as has already occurred in VTM-1.0, quadtree partitioning may be performed before MTT partitioning. That is, the QT partitioning step is performed first, and then the MTT partitioning step is performed. However, the present invention is not limited to this, and it is also applicable to a configuration in which MTT partitioning is performed first and QT partitioning is performed later. Therefore, as described above, Fig.10 and 11 The starting point in does not have to be a CTU, but can also be a partition block obtained from one or more previous partitioning steps.

[0176] In addition, according to the present invention, the maximum boundary multi-type partitioning depth may be greater than or equal to the maximum non-boundary multi-type partitioning depth. For example, if multi-type partitioning with the boundary direction as the partitioning direction is not performed in the boundary CTU, the maximum boundary multi-type partitioning depth may be equal to the maximum non-boundary multi-type partitioning depth. In this case, there is no partitioning step that increases the internal variable BTTBPDepth (or MTTBPDepth). Still as described above, the device for dividing an image into coding units adopts different methods to determine the non-boundary multi-type partitioning depth and the maximum boundary multi-type partitioning depth, respectively.

[0177] However, in the third embodiment in which the maximum boundary multi-type partition depth is predetermined, the predetermined value should be greater than the maximum non-boundary multi-type partition depth so that the MTT partition in the boundary CTU is more flexible.

[0178] Fig.13 Flow chart of the maximum BT / TT (or MTT in general) depth extension of the present invention. The condition "inside boundary CTU" indicates whether the current CTU / CU is located in a boundary CTU. If the condition is false (N), the normal MaxBTTDepth of the SPS is used as the non-border multi-type partition depth to limit the maximum possible BT / TT (or other MTT) depth. Otherwise, MaxBttdepth will be extended and determined according to any embodiment of the present invention.

[0179] As described above, the image segmentation device 800 may be embedded in Figure 1 and Figure 2The video encoding device 100 or decoding device 200 shown in the figure. Therefore, the present invention also provides a device 100 for encoding an image of a video sequence, that is, an encoding device. The encoding device 100 includes: an image division device 800 according to any embodiment described in the present invention; an image encoding unit, which is used to encode a coding unit; and a bit stream forming unit for generating a bit stream, wherein the bit stream includes the encoded coding unit and division information indicating how to divide the coding tree unit.

[0180] In particular, for each CTU or partition block in the intermediate step, the partition information may include information indicating which partition type mode among the QT and several MTT partition modes is used. For example, for each partition of the partition block or CTU, the bitstream may include a parameter indicating the partition mode. Alternatively, a specific partition mode may be forced to be used.

[0181] It should be noted that the partitioning mode of the boundary CTU and the non-border CTU may be different. For example, as described above, the partitioning mode may be indicated based on the CTU of the boundary CTU and the non-border CTU (and / or based on the partition block). Alternatively, the partitioning mode of all CTUs on a certain boundary (vertical or horizontal) may be the same and set in the signaling associated with one or more video pictures.

[0182] However, the partitioning mode of the boundary CTU may be determined differently from the partitioning mode of the non-boundary CTU. For example, for the boundary CTU (and partition block), a predetermined partitioning mode may be mandatory, that is, the standard may define a fixed partitioning mode or algorithm for determining the partitioning mode of the boundary CTU, for example, based on its position and / or size within the boundary and / or the partitioning mode of the adjacent CTU.

[0183] Accordingly, the present invention also provides a method for encoding an image of a video sequence. The method comprises: a step of dividing an image according to any one of the embodiments described in the present invention; an image encoding step of encoding the coding unit; and a bit stream forming step of generating a bit stream, wherein the bit stream comprises the encoded coding unit and division information indicating how to divide the coding tree unit.

[0184] The present invention also provides an apparatus 200 and method for decoding an image of a video sequence. If embedded in the decoding apparatus 200, the apparatus 800 is used to determine the division of a (decoded) image into coding units. The decoding apparatus 200 comprises: a bitstream parser for parsing a bitstream comprising encoded coding units; an apparatus 800 for determining the image division according to any embodiment; and an image decoding unit for decoding the encoded coding units based on the determined image division.

[0185] Accordingly, the image decoding method comprises: a step of parsing a bit stream including the encoded coding unit; a step of determining image division according to any embodiment of the present invention; and an image decoding step of decoding the encoded coding unit based on the determined image division.

[0186] On the decoder side, the maximum BT / TT depth will be extended on the boundary CTU / CU, that is, the maximum boundary multi-type partition depth will be used for the boundary CTU to be decoded. The extension can be simply parsed from the SPS or derived based on a certain condition. Based on embodiment one, the maximum BTT depth can be extended with the BTT boundary partition depth. And based on embodiment two, the maximum BTT depth can be extended with the ratio of the boundary partition block and the BTT boundary partition depth.

[0187] If, in the process of encoding and decoding the video image by the codec and the corresponding coding and decoding method, the division is performed (or the division is determined) according to the third embodiment, the maximum boundary multi-type division depth is predefined. At this time, the encoding / decoding bit stream may also include a coding sequence parameter set, including the maximum boundary multi-type division depth. At the decoder side, the image division determination device 800 is also used to obtain a second maximum multi-type division depth from the sequence parameter set.

[0188] Fig.14 A conceptual or schematic block diagram of an embodiment of a coding system 300 (e.g., an image coding system 300), wherein the coding system 300 includes a source device 310 for providing encoded data 330 (e.g., an encoded image 330) to be sent to a destination device 320, etc., for decoding the encoded data 330.

[0189] The source device 310 includes the encoder 100 or encoding unit 100 , and may additionally (ie, optionally) include an image source 312 , a pre-processing unit 314 (eg, an image pre-processing unit 314 ), and a communication interface or communication unit 318 .

[0190] The image source 312 may include or may be any type of image capture device, such as a device for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Hereinafter, unless otherwise specifically stated, all of these types of images and any other types of images will be referred to as “images” or “pictures”, and the previous explanation of the term “image” (including “video images” and “still images”) still applies unless otherwise specified.

[0191] A (digital) image is or can be considered as a two-dimensional array or matrix of pixels with intensity values. The pixels in the array may also be called pixels (picture elements for short) or pictures. The size and / or resolution of an image is defined by the number of pixels in the array or image in the horizontal and vertical directions (or axes). To represent colors, three color components are usually used, i.e. the image can be represented as or includes three pixel arrays. In the RBG format or color space, the image includes corresponding arrays of red, green and blue pixels. However, in video coding, each pixel is usually represented by a luminance / chrominance format or color space, for example, YCbCr, including a luminance component represented by Y (sometimes also represented by L) and two chrominance components represented by Cb and Cr. The luminance (luma) component Y represents the brightness or grayscale intensity (for example, in a grayscale image), while the two chrominance (chroma) components Cb and Cr represent the chrominance or color information components.

[0192] Thus, an image in the YCbCr format includes a luma pixel array of luma pixel values ​​(Y) and two chroma pixel arrays of chroma values ​​(Cb and Cr). An image in the RGB format can be converted or transformed into the YCbCr format and vice versa, a process also known as color conversion or color transformation. If the image is monochrome, the image may include only a luma pixel array.

[0193] The image source 312 may be a camera or memory (e.g., an image memory including or storing previously captured or generated images) for capturing images, and / or any type of (internal or external) interface to acquire or receive images. For example, the camera may be a local or integrated camera integrated in the source device, and the memory may be a local or integrated memory (e.g., integrated in the source device). For example, the interface may be an external interface for receiving images from an external video source (e.g., an external image capture device such as a camera or an external memory) or an external image generating device (e.g., an external computer graphics processor, a computer, or a server). The interface may be any type of interface according to any proprietary or standardized interface protocol, such as a wired or wireless interface, an optical interface, etc. The interface for acquiring image data 313 may be the same as the communication interface 318, or may be part of the communication interface 318. The communication interface may be any interface such as an Ethernet, WLAN, Bluetooth, or LTE interface, or may be any wired or wireless interface, such as a satellite or optical interface. The transmission may be a peer-to-peer transmission, a broadcast transmission, or a multicast transmission.

[0194] In distinction from the pre-processing unit 314 and the processing performed by the pre-processing unit 314 , the image or image data 313 may also be referred to as an original image or original image data 313 .

[0195] The preprocessing unit 314 is used to receive (raw) image data 313 and preprocess the image data 313 to obtain a preprocessed image 315 or preprocessed image data 315. The preprocessing performed by the preprocessing unit 314 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or denoising.

[0196] The encoder 100 is configured to receive pre-processed image data 315 and provide encoded image data (more details have been described above, for example, based on Figure 1 description).

[0197] The communication interface 318 of the source device 310 can be used to receive the encoded image data and transmit it directly to another device (e.g., the destination device 320 or any other device) for storage or direct reconstruction, or to process the encoded image data before storing the encoded data 330 and / or transmitting the encoded data 330 to another device (e.g., the destination device 320, or any other device for decoding or storage), respectively.

[0198] The destination device 320 includes the decoder 200 or the decoding unit 200 , and may additionally (ie, optionally) include a communication interface or a communication unit 322 , a post-processing unit 326 , and a display device 328 .

[0199] The communication interface 322 of the destination device 320 is used to receive the encoded image data or the encoded data 330, for example, directly from the source device 310 or any other source (eg, a memory such as an encoded image data memory).

[0200] The communication interface 318 and the communication interface 322 may be used to transmit or receive the encoded image data or the encoded data 330, respectively, via a direct communication link (a direct wired or wireless connection, etc.) between the source device 310 and the destination device 320, or via any type of network (e.g., a wired (e.g., optical cable, power line, copper cable or coaxial cable, or based on any other medium) or a wireless network or any combination thereof, or any type of private and public network), or any combination thereof.

[0201] For example, the communication interface 318 may be used to package the encoded image data into a suitable format (eg, data packets) for transmission over a communication link or communication network, and may also include data loss protection and data loss recovery.

[0202] The communication interface 322 is a corresponding part of the communication interface 318, and may be used, for example, to depacketize the encoded data 330 to obtain encoded image data, and may also be used to perform data loss protection and data loss recovery (eg, including error concealment).

[0203] Both the communication interface 318 and the communication interface 322 can be configured as a one-way communication interface (eg Fig.14 The invention also provides a method for transmitting the encoded image data 330 from the source device 310 to the destination device 320, as indicated by the arrow in the figure (indicated by the encoded image data 330 pointing from the source device 310 to the destination device 320), or a two-way communication interface, and can be used to send and receive messages, etc., for example, to establish a connection, confirm and / or resend lost or delayed data (including image data), and exchange any other information related to the communication link and / or data transmission (for example, encoded image data transmission).

[0204] The decoder 200 is used to receive encoded image data and provide decoded image data or decoded image (more details have been described, for example, based on Figure 2 description).

[0205] The post-processor 326 of the destination device 320 is used to post-process the decoded image data, such as the decoded image, to obtain post-processed image data 327, such as the post-processed image 327. For example, the post-processing performed by the post-processing unit 326 may include color format conversion (e.g., from YCbCr to RGB), color correction, cropping or resampling, or any other processing, such as processing for preparing the decoded image data for display by the display device 328, etc.

[0206] The display device 328 of the destination device 320 is used to receive the post-processed image data 327 to display the image to a user or viewer, etc. The display device 328 can be or include any type of display for representing the reconstructed image, such as an integrated or external display or monitor. For example, the display may include a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or any other type of display including a beam emitter, a hologram, or 3D / VR glasses.

[0207] although Fig.14 The source device 310 and the destination device 320 are described as separate devices, but the device embodiments may also include both devices or both functions, namely the source device 310 or the corresponding functions and the destination device 320 or the corresponding functions. In these embodiments, the source device 310 or the corresponding functions and the destination device 320 or the corresponding functions may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0208] Based on the description, it is obvious to a skilled person that Fig.14 The presence and (precise) division of different units or functionalities in the source device 310 and / or destination device 320 shown may vary depending on the actual device and application.

[0209] therefore, Fig.14 The source device 310 and the destination device 320 shown are merely exemplary embodiments of the present invention, and the present invention is not limited to the embodiment of the present invention. Fig.14 Those embodiments shown.

[0210] Source device 310 and destination device 320 may include any variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device, a broadcast receiver device, etc., and may not use or use any type of operating system.

[0211] Fig.15A schematic diagram of a video encoding device 1000 provided in an embodiment of the present invention. The video encoding device 1000 is suitable for implementing the disclosed embodiments described herein, for example, as an encoder or a decoder. The video encoding device 1000 includes: an input port 1010 and a receiver unit (Rx) 1020 for receiving data; a processor, a logic unit or a central processing unit (CPU) 1030 for processing data; a transmitter unit (Tx) 1040 and an output port 1050 for transmitting data; and a memory 1060 for storing data. The video encoding device 1000 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the input port 1010, a receiver unit 1020, a transmitter unit 1040, and an output port 1050 for inputting and outputting optical or electrical signals.

[0212] The present invention can be implemented in a device. Such a device can be a combination of software and hardware. For example, intra-frame prediction and deblocking filtering can be performed by a general-purpose processor, a digital signal processor (DSP), a field programmable gate array (FPGA), or other chips. However, the present invention is not limited to implementation on programmable hardware. It can be implemented on an application-specific integrated circuit (ASIC) or by a combination of the above hardware components.

[0213] The present invention also provides a computer readable medium storing instructions, which, when executed by a processing circuit, causes the processing circuit to perform any of the disclosed methods for dividing, encoding and decoding images. The computer readable medium may be any medium for storing programs, such as a DVD, CD, USB (flash) drive, hard disk, and server storage available on the Internet.

[0214] The encoder and / or decoder can be implemented in various devices such as TV, set-top box, PC, tablet, smart phone, etc. It can be software or application to implement the steps of the method.

[0215] In summary, the present invention provides an apparatus and method for dividing an image into coding units. An image is divided into coding tree units (CTUs), and the coding tree units are hierarchically divided. The hierarchical division includes multiple types of divisions, such as binary tree or quadtree division. For CTUs that are completely within the image and CTUs that are located at the boundary, a corresponding multi-type division depth can be selected. The present invention provides multiple types of divisions of the image boundary portion, which can enhance flexibility.

Claims

1. A device for dividing an image into coding units, characterized in that The apparatus comprises a processing circuit for: Divide the image into a plurality of coding tree units CTUs, the plurality of coding tree units including a non-border CTU and a border CTU, wherein the non-border CTU has a certain size in a horizontal direction and a vertical direction, a portion of the border CTU falls within an image defined by a horizontal image boundary or a vertical image boundary, and a size of a portion of the border CTU located within the image in a direction perpendicular to the horizontal image boundary or the vertical image boundary is smaller than the certain size; The non-boundary CTU and the boundary CTU are divided into coding units (CUs) according to the hierarchy, where The hierarchical division of the boundary CTU includes multi-type division, and the division depth of the multi-type division of the hierarchical division of the boundary CTU does not exceed the maximum boundary multi-type division depth, wherein the maximum boundary multi-type division depth is the sum of the adaptive boundary multi-type division depth and the predefined multi-type division depth, and the adaptive boundary multi-type division depth does not increase when the division direction is perpendicular to the horizontal image boundary or the vertical image boundary.

2. The device according to claim 1, characterized in that The partition depth of the multi-type partition of the hierarchical partition of the non-boundary CTU does not exceed the maximum non-boundary multi-type partition depth, and the predefined multi-type partition depth is equal to the maximum non-boundary multi-type partition depth.

3. The device according to claim 1 or 2, characterized in that: When the boundary CTU is located at the lower right corner of the image, horizontal binary or ternary tree (BTT) partitioning is performed, and then vertical BTT partitioning is performed.

4. The device according to claim 1, characterized in that The adaptive boundary multi-type division depth is a multi-type division depth with the horizontal image boundary direction or the vertical image boundary direction as the division direction.

5. The device according to claim 4, characterized in that The multi-type division depth with the horizontal image boundary direction or the vertical image boundary direction as the division direction is a binary tree division depth with the horizontal image boundary direction or the vertical image boundary direction as the division direction.

6. The device according to any one of claims 1 to 5, characterized in that The hierarchical division of the boundary CTU also includes quadtree division.

7. The device according to claim 2, characterized in that The maximum boundary multi-type partition depth is greater than or equal to the maximum non-boundary multi-type partition depth.

8. A device for encoding an image of a video sequence, characterized in that include: The device for dividing an image into coding units according to any one of claims 1 to 7; An image encoding unit, used for encoding the encoding unit; The bitstream forming unit is used to generate a bitstream, wherein the bitstream includes the encoded coding unit and division information indicating how to divide the coding tree unit.

9. The device for encoding an image of a video sequence according to claim 8, characterized in that The bitstream also includes a coding sequence parameter set, and the coding sequence parameter set includes the maximum boundary multi-type partitioning depth.

10. A device for decoding an image of a video sequence, characterized in that include: A bitstream parser for parsing a bitstream including the encoded coding units; The device for dividing an image into coding units according to any one of claims 1 to 7; An image decoding unit is used to decode the encoded coding unit based on the determined image division.

11. The device for decoding an image of a video sequence according to claim 10, characterized in that The bitstream further includes a coding sequence parameter set, the coding sequence parameter set including the maximum boundary multi-type partition depth, The image decoding unit is further used to obtain a second maximum multi-type division depth from the sequence parameter set.

12. A method for dividing an image into coding units, characterized in that The method comprises: Divide the image into a plurality of coding tree units CTUs, the plurality of coding tree units including a non-border CTU and a border CTU, wherein the non-border CTU has a certain size in a horizontal direction and a vertical direction, a portion of the border CTU falls within an image defined by a horizontal image boundary or a vertical image boundary, and a size of a portion of the border CTU located within the image in a direction perpendicular to the horizontal image boundary or the vertical image boundary is smaller than the certain size; The non-boundary CTU and the boundary CTU are divided into coding units (CUs) according to the hierarchy, where The hierarchical division of the boundary CTU includes multi-type division, and the division depth of the multi-type division of the hierarchical division of the boundary CTU does not exceed the maximum boundary multi-type division depth, wherein the maximum boundary multi-type division depth is the sum of the adaptive boundary multi-type division depth and the predefined multi-type division depth, and the adaptive boundary multi-type division depth does not increase when the division direction is perpendicular to the horizontal image boundary or the vertical image boundary.

13. The method according to claim 12, characterized in that The partition depth of the multi-type partition of the hierarchical partition of the non-boundary CTU does not exceed the maximum non-boundary multi-type partition depth, and the predefined multi-type partition depth is equal to the maximum non-boundary multi-type partition depth.

14. The method according to claim 12 or 13, characterized in that When the boundary CTU is located at the lower right corner of the image, horizontal binary tree or ternary tree (BTT) partitioning is performed, and then vertical BTT partitioning is performed.

15. The method according to claim 12, characterized in that The adaptive boundary multi-type division depth is a multi-type division depth with the horizontal image boundary direction or the vertical image boundary direction as the division direction.

16. The method according to claim 15, characterized in that The multi-type division depth with the horizontal image boundary direction or the vertical image boundary direction as the division direction is a binary tree division depth with the horizontal image boundary direction or the vertical image boundary direction as the division direction.

17. The method according to any one of claims 12 to 16, characterized in that The hierarchical division of the boundary CTU also includes quadtree division.

18. The method according to claim 13, characterized in that The maximum boundary multi-type partition depth is greater than or equal to the maximum non-boundary multi-type partition depth.

19. A method for encoding images of a video sequence, characterized in that The method comprises: The step of dividing the image into coding units according to any one of claims 12 to 18; An image encoding step of encoding the encoding unit; A bitstream forming step is to generate a bitstream including the encoded coding unit and division information indicating how to divide the coding tree unit.

20. A method for decoding an image of a video sequence, characterized in that The method comprises: The step of parsing a bitstream comprising the encoded coding units; The step of dividing the image into coding units according to any one of claims 12 to 18; An image decoding step of decoding the encoded coding unit based on the determined image division.

21. A computer-readable medium storing instructions, characterized in that: When the instructions are executed by a processing circuit, the processing circuit is caused to perform the method according to any one of claims 12 to 18.

Images