Block segmentation of signaling images and video

Abstract

A video system applying constraints on block segmentation is provided. The system receives, from a bitstream, a partition control parameter that specifies that a maximum block size for enabling ternary tree splitting is constrained to 64 or less (1410). The system receives data from a bitstream for a block of pixels to be decoded as a current block of a current picture of a video (1420). The system recursively splits the current block into one or more partitions (1430), wherein a ternary tree partition is not allowed for a portion of the current block unless the partition is less than or equal to the maximum block size (1435). The system reconstructs the one or more partitions of the current block (1440).

Description

[0001] Cross-references

[0002] This case is a divisional application of patent application number 202080070109.6, entitled "Block Segmentation of Transmitted Images and Videos".

[0003] This invention claims priority to U.S. Provisional Patent Applications filed on October 14, 2019, October 15, 2019, October 18, 2019, November 4, 2019, and January 2, 2020, with serial numbers 62 / 914,577, 62 / 915,049, 62 / 916,891, 62 / 930,084, and 62 / 956,359, respectively. The entire contents of these U.S. Provisional Patent Applications are incorporated herein by reference. Technical Field

[0004] This invention generally relates to the encoding and decoding of video and images. More specifically, this invention relates to a method for segmenting the structure of blocks (e.g., coding units) in transmitted encoded or decoded video or images. Background Technology

[0005] Unless otherwise indicated herein, the methods described in this section are not prior art to the claims listed below and are not recognized as prior art by virtue of being included in this section.

[0006] In High-Efficiency Video Coding (HEVC), the encoded image is segmented into non-overlapping square block regions represented by Codec Tree Units (CTUs). The encoded image can be represented by slices, each containing an integer number of CTUs. Individual CTUs within a slice are processed in raster scanning order. Bidirectional Prediction (B) slices can be decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict the sample values ​​of each block. Intra-frame (I) slices are decoded using only intra-frame prediction. Prediction (P) slices are decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict the sample values ​​of each block.

[0007] The CTU can be further subdivided into multiple non-overlapping coding units (CUs) to accommodate various local motion and texture characteristics. A coding tree built on a quadtree (QT) split structure is used to represent the multiple CU splits from the CTU. A CTU is defined for M×M pixel samples, where M is one of 64, 32, or 16. A CTU can include a single unit of the same M×M size, or be split into four smaller units of size M / 2×M / 2. The CTU associated with the root node of the code-decode tree, along with the four M / 2×M / 2 units, are child quadtree nodes split from the root node of the code-decode tree. Each of the four quadtree nodes can become a parent node for another split where the size is halved in each spatial dimension, resulting in four child nodes. Quadtree splits can be applied recursively until the leaf size is reached. The leaf nodes of the coding tree are associated with CUs. The minimum and maximum CU sizes can be specified in the Sequence Parameter Set (SPS).

[0008] One or more prediction units (PUs) can be specified for each CU. The prediction unit, along with the associated CU syntax, serves as the basic unit for transmitting prediction sub-information. The specified prediction process is applied to the values ​​of the relevant pixel samples within the prediction PU. A CU can be split into one, two, or four PUs, depending on the selected PU type.

[0009] The CU can further utilize a Residual Quadtree (RQT) structure to segment the data used to represent the correlated prediction residual signals. The leaf nodes of the RQT correspond to the generated Transform Units (TUs). A Transform Unit comprises a Transform Block (TB) of 8×8, 16×16, or 32×32 luminance samples, or four TBs of 4×4 luminance samples, and two corresponding TBs of chroma samples from an image in a 4:2:0 color format. Integer transforms are applied to the TBs, and the quantized coefficient values ​​are encoded and decoded in the bitstream. The minimum and maximum TB sizes are specified in the sequence parameter set.

[0010] In HEVC, the terms coded tree block, coded block, prediction block, and transform block are defined as 2-D sample arrays representing a color component from the associated CTU, CU, PU, ​​and TU, respectively. Therefore, the CTU contains one luma CTB, two chroma CTBs, and associated syntax elements in a color image that is not encoded using three separate color planes. The transmitted coded tree segmentation is typically applied to both luma and chroma blocks, but exceptions may be made when certain minimum size constraints are encountered. Summary of the Invention

[0011] The following overview is illustrative only and is not intended to be limiting in any way. That is, the following overview is meant to introduce the concepts, brilliance, benefits, and advantages of the novel and non-obvious techniques described herein. Options and not all embodiments are further described in the detailed description. Therefore, the following overview is not intended to identify the essential features of the claimed subject matter, nor is it intended to determine the scope of the claimed subject matter.

[0012] This invention provides some embodiments of a video system that applies constraints to block segmentation. The system receives a segmentation control parameter from a bitstream specifying that a maximum block size for enabling ternary tree (TT) splitting is constrained to 64 or less. The system receives data from a bitstream of a pixel block of a current image to be decoded into a video. The system recursively splits the current block into one or more segments, wherein TT splitting is not allowed for a segment of the current block unless the segment is less than or equal to the maximum block size. The system reconstructs the one or more segments of the current block.

[0013] In some embodiments, the segmentation control parameter is signaled in a strip header that includes the current block. The segmentation control parameter is signaled in an image header of the current image. The segmentation control parameter may be signaled in a sequence parameter set (SPS) of a video image sequence that includes the current image. In some embodiments, the SPS or image header sets a preset value for the segmentation control parameter, which may be overridden by the strip header that includes the current block.

[0014] In some embodiments, the segmentation control parameter specifies the maximum block size by referencing a minimum size of a quadtree-splitting leaf block, for example by using the difference between (i) the base-2 logarithm of the maximum block size of a block that can be further segmented by a ternary tree and (ii) the base-2 logarithm of the minimum size of the quadtree-splitting leaf block. The minimum size of the quadtree-splitting leaf block is constrained to be a smaller value between 64 and a coding tree block size. The maximum block size used to enable TT splitting is further constrained to be less than a coding tree block size.

[0015] The minimum size of the quadtree split leaf block is specified by reference to a minimum coding block size, and the minimum size of the quadtree split leaf block is specified by the difference between (i) the base-2 logarithm of the minimum size of the quadtree split leaf block and (ii) the base-2 logarithm of the minimum coding block size.

[0016] In some embodiments, the maximum block size is determined based on the size of the Virtual Pipeline Data Unit (VPDU) and is defined based on ensuring that each VPDU can be fully processed by the same pipeline stage. Attached Figure Description

[0017] The accompanying drawings are included to provide a further understanding of the invention and are included and constitute a part of this invention. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. It will be understood that the invention does not need to be drawn to scale, as some parts are shown out of scale to clearly illustrate the concepts of the invention.

[0018] Figure 1 The diagram illustrates a coding tree structure used to represent dividing a CTU into multiple CUs. Solid lines indicate CU boundaries and dashed lines indicate TU boundaries.

[0019] Figure 2 The diagram shows eight partitioning types defined in HEVC for splitting a CU into multiple PUs.

[0020] Figure 3 Various splitting types that divide a CU into two segments are shown, including different symmetric and asymmetric splitting types that can divide the CU vertically or horizontally.

[0021] Figure 4 An example of block partitioning and its corresponding binary tree is shown.

[0022] Figure 5 An example of block partitioning and its corresponding QTBT is shown.

[0023] Figure 6 This demonstrates how to split a CU into multiple smaller CUs using one of five splitting types.

[0024] Figure 7 The diagram illustrates the compatible and incompatible segmentation structures of pixel blocks when VPDU-based constraints are applied.

[0025] Figure 8 The constraint parameters used to control the segmentation structure of pixel blocks are conceptually illustrated.

[0026] Figure 9 An exemplary video encoder is shown that transmits constraints regarding block segmentation.

[0027] Figure 10 A portion of the video encoder implementing block segmentation constraints is shown.

[0028] Figure 11 The process for constraining the transmission block is conceptually illustrated.

[0029] Figure 12 An exemplary video decoder with constraints applied to block segmentation is shown.

[0030] Figure 13 A portion of the video decoder implementing block segmentation constraints is shown.

[0031] Figure 14 The process of applying block partitioning constraints is conceptually illustrated.

[0032] Figure 15 An electronic system implementing some embodiments of the present invention is conceptually illustrated. Detailed Implementation

[0033] In the following detailed description, numerous specific details are given by way of example to provide a thorough understanding of the invention. Any variations, derivatives, and / or extensions based on the teachings described herein are within the scope of protection of this invention. In some instances, in order not to unnecessarily obscure aspects of the teachings of this invention, well-known methods, processes, elements, and / or circuits of one or more exemplary embodiments disclosed herein are not described in detail at a relatively high level.

[0034] I. Block partitioning structure

[0035] Figure 1 The diagram illustrates a coding tree structure used to represent dividing a CTU into multiple CUs. Solid lines indicate CU boundaries and dashed lines indicate TU boundaries. Figure 2 This shows eight types of partitioning defined in HEVC that split a CU into multiple PUs.

[0036] A. Binary Tree (BT) Block Segmentation

[0037] In some embodiments, a binary tree block splitting structure is used to split blocks. In a binary tree splitting structure, a block can be recursively split into two smaller blocks or into smaller segments.

[0038] Figure 3 Various splitting types for dividing a CU into two segments are illustrated, including different symmetric and asymmetric splitting types that can split the CU vertically or horizontally. For some embodiments, the most efficient and simplest splitting types are symmetric horizontal and vertical splitting, and only these two splitting types are used in binary tree splitting structures.

[0039] For a given block of size M×N, the syntax element `bt_split_flag` can be signaled to indicate whether the block should be split into two smaller blocks. If the flag is true, another syntax element `bt_split_mode` is signaled to indicate which split type (i.e., horizontal or vertical) was used. If horizontal splitting was used, the block is split into two smaller blocks of size M×N / 2. Otherwise, if vertical splitting was used, the block is split into two smaller blocks of size M / 2×N. The binary tree splitting process can be applied recursively until the current block size (width or height) reaches the minimum block size (width or height), which can be defined in a higher-level syntax set (such as SPS). Because binary trees have two splitting types (i.e., horizontal and vertical), the minimum block width and height can be indicated. A non-horizontal split is implied when the split would result in a block height smaller than the indicated minimum. A non-vertical split is implied when the split would result in a width smaller than the indicated minimum.

[0040] Figure 4 An example of a block split (left) and its corresponding binary tree (right) is shown. For each split (i.e., non-leaf) node in the binary tree, a flag is sent to indicate which split type (horizontal or vertical) was used, with "0" indicating a horizontal split and "1" indicating a vertical split.

[0041] B. Quadtree + Binary Tree (QTBT) Block Partitioning

[0042] In some embodiments, a quadtree plus binary tree (QTBT) structure is used to partition a CU into smaller CUs. Each CTU may contain one CU or be split into multiple smaller CUs using a quadtree splitting structure first. Quadtree splitting can be recursively applied to CUs until the current CU size reaches the minimum quadtree leaf node size. If the current quadtree block is not larger than the maximum binary tree root node size, it can be further split using a binary tree splitting structure. Binary tree splitting can be recursively applied until the current CU size (width or height) reaches the minimum binary tree leaf node size (width or height) or the current binary tree depth reaches the maximum binary tree depth. There are two splitting types: symmetric horizontal splitting and symmetric vertical splitting. Binary tree leaf nodes are associated with CUs. For each split (i.e., non-leaf) node in the binary tree, a flag is sent to indicate which splitting type (horizontal or vertical) was used. "0" indicates a horizontal splitting and "1" indicates a vertical splitting. The minimum size of a quadtree leaf node, the maximum size of a binary tree root node, the minimum width and height of a binary tree leaf node, and the maximum depth of a binary tree can be indicated in the higher-level syntax set. Figure 5 An example of block splitting (left) and its corresponding QTBT (right) is shown. Solid lines indicate quadtree splitting and dashed lines indicate binary tree splitting.

[0043] C. Multi-Type Tree (MTT) Block Segmentation

[0044] In the Multi-Type Tree (MTT) structure, horizontal and vertical center-side ternary trees (TT) are further added to the QTBT structure for splitting CUs. Figure 6 This illustrates splitting a CU into multiple smaller CUs using one of five splitting types. Splitting types (d) and (e) are called ternary trees or ternary tree splitting types and can be used to split a block into three smaller blocks, all with a reduced size in one spatial dimension while maintaining the same size in other spatial dimensions. The binary tree depth btDepthC of the current CU is defined in the MTT structure as follows:

[0045] btDepthC=log2((widthR*heightR) / (widthC*heightC))

[0046] Where widthC and heightC represent the width and height of the current CU, respectively, widthR and heightR represent the width and height of the corresponding MTT root node, respectively, and the function log2(x) is the logarithm of x with base 2.

[0047] In some embodiments, each coding tree unit (CTU) can be partitioned into one or more smaller CUs by a quadtree with embedded multi-type trees (using binary and ternary splits). The syntax element `split_cu_flag` is first signaled to indicate whether the current coding unit is to be further split into more than one coding unit. When `split_cu_flag` equals 0, the current CU is a leaf node and is not further split. Otherwise, the current CU is further split, and the syntax element `split_qt_flag` is further signaled to indicate whether the current coding unit is split by a quadtree split. When `split_qta_flag` equals 1, the current CU is further split by a quadtree split into four CUs, each with half the horizontal and vertical dimensions. Otherwise, the syntax elements `mtt_split_cu_vertical_flag` and `mtt_split_cu_binary_flag` are signaled to indicate the direction (vertical or horizontal) and type (binary or ternary) of the selected MTT split.

[0048] II. Constraints on block size used for partitioning

[0049] Virtual Pipeline Data Units (VPDUs) are defined as non-overlapping M×M-luminance (L) / N×N-chrominance (C) units in an image. In a hardware decoder, consecutive VPDUs are processed simultaneously by multiple pipeline stages, and different stages process different VPDUs simultaneously. The size of a VPDU is roughly proportional to the buffer size of most pipeline stages. In HEVC hardware decoders, the VPDU size is typically set to match the maximum transform block (TB) size, referred to as the maximum allowed block size, or MaxTbSizeY.

[0050] In some embodiments, to reduce the VPDU size in the encoded video, CU segmentation is constrained by the following: (1) For each VPDU containing one or more CUs, the CU is fully contained within the VPDU. (2) For each CU containing one or more VPDUs, the VPDU is fully contained within the CU. (3) The processing order of CUs cannot leave a VPDU unaccessed later. This ensures that each VPDU can be fully processed by the same pipeline stage.

[0051] Figure 7 The diagram illustrates the compatible and incompatible segmentation structures of pixel blocks when VPDU-based constraints are applied. The diagram shows block 700 (e.g., CTU) that can be further segmented for encoding and decoding. Block 700 is 128×128 luminance samples in size. Block 700 is processed into four VPDUs (701-704) (separated by dashed lines), each VPDU corresponding to a 64×64 luminance sample area.

[0052] The diagram illustrates various exemplary schemes (a) and (i) for dividing block 700 into multiple

[0053] CU. Some partitioning schemes are compatible with VPDU while others are not. Specifically, because each

[0054] A VPDU contains one or more CUs, and the CUs are fully contained within the VPDU. For each CU containing one or more VPDUs, the VPDU is fully contained within the CU. Partition schemes (a), (b), (c), and (d) result in CU-VPDU compatibility. This ensures that it is not necessary to...

[0055] The VPDU is partially processed as part of a CU and then revisited later as part of another CU.

[0056] The partitioning schemes (e), (f), (g), (h), (i), (j), and (k) are given. Segment (i) generates multiple CUs compatible with the VPDU because each scheme requires at least one VPDU to be partially processed as part of one CU and later revisited as part of another CU. In other words, there exists at least one VPDU spanning multiple pipeline stages.

[0057] In some embodiments, when the TT splitting allowed for the current block is determined based on VPDU constraints, such as the maximum allowed block size due to pipeline limitations, the permission for TT splitting can be determined based on subsequent parameters:

[0058] -Ternary Split Mode ttSplit

[0059] - width of the coded block in the luminance sample, cbWidth

[0060] - height of the coded block in the luminance sample (cbHeigth)

[0061] - The position (x0, y0) of the top-left luminance sample of the considered coded block relative to the top-left luminance sample of the image.

[0062] -Multi-type tree depth mttDepth

[0063] -MaxMttDepth of the multi-type tree with offset

[0064] -Maximum ternary tree size maxTtSize

[0065] - The variable treeType specifies whether a single tree (SINGLE_TREE) or a dual tree is used to split the encoded tree nodes, and when a dual tree is used, whether the luma (DUAL_TREE_LUMA) or chroma component (DUAL_TREE_CHROMA) is currently being processed.

[0066] - The variable modeType specifies whether intra-frame (MODE_INTRA), IBC (MODE_IBC), and inter-frame coding modes can be used (MODE_TYPE_ALL), or whether only intra-frame and IBC coding modes can be used (MODE_TYPE_INTRA), or whether only inter-frame coding modes (MODE_TYPE_INTER) can be used for coding units within coding tree nodes.

[0067] These parameters determine the variable "allowTtSplit". The variable cbSize is derived from ttSplit based on the following:

[0068]

[0069]

[0070] The variable allowTtSplit is derived from the following: (VPDU size indicated by MaxTbSizeY)

[0071] - If one or more subsequent conditions are true, allowTtSplit is set to FALSE:

[0072] -cbSize is less than or equal to 2 * MinTtSizeY

[0073] -cbWidth is greater than Min(MaxTbSizY,maxTtSize)

[0074] -cbHeight is greater than Min(MaxTbSizY,maxTtSize)

[0075] -mttDepth is greater than or equal to maxMttDepth

[0076] -x0+cbWidth is greater than pic_width_in_luma_samples

[0077] -y0+cbHeight is greater than pic_height_in_luma_samples

[0078] -treeType equals DUAL_TREE_CHROMA and (cbWidth / SubWidthC)*(cbHeight / SubHeightC) is less than or equal to 32.

[0079] -treeType equals DUAL_TREE_CHROMA, (cbWidth / SubWidthC) equals 8, and TTSplit equals SPLIT_TT_VER.

[0080] -treeType equals DUAL_TREE_CHROMA and modeType equals

[0081] MODE_TYPE_INTRA

[0082] -cbWidth*cbHeight equals 64 and modeType equals...

[0083] MODE_TYPE_INTER

[0084] - Otherwise, allow TTSplit to be set to TRUE.

[0085] In some embodiments, high-level syntax parameters relating to the CU segmentation structure are specified based on pipeline processing constraints, i.e., VPDU-related constraints (e.g., maximum allowed block size) are implemented when the CU is segmented. In some embodiments, the video codec system may include one or more syntax elements for deriving a value for the maximum allowed block size to enable TT splitting in the relevant data structure.

[0086] In some embodiments, the maximum allowed block size for enabling TT splits is constrained by bitstream consistency requirements. Therefore, to determine whether TT splits are permissible for the current CU, the video codec system may only test whether the width or height of the current CU is greater than the maximum allowed block size for enabling TT splits, without further testing the VPDU size for each current CU. For example, when the VPDU size is equal to 64, the video codec system may constrain the maximum allowed block size for enabling TT splits transmitted in the bitstream to be less than or equal to 64.

[0087] In some embodiments, the parameter maxTtSize is constrained to be less than or equal to 64, wherein the variable maxTtSize is derived from one or more encoded syntax element values ​​indicating the maximum allowed block size for enabling TT splitting. In some embodiments, the variable maxTtSize is derived from the encoded / decoded high-level syntax element log2_diff_max_tt_min_t, according to:

[0088] maxTtSize=1<<(log2_diff_max_tt_min_tt+log2MinMaxTT)

[0089] The maximum permissible value of log2_diff_tt_min_tt is equal to the constraint Min(6,CtbLog2SizeY) – log2MinMaxTT. The variable log2MinMaxTT indicates the logarithm of the minimum permissible value of the maximum TT size with a base of 2, and CtbLog2SizeY indicates the logarithm of the specified CTB size with a base of 2.

[0090] The process that determines whether a TT split is allowed for the current CU can be further simplified. Therefore, the variable allowTtSplict is derived as follows:

[0091] - If one or more of the following conditions are true, allowTtSplit is set to equal to

[0092] FALSE:

[0093] -cbsize is less than or equal to 2 * MinTtSizeY

[0094] -cbheight is greater than maxTtSize

[0095] -cbheight is greater than maxTtSize

[0096] -mttDepth is greater than or equal to maxMttDepth

[0097] -x0+cbWidth is greater than pic_width_in_luma_samples

[0098] -y0+cbHeight is greater than pic_height_in_luma_samples

[0099] -treeType equals DUAL_TREE_CHROMA and (cbWidth / SubWidthC)*(cbHeight / SubHeightC) is less than or equal to 32.

[0100] -treeType equals DUAL_TREE_CHROMA, (cbwidth / subwidthC) equals 8, and ttSplit equals SPLIT_TT_VER.

[0101] -treeType equals DUAL_TREE_CHROMA and modeType equals

[0102] MODE_TYPE_INTRA

[0103] -cbWidth*cbHeight equals 64 and modetype equals...

[0104] MODE_TYPE_INTER

[0105] Otherwise, allowTtSplit is set to TRUE.

[0106] To satisfy VDPU constraints, in some embodiments, the variables minQtSizeY and minQtSizeC are derived from one or more encoded syntax element values ​​of QT leaf nodes, which respectively indicate the minimum allowed block size for luma and chroma. When intra-tree mode is enabled, the values ​​of minQtSizeY and minQtSizeC are constrained to be less than or equal to 64 in the intra-slice. In some embodiments, when intra-tree mode is enabled, the values ​​of minQtSizeY and minQtSizeC are constrained to be less than or equal to 64 in the intra-slice.

[0107] Min(64,CtbSizeY), where the variables minQtSizeY and minQtSizeC are derived from one or more encoded syntax element values ​​of QT leaf nodes, respectively, which indicate the minimum allowed block size of luminance samples and chrominance samples, and the variable CtbSizeY indicates the CTB size in luminance samples of an encoded video sequence.

[0108] In some embodiments, when intra-frame dual-tree mode is enabled, a current CU with a size greater than 64 can be further split into multiple smaller CUs, each of which is contained within a VPDU region split by a quadtree.

[0109] In some embodiments, the values ​​of the variables minQtSizeY and minQtSizeC are constrained to be less than or equal to Min(64, CtbSizeY) for all stripe types. Therefore, when a current CU portion of size greater than 64 intersects (or crosses) the right or bottom image boundary, the current CU can be further split into multiple smaller CUs by quadtree splitting.

[0110] III. Constraints on Segmentation Depth

[0111] In some embodiments, constraint parameters related to the maximum MTT depth are specified. These constraint parameters are used to control the CU segmentation structure based on a quadtree with an embedded multi-type tree using binary and / or ternary splitting. According to one aspect of the invention, the maximum MTT depth is constrained to be greater than 0 when the minimum QT size is greater than the minimum code block size. In some embodiments, the maximum MTT depth is constrained to be greater than 0 when the minimum QT size is greater than the minimum code block size.

[0112] In some embodiments, the SPS syntax element log2_ctu_size_minus5 plus 5 specifies the luminance coding tree block size (logarithm of base 2) for each CTU. The syntax element log2_ctu_size_minus5 can be used to derive ctblog2SizeY and ctbSizeY according to the following:

[0113] CtbLog2SizeY=log2_ctu_size_minus5+5

[0114] CtbSizeY = 1 <CtbLog2SizeY

[0115] In some embodiments, the SPS syntax element log2_min_luma_coding_block_size_minus plus 2 specifies the logarithm of the minimum luma coding block size, with the base being 2. The syntax element log2_min_luna_coding_block_size_minus2 can be used to derive the minimum size of the CB in a luma sample, denoted as MinCbLog2SizeY or...

[0116] MinCbSizeY. In some embodiments, MinCbSizeY is constrained to be less than or equal to...

[0117] Min(64, CtbSizeY). MinCbLog2SizeY and MinCbSizeY can be derived as follows:

[0118] MinCbLog2SizeY=log2_min_luma_coding_block_size_minus2+2

[0119] MinCbSizeY=1< <MinCbLog2SizeY

[0120] In some embodiments, the slice header (SH) syntax element slice_log2_diff_qt_min_cb_luma specifies the difference between the base-2 logarithm of the minimum size of the luminance samples generated from the QT split of the CTU and the base-2 logarithm of the minimum coded block size of the luminance samples used for the luminance CU in the current slice.

[0121] In some embodiments, the slice header (SH) syntax element slice_max_mtt_hierarchy_depth_luma specifies the maximum hierarchical depth of the coding units generated from the multi-type tree splitting of the quadtree leaves of the current slice. When slice_max_mtt_hierarchy_depth_luma is non-zero, the syntax elements slice_log2_diff_max_bt_min_qt_luma and slice_log2_diff_max_tt_min_qt_luma are encoded and decoded in the current slice header. The slice header syntax element slice_log2_diff_max_bt_min_qt_luma specifies the difference between the maximum size (width or height) (or logarithm base 2) of the luma coding block that can be split using binary splitting and the minimum size (width or height in luma samples) (logarithm base 2) of the luma leaf block generated from the quadtree splitting of the CTU in the current slice. The SH syntax element slice_log2_diff_max_tt_min_qt_luma specifies the difference between the maximum size (width or height in the luminance sample) of the luminance coding block that can be split using ternary splitting (logarithm of 2) and the minimum size (width or height in the luminance sample) of the luminance leaf block generated from the quadtree splitting of the CTU in the current slice (logarithm of 2).

[0122] The variable MinQtLog2SizeY or MinQtSizeY indicates the minimum size of the luminance sample of the luminance leaf block generated from the QT split of CTU.

[0123] The values ​​of slice_log2_diff_max_bt_min_qt_luma and slice_log2_diff_max_tt_min_qt_luma are constrained to the range from 0 to CtbLog2SizeY - MinQtLog2SizeY, inclusive. MinQtSizeY is constrained to the range (MinCbSizeY, Min(64, CtbSizeY)), inclusive of MinCbSizeY and Min(64, CtbSizeY).

[0124] The syntax element slice_log2_diff_min_qt_min_cb_luma can be used to derive MinQtLog2SizeY and MinQtSizeY as follows:

[0125] MinQtLog2SizeY=MinCbLog2SizeY+

[0126] slice_log2_diff_min_qt_min_cb_luma

[0127] MinQtSizeY = 1 <MinQtLog2SizeY

[0128] Because the value of MinQtSizeY is constrained to the range (MinCbSizeY, Min(64, CtbSizeY)), the range of slice_log2_diff_min_qt_min_cb_luma is 0 to...

[0129] Min(6, CtbLog2SizeY) – MinCbLog2SizeY, including 0 and

[0130] Min(6,CtbLog2SizeY)–MinCbLog2SizeY.

[0131] The variable MaxBtSizeY indicates the maximum size of the luminance samples in the luminance coded block that can be partitioned using binary splitting. The variable MaxBtSizeY is derived as follows:

[0132] MaxBtSizeY =

[0133] 1<<(MinQtLog2SizeY+slice_log2_diff_max_bt_min_qt_luma)

[0134] In some embodiments, the maximum BT size is constrained to be greater than or equal to the minimum QT size. Thus, when the current coding block (as a result of segmentation) reaches the minimum QT size, the coding block can be further recursively segmented using binary splitting until the current coding block equals the minimum coding block size. In some embodiments, the maximum TT size is constrained to be greater than or equal to the minimum QT size. Thus, when the current coding block (as a result of segmentation) reaches the minimum QT size, the coding block can be further iteratively segmented using ternary splitting until the current coding block equals the minimum coding block size.

[0135] The variable MaxTtSizeY indicates the maximum size of the luminance samples in a luminance-coded block that can be split using ternary splitting. The variable MaxTtSizeY can be derived from the syntax element slice_log2_diff_max_tt_min_qt_luma, as follows:

[0136] MaxTtSizeY =

[0137] 1<<(MinQtLog2SizeY+slice_log2_diff_max_tt_min_qt_luma)

[0138] Because MaxTtSizeY is in the range from MinQtSize to Min(64,CtbSizeY), slice_log2_diff_max_tt_min_qt_luma is in the range from 0 to Min(6,CtbLog2SizeY)–MinQtLog2SizeY.

[0139] In some embodiments, the slice header syntax element slice_log2_diff_min_qt_min_cb_chroma,

[0140] `slice_log2_diff_max_bt_min_qt_chroma` and `slice_log2_diff_max_tt_min_qt_chroma` specify the minimum QT size, maximum BT size, and maximum TT size (in luminance samples) of the chroma-coded block, respectively. In some embodiments, the preset values ​​of these three SH syntax elements are specified by the corresponding syntax elements in SPS. For example, in some embodiments, there may be corresponding SPS parameters `sps_log2_diff_min_qt_min_cb_chroma`,

[0141] sps_log2_diff_max_bt_min_qt_chroma and sps_log2_diff_max_tt_min_qt_chroma specify the minimum QT size, maximum BT size, and maximum TT size (in luminance samples) of the chroma coding block at the SPS level, respectively.

[0142] In some embodiments, the derived maximum BT size and maximum TT size can be greater than or equal to the minimum QT size. Therefore, when the current coding block reaches the minimum QT size and binary or ternary splitting is enabled, the current coding block can be recursively split using binary or ternary splitting until the current coding block equals the minimum coding block size. However, when slice_max_mtt_hierarchy_depth_luma equals 0, both binary and ternary splitting are disabled. In some embodiments, if the minimum QT size is greater than the minimum coding block size, no coding block can be split into the minimum coding block, as this can cause problems with segmenting CUs along the image boundary, where each CU exceeding the boundary needs to be further iteratively segmented until the entire CU is within the image boundary. In some embodiments, when slice_log2_diff_min_qt_min_cb_luma is non-zero, syntax slice_max_mtt_hierarchy_depth_luma is constrained to be greater than 0.

[0143] Figure 8 The constraint parameters for controlling the segmentation structure of pixel block 800 are conceptually illustrated. Pixel block 800 can be segmented using both QT and MTT CTUs. The diagram together shows CTU 800 and tree 810, which illustrates a partial segmentation structure of CTU 800.

[0144] The CTU 800 measures 128 x 128. The CTU 800 is constrained to not...

[0145] MTT (e.g., TT) splits are used for splits larger than 64. In other words, MTT splits are not allowed for CTU 800, but QT splits are allowed for CTU 800s of 64×64 or smaller. To ensure that each VPDU can be processed completely by the same pipeline stage, this maximum allowable block size for MTT can be determined based on the size of the VPDU.

[0146] The maximum permissible size for TT (e.g., variable MaxBtSizeY or MaxTtSizeY) can also be derived by referencing the minimum size of the QT leaf (variable minQTSizeY). For example, the maximum permissible size for MTT can be expressed as the logarithm of MaxTtSizeY with base 2.

[0147] The minimum size of the QT leaf can be signaled by referring to the minimum size of the CB (variable MinCbSizeY). For example, the minimum size of the QT leaf can be increased to the difference between the base 2 logarithm of MinQtSizeY and the base 2 logarithm of MinCbSizeY, as shown in slice_log2_diff_min_qt_min_cb_luma or the corresponding signal in the image header or SPS.

[0148] IV. Transmitting Block Segmentation Structure

[0149] In some embodiments, information regarding the maximum MTT depth is signaled based on the difference between the base-2 logarithm of the minimum QT size and the base-2 logarithm of the minimum coding block size. In some embodiments, MaxMttDepthY indicates that the maximum hierarchical depth of the coding unit generated from the multi-type tree splitting of the quadtree leaves is in the range (0, 2*(CtbLog2SizeY - MinCbLog2SizeY)), inclusive of 0 and 2*(CtbLog2SizeY - MinCbLog2SizeY). Variables

[0150] MaxMttDepthY can be derived as follows:

[0151] MaxMttDepthY=slice_max_mtt_hierarchy_depth_luma+

[0152] ((1+slice_log2_diff_min_qt_min_cb_luma)>>1).

[0153] In some embodiments, constraint parameters for controlling the CU segmentation structure are specified. In some embodiments, when the current coding block reaches (or has) the minimum QT size, it is allowed to be further recursively segmented until the minimum coding block size, as long as at least one MTT splitting type (BT and TT) is still enabled. The video encoder may specify a maximum BT size or a maximum TT size that is less than the minimum QT size, where at least one maximum BT size and a maximum TT size are constrained to be greater than or equal to the minimum QT size when the current MTT depth is non-zero. Therefore, the specified MTT structure can disable one of the MTT splitting types for coding blocks with size constraints greater than the minimum QT size. Specifically, by specifying a corresponding size constraint equal to the minimum coding block size, the video encoder can disable only one MTT type.

[0154] In some embodiments, the first syntax element for the maximum size of the first MTT type specifies the difference between the maximum size (logarithm to the base 2) in the luminance samples of the first MTT type and the smallest decoder block size (logarithm to the base 2) in the luminance samples of the luminance CU in the current strip. When the maximum size of the first MTT type is less than the minimum QT size and the minimum...

[0155] When the QT size is greater than the minimum coding block size, the second syntax element for the maximum size of the second MTT type specifies the difference between the maximum size (logarithm to the base 2) of the luminance sample of the first MTT type and the minimum size (width or height) (logarithm to the base 2) of the luminance sample of the luminance leaf block generated from the quadtree split of the CTU of the current slice. Otherwise, the second syntax element for the maximum size of the second MTT type specifies the difference between the maximum size (logarithm to the base 2) of the luminance sample of the second MTT type and the minimum coding block size (logarithm to the base 2) of the luminance sample of the luminance CU in the current slice.

[0156] Table 1 provides an exemplary syntax table for specifying stripe headers (SH) for block segmentation structures. The syntax allows the specified maximum size to be less than the minimum QT size of an MTT type.

[0157] Table 1

[0158]

[0159]

[0160] The variables MaxBtSizeY and MaxTtSizeY are derived as follows:

[0161] MaxBtSizeY=1<<(MinCbLog2SizeY+slice_log2_max_bt_luma)

[0162] MaxTtSizeY=1<< (((MaxBtSizeY<MinQtSizeY&&MinQtLog2SizeY> MinCbLog2SizeY)? MinQtLog2SizeY:MinCbLog2SizeY)+ slice_log2_max_tt_luma).

[0163] In some other embodiments, the variables MaxBtSizeY and MaxTtSizeY are derived as follows:

[0164] MaxBtSizeY=1<<(MinCbLog2SizeY+slice_log2_max_bt_luma)

[0165] MaxTtSizeY=1<<(((MaxBtSizeY <MinQtSizeY)? MinQtLog2SizeY:MinCbLog2SizeY)+slice_log2_max_tt_luma).

[0166] In some other embodiments, the first syntax element for the maximum size of the first MTT type specifies the difference between the base-2 logarithm of the maximum size of the luminance samples of the first MTT type and the base-2 logarithm of the minimum coded block size of the luminance samples of the luminance CU in the current stripe. The second syntax element for the maximum size of the second MTT type specifies the base-2 logarithm of the maximum size of the luminance samples of the second MTT type and the base-2 logarithm of the minimum coded block size of the luminance samples of the luminance CU in the current stripe. The variables MaxBtSizeY and...

[0167] MaxTtSizeY is derived as follows:

[0168] MaxBtSizeY=1<<(MinCbLog2SizeY+slice_log2_max_bt_luma)

[0169] MaxTtSizeY=1<<(MinCbLog2SizeY+slice_log2_max_tt_luma)

[0170] In some embodiments, at least one value of the MaxBtSizeY variable MaxBtSizeY is constrained to be greater than or equal to MinQtSizeY.

[0171] It is worth noting that when the dual-tree segmentation mode is enabled (qtbtt_dual_tree_intra_flag equals 1), the proposed method and related embodiments can be further applied to specify other relevant syntax elements to indicate the maximum BT size and maximum TT size in the stripe header of the intra-strip for splitting chroma codec blocks. The proposed method and related embodiments can also be further applied to specify other relevant syntax elements to indicate the default maximum BT size and default maximum TT size in SPS.

[0172] In some embodiments, when the current coding block reaches the minimum QT size, it is allowed to be further recursively split until the minimum coding block size is reached (e.g., enabling binary or ternary splitting for coding block sizes less than or equal to the minimum QT size without any constraint on the maximum size of the binary or ternary split). In some embodiments, under the constraint that the maximum BT size is greater than or equal to the minimum QT size, the video encoder can transmit a maximum TT size less than the minimum QT size. Therefore, the specified MTT structure can disable ternary splitting for coding blocks whose size is greater than the specified size constraint, i.e., less than the minimum QT size. Specifically, by specifying the corresponding maximum size constraint as equal to the minimum coding block size, the video encoder can effectively disable ternary splitting.

[0173] In some embodiments, the syntax element for the maximum size of the sending binary split type specifies the difference between the maximum size (logarithm base 2) of the luminance coding block (luminance sample) that can be split using binary splitting and the minimum coding block size (width or height of the luminance sample) (logarithm base 2) of the luminance leaf block (which is generated from the quadtree splitting of the CTU of the current slice). Another syntax element for the maximum size of the sending ternary type specifies the difference between the maximum size (width or height of the luminance sample) (logarithm base 2) of the luminance coding block that can be split using ternary splitting and the minimum block size (logarithm base 2) of the luminance sample of the luminance CU in the current slice. Table 2 provides an exemplary syntax table for slice headers that can support ternary split types where the specified maximum size is smaller than the minimum QT size.

[0174] Table 2

[0175]

[0176]

[0177] The variable MaxBtSizeY indicates the maximum size of the luminance CB that can be split by BT. The variable MaxTtSizeY indicates the maximum size of the luminance CB that can be split by TT. MaxBtSizeY and MaxTtSizeY are in the range from MinQtSize to Min(64,CtbSizeY), including...

[0178] MinQtSize and Min(64,CtbSizeY). MaxBtSizeY and MaxTtSizeY are derived as follows:

[0179] MaxBtSizeY =

[0180] 1<<(MinQtLog2SizeY+slice_log2_diff_max_bt_min_qt_luma)

[0181] MaxTtSizeY =

[0182] 1<<(MinCbLog2SizeY+slice_log2_diff_min_tt_min_cb_luma).

[0183] The variable MaxBtSizeC indicates the maximum size of the chroma CB that can be split by BT. The variable MaxTtSizeY indicates the maximum size of the chroma CB that can be split by TT. The variables MaxBtSizeC and MaxTtSizeC are derived as follows:

[0184] MaxBtSizeC =

[0185] 1<<(MinQtLog2SizeC+slice_log2_diff_max_bt_min_qt_chroma)

[0186] MaxTtSizeC =

[0187] 1<<(MinCbLog2SizeY+slice_log2_diff_min_tt_min_cb_chroma)

[0188] It is worth noting that when the dual-tree segmentation mode is enabled (qtbtt_dual_tree_intra_flag equals 1), the methods and related embodiments described in Sections II to IV can be further applied to other relevant syntax elements for sending information in the slice header of the intra-frame slice to derive the maximum MTT depth, in order to split the chroma-coded blocks. The proposed scheme can be further applied to sending relevant constraint parameters in the image header. The proposed scheme and related embodiments can also be applied to other relevant syntax elements for sending information in the SPS to derive the preset maximum MTT depth. For example, there are image header parameters and SPS parameters corresponding to the slice header parameters, such as slice_log2_diff_max_tt_min_qt_luma and slice_log2_diff_min_qt_min_cb_luma, which set preset values ​​for these parameters at the sequence or image level.

[0189] Any of the aforementioned proposed schemes can be implemented in an encoder and / or decoder. For example, any proposed method can be implemented in the entropy coding (or high-level syntax coding) module of the encoder, and / or the entropy decoding (or high-level syntax decoding) module of the decoder. Alternatively, any of the aforementioned proposed methods can be implemented as circuitry integrated into the entropy coding (or high-level syntax coding) module of the encoder and / or the entropy decoding (or high-level syntax decoding) module of the decoder. Any of the aforementioned proposed methods can also be implemented in an image encoder and / or decoder, wherein the generated bitstream corresponds to an encoded / decoded frame using only intra-frame image prediction.

[0190] V. Exemplary Video Encoder

[0191] Figure 9 An exemplary video encoder 900 is shown, illustrating constraints regarding block segmentation. As shown, the video encoder 900 receives an input video signal from a video source 905 and encodes the signal into a bitstream 995. The video encoder 900 has various elements or modules for encoding the signal from the video source 905, including at least some elements selected from a transform module 910, a quantization module 911, an inverse quantization module 914, an inverse transform module 915, an intra-frame image estimation module 920, an intra-frame prediction module 925, a motion compensation module 930, a motion estimation module 935, a loop filter 945, a reconstructed image buffer 950, an MV buffer 965, an MV prediction module 975, and an entropy encoder 990. The motion compensation module 930 and the motion estimation module 935 are part of an inter-frame prediction module 940.

[0192] In some embodiments, modules 910-990 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 910-990 are modules of hardware circuitry implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 910-990 are shown as separate modules, some modules may be combined into a single module.

[0193] Video source 905 provides a raw video signal representing pixel data for each uncompressed video frame. Subtractor 908 calculates the difference between the raw video pixel data from video source 905 and predicted pixel data 913 from motion compensation module 930 or intra-frame prediction module 925. Transform module 910 transforms this difference (or residual pixel data or residual signal 909) into transform coefficients (e.g., by performing a discrete cosine transform, or DCT). Quantization module 911 quantizes these transform coefficients into quantized data (or quantized coefficients) 912, which is encoded into bitstream 995 by entropy encoder 990.

[0194] Inverse quantization module 914 dequantizes the quantized data (or quantized coefficients) 912 to obtain transform coefficients, and inverse transform module 915 performs an inverse transform on the transform coefficients to generate a reconstructed residual 919. The reconstructed residual 919 is added to the predicted pixel data 913 to generate reconstructed pixel data 917. In some embodiments, the reconstructed pixel data 917 is temporarily stored in a linear buffer (not shown) for intra-frame image prediction and spatial MV prediction. The reconstructed pixels are filtered by a loop filter 945 and stored in a reconstructed image buffer 950. In some embodiments, the reconstructed image buffer 950 is an external storage area of ​​the video encoder 900. In some embodiments, the reconstructed image buffer 950 is an internal storage area of ​​the video encoder 900.

[0195] Intra-frame image estimation module 920 performs intra-frame prediction based on reconstructed pixel data 917 to generate intra-frame prediction data. The intra-frame prediction data is provided to entropy encoder 990 to be encoded into bitstream 995. The intra-frame prediction data is also used by intra-frame prediction module 925 to generate predicted pixel data 913.

[0196] The motion estimation module 935 performs inter-frame prediction by generating MVs (Motion Rendering Models) to parameterize the pixel data of previously decoded frames stored in the reconstructed image buffer 950. These MVs are then provided to the motion compensation module 930 to generate the predicted pixel data.

[0197] The video encoder 900 uses MV prediction to generate a predicted MV instead of encoding the complete actual MV in the bitstream, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 995.

[0198] The MV prediction module 975 generates a predicted MV based on a reference MV (i.e., a motion-compensated MV used to perform motion compensation) generated for encoding previous video frames. The MV prediction module 975 retrieves the reference MV from previous video frames from the MV buffer 965. The video decoder 900 stores the MV generated for the current video frame in the MV buffer 965 as a reference MV for generating the predicted MV.

[0199] The MV prediction module 975 uses a reference MV to create a predicted MV. This predicted MV can be calculated by spatial MV prediction or temporal MV prediction. The difference between the predicted MV and the motion-compensated MV (MC MV) of the current frame is encoded into bitstream 995 by the entropy encoder 990.

[0200] By using entropy coding techniques such as context-adjusted binary arithmetic coding (CABAC) or Huffman coding, the entropy encoder 990 encodes various parameters and data into a bitstream 995. The entropy encoder 990 encodes various header elements, flags, quantized transform coefficients 912, and residual motion datasets as syntax elements into the bitstream 995. The bitstream 995 is then stored in a storage device or transmitted to the decoder via a communication medium such as a network.

[0201] The loop filter 945 performs filtering or smoothing operations on the reconstructed pixel data to reduce artifacts in the encoded solution, especially at pixel block boundaries. In some embodiments, the filtering operation performed includes Sample Adaptive Offset (SAO). In other embodiments, the filtering operation includes an Adaptive Loop Filter (ALF).

[0202] Figure 10 A video encoder 900 implementing block segmentation constraints is illustrated. Specifically, a segmentation engine 1010 generates a set of segmentation constraints 1020 for an entropy encoder 990 based on hardware constraints (e.g., VPDU size) and a set of rate / distortion information 1015. The entropy encoder 990 encodes or transmits the set of segmentation constraints 1020 to a bitstream 995. Segmentation constraints 1020 may include constraints applicable at different video levels (e.g., sequence, image, strip, block) and transmitted by the entropy encoder 990 in strip headers, image headers, SPS, etc. Segmentation constraints 1020 may include maximum MTT block size, minimum QT leaf size, etc. The segmentation engine 1010 also provides a segmentation structure 1030 to a transform module 910 such that the transform module can perform transform operations on the current block (e.g., CTU) of the encoded / decoded pixels according to the segmentation structure 1030.

[0203] Figure 11 A process 1100 for transmitting block segmentation constraints is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implement an encoder 900 and execute process 1100 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing a decoder 900 executes process 1100.

[0204] The encoder transmits (in block 1110) segmentation control parameters in the bitstream, specifying that the maximum block size for enabling ternary tree splitting is constrained to 64 or less.

[0205] In some embodiments, segmentation control parameters are signaled in the strip header of the strip that includes the current block. The segmentation control parameters may be signaled in the image header of the current image. The segmentation control parameters may be signaled in the sequence parameter set (SPS) of a sequence of video images that includes the current image. In some embodiments, the SPS or image header sets preset values ​​for the segmentation control parameters, which may be overwritten by the strip header that includes the current block.

[0206] In some embodiments, the segmentation control parameter specifies the maximum block size by referencing the minimum size of the quadtree split leaf block, for example, as the difference between (i) the base-2 logarithm of the maximum block size from which the block can be further split by ternary tree splitting and (ii) the base-2 logarithm of the minimum size of the quadtree split leaf block. The minimum size of the quadtree split leaf block is constrained to be the smaller of 64 and the coding tree block size. The maximum block size used to enable TT splitting is further constrained to be smaller than the coding tree block size.

[0207] The minimum size of a quadtree leaf block can be specified by referring to the minimum coding block size, and the minimum size of a quadtree split leaf block is specified by using the difference between (i) the base of the minimum size of the quadtree split leaf block and (ii) the base of the minimum coding block size.

[0208] In some embodiments, the maximum block size is determined based on the size of the Virtual Pipeline Data Unit (VPDU) and defined based on ensuring that each VPDU can be fully processed by the same pipeline stage.

[0209] The encoder (in block 1120) receives the raw pixel data of the current block of the current image of the video to be encoded into the bitstream.

[0210] The encoder recursively splits the current block (in block 1130) into one or more segments. The encoder does not allow (in block 1135) any TT splits used for the current block unless the segment is less than or equal to the maximum block size.

[0211] The encoder (in block 1140) encodes one or more segments of the current block into a bit stream.

[0212] VI. Exemplary Video Encoder

[0213] Figure 12An exemplary video decoder 1200 applying constraints to block segmentation is shown. As shown, the video decoder 1200 is an image decoding or video decoding circuit that receives a bitstream 1295 and decodes the contents of the bitstream into pixel data of video frames for display. The video decoder 1200 has multiple elements or modules for decoding the bitstream 1295, including some elements selected from an inverse quantization module 1211, an inverse transform module 1210, an intra-frame prediction module 1225, a motion compensation module 1230, a loop filter 1245, a decoded image buffer 1250, an MV buffer 1265, an MV prediction module 1275, and a parser 1290. The motion compensation module 1230 is part of an inter-frame prediction module 1240.

[0214] In some embodiments, modules 1210-1290 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 1210-1290 are modules of hardware circuitry implemented by one or more ICs of an electronic device. Although modules 1210-1290 are shown as separate modules, some modules may be combined into a single module.

[0215] Parser 1290 (or entropy decoder) performs initial parsing based on the syntax defined by the video or image codec standard. The parsed syntax elements include various header elements, flags, and quantized data (or quantized coefficients) 1212. Parser 1290 parses the various syntax elements using entropy coding techniques such as Context-Adaptive Arithmetic Coding (CABAC) or Huffman coding.

[0216] Inverse quantization module 1211 dequantizes the quantized data (or quantized coefficients) 1212 to obtain transform coefficients, and inverse transform module 1210 performs an inverse transform on the transform coefficients to generate a reconstructed parameter signal 1219. The reconstructed residual signal 1219 is added to the predicted pixel data 1213 from intra-frame prediction module 1225 or motion compensation module 1230 to generate decoded pixel data 1217. The decoded pixel data is filtered by loop filter 1245 and stored in decoded image buffer 1250. In some embodiments, decoded image buffer 1250 is an external storage area of ​​video decoder 1200. In some embodiments, decoded image buffer 1250 is an internal storage area of ​​video decoder 1200.

[0217] Intra-prediction module 1225 receives intra-prediction data from bitstream 1295 and generates predicted pixel data 1213 from decoded pixel data 1217 stored in decoded image buffer 1250 based on it. In some embodiments, decoded pixel data 1217 is also stored in a linear buffer (not shown) for intra-image prediction and spatial MV prediction.

[0218] In some embodiments, the contents of the decoded image buffer 1250 are used for display. The display device 1255 retrieves the contents of the decoded image buffer 1250 directly for display or retrieves the contents of the decoded image buffer to a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded image buffer 1250 via pixel transfer.

[0219] Motion compensation module 1230 generates predicted pixel data 1213 from decoded pixel data 1217 stored in decoded image buffer 1250 based on motion compensation MV (MC MV). These motion compensation MVs are decoded by adding residual motion data from bitstream 1295 to the predicted MVs from MV prediction module 1275.

[0220] The MV prediction module 1275 generates a predicted MV, such as a motion-compensated MV, based on a reference MV generated for decoding previous video frames, to perform motion compensation. The MV prediction module 1275 retrieves the reference MV of the previous video frame from the MV buffer 1265. The video decoder 1200 stores the motion-compensated MV generated for decoding the current video frame in the MV buffer 1265 as a reference MV for generating the predicted MV.

[0221] The loop filter performs filtering or smoothing operations on the decoded pixel data 1217 to reduce encoding / decoding artifacts, especially at pixel block boundaries. In some embodiments, the filtering operation performed includes Sample Adaptive Shift (SAO). In some embodiments, the filtering operation includes Adaptive Loop Filtering (ALF).

[0222] Figure 13 A portion of a video decoder 1200 implementing block segmentation constraints is shown. Specifically, an entropy decoder 1290 parses block segmentation-related syntax elements from a bitstream 1295 and generates a set of segmentation constraints 1320. Segmentation constraints may include constraints applicable at different levels of the video hierarchy (e.g., sequence, image, strip, block) and may be signaled in strip headers, image headers, SPS, etc. Segmentation constraints 1320 may include a maximum MTT block size, a minimum QT leaf size, etc. Based on segmentation constraints 1320, the decoder may disallow certain splitting patterns that exceed the maximum block size.

[0223] The generated segmentation constraint 1320 is applied to the segmentation engine 1310, which generates a segmentation structure 1330 for the inverse transformation module 1210 so that it can perform an inverse transformation operation on a single segment (e.g., CU) according to the segmentation structure.

[0224] Figure 14A process 1400 applying block segmentation constraints is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implement a decoder to execute process 1400 by performing execution stored in a computer-readable medium. In some embodiments, an electronic device implementing decoder 1200 executes process 1400.

[0225] The decoder (in block 1410) receives segmentation control parameters from the bitstream, specifying the maximum block size for enabling ternary tree (TT) splitting. In some embodiments, this maximum block size is constrained to 64 or less.

[0226] In some embodiments, segmentation control parameters are sent in the strip header of the strip that includes the current block. These segmentation control parameters may be sent in an image header of the current image. Alternatively, they may be sent in a sequence parameter set (SPS) of a sequence of video images that includes the current image. In some embodiments, the SPS or image header sets preset values ​​for the segmentation control parameters, which may be overridden by the strip header that includes the current block.

[0227] In some embodiments, the segmentation control parameter specifies the maximum block size by referring to the minimum size of the quadtree-splitting leaf block, for example, by using the difference between (i) the base-2 logarithm of the maximum block size that can be further segmented by a ternary tree and (ii) the base-2 logarithm of the minimum size of the quadtree-splitting leaf block. The minimum size of the quadtree-splitting leaf block is constrained to the smaller of 64 and the coding tree block size. The maximum block size used to enable TT splitting can be further constrained to be smaller than the coding tree block size.

[0228] The minimum size of a quadtree split leaf block can be specified by referring to the minimum coding block size, and the minimum size of a quadtree split leaf block can be specified by using the difference between (i) the degree of the minimum size of the quadtree split leaf block with a base of 2 and (ii) the logarithm of the minimum coding block size with a base of 2.

[0229] In some embodiments, the maximum block size is determined based on the size of the Virtual Pipeline Data Unit (VPDU) and defined based on ensuring that each VPDU can be fully processed by the same pipeline stage.

[0230] In some embodiments, segmentation control parameters are signaled in the strip header of the strip that includes the current block. These segmentation control parameters may be signaled in the image header of the current image. Alternatively, they may be signaled in the sequence parameter set (SPS) of a sequence of video images that includes the current image. In some embodiments, the SPS or image header sets preset values ​​for the segmentation control parameters, which may be overridden by the strip header that includes the current block.

[0231] The decoder (in block 1420) receives the pixel blocks of the current block of the current image from the bitstream to be decoded into video.

[0232] The decoder (in block 1430) recursively splits the current block into one or more segments. The decoder (in block 1435) disables TT splitting for the current block unless the segment is less than or equal to the maximum block size.

[0233] The decoder (in block 1440) reconstructs one or more segments of the current block.

[0234] In some embodiments, the encoder may transmit (or generate) one or more syntax elements in the bitstream such that the decoder may parse the one or more syntax elements from the bitstream.

[0235] VII. Exemplary Electronic System

[0236] Many of the features and applications described above are implemented as software processes, which are specified as a set of instructions recorded on a computer-readable storage medium (also known as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), the processing unit performs the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, high-speed drives, random access memory (RAM) chips, hard disk drives, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or over wired connections.

[0237] In this specification, the term "software" means firmware residing in read-only memory or applications stored in magnetic storage that can be read from memory and processed by a processor. Additionally, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger assembly while maintaining the independence of the software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs, together with the software inventions described herein, is within the scope of this invention. In some embodiments, when installed to operate one or more electronic systems, software defines one or more specific machine implementations, their execution, and the operation of the software programs.

[0238] Figure 15An electronic system 1500, which may implement some embodiments of the present invention, is conceptually illustrated. The electronic system 1500 may be a computer (e.g., a desktop computer, personal computer, tablet computer, etc.), a telephone, a PDA, or any suitable electronic device. Such an electronic device includes various types of computer-readable media with interfaces for various other types of computer-readable media. The electronic system 1500 includes a bus 1505, a processing unit 1510, a graphics processing unit (GPU) 1515, system memory 1520, a network 1525, read-only memory 1530, permanent storage device 1535, an input device 1540, and an output device 1545.

[0239] Bus 1505 collectively represents all systems, peripherals, and chip assembly buses that communicatively connect multiple internal devices of electronic system 1500. For example, bus 1505 communicatively connects processing unit 1510, GPU 1515, read-only memory 1530, system memory 1520, and permanent storage device 1535.

[0240] From these various storage units, the processing unit 1510 retrieves instructions to be executed and data to be processed to execute the process of the present invention. In different embodiments, the processing unit may be a single processor or a multi-core processor. In some embodiments, it is transmitted and executed by a GPU 1515. The GPU 1515 can offload various computations or perform image processing provided by the processor unit 1510.

[0241] Read-only memory (ROM) stores static data and instructions used by the processing unit 1510 and other modules of the electronic system. On the other hand, permanent storage device 1535 is a read-write storage device. This device is a volatile storage unit that stores instructions and data even when the electronic system 1500 is turned off. Some embodiments of the present invention use mass storage devices (such as magnetic or optical disks and their corresponding hard disk drives) as permanent storage device 1535.

[0242] Other embodiments use removable storage devices (such as floppy disks, fast storage devices, etc., and their corresponding hard disk drives) as permanent storage devices. Like permanent storage device 1535, system memory 1520 is a read-write storage device. However, unlike storage device 1535, system memory 1520 is volatile read-write memory, such as random access memory. System memory 1520 stores some instructions and data used by the processor during operation. In some embodiments, processes according to the invention are stored in system memory 1520, permanent storage device 1535, and / or read-only memory 1530. For example, various storage units include instructions for processing multimedia according to some embodiments. From these memory units, processing unit 1510 retrieves instructions to be executed and data to be processed to execute processes of some embodiments.

[0243] Bus 1505 is also connected to input and output devices 1540 and 1545. Input device 1540 enables the user to communicate information and select commands to the electronic system. Input device 1540 includes a keypad and positioning device (also known as a cursor control device), a camera (e.g., a webcam), a microphone, or a similar device for receiving voice commands. Input device 1545 displays images generated by the electronic system or other output data. Output device 1545 includes a printer and a display device, such as a cathode ray tube (CRT) or liquid crystal display (LCD), as well as a speaker or similar audio output device. Some embodiments include a touch screen, which serves as both an input and output device.

[0244] Ultimately, as Figure 15 As shown, bus 1505 also couples electronic system 1500 to network 1525 via network interface card (not shown). In this way, the computer can be part of a computer network (such as a local area network (LAN), wide area network (WAN), intranet, or network of networks (such as the Internet)). Any or all elements of electronic system 1500 can be used in conjunction with this invention.

[0245] Some embodiments include electronic components, such as microprocessors, and their readable storage of computer program instructions, as well as memory or computer-readable media (or computer-readable storage media, their readable media, or machine-readable storage media). Examples of such computer-readable media include RAM, ROM, read-only optical discs (CD-ROM), recordable optical discs (CD-R), rewritable optical discs (CD-RW), read-only bit universal optical discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), high-speed memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state drives, read-only and recordable Blu-ray discs, ultra-high-density optical discs, any other optical or magnetic media, and floppy disks. Computer-readable media may store a computer program executed by at least one processing unit, and a set of instructions including instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that generated by a compiler, and files containing high-level code executed by a calculator, electronic component, or microprocessor using an annotator.

[0246] While the foregoing discussion primarily refers to microprocessors or multi-core processors that execute software, many of the features and applications discussed above are executed by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself. Furthermore, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0247] As used in the specification and any claim of this application, the terms "computer," "server," "processor," and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. For illustrative purposes, the terms "displayed" mean displayed on an electronic device. As used in the specification and any claim of this application, the terms "computer-readable medium," "computer-readable media," and "machine-readable medium" are limited to tangible, physical objects that store information in a computer-readable form. These terms exclude any wireless signals, wired download signals, and any other transient signals.

[0248] Although the invention has been described with reference to various specific details, those skilled in the art will recognize that the invention can be presented in other specific forms without departing from the spirit of the invention. Furthermore, some figures (including...) Figure 11 as well as Figure 14 The process is conceptually illustrated. The specific operations of these processes may not be performed in the order shown and described. Specific operations may not be performed as a single, consecutive operation, and different specific operations may be performed in different embodiments. Furthermore, a process may be implemented using various subprocesses or as part of a larger macro process. Therefore, those skilled in the art will understand that the invention is not limited by the foregoing illustrative details, but rather by the scope of the appended claims.

[0249] Notes

[0250] The topics described herein sometimes illustrate different elements included in or connected to other elements. It is understood that the architectures depicted are merely examples, and many other architectures can be implemented to achieve the same functionality. Conceptually, any arrangement of elements that achieve the same functionality is effectively “associated” to achieve the desired functionality. Therefore, any two elements combined herein to achieve a particular function can be considered “associated” with each other to achieve the desired functionality, regardless of the architecture or intermediate elements. Similarly, any two elements so associated can also be considered “operably connected” or “operably coupled” to each other to achieve the desired functionality, and any two elements that can be so associated can also be considered “operably coupled” to each other to achieve the desired functionality. Specific examples of operably coupled elements include, but are not limited to, physically matchable and / or physically interacting elements and / or wirelessly knowable and / or wirelessly interacting elements and / or logically interacting and / or logically interactable elements.

[0251] Furthermore, regarding the use of any plural and / or singular terms in this document, those skilled in the art will appropriately convert them from plural to singular and / or from singular to plural depending on the context and application. For clarity, various singular / plural substitutions may be explicitly stated herein.

[0252] Furthermore, those skilled in the art will understand that, generally, the terms used herein, especially those used in the appended claims (such as the subject of the appended claims), are generally meant as “open-ended” terms. For example, the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” and the term “includes” should be interpreted as “including but not limited to,” etc. Those skilled in the art will further understand that if a particular number of the cited claims are intentional, this intention will be explicitly stated in the claims, and if such statements are not used, this intention will not exist. For example, to aid understanding, the appended claims may include the use of the introductory phrases “at least one” and “one or more” to introduce the claims. However, the use of such phrases should not be construed as implying that the patent application scope statement introduced by the indefinite article "a" or "an" limits any particular patent application scope containing such an introduced patent application scope statement to an implementation containing only one such expression, even when the same patent application scope includes the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an," "a" and / or "an" should be interpreted as meaning "at least one" or "one or more," and the same applies to definite articles introducing patent application scope statements. Furthermore, even if a specific number of introduced patent application scope statements are explicitly listed, those skilled in the art will recognize that such statements should be interpreted as meaning at least one of the listed numbers; for example, the plain expression "two statements" without other modifications means at least two statements, or two or more statements. Furthermore, in the use of conventions such as "at least one A, B, and C, etc.", this construction is generally intended to be understood by those skilled in the art. For example, "the system has at least one A, B, and C" includes, but is not limited to, the system having A alone, having B alone, having C alone, having A and B together, having A and C together, having B and C together, and / or having A, B, and C together. In these cases where conventions similar to "at least one A, B, or C" are used, this construction is generally intended to be understood by those skilled in the art. For example, "the system has at least one A, B, or C" includes, but is not limited to, the system having A alone, having B alone, having C alone, having A and B together, having A and C together, having B and C together, and / or having A, B, and C together. Those skilled in the art will further understand that, in fact, in the description, claims, or illustrations, any separator and / or phrase indicating two or more alternative terms will be understood to consider the possibility of including one of the terms, either term, or both terms. For example, the phrase “A or B” would be understood to include the possibility of “A or B” or “A and B”.

[0253] As can be understood from the foregoing, various embodiments of the present invention have been described herein for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the invention. Therefore, the various embodiments described herein are not intended to be limiting, and the true scope and spirit are indicated by the claims following.

Claims

1. A decoding method, comprising: Receive segmentation control parameters from the bitstream, which specify the maximum size of the leaf block used to enable ternary tree splitting; Receive data from the bitstream of the pixel blocks of the current image to be decoded into video; Recursively split the current block into one or more segments, where ternary tree splits are not allowed for the current block unless the segment is less than or equal to the maximum size of the leaf block of the ternary tree split; and Reconstruct the one or more segments of the current block; The smaller value between 64 and the size of the encoded tree block is used as a specific value, which is the maximum value of the maximum size of the leaf block of the ternary tree split.

2. The decoding method according to claim 1, characterized in that, The splitting control parameter is specified by using (i) the base-2 logarithm of the maximum size of the leaf block that can be split by a ternary tree and (ii) the base-2 logarithm of the minimum size of the leaf block that can be split by a quadtree.

3. The decoding method according to claim 2, characterized in that, The maximum size of the leaf block split in the ternary tree is determined based on the size of the Virtual Pipeline Data Unit (VPDU).

4. The decoding method according to claim 3, characterized in that, The maximum size of the leaf block of the ternary tree split is defined to ensure that each virtual pipeline data unit can be fully processed by the same pipeline stage.

5. A video encoding method, comprising: The segmentation control parameter in the bitstream specifies the maximum size of the leaf block used to enable ternary tree splitting; Receive the raw pixel data of the current block of the current image of the video to be encoded into the bitstream; Recursively split the current block into one or more segments, where ternary tree splits are not allowed for the current block unless the segment is less than or equal to the maximum size of the leaf block of the ternary tree split; and Encode the one or more segments of the current block into the bit stream; The smaller value between 64 and the size of the encoded tree block is used as a specific value, which is the maximum value of the maximum size of the leaf block of the ternary tree split.

6. An electronic device, Includes a processor and memory, the processor executing program instructions stored in the memory to cause the device to perform a decoding method, the method including: Receive segmentation control parameters from the bitstream, which specify the maximum size to enable ternary tree splitting; Receive data from the bitstream of the pixel blocks of the current image to be decoded into video; Recursively split the current block into one or more segments, where ternary tree splits are not allowed for the current block unless the segment is less than or equal to the maximum size of the leaf block of the ternary tree split; and Reconstruct the one or more segments of the current block; The smaller value between 64 and the size of the encoded tree block is used as a specific value, which is the maximum value of the maximum size of the leaf block of the ternary tree split.

Images