CU partitioning method and apparatus using transmission predetermined partition in video coding and decoding
By using shortcut mode to partition blocks in video encoding and decoding technology, using predetermined partition indexes and multi-type tree structures, the problem of low partition processing efficiency in the prior art is solved, and more efficient video encoding and decoding is achieved.
Patent Information
- Application Number
- CN202380029672.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-13
- Filing Date
- 2023-04-11
- Publication Date
- 2024-11-12
AI Technical Summary
The existing video encoding and decoding technology has problems with inefficiency in partition processing, especially when processing complex video signals, resulting in impaired video quality and low encoding efficiency.
The block is partitioned using a shortcut mode, indicating whether to divide the current block through sending or parsing syntax, and using a predetermined partition to directly indicate the target partition during segmentation, reducing partition decision steps and improving partition sending efficiency. Specific methods include partitioning using quad-tree, multi-type trees, or combinations thereof, and simplifying the partitioning process by predetermined partition indexes and shortcuts.
It improves the partition transmission efficiency of video encoding and decoding, reduces syntax overhead, improves video quality and encoding efficiency, and is suitable for various video signals, including 3D video signals.
Smart Images

Figure CN118947115A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This disclosure is part of a non-provisional application that claims priority to U.S. Provisional Patent Application No. 63 / 330,343, filed on April 13, 2022. The contents of the above-listed applications are incorporated herein by reference in their entirety. Technical Field
[0003] The present disclosure generally relates to video coding systems. In particular, the present invention relates to partitioning a block using a shortcut mode to indicate a target partition among predetermined partitions. Background Art
[0004] Versatile video coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The standard was published as an ISO standard in February 2021: ISO / IEC23090-3:2021, Information technology - Codec representation of immersive media - Part 3: Versatile video coding. VVC is based on its previous generation High Efficiency Video Coding (HEVC) by adding more codec tools to improve codec efficiency and handle various types of video sources including three-dimensional (3D) video signals.
[0005] Figure 1AAn example adaptive inter / intra video coding system in combination with loop processing is shown. For intra prediction, prediction data is derived based on previously encoded and decoded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed at the encoder end and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by transform (T) 118 followed by quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to be included in a video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with auxiliary information (such as auxiliary information such as motion and coding modes associated with intra prediction and inter prediction) and other information (parameters associated with loop filters applied to underlying image areas, etc.). As Figure 1A As shown, auxiliary information associated with intra prediction 110, inter prediction 112 and loop filter 130 is provided to entropy encoder 122. When inter prediction mode is used, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (Inverse Quantization, referred to as IQ) 124 and inverse transformation (Inverse Transformation, referred to as IT) 126 to restore the residual. The residual is then added back to the prediction data 136 in reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in the reference picture buffer 134 and used for prediction of other frames.
[0006] like Figure 1A As shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC 128 may be subject to various damages. Therefore, before the reconstructed video data is stored in the reference picture buffer 134, a loop filter 130 is generally applied to the reconstructed video data to improve the video quality. For example, a deblocking filter (DF for short), a sample adaptive offset (SAO for short), and an adaptive loop filter (ALF for short) may be used. The loop filter information may need to be incorporated into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be incorporated into the bitstream. In Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an example structure of a typical video encoder. It may correspond to a High Efficiency Video Coding (HEVC) system, VP8, VP9, H.264, or VVC.
[0007] like Figure 1B As shown, the decoder can use similar or partially identical functional blocks as the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. The decoder uses an entropy decoder 140 instead of an entropy encoder 122 to decode the video bitstream into quantized transform coefficients and required coding information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 at the decoder end does not need to perform a pattern search. Instead, the decoder only needs to generate an intra-frame prediction based on the intra-frame prediction information received from the entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the intra-frame prediction information received from the entropy decoder 140 without motion estimation.
[0008] According to VVC, the input picture is divided into non-overlapping square block areas called coding tree units (CTUs), similar to HEVC. Each CTU can be divided into one or more smaller-sized coding units (CUs). The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into prediction units (PUs) as a unit to apply prediction processing such as inter-frame prediction, intra-frame prediction, etc.
[0009] The VVC standard incorporates various new codec tools to further improve the codec efficiency beyond the HEVC standard. Among the various new codec tools, some codec tools related to the present invention are summarized as follows.
[0010] The invention discloses a method and a device for improving partition transmission efficiency. Summary of the invention
[0011] A method and apparatus for video encoding and decoding are disclosed. According to the method, pixel data associated with a current block is received at an encoder end, or codec data associated with a current block to be decoded is received at a decoder end. The current block is divided into one or more sub-blocks. The division process includes: sending or parsing a first syntax, wherein the first syntax indicates whether to divide the current block; and in response to the first syntax indicating that the current block is divided, sending or parsing a second syntax, wherein the second syntax indicates whether to use a shortcut mode to directly indicate a target partition from a predetermined partition. The one or more sub-blocks are encoded or decoded.
[0012] In one method, the method further includes sending or parsing a third syntax to indicate that one of the predetermined partitions is applied to the current block in response to the second syntax indicating that the shortcut mode is applied to the current block. In another method, the method further includes sending or parsing a fourth syntax for each of a plurality of sub-blocks to indicate whether each sub-block is further divided in response to the first syntax indicating that the one of the predetermined partitions is applied to the current block, the plurality of sub-blocks being generated by applying the one of the predetermined partitions to the current block.
[0013] According to another method, the partitioning process includes: sending or parsing a first grammar, wherein the first grammar indicates whether to partition a current block; and in response to the first grammar, sending or parsing a second grammar, wherein the second grammar indicates one or more conditions, wherein the one or more conditions include whether one of the predetermined partitions is applied to the current block.
[0014] In one method, when the second syntax corresponds to a specific value, the current block is not partitioned using any predetermined partition.
[0015] In one embodiment, when the first syntax indicates that the current block is divided, the second syntax is sent or parsed. In one embodiment, the method further includes sending or parsing a third syntax for each of a plurality of sub-blocks to indicate whether each sub-block is further divided in response to the second syntax indicating that one of the predetermined partitions is applied to the current block, and the plurality of sub-blocks are generated by applying the one of the predetermined partitions to the current block. In one embodiment, when the second syntax is equal to a specific value, the current block is partitioned using one or more partition types including quadtree partitioning, multi-type tree (MTT) partitioning, or a combination thereof.
[0016] In one embodiment, when the first syntax corresponds to false, the second syntax is sent or parsed. In one embodiment, the method further includes, in response to the second syntax indicating that one of the predetermined partitions is applied to the current block, sending or parsing a third syntax for each of a plurality of sub-blocks to indicate whether each sub-block is further divided, the plurality of sub-blocks being generated by applying the one of the predetermined partitions to the current block. In one embodiment, when the second syntax is equal to a specific value, the current block is not divided.
[0017] In one embodiment, the one or more conditions further include whether quadtree (QT) partitioning is applied to the current block. In one embodiment, when the second syntax corresponds to a first specific value, the current block is partitioned using quadtree (QT) partitioning, and when the second syntax corresponds to a second specific value, the current block is partitioned using multi-type tree (MTT). BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1A An example adaptive inter / intra video codec system incorporating loop processing is shown.
[0019] Figure 1B Shown with Figure 1A The decoder corresponding to the encoder in .
[0020] Figure 2 An example of dividing a picture into coding tree units (Coding Tree Unit, CTU for short) is shown.
[0021] Figure 3 Examples of multi-type tree structures corresponding to vertical binary segmentation (SPLIT_BT_VER), horizontal binary segmentation (SPLIT_BT_HOR), vertical ternary segmentation (SPLIT_TT_VER), and horizontal ternary segmentation (SPLIT_TT_HOR) are shown.
[0022] Figure 4 An example of a transmission mechanism of partition split information in a quadtree with a nested multi-type tree codec tree structure is shown.
[0023] Figure 5 An example of dividing a CTU into multiple CUs using quadtree and nested multi-type tree codec block structures is shown, where bold block edges represent quadtree partitions and the remaining edges represent multi-type tree partitions.
[0024] Figure 6 Some examples of disabling TT splitting when the width or height of the luma codec block is greater than 64 are shown.
[0025] Figure 7An example of a new partition type according to an embodiment of the present invention, where a BT is then further split into TTs or BTs.
[0026] Figure 8 A grammar sending mechanism according to an embodiment of the present invention is shown.
[0027] Fig. 9 An example of 2 predetermined partitions is shown.
[0028] Fig.10 An example of 6 predetermined partitions is shown.
[0029] Fig.11 An example of 14 predetermined partitions is shown.
[0030] Fig.12 An example of a grammar sending mechanism according to another embodiment of the present invention is shown.
[0031] Fig.13 An example of a grammar sending mechanism according to yet another embodiment of the present invention is shown.
[0032] Fig.14 A syntax sending mechanism according to yet another embodiment of the present invention is shown.
[0033] Fig.15 An example of a new BT is shown, which allows any kind of segmentation boundary location.
[0034] Fig.16 An example of any boundary concept is shown, in which an arbitrary boundary for TT and an arbitrary boundary for QT are used.
[0035] Fig.17 An example of tilted partitioning is shown.
[0036] Fig.18 An example of a partition being divided into two parts is shown: one is a corner rectangle and the other is the remaining L-shaped partition.
[0037] Fig.19 An example of a four-part partitioning is shown by partitioning a parent CU into 4 parts, all of which are in the same cutting direction.
[0038] Fig. 20 An example of partitioning is shown, where one CU is divided into two parts, one is a rectangle and the other is the remaining samples.
[0039] Fig.21 An example of 3D partitioning is shown by grouping temporally adjacent samples together for encoding and decoding.
[0040] Fig. 22 An example showing segmentation landmark prediction by a neural network.
[0041] Fig.23 A flow chart showing an exemplary video coding system incorporating a syntax transmission mechanism according to an embodiment of the present invention.
[0042] Fig.24 A flow chart showing an exemplary video coding system incorporating a syntax sending mechanism according to another embodiment of the present invention. DETAILED DESCRIPTION
[0043] It is readily understood that the components of the present invention as generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the systems and methods of the present invention, as shown in the figures, is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. References in this specification to "an embodiment," "some embodiments," or similar language mean that specific features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in an embodiment" or "in some embodiments" appearing in various places throughout this specification do not necessarily all refer to the same embodiment.
[0044] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the present invention may be implemented without one or more specific details or using other methods, components, etc. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring various aspects of the present invention. The illustrated embodiments of the present invention will be best understood by reference to the accompanying drawings, in which the same components are represented by the same numbers throughout. The following description is by way of example only, and simply describes some selected embodiments of the apparatus and method consistent with the present invention as claimed herein.
[0045] Use tree structure to divide CTU
[0046] The picture is divided into a series of coding tree units (CTUs). The concept of CTU is the same as that of HEVC. For a picture with three sample arrays, a CTU consists of an N×N luma sample block and two corresponding chroma sample blocks. Figure 2 An example of a picture 210 divided into CTUs (shown as small squares) having 9 rows and 11 columns is shown.
[0047] The maximum allowed size of a luma block in a CTU is specified as 128x128 (although the maximum size of a luma transform block is 64x64).
[0048] In HEVC, CTUs are divided into CUs to accommodate various local characteristics using a quadtree or quadtree (QT) structure represented as a codec tree. The decision to encode and decode a picture area using inter-picture (temporal) or intra-picture (spatial) prediction is made at the leaf-CU level. Each leaf-CU can be further partitioned into one, two, or four PUs according to the PU partition type. Within a PU, the same prediction process is applied, and related information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the leaf-CU can be divided into transform units (TUs) according to another quadtree structure similar to the codec tree of the CU. One of the key features of the HEVC structure is that it has multiple partition concepts, including CU, PU, and TU.
[0049] In VVC, a quadtree with nested multi-type trees using a partitioning structure of binary and ternary partitioning replaces the concept of multiple partitioning unit types, that is, it eliminates the independence of the concepts of CU, PU, and TU, unless the size of the CU is too large for the maximum transform length, and supports more flexible CU partitioning shapes. In the codec tree structure, the CU can be a square or a rectangle. The coding tree unit (CTU) is first divided into a quadtree (also known as a quadtree) structure. Then the quadtree leaf nodes can be further divided into multi-type tree structures. For example Figure 3 As shown, there are four types of splits in the multi-type tree structure, vertical binary split (SPLIT_BT_VER 310), horizontal binary split (SPLIT_BT_HOR 320), vertical ternary split (SPLIT_TT_VER 330), and horizontal ternary split (SPLIT_TT_HOR 340). The multi-type tree leaf node is called a coding unit (CU for short), and this segmentation will be used for prediction and transform processing without further division unless the CU is too large for the maximum transform length. This means that in most cases, CU, PU, and TU have the same block size in a quadtree with a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is smaller than the width or height of the CU color component.
[0050] Figure 4The transmission mechanism of partition split information in a quadtree with nested multi-type trees is shown. If the block is not allowed to be split, a leaf node (i.e., leaf 420) is reached and the block is a CU. If the block is allowed to be split, the syntax split_cu_flag is sent to indicate whether the split is applied to the block. If the block is not split (i.e., path "0"), a leaf node (e.g., leaf 430) is reached and the block is a CU. The codec tree unit (CTU) 410 is regarded as the root of the quadtree and is first divided by the quadtree structure. When the syntax split_cu_flag indicates that the block is split (i.e., path "1"), the syntax split_qt_flag is sent to indicate whether the quadtree is used. If the syntax split_qt_flag indicates that the quadtree is used (i.e., path "1"), the block is divided into 4 sub-blocks 440. Then, each quad tree leaf node (when large enough to allow it) is further divided by the multi-type tree structure. In the multi-type tree structure, the first flag (mtt_split_cu_flag) is sent to indicate whether the node is further divided; when the node is further divided, the second flag (mtt_split_cu_vertical_flag) is sent to indicate the direction of the division, and then the third flag (mtt_split_cu_binary_flag) is sent to indicate whether the division is binary or ternary. When ternary division is used, the block is divided into 3 sub-blocks 450. When binary division is used, the block is divided into 2 sub-blocks 460. Based on the values of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the derivation of the division mode (MttSplitMode) of the multi-type tree CU is shown in Table 1.
[0051] Table 1 – Derivation of MttSplitMode based on multi-type tree syntax elements
[0052]
[0053] Figure 5 The CTU is shown to be divided into multiple CUs with quadtree and nested multi-type tree codec block structures, where the bold block edges represent quadtree partitions and the remaining edges represent multi-type tree partitions. The quadtree with nested multi-type tree partitions provides a content-adaptive codec tree structure composed of CUs. The size of the CU can be as large as the CTU or as small as 4×4 in units of luma samples. For the 4:2:0 chroma format, the maximum chroma CB size is 64×64 and the minimum size chroma CB consists of 16 chroma samples.
[0054] In VVC, the maximum supported luma transform size is 64 × 64 and the maximum supported chroma transform size is 32 × 32. When the width or height of the CB is larger than the maximum transform width or height, the CB is automatically split horizontally and / or vertically to meet the transform size limit in that direction.
[0055] The following parameters are defined and specified by the SPS syntax elements for quadtree with nested multi-type tree codec tree scheme.
[0056] –CTU size: the root node size of the quadtree
[0057] –MinQTSize: Minimum allowed quadtree leaf node size
[0058] –MaxBtSize: Maximum allowed binary tree root node size
[0059] –MaxTtSize: Maximum allowed ternary tree root node size
[0060] –MaxMttDepth: The maximum allowed hierarchical depth of the multi-type tree split from the leaves of the quad tree
[0061] –MinBtSize: The minimum allowed binary tree leaf node size
[0062] –MinTtSize: The minimum allowed ternary tree leaf node size
[0063] In one example of a quadtree with a nested multi-type tree codec tree structure, the CTU size is set to 128×128 luma samples and two corresponding 64×64 blocks of 4:2:0 chroma samples, MinQTSize is set to 16×16, MaxBtSize is set to 128×128, MaxTtSize is set to 64×64, MinBtSize and MinTtSize (width and height) are set to 4×4, and MaxMttDepth is set to 4. Tree partitioning is first applied to the CTU to generate quadtree leaf nodes. The size of the quadtree leaf node can range from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf QT node is 128×128, the binary tree will not be further split because the size exceeds MaxBtSize and MaxTtSize (i.e., 64×64). Otherwise, the quadtree leaf node may be further split by the multi-type tree. Therefore, the quadtree leaf node is also the root node of the multitree, and its multitree depth (mttDepth) is 0. When the multitree depth reaches MaxMttDepth (i.e. 4), no further splitting is considered. When the width of the multi-type tree node is equal to MinBtSize and less than or equal to 2*MinTtSize, no further horizontal splitting is considered. Similarly, when the height of the multi-type tree node is equal to MinBtSize and less than or equal to 2*MinTtSize, no further vertical splitting is considered.
[0064] To allow 64×64 luma blocks and 32×32 chroma pipeline design in the VVC hardware decoder, TT splitting is disabled when the width or height of the luma codec block is greater than 64, such as Figure 6 As shown, where block 610 corresponds to a 128x128 luma CU. TT splitting is also prohibited when the width or height of the chroma codec block is greater than 32. Block 620 corresponds to vertical binary splitting, and block 630 corresponds to horizontal binary splitting. Block 640 corresponds to a 128×128 block, which is split into four 64×64 blocks, and the vertical TT splitting is applied to the 64×64 block in the upper left corner. Block 650 corresponds to a 128×128 block, which is split into four 64×64 blocks, and the horizontal TT splitting is applied to the 64×64 block in the upper left corner.
[0065] In VVC, the codec tree scheme supports the ability to have separate block tree structures for luma and chroma. For P and B slices, the luma and chroma CTBs in one CTU must share the same codec tree structure. However, for I slices, luma and chroma can have separate block tree structures. When separate block tree mode is applied, the luma CTB is partitioned into CUs by one codec tree structure, and the chroma CTB is partitioned into chroma CUs by another codec tree structure. This means that a CU in an I slice may consist of codec blocks for the luma component or codec blocks for two chroma components, while a CU in a P or B slice always consists of codec blocks for all three color components unless the video is monochrome.
[0066] According to CTU partition statistics based on multiple test sequences, we found that when the parent CU's partition type is horizontal binary split (HBT) or vertical binary split (VBT), the "sub-CU's partition type is not split" is quite high. Therefore, we can take advantage of this correlation and propose to send one or more shortcut syntaxes after sending split_cu_flag to reduce syntax overhead. Due to the high dependence of the current partition mode on the parent partition mode under certain conditions, it may be beneficial to use a shortcut mode to quickly determine the specific partition mode selected for the current block instead of going through multiple decision steps. Therefore, the present application discloses several methods.
[0067] Method 1: New Partition Shortcut
[0068] In this proposed method, some new partition types are presented.
[0069] like Figure 7 As shown in Figure 2, a new partition type is proposed. In this partition, we have a BT, which is then further divided into TT or BT. However, unlike the traditional VVC where we need to send BT in depth 1 and then TT or BT in depth 2, this method only uses one flag to indicate the BT / TT segmentation result of 2 depths. For example, in Figure 7 In , we can use a flag to indicate this partition (called a "double-deep TTBT partition"), and if partitioning is enabled, then further bits can indicate something like Figure 7 A specific direction (710-740) is shown among the four directions.
[0070] In a more general way, the encoder can first analyze which partition shape is most suitable for the content, and send the choice of partition type in the picture header. Figure 7If a partition in the 4-bit range is suitable, it can be sent in the header to inform the decoder that it is allowed to use this partition. That is, the encoder can perform a statistical analysis to pick one or more complex partitions that have a higher probability of being selected. In some cases, more complex partitions (than Figure 7 More complex) can be pre-defined based on the current content and sent in the picture header (e.g. using one segmentation step to reach depth 3 segmentation for BT or TT). Therefore, the decoder can use it as a shorter shortcut. Although Figure 7 An example of a predetermined partition is shown, but the partition set is not limited to this specific example. In the following, more examples using predetermined partitions, shortcuts and signaling will be described.
[0071] Method 2
[0072] As described in Method 1, when a block is split, a predetermined partition may be used. When split_cu_flag is true (e.g., Figure 4 The CU shown is split), one or more syntaxes can be sent to specify whether any shortcut is used to represent the CU partition. If a shortcut is used, for each sub-partition / sub-CU, we can define whether it can be further split. An embodiment of method 2 is as follows:
[0073] Sends a flag to indicate whether the shortcut is applied.
[0074] If the shortcut is applied, an index is further sent to indicate which predetermined partition is applied.
[0075] Figure 8 An example of split flag transmission according to an embodiment of the present invention is shown. When split_cu_flag is true, a shortcut syntax (e.g., predefined_partition_flag) is sent to indicate whether a shortcut is applied. If predefined_partition_flag is true, predefined_partition_idx is sent to indicate a specific partition selected from a predetermined partition. After the partition is known, split_cu_flag is sent to each sub-CU to indicate whether each sub-CU is further divided. For Figure 4 The predefined partition shown, predefined_partition_idx is used to indicate which of the four partitions is selected.
[0076] Fig. 9An example of a set of predefined partitions according to an embodiment of the present invention is shown. In this example, the set of predefined partitions includes two partitions, where predefined_partition_idx=0 corresponds to a horizontal binary partition and predefined_partition_idx=1 corresponds to a vertical binary partition. Fig.10 Another example of a set of predetermined partitions is shown, where the set includes 6 partitions indexed from 0 to 5. Fig.11 Another example of a set of predetermined partitions is shown, where the set includes 14 partitions indexed from 1 to 13. Figures 9 to 11 The embodiments of predefined_partition_idx shown are used to illustrate a predetermined partition set. They should not be construed as limiting the present invention. In addition, different codec algorithms (e.g., truncated unary codecs instead of fixed-length codecs) are used to encode predefined_partition_idx. Additional syntax can be introduced into the existing codec tree syntax table. For example, the VVC-based syntax table according to the present embodiment, as shown in Table 2, can be formed by modifying the syntax table of the existing method.
[0077] Table 2. Example codec tree syntax for method 2 according to one embodiment
[0078]
[0079]
[0080] As shown in Table 2, as shown in Note_A(1), a new syntax predefined_partition_flag is added. If predefined_partition_flag is true, another new syntax predefined_partition_idx is added, as shown between Note_A(2) and Note_A(3). Otherwise, (i.e., predefined_partition_flag is false), the traditional partition tree syntax is used, as shown between Note_A(4) and Note_A(5).
[0081] In Table 2, predefined_partition_flag equal to 1 specifies that the partition of the current CU is one of the predetermined partitions. predefined_partition_flag equal to 0 specifies that the partition of the current CU is not one of the predetermined partitions. predefined_partition_idx specifies which predetermined partition the current CU belongs to.
[0082] According to the second embodiment of method 2, we simply remove predefined_partition_flag and change the definition of predefined_partition_idx. When predefined_partition_idx is equal to n, it means that the partition of the current CU is not one of the predefined partitions, and we need to send split_qt_flag, such as Fig.12 As shown. On the contrary, when predefined_partition_idx is not equal to n, the value of predefined_partition_idx represents one of the predetermined partitions. We propose to use variable length codec to encode predefined_partition_idx. Therefore, we need to collect statistics of CU partitions. The syntax transmission mechanism is shown in Table 3.
[0083] Table 3. Example codec tree syntax of method 2 according to another embodiment
[0084]
[0085]
[0086] As shown in Table 3, a new syntax predefined_partition_idx is added as shown in Note_B(1). When predefined_partition_idx is equal to n, the traditional partition tree syntax starting from split_qt_flag is used as shown between Note_B(2) and Note_B(3).
[0087] Method 3
[0088] In this approach, we propose to change the semantics of split_cu_flag as follows:
[0089] When split_cu_flag is equal to 1, it indicates that the current CU can be further split and the partition of the current CU is not one of the predetermined partitions.
[0090] When split_cu_flag is equal to 0, it indicates that the current CU cannot be further split or the partition of the current CU is one of the predetermined partitions.
[0091] like Fig.13 As shown, when split_cu_flag is equal to 0 (ie, false), we further send predefined_partition_idx to specify the partition of the current CU. When predefined_partition_idx is equal to n, it means that the current CU is not further split. n corresponds to a predetermined value, such as a positive integer.
[0092] The syntax predefined_partition_idx specifies which predefined partition the current CU belongs to when it is not equal to n. If the CU size is too small to be split, the decoder can deduce the values of split_cu_flag and predefined_partition_idx. We propose to use variable length codec to encode predefined_partition_idx. Therefore, we need to collect statistics of CU partitions. If predefined_partition_idx is not equal to n, for each sub-partition / sub-CU, we can define whether it can be further split by sending split_cu_flag for each sub-partition / sub-CU.
[0093] An example of the syntax table corresponding to this embodiment is shown in Table 4.
[0094] Table 4. Example codec tree syntax for method 3 according to one embodiment
[0095]
[0096] As shown in Table 4, when split_cu_flag is false, a new syntax predefined_partition_idx is added as shown between Note_C(1) and Note_C(3).
[0097] Method 4
[0098] According to this method, we simply remove predefined_partition_flag and combine predefined_partition_idx and split_qt_flag into a new syntax split_qt_and_predefined_partition_idx. When split_qt_and_predefined_partition_idx is equal to n, this means that the partition of the current CU is not one of the predetermined partitions, and quadtree split is applied to the current CU. When split_qt_and_predefined_partition_idx is equal to m, this means that the partition of the current CU is not one of the predetermined partitions, and quadtree split is not applied to the current CU. When split_qt_and_predefined_partition_idx is not equal to n and is not equal to m, it specifies which predetermined partition the current CU belongs to. Similarly, n and m correspond to predetermined values, such as two different integer values. We propose to use variable length codec to encode split_qt_and_predefined_partition_idx. Therefore, we need to collect statistics of CU partitions.
[0099] Fig.14 An example of syntax transmission corresponding to method 4 is shown. Fig.14 As shown, when split_cu_flag is true, the combined syntax split_qt_and_predefined_partition_idx is sent. When split_qt_and_predefined_partition_idx is not equal to n and not equal to m, split_cu_flag is sent for each sub-CU. When split_qt_and_predefined_partition_idx is equal to n, quadtree partitioning is applied. When split_qt_and_predefined_partition_idx is equal to m, non-quadtree partitioning is used.
[0100] split_qt_and_predefined_partition_idx equal to n specifies that the partition of the current CU is not one of the predetermined partitions, and quadtree split is applied to the current CU. split_qt_and_predefined_partition_idx equal to m specifies that the partition of the current CU is not one of the predetermined partitions, and quadtree split is not applied to the current CU.
[0101] When split_qt_and_predefined_partition_idx is neither n nor m, it specifies which predetermined partition the current CU belongs to.
[0102] Table 5. Example codec tree syntax for method 4 according to one embodiment
[0103]
[0104]
[0105] The syntax corresponding to method 4 is shown in Table 5, where the italic text corresponds to the deleted text. As shown in Table 5, the syntax split_qt_flag is replaced by split_qt_and_predefined_partition_idx, as shown in Note_D(1). In addition, the statement "if(!split_qt_flag)" is replaced by "if(split_qt_and_predefined_partition_idx==m)", as shown in Note_D(2).
[0106] Method 5: Arbitrary Boundaries of BT
[0107] In this proposed method, a new type of BT (binary segmentation) is proposed. Different from the traditional BT, the new BT can have any segmentation boundary position, such as Fig.15 For example, if the CU is 32x32, any horizontal BT divides it into 2 sub-CUs, one of which is equal to 32xN and the other is equal to 32x(32-N), where N is a value between 1 and 32.
[0108] In one sub-embodiment, the partition position (ie, N value) may be sent from the encoder to the decoder, or, in another sub-embodiment, it may be derived at the decoder side by some template matching method.
[0109] This new form of BT will benefit a lot of content since object boundaries are usually aligned with the middle of the parent CU (i.e., the traditional VBT, HBT cases will not match perfectly).
[0110] In one sub-embodiment, at any boundary of BT, in order to save syntax overhead, this partitioning can be turned on / off for different CTUs / CTU rows / tiles / slices / pictures.
[0111] In a sub-embodiment, in a BT with arbitrary boundaries, in order to simplify the transformation algorithm, it may be possible to restrict the application of “arbitrary boundary BT” to only the skip mode.
[0112] In arbitrary boundary BT, in order to simplify the intra prediction algorithm, in one sub-embodiment, it is possible to restrict the application of "arbitrary boundary BT" to only inter-frame mode. In another embodiment, some partition dimensions will not be allowed for intra-frame coding (for example, if 32x32 is divided into 7x32 and 25x32 → intra-frame coding is not allowed, but if 32x32 is divided into 16x32 and 16x32 → intra-frame coding is allowed)
[0113] In a sub-embodiment, in any boundary BT, in order to simplify the transform algorithm, the residual block can be padded to adapt to the transform kernel size. For example, if the sub-CU is 7x16, then we originally need a 7x16 transform kernel. However, according to an embodiment of the present invention, we can pad 7x16 to 8x16, so that a transform kernel of size 8x16 can be applied.
[0114] Method 6: Arbitrary segmentation boundary
[0115] The method is to apply the arbitrary boundary concept (similar to method 5) to other types of partitions, such as arbitrary boundaries for TT (1610) and arbitrary boundaries for QT (1620), as shown in Fig.16 As shown, the values of N and M can be arbitrary.
[0116] In one sub-embodiment, the partition positions (ie, the N or M values as described above) can be sent from the encoder to the decoder. In another sub-embodiment, the partition positions can be derived at the decoding end by some template matching method.
[0117] We can also apply the transform simplification or intra prediction simplification method in method 5 to method 6.
[0118] Method 7: GPM-Like Partitioning
[0119] In this method, the parent CU is divided into two parts, and the partition boundary can be inclined (i.e., non-horizontal boundary or non-vertical boundary), such as Fig.17 As shown. The partitioning method is similar to the GPM mode of VVC. Partitions A and B are different CUs, and separate coding processes can be applied, such as coding mode, candidate list, residual coding (for example, it can use shape adaptive DCT for the transform part).
[0120] For content-related features, in one embodiment, it uses the picture / tile / slice / CTU-row / CTU header to control the on-off of the GPM-partition type. For example, the encoder can perform picture analysis (such as MV field analysis) to guess whether to turn on the GPM CU partition.
[0121] In addition, a GPM-TT partition is proposed. That is, the partition is based on the TT partition, but the partition boundary can be tilted.
[0122] Method 8: Corner Partition
[0123] In this method, a parent CU is divided into two parts (1810): one is a corner-rectangular partition and the other is a remaining L-shaped partition, such as Fig.18 The ratio between the parent block and the corner block can be 2:1, 4:1 or other ratios.
[0124] In another sub-embodiment, an arbitrary boundary concept is applied, such as Fig.18 Partition 1820 in.
[0125] In one sub-embodiment, the partition positions (ie, the N or M values as described above) can be sent from the encoder to the decoder. In another sub-embodiment, the partition positions can be derived at the decoding end by some template matching method.
[0126] The residual transformation of L-type partition can be:
[0127] Sub-embodiment 1: Filled into a rectangle and transformed by traditional transformation
[0128] Sub-embodiment 2: Divided into two rectangular sub-residual blocks and transformed separately
[0129] Sub-Example 3: A Specially Designed L-Shaped Transformation Kernel
[0130] We can also apply the transform simplification or intra prediction simplification method in method 5 to the “arbitrary boundary portion” in method 8.
[0131] Method 9: Other Brute-Force Partitioning
[0132] More partition types are proposed. In one embodiment, 7:1 or 1:7 BT is used. This method is applicable to HBT or VBT. Fig.19 The 1:7 example (1920) shown is for a 1:7 VBT case. Other allocation ratios (e.g., 15:1, 1:15, 31:1, 1:32, etc.) may also be used.
[0133] By dividing the mother CU into 4 parts all along the same cutting direction, an example of four-part partitioning is revealed. A partitioning example (1910) is shown as Fig.19 It can be based on horizontal cutting or vertical cutting (e.g. Fig.19 Partition 1910 in).
[0134] Some examples of ratios between the 4 parts are as follows:
[0135] [A]1:2:4:1
[0136] [B]1:4:2:1
[0137] [C]4:2:1:1
[0138] [D]1:1:2:4
[0139] Other sequences may also be used.
[0140] Method 10: Chunks within chunks
[0141] In this method, a "block in block" partition is proposed. Fig. 20 As shown, a CU is divided into two parts (2010 and 2020), one part is a rectangle (2010), and the other part is the remaining samples (2020).
[0142] This partitioning can be beneficial to codec gain. For some special content, the old BT / QT / TT is not easy to approach good codec gain, but it is very suitable for "block in big block" partitioning.
[0143] In one sub-embodiment, the internal rectangle position (the offset relative to the upper left corner of the parent node) can be sent from the encoder to the decoder. In another sub-embodiment, the internal rectangle position can be derived at the decoding end by some template matching method.
[0144] For the residual part of the surrounding sample partition, some methods have been proposed:
[0145] Sub-embodiment 1: Filled into a rectangle and transformed by traditional transformation
[0146] Sub-embodiment 2: Split into several rectangular sub-residual blocks and transform them separately
[0147] Sub-Example 3: A Specially Designed Surrounding Transform Kernel
[0148] Method 11: 3D Partitioning Based on
[0149] Instead of the traditional CU partitioning that groups spatially adjacent samples together for encoding and decoding, the proposed method groups temporally adjacent samples together for encoding and decoding. The partitioning can be not only 2D (spatial adjacent samples, coordinates X, Y), but also 3D (spatial (X, Y) and temporal (coordinate: T)) grouping. Fig.21As shown, a codec unit may include not only samples within the same frame (e.g., frame 1), but also samples from two consecutive frames (e.g., frames 2 and 3) (2110). In another embodiment, it may include more consecutive frames. The tree partitioning will be based on 3D, which can divide the samples into sub-blocks in the temporal direction (i.e., T coordinate) and the spatial direction (i.e., X, Y coordinates). Samples from frames 2 and 3 can be used with temporal combined coding (2120) or not with temporal combined coding (2130).
[0150] The benefit is that in some static backgrounds, 3D-based transforms will outperform traditional 3D-based video codecs.
[0151] 3D tree partitioning can be turned on / off based on N-frame level, tile level, or even CTU level. The partitioning order can be temporal partitioning first and then spatial partitioning. In other words, spatial partitioning is performed first, and then temporal partitioning is performed.
[0152] Method 12: Neural Network Segmentation Landmark Prediction
[0153] Neural network is an emerging technology for circuits or algorithms that uses some neural units (neurons) connected into a network. It is powerful in image analysis, image recognition, etc.
[0154] In this method, we propose to use a neural network to predict the segmentation mark of a parent CU. The input of the neural network is a large range of pixels of the neighboring samples of the current "parent-CU", and the output of the neural network is the estimated prediction result of the segmentation mark inside the "parent-CU". The current "parent-CU" is the current area to be predicted for the segmentation mark, and the predicted segmentation mark will indicate that the decoder knows how to segment the current "parent-CU". Please note that the proposed scheme is applied on the decoder side. In other words, the neural network (NN) needs to be applied on the decoder side.
[0155] At the decoding end, when NN obtains the prediction result of the segmentation information, there are two sub-implementations.
[0156] Sub-embodiment 1: The prediction result is directly applied, but the segmentation information is limited. In other words, only part of the segmentation information is predicted, and the rest is sent by the encoder. For example, the encoder can only send BT (but cannot tell the decoder HBT or VBT), and the determination of HBT or VBT depends entirely on NN analysis.
[0157] Sub-embodiment 2: The prediction result is called the reordering priority. In other words, all the segmentation directions are reordered according to the neural network analysis, and the encoder needs to send a reordering index to the decoder. For example, the encoder sends index 0, and HBT and VBT are first sorted by the NN prediction result. If HBT is better than VBT after sorting, HBT represents index 0 and VBT represents index 1.
[0158] The benefit of the proposed method is saving the syntax rate for segmenting the tokens.
[0159] like Fig. 22 As shown, the NN input is the entire CTU range (except the current parent CU area, which is not available because it has not been decoded yet). After NN analysis, the split flag is predicted (or assisted in prediction), and the "current parent CU" can be further split according to the predicted split flag.
[0160] The aforementioned shortcut sending method can be implemented in an encoder and / or a decoder. For example, the proposed method can be implemented in an inter-frame coding and decoding module of an encoder and / or an inter-frame coding and decoding module of a decoder. For example, the proposed method can be implemented in an intra-frame coding and decoding module of an encoder and / or an intra-frame coding and decoding module of a decoder. For example, the proposed method can be implemented in a tree segmentation coding and decoding module of an encoder and / or a tree segmentation coding and decoding module of a decoder. In addition, the sending related to the proposed method can be implemented using the entropy encoder 122 in the encoder or the entropy decoder 140 in the decoder.
[0161] Fig.23 A flow chart of an exemplary video codec system is shown, which incorporates a syntax transmission mechanism according to an embodiment of the present invention. The steps shown in the flow chart can be implemented as program code that can be executed on one or more processors (e.g., one or more CPUs) at the encoder end. The steps shown in the flow chart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flow chart. According to the method, in step 2310, pixel data associated with the current block is received at the encoder end, or codec data associated with the current block to be decoded is received at the decoder end. In step 2320, the current block is divided into one or more sub-blocks, wherein the dividing the current block includes: sending or parsing a first syntax, wherein the first syntax indicates whether to divide the current block; in response to the first syntax indicating that the current block is divided, sending or parsing a second syntax, wherein the second syntax indicates whether to use a shortcut to directly indicate a target partition from a predetermined partition. The one or more sub-blocks are encoded or decoded in step 2330.
[0162] Fig.24A flow chart of an exemplary video codec system is shown, which incorporates a syntax transmission mechanism according to another embodiment of the present invention. According to the method, in step 2410, pixel data associated with the current block is received at the encoder end, or codec data associated with the current block to be decoded is received at the decoder end. In step 2420, the current block is divided into one or more sub-blocks, wherein dividing the current block includes: sending or parsing a first syntax, wherein the first syntax indicates whether to split the current block; in response to the first syntax, sending or parsing a second syntax, wherein the second syntax indicates one or more conditions, including whether one of the predetermined partitions is applied to the current block. In step 2430, the one or more sub-blocks are encoded or decoded.
[0163] The flowchart shown is intended to illustrate an example of video coding according to the present invention. A person skilled in the art may modify each step, rearrange the steps, split the steps, or combine the steps to implement the present invention without departing from the spirit of the present invention. In the present disclosure, specific syntax and semantics are used to illustrate examples to implement embodiments of the present invention. A person skilled in the art may implement the present invention by replacing the above syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present invention.
[0164] The above description is presented to enable those skilled in the art to implement the present invention provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but to conform to the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are described in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention can be implemented.
[0165] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, one embodiment of the present invention can be a program code integrated into one or more circuits in a video compression chip or integrated into video compression software to perform the processing described herein. An embodiment of the present invention can also be a program code to be executed on a digital signal processor (Digital Signal Processor, referred to as DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (field programmable gate array, referred to as FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software code or firmware code that defines the specific method embodied in the present invention. The software code or firmware code can be developed in different programming languages and different formats or styles. The software code can also be compiled for different target platforms. However, the different code formats, styles and languages of the software code and other ways of configuring the code to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.
[0166] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects as illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes within the equivalent meaning and scope of the claims should be included within their scope.
[0167] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects as illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes within the equivalent meaning and scope of the claims should be included within their scope.
Claims
1. A video encoding and decoding method, the method comprising: Receiving pixel data associated with a current block at an encoding end, or receiving codec data associated with the current block to be decoded at a decoding end; Divide the current block into one or more sub-blocks, wherein dividing the current block comprises: Sending or parsing a first syntax, wherein the first syntax indicates whether to split the current block; and In response to the first syntax indicating that the current block is split, sending or parsing a second syntax, wherein the second syntax indicates whether to use a shortcut to directly indicate a target partition from a plurality of predetermined partitions; and The one or more sub-blocks are encoded or decoded.
2. The video encoding and decoding method according to claim 1, characterized in that: Further including: In response to the second syntax indicating that the shortcut mode is applied to the current block, a third syntax is sent or parsed to indicate that a predetermined partition among the plurality of predetermined partitions is applied to the current block.
3. The video encoding and decoding method according to claim 2, characterized in that: Further including: In response to the third syntax indicating that the predetermined partition of the plurality of predetermined partitions is applied to the current block, sending or parsing a fourth syntax for each of a plurality of sub-blocks to indicate whether each of the plurality of sub-blocks is further divided, the plurality of sub-blocks being generated by applying the predetermined partition of the plurality of predetermined partitions to the current block.
4. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors, configured to: Receiving pixel data associated with a current block at an encoding end, or receiving codec data associated with the current block to be decoded at a decoding end; The current block is divided into one or more sub-blocks, wherein: Dividing the current block includes: Sending or parsing a first syntax, wherein the first syntax indicates whether to split the current block; and In response to the first syntax indicating that the current block is split, sending or parsing a second syntax, wherein the second syntax indicates whether to use a shortcut to directly indicate a target partition from a plurality of predetermined partitions; and The one or more sub-blocks are encoded or decoded.
5. The video encoding and decoding device according to claim 4, characterized in that: In response to the second syntax indicating that the shortcut mode is applied to the current block, the one or more electronic devices or processors send or parse a third syntax to indicate that a predetermined partition among the plurality of predetermined partitions is applied to the current block.
6. The video encoding and decoding device according to claim 5, characterized in that: In response to the third grammar indicating that the predetermined partition among the multiple predetermined partitions is applied to the current block, the one or more electronic devices or processors send or parse a fourth grammar for each of a plurality of sub-blocks to indicate whether each of the plurality of sub-blocks is further divided, and the plurality of sub-blocks are generated by applying the predetermined partition among the multiple predetermined partitions to the current block.
7. A video encoding and decoding method, the method comprising: Receiving pixel data associated with a current block at an encoding end, or receiving codec data associated with the current block to be decoded at a decoding end; Divide the current block into one or more sub-blocks, wherein dividing the current block comprises: Sending or parsing a first syntax, wherein the first syntax indicates whether to split the current block; and In response to the first grammar, sending or parsing a second grammar, wherein the second grammar indicates one or more conditions, the one or more conditions including whether a predetermined partition among a plurality of predetermined partitions is applied to the current block; and The one or more sub-blocks are encoded or decoded.
8. The video encoding and decoding method according to claim 7, characterized in that: When the second syntax corresponds to a specific value, the current block is not partitioned using any predetermined partition.
9. The video encoding and decoding method according to claim 8, characterized in that: When the first syntax indicates that the current block is segmented, the second syntax is sent or parsed.
10. The video encoding and decoding method according to claim 9, characterized in that: In response to the second syntax indicating that a predetermined partition of the plurality of predetermined partitions is applied to the current block, a third syntax for each of a plurality of sub-blocks is sent or parsed to indicate whether each of the plurality of sub-blocks is further divided, the plurality of sub-blocks being generated by applying the predetermined partition of the plurality of predetermined partitions to the current block.
11. The video encoding and decoding method according to claim 9, characterized in that: When the second syntax is equal to the specific value, the current block is partitioned using one or more partition types, including quadtree partitioning, multi-type tree partitioning, or a combination of the two.
12. The video encoding and decoding method according to claim 8, characterized in that: When the first grammar corresponds to false, the second grammar is sent or parsed.
13. The video encoding and decoding method according to claim 12, characterized in that: In response to the second syntax indicating that the predetermined partition of the multiple predetermined partitions is applied to the current block, sending or parsing a third syntax for each sub-block of a plurality of sub-blocks to indicate whether each sub-block of the plurality of sub-blocks is further divided, the plurality of sub-blocks being generated by applying the predetermined partition of the multiple predetermined partitions to the current block.
14. The video encoding and decoding method according to claim 12, characterized in that: When the second syntax is equal to the specific value, the current block is not split.
15. The video encoding and decoding method according to claim 7, characterized in that: The one or more conditions also include whether quadtree partitioning is applied to the current block.
16. The video encoding and decoding method according to claim 15, characterized in that: When the second syntax corresponds to a first specific value, the current block is partitioned using quadtree partitioning, and when the second syntax corresponds to a second specific value, the current block is partitioned using multi-type tree partitioning.
17. An apparatus for video encoding, the apparatus comprising one or more electronic devices or processors for: Receiving pixel data associated with a current block at an encoding end, or receiving codec data associated with the current block to be decoded at a decoding end; The current block is divided into one or more sub-blocks, wherein: Dividing the current block includes: Sending or parsing a first syntax, wherein the first syntax indicates whether to split the current block; and In response to the first grammar, sending or parsing a second grammar, wherein the second grammar indicates one or more conditions, the one or more conditions including whether a predetermined partition among a plurality of predetermined partitions is applied to the current block; and The one or more sub-blocks are encoded or decoded.
