Method and apparatus for video coding, device and computer readable medium

Abstract

Embodiments of the present disclosure relate to a method and apparatus for video coding, device and computer readable medium. A method for video decoding includes determining a maximum transform size associated with a video sequence based on a first high level syntax element of the video sequence, the maximum transform size corresponding to a maximum transform unit area. When a sub-block transform (SBT) mode is enabled, a maximum block size of the SBT mode is allowed to be constrained by the maximum transform size. Wherein the maximum block size of the SBT mode is allowed to be constrained by the maximum transform size includes allowing a width and a height of a maximum coding unit (CU) of the SBT mode to be derived as a minimum value between the maximum transform size indicated by the first high level syntax element and a width and a height of a maximum CU of the SBT mode allowed to be indicated in a second high level syntax element.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 804,547, filed February 28, 2020, entitled "Maximum Change Size Control," which in turn claims priority to U.S. Patent Application No. 62 / 813,665, filed March 4, 2019, entitled "Maximum Change Size Control." The entire disclosure of the earlier application is incorporated herein by reference. Technical Field

[0003] This disclosure is made for next-generation video coding and decoding technologies (such as Universal Video Coding (VVC)) that go beyond HEVC (High Efficiency Video Coding and Decoding). More specifically, a scheme for controlling the maximum transform size is proposed, and the interaction between the maximum transform size and transform partitioning schemes (e.g., Subblock Transform (SBT) and Intra-Frame Sub-Partitioning (ISP)) is also discussed. Background Technology

[0004] The ITU-T VCEG (Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU)) (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) (Standardization Subcommittee of the Joint Technical Committee of the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) ISO / IEC JTC 1) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (1st edition), 2014 (2nd edition), 2015 (3rd edition), and 2016 (4th edition). Since then, they have been studying the potential need for standardization of future video coding and decoding technologies with compression capabilities significantly exceeding the HEVC standard (including its extensions). In October 2017, they issued a joint solicitation for proposals on video compression capabilities exceeding HEVC (CfP). As of February 15, 2018, a total of 22 CfP responses were submitted for Standard Dynamic Range (SDR), 12 CfP responses for High Dynamic Range (HDR), and 12 CfP responses for 360 video categories. In April 2018, all received CfP responses were evaluated at the 122 Moving Picture Experts Group (MPEG) / 10th Joint Video Development Group – Joint Video Experts Group (JVET) meeting. Following careful evaluation, JVET officially launched the standardization of a next-generation video codec beyond HEVC, the so-called Universal Video Codec (VVC). The current version of VTM (VVC test model) is VTM4. Summary of the Invention

[0005] According to one aspect of this disclosure, a video decoding method includes: determining a maximum transform size associated with the video sequence based on a first high-level syntax element, the maximum transform size corresponding to a maximum transform unit region. When Subblock Transform (SBT) mode is enabled, the maximum block size of the SBT mode is allowed to be constrained by the maximum transform size. Specifically, allowing the maximum block size of the SBT mode to be constrained by the maximum transform size includes: allowing the width and height of the maximum coding unit (CU) of the SBT mode to be derived as the minimum value between the maximum transform size indicated by the first high-level syntax element and the width and height of the maximum CU allowed for the SBT mode indicated in a second high-level syntax element.

[0006] According to one aspect of this disclosure, a video coding method includes: determining a maximum transform size associated with a video sequence, the maximum transform size corresponding to a maximum transform unit region; and encoding the video sequence using the maximum transform size. When Subblock Transform (SBT) mode is enabled, the maximum block size of the SBT mode is allowed to be constrained by the maximum transform size. Specifically, allowing the maximum block size of the SBT mode to be constrained by the maximum transform size includes: allowing the width and height of the maximum coding unit (CU) of the SBT mode to be derived as the minimum between the maximum transform size indicated by a first high-level syntax element and the width and height of the maximum CU allowed for the SBT mode indicated in a second high-level syntax element.

[0007] According to one aspect of this disclosure, a video decoding apparatus is characterized by comprising: a determining module configured to determine a maximum transform size associated with the video sequence based on a first high-level syntax element, the maximum transform size corresponding to a maximum transform unit region; and a constraining module configured to allow the maximum block size of the SBT mode to be constrained by the maximum transform size when the Subblock Transform (SBT) mode is enabled. The constraining module is further configured to allow the width and height of the maximum coding unit (CU) of the SBT mode to be derived as the minimum value between the maximum transform size indicated by the first high-level syntax element and the width and height of the maximum allowed CU of the SBT mode indicated in a second high-level syntax element.

[0008] According to one aspect of this disclosure, a video encoding apparatus includes: a determining module configured to determine a maximum transform size associated with a video sequence, the maximum transform size corresponding to a maximum transform unit region; an encoding module configured to encode the video sequence using the maximum transform size; and a constraint module configured to allow the maximum block size of the SBT mode to be constrained by the maximum transform size when the Subblock Transform (SBT) mode is enabled. The constraint module is further configured to: allow the width and height of the maximum coding unit (CU) of the SBT mode to be derived as the minimum value between the maximum transform size indicated by a first high-level syntax element and the width and height of the maximum CU allowed for the SBT mode indicated in a second high-level syntax element.

[0009] According to one aspect of this disclosure, an apparatus for performing maximum transform size control on the decoding of a video sequence includes: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to the instructions of the program code to perform the video decoding and / or encoding method described in the embodiments of this application.

[0010] According to one aspect of this disclosure, a non-volatile computer-readable medium stores instructions and a bitstream consisting of a video sequence, the instructions comprising one or more instructions that, when executed by one or more processors of a device performing encoding or decoding of the video sequence, cause the one or more processors to perform the video decoding and / or encoding methods described in embodiments of this application. Attached Figure Description

[0011] Other features, properties, and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, wherein:

[0012] Figure 1 This is a flowchart of an example process for performing maximum transform size control on the encoding or decoding of a video sequence.

[0013] Figure 2 This is a simplified block diagram of a communication system according to an embodiment of the present disclosure.

[0014] Figure 3 This is a schematic diagram of the placement of video encoders and decoders in a streaming environment.

[0015] Figure 4 This is a functional block diagram of a video decoder according to an embodiment of the present disclosure.

[0016] Figure 5 This is a functional block diagram of a video encoder according to an embodiment of the present disclosure.

[0017] Figure 6This is a schematic diagram of a computer system according to an embodiment.

[0018] The problem to be solved

[0019] In the latest VVC draft, the maximum TU size is a fixed number of 64, meaning there is no ability to control the maximum TU size. However, since the maximum TU size affects the hardware complexity of the encoder implementation (e.g., pipeline intermediate buffer size, number of multipliers, etc.), it may be necessary to control the maximum TU size in VVC.

[0020] The latest VVC draft already includes SBT and ISP, and needs to handle the interaction between SBT, ISP, and the maximum TU size. For example, in SBT, the SPS flag `sps_sbt_max_size_64_flag` is signaled to indicate whether the maximum SBT size is 32 or 64 bits. When `sps_sbt_max_size_64_flag` is true and the maximum TU size is 32 bits, the current VVC draft cannot handle this and may trigger an encoder crash.

[0021] Currently, ISP mode is allowed for all CU sizes. However, when the maximum transform size is set to less than 64, a conflict arises between performing implicit transform partitioning and performing explicit transform partitioning using ISP with signaling. For example, when the maximum transform size is 16, for a 64×16 TU, without ISP, it should be implicitly partitioned into four 16×16 TUs. However, with ISP, it can be partitioned using vertical ISP partitioning, resulting in the same four 16×16 TUs, but with signaling. Detailed Implementation

[0022] In HEVC, a quadtree structure, represented as a coding tree, is used to partition a coding tree unit (CTU) into multiple coding units (CUs) to accommodate various local features. At the CU level, a decision is made as to whether to use inter-frame picture (temporal) prediction or intra-frame picture (spatial) prediction to encode the picture region. Depending on the prediction unit (PU) partitioning type, each CU can be further partitioned into one, two, or four PUs. Within a PU, the same prediction process is applied, and relevant information is sent to the decoder based on the PU. After obtaining residual blocks by applying a prediction process based on the PU partitioning type, the CUs can be divided into multiple transform units (TUs) according to another quadtree structure similar to the coding tree of the CU. One of the key features of the HEVC structure is that it has multiple partitioning concepts, including CUs, PUs, and TUs.

[0023] The Quadtree Plus Binary Tree (QTBT) structure eliminates the concept of multiple partition types; that is, it eliminates the separation of CU, PU, ​​and TU concepts and supports more flexible CU partition shapes. In a QTBT block structure, a CU can have a square or rectangular shape. For example... Figure 1 As shown, the coding unit (CTU) is first partitioned using a quadtree structure. Then, the leaf nodes of the quadtree are further partitioned using a binary tree structure. In the binary tree partitioning, there are two types of partitioning: symmetrical horizontal partitioning and symmetrical vertical partitioning. The leaf nodes of the binary tree are called coding units (CUs), and this partitioning is used for prediction and transform processing without any further partitioning. This means that CUs, PUs, and TUs have the same block size in the QTBT coding block structure. In JEM, CUs sometimes consist of coding blocks (CBs) of different color components; for example, in the case of P and B slices in a 4:2:0 chroma format, a CU contains one luma CB and two chroma CBs. Sometimes, CUs contain CBs of a single component; for example, in the case of I slices, a CU contains only one luma CB or only two chroma CBs.

[0024] The following parameters are defined for the QTBT partitioning scheme:

[0025] CTU size: The size of the root node of the quadtree, the same concept as in HEVC;

[0026] MinQTSize: The minimum allowed size of a quadtree leaf node;

[0027] MaxBTSize: The maximum allowed size of the root node of a binary tree;

[0028] MaxBTDepth: The maximum allowed binary tree depth;

[0029] MinBTSize: The minimum allowed size of a binary leaf node.

[0030] In one example of a QTBT partitioning structure, the CTU size is set to 128×128 luminance samples with two corresponding 64×64 chroma sample blocks. MinQTSize is set to 16×16, MaxBTSize to 64×64, MinBTSize (for width and height) to 4×4, and MaxBTDepth to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes ranging from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If a quadtree leaf node is 128×128, it will not be further partitioned into binary trees because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, the quadtree leaf node will be further partitioned into binary trees. Therefore, the quadtree leaf node is also the root node of the binary tree, and its binary tree depth is 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), further partitioning is not considered. When the width of a binary tree node equals MinBTSize (i.e., 4), no further horizontal segmentation is considered. Similarly, when the height of a binary tree node equals MinBTSize, no further vertical segmentation is considered. Leaf nodes of the binary tree are further processed through prediction and transformation without any further partitioning. In JEM, the maximum CTU size is 256×256 luminance samples.

[0031] In block partitioning using QTBT, for binary tree partitions (i.e., non-leaf nodes), a flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used, where 0 indicates a horizontal partition and 1 indicates a vertical partition. For quadtree partitions, no partition type indication is needed because quadtree partitions always partition blocks horizontally and vertically to produce four child blocks of equal size.

[0032] Furthermore, the QTBT scheme supports the flexibility of having separate QTBT structures for luma and chroma. Currently, for P-slices and B-slices, the luma and chroma CTBs within a single CTU share the same QTBT structure. However, for I-slices, the luma CTB is divided into multiple CUs using a QTBT structure, and the chroma CTB is divided into multiple chroma CUs using a different QTBT structure. This means that a CU in an I-slice consists of a block of code for the luma component or two blocks of code for the chroma components, while a CU in a P-slice or B-slice consists of blocks of code for all three color components.

[0033] In HEVC, inter-frame prediction for small blocks is restricted to reduce memory accesses for motion compensation. This prevents 4×8 and 8×4 blocks from supporting bidirectional prediction, and 4×4 blocks from supporting inter-frame prediction altogether. These restrictions are eliminated in QTBT, implemented in JEM-7.0.

[0034] A Multi-Type Tree (MTT) structure is proposed. MTT is a more flexible tree structure than QTBT. In MTT, in addition to quadtrees and binary trees, horizontal and vertical center-side ternaries are introduced. The main advantages of ternary partitioning are: it complements quadtree and binary tree partitioning; ternary partitioning can capture objects located at the block center, while quadtrees and binary trees are always partitioned along the block center. The proposed ternary partitions always have a width and height of 2, thus requiring no additional transformations.

[0035] The two-level tree design is primarily aimed at reducing complexity. Theoretically, the time complexity of traversing a tree is T. D Where T represents the number of partition types and D is the depth of the tree.

[0036] In HEVC, the main transforms are 4-point, 8-point, 16-point, and 32-point DCT-2, and the transform kernel matrix is ​​represented by an 8-bit integer, i.e., an 8-bit transform kernel. The transform kernel matrix of a smaller DCT-2 is a subset of that of a larger DCT-2.

[0037] The DCT-2 kernel exhibits symmetric / antisymmetric characteristics, thus supporting a so-called "partial butterfly" implementation to reduce the number of operation counts (multiplication, addition / subtraction, shifting), and the same matrix multiplication results can be obtained using partial butterfly.

[0038] In the current VVC, in addition to the 4-point, 8-point, 16-point and 32-point DCT-2 transforms that are the same as HEVC, additional 2-point and 64-point DCT-2 are also included.

[0039] In addition to DCT-2 and 4×4 DST-7 already used in HEVC, the Adaptive Multiple Transform (AMT, or Enhanced Multiple Transform (EMT), or Multiple Transform Selection (MTS)) scheme has been used in VVC for residual coding of inter-frame and intra-frame coded blocks. It uses multiple transforms selected from the DCT / DST family, in addition to the current transforms in HEVC. The newly introduced transform matrices are DST-7 and DCT-8.

[0040] All master transform matrices in a VVC are represented using 8 bits. AMT is applied to CUs with a width and height less than or equal to 32, and whether AMT is applied is controlled by a flag called mts_flag. When mts_flag equals 0, only DCT-2 is used to encode the residuals. When mts_flag equals 1, two binary numbers (bins) are used to further signal the index mts_idx to specify the horizontal and vertical transforms to be used.

[0041] Intra-Frame Sub-Partition (ISP) coding mode divides the luma intra-frame prediction block vertically or horizontally into 2 or 4 sub-partitions based on the block size dimension.

[0042] For each of these sub-partitions, a residual signal is generated by entropy decoding the coefficients sent by the encoder, followed by inverse quantization and inverse transform. Intra-frame prediction is then performed on the sub-partition, and finally, the corresponding reconstructed sample is obtained by adding the residual signal to the prediction signal. Therefore, the reconstructed values ​​of each sub-partition can be used to generate the prediction for the next sub-partition, and this process is repeated. All sub-partitions share the same intra-frame mode.

[0043] The ISP algorithm will only test using intra-modes that are part of the MPM list. Therefore, if a block uses ISP, the MPM flag will be inferred to be 1. Furthermore, if ISP is used for a block, the MPM list will be modified to exclude DC modes and prioritize the horizontal intra-modes used for ISP horizontal segmentation and the vertical intra-modes used for vertical segmentation.

[0044] In ISP, because the transformation and rebuild are performed separately for each subpartition, each subpartition can be regarded as a subTU.

[0045]

[0046]

[0047] In JVET-J0024, JVET-K0139, and JVET-L0358, a Spatial Variation Transform (SVT) scheme was proposed. Using SVT, for inter-frame prediction residuals, the coded block contains only residual blocks, but the residual blocks are smaller than the coded block size; therefore, the transform size in SVT is smaller than the coded block size. For regions not covered by residual blocks or transforms, the residual is assumed to be zero.

[0048] The recommended SBT is shown below when used on top of VVC, with the added text highlighted in gray. It can be seen that the SBT method requires signaling additional overhead bits (cu_sbt_flag, cu_sbt_quad_flag, cu_sbt_horizontal_flag, and cu_sbt_pos_flag) to indicate the sub-block type (horizontal or vertical), size (half or quarter), and position (left or right, top or bottom).

[0049]

[0050]

[0051]

[0052]

[0053]

[0054] A value of 0 for "sps_sbt_enabled_flag" indicates that subblock transformation is disabled for inter-frame prediction CUs. A value of 1 for "sps_sbt_enabled_flag" indicates that subblock transformation is enabled for inter-frame prediction CUs.

[0055] A slice_sbt_max_size_64_flag value of 0 specifies that the maximum CU width and height allowed for sub-block transformations is 32. A slice_sbt_max_size_64_flag value of 1 specifies that the maximum CU width and height allowed for sub-block transformations is 64.

[0056] maxSbtSize=slice_sbt_max_size_64_flag? 64:32

[0057] A value of 1 for cu_sbt_flag[x0][y0] indicates that subblock transformation is used for the current coding unit. A value of 0 for cu_sbt_flag[x0][y0] indicates that subblock transformation is not used for the current coding unit.

[0058] If cu_sbt_flag[x0][y0] does not exist, its value is inferred to be 0.

[0059] When using sub-block transform, the coding unit is flattened into two transform units, one of which has a residual and the other does not.

[0060] The value of cu_sbt_quad_flag[x0][y0] equal to 1 indicates that, for the current coding unit, the sub-block transform includes a transform unit of 1 / 4 the size of the current coding unit. The value of cu_sbt_quad_flag[x0][y0] equal to 0 indicates that, for the current coding unit, the sub-block transform includes a transform unit of 1 / 2 the size of the current coding unit.

[0061] If cu_sbt_quad_flag[x0][y0] does not exist, its value is inferred to be 0.

[0062] `cu_sbt_horizontal_flag[x0][y0]` equal to 1 specifies that the current coding unit is flattened into 2 transform units by horizontal partitioning. `cu_sbt_horizontal_flag[x0][y0]` equal to 0 specifies that the current coding unit is flattened into 2 transform units by vertical partitioning.

[0063] When cu_sbt_horizontal_flag[x0][y0] does not exist, its value is derived as follows:

[0064] If cu_sbt_quad_flag[x0][y0] equals 1, then set cu_sbt_horizontal_flag[x0][y0] to be equal to allowSbtHoriQuad.

[0065] Otherwise (cu_sbt_quad_flag[x0][y0] equals 0), set cu_sbt_horizontal_flag[x0][y0] to allowSbtHoriHalf.

[0066] A value of 1 for `cu_sbt_pos_flag[x0][y0]` indicates that `tu_cbf_luma`, `tu_cbf_cb`, and `tu_cbf_cr` of the first transform unit in the current coding unit do not exist in the bitstream. A value of 0 for `cu_sbt_pos_flag[x0][y0]` indicates that `tu_cbf_luma`, `tu_cbf_cb`, and `tu_cbf_cr` of the second transform unit in the current coding unit do not exist in the bitstream.

[0067] The input to this process is:

[0068] The luminance position (xTbY, yTbY) specifies the top-left sample of the current luminance transform block relative to the top-left luminance sample of the current image.

[0069] The variable nTbW specifies the width of the current transform block.

[0070] The variable nTbH specifies the height of the current transform block.

[0071] The variable cIdx specifies the color component of the current block.

[0072] The array d[x][y] of the scaling transformation coefficients (nTbW)x(nTbH), where x = 0..nTbW-1 and y = 0..nTbH-1.

[0073] The output of this process is an array r[x][y] of residual samples (nTbW)x(nTbH), where x = 0..nTbW-1 and y = 0..nTbH-1.

[0074] If cu_sbt_flag[xTbY][yTbY] equals 1, then based on cu_sbt_horizontal_flag[xTbY][yTbY] and cu_sbt_pos_flag[xTbY][yTbY], derive the variable trTypeHor for the specified horizontal transform kernel and the variable trTypeVer for the specified vertical transform kernel in Table 8-X.

[0075] Otherwise (cu_sbt_flag[xTbY][yTbY] equals 0), then based on mts_idx[xTbY][yTbY] and CuPredMode[xTbY][yTbY], derive the variable trTypeHor for the specified horizontal transform kernel and the variable trTypeVer for the specified vertical transform kernel in Table 8-9.

[0076] The (nTbW)x(nTbH) array r of the residual samples is derived as follows:

[0077] By calling a one-dimensional transformation process for each column x = 0..nTbW–1 (which has a transformation block height nTbH), each (vertical) column of the scaled transformation coefficients d[x][y] (x = 0..nTbW-1, y = 0..nTbH-1) is transformed to e[x][y] (x = 0..nTbW-1, y = 0..nTbH-1). The list d[x][y] of y = 0..nTbH-1 and the transformation type variable trType set to equal trTypeVer are taken as input, and the output is the list e[x][y] of y = 0..nTbH-1.

[0078] The intermediate sample values ​​g[x][y] for x = 0..nTbW-1 and y = 0..nTbH-1 are derived as follows:

[0079] g[x][y]=Clip3(CoeffMin,CoeffMax,(e[x][y]+256)>>9)

[0080] By calling a one-dimensional transformation process for each row y = 0..nTbH–1 (which has a transform block width nTbW), each (horizontal) row of the resulting array g[x][y] (x = 0..nTbW–1, y = 0..nTbH–1) is transformed into r[x][y] (x = 0..nTbW–1, y = 0..nTbH–1). The list g[x][y] of x = 0..nTbW–1 and the transformation type variable trType, set to equal trTypeHor, are taken as input, and the output is the list r[x][y] of x = 0..nTbW–1.

[0081]

[0082] Different YUV formats exist. For the 4:2:0 format, LM prediction applies a six-tap interpolation filter to obtain the corresponding... Figure 6 The chromaticity sample shown is a downsampled luminance sample. The downsampled luminance sample Rec'L[x,y] is calculated from the reconstructed luminance sample using the following formula:

[0083] Rec' L [x,y]=(2×Rec L [2x,2y]+2×Rec L [2x,2y+1]+

[0084] Rec L [2x-1,2y]+Rec L [2x+1,2y]+

[0085] Rec L [2x-1,2y+1]+Rec L [2x+1,2y+1]+4)>>3

[0086] Figure 1 This is a flowchart of an example process 100 for a method of performing maximum transform size control on the encoding or decoding of a video sequence. In some implementations, Figure 1 One or more process blocks can be executed by the decoder. In some implementations, Figure 1 One or more process blocks can be executed by another device or a group of devices that are separate from or include the decoder, such as an encoder.

[0087] like Figure 1 As shown, process 100 may include: the decoder identifying high-level syntax elements associated with the video sequence (box 110).

[0088] like Figure 1 As further shown in the figure, process 100 may include: the decoder determining the maximum transform size associated with the video sequence based on the identification of high-level syntax elements associated with the video sequence (box 120).

[0089] like Figure 1 As further shown in the figure, process 100 may include: the decoder decoding the video sequence based on determining the maximum transform size associated with the video sequence (box 130).

[0090] like Figure 1 As further shown in the figure, process 100 may include: the decoder decoding the video sequence based on using the maximum transform size, and transmitting the video sequence (box 140).

[0091] As used herein, advanced syntax elements can refer to any of the Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Slice Header, Patch Header, Patch Group Header, etc. Furthermore, the CTU (Code Tree Unit, which is the largest CU size) header can refer to the syntax element that signals each CTU, for example, as header information. Additionally, "transform size" can refer to the maximum transform width and / or height, or the maximum transform unit region size.

[0092] According to an embodiment, the maximum transform size is signaled in the high-level syntax element or CTU header. The minimum transform size is not signaled but is set to a default value. Example values ​​for the minimum transform size include lengths of 4, 8, and 16. In one embodiment, limiting the maximum transform size must be one of several predefined values. Examples of predefined values ​​include lengths of 16, 32, and 64.

[0093] In one embodiment, the maximum transform size is signaled as a logarithm of the constant minus the maximum transform size. For example, if the minimum supported maximum transform size is set to 16, the maximum transform size is signaled as log2_max_transform_size_minus_4; that is, if the maximum transform size is 64, then a signal value of 2 is sent, and for a maximum transform size of 32, a signal value of 1 is sent. In another example, the minimum possible maximum transform size is set to 32, and for maximum transform sizes of 32 and 64, signal values ​​of 0 and 1 are sent, respectively.

[0094] According to an embodiment, only the maximum transform unit region size is signaled in the high-level syntax element or CTU header. The minimum transform unit region size is not signaled but is set to a default value, with example values ​​including 16 samples, 32 samples, and 64 samples. In an embodiment, the maximum transform unit region size is limited to at least the default value. Examples of default values ​​include 64 samples, 128 samples, 256 samples, 512 samples, 1024 samples, 2048 samples, or 4096 samples. In an embodiment, the maximum transform unit region size is signaled by the logarithm of the smallest possible maximum transform unit region size. For example, if the smallest possible maximum transform unit region size is set to 256, the logarithm of the maximum transform size divided by 256 is signaled; that is, if the maximum transform unit region sizes are 256, 512, 1024, 2048, and 4096, then 0, 1, 2, 3, and 4 are signaled.

[0095] According to an embodiment, the maximum SBT size is constrained based on the maximum transform size or transform unit size that is signaled.

[0096]

[0097] In one embodiment, when the maximum transform size is less than 64, sps_sbt_max_size_64_flag is not signaled, but instead is exported as the default value, indicating that the maximum CU width and height allowed for the SBT is 32 luminance samples.

[0098] In another embodiment, the actual maximum CU width and height of the SBT are further adjusted by a high-level syntax element indicating the maximum transform size, and the actual maximum CU width and height of the SBT are derived as the minimum between the maximum transform size and the maximum CU width and height of the SBT signaled in the high-level syntax. For example, if the maximum transform size is 16, then regardless of whether slice_max_sbt_size_64_flag is signaled as 0 (allowing a maximum CU width and height of 32 for the SBT) or 1 (allowing a maximum CU width and height of 64 for the SBT), the maximum CU width and height of the SBT are 16. The proposed changes to the VVC specification text are described below:

[0099] A slice_sbt_max_size_64_flag value of 0 specifies that the maximum CU width and height allowed for sub-block transformations is 32. A slice_sbt_max_size_64_flag value of 1 specifies that the maximum CU width and height allowed for sub-block transformations is 64.

[0100] maxSbtSize = min(max_transform_size, slice_sbt_max_size_64_flag?64:32), where max_transform_size defines the maximum transformation size.

[0101] It is recommended to apply SBT only when the CU width and height are not greater than the maximum TU size.

[0102] Instead of indicating which transform unit has non-zero coefficients as in SBT, it is suggested that a signal be sent first to indicate which sub-part of the coding unit (e.g., using SBT partitions, i.e., left half, right half, upper half, lower half, left quarter, right quarter, upper quarter, or lower quarter) has at least one non-zero coefficient, and each part of the coding unit may include one or more TUs.

[0103] In one embodiment, if a sub-part size is larger than the maximum TU size, it is divided into multiple TUs, and each TU is no larger than the maximum TU size. For example, if the maximum TU size is 16 lengths and it is determined that only the right half (32×32) of the 64×32 CU is associated with non-zero coefficients, while the left half (32×32) is associated with only zero coefficients, then the right half is further divided into four 16×16 TUs, and each 16×16 TU can have non-zero coefficients. The 64×32 CU includes only the right half (texture sub-part) associated with non-zero coefficients, and the right half 32×32 sub-part is further divided into four 16×16 TUs.

[0104] In one embodiment, when the size of a subpart is greater than the maximum TU size, it is divided into multiple TUs, each of which is no larger than the maximum TU size. If all TUs of a subpart except the last TU in the encoding order are encoded with zero CBF, the CBF of the last TU of that subpart is not signaled, but rather the CBF of the last TU of that subpart is derived using a non-zero CBF value.

[0105] According to the embodiment, instead of using the quarter-partitioning method used in SBT (which divides the CU horizontally or vertically into 3:1 or 1:3 sub-partitions), it is suggested that when the CU width (W) is greater than the height (H), instead of using a vertically divided 3:1 or 1:3 partition, the CU should be divided into a left HxH and a right (WH)xH sub-partition, or a right HxH and a left (WH)xH sub-partition. When W is greater than 4*H, the suggested partitioning differs from 3:1 and 1:3 partitioning. Similarly, when the CU width (W) is less than the height (H), instead of using a horizontally divided 3:1 or 1:3 partition, the CU should be divided into an upper HxH and a lower (WH)xH sub-partition, or a lower HxH and an upper (WH)xH sub-partition. When H is greater than 4*W, the suggested partitioning differs from 3:1 and 1:3 partitioning.

[0106] In one example, instead of partitioning a 128×16 CU into a 96×16 zero TU and a 32×16 non-zero TU, it is suggested to partition it into a 112×16 zero TU and a 16×16 non-zero TU. In another example, instead of partitioning a 128×8 CU into a 96×8 zero TU and a 32×8 non-zero TU, it is suggested to partition it into a 120×8 zero TU and an 8×8 non-zero TU.

[0107] In one embodiment, if the CU size is W×H and a horizontal 1:3 or 3:1 partition is used, if 0.25W is greater than the maximum TU size (max_transform_size), then instead of dividing the CU into a 0.25W×H TU and a 0.75W×H TU, the CU is divided into a max_transform_size×H TU and a (W-max_transform_size)×H TU.

[0108] In one example, for a 128×8 CU, if the maximum TU size is 16, instead of partitioning it into a 96×8 zero TU and a 32×8 non-zero TU, it is recommended to partition it into a 112×8 zero TU and a 16×8 non-zero TU.

[0109] In one embodiment, if the CU size is W×H and a vertical 1:3 or 3:1 partitioning is used, if 0.25H is greater than the maximum TU size (max_transform_size), then instead of dividing the CU into a Wx0.25H TU and a Wx0.75H TU, the CU is divided into a max_transform_size×H TU and a (W-max_transform_size)×H TU.

[0110] Unlike the high-level syntax slice_max_sbt_size_64_flag in the current SBT design, which signals the maximum allowed CU width and height of the SBT, this design does not signal the maximum allowed transform size of the SBT.

[0111] In one embodiment, even if the current CU width or height exceeds the maximum CU width and height allowed for SBT, SBT partitioning is permitted and signaling is allowed as long as the width and height of the resulting non-zero sub-TU do not exceed the maximum TU width and height. For example, if the maximum TU size (width and height) is 32 points and the current CU is 64×32, then vertically partitioning the current CU into two 32×32 TUs, or a non-zero 16×32 TU and a zero 48×32 TU, is permitted. However, horizontal partitioning of the current CU is not permitted because the resulting transform width is 64, exceeding the maximum transform size.

[0112] The availability of SBT partitions and orientations can be based on whether the obtained non-zero sub-TUs satisfy the constraint of the maximum TU. If unavailable, the relevant flag is not signaled but inferred. In one example, the CU is 64×16, and the maximum transform size is 16. In this case, half-SBT partitioning is unavailable, so cu_sbt_quad_flag is not signaled but inferred to be true. Furthermore, horizontal SBT partitioning is also unavailable, so cu_sbt_horizontal_flag is not signaled but inferred to be false.

[0113] It is recommended that no signal be sent for any SBT partitions that result in a non-zero TU with a height or width greater than the maximum TU size.

[0114] Currently, ISP mode is allowed for all CU sizes. However, when the maximum transform size is set to less than 64, a conflict arises regarding whether to perform implicit transform partitioning or use explicit transform partitioning with signaling ISP to resolve the issue. It is recommended to limit the maximum CU size allowed for ISP so that ISP is only applied to CUs without implicit transform partitioning.

[0115] In one embodiment, the changes to the syntax table are as follows.

[0116]

[0117] When the size of the CU is greater than the maximum size of the TU, after dividing the CU into multiple TUs using implicit transformation partitioning, an ISP can be further applied to each TU (of size w×h) to indicate whether the TU should be further divided into multiple smaller TUs of 0.5w×h, or w×0.5h, or 0.25w×h, or w×0.25h.

[0118] In one example, if the CU size is 64×16 and the maximum TU size is 16, the CU is first implicitly divided into four 16×16 TUs. Then, for each 16×16 TU, the ISP is applied to signal whether it should be further divided into four 4×16 or four smaller 16×4 TUs.

[0119] It is recommended to send separate signals to notify the maximum change size for each color component.

[0120] In one embodiment, a maximum transformation size is signaled for the luminance component and a maximum transformation size is signaled for the chrominance component.

[0121] It is recommended to signal only the maximum transformation size of one color component, while the maximum transformation sizes of other color components are implicitly derived.

[0122] In one embodiment, a maximum transform size is signaled for a first component, and for another component, the maximum horizontal and / or vertical transform sizes are adjusted accordingly based on the downsampling rate of the first component.

[0123] In one example, if the sample of the current color component is downsampled by N relative to the first color component in the horizontal (and / or vertical) direction, that is, the first color component has N times the number of samples relative to the current sample along the horizontal (and / or vertical) axis, then the maximum horizontal (and / or vertical) transformation size applied to the current color component is the maximum transformation size divided by N.

[0124] In one embodiment, a maximum transform size is signaled for the luminance component, and if it is a YUV 444 format, the maximum transform size applied to the chrominance component is set to the same as the maximum transform size applied to the luminance component.

[0125] In one embodiment, a maximum transform size is signaled for the luminance component. If it is a YUV 422 format, the maximum horizontal transform size applied to the chrominance component is set to half the maximum transform size applied to the luminance component, and the maximum vertical transform size applied to the chrominance component is set to the same as the maximum transform size applied to the luminance component.

[0126] It is recommended to apply different zero-out transformation schemes for different color components.

[0127] It is recommended to apply different transform-to-zero schemes only when the downsampling rates of different color components are different.

[0128] although Figure 1 An example block diagram of process 100 is shown, but in some implementations, process 100 may include more than Figure 1 The number of boxes, fewer boxes, different boxes, or different arrangements of boxes depicted in the process. Alternatively, two or more boxes in process 100 may be executed in parallel.

[0129] Figure 2A simplified block diagram of a communication system (200) according to an embodiment of the present disclosure is shown. The communication system (200) may include at least two terminal devices (210, 220) interconnected via a network (250). For unidirectional data transmission, a first terminal device (210) may encode video data at a local location for transmission over the network (250) to another terminal device (220). A second terminal device (220) may receive encoded video data from the other terminal device via the network (250), decode the encoded video data, and display the recovered video data. Unidirectional data transmission is common in applications such as media services.

[0130] Figure 2 A second pair of terminal devices (230, 240) supporting bidirectional transmission of encoded video is illustrated, which may occur, for example, during a video conference. For bidirectional data transmission, each terminal device (230, 240) may encode video data acquired at a local location for transmission over a network (250) to the other terminal device. Each terminal device (230, 240) may also receive encoded video data transmitted by the other terminal device and may decode the encoded video data and display the recovered video data on a local display device.

[0131] exist Figure 2 In this disclosure, the terminal devices (210-240) may be servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (250) refers to any number of networks that transmit encoded video data between the terminal devices (210-240), including, for example, wired and / or wireless communication networks. The communication network (250) may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, unless explained below, the architecture and topology of the network (250) may be irrelevant to the operation of this disclosure.

[0132] As an application example of the disclosed subject matter Figure 3 The placement of the video decoder and encoder in a streaming environment is illustrated. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0133] The streaming system may include an acquisition subsystem (313) that may include a video source (301) such as a digital camera, which creates, for example, an uncompressed video sample stream (302). The sample stream (302) is depicted as a thick line compared to the encoded video bitstream to emphasize that it is a high-data-volume video sample stream, and the sample stream (302) may be processed by an encoder (303) coupled to the camera (301). The encoder (303) may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video bitstream (304) is depicted as a thin line compared to the sample stream (302) to emphasize the lower-data-volume encoded video bitstream, which may be stored on a streaming server (305) for future use. One or more streaming clients (306, 308) may access the streaming server (305) to retrieve a copy (307, 309) of the encoded video bitstream (304). The client (306) may include a video decoder (310). The video decoder (310) decodes an incoming copy (307) of the encoded video bitstream and produces an output video sample stream (311) that can be displayed on a display (312) or another presentation device (not shown). In some streaming systems, the video bitstream (304, 307, 309) may be encoded according to certain video codec / compression standards. Examples of these standards include ITU-T Recommendation H.265. The video codec standards under development are informally referred to as Multi-Functional Video Codec (VVC). The topics disclosed can be used in the context of VVC.

[0134] Figure 4 This is a functional block diagram of a video decoder (310) according to an embodiment of the present invention.

[0135] The receiver (410) may receive one or more encoded video sequences to be decoded by the video decoder (310); in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences may be received from a channel (412), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (410) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver (410) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (410) and the entropy decoder / parser (420) (hereinafter referred to as the "parser"). The buffer memory (415) may not be required when the receiver (410) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, or the buffer memory may be made smaller. Of course, for use on packet networks such as the Internet, a buffer memory (415) may also be required, which may be relatively large and may advantageously have an adaptive size.

[0136] The video decoder (310) may include a parser (420) to reconstruct symbols (421) from an entropy-coded video sequence. These symbols may include information for managing the operation of the video decoder (310), and potential information for controlling a display device such as a display 312, which is not part of the decoder but may be coupled to it. Figure 4As shown in the diagram. The control information for the display device may be in the form of a parameter set fragment (not shown) of Supplementary Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (420) may parse / decode the received encoded video sequence. The encoding and decoding of the encoded video sequence may be performed according to video encoding and decoding technologies or standards, and may follow principles known to those skilled in the art, including variable-length encoding and decoding, Huffman coding, arithmetic encoding and decoding with or without context sensitivity, etc. The parser (420) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup of pixels in the encoded video sequence for use in the video decoder based on at least one parameter corresponding to a group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The entropy decoder / parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter (QP) values, motion vectors, and so on.

[0137] The parser (420) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415) to create symbols (421). The parser (420) can receive encoded data and selectively decode specific symbols (421). In addition, the parser (420) can determine whether to provide specific symbols (421) to the motion compensation prediction unit (453), the scaler / inverse transform unit (451), the intra-frame prediction unit (452), or the loop filter (456).

[0138] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (421) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed by the parser (420) from the encoded video sequence. For brevity, the flow of such subgroup control information between the parser (420) and the various units described below is not described.

[0139] In addition to the functional blocks already mentioned, the video decoder (310) can be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0140] The first unit is the scaler / inverse transform unit (451). The scaler / inverse transform unit (451) receives quantization transform coefficients as symbols (421) and control information from the parser (420), including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (451) can output a block containing sample values, which can be input into the aggregator (455).

[0141] In some cases, the output samples of the scaler / inverse transform unit (451) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) uses surrounding reconstructed information extracted from the (partially reconstructed) current image (454) to generate blocks of the same size and shape as the blocks being reconstructed. In some cases, the aggregator (455) adds the predictive information generated by the intra-picture prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451) based on each sample.

[0142] In other cases, the output samples of the scaler / inverse transform unit (451) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (453) can access the reference image memory (457) to extract samples for prediction. After motion compensation is performed on the extracted samples according to the symbols (421), these samples can be added by the aggregator (455) to the output of the scaler / inverse transform unit (referred to in this case as residual samples or residual signals) to generate output sample information. The motion compensation prediction unit's retrieval of prediction samples from the address in the reference image memory can be controlled by motion vectors, and the motion vectors are available to the motion compensation prediction unit (453) in the form of the symbols (421), which, for example, include X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (457) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0143] The output samples of the aggregator (455) can be employed by various loop filtering techniques in the loop filter unit (456). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream, and these parameters can be used as symbols (421) from the parser (420) in the loop filter unit (456). However, video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0144] The output of the loop filter unit (456) can be a sample stream, which can be output to a display device (312) and stored in a reference image memory (457) for subsequent inter-frame image prediction.

[0145] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. Once the encoded images have been fully reconstructed and the encoded images (by, for example, a parser (420)) are identified as reference images, the current image (454) can become part of the reference image memory (457), and a new current image memory can be reallocated before the reconstruction of subsequent encoded images begins.

[0146] The video decoder (310) can perform decoding operations according to a predetermined video compression technique, such as that documented in the ITU-T H.265 standard. The encoded video sequence is considered to conform to the syntax specified by the video compression technique or standard used, in the sense that it follows the syntax of the video compression technique or standard as defined in the video compression technique literature or standard, particularly in the configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further defined by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0147] In this embodiment, the receiver (410) may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by the video decoder (310) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0148] Figure 5 This is a functional block diagram of a video encoder (303) according to an embodiment of the present disclosure.

[0149] The video encoder (303) can receive video samples from a video source (301) (not part of the encoder), which can capture video images to be encoded by the video encoder (303).

[0150] A video source (301) can provide a sequence of source videos encoded by a video encoder (303) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (301) can be a storage device storing previously prepared video. In a video conferencing system, the video source (301) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art can readily understand the relationship between pixels and samples. The following focuses on describing samples.

[0151] According to an embodiment, the video encoder (303) can encode and compress images of a source video sequence into an encoded video sequence (543) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (550). The controller controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller may include rate control related parameters (e.g., image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of the controller (550) can be readily identified by those skilled in the art, as they may belong to the video encoder (303) optimized for a particular system design.

[0152] Some video encoders operate in a manner readily recognizable to those skilled in the art as an "encoding loop." In a simplified description, an encoding loop may include an encoding portion of an encoder (530) (hereinafter referred to as the "source coder," which is responsible for creating symbols based on the input image to be encoded and the reference image) and a (local) decoder (533) embedded within the video encoder (303). The "local" decoder (533) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates the sample data (since any compression between the symbols and the encoded video stream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream is input to a reference image memory (534). Since the decoding of the symbol stream produces bit-precise results independent of the decoder's location (local or remote), the contents of the reference image memory also correspond bit-precisely between the local and remote encoders. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" during the prediction process. The basic principles of this reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) are well known to those skilled in the art.

[0153] The operation of the "local" decoder (533) can be combined with the above. Figure 4 The "remote" decoder (310) described in detail is the same. However, another brief reference is provided. Figure 5 When symbols are available and the entropy encoder (545) and parser (420) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding part of the video decoder (310) (including the channel (412), receiver (410), buffer memory (415) and parser (420)) may not be fully implemented in the local decoder (533).

[0154] It can then be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also exist in the corresponding encoder in essentially the same functional form. Descriptions of encoder techniques can be simplified, as they are inverses of fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0155] As part of the operation, the source encoder (530) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence designated as "reference frames." In this manner, the encoding engine (532) encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which may be selected as a prediction reference for the input frame.

[0156] The local video decoder (533) can decode encoded video data of a frame that can be designated as a reference frame, based on symbols created by the source encoder (530). The operation of the encoding engine (532) can advantageously be a lossy process. When the encoded video data can be decoded by the video decoder (533), Figure 5 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (533) replicates the decoding process, which can be performed by the video decoder on the reference frame, and allows the reconstructed reference frame to be stored in a reference image memory (534). In this way, the encoder (303) can locally store a copy of the reconstructed reference frame that shares the same content (no transmission errors) as the reconstructed reference frame to be obtained by the remote video decoder.

[0157] The predictor (535) can perform a prediction search against the encoding engine (532). That is, for a new frame to be encoded, the predictor (535) can search in the reference image memory (534) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (535) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (535), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (534).

[0158] The controller (550) can manage the encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters for encoding video data.

[0159] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (545). The entropy encoder can perform lossless compression on the symbols generated by the various functional units according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0160] The transmitter (540) can buffer the encoded video sequence created by the entropy encoder (545) in preparation for transmission via a communication channel (560), which may be a hardware / software link to a storage device storing the encoded video data. The transmitter (540) can combine the encoded video data from the source encoder (530) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0161] The controller (550) manages the operation of the video encoder (303). During encoding, the controller (550) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types.

[0162] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs support different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of the variations of I-pictures and their respective applications and characteristics.

[0163] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0164] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0165] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction.

[0166] The video encoder (303) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (303) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0167] In this embodiment, the transmitter (540) may transmit additional data and encoded video. The source encoder (30) may include such data as a portion of an encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR layers, redundant images and slices, SEI messages, VUI parameter set fragments, etc.

[0168] Furthermore, the proposed methods can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium to perform one or more of the proposed methods.

[0169] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 6 A computer system 600 is shown, which is adapted to implement certain embodiments of the disclosed subject matter.

[0170] The computer software can be encoded using any suitable machine code or computer language, and code including instructions can be created through mechanisms such as assembly, compilation, and linking. These instructions can be executed directly by the computer's central processing unit (CPU), graphics processing unit (GPU), or through decoding, microcode, or other means.

[0171] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0172] Figure 6 The components shown for computer system 600 are exemplary in nature and are not intended to limit the scope or functionality of computer software implementing embodiments of this disclosure. Nor should the configuration of the components be construed as having any dependency or requirement on any component or combination thereof shown in the exemplary embodiments of computer system 600.

[0173] Computer system 600 may include certain human-machine interface (HMI) input devices. These HMI input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The HMI device may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0174] Human-machine interface input devices may include one or more of the following (only one is shown): keyboard 601, mouse 602, touchpad 603, touch screen 610, data glove, joystick 605, microphone 606, scanner 607, and camera 608.

[0175] Computer system 600 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. These HMI output devices may include tactile output devices (e.g., tactile feedback via touchscreen 610, data gloves, or joystick 605, but may also include tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers 609, headphones (not shown)), visual output devices (e.g., screen 610 including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light-emitting diode (OLED) screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which may output two-dimensional or higher-dimensional visual outputs through means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0176] The computer system 600 may also include human-accessible storage devices and related media, such as optical media including high-density read-only / rewritable optical discs (CD / DVD ROM / RW) 620 or similar media 621, thumb drives 622, removable hard disk drives or solid-state drives 623, conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated devices based on ROM / ASIC / PLD such as security software protectors (not shown), and so on.

[0177] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0178] Computer system 600 may also include one or more interfaces to one or more communication networks. For example, the network may be wireless, wired, or optical. The network may also be a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), vehicular and industrial network, real-time network, latency-tolerant network, etc. Networks also include Ethernet, wireless LANs, cellular networks (Global System for Mobile Communications (GSM), 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicular and industrial networks (including CANBus), etc. Some networks typically require external network interface adapters for connection to certain general-purpose data ports or peripheral buses 649 (e.g., the Universal Serial Bus (USB) port of computer system 600); other systems are typically integrated into the core of computer system 600 via connection to system buses as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks, computer system 600 can communicate with other entities. The communication can be unidirectional, used only for receiving (e.g., wireless television), unidirectional, used only for sending (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as through a local area or wide area digital network to other computer systems. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

[0179] The aforementioned human-machine interface device, human-accessible storage device, and network interface can be connected to the core 640 of the computer system 600.

[0180] Core 640 may include one or more central processing units (CPUs) 641, graphics processing units (GPUs) 642, dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) 643, task-specific hardware accelerators 644, etc. These devices, along with read-only memory (ROM) 645, random access memory (RAM) 646, internal mass storage (e.g., internal non-user-accessible hard disk drives, solid-state drives (SSDs), etc.) 647, can be connected via system bus 648. In some computer systems, system bus 648 may be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus 648 or connected via peripheral bus 649. Peripheral bus architectures include peripheral component interconnect (PCI), USB, etc.

[0181] The CPU 641, GPU 642, FPGA 643, and accelerator 644 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 645 or RAM 646. Transient data can also be stored in RAM 646, while permanent data can be stored, for example, in internal mass storage 647. Fast storage and retrieval of any memory device can be achieved through the use of a cache memory, which can be closely associated with one or more CPUs 641, GPUs 642, mass storage 647, ROM 645, RAM 646, etc.

[0182] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of this disclosure, or they may be media and code well-known and usable by those skilled in the art of computer software.

[0183] By way of example and not limitation, a computer system having architecture 600, particularly core 640, can provide functionality as a processor (including CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of core 640 that is non-volatile, such as core-internal mass storage 647 or ROM 645. Software implementing various embodiments of this disclosure can be stored in such a device and executed by core 640. Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause core 640, particularly the processor therein (including CPU, GPU, FPGA, etc.), to execute a specific process or a specific portion of a specific process described herein, including defining data structures stored in RAM 646 and modifying such data structures according to a software-defined process. Alternatively or as an alternative, the computer system can provide functionality that is logically hardwired or otherwise included in circuitry (e.g., accelerator 644), which can replace or operate with the software to execute the specific process or a specific portion of a specific process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing the software (such as an integrated circuit (IC)), circuitry containing the logic to perform the execution, or both. This disclosure includes any suitable combination of hardware and software.

[0184] While this disclosure has described several exemplary embodiments, various modifications, arrangements, and equivalent substitutions of the embodiments are within the scope of this disclosure. Therefore, it should be understood that those skilled in the art can design various systems and methods that, while not explicitly shown or described herein, embody the principles of this disclosure and are thus within its spirit and scope.

[0185] Acronyms:

[0186] HEVC: High Efficiency Video Coding

[0187] HDR: High Dynamic Range

[0188] SDR: Standard Dynamic Range

[0189] VVC: Versatile Video Coding

[0190] JVET: Joint Video Exploration Team

[0191] MPM: Most Probable Mode

[0192] WAIP: Wide-Angle Intra Prediction

[0193] CU: Coding Unit

[0194] PU: Prediction Unit

[0195] ISP: Intra Sub-Partitions

[0196] SBT: Sub-block transform

[0197] CBF: Coded block flag

Claims

1. A method for video decoding, characterized in that, The method includes: Based on the first high-level syntax element of the video sequence, determine the maximum transform size associated with the video sequence, the maximum transform size corresponding to the maximum transform unit region; When the Subblock Transform (SBT) mode is enabled, the maximum block size of the SBT mode is allowed to be constrained by the maximum transform size; Wherein, the maximum block size allowed for the SBT mode is constrained by the maximum transform size, including: The width and height of the maximum CU allowed for the SBT mode are derived as the minimum between the maximum transform size indicated by the first high-level syntax element and the width and height of the maximum CU allowed for the SBT mode indicated in the second high-level syntax element.

2. The method according to claim 1, characterized in that, The advanced syntax elements are Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Slice Header, Patch Header, Patch Group Header, or Code Tree Unit (CTU) Header.

3. The method according to claim 1, characterized in that, The maximum transformation size corresponds to the maximum transformation width and height.

4. The method according to any one of claims 1-3, characterized in that, When the intra-fractional sub-partition (ISP) mode is enabled, the maximum coding unit size is constrained by the maximum transform size.

5. The method according to claim 4, characterized in that, The ISP mode is only applied to coding units that do not have implicit transform splitting.

6. The method according to claim 3, characterized in that, The SBT mode is applied only when the maximum transform width and height are not greater than the maximum transform unit TU.

7. A video encoding method, characterized in that, The method includes: Determine the maximum transform size associated with the video sequence, the maximum transform size corresponding to the maximum transform unit region; The video sequence is encoded using the maximum transform size; When the Subblock Transform (SBT) mode is enabled, the maximum block size of the SBT mode is allowed to be constrained by the maximum transform size; Wherein, the maximum block size allowed for the SBT mode is constrained by the maximum transform size, including: The width and height of the maximum CU allowed in the SBT mode are derived as the minimum between the maximum transform size indicated by the first high-level syntax element and the width and height of the maximum CU allowed in the SBT mode indicated by the second high-level syntax element.

8. A video encoding / decoding device, characterized in that, include: At least one memory is configured to store program code; At least one processor is configured to read the program code and execute the method as described in any one of claims 1 to 7, as instructed by the program code.

9. A video decoding apparatus, characterized in that, include: The determination module is configured to determine, based on a first high-level syntax element of the video sequence, the maximum transform size associated with the video sequence, the maximum transform size corresponding to a maximum transform unit region; The constraint module is configured to allow the maximum block size of the SBT mode to be constrained by the maximum transform size when the sub-block transform SBT mode is enabled; The constraint module is further configured as follows: The width and height of the maximum CU allowed for the SBT mode are derived as the minimum between the maximum transform size indicated by the first high-level syntax element and the width and height of the maximum CU allowed for the SBT mode indicated in the second high-level syntax element.

10. A video encoding apparatus, characterized in that, include: The determination module is configured to determine the maximum transform size associated with the video sequence, the maximum transform size corresponding to the maximum transform unit region; An encoding module is configured to encode the video sequence using the maximum transform size; The constraint module is configured to allow the maximum block size of the SBT mode to be constrained by the maximum transform size when the sub-block transform SBT mode is enabled; The constraint module is further configured as follows: The width and height of the maximum CU allowed in the SBT mode are derived as the minimum between the maximum transform size indicated by the first high-level syntax element and the width and height of the maximum CU allowed in the SBT mode indicated by the second high-level syntax element.

11. A non-volatile computer-readable medium, characterized in that, For storing instructions and bitstreams consisting of video sequences, the instructions including one or more instructions that, when executed by one or more processors of a device that performs encoding or decoding of the video sequences, cause the one or more processors to perform the method as described in any one of claims 1 to 7.

Images