Video encoding method and corresponding video encoding apparatus

Abstract

The present application provides a video encoding method and related device. The video encoding method includes receiving input data; determining a block partition structure of a current block, dividing the current block into one or more coding blocks, wherein each PE group has a plurality of PEs performing RDO tasks in parallel, and each PE group is associated with a specific block size; testing a plurality of coding modes for each partition of the current block and a corresponding sub-partition divided from each partition by parallel PEs of each PE group; determining the block partition structure of the current block and a corresponding coding mode of each coding block in the current block according to a rate-distortion cost related to the coding modes tested by the PE groups; and entropy encoding one or more coding blocks in the current block according to the corresponding coding modes determined by the PE groups. The video encoding method and related device of the present application can save bandwidth.

Description

[Technical Field]

[0001] This invention relates to hierarchical architecture in video encoders. In particular, it relates to rate-distortion optimization for determining the block partition structure and corresponding coding mode in video coding. [Background Technology]

[0002] Versatile Video Coding (VVC) is the latest video decoding standard developed by the Joint Collaborative Video Coding Group (JCT-VC) of video decoding experts from the ITU-T Study Group. The VVC standard relies on a block-based decoding structure that divides each image into multiple Code Tree Units (CTUs). A CTU consists of an NxN luma sample block and one or more corresponding chroma sample blocks. For example, each 4:2:0 chroma subsample CTU consists of a 128x128 luma code tree block (CTB) and two 64x64 chroma CTBs. Each CTB in a CTU is further recursively divided into one or more code blocks (CBs) within a Code Unit (CU) for encoding or decoding to accommodate various local characteristics. Compared to the quadtree (QT) structure used in the High Efficiency Video Decoding (HEVC) standard, flexible CU structures such as the Quad-Tree-Binary-Tree (QTBT) structure can improve coding performance. Figure 1 An example of splitting the CTB using a QTBT structure is shown, where the CTB is adaptively partitioned using a quadtree structure, and then each quadtree leaf node is adaptively partitioned using a binary tree structure. The binary tree leaf nodes are represented as CBs for prediction and transformation, without further partitioning. In addition to binary tree partitioning, ternary tree partitioning can be chosen after quadtree partitioning to capture objects centered on the quadtree leaf nodes. A horizontal ternary tree partition divides a quadtree leaf node into three partitions: the top and bottom partitions are each a quarter the size of the quadtree leaf node, and the middle partition is half the size of the quadtree leaf node. A vertical ternary tree partition divides a quadtree leaf node into three partitions: the left and right partitions are each a quarter the size of the quadtree leaf node, and the middle partition is half the size of the quadtree leaf node. In this flexible structure, the CTB is first partitioned using a quadtree structure, and then the quadtree leaf nodes are further partitioned into subtree structures containing binary and ternary partitions. The leaf nodes of the subtrees are represented as CBs.

[0003] Prediction decisions in video encoding or decoding are made at the CU level, where each CU is decoded by a selected decoding mode or a combination thereof. After obtaining the residual signal generated by the prediction process, the residual signal belonging to the CU is further transformed into transform coefficients for compact data representation. These transform coefficients are quantized and sent to the decoder.

[0004] Traditional video encoders used to encode video images into bitstreams, such as Figure 2 As shown. The encoding process of a traditional video encoder can be divided into four stages: preprocessing stage 22, integer motion estimation (IME) stage 24, rate-distortion optimization (RDO) stage 26, and loop filtering and entropy coding stage 28. In RDO stage 26, a single processing element (PE) is used to search for the optimal decoding module for encoding the target NxN block within the CTU. PE is a general term referring to a hardware element that executes a stream of instructions to perform arithmetic and logical operations on data. The PE executes scheduled RDO tasks to encode the target NxN block. The scheduling of PEs is called PE thread, which shows the RDO tasks assigned to the PE in multiple PE calls. The term PE call or PE run refers to a fixed time interval during which the PE executes one or more tasks. For example, the first PE thread, containing M+1 PE calls, is dedicated to calculating the rate and distortion cost of encoding an 8x8 block using multiple decoding modes. Similarly, the second PE thread, also containing M+1 PE calls, is dedicated to calculating the rate and distortion cost of encoding a 16x16 block using multiple decoding modes. Within each PE thread, the PE sequentially tests various decoding modes to select the optimal mode for the block partition corresponding to the allocated block size. The VVC standard supports more video decoding tools, thus requiring more decoding mode testing in each PE thread, resulting in a longer chain of PE threads in RDO stage 26. Consequently, making the optimal decoding mode decision requires longer latency, and the video encoder throughput becomes lower. The following is a brief introduction to several decoding tools introduced in the VVC standard.

[0005] Merge mode with MVD (MMVD) for CUs encoded by the merge mode implicitly derives motion information directly for predictive sample generation. The MMVD merge mode introduced in the VVC standard further refines the selected merge candidates by transmitting Motion Vector Difference (MVD) information. The MMVD flag is transmitted immediately after the regular merge flag to specify whether the MMVD mode is used for the CU. The MMVD information transmitted in the bitstream includes the MMVD candidate flag, an index specifying the motion amplitude, and an index indicating the motion direction. In the MMVD mode, one of the top two candidates in the Merge list is selected as the MV basis. The MMVD candidate flag is transmitted to specify which of the top two merge candidates to use. The distance index specifies the motion amplitude information and indicates a predefined offset from the starting point. The offset is added to the horizontal or vertical component of the starting MV. The relationship between the distance index and the predefined offset is shown in Table 1.

[0006] Table 1 – Relationship between Distance Index and Predefined Offset

[0007] Distance Index 0 1 2 3 4 5 6 7 Offset (in brightness samples) 1 / 4 1 / 2 1 2 4 8 16 32

[0008] The direction index indicates the direction of the MVD relative to the starting point. The direction index represents one of four directions, both horizontal and vertical. It's important to note that the meaning of the MVD sign can vary depending on the information of the starting MV. For example, when the starting MV(s) is a uni-prediction MV or a bi-prediction MV, and both lists point to the same direction as the current image, the signs shown in Table 2 specify the sign of the MV offset added to the starting MV. If the Picture Order Count (POC) of both reference images is greater than the POC of the current image, or if the POCs of both reference images are less than the POC of the current image, then both lists point to the same direction as the current image. When the starting MV is a bi-prediction MV, the two MVs point to different directions as the current image, and the difference in POC in list 0 is greater than the difference in POC in list 1, the signs in Table 2 specify that the sign of the MV offset added to the list 0 MV component of the starting MV has the opposite sign to the sign of the list 1 MV. Otherwise, when the offset of the POC in List 1 is greater than the offset of the POC in List 0, the signs in Table 2 specify that the signs of the MV offsets of the List 1 MV component added to the starting MV are opposite to the signs of the List 0 MV. MVD is scaled based on the offset of the POC in each direction. If the offsets of the POCs in the two lists are the same, no scaling is required; otherwise, if the offset of the POC in List 0 is greater than the offset in List 1, the MVD of List 1 is scaled using the POC offsets of List 0 and List 1. If the POC offset of List 1 is greater than that of List 0, the MVD of List 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MV.

[0009] Table 2 – Symbols of MV Offsets Specifyed by Direction Index

[0010] Directional IDX 00 01 10 11 X-axis + - N / A N / A y-axis N / A N / A + -

[0011] Bi-prediction with CU-level weight (BCW) generates a bi-prediction signal by averaging two prediction signals obtained from two different reference images and / or using two different motion vectors from the HEVC standard. In the VVC standard, the bi-prediction mode is extended beyond simple averaging to allow for a weighted average of the two prediction signals.

[0012] P bi-pred =((8-w)*P0+w*P1+4)>>3

[0013] In the VVC standard, weighted average bidirectional prediction allows five weights w∈{-2,3,4,5,10}. In each bidirectional prediction CU, the weights w are determined in one of two ways: 1) for non-merge CUs, the weight index is sent after the motion vector difference; 2) for merge CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applicable to CUs with 256 or more luminance samples, meaning the CU width multiplied by the CU height must be greater than or equal to 256. For low-latency images, all five weights are used. For non-low-latency images, only three weights w∈{3,4,5} are used.

[0014] A fast search algorithm is applied to find weight indices without significantly increasing the encoder complexity of the video encoder. When used in conjunction with Adaptive Motion Vector Resolution (AMVR), unequal weights are conditionally checked only for 1-pixel and 4-pixel motion vector precision if the current image is a low-latency image. When BCM is combined with affine mode, unequal weights are only performed on unequal weights if the affine mode is selected as the current best mode. Unequal weights are conditionally checked only if the two reference images in bidirectional prediction are identical. Unequal weights are not searched when certain conditions are met, depending on the POC distance between the current image and its reference images, the coding QP, and the temporal level.

[0015] The BCW weight index uses a bin of context-decoded bits, followed by a bin of bypass-coded bits. The first bin of context-decoded bits indicates whether equal weights are used; if unequal weights are used, the bypass-coded bits are used to indicate which unequal weights were applied. Weighted Prediction (WP) is a decoding tool supported by the H.264 / AVC and HEVC standards that efficiently decodes video content (with attenuation). Support for WP has also been added to the VVC standard. WP allows signaling weighted parameters (weights and offsets) for each reference picture in the L0 and L1 lists. The weights and offsets of the corresponding reference pictures are applied during motion compensation. WP and BCW are designed for different types of video content. To avoid interaction between WP and BCW (which would complicate the VVC decoder design), if the CU uses WP, the BCW weight index is not signaled, and w is inferred to be 4, meaning equal weights are applied. For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. This can be applied to both normal merge mode and inherited affine merge mode. For constructed affine merge mode, affine motion information is constructed based on motion information from up to 3 blocks. The BCW index of a CU using constructed affine merge mode is simply set to be equal to the BCW index of the first control point MV. In the VVC standard, Combined Inter and Intra Prediction (CIIP) and BCW cannot be jointly applied to a CU. When a CU decodes using CIIP mode, the CBW index of the current CU is set to 4, meaning that equal weights are applied.

[0016] Multiple Transform Selection (MTS) for core transforms, in addition to the DCT-II transform already used in the HEVC standard, is used for residual coding of inter-frame and intra-frame decoded blocks. It provides the flexibility to select transform decoding settings from multiple transforms (e.g., DCT-II, DCT-VIII, and DST-VII). The newly introduced transform matrices are DST-VII and DCT-VIII. Table 3 shows the basic functions of the DST and DCT transforms.

[0017] Table 3 - Transform basis functions for DCT-II / VIII and DSTVII for N-point input.

[0018]

[0019] To maintain the orthogonality of the transform matrix, the quantization of the transform matrix is ​​more precise than that in the HEVC standard. To keep the intermediate values ​​of the transformed coefficients within the 16-bit range, all coefficients after horizontal and vertical transforms are 10-bit coefficients. To control the MTS scheme, separate enable flags are specified for intra-frame and inter-frame prediction at the Sequence Parameter Set (SPS) level. When MTS is enabled at the SPS, a CU-level flag is sent to indicate whether MTS is applied. MTS applies only to the luma component. MTS signaling is skipped when one of the following conditions is met: the last significant coefficient of the luma transform block (TB) is less than 1 (i.e., DC only); or the last significant coefficient of the luma TB is located within the MTS zero-out region.

[0020] If the MTS CU flag is equal to 0, DCT-II is applied in both directions. However, if the MTS CU flag is equal to 1, two additional flags are sent to indicate the transform type in the horizontal and vertical directions, respectively. The transform and flag signaling mapping table is shown in Table 4. Transform selection for intra-ISP and implicit MTS is unified by removing intra-mode and block-shape dependency. If the current block is encoded in ISP mode, or if the current block is an intra-block and both intra- and inter-frame display MTS are enabled, only DST-VII is used for the horizontal and vertical transform cores. Regarding transform matrix precision, an 8-bit primary transform core is used. Therefore, all transform cores used in the HEVC standard remain unchanged, including 4-point DCT-II and DST-VII, 8-point, 16-point, and 32-point DCT-II. Furthermore, other transform cores, including 64-point DCT-II, 4-point DCT8, 8-point, 16-point, 32-point DST-VII, and DCT-VIII, use the 8-bit primary transform core.

[0021] Table 4 - Conversion and Flag Signaling Mapping Table

[0022]

[0023] To reduce the complexity of large-sized DST-VII and DCT-VIII blocks, the high-frequency transform coefficients are zeroed out for DST-VII and DCT-VIII blocks with a size (width or height, or width and height) equal to 32. Only the coefficients in the 16x16 low-frequency region are retained.

[0024] Similar to the HEVC standard, the transform skip mode can be used to decode the block residuals. To avoid redundancy in syntax decoding, the transform skip flag is not sent when the CU-level MTS CU flag is not equal to 0. Note that when the Low-Frequency Non-Separable Transform (LFNST) or Matrix-based Intra Prediction (MIP) is activated for the current CU, the implicit MTS transform is set to DCT-II. Furthermore, implicit MTS can still be enabled when MTS is enabled for inter-frame decoding blocks.

[0025] Geometric Partitioning Mode (GPM) is supported in the VVC standard for inter-frame prediction. GPM uses CU-level flags as a merging mode for signal transmission. Other merging modes include regular merging mode, MMVD mode, CCIP mode, and sub-block merging mode. For each possible CU size w×h = 2 m ×2 n (where m, n ∈ {3…6}), GPM supports a total of 64 partitions, excluding 8x64 and 64x8. When using this mode, a CU is divided into two parts by a geometrically positioned straight line, such as... Figure 3 As shown. Figure 3 An example of GPM partitioning is shown. The position of the dividing line is mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU uses its own motion for inter-frame prediction; each partition only allows unidirectional prediction, i.e., each part has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that only two motion-compensated predictors are computed for each CU, which is the same as traditional bidirectional prediction.

[0026] If the geometric partitioning pattern is used for the current CU, the geometric partition index indicating the partitioning pattern (angle and offset) of the geometric partition, along with two merge indices (one per partition), are further signaled. The number of maximum GPM candidate sizes is explicitly represented in the SPS, and syntax binarization of the GPM merge index is specified. After predicting each part of the geometric partition, sample values ​​along the edges of the geometric partition are adjusted through an adaptive weighted mixing process to obtain the predicted signal for the entire CU. As with other prediction patterns, the transformation and quantization processes are applied to the entire CU. Finally, the motion field of the CU predicted using the geometric partitioning pattern is stored.

[0027] The unidirectional prediction candidate list is directly derived from the merge candidate list constructed based on the extended merge prediction process. Let n denote the index of the unidirectional predicted motion in the geometric unidirectional prediction candidate list. The LX motion vector of the nth extended merge candidate (where X equals the parity of n) is used as the nth unidirectional predicted motion vector for the geometric segmentation pattern. For example, the unidirectional predicted motion vector for merge index 0 is L0 MV, the unidirectional predicted motion vector for merge index 1 is L1 MV, the unidirectional predicted motion vector for merge index 2 is L0 MV, and the unidirectional predicted motion vector for merge index 3 is L1 MV. In the absence of a corresponding LX motion vector for the nth extended merge candidate, the L(1-X) motion vector of the same candidate is used as the unidirectional predicted motion vector for the geometric segmentation pattern.

[0028] After using its own motion prediction geometry for each part, a blend is applied to the two predicted signals to derive samples around the geometry partition edges. The blending weights at each location of the CU are derived based on the distance between each location and the partition edge.

[0029] The distance between the position (x, y) and the partition edge is derived as follows:

[0030]

[0031]

[0032]

[0033]

[0034] Where i, j are the indices of the angle and offset of the geometric partition, which depend on the partition index of the sending set. ρ x,j and ρ y,j The sign depends on the angle index i.

[0035] The weights of each part of the geometric partition are derived as follows:

[0036] wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y)

[0037]

[0038] w1(x,y)=1-w0(x,y)

[0039] partIdx depends on the angle index i.

[0040] The motion vectors Mv1 from the first part of the geometric partition, Mv2 from the second part of the geometric partition, and the combined motion vectors of Mv1 and Mv2 are stored in the motion field of the CU of the geometric partition pattern decoder. The type of motion vector stored at each individual location in the motion field is determined as follows:

[0041] sType=abs(motionIdx)<32?2: (motionIdx≤0?(1-partIdx):partIdx)

[0042] Where motionIdx equals d(4x+2, 4y+2), it is recalculated according to the above equation. partIdx depends on the angle index i. If sType equals 0 or 1, Mv0 or Mv1 is stored in the corresponding motion field; otherwise, if sType equals 2, a combined motion vector from Mv0 and Mv2 is stored. The combined motion vector is generated using the following procedure: if Mv1 and Mv2 come from different lists of reference images (one from L0 and the other from L1), Mv1 and Mv2 are simply combined to form a bidirectional predicted motion vector; otherwise, if Mv1 and Mv2 come from the same list, only the unidirectional predicted motion vector Mv2 is stored.

[0043] In the VVC standard, when a CU uses combined mode decoding, if the CU contains at least 64 luma samples (i.e., CU width multiplied by CU height equal to or greater than 64), and if both the CU width and height are less than 128 luma samples, an additional flag is sent to indicate whether the Combined Inter-and Intra Prediction (CIIP) mode should be applied to the current CU. As the name suggests, CIIP mode combines inter-prediction and intra-prediction signals. CIIP mode P _inter The inter-frame prediction signal is derived using the same inter-frame prediction process as in the regular combining mode; the intra-frame prediction signal P _intraThis is derived from the standard intra-frame prediction process in planar mode. Then, a weighted average is used to combine the intra-frame and inter-frame prediction signals, where the weight values ​​are calculated based on the decoding modes of the top and left adjacent blocks as follows: If the top adjacent block is available and intra-coded, the variable isIntraTop is set to 1; otherwise, isIntraTop is set to 0. If the left adjacent block is available and intra-coded, the variable isIntraLeft is set to 1; otherwise, isIntraLeft is set to 0. If the sum of the two variables isIntraTop and isIntraLeft equals 2, the weight value wt is set to 3; otherwise, if the sum of the two variables equals 1, the weight value wt is set to 2; otherwise, the weight value wt is set to 1. The CIIP prediction is calculated as follows:

[0044] P CIIP =((4-wt)*P inter +wt*P intra +2)>>2 [Summary of the Invention]

[0045] To address the aforementioned problems, a video encoding method and related apparatus are provided. The following description is illustrative only and is not intended to limit the scope in any way. That is, the following overview is provided to introduce the concepts, key points, benefits, and advantages of the novel and non-obvious techniques described herein. Options, but not all, of the implementations will be further described in the detailed description below. Therefore, the following overview is not intended to identify the essential features of the claimed subject matter, nor is it intended to determine the scope of the claimed subject matter.

[0046] Some embodiments of this disclosure provide a video coding method that optimizes rate distortion in a video coding system using a layered architecture, including: receiving input data associated with a current block in a video image; determining the block partitioning structure of the current block; determining a corresponding decoding mode for each decoded block in the current block using multiple processing unit groups; dividing the current block into one or more decoded blocks according to the block partitioning structure, wherein each processing unit group has multiple processing units that execute processing unit tasks in parallel, and each processing unit group is associated with a specific block size; for each processing unit group, the current block is divided into one or more partitions, each partition has a specific block size associated with the processing unit group, and each partition is divided into sub-partitions according to one or more partition types; determining the block partitioning structure and decoding mode of the current block includes: testing multiple decoding modes for each partition of the current block and the corresponding sub-partitions divided from each partition by the parallel processing units of each processing unit group; determining the block partitioning structure of the current block and the decoding mode corresponding to each decoded block in the current block based on the rate distortion cost associated with the decoding modes tested by the processing unit groups; and performing entropy coding on one or more decoded blocks in the current block according to the corresponding decoding modes determined by the processing unit groups.

[0047] Some embodiments of this disclosure provide a video encoding apparatus that performs rate distortion optimization through a layered architecture in a video encoding system. The video encoding apparatus includes one or more electronic circuits configured to: receive input data associated with a current block in a video image; determine the block partitioning structure of the current block; determine a corresponding decoding mode for each decoding block in the current block using multiple processing unit groups; and divide the current block into one or more decoding blocks according to the block partitioning structure. Each processing unit group has multiple processing units that execute processing unit tasks in parallel, and each processing unit group is associated with a specific block size. For each processing unit group, the current block is divided into... One or more partitions, each partition having a specific block size associated with a processing unit group, and each partition being divided into sub-partitions according to one or more partition types, determining the block partition structure and decoding mode of the current block includes: testing multiple decoding modes for each partition of the current block and the corresponding sub-partitions divided from each partition by the parallel processing units of each processing unit group; determining the block partition structure of the current block and the decoding mode corresponding to each decoding block in the current block based on the rate distortion cost associated with the decoding modes tested by the processing unit group; and entropy encoding the paths of one or more decoding blocks in the current block according to the corresponding decoding modes determined by the processing unit group.

[0048] The video encoding method and related apparatus of the present invention can save bandwidth. [Attached Image Description]

[0049] The accompanying drawings are included to provide a further understanding of this disclosure, and are incorporated in and constitute a part of this disclosure. The drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the disclosure. It is understood that the drawings are not necessarily drawn to scale, as some components may be shown out of proportion to their actual dimensions in order to clearly illustrate the concepts of the disclosure.

[0050] Figure 1 An example of splitting CTB using the QTBT structure is shown.

[0051] Figure 2 This demonstrates video encoding processing for each block size using a single PE, based on the traditional video encoder.

[0052] Figure 3 An example of a GPM partition is shown.

[0053] Figure 4 A high-throughput video encoder with a hierarchical architecture for data processing in the RDO stage is shown according to an embodiment of the present invention.

[0054] Figure 5An exemplary timing diagram for data processing in the first and second PEs of PE group 0.

[0055] Figure 6 An embodiment of a hierarchical architecture for the RDO phase is shown, which uses multiple PEs from PE group 0 and PE group 1 to process 128×128 CTUs.

[0056] Figure 7 An example is shown where one of two PE tables containing different decoding modes is adaptively selected based on predefined conditions.

[0057] Figure 8 An example is shown where PEs in PE group 0 share source buffers and adjacent buffers.

[0058] Figure 9 An example is shown in which prediction samples are directly passed between parallel PEs in a PE group to generate GPM predictors.

[0059] Figure 10 An example is shown in which prediction samples are directly passed between parallel PEs in a PE group to generate CIIP predictors.

[0060] Figure 11 An example is shown where prediction samples are directly passed between parallel PEs in a PE group to generate bidirectional AMVP predictors.

[0061] Figure 12A An example is shown in which prediction samples are directly passed between parallel PEs in a PE group for the generation of BCW predictors.

[0062] Figure 12B Another embodiment is shown in which prediction samples are directly passed between parallel PEs in a PE group to generate BCW predictors.

[0063] Figure 13 An example of a buffer for sharing adjacent reconstruction samples between different PEs in a parallel PE architecture is shown.

[0064] Figure 14 An example of dynamically terminating some PEs in a parallel PE architecture to save power is shown.

[0065] Figure 15 An example of residual sharing for different transform decoding settings in a parallel PE architecture is shown.

[0066] Figure 16 An example of sharing SATD units among PEs in a parallel PE architecture is shown.

[0067] Figure 17This is a flowchart illustrating the encoding of CTB video data by multiple PE groups, each with a parallel PE, according to an embodiment of the present invention.

[0068] Figure 18 An exemplary system block diagram is shown for a video coding system that incorporates a high-throughput video processing method or a combination of multiple methods according to embodiments of the present invention.

Detailed Implementation Methods

[0069] It is readily understood that the components of the present invention, as generally described and illustrated in the accompanying drawings, can be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the systems and methods of the present invention as illustrated in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention.

[0070] Throughout this specification, references to "one embodiment," "some embodiments," or similar language mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. Therefore, the phrases "in one embodiment" or "in some embodiments" appearing throughout this specification do not necessarily refer to the same embodiment, which may be implemented alone or in combination with one or more other embodiments. Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the invention can be practiced without one or more specific details, or using other methods, components, etc. In other instances, well-known structures or operations are not shown or illustrated. Detailed description is provided to avoid obscuring aspects of the invention.

[0071] High throughput video encoder Figure 4A high-throughput video encoder with a hierarchical architecture for data processing in the RDO stage is illustrated according to an embodiment of the present invention. The encoding process of the high-throughput video encoder is generally divided into four encoding stages: preprocessing stage 42, IME stage 44, RDO stage 46, and loop filtering and entropy coding stage 48. Preprocessing stage 42, IME stage 44, RDO stage 46, and loop filtering and entropy coding stage 48 sequentially process data in video images to generate a bitstream. A common motion estimation architecture consists of Integer Motion Estimation (IME) and Fractional Motion Estimation (FME), where IME performs an integer pixel search over a large area, while FME performs a sub-pixel search around the optimally selected integer pixels. Multiple PE groups in RDO stage 46 are used to determine the block partitioning structure of the current block, and these PE groups are also used to determine the corresponding decoding mode for each decoded block in the current block. The video encoder splits the current block into one or more decoding blocks according to the block partitioning structure and encodes each decoding block according to the decoding mode determined in RDO phase 46. In RDO phase 46, each PE group has multiple parallel PEs, each PE handling the RDO tasks allocated in one PE thread. Each PE group sequentially calculates the rate distortion performance of the decoding mode tested on one or more partitions, each partition having a specific block size and sub-partitions that add up to the specific block size. For each PE group, the current block is divided into one or more partitions, each partition having a specific block size associated with the PE group, and each partition is divided into sub-partitions according to one or more partition types. For example, each partition is divided into sub-partitions using two partition types: horizontal binary tree partitioning and vertical binary tree partitioning. In some embodiments, the partitions and sub-partitions of the first PE group include a 128x128 partition, a top 128x64 sub-partition, a bottom 128x64 sub-partition, a left 64x128 sub-partition, and a 128x64 sub-partition. In another example, each partition is divided into sub-partitions using four partition types: horizontal binary tree partitions, vertical binary tree partitions, horizontal ternary tree partitions, and vertical ternary tree partitions. One PE in each PE group tests various decoding modes on each partition of the current block, which has a specific block size and corresponding sub-partitions split from each partition. The optimal block partitioning structure for the current block and the optimal decoding mode for the decoding block are thus determined based on the rate distortion cost associated with the decoding modes tested in RDO stage 46.

[0072] Each PE tests a decoding mode or one or more candidates of a decoding mode in a PE call, or each PE tests a decoding mode or candidates of a decoding mode in multiple PE calls. A PE call is a time interval. The buffer size required by a PE in each PE group can be further optimized based on the specific block size associated with the PE group. For each decoding mode or candidate of a decoding mode, the video data in the partition or subpartition can be computed using a low-complexity Rate Distortion Optimization (RDO) operation, followed by a high-complexity RDO operation. The low-complexity and high-complexity RDO operations of the decoding mode or candidate of a decoding mode can be computed by one or more PEs. Figure 5 An exemplary timing diagram illustrating data processing in the first and second PEs of PE group 0 is provided. In this example, the first and second PEs are assigned to test normal inter-frame candidate modes, where the first PE performs prediction in a low-complexity RDO operation, while the second PE performs Differential Pulse Code Modulation (DPCM) in a high-complexity RDO operation. Figure 5 In the example shown, PE group 0 is associated with a 128x128 block that allows for two possible partition types. The 128x128 block can be divided into two horizontal sub-partitions H1 and H2 via a horizontal binary tree partition, or into two vertical sub-partitions V1 and V2 via a vertical binary tree partition, or the 128x128 block can remain unpartitioned. Figure 5In this context, the first PE computes a low-complexity RDO operation (e.g., PE1_0) in each PE call, while the second PE computes a high-complexity RDO operation (e.g., PE2_1) in each PE call. The first PE in PE group 0 predicts the first horizontal binary tree sub-partition H1 using normal inter-frame candidate mode at PE call PE1_0, and predicts the first vertical binary tree sub-partition V1 using normal inter-frame candidate mode at PE call PE1_1. The first PE predicts the second horizontal binary tree sub-partition H2 using normal inter-frame candidate mode at PE call PE1_2, and predicts the second vertical binary tree sub-partition V2 using normal inter-frame candidate mode at PE call PE1_3. The first PE predicts the non-segmented partition N using normal inter-frame candidate mode at PE call PE1_4. The second PE performs DPCM on the first horizontal binary tree sub-partition H1 at PE call PE2_1, and performs DPCM on the first vertical binary tree sub-partition V1 at PE call PE2_2. The second partitioning process (PE) performs a DPCM on the second horizontal binary sub-partition H2 in call PE2_3, a DPCM on the second vertical binary sub-partition V2 in call PE2_4, and a DPCM on the unsplit partition N in call PE2_5. In this example, the high-complexity RDO operations performed by the second PE are processed in parallel with the low-complexity RDO operations of subsequent partitions / subpartitions. For example, after processing the low-complexity RDO operation of the current partition at PE1_0, the high-complexity RDO operation of the current partition at PE2_1 is processed in parallel with the low-complexity RDO operation of the subsequent partition at PE1_1.

[0073] Figure 6An embodiment of a hierarchical architecture for the RDO stage is illustrated, employing multiple PEs from PE groups 0 and 1 to process 128×128 CTUs. PE group 0 is used to compute the rate-distortion performance of various decoding modes applied to a non-split 128x128 partition and sub-partitions split from the 128x128 partition. PE group 0 determines the optimal decoding mode corresponding to the optimal block partitioning structure among the non-split 128x128 partition, two 128x64 sub-partitions, and two 64x128 sub-partitions. In this embodiment, the block partitioning test order in PE group 0 is horizontal binary tree sub-partitions H1 and H2, vertical binary tree sub-partitions V1 and V2, and then the non-split partition N. In this embodiment, four PEs are allocated in PE group 0, where each PE is used to evaluate the rate-distortion performance of one or more corresponding decoding modes applied to the 128x128 partition and sub-partitions. For example, the decoding modes evaluated by the four PEs are normal inter-frame mode, merge mode, affine mode, and intra-frame mode, respectively. In each PE thread of PE group 0, four PE calls are used to apply the corresponding decoding mode to each partition or subpartition to calculate rate distortion performance. The optimal decoding mode and optimal block partitioning structure for PE group 0 are selected by comparing the rate distortion costs across the four PE threads. Similarly, PE group 1 is used to test the rate distortion performance of various decoding modes applied to four 64x64 partitions of a 128x128 CTU and subpartitions split from the four 64x64 partitions. In this embodiment, the block partitioning test order in PE group 1 is the same as in PE group 0; however, six parallel PEs are used to evaluate the rate distortion performance of the corresponding decoding modes applied to 64x64 partitions, 64x32 subpartitions, and 32x64 subpartitions. In each PE thread of PE group 1, three PE calls are used to apply the corresponding decoding mode to each partition or subpartition. The optimal decoding mode and optimal block partitioning structure for PE group 1 are selected by comparing the rate distortion costs across the six PE threads. In addition to... Figure 6In addition to PE groups 0 and PE groups 1, there are PE groups in the RDO stage used to test various decoding modes at other block sizes. The optimal block partitioning structure for each CTU and the optimal decoding mode for the decoded blocks within the CTU are selected based on the lowest combined rate distortion cost calculated by the PE group. For example, if the combined rate distortion cost is the lowest when considering the rate distortion cost corresponding to the merge candidate applied to the left horizontal sub-partition H1 of the 64x128 in PE group 0, the CIIP candidate is applied to the 64x64 unsplitted partition N in the upper right corner of the CTU in PE group 1, and the affine candidate is applied to the 64x64 unsplitted partition N in the lower right corner of the CTU in PE group 1, then the optimal block partitioning structure of the CTU is first split by vertical binary tree partitioning, and then the right binary tree partition is further split by horizontal binary tree partitioning. The resulting decoded blocks in the CTU are one 64x128 decoded block and two 64x64 decoded blocks, and the corresponding decoding modes used to encode these decoded blocks are merge, CIIP, and affine modes, respectively.

[0074] In various embodiments of the high-throughput video encoder, encoder latency of the PE group is reduced while maintaining the highest rate distortion performance by shortening the original PE thread chain in each PE group by employing more than one parallel PE. The high-throughput video encoder of the present invention increases encoder throughput to support ultra-high definition (UHD) video encoding. The required buffer size of the PEs in various embodiments of the hierarchical architecture can be optimized according to the specific block size of each PE group. Each PE group is designed to process a specific block size, and the required buffer size for each PE group is related to the corresponding specific block size. For example, smaller buffers are used for PEs in PE groups that process smaller-sized blocks. Figure 6In the illustrated embodiment, the buffer size for PE group 0 is determined by considering the buffer size required to process 128x128 blocks, while the buffer size for PE group 1 is determined by considering only the buffer size required to process 64x64 blocks. The required buffer size for each PE group can be optimized based on the specific block size associated with each PE group, since each PE group only makes pattern decisions for partitions of a specific size or subpartitions summed to a specific block size. The required buffer size for each PE group can be further reduced by setting the same block partitioning test order for all PEs in the PE group; for example, the order in PE group 0 is horizontal binary tree partitioning, vertical binary tree partitioning, followed by no partitioning. Theoretically, three sets of reconstruction buffers are needed to store reconstruction samples corresponding to the three block partitioning types. However, after testing horizontal and vertical binary tree subpartitions, only two sets of reconstruction buffers are needed when testing the non-partitioned partitions. One set of reconstruction buffers is initially used to store reconstruction samples for horizontal binary tree subpartitions, and the other set is initially used to store reconstruction samples for vertical binary tree subpartitions. A better binary tree partitioning type corresponding to lower combinatorial rate distortion cost is selected, and the set of reconstruction buffers containing reconstruction samples of binary tree subpartitions with higher combinatorial rate distortion costs is released. When processing non-partitioned partitions, reconstruction samples of the non-partitioned partitions can be stored in the released reconstruction buffers. To further consider the improvement in decoding throughput and hardware resource optimization with respect to the RDO stage architecture, this disclosure provides the following method implemented in the proposed hierarchical architecture.

[0075] Method 1: Combining Decoding Tools or Decoding Patterns with Similar Attributes in the PE Thread Some embodiments of the present invention further reduce the required resources while improving coding throughput by combining decoding tools or decoding patterns with similar attributes in the same PE thread. Table 5 shows the decoding patterns tested by the six PEs in the PE group according to an embodiment of combining decoding tools or decoding patterns with similar attributes in the same PE thread. Call 0, Call 1, Call 2, and Call 3 represent four PE calls of the PE thread, used sequentially to process the current partition or sub-partition within the CTB. Each PE thread is scheduled to test one or more dedicated decoding tools, decoding patterns, and candidates in each PE call. In this embodiment, the first PE tests a normal inter-frame candidate mode to encode the current partition or sub-partition, wherein a one-way prediction candidate is tested, followed by a two-way prediction candidate. The second PE encodes the current partition or sub-partition using an intra-angular candidate mode. The third PE encodes the current partition or sub-partition using an affine candidate mode, and the fourth PE encodes the current partition or sub-partition using an MMVD candidate mode. The fifth PE applies the GEO candidate mode, and the sixth PE applies the inter-frame merge candidate mode to encode the current partition or sub-partition. As shown in Table 5, decoding tools or decoding modes with similar attributes are grouped together in the same PE thread. For example, the evaluation of the inter-frame merge mode can be placed in PE thread 1, and the evaluation of the affine mode can be placed in PE thread 3. If decoding tools or decoding modes with similar attributes are not placed in the same PE thread, each PE needs more hardware circuitry to support multiple decoding tools. For example, if some MMVD candidate modes are tested by PE 1 and some MMVD candidate modes are tested by PE 4, the hardware implementation requires two sets of MMVD hardware circuitry, one for PE 1 and the other for PE 4. If all MMVD candidate modes are tested by PE 4, as shown in Table 5, then PE 4 only needs one set of MMVD hardware circuitry. According to the embodiments shown in Table 5, decoding tools or decoding modes with similar attributes are arranged to be executed by the same PE thread. For example, affine-related decoding tools are all placed in PE thread 3, MMVD-related decoding tools are all placed in PE thread 4, and GEO-related decoding tools are all placed in PE thread 5.

[0076] Table 5

[0077]

[0078] Method 2: Adaptive Decoding Mode for PE Threads In some embodiments of the layered architecture, a decoding mode associated with one or more PE threads in a PE group is adaptively selected based on one or more predefined conditions. Some embodiments of the predefined conditions are associated with information comparisons between the current partition / subpartition and one or more neighboring blocks of the current partition / subpartition, the current temporal layer ID, a historical MV list, or preprocessing results. For example, the preprocessing results may correspond to search results from the IME stage. In some embodiments, the predefined conditions involve comparisons between the decoding mode, block size, block partition type, motion vector, reconstructed samples, residuals, or coefficients of the current partition / subpartition and one or more neighboring blocks. For example, a predefined condition is met when the number of neighboring blocks encoded in intra-frame mode is greater than or equal to a threshold TH1. In another example, a predefined condition is met when the current time identifier is less than or equal to a threshold TH2. According to Method 2, one or more predefined conditions are checked to adaptively select a decoding mode for the PEs in the PE group. When one or more predefined conditions are met, the PE evaluates a pre-specified decoding mode; otherwise, the PE evaluates a default decoding mode. In one embodiment of adaptively selecting a decoding mode for the current partition, when any adjacent block of the current partition is encoded in intra-frame mode, a predetermined condition is met: if at least one adjacent block is encoded in intra-frame mode, then a PE table with more intra-frame modes is tested on the current partition; otherwise, a PE table with fewer or no intra-frame modes is tested on the current partition. Figure 7 An example is shown where one of two PE tables containing different decoding modes is adaptively selected based on predefined conditions. If the predefined conditions are met, PEs 0 through 4 evaluate the decoding modes in PE table A; otherwise, PEs 0 through 4 evaluate the decoding modes in PE table B. Figure 7 In this context, n is an integer greater than or equal to 0. The three calls in each PE thread are based on, for example... Figure 7 The predefined conditions in the table shown are used for adaptive selection; however, in other examples, more or fewer calls in one or more PE threads can be adaptively selected based on one or more predefined conditions. Decoding modes can also be adaptively switched between calls. For example, when the rate distortion cost of PE computation in call(n) is too high for a particular mode, the next PE call(n+1) in the PE thread adaptively runs another mode, or the next PE call(n+1) skips decoding entirely.

[0079] Method 3: Sharing Buffers Among PEs in the Same PE Group. In some embodiments of the hierarchical architecture, certain buffers can be shared among PEs within the same PE group by unifying the data scanning order among PE threads. For example, the shared buffer is one or a combination of the source sample buffer, adjacent reconstructed sample buffer, adjacent motion vector buffer, and adjacent side information buffer. By unifying the source sample loading method among PE threads in a specific scanning order, only one set of source sample buffers needs to be shared with all PEs in the same PE group. After decoding each PE in the current PE group is complete, each PE outputs the final decoding result to the reconstructed buffer, coefficient buffer, side information buffer, and updated adjacent buffers. The video encoder compares the rate distortion cost to determine the best decoding result for the current PE group. Figure 8 An example of shared source and adjacency buffers among PEs in PE group 0 is shown. The CTU source buffer 82 and adjacency buffer 84 are shared between PEs 0 to PE Y0 in PE group 0 by unifying the data scan order among PE threads. In the first call, each PE in PE group 0, such as PE0_0, PE1_0, PE2_0, ..., PEY0_0, encodes the current partition or subpartition using an assigned decoding mode. Then, the multiplexer 86 selects the optimal decoding mode for the current partition / subpartition based on the rate distortion cost. The corresponding decoding results for the optimal decoding mode, such as reconstructed samples, coefficients, mode, MV, and adjacency information, are stored in the arrangement buffer 88.

[0080] In GPM's parallel PE, hardware sharing occurs because the current decoding block is divided into two parts by geometrically positioned straight lines, and each part of the geometric partition within the current decoding block uses its own motion for inter-frame prediction. The GPM candidate list is derived directly from the merge candidate list, for example, from merge candidates 0 and 1, merge candidates 1 and 2, merge candidates 0 and 2, merge candidates 3 and 4, merge candidates 4 and 5, and merge candidates 3 and 5, resulting in six GPM candidates. After obtaining the merged prediction samples corresponding to each part of the geometric partition based on the two merge candidates, the merged prediction samples around the edges of the geometric partition are blended to obtain the GPM prediction samples. In traditional hardware designs for computing GPM prediction samples, additional buffer resources are required to store the merged prediction samples. Using a parallel PE thread design, one embodiment of the GPM PE directly shares the merged prediction samples from two or more merged PEs without temporarily storing the merged prediction samples in a buffer. One advantage of this parallel PE design with hardware sharing is bandwidth savings, because the GPM PE directly uses merged prediction samples from the Merge PE for GPM arithmetic computations, rather than fetching reference samples from a buffer. Other benefits of directly passing predictors from the Merge PE to the GPM PE include reduced circuitry in the GPM PE and savings in motion compensation (MC) buffers. Figure 9 An example of a parallel PE design is illustrated, featuring hardware sharing for merging and GPM decoding tools. In this example, if GPM0 tested by PE 4 requires merging merged prediction samples from candidates 0, 1, and 2 to generate a GPM prediction sample, it will share the merged prediction samples from candidates 0, 1, and 2 from PE 1, 2, and 3, respectively. Similarly, if GPM1 tested by PE 4 requires merging merged prediction samples from candidates 3, 4, and 5 to generate a GPM prediction sample, PE 4 will share the merged prediction samples from candidates 3, 4, and 5 generated from PE 1, 2, and 3, respectively.

[0081] Utilizing a parallel PE design, when testing two or more GPM candidates, the implementation adaptively skips tasks assigned to one or more remaining GPM candidates based on the rate distortion cost of the current GPM candidate. PE calls initially assigned to the remaining GPM candidates may be reassigned to perform other tasks or may remain idle. The order of merging candidates is first sorted by the number of bits required for Motion Vector Difference (MVD) from best to worst (i.e., from least to most MVD bits). For example, one or more GPM candidates are tested in the first PE call, which combine with merged candidates associated with fewer MVD bits. If the rate distortion cost computed in the first PE call is greater than the current best rate distortion cost of another decoding tool, the GPM task for the remaining GPM candidates is skipped. This is based on the assumption that the GPM candidate combining merged candidates associated with the fewest MVD bits is the best GPM candidate among all GPM candidates. If this best GPM candidate cannot generate a better predictor compared to predictors generated by other decoding tools, then other GPM candidates are not worth trying. Figure 9 An example of a parallel PE thread design is shown. Figure 9 In the example shown, the number of bits required for the MVD of merge candidates Merge0, Merge1, and Merge2 is less than the number of bits required for the MVD of merge candidates Merge3, ​​Merge4, and Merge5; GPM0 requires predicted samples from Merge0, Merge1, and Merge2, and GPM1 requires predicted samples from Merge3, ​​Merge4, and Merge5. If the rate distortion cost of GPM0 is worse than the current best rate distortion cost, the original task assigned to GPM1 is skipped. In some other embodiments, merge candidates are sorted by the sum of absolute transformed differences (SATD) or the sum of absolute differences (SAD) between the current source samples and the predicted samples. SATD or SAD can be computed before starting PE threads 1 to 4 by only computing the predicted samples at certain specific locations in the block partition. Since the MV of each merge candidate is known, the predicted samples at certain specific locations can be estimated to derive the distortion value. For example, if the current partition has 64x64 samples, before performing PE threads 1 to 4, estimate the location of every 7 sample points (every 8). thThe predicted values ​​of the sample points (e.g., the predicted values ​​at sample point positions 0, 8, 16...) are collected, resulting in a total of (64 / 8)*(64 / 8) = 64 predicted samples. The SATD or SAD of these 64 sample points in the current partition can be calculated. Merge candidates are sorted according to SATD or SAD, with those having lower SATD or SAD being used first in the GPM derivation.

[0082] Hardware sharing in the parallel PE of CIIP predicts the current block decoded by CIIP by combining inter-frame prediction samples and intra-frame prediction samples. Inter-frame prediction samples are derived based on an inter-frame prediction process using merged candidates, while intra-frame prediction samples are derived based on an intra-frame prediction process using planar modes. A weighted average is used to combine the intra-frame and inter-frame prediction samples, where weight values ​​are calculated based on the decoding modes of the top and left adjacent blocks. This is achieved by utilizing... Figure 10 The parallel PE thread design of the illustrated embodiment allows CIIP candidates tested in PE thread 3 to directly share prediction samples from intra-candidates in PE thread 2 and merged candidates in PE thread 1. Conventional CIIP encoding methods require either re-acquiring reference pixels or retrieving merged and intra-prediction samples stored in a buffer. Compared to traditional methods, Figure 10 The illustrated embodiment saves bandwidth because predicted samples are passed directly from PE1 and PE2 to PE3, reducing the circuitry required to test CIIP candidates in the PEs and saving MC buffers for these PEs. Figure 10 In this process, the first CIIP candidate (CIIP0) requires prediction samples from the first merge candidate (Merge0) and the first intra-plane mode (Intra0), while the second CIIP candidate (CIIP1) requires prediction samples from the second merge candidate (Merge1) and the second intra-plane mode (Intra1). The prediction samples in the PEs for Merge0 and Intra0 are shared with the PE for CIIP0, and the prediction samples in the PEs for Merge1 and Intra1 are shared with the PE for CIIP1. The first intra-plane mode (Intra0) and the second intra-plane mode (Intra1) are essentially the same, as shown below. Figure 10 The illustrated embodiment does not have a sufficient prediction buffer to cache intra-prediction samples for the current block partition, so the Intra1 PE must generate prediction samples again in planar mode. In another embodiment with sufficient prediction buffer capacity, no additional PE call is required to Intra1, because the prediction samples generated by Intra0 can be buffered and then computed by the PE (PE computing CIIP1) for combination with Merge1.

[0083] Through a parallel PE design, one or more PEs can adaptively skip some CIIP candidates by computing tasks within CIIP candidates based on the rate distortion performance of the predictions produced by the previous CIIP candidate in the same PE thread. In one embodiment, if two or more CIIP candidates are tested in a PE thread, the original tasks assigned to subsequent CIIP candidates are skipped when the rate distortion cost associated with the current CIIP candidate is greater than the current best cost, by ordering the merged candidates from best (e.g., fewest MVD bits, lowest SATD, or lowest SAD) to worst (e.g., most MVD bits, highest SATD, or highest SAD). For example, if the SAD of the first merged candidate (Merge0) is lower than that of the second merged candidate (Merge1), the second CIIP candidate (CIIP1) is skipped if the rate distortion performance of the first CIIP candidate (CIIP0) is worse than the current best rate distortion performance of another decoding tool. This is because if the merged candidates are correctly ordered, the rate distortion performance of the second CIIP candidate is likely to be worse than that of the first CIIP candidate.

[0084] Hardware sharing in the parallel PE for AMVP-BI predicts the current block decoded using Bi-directional Advance Motion Vector Prediction (AMVP-BI) by combining unidirectional prediction samples from AMVP list 0 (L0) and list 1 (L1). Figure 11 The parallel PE design of the illustrated embodiment allows AMVP-BI candidates tested in PE thread 3 to directly share prediction samples from AMVP-UNI_L0 candidates tested in PE thread 1 and AMVP-UNI_L1 candidates tested in PE thread 2. Traditional methods of AMVP-BI encoding obtain reference pixels stored in a buffer. Compared to traditional methods, Figure 11 The illustrated embodiment saves bandwidth because the predicted samples are passed directly from PE1 and PE2 to PE3, effectively reducing the circuitry required for testing the AMVP-BI PEs and saving the MC buffers of these PEs. Figure 11 In this process, the PE (Programmer) needs prediction samples from List 0 (one-way AMVP) and List 1 (one-way AMVP) to compute AMVP-BI. The prediction samples in the PE that computes AMVP-UNI_L0 and AMVP-UNI_L1 are shared with the PE that computes AMVP-BI.

[0085] Hardware sharing in the parallel PE used for BCW generates the predictor of the current block for BCW decoding by weighted averaging two unidirectional prediction signals obtained from two different reference lists L0 and L1. This is achieved using... Figure 12AThe parallel PE design of the illustrated embodiment directly shares prediction samples from PE thread 1 (testing AMVP-UNI_L0) and PE thread 2 (testing AMVP-UNI_L1) with BCW0 tested in PE thread 3 and BCW1 tested in PE thread 4. Traditional BCW encoding methods require acquiring reference pixels stored in a buffer. Compared to traditional methods, Figure 12A The illustrated embodiment saves bandwidth because the predicted samples are passed directly from PE1 and PE2 to PE3 and PE4, reducing the circuitry required to compute BCW0 and BCW1 in the PEs and saving MC buffers in these PEs. Figure 12A In the test BCW0, the PE (Programmer) acquires prediction samples from List 0 and List 1 unidirectional AMVP, and then tests the combination of these two predictors by weighting the prediction samples according to weight patterns 1 and 2. The PE test BCW1 also acquires prediction samples from List 0 and List 1 unidirectional AMVP, and then tests the combination of these two predictors by weighting the prediction samples according to weight patterns 3 and 4. The prediction samples in the PEs testing AMVP-UNI_L0 and AMVP-UNI_L1 are shared with the PE in test BCW0. Figure 12B Another embodiment of the parallel PE design is shown, where instead of allocating two PEs to test the rate distortion performance of BCW, only one PE is used. Figure 12A In contrast, the advantage of this design is that it can skip the second BCW candidate (BCW1) based on the rate distortion cost of the first BCW candidate (BCW0). Similar to embodiments of parallel PE designs like GPM and CIIP, if the rate distortion cost of the current BCW candidate is greater than the current optimal rate distortion cost, the remaining BCW candidates are skipped. For example, as... Figure 12B As shown, if the PE test BCW0 combines the AMVP L0 and AMVP L1 one-way prediction samples with weight patterns 1 and 2, and the rate distortion cost of both combinations is lower than the current best rate distortion cost, then the BCW1 candidate is skipped. It is assumed that the predictors generated according to weight patterns 1 and 2 will be superior to those generated according to weight patterns 3 and 4.

[0086] Adjacent sharing in parallel PEs, according to embodiments of the present invention, allows for the sharing of buffers for adjacent reconstructed samples among different PEs. For example, only one set of adjacent buffers is needed because both intra-frame PEs and matrix-based intra-prediction (MIP) PEs can obtain adjacent reconstructed samples from this shared buffer. Figure 13As shown, PE 1 tests intra-frame prediction, while PE 2 tests MIP prediction. The block segmentation test order is horizontal binary tree partition 1 (HBT1), vertical binary tree partition 1 (VBT1), horizontal binary tree partition 2 (HBT2), and vertical binary tree partition 2 (VBT2). The first PE call in PE thread 1 and the first PE call in PE thread 2 both require adjacent reconstructed samples from horizontal binary tree partition 1 to derive the prediction sample. Using a parallel PE design, a set of adjacent buffers can be shared for these two PEs. Similarly, the second PE call in PE thread 1 and the second PE call in PE thread 2 both require adjacent reconstructed samples from vertical binary tree partition 1 to derive the prediction sample; therefore, the adjacent buffers pass the corresponding adjacent reconstructed samples to these two PEs.

[0087] In some embodiments of a multi-PE design, on-the-fly terminating processing involves prematurely terminating the remaining processing of at least one other PE thread based on the cumulative rate distortion cost of the parallel PEs. For example, if the current cumulative rate distortion cost of one PE thread is significantly lower than that of the other PE threads (i.e., the current cumulative rate distortion cost is significantly lower than the cumulative rate distortion cost of each of the other PE threads), the remaining processing of the other PE threads is terminated early to save power. Figure 14 This example demonstrates how to prematurely terminate two parallel PE threads based on the cumulative rate-distortion cost of three parallel PE threads. In this example, at a certain point before the completion of decoding, where decoding is tested via parallel PEs, if the cumulative rate-distortion cost of PE thread 1 is significantly lower than that of PE threads 2 and 3, the remaining processing of PE threads 2 and 3 is prematurely shut down in video encoding. For example, the offset between the cumulative rate-distortion cost of each of PE threads 2 and 3 and that of PE thread 1 is greater than a predefined threshold. Assuming that the difference between the cumulative rate-distortion costs of PE threads 1 and 2 and the difference between the cumulative rate-distortion costs of PE threads 1 and 3 both exceed the threshold at the check time, the final rate-distortion cost of PE threads 2 and 3 will certainly exceed the final rate-distortion cost of PE thread 1.

[0088] The MTS shared multiple transform selection (MTS) scheme for parallel PE architectures utilizes multiple transform selections to process residuals. For example, different transforms include DCT-II, DCT-VIII, and DST-VII. Figure 15 An example of residual sharing for transform decoding implemented through parallel PE design is described. Figure 15In order to test the same predictions using two different transform decoders for DCT-II and DST-VII, one PE can share its residuals with another PE through a parallel PE design. This allows for the hardware advantage of having only a single residual buffer by sharing the residuals between the DCT-II and DST-VII transform decoders. Figure 15 In this case, the circuitry related to the prediction processing in PE 2 can be omitted because the residuals generated from the same predictor can be directly passed from PE 1.

[0089] Low-complexity SATD dynamic on-the-fly re-allocation utilizes a parallel PE design, allowing SATD units to be shared among parallel PEs. Figure 16 An embodiment of sharing SATD units from one PE to another is illustrated. In this embodiment, PE 1 encodes the current block partition in merge mode during the first PE call, and then encodes the current or subsequent block partition in MMVD mode. PE 2 encodes the current block partition in BCW mode during the first PE call, and encodes the current or subsequent block partition in AMVP mode during the second PE call. Assuming that the merge, BCW, MMVD, and AMVP PEs require 2, 90, 50, and 50 sets of SATD units respectively, PE 2, which computes BCW candidates, can borrow 40 sets of SATD units from PE 1, which computes merge candidates. By allowing dynamic reallocation of SATD units between parallel PEs, low-complexity rate distortion optimization decision circuitry can be used more efficiently.

[0090] Representative flowchart for high-throughput video coding Figure 17This is a flowchart illustrating an embodiment of a video coding system that encodes video data using a hierarchical architecture with parallel PE groups. In step S1702, the video coding system receives the current decoded block (CTB) in the current video frame. According to this embodiment, the current CTB is a luminance CTB with 128x128 samples. In this embodiment, the maximum size of the decoded block (CB) is set to 128x128, and the minimum size of the CB is set to 2x4 or 4x2. Steps S17040, S17041, S17042, S17043, S17044, and S17045 correspond to PE groups 0, PE group 1, PE group 2, PE group 3, PE group 4, and PE group 5, respectively. PE group 0 is associated with a specific block size of 128x128, and PE groups 1, 2, 3, 4, or 5 are associated with specific block sizes of 64x64, 32x32, 16x16, 8x8, or 4x4. For PE group 0, in step S17040, the current CTB is set as a 128x128 partition and divided into sub-partitions according to the preset partition type. For example, the preset partition types are horizontal binary tree partitioning and vertical binary tree partitioning. Therefore, according to the horizontal binary tree partitioning, the current CTB is divided into two 128x64 sub-partitions, and according to the vertical binary tree partitioning, the current CTB is divided into two 64x128 sub-partitions. In step S17041, for PE group 1, the current CTB is first divided into four 64x64 partitions, and each 64x64 partition is divided into sub-partitions according to the preset partition type. PE groups 2 to PE groups 4 perform similar processing steps, dividing the current CTB into partitions and sub-partitions. For simplicity, these steps are not detailed in the text. Figure 17 As shown in the diagram. For PE group 5, in step S17045, the current CTB is divided into 4x4 partitions, and each 4x4 partition is further divided into sub-partitions according to a preset partition type. Each PE group contains multiple parallel PEs. In step S17060, the PEs in PE group 0 test a set of decoding patterns on each 128x128 partition and each sub-partition. In step S17061, the PEs in PE group 1 test a set of decoding patterns on each 64x64 partition and each sub-partition. The PEs in PE groups 2, 3, or 4 also test a set of decoding patterns on each corresponding partition and sub-partition. In step S17065, the PEs in PE group 5 test a set of decoding patterns on each 4x4 partition and sub-partition. In step S1708, the video coding system determines the block partitioning structure of the current CTB for dividing it into CBs, and the video coding system also determines the corresponding decoding pattern for each CB based on the rate distortion cost of the tested decoding patterns. In step S1710, the video coding system performs entropy coding on the CBs in the current CTB.

[0091] An exemplary video encoder implementing the present invention can implement embodiments of the invention within a video encoder. For example, the disclosed methods can be implemented in one or a combination of an entropy coding module, an inter-frame, intra-frame, or prediction module, and a transform module of the video encoder. Optionally, any disclosed method can be implemented as circuitry coupled to the entropy coding module, the inter-frame, intra-frame, or prediction module, and the transform module of the video encoder to provide the information required by any module. Figure 18 An exemplary system block diagram of a video encoder 1800 for implementing one or more embodiments of the present invention is shown. The video encoder 1800 receives input video data of a current picture, consisting of multiple CTUs. Each CTU consists of a CTB of one luminance sample and one or more corresponding CTBs of chrominance samples. A hierarchical architecture is used in the RDO stage, with each CTB processed by multiple PE groups consisting of parallel processing PEs. The PEs process each CTB in parallel to test various decoding modes at different block sizes. For example, each PE group is associated with a specific block size, and the PE threads in each PE group calculate the rate distortion rate to apply various decoding modes to partitions and corresponding sub-partitions with a specific block size. The optimal block partitioning structure for dividing the CTB into CBs and the optimal decoding mode for each CB are determined based on the lowest combined rate distortion rate. In some embodiments of the invention, hardware is shared between parallel PEs within a PE group to reduce the bandwidth, circuitry, or buffers required for encoding. For example, prediction samples are shared directly between parallel PEs without temporarily storing prediction samples in buffers. In another example, a set of adjacent buffers storing adjacent reconstructed samples is shared between parallel PE threads within a PE group. In yet another example, SATD units can be dynamically shared among parallel PE threads within a PE group. Figure 18 In this process, the intra-frame prediction module 1810 provides intra-frame predictors based on the reconstructed video data of the current frame. The inter-frame prediction module 1812 performs motion estimation (ME) and motion compensation (MC) to provide inter-frame predictors based on reference video data from one or more other frames. The intra-frame prediction module 1810 or the inter-frame prediction module 1812 uses switch 1814 to provide the selected predictor of the current decoded block in the current frame to adder 1816 to form a residual by subtracting the selected predictor from the original video data of the current decoded block. The residual of the current decoded block is further processed by transform module (T) 1818 and quantization module (Q) 1820. In a hardware-shared example, the residual is shared between parallel PE threads for transform processing according to different transform decoding settings. The transformed and quantized residuals are then encoded by entropy encoder 1834 to form a video bitstream. The transform and quantization residuals of the current block are also processed by the inverse quantization module (IQ) 1822 and the inverse transform module (IT) 1824 to recover the prediction residuals. For example... Figure 18As shown, the residual is recovered by adding the selected predictor back at the reconstruction module (REC) 1826 to produce reconstructed video data. The reconstructed video data can be stored in a reference picture buffer (Ref.Pict.Buffer) 1832 and used for prediction of other pictures. Due to encoding processing, the reconstructed video data from REC 1826 may be subject to various degradations; therefore, at least one in-loop processing filter (ILPF) 1828 is conditionally applied to the luma and chroma components of the reconstructed video data before they are stored in the reference picture buffer 1832 to further improve picture quality. An example of ILPF 1828 is a deblocking filter. Syntax elements are provided to the entropy encoder 1834 to be incorporated into the video bitstream.

[0092] Figure 18 The various components of the video encoder 1800 can be implemented by hardware components, one or more processors configured to execute program instructions stored in memory, or a combination of hardware and processors. For example, a processor executes program instructions to control the reception of input data for the current block for video encoding. The processor is equipped with one or more processing cores. In some examples, the processor executes program instructions to perform functions in some components of the encoder 1800, and memory electrically coupled to the processor is used to store program instructions, information corresponding to the reconstructed image of the block, and / or intermediate data during the encoding or decoding process. In some examples, the video encoder 1800 can transmit information by including one or more syntax elements in the video bitstream, and the corresponding video decoder derives such information by parsing and decoding one or more syntax elements. In some embodiments, the memory buffer includes a non-transitory computer-readable medium, such as semiconductor or solid-state memory, random access memory (RAM), read-only memory (ROM), hard disk, optical disk, or other suitable storage media. The memory buffer can also be a combination of two or more of the non-transitory computer-readable media listed above.

[0093] Embodiments of the high-throughput video coding processing method can be implemented in circuitry integrated into a video compression chip or in program code integrated into video compression software to perform the aforementioned processing. For example, the encoder-decoder block can be implemented in program code to be executed on a computer processor, digital signal processor (DSP), microprocessor, or field-programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the invention by executing machine-readable software code or firmware code that defines the specific methods embodied in the invention.

[0094] This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered illustrative rather than limiting in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All variations within the equivalent meaning and scope of the claims should be included within their scope.

[0095] The use of ordinal terms such as "first" and "second" in this disclosure and claims is for illustrative purposes only. It does not in itself imply any order or relationship.

[0096] The steps of the methods described in conjunction with the aspects disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module (e.g., including executable instructions and associated data) and other data can reside in data memory, such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of computer-readable storage medium known in the art. The sample storage medium can be coupled to a machine, such as a computer / processor (which may be referred to herein as a "processor" for convenience), such that the processor can read information (e.g., code) from a location and write information to the storage medium. The sample storage medium can be integrated into the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in a user device. Alternatively, the processor and storage medium can reside in a user device as discrete components. Furthermore, in some aspects, any suitable computer program product may include a computer-readable medium comprising code relating to one or more aspects of this disclosure. In some aspects, the computer software product may include packaging material.

[0097] It should be noted that, although not explicitly stated, one or more steps of the methods described herein may include steps for storing, displaying, and / or outputting data as needed for a particular application. In other words, any data, records, fields, and / or intermediate results discussed in the methods may be stored, displayed, and / or output to another device as needed for a particular application. While the foregoing describes embodiments of the invention, other and further embodiments of the invention can be devised without departing from its basic scope. The various embodiments or portions thereof presented herein may be combined to produce further embodiments. The above description represents the best contemplated mode for carrying out the invention. This description is intended to illustrate the general principles of the invention and should not be construed as limiting. The scope of the invention is best determined by reference to the appended claims.

[0098] The preceding paragraphs describe many aspects. It is evident that the teachings of this invention can be implemented in various ways, and any specific configuration or function in the disclosed embodiments represents only one representative case. Those skilled in the art will understand that all aspects disclosed in this invention can be applied independently or incorporated into it.

[0099] Although the invention has been described by way of example and preferred embodiments, it should be understood that the invention is not limited thereto. Those skilled in the art can make various modifications and alterations without departing from the scope and spirit of the invention. Therefore, the scope of the invention should be defined and protected by the appended claims and their equivalents.

Claims

1. A video coding method, comprising optimizing rate distortion in a video coding system through a layered architecture, including: Receive input data associated with the current block in the video image; The block partitioning structure of the current block is determined. A corresponding decoding mode is determined for each decoding block in the current block using multiple processing unit groups. The current block is then divided into one or more decoding blocks according to the block partitioning structure. Each processing unit group has multiple processing units that execute processing unit tasks in parallel, and each processing unit group is associated with a specific block size. For each processing unit group, the current block is divided into one or more partitions. Each partition has a specific block size associated with that processing unit group, and each partition is further divided into sub-partitions according to one or more partition types. Determining the block partitioning structure and decoding mode of the current block includes: Each processing unit in each processing unit group performs parallel testing of multiple decoding modes on each partition of the current block and the corresponding sub-partitions derived from each partition; and Based on the rate distortion cost associated with the decoding mode tested by the processing unit group, the block partitioning structure of the current block and the decoding mode corresponding to each decoding block in the current block are determined; and The current block or one or more decoded blocks are entropy encoded according to the corresponding decoding mode determined by the processing unit group.

2. The video coding method of claim 1, wherein, The buffer size required for each processing unit group is related to the specific block size of that processing unit group.

3. The video encoding method of claim 2 further includes setting the same block partition test order for all processing units in the processing unit group, and releasing a set of reconstruction buffers storing reconstruction samples associated with one of the at least two partition types based on the rate distortion cost associated with the at least two partition types, to store reconstruction samples associated with the other partition type.

4. The video coding method of claim 1, wherein, The one or more partition types used to divide each partition in the current block into subpartitions include one or a combination of horizontal binary tree partitions, vertical binary tree partitions, horizontal ternary tree partitions, and vertical ternary tree partitions.

5. The video encoding method as described in claim 1, characterized in that, In a single processing unit call, the processing unit tests the decoding pattern or one or more candidates of the decoding pattern, or in multiple processing unit calls, the processing unit tests the decoding pattern or one candidate of the decoding pattern.

6. The video encoding method as described in claim 1, characterized in that, In a processing unit call, the processing unit computes a low-complexity processing unit operation, followed by a high-complexity processing unit operation, or in a processing unit call, the processing unit computes either a low-complexity processing unit operation or a high-complexity processing unit operation.

7. The video encoding method as described in claim 1, characterized in that, The first processing unit in the processing unit group calculates the low-complexity processing unit operation of the decoding mode, and the second processing unit in the same processing unit group calculates the high-complexity processing unit operation of the decoding mode, wherein the low-complexity processing unit operation of the subsequent partition calculated by the first processing unit and the high-complexity processing unit operation of the current partition calculated by the second processing unit are executed in parallel.

8. The video encoding method as described in claim 1, characterized in that, Combine decoding tools or decoding patterns with similar properties to test within the same processing unit thread in each processing unit group.

9. The video encoding method as described in claim 1, characterized in that, The parallel testing of multiple decoding modes by the processing units of the processing unit group on a partition or sub-partition further includes checking one or more predefined conditions, and when the one or more predefined conditions are met, adaptively testing the selected encoding / decoding mode by at least one processing unit in the processing unit group.

10. The video encoding method as described in claim 9, characterized in that, The one or more predefined conditions are associated with information comparisons between the partition / subpartition and one or more adjacent blocks of the partition / subpartition, current time identifiers, historical motion vector lists, or preprocessing results; wherein the information between the partition / subpartition and one or more adjacent blocks of the partition / subpartition includes decoding mode, block size, block partition type, MV, reconstructed samples, or residuals.

11. The video encoding method as described in claim 9, characterized in that, When one or more predefined conditions are met, one or more processing units skip decoding in one or more processing unit calls.

12. The video encoding method as described in claim 11, characterized in that, One of the predefined conditions is met when the cumulative rate distortion cost associated with a processing unit is higher than the cumulative rate distortion cost associated with each of the other processing units by a predetermined threshold.

13. The video encoding method as described in claim 1, characterized in that, By unifying the data scanning order among processing units, one or more buffers can be shared among processing units in the same processing unit group.

14. The video encoding method as described in claim 1, characterized in that, The current processing units in the current processing unit group directly share the prediction samples from one or more processing units in the current processing unit group, without temporarily storing the prediction samples in a buffer.

15. The video encoding method as described in claim 14, characterized in that, The current processing unit obtains the prediction sample from the one or more processing units that test merge candidates on each partition or subpartition, and tests one or more geometric partition pattern candidates on each partition or subpartition.

16. The video encoding method as described in claim 15, characterized in that, The GPM task initially assigned to the current processing unit is adaptively skipped based on the rate distortion cost associated with the prediction result of the current processing unit.

17. The video encoding method as described in claim 14, characterized in that, The current processing unit obtains the prediction sample from a processing unit that tests the intra-plane mode of the test frame and the one or more processing units that test the merged candidates on the partition or sub-partition. The current processing unit tests one or more combined inter-frame and intra-frame prediction candidates on each partition or sub-partition.

18. The video encoding method as described in claim 17, characterized in that, The CIIP task initially assigned to the current processing unit is adaptively skipped based on the rate distortion cost associated with the prediction result of the current processing unit.

19. The video encoding method as described in claim 14, characterized in that, The prediction sample is obtained by the current processing unit testing one or more bidirectional advanced motion vector prediction candidates on each partition or subpartition by the processing unit testing one or more bidirectional advanced motion vector prediction candidates on each partition or subpartition.

20. The video encoding method as described in claim 19, characterized in that, The prediction sample is obtained by the current processing unit testing one or more bidirectional prediction candidates with decoding unit-level weights on each partition or subpartition, by the processing unit testing one or more bidirectional prediction candidates with decoding unit-level weights on each partition or subpartition.

21. The video encoding method as described in claim 1, characterized in that, A set of adjacent buffers that store adjacent reconstructed samples are shared among multiple processing units in a processing unit group.

22. The video encoding method as described in claim 1, characterized in that, It also includes generating residuals for each decoded block in the current block and sharing these residuals among multiple processing units for transform processing based on different transform decoding settings.

23. The video encoding method as described in claim 1, characterized in that, The absolute transformation difference and unit are dynamically shared among processing units within a processing unit group.

24. A video encoding apparatus for rate distortion optimization in a video encoding system using a layered architecture, the video encoding apparatus comprising one or more electronic circuits configured to: Receive input data associated with the current block in the video image; The block partitioning structure of the current block is determined. A corresponding decoding mode is determined for each decoding block in the current block using multiple processing unit groups. The current block is then divided into one or more decoding blocks according to the block partitioning structure. Each processing unit group has multiple processing units that execute processing unit tasks in parallel, and each processing unit group is associated with a specific block size. For each processing unit group, the current block is divided into one or more partitions. Each partition has a specific block size associated with that processing unit group, and each partition is further divided into sub-partitions according to one or more partition types. Determining the block partitioning structure and decoding mode of the current block includes: Each processing unit in each processing unit group performs multiple decoding modes in parallel for each partition of the current block and the corresponding sub-partitions divided from each partition; as well as Based on the rate distortion cost associated with the decoding mode tested by the processing unit group, the block partitioning structure of the current block and the decoding mode corresponding to each decoding block in the current block are determined. as well as The current block or one or more decoded blocks are entropy encoded according to the corresponding decoding mode determined by the processing unit group.

Images