Intra mode JVET coding

Abstract

To provide a method of properly partitioning a video coding block for JVET.SOLUTION: A set of MPMs includes a set of other than 6 intra prediction coding modes and can be encoded using truncated unary binarization, 16 selected intra prediction coding modes can be encoded using 4 bits of fixed length code and remaining non-selected coding modes can be encoded using truncated binary coding. A JVET coding tree unit can be coded as a root node in a QTBT structure. The QTBT structure has a quadtree branching from the root node and binary trees branching from each of the quadtree's leaf nodes, uses asymmetric binary partitioning to split a coding unit represented by a quadtree leaf node into child nodes, and represents the child nodes as leaf nodes in a binary tree branching from the quadtree leaf node.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates to the field of video coding, and more specifically, efficient intra-mode coding. [Background technology]

[0002] Technical improvements in evolving video coding standards show a trend towards increasing coding efficiency to achieve higher bitrates, higher resolutions, and better video quality. The Joint Video Exploration Team is developing a new video coding scheme called JVET. Similar to other video coding schemes such as High Efficiency Video Coding (HEVC), JVET is a block-based hybrid spatial and temporal predictive coding scheme. However, compared to HEV, JVET includes many changes to the bitstream structure, syntax, constraints, and mappings to generate the decoded multiple pictures. JVET is implemented in the Joint Exploration Model (JEM) encoder and decoder.

[0003] The current JVET standard describes a total of 67 intra prediction modes, including planar, DC, and 65 directional angular intra modes. To efficiently code these 67 modes, all intra modes are subdivided into three sets, including a set of 6 most probable modes (MPM), a set of 16 selected modes, and a set of 45 non-selected modes.

[0004] The six MPMs are derived from the modes of available neighboring blocks, derived intra modes and default intra modes. The intra modes of the five neighboring blocks of the current block are shown in Figure 1a. These are left (L), top (A), bottom left (BL), top right (AR) and top left (AL) and are used to form the MPM list of the current block. The initial MPM list is created by inserting the five neighboring intra modes, the planar mode and the DC mode into the MPM list. A pruning process is used to remove duplicated modes so that only unique modes can be included in the MPM list. The order in which multiple initial modes are included is left, top, planar, DC, bottom left, top right and top left.

[0005] If the MPM list is not full, derived modes are added, these intra modes are derived by adding "-1" or "+1" to the angular modes already included in the MPM list. If the MPM list is not yet complete, several default modes are added in the following order: vertical, horizontal, mode 2, and diagonal modes. This process results in a unique list of six MPM modes.

[0006] For entropy coding of the six MPMs, a truncated unary binarization shown in Fig. 1b is currently used. The first three bins of MPM modes are coded with multiple contexts that depend on the MPM mode associated with the currently signaled bin. MPM modes are classified into one of three categories: (a) modes that are predominantly horizontal (i.e., the MPM mode number is smaller than the diagonal mode number), (b) modes that are predominantly vertical (i.e., the MPM mode number is larger than the diagonal mode number), and (c) non-angular (DC and planar) classes. Thus, three contexts are used to signal the MPM index based on this classification.

[0007] The coding for selecting the remaining 61 non-MPMs is done as follows: The 61 non-MPMs are first divided into two sets: selected and non-selected mode sets. The selected mode set contains 16 modes, and the rest (45 modes) are assigned to the non-selected mode set. The mode set to which the current mode belongs is indicated by a flag in the bitstream. If the indicated mode is in the selected mode set, the selected mode is signaled with a 4-bit fixed length code, and if the indicated mode is from the non-selected mode set, the selected mode is signaled with a truncated binary code. As an example, the selected mode set is generated by subsampling the 61 non-MPM modes as follows:

[0008] Selection mode set = {0,4,8,12,16,20…60} Unselected mode set = {1, 2, 3, 5, 6, 7, 9, 10…59} The current JVET intra-mode coding is summarized in Figure 1b below.

[0009] As shown in Fig. 1b, the last two entries of the MPM list require six bins, which is the same as the number of bins assigned to the 16 selection modes. Such a configuration has no advantage in terms of coding performance for the last two modes of the MPM list. Also, since the first three bins of the MPM mode are coded with context-based entropy coding, the coding complexity of the six bins of the MPM mode is higher than that of the six bins of the selection mode.

[0010] What is needed is a system and method that reduces the coding burden and bandwidth associated with intra-mode coding. Summary of the Invention

[0011] This disclosure provides a video coding method for JVET intra prediction, the video coding method including: defining a set of unique intra-predictive coding modes, which in some embodiments may be 67 modes; identifying and instantiating in memory a subset of unique MPM intra-predictive coding modes from the set of unique intra-predictive coding modes, which in some embodiments may be 5 or less of 7 or more. The method also provides identifying and instantiating in memory a subset of selected unique intra-predictive coding modes, which in some embodiments may include 16 coding modes, from the set of unique intra-predictive coding modes other than the subset of unique MPM intra-predictive coding modes; and identifying and instantiating in memory a subset of non-selected unique intra-predictive coding modes that constitute a balance of the intra-prediction modes from the set of unique intra-predictive coding modes other than the subset of unique MPM intra-predictive coding modes and other than the selected subset of unique intra-predictive coding modes. The subset of unique MPM intra-predictive coding modes is then coded using truncated unary binarization.

[0012] This disclosure also provides a video coding system for JVET intra prediction, in some embodiments the video coding system comprising: instantiating in memory a set of 67 unique intra-predictive coding modes; instantiating in memory a subset of unique MPM intra-predictive coding modes from the set of unique intra-predictive coding modes; instantiating in memory a subset of 16 unique selected intra-predictive coding modes from the set of unique intra-predictive coding modes other than the subset of unique MPM intra-predictive coding modes; instantiating in memory a subset of unselected unique intra-predictive coding modes from the set of unique intra-predictive coding modes that is other than the subset of unique MPM intra-predictive coding modes and other than the subset of unique selected intra-predictive coding modes; encoding the subset of unique MPM intra-predictive coding modes using truncated unary binarization; and encoding the subset of 16 selected unique intra-predictive coding modes using a 4-bit fixed length code. [Brief description of the drawings]

[0013] Further details of the invention are explained using the accompanying drawings. [Figure 1a] Indicates the neighboring blocks related to the current coding block. [Figure 1b] 1 shows a table of the current JVET coding for intra-mode prediction. [Figure 1c] This shows the division of a frame into multiple Coding Tree Units (CTUs). [Diagram 2] 1 illustrates an exemplary partitioning of a CTU into multiple Coding Units (CUs) using quadtree partitioning and symmetric bipartitioning. [Diagram 3]We show a QTBT (quadtree plus binary tree) representation of the partition in Figure 2. [Figure 4] We present four possible types of asymmetric bisection of a CU into two smaller CUs. [Diagram 5] 1 illustrates an example partitioning of a CTU into multiple CUs using quadtree partitioning, symmetric bisection, and asymmetric bisection. [Figure 6] We show the QTBT representation of the partition in Fig. 5 . [Figure 7] 1 shows a simplified block diagram of CU coding in the JVET encoder. [Figure 8] 67 possible intra prediction modes for the luma component of JVET are shown. [Figure 9] 1 shows a simplified block diagram of CU coding in the JVET encoder. [Figure 10] 1 illustrates an embodiment of a method for CU coding in a JVET encoder. [Figure 11] 1 shows a simplified block diagram of CU coding in the JVET encoder. [Figure 12] FIG. 1 shows a simplified block diagram of CU decoding in the JVET decoder. [Figure 13] 1 shows an alternative simplified block diagram of JVET coding for intra-mode prediction. [Figure 14] 13 shows a table of alternative JVET coding for intra-mode prediction. [Figure 15] 1 illustrates an embodiment of a computer system adapted and / or configured to process a method of CU coding. [Figure 16] 1 illustrates an embodiment of an encoding / decoding system for CU encoding / decoding in a JVET encoder / decoder. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] FIG. 1 illustrates the division of a frame into Coding Tree Units (CTUs) 100. A frame may be an image of a video sequence. A frame may include a matrix or a set of matrices with pixel values ​​that represent intensity measurements in the image. A video sequence may then be generated by the set of matrices. The pixel values ​​may be defined to represent color and brightness in full-color video coding, where pixels are divided into three channels. For example, in the YCbCr color space, pixels have a luminance value Y that represents the intensity of the gray level of the image, and two chrominance values ​​Cb, Cr that represent the color difference from gray to blue and red. In other embodiments, the pixel values ​​may be represented by values ​​in different color spaces or models. The resolution of the video determines the number of pixels in a frame. A higher resolution results in more pixels and a sharper image, but also higher bandwidth, storage, and transmission requirements.

[0015] Frames of a video sequence may be encoded and decoded using JVET. JVET is a video coding scheme being developed by the Joint Video Exploration Team. Versions of JVET have been implemented in the Joint Exploration Model (JEM) decoder and decoder. Similar to other video coding schemes such as High Efficiency Video Coding (HEVC), JVET is a block-based hybrid spatial and temporal predictive coding scheme. In coding with JVET, a frame is first divided into square blocks called CTU100, as shown in FIG. 1. For example, the CTU100 may be blocks of 128x128 pixels.

[0016] FIG. 2 shows an example division of a CTU 100 into multiple CUs 102. Each CTU 100 in a frame may be divided into one or more Coding Units (CUs) 102. One or more CUs 102 may be used for prediction and transformation as described below. Unlike HEVC, in JVET, the CUs 102 may be rectangular or square and may be coded without further division into prediction units or transform units. The CUs 102 may be as large as their root CTU 100 or may be smaller subdivisions of the root CTU 100 as small as a 4×4 block.

[0017] In JVET, the CTU 100 may be split into multiple CUs 102 according to a quadtree plus binary tree (QTBT) scheme. In this scheme, the CTU 100 may be recursively split into multiple square blocks according to a quadtree, and these square blocks may be recursively split horizontally or vertically according to a binary tree. Multiple parameters may be set to control the split according to the QTBT, such as the CTU size, the minimum size of the leaf nodes of the quadtree and binary tree, the maximum size of the leaf nodes of the binary tree, and the maximum depth of the binary tree.

[0018] In some embodiments, JVET can restrict the binary partitioning of the binary tree portion of QTBT to a symmetric partitioning, where blocks are split in half either vertically or horizontally along the midline.

[0019] 2 illustrates a CTU 100 partitioned into multiple CUs 102, with solid lines indicating a quadtree partition and dashed lines indicating a symmetric binary tree partition. As illustrated, the binary partition allows for symmetric horizontal and vertical partitioning to define the structure of the CTU and its subdivision into multiple CUs.

[0020] FIG. 3 shows a QTBT representation of the partition of FIG. 2. A quadtree root node represents the CTU 100 with each child node of the quadtree section representing one of the four square blocks partitioned from the parent square block. The square blocks represented by the quadtree leaf nodes are partitioned symmetrically zero or more times using a binary tree, and the quadtree leaf nodes are root nodes of the binary tree. At each level of the binary tree section, the block may be partitioned symmetrically vertically or horizontally. A flag set to "0" indicates that the block is partitioned symmetrically horizontally, and a flag set to "1" indicates that the block is partitioned symmetrically vertically.

[0021] In other embodiments, the JVET may allow either symmetric or asymmetric bipartitioning in the binary tree portion of the QTBT. Asymmetric motion partitioning (AMP) is possible in a different context of HEVC when partitioning multiple prediction units (PUs). However, when partitioning multiple CUs 102 in the JVET according to the QTBT structure, asymmetric bipartitioning may result in improved partitioning relative to symmetric bipartitioning when multiple correlated areas of a CU 102 are not located on either side of a midline through the center of the CU 102. As a non-limiting example, if a CU 102 shows one object close to the center of the CU and another object on the side of the CU 102, the CU 102 may be partitioned asymmetrically to place each object in a separate smaller CU 102 of different size.

[0022] 4 illustrates four possible types of asymmetric bisections, where a CU 102 is split into two smaller CUs 102 along a line that crosses the length or height of the CU 102, where one of the two smaller CUs 102 is 25% of the size of the parent CU 102 and the other is 75% of the size of the parent CU 102. With the four types of asymmetric bisections illustrated in FIG. 4, the CU 102 can be split along a line that is 25% away from the left side of the CU 102, 25% away from the right side of the CU 102, 25% away from the top of the CU 102, or 25% away from the bottom of the CU 102. In another embodiment, the asymmetric bisection line along which the CU 102 is split can be located in any other location such that the CU 102 is not symmetrically split into halves.

[0023] 5 shows a non-limiting example of a CTU 100 that has been split into multiple CUs 102 using a scheme that allows for both symmetric and asymmetric bisection in the binary tree portion of the QTBT. In FIG. 5, the dashed lines indicate asymmetric bisection lines, and the parent CU 102 has been split using one of the split types shown in FIG.

[0024] Figure 6 shows a QTBT representation of the partition of Figure 5. In Figure 6, two solid lines extending from a node indicate a symmetric partition in the binary tree portion of the QTBT, and two dashed lines extending from a node indicate an asymmetric partition in the binary tree portion.

[0025] Syntax may be coded into the bitstream indicating how the CTU 100 was split into CUs 102. As a non-limiting example, syntax may be coded into the bitstream to indicate which nodes were split with a quadtree split, a symmetric bisection, and an asymmetric bisection. Similarly, syntax may be coded into the bitstream for nodes split using an asymmetric bisection to indicate which type of asymmetric bisection was used, such as one of the four types shown in FIG. 4.

[0026] In some embodiments, the use of asymmetric splitting can be limited to splitting CUs 102 at leaf nodes of the QTBT quadtree portion. In these embodiments, the CUs 102 of child nodes split from a parent node at the quadtree portion using quadtree splitting can be final CUs 102 or can be further split using quadtree splitting, symmetric bisection, or asymmetric bisection. The child nodes of a bisection split using symmetric bisection can be final CUs 102 or can be further split recursively one or more times using only symmetric bisection. The child nodes of a bisection split from a QT leaf node using asymmetric bisection can be final CUs 102 that are not further split.

[0027] In these embodiments, the use of asymmetric splitting can reduce search complexity and / or limit overhead bits by limiting the splitting of quadtree leaf nodes. Because only quadtree leaf nodes are split in an asymmetric split, the use of asymmetric splitting can directly indicate the end of a branch of the QT portion without other syntax or further signaling.

[0028] Similarly, because asymmetrically split nodes cannot be split any further, the use of asymmetric split on a node can also directly indicate that its asymmetrically split child nodes are final CUs 102 without other syntax or further signaling.

[0029] In alternative embodiments, such as when limiting search complexity and / or limiting the number of additional bits is less of an issue, asymmetric splitting may be used to split multiple nodes generated by quadtree splitting, symmetric bisection, and / or asymmetric bisection.

[0030] After quadtree and binary tree partitioning using any of the QTBT structures described above, the blocks represented by the leaf nodes of the QTBT indicate the final CU 102 to be coded, such as coded using inter prediction or intra prediction. For slices or full frames coded with inter prediction, different partitioning structures can be used for the luma and chroma components. For example, for inter slices, the CU 102 can have coding blocks (CBs) of different color components, such as one luma CB and two chroma CBs. For slices or full frames coded with intra prediction, the partitioning structures for the luma and chroma components are the same.

[0031] In an alternative embodiment, the JVET may use a two-level coding block structure as an alternative or extension to the QTBT partitioning described above. In the two-level coding block structure, the CTU 100 may first be partitioned into multiple base units (BUs) at a higher level. The multiple BUs may then be partitioned into multiple operating units (OUs) at a lower level.

[0032] In an embodiment employing a two-level coding block structure, at a high level, the CTU 100 may be partitioned into BUs according to one of the QTBT structures described above, or according to a quadtree (QT) structure such as that used in HEVC. A block may only be partitioned into four equal-sized sub-blocks. As a non-limiting example, the CTU 102 may be partitioned into BUs according to the QTBT structure described above with respect to Figures 5-6. The leaf nodes of the quadtree portion may be partitioned using a quadtree partition, a symmetric binary tree partition, or an asymmetric binary tree partition.

[0033] In this example, the final leaf nodes of the QTBT can be multiple BUs instead of multiple CUs. At the lower level of the two-level coding block structure, each BU split from the CTU 100 may be further split into one or more OUs. In some embodiments, if a BU is square, it may be split into multiple OUs using quadtree splitting or bipartitioning, such as symmetric or asymmetric bipartitioning. However, if a BU is not square, it may be split into multiple OUs using only bipartitioning. Limiting the types of splitting that can be used for non-square BUs may limit the number of bits used to indicate the type of splitting used to generate multiple BUs.

[0034] Although the following description describes coding of CU 102, in an embodiment using a two-level coding block structure, BUs and OUs can be coded instead of CU 102. As a non-limiting example, BUs can be used for higher level coding operations such as intra-prediction or inter-prediction, and smaller OUs can be used for lower level coding operations such as transforming and generating transform coefficients. Thus, the syntax coded for BUs indicates whether they are coded with intra-prediction or inter-prediction, or indicates information identifying a specific intra-prediction mode or motion vector used to code the BUs. Similarly, the syntax for OUs can identify a specific transform operation or quantized transform coefficients used to code the OUs.

[0035] 7 shows a simplified block diagram of CU coding in the JVET encoder. The main stages of video coding are, as described above, partitioning to identify multiple CUs 102, then encoding the multiple CUs 102 using prediction at 704 or 706 to generate residual CUs 710, transforming at 712, quantizing at 716, and entropy coding at 720.

[0036] The encoder and encoding process shown in FIG. 7 also include a decoding process, which will be described in more detail below. Given a current CU 102, the encoder may obtain a predicted CU 702 using intra prediction spatially at 704 or inter prediction temporally at 706. The basic idea of ​​predictive coding is to transmit a difference or residual signal between an original signal and a prediction of the original signal. At the receiver side, the original signal may be reconstructed by adding the residual and the prediction, as described below. The difference signal is less correlated than the original signal, so fewer bits are required for transmission.

[0037] A slice coded entirely by the intra-predicted CU 102, such as an entire image or a portion of an image, may be an "I" slice that is decoded without reference to other slices and may be a point where decoding may begin. Slices coded with at least some inter-predicted CUs may be predicted (P) slices or bi-predictive (B) slices that may be decoded based on one or more reference images. P slices may use intra and inter prediction on previously coded slices. For example, P slices can be compressed more than "I" slices using inter prediction, but require coding of previously coded slices to code them. B slices may use intra or inter prediction with interpolated prediction from two different frames to use data from slices before and / or after coding, improving the accuracy of the motion estimation process. In some cases, P and B slices may be coded or alternatively coded using intra block copies where data from other parts of the same slice are used.

[0038] As described below, intra prediction or inter prediction may be performed based on multiple CUs 734 reconstructed from multiple previously coded CUs 102, such as multiple neighboring CUs 102 or multiple CUs 102 in a reference image.

[0039] When CU 102 is spatially coded using intra prediction at 704, an intra prediction mode that optimally predicts pixels of CU 102 based on samples from neighboring CUs 102 in the image may be identified.

[0040] When coding the luma component of a CU, the encoder can generate a list of candidate intra prediction modes. Whereas HEVC had 35 possible intra prediction modes for the luma component, JVET has 67 possible intra prediction modes for the luma component. These include a planar mode that uses a three-dimensional plane of values ​​generated from neighboring pixels, a DC mode that uses values ​​averaged from neighboring pixels, and the 65 directional modes shown in Figure 8 that use values ​​copied from neighboring pixels along indicated directions.

[0041] When generating a list of candidate intra-prediction modes for the luma component of a CU, the number of candidate modes on the list depends on the size of the CU. The candidate list may include modes from a set of six most probable modes (MPM) of CU 102 identified based on a subset of the 35 modes of HEVC with the lowest Sum of Absolute Transform Difference (SATD) cost, new directional modes added to the JVET adjacent to the candidates identified from the HEVC modes, intra-prediction modes used for adjacent blocks previously coded, and a list of default modes.

[0042] When coding the chroma components of a CU, a list of candidate intra-prediction modes can be generated. The list of candidate modes includes modes generated by cross-component linear model projection from luma samples, intra-prediction modes identified in luma CB at specific array positions within a chroma block, and chroma prediction modes previously identified in neighboring blocks. The encoder identifies the candidate modes on the list with the lowest rate distortion cost and uses those intra-prediction modes when coding the luma and chroma components of the CU. Syntax can be coded into the bitstream indicating the intra-prediction modes used to code each CU 102.

[0043] After the optimal intra-prediction modes of CU 102 are selected, the encoder may use those modes to generate the predicted CU 402. If the selected mode is a directional mode, a 4-tap filter may be used to improve the accuracy of the directionality. The top or left columns or rows of the prediction block may be adjusted with a boundary prediction filter, such as a 2-tap or 3-tap filter.

[0044] The predicted CU702 may be further smoothed by a position dependent intra prediction combination (PDPC) process, which adjusts the predicted CU702 generated based on filtered samples of adjacent blocks using unfiltered samples of adjacent blocks, or adaptive reference sample smoothing using a 3-tap or 5-tap low-pass filter to process multiple reference samples.

[0045] Once CU 102 is temporally coded using inter prediction at 706, a set of motion vectors (MVs) can be identified that point to samples in reference pictures that best predict pixels of CU 102. Inter prediction exploits temporal redundancy between slices by representing the displacement of a block of pixels within a slice. The displacement is determined according to values ​​of pixels in a previous or subsequent slice through a process called motion compensation. The motion vectors and associated reference indices indicating pixel displacement relative to a particular reference picture can be provided in a bitstream to a decoder along with residuals between original and motion compensated pixels. The decoder can use the signaled motion vectors and reference indices of the residuals to reconstruct blocks of pixels in a reconstructed slice.

[0046] In JVET, motion vectors are stored with 1 / 16 pel precision, and the difference between the motion vector and a CU's predicted motion vector may be coded with 1 / 4 pel resolution or integer pel resolution.

[0047] In JVET, multiple motion vectors may be determined for multiple sub-CUs within CU 102 using techniques such as advanced temporal motion vector prediction (ATMP), spatial-temporal motion vector prediction (STMP), affine motion compensation prediction, pattern matched motion vector derivation (PMMVD), and / or bi-directional optical flow (BIO).

[0048] The encoder may use ATMVP to determine a time vector for CU 102 that points to a corresponding block in a reference image. The time vector may be determined based on motion vectors and reference images determined for previously coded neighboring CUs 102. A motion vector for each sub-CU in CU 102 may be determined using the reference block indicated by the time vector for the entire CU 102.

[0049] STMVP may determine a motion vector for a sub-CU by scaling and averaging, along with a temporal vector, motion vectors determined for neighboring blocks previously coded with inter prediction.

[0050] Affine motion compensation prediction may be used to predict a field of motion vectors for each sub-CU in the block based on two control motion vectors identified at the top corners of the block. For example, motion vectors for sub-CUs may be derived based on the top corner motion vectors identified for each 4x4 block in CU 102.

[0051] PMMVD can use bilateral matching or template matching to identify an initial motion vector for the current CU 102. Bilateral matching identifies the current CU 102 and a reference block in two different reference images along the motion trajectory, while template matching can search for corresponding blocks in the current CU 102 and a reference image identified by a template.

[0052] The initial motion vector identified for CU 102 may then be refined for each sub-CU individually. BIO may be used when performing inter prediction in bi-prediction based on previous and next reference images to identify a motion vector for a sub-CU based on the gradient of the difference between two reference images.

[0053] In some circumstances, local illumination compensation (LIC) may be used at the CU level to determine the values ​​of the scaling factor parameter and offset parameter based on samples neighboring the current CU 102 and corresponding samples neighboring the reference block identified by the candidate motion vector. In JVET, LIC parameters may be modified and signaled at the CU level. In some of the methods described above, the motion vectors determined for each sub-CU of the CU may be signaled to the decoder at the CU level. For other methods such as PMMVD and BIO, the motion information is not signaled in the bitstream to save overhead, and the decoder may derive the motion vectors in the same process.

[0054] After the motion vectors for CU 102 are identified, the encoder may use those motion vectors to generate the predictive CU 702. In some cases, if multiple motion vectors are identified for an individual sub-CU, overlapped block motion compensation (OBMC) may be used when combining those motion vectors with motion vectors previously identified for one or more neighboring sub-CUs to generate the predictive CU 702.

[0055] Using bi-prediction, JVET may identify multiple motion vectors using decoder-side motion vector refinement (DMVR). In DMVR, a bilateral template matching process may be used to identify a motion vector based on two motion vectors identified in bi-prediction. In DMVR, a weighted combination of multiple predictive CUs 702 generated with each of the two motion vectors is identified, and the two motion vectors may be refined by replacing them with a new motion vector that best represents the combined predictive CU 702.

[0056] The two refined motion vectors can be used to generate the final predicted CU 702 . As described above, once a predicted CU 702 has been identified via intra prediction at 704 or inter prediction at 706 , the encoder may subtract the predicted CU 702 from the current CU 102 at 708 to identify a residual CU 710 .

[0057] The encoder may convert 712 the residual CU 710 into transform coefficients 714 that represent the residual CU 710 in the transform domain using one or more transform operations, such as, for example, a discrete cosine block transform (DCT) to transform the data into the transform domain. JVET allows more types of transform operations than HEVC, such as DCT-II, DST-VII, DST-VII, DCT-VIII, DST-I, and DCT-V operations. The possible transform operations are grouped into subsets, and an indication of which subsets and which specific operations within those subsets were used may be signaled by the encoder. In some cases, a large block size transform is used to zero out high frequency transform coefficients in CUs 102 that are larger than a certain size, so that only low frequency transform coefficients are kept for those CUs 102.

[0058] In some cases, a mode dependent non-separable secondary transform (MDNSST) may be applied to the low frequency transform coefficients 714 after the forward core transform. The MDNSST operation may use a Hypercube-Givens Transform (HyGT) based on the rotation data. When used, an index value identifying the particular MDNSST operation may be signaled from the encoder.

[0059] At 716, the encoder may quantize the transform coefficients 714 into quantized transform coefficients 716. The quantization of each coefficient may be calculated by dividing the value of the coefficient by a quantization step derived from a quantization parameter (QP). In some embodiments, Qstep is 2 (QP-4) / 6 Quantization can aid in data compression because the high precision transform coefficients 714 can be converted to quantized transform coefficients 716, which have a finite number of possible values.

[0060] Quantization of the transform coefficients may therefore limit the amount of bits generated and transmitted by the transform process. However, quantization is a lossy operation and quantization losses cannot be recovered, but the quantization process represents a trade-off between the quality of the reconstructed sequence and the amount of information required to represent the sequence. For example, a lower QP value may result in a higher quality decoded video but may require a larger amount of data to represent and transmit. In contrast, a higher QP value may result in a lower quality reconstructed video sequence but require less data and bandwidth.

[0061] JVET may utilize a variance-based adaptive quantization technique that allows each CU 102 to use a different quantization parameter for its coding process (instead of using the same frame QP in the coding of each CU 102 of a frame). The variance-based adaptive quantization technique adaptively lowers the quantization parameter for certain blocks and increases it for other blocks. To select a particular QP for a CU 102, the variance of the CU is calculated. That is, if the variance of the CU is higher than the average variance of the frame, a higher QP than the QP of the frame may be set for the CU 102. If the CU 102 exhibits a lower variance than the average variance of the frame, a lower QP may be assigned.

[0062] At 720, the encoder may identify final compressed bits 722 by entropy coding the quantized transform coefficients 718. Entropy coding aims to remove statistical redundancy in the information being transmitted. In JVET, the quantized transform coefficients 718 may be coded using Context Adaptive Binary Arithmetic Coding (CABAC), which uses a probability measure to remove statistical redundancy. For CUs 102 with non-zero quantized transform coefficients 718, the quantized transform coefficients 718 may be converted to binary. Each bit ("bin") of the binary representation may then be coded using a context model. The CU 102 is divided into three regions, each with its own set of context models to use for the pixels in the region.

[0063] Multiple scan passes may be performed to code multiple bins. During the pass that codes the first three bins (bin0, bin1, bin2), an index value indicating which context model to use for a bin may be identified by summing that bin's position over up to five neighboring quantized transform coefficients 718 previously coded as identified by the template.

[0064] The context model can be based on the probability that the value of a bin is '0' or '1'. As values ​​are coded, the probabilities of the context model can be updated based on the number of actual '0' and '1' values. Whereas HEVC used a fixed table to reinitialize the context model for each new picture, in JVET, the probabilities of the context models for multiple new inter-predicted pictures can be initialized based on the context models generated for previously coded inter-predicted pictures.

[0065] The encoder may generate a bitstream that includes entropy coded bits 722 of the residual CUs 710, prediction information such as a selected intra prediction mode or motion vectors, an indicator of how the CUs 102 were split from the CTU 100 according to the QTBT structure, and / or other information about the encoded video. The bitstream may be decoded at a decoder, as described below.

[0066] In addition to using the quantized transform coefficients 718 to identify the final compressed bits 722, the encoder may also use the quantized transform coefficients 718 to generate the reconstructed CUs 734 by following the same decoding process that the decoder uses to generate the reconstructed CUs 734. Thus, once the transform coefficients are calculated and quantized by the encoder, the quantized transform coefficients 718 may be sent to a decoding loop in the encoder. After quantizing the transform coefficients of the CUs, the decoding loop may cause the encoder to generate the same reconstructed CUs 734 that the decoder generates in the decoding process. Thus, the encoder may use the same reconstructed CUs 734 that the decoder uses for the neighboring CUs 102 or reference images when performing intra prediction or inter prediction of the new CU 102. The reconstructed CUs 102, reconstructed slices, or a fully reconstructed frame may serve as a reference for further prediction stages.

[0067] In order to obtain pixel values ​​of a reconstructed image, an inverse quantization process may be performed in the decoding loop of the encoder (for the same operation of the decoder, see below). To inverse quantize a frame, for example, the quantized value of each pixel of the frame is multiplied by a quantization step such as Qstep described above to obtain reconstructed inverse quantized transform coefficients 726. For example, in the decoding process shown in FIG. 7 in the encoder, the quantized transform coefficients 718 of the residual CU 710 may be inverse quantized at 724 to obtain the inverse quantized transform coefficients 726. If an MDNSST operation was performed in the encoding, the operation may be reversed after inverse quantization.

[0068] At 728, the inverse quantized transform coefficients 726 may be inverse transformed to identify a reconstructed residual CU 730, such as by applying a DCT to the values ​​to obtain a reconstructed image. At 732, the reconstructed residual CU 730 may be added to the corresponding predicted CU 702 identified in the intra prediction at 704 or the inter prediction at 706 to identify a reconstructed CU 734.

[0069] At 736, one or more filters may be applied to the reconstructed data during the decoding process (at the encoder or, as described below, at the decoder), either at the picture level or at the CU level. For example, the encoder may apply a deblocking filter, a sample adaptive offset (SAO) filter, and / or an adaptive loop filter (ALF). The encoder's decoding process may implement filters that estimate optimal filter parameters that can address potential artifacts in the reconstructed image and send them to the decoder. Such improvements improve the objective and subjective quality of the reconstructed video.

[0070] In deblocking filtering, pixels near sub-CU boundaries are modified, whereas in SAO, pixels within the CTU 100 can be modified using either edge offset or band offset classification. JVET's ALF can use a circularly symmetric filter for each 2x2 block. An indication of the size and identity of the filter used for each 2x2 block can be signaled.

[0071] If the reconstructed images are reference images, then at 706 they may be stored in a reference buffer 738 for inter-prediction of future CUs 102 . In the above steps, JVET can use a content adaptive clipping operation to adjust color values ​​between upper and lower clipping bounds. The clipping bounds can be changed for each slice, and parameters identifying the bounds can be signaled in the bitstream.

[0072] FIG. 9 shows a simplified block diagram of CU coding in a JVET decoder. The JVET decoder may receive a bitstream including information about the coded CUs 102. The bitstream may indicate how the CUs 102 of an image were split from the CTU 100 according to the QTBT structure. As a non-limiting example, the bitstream may identify how the CUs 102 were split from each CTU 100 in the QTBT using a quadtree split, a symmetrical bisection, and / or an asymmetrical bisection. The bitstream may also indicate prediction information of the CUs 102, such as intra-prediction modes or motion vectors, and a number of bits 902 representing entropy-coded residual CUs.

[0073] At 904, a decoder may decode the entropy coded bits 902 using the CABAC context model signaled in the bitstream by the encoder. The decoder may use the parameters signaled by the encoder to update the probabilities of the context model in the same manner as they were updated during encoding.

[0074] After reversing the entropy coding at 904 to determine quantized transform coefficients 906, the decoder may inverse quantize them at 908 to determine inverse quantized transform coefficients 910. If an MDNSST operation was performed in the encoding, that operation may be reversed by the decoder after inverse quantization.

[0075] At 912, the dequantized transform coefficients 910 may be inverse transformed to identify a reconstructed residual CU 914. At 916, the reconstructed residual CU 914 may be added to a corresponding predicted CU 926 identified in the intra prediction at 922 or the inter prediction at 924 to identify a reconstructed CU 918.

[0076] At 920, one or more filters may be applied to the reconstructed data, either at the picture level or at the CU level. For example, the decoder may apply a deblocking filter, a sample adaptive offset (SAO) filter, and / or an adaptive loop filter (ALF). As described above, an in-loop filter in the encoder's decoding loop may be used to estimate optimal filter parameters to improve the objective and subjective quality of the frame. At 920, these parameters are sent to the decoder to filter the reconstructed frame to match the filtered reconstructed frame in the encoder.

[0077] After the reconstructed images are generated by applying the filters signaled to identify the reconstructed CUs 918, the decoder may output the reconstructed images as output video 928. If the reconstructed images are used as reference images, they may be stored 924 in a reference buffer 930 for inter prediction of future CUs 102.

[0078] Figure 10 shows an embodiment of a method of CU coding 1000 in a JVET decoder. In the embodiment shown in Figure 10, an encoded bitstream 902 may be received in step 1002, a CABAC context model associated with the encoded bitstream 902 may be determined in step 1004, and then the encoded bitstream 902 may be decoded in step 1006 using the determined CABAC context model.

[0079] At step 1008 , a plurality of quantized transform coefficients 906 associated with the encoded bitstream 902 may be determined, and at step 1010 , inverse quantized transform coefficients 910 may be determined from the plurality of quantized transform coefficients 906 .

[0080] In step 1012, it may be determined whether an MDNSST operation was performed during encoding and / or whether the bitstream 902 includes an indication that an MDNSST operation was applied to the bitstream 902. If it is determined that an MDNSST operation was performed during the encoding process or that the bitstream 902 includes an indication that an MDNSST operation was applied to the bitstream 902, an inverse MDNSST operation 1014 may be performed on the bitstream 902 in step 1016 before the inverse transform operation 912 is performed. Alternatively, if there is no application of an inverse MDNSST operation in step 1014, then in step 1016, the inverse transform operation 912 may be performed on the bitstream 902. The inverse transform operation 912 in step 1016 may determine and / or construct a reconstructed residual CU 914.

[0081] In step 1018, the reconstructed residual CU 914 from step 1016 may be combined with a predicted CU 918. The predicted CU 918 may be one of the intra-prediction CU 922 determined in step 1020 and the inter-prediction unit 924 determined in step 1022.

[0082] In step 1024, any one or more filters 920 may be applied to the reconstructed CU 914 and output in step 1026. In some embodiments, filters 920 may not be applied in step 1024.

[0083] In some embodiments, in step 1028, the reconstructed CU 918 may be stored in a reference buffer 930. FIG. 11 shows a simplified block diagram 1100 of CU coding in a JVET encoder. In step 1102, the JVET coding tree unit may be represented as a root node of a quadtree plus binary tree (QTBT) structure. In some embodiments, the QTBT may have a quadtree branching from the root node and / or a binary tree branching from one or more leaf nodes of the quadtree. The representation from step 1102 may proceed to steps 1104, 1106 or 1108.

[0084] In step 1104, an asymmetric bisection may be used to divide the represented quadtree node into two blocks of unequal size. In some embodiments, the divided blocks may be represented in a binary tree branching from the quadtree node as leaf nodes representing final coding units. In some embodiments, the binary tree branching from the quadtree node as leaf nodes indicates final coding units for which further division is not permitted. In some embodiments, the asymmetric bisection divides the coding unit into blocks of unequal size, with a first block representing 25% of the quadtree node and a second block representing 75% of the quadtree node.

[0085] In step 1106, a quadtree partition may be used to partition the represented quadtree node into four square blocks of equal size. In some embodiments, the partitioned blocks may be represented as quadtree nodes representing final coding units, or as child nodes that are partitioned again by a quadtree partition, a symmetric binary tree partition, or an asymmetric binary tree partition.

[0086] In step 1108, a quadtree split may be used to split the represented quadtree node into two blocks of equal size. In some embodiments, the split blocks may be represented as quadtree nodes representing final coding units, or as child nodes that are split again by a quadtree split, a symmetric binary tree split, or an asymmetric binary tree split.

[0087] In step 1110, the child nodes from step 1106 or step 1108 may be represented as child nodes configured to be encoded. In some embodiments, the child nodes may be represented by leaf nodes of a binary tree in the JVET.

[0088] In step 1112, multiple coding units from steps 1104 or 1110 may be encoded using JVET. Figure 12 shows a simplified block diagram 1200 of CU decoding in a JVET decoder. In the embodiment shown in Figure 12, a bitstream may be received in step 1202 indicating how a coding tree unit has been split into multiple coding units according to a QTBT structure. The bitstream may indicate how the quadtree nodes are split in at least one of a quadtree split, a symmetric bisection, or an asymmetric bisection.

[0089] In step 1204, a number of coding units represented by a number of leaf nodes of the QTBT structure may be identified. In some embodiments, the number of coding units may indicate whether the node was split from a quadtree leaf node using an asymmetric bisection. In some embodiments, the coding unit may indicate that the node represents a final coding unit to be decoded.

[0090] In step 1206, the identified one or more coding units may be decoded using the JVET. FIG. 13 illustrates an alternative simplified block diagram 1300 of JVET coding for intra mode prediction. In the embodiment illustrated in FIG. 13, a set of MPMs may be identified and instantiated in memory at step 1302, a set of 16 selected modes may be identified and instantiated in memory at step 1304, and a balance of 67 modes may be defined and instantiated in memory at step 1304. In some embodiments, the set of MPMs may be reduced from a standard set of 6 MPMs. In some embodiments, the set of MPMs may include 5 unique modes, the selected modes may include 16 eigenmodes, and the set of non-selected modes may include the remaining 46 non-selected eigenmodes. However, in alternative embodiments, the set of MPMs may include fewer unique modes, the selected modes may remain fixed at 16 unique modes, and the set size of the non-selected eigenmodes may be appropriately adjusted to accommodate a total of 67 modes. As a non-limiting example, in some embodiments where the set of MPMs includes five unique modes instead of six MPMs, a truncated unary binarization is used, and if a new binarization for the five MPMs is utilized, the number of bins assigned to the MPM modes may be equal to or less than five bins.

[0091] Thus, in some embodiments, 16 selected modes from the 62 remaining intra modes are generated by uniformly subsampling these 62 intra modes, each coded with a 4-bit fixed length code. As a non-limiting example, assuming the remaining 62 modes are indexed as {0, 1, 2, ..., 61}, then the 16 selected modes = {0, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60}. The remaining 46 unselected modes = {1, 2, 3, 5, 6, 7, 9, 10 ... 59, 61}, and such 46 unselected modes may be coded with a truncated binary code.

[0092] Figure 14 illustrates an alternative JVET coding table 1400 for intra mode prediction according to Figure 13. In the embodiment illustrated in Figure 14, a number of intra prediction modes 1402 are shown to include 5 MPMs, 16 selected modes, and 46 non-selected modes, where a number of bin strings 1404 for the MPMs may be coded using truncated unary binarization, the 16 selected modes may be coded using 4 bits of a fixed length code, and the 46 non-selected modes may be coded using truncated binary coding.

[0093] In the alternative embodiment of Figure 13, six MPMs are available, but only the first five MPMs in the MPM list are binarized as shown in Figure 14 and coded in a manner based on the current context as described in the current JVET. The sixth MPM in the MPM list is considered as one of the 16 selection modes and is coded with a 4-bit fixed length code along with the other 15 selection modes.

[0094] As a non-limiting example, if the remaining 61 modes are indexed as {0, 1, 2, ..., 60}, the 15 selected modes may be obtained by evenly subsampling the remaining 61 intra modes as follows: The set of 15 selected modes may be {0, 5, 10, 14, 18, 22, 26, 30, 34, 38, 42, 46, 50, 55, 60}, where the 15 selected modes plus the 6th The MPMs are coded with 4 bits of a fixed length code as the set {6th MPM, 0, 5, 10, 14, 18, 22, 26, 30, 34, 38, 42, 46, 50, 55, 60}, and the balance of the 46 non-selected modes are denoted as the set of non-selected modes = {1, 2, 3, 4, 6, 7, 8, 9, 11, 12 ... 49, 51, 52, 53, 54, 56, 57, 58, 59} and coded with a truncated binary code.

[0095] In a further alternative embodiment of FIG. 13, only the first five MPMs of the MPM list are binarized as shown in FIG. 14 and coded in a manner based on the current context as described in the current JVET standard. In such an embodiment, the sixth MPM of the MPM list is considered as one of the 16 selected modes and coded with a 4-bit fixed length code along with the other 15 selected modes. Thus, the selection of the other 15 selected modes may be established using any known convenient and / or desired selection process. As non-limiting examples, they may be selected with respect to the MPM mode, or with respect to (content-based) statistically well-known modes, or with respect to trained or historically well-known modes, or using other methods or processes.

[0096] Again, the selection of five MPMs is merely a non-limiting example, and in alternative embodiments, the set of MPMs may be further reduced to four or three MPMs, or expanded to more than six. There are still 16 selected modes, and the balance of the 67 (or other known, convenient, and / or desired total number) intra-coding modes is included in the set of non-selected intra-coding modes. That is, embodiments are contemplated in which the total number of intra-coding modes is greater or less than 67, in which the set of MPMs includes any known, convenient, or desired number of MPMs, and the amount of selected modes may be any known, convenient, and / or desired amount.

[0097] Execution of the sequences of instructions necessary to implement the embodiments may be performed by a computer system 1500 as shown in Figure 15. In one embodiment, execution of the sequences of instructions is performed by a single computer system 1500. According to other embodiments, multiple computer systems 1500 connected by communication links 1515 may execute the sequences of instructions in coordination with each other. Although the following provides a description of only one computer system 1500, it should be understood that any number of computer systems 1500 may be used to implement the embodiments.

[0098] A computer system 1500 according to one embodiment will now be described with reference to Figure 15, which is a block diagram of several functional components of computer system 1500. As used herein, the term computer system 1500 is used broadly to describe any computing device capable of storing and independently executing one or more programs.

[0099] Each computer system 1500 may include a communication interface 1514 coupled to the bus 1506. The communication interface 1514 provides two-way communication between the multiple computer systems 1500. The communication interface 1514 of each computer system 1500 sends and receives electrical, electromagnetic, or optical signals that include data streams representing various types of signal information, such as commands, messages, and data. The communication link 1515 links one computer system 1500 with another computer system 1500. For example, the communication link 1515 is a LAN, in which case the communication interface 1514 is a LAN card, or the communication link 1515 is a PSTN, in which case the communication interface 1514 is an integrated services digital network (ISDN) card or modem, or the communication link 1515 is the Internet, in which case the communication interface 1514 may be a dial-up, cable, or wireless modem.

[0100] Computer system 1500 may transmit and receive messages, data, and instructions, including programs, i.e., applications, code, via its corresponding communications link 1515 and communications interface 1514. Received program code may be executed by the respective processor 1507 upon receipt and / or stored in storage device 1510 or other associated non-volatile medium for later execution.

[0101] In one embodiment, computer system 1500 operates in conjunction with a data storage system 1531, e.g., data storage system 1531 including a database 1532 readily accessible by computer system 1500. Computer system 1500 communicates with data storage system 1531 via a data interface 1533. Data interface 1533 coupled to bus 1506 sends and receives electrical, electromagnetic, or optical signals including data streams representing various types of signal information, e.g., commands, messages, and data. In various embodiments, the functions of data interface 1533 may be performed by communication interface 1514.

[0102] Computer system 1500 includes a bus 1506 or other communication mechanism for communicating instructions, messages and data, collectively information, and one or more processors 1507 coupled to bus 1506 for processing information. Computer system 1500 also includes a main memory 1508, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1506 for storing dynamic data and instructions executed by the one or more processors 1507. Main memory 1508 may also be used for storing temporary data, i.e., variables, or other intermediate information during execution of instructions by the one or more processors 1507.

[0103] The computer system 1500 may further include a read-only memory (ROM) 1509 or other static storage device coupled to the bus 1506 for storing static data and instructions for the one or more processors 1507. A storage device 1510, such as a magnetic disk or optical disk, may also be provided and coupled to the bus 1506 for storing data and instructions for the one or more processors 1507.

[0104] Computer system 1500 may be connected via bus 1506 to a display device 1511, such as, but not limited to, a cathode ray tube (CRT) or liquid-crystal display (LCD) monitor, for displaying information to a user. An input device 1512, such as alphanumeric and other keys, is connected to bus 1506 for communicating information and command selections to processor 1507.

[0105] According to one embodiment, each computer system 1500 performs certain operations by its corresponding one or more processors 1507 executing one or more sequences of one or more instructions contained in main memory 1508. Such instructions may be read into main memory 1508 from another computer-usable medium, such as ROM 1509 or storage device 1510. Execution of the sequences of instructions contained in main memory 1508 causes the one or more processors 1507 to perform the processes described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and / or software.

[0106] The term "computer usable medium" as used herein refers to any medium that provides information or can be used by one or more processors 1507. Such media can have many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media, i.e., media that can retain information without electrical power, include ROM 1509, CD ROM, magnetic tape, and magnetic disks. Volatile media, i.e., media that cannot retain information without electrical power, includes main memory 1508. Transmission media include coaxial cables, copper wire, and optical fibers, including the wires that comprise bus 1506. Transmission media can also be in the form of carrier waves, i.e., electromagnetic waves that are modulated in frequency, amplitude, or phase to transmit information signals. Further, transmission media can be in the form of acoustic or light waves, such as those generated during radio wave or infrared data communications.

[0107] In the foregoing specification, the embodiments have been described with reference to specific components thereof. However, it will be apparent that various changes and modifications can be made without departing from the broader spirit and scope of the embodiments. For example, it will be understood that the specific ordering and combination of process actions shown in the process flow diagrams described herein are merely exemplary, and that the embodiments can be implemented using different or additional process actions, or using different combinations or orderings of process actions. Thus, the specification and drawings should be considered in an illustrative and not restrictive sense.

[0108] It should also be noted that the present invention can be implemented in a variety of computer systems. The various techniques described herein may be embodied in hardware or software, or a combination of both. Preferably, these techniques are embodied in programmable computer programs running on a number of computers, each including a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The program code is applied to data entered using the input device to perform the functions described above and generate output information. The output information is applied to one or more output devices. Each program is preferably implemented in a general procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be embodied in assembly or machine language. In either case, the language may be a compiled or interpreted language. Each such computer program is preferably stored in a general purpose or special purpose programmable computer-readable storage medium or device (e.g., ROM or magnetic disk) for configuring and operating a computer when the storage medium or device is read by the computer to perform the procedures described above. The system is also contemplated as embodied as a computer readable storage medium configured to have a computer program thereon, the storage medium so configured causing the computer to operate in a particular, predetermined manner. Furthermore, the storage element of an exemplary computing application may be a relational or sequential (flat file) type computing database capable of storing data in various combinations and configurations.

[0109] FIG. 16 is a schematic diagram of a source device 1612 and a destination device 1610 incorporating features of the systems and devices described herein. As shown in FIG. 16, the exemplary video coding system 1610 includes a source device 1612 and a destination device 1616, where in this example, the source device 1612 generates encoded video data. Thus, the source device 1612 may be referred to as a video encoder. The destination device 1616 may decode the encoded video data generated by the source device 1612. Thus, the destination device 1616 may be referred to as a video decoder. The source device 1612 and the destination device 1616 may be examples of video coding devices.

[0110] The destination device 1616 may receive the encoded video data from the source device 1612 over the channel 1616. The channel 1616 may comprise any type of medium or device that can move the encoded video data from the source device 1612 to the destination device 1616. In one example, the channel 1616 may comprise a communication medium that allows the source device 1612 to transmit the encoded video data directly to the destination device 1616 in real time.

[0111] In this example, the source device 1612 may modulate the encoded video data according to a communication standard, such as a wireless communication protocol, and transmit the modulated video data to the destination device 1616. The communication medium may include a wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or other equipment that enables communication from the source device 1612 to the destination device 1616. In another example, the channel 1616 may correspond to a storage medium that stores the encoded video data generated by the source device 1612.

[0112] 16, source device 1612 includes a video source 1618, a video encoder 1620, and an output interface 1622. In some cases, output interface 1628 may include a modulator / demodulator (modem) and / or a transmitter. In source device 1612, video source 1618 may include sources such as a video capture device such as a video camera, a video archive containing previously captured video data, a video feed interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of such sources.

[0113] The video encoder 1620 may encode captured, pre-captured, or computer-generated video data. Input images may be received by the video encoder 1620 and stored in an input frame memory 1621. A general-purpose processor 1623 may load information therefrom and perform the encoding. Programs for driving the general-purpose processor may be loaded from a storage device, such as the exemplary memory module shown in FIG. 16. The general-purpose processor performs the encoding using the processing memory 1622, and output of the encoded information by the general-purpose processor may be stored in a buffer, such as an output buffer 1626.

[0114] The video encoder 1620 may include a resampling module 1625 configured to code (e.g., encode) the video data in a scalable video coding scheme that defines at least one base layer and at least one enhancement layer. The resampling module 1625 may resample at least some of the video data as part of the encoding process, where the resampling may be performed adaptively using a resampling filter.

[0115] The encoded video data, e.g., a coded bitstream, may be transmitted directly to a destination device 1616 via an output interface 1628 of the source device 1612. In the example of FIG. 16, the destination device 1616 includes an input interface 1638, a video decoder 1630, and a display device 1632. In some cases, the input interface 1628 may include a receiver and / or a modem. The input interface 1638 of the destination device 1616 receives the encoded video data via the channel 1616. The encoded video data may include various syntax elements generated by the video encoder 1620 that represent the video data. Such syntax elements may be included in the encoded video data that is transmitted over a communication medium, stored on a storage medium, or stored on a file server.

[0116] The encoded video data may also be stored on a storage medium or file server for later access by the destination device 1616 for decoding and / or playback. For example, the coded bitstream may be temporarily stored in an input buffer 1631 and then loaded into a general purpose processor 1633. Programs for driving the general purpose processor may be loaded from a storage device or memory. The general purpose processor may perform the decoding using a process memory 1632. The video decoder 1630 may also include a resampling module 1635 similar to the resampling module 1625 used in the video encoder 1620.

[0117] 16 shows the resampling module 1635 separate from the general purpose processor 1633, those skilled in the art will appreciate that the resampling function is performed by a program executed by a general purpose processor, and that processing in a video encoder is accomplished using one or more processors. The decoded image or images may be stored in an output frame buffer 1636 and then sent to an input interface 1638.

[0118] The display device 1638 may be integrated with the destination device 1616 or may be external. In some examples, the destination device 1616 may include an integrated display device or be configured to interface with an external display device. In other examples, the destination device 1616 may be a display device. In general, the display device 1638 displays the decoded video data to a user.

[0119] The video encoder 1620 and the video decoder 1630 may operate according to a video compression standard. ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) are currently studying the potential need for standardization of future video coding techniques with compression capabilities significantly beyond those of the current High Efficiency Video Coding HEVC standard (including its current and near-future extensions for screen content coding and high dynamic range coding). These groups are working together in this research effort in a joint collaboration known as the Joint Video Exploration Team (JVET) to evaluate compression technology designs proposed by their experts in this field. A recent record of JVET development is described in Algorithm Description of Joint Exploration Test Model 5 (JEM 5), JVET-E1001-V2, by J. Chen, E. Alshina, G. Sullivan, J. Ohm, and J. Boyce.

[0120] Additionally or alternatively, the video encoder 1620 and the video decoder 1630 may operate according to other proprietary or industry standards that work with the disclosed JVET features, i.e., the ITU-T H.264 standard, alternatively MPEG-4, Part 10, Advanced Video Coding (AVC), or other standards such as extensions of such standards. Thus, the techniques of this disclosure are not limited to any particular coding standard or technique, although they are newly developed for JVET. Other examples of video compression standards and techniques include MPEG-2, ITU-T H.263, and proprietary or open source compression and related formats.

[0121] The video encoder 1620 and the video decoder 1630 may be embodied in hardware, software, firmware, or any combination thereof. For example, the video encoder 1620 and the decoder 1630 may use one or more processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, or any combination thereof.

[0122] If the video encoder 1620 and decoder 1630 are embodied partially in software, the device may store the software instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of the video encoder 1620 and video decoder 1630 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (codec) within the respective device.

[0123] Aspects of the subject matter described herein may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer, such as the general purpose processors 1623 and 1633 described above. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Aspects of the subject matter described herein may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0124] Examples of memory include random access memory (RAM), read-only memory (ROM), or both. The memory may store instructions, such as source code or binary code, for performing the techniques described above. The memory may also be used to store variables or other intermediate information during execution of instructions to be executed by a processor, such as processors 1623 and 1633.

[0125] The storage device may store a plurality of instructions, such as source code or binary code, for carrying out the techniques described above. The storage device may further store data used and manipulated by the computer processor. For example, the storage device in the video encoder 1620 or the video decoder 1630 may be a database accessed by the computer system 1623 or 1633.

[0126] Other examples of storage devices include random access memory (RAM), read-only memory (ROM), a hard drive, a magnetic disk, an optical disk, a CD-ROM, a DVD, a flash memory, a USB memory card, or any other medium from which a computer can read.

[0127] The memory or storage device may be an example of a non-transitory computer-readable storage medium for use by or in connection with a video encoder and / or decoder. The non-transitory computer-readable storage medium includes instructions for controlling a computer system configured to perform functions described by certain embodiments. The instructions, when executed by one or more computer processors, may be configured to perform those described in certain embodiments.

[0128] Also, it should be noted that some embodiments are described as processes that are shown as flow diagrams or block diagrams. Although each has been described as sequential operations, many of the operations may be performed in parallel or simultaneously. Additionally, the order of the operations may be rearranged. The processes may have additional steps not included in the drawings.

[0129] Certain embodiments may be implemented in a non-transitory computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, system or machine. The computer-readable storage medium includes instructions for controlling a computer system to perform the methods described by certain embodiments. The computer system may include one or more computing devices. When executed by one or more computer processors, the instructions may be configured to perform those described in certain embodiments.

[0130] As used in this description and throughout the claims which follow, "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Also, as used in this description and throughout the claims which follow, the meaning of "in" includes "in" and "on," unless the context clearly dictates otherwise.

[0131] Although the exemplary embodiments of the present invention have been described in detail and language specific to the structural features and / or methodological operations set forth above, those skilled in the art will readily appreciate that many additional modifications are possible in the exemplary embodiments without substantially departing from the novel teachings and advantages of the present invention. Moreover, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or operations set forth above. Accordingly, these and all such modifications are intended to be included within the scope of the present invention, which is to be construed in breadth and scope in accordance with the appended claims.

Claims

1. 1. A method for decoding video data from a bitstream, comprising: (a) receiving the bitstream indicating how a coding tree unit has been partitioned into a plurality of coding units, where the plurality of coding units are rectangular; (b) determining a first set of Most Probable Modes (MPMs) for the current block of video data, wherein the first set of MPMs is selectable based on an MPM index, one of the first set of MPMs selectable based on the MPM index includes a true horizontal mode, another of the first set of MPMs selectable based on the MPM index includes a true vertical mode, and another of the first set of MPMs selectable based on the MPM index includes an angular mode, wherein the first set of MPMs includes only five distinct modes; (c) deriving from the bitstream (i) an MPM flag including a total of one bit and (ii) a further index, where at least one of the MPM flag and the further index indicates whether an intra mode for predicting the current block is one of the first set of MPMs; (d) if at least one of the MPM flag and the further index is used to indicate that the intra mode for predicting the current block is one of the first set of MPMs selectable based on the MPM index, selecting an intra mode for the current block based on the MPM index decoded from the bitstream of one of the first set of MPMs; (e) if at least one of the MPM flag and the further index indicates that the intra mode for predicting the current block is not one of the first set of MPMs, determining (i) at least one mode of a second set, and (ii) at least one mode of a third set, according to the MPM flag and the further index; (f) wherein the first set, the second set, and the third set include different modes, and the combination of the first set, the second set, and the third set includes 67 different modes; (g) determining an intra mode for the current block for at least one mode in the second set based on a first combination of the MPM flag and the further index that does not include any of the first set of MPMs selectable based on the MPM index included in the first set of MPMs; (h) determining an intra mode for the current block for at least one mode in the third set based on a second combination of the MPM flag and the further index that does not include any of the first set of MPMs selectable based on the MPM index included in the first set of MPMs.

2. 1. A bitstream of compressed video data for decoding by a decoder, the bitstream including a computer readable storage medium storing compressed video data, the bitstream comprising: (a) the bitstream includes data indicating how a coding tree unit is partitioned into a plurality of coding units, the plurality of coding units being rectangular; (b) the bitstream includes data suitable for determining a first set of Most Probable Modes (MPMs) for a current block of the video data, wherein the first set of MPMs is selectable based on an MPM index, one of the first set of MPMs selectable based on the MPM index includes a true horizontal mode, another of the first set of MPMs selectable based on the MPM index includes a true vertical mode, and another of the first set of MPMs selectable based on the MPM index includes an angular mode, and the first set of MPMs includes only five distinct modes; (c) the bitstream includes data suitable for deriving from the bitstream (i) an MPM flag including a total of one bit and (ii) a further index, where at least one of the MPM flag and the further index indicates whether an intra mode for predicting the current block is one of the first set of MPMs; (d) the bitstream includes data suitable for selecting an intra mode for the current block based on the MPM index decoded from the bitstream of one of the first set of MPMs when at least one of the MPM flag and the further index is used to indicate that the intra mode for predicting the current block is one of the first set of MPMs selectable based on the MPM index; (e) the bitstream includes data suitable for determining, when at least one of the MPM flag and the further index indicates that the intra mode for predicting the current block is not one of the first set of MPMs, (i) at least one mode of a second set, and (ii) at least one mode of a third set, according to the MPM flag and the further index; (f) wherein the first set, the second set, and the third set include different modes, and the combination of the first set, the second set, and the third set includes 67 different modes; (g) the bitstream includes data suitable for determining an intra mode of the current block for at least one mode of the second set based on a first combination of the MPM flag and the further index that does not include any of the first set of MPMs selectable based on the MPM index included in the first set of MPMs; (h) the bitstream includes data suitable for determining an intra mode of the current block for at least one mode of the third set based on a second combination of the MPM flag and the other index that does not include any of the first set of MPMs selectable based on the MPM index included in the first set of MPMs.

3. 1. A method for encoding video data by an encoder, comprising: (a) providing a bitstream indicating how a coding tree unit has been partitioned into a plurality of coding units, the plurality of coding units being rectangular; (b) the bitstream includes data suitable for determining a first set of Most Probable Modes (MPMs) for a current block of the video data, wherein the first set of MPMs is selectable based on an MPM index, one of the first set of MPMs selectable based on the MPM index includes a true horizontal mode, another of the first set of MPMs selectable based on the MPM index includes a true vertical mode, and another of the first set of MPMs selectable based on the MPM index includes an angular mode, and the first set of MPMs includes only five distinct modes; (c) the bitstream includes data suitable for deriving from the bitstream an MPM flag and another index including a total of one bit, where at least one of the MPM flag and the other index indicates whether an intra mode for predicting the current block is one of the first set of MPMs; (d) the bitstream includes data suitable for selecting an intra mode for the current block based on the MPM index decoded from the bitstream of one of the first set of MPMs when at least one of the MPM flag and the further index is used to indicate that the intra mode for predicting the current block is one of the first set of MPMs selectable based on the MPM index; (e) the bitstream includes data suitable for determining, when at least one of the MPM flag and the further index indicates that the intra mode for predicting the current block is not one of the first set of MPMs, (i) at least one mode of a second set, and (ii) at least one mode of a third set, according to the MPM flag and the further index; (f) wherein the first set, the second set, and the third set include different modes, and the combination of the first set, the second set, and the third set includes 67 different modes; (g) the bitstream includes data suitable for determining an intra mode of the current block for at least one mode of the second set based on a first combination of the MPM flag and the further index that does not include any of the first set of MPMs selectable based on the MPM index included in the first set of MPMs; (h) the bitstream includes data suitable for determining an intra mode of the current block for at least one mode in the third set based on a second combination of the MPM flag and the further index that does not include any of the first set of MPMs selectable based on the MPM index included in the first set of MPMs.

Images