History-based Motion Vector Prediction
By using a method based on historical motion vector predictors in video encoding and decoding, the problem of increasing demand for high-resolution video bandwidth and low encoding and decoding efficiency is solved, and more efficient video compression and programming is achieved Decoding performance.
Patent Information
- Application Number
- CN202080025301.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-03-27
- Filing Date
- 2020-03-26
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-03-26
AI Technical Summary
When existing video encoding and decoding technologies process high-resolution videos, bandwidth requirements increase and codec efficiency is low, making it difficult to effectively utilize historical motion vector information to improve compression performance.
Using a method based on historical motion vector predictor (HMVP), the motion vector prediction table is generated and used to optimize the conversion process between the video area and the bitstream by initializing and selecting multiple stored MVP tables, using the previously coded coded Move information to improve encoding and decoding efficiency.
It improves the compression performance of video encoding and codec, reduces bandwidth requirements, and enhances the encoding and codec efficiency, and is suitable for existing and future video encoding and codec standards.
Smart Images

Figure CN113647108B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is filed in order to timely claim the priority and benefits of International Patent Application No. PCT / CN2019 / 079826 filed on March 27, 2019, in accordance with the applicable patent laws and / or the Paris Convention. The entire disclosure of International Patent Application No. PCT / CN2019 / 079826 is incorporated by reference as part of the disclosure of this application. Technical Field
[0003] This patent document relates to video coding and decoding technology, devices and systems. Background Art
[0004] Despite advances in video compression, digital video still accounts for the largest amount of bandwidth used on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, the demand for bandwidth used by digital video is expected to continue to grow. Summary of the invention
[0005] Devices, systems and methods are described that are related to digital video encoding and decoding, and in particular to using a history-based motion vector predictor (HMVP). The methods described can be applied to existing video encoding and decoding standards (e.g., High Efficiency Video Codec (HEVC)) and future video encoding and decoding standards or video codecs.
[0006] In one representative aspect, a method of video processing is disclosed. The method includes initializing a motion vector predictor (MVP) table from a plurality of stored MVP tables during a conversion between a video region and a bitstream representation of the video region. Each of the plurality of MVPs includes motion information previously considered during the conversion. The conversion is performed using an MVP table selected from the plurality of stored MVP tables.
[0007] In another exemplary aspect, another method of video processing is disclosed. The method includes initializing one or more entries of a motion vector predictor (MVP) table from one or more stored MVP tables during a conversion between a video region and a bitstream representation of the video region, wherein each MVP table includes motion information previously considered during the conversion, a flag indicating that the motion information is inherited from a previous MVP table or added as new motion information to the current region; and performing the conversion using an MVP table selected from the plurality of stored MVP tables.
[0008] In another exemplary aspect, another method of video processing is disclosed. The method includes, for a first region of a video and a conversion between a bitstream representation of the first region, initializing a first historical motion vector predictor (HMVP) table associated with the first region based on at least one of a plurality of HMVP tables stored, the plurality of HMVP tables being used to store motion information associated with a region encoded and decoded before the first region; and performing the conversion by using the initialized first HMVP table including at least one motion candidate.
[0009] In another exemplary aspect, another method of video processing is disclosed. The method includes converting between a first region of a video and a bitstream representation of the first region, generating a new historical motion vector predictor (HMVP) table associated with the first region based on a plurality of HMVP tables associated with other regions, the plurality of HMVP tables being used to store motion information associated with other regions encoded and decoded before the first region; and performing the conversion by using the generated new HMVP table including at least one motion candidate.
[0010] In yet another representative aspect, the above method is embodied in the form of processor executable code and stored in a computer readable program medium.
[0011] In yet another representative aspect, a device configured or operable to perform the above method is disclosed. The device may include a processor programmed to implement the method.
[0012] In yet another representative aspect, a video decoder device may implement a method as described herein.
[0013] The above and other aspects and features of the disclosed technology are described in more detail in the drawings, the description and the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 An example of constructing a merge candidate list is shown.
[0015] Figure 2 Examples of locations of spatial candidates are shown.
[0016] Figure 3 An example of candidate pairs subject to redundancy checking for spatial merge candidates is shown.
[0017] Figure 4A and 4B An example of the position of the second prediction unit (PU) based on the size and shape of the current block is shown.
[0018] Figure 5 An example of motion vector scaling for temporal merge candidates is shown.
[0019] Figure 6 Examples of candidate positions for time-domain merge candidates are shown.
[0020] Figure 7 An example of generating a combined bi-predictive merge candidate is shown.
[0021] Figure 8 An example of constructing motion vector prediction candidates is shown.
[0022] Fig. 9 An example of motion vector scaling for spatial motion vector candidates is shown.
[0023] Figures 10A-10B A diagram showing partitioning of a CU into two triangular prediction units (two partitioning modes) is shown.
[0024] Fig.11 Example locations of neighboring blocks are shown.
[0025] Fig.12 An example of adjacent blocks (A and L) used for context selection in TPM flag encoding and decoding is shown.
[0026] Fig.13A An example of a 4-parameter affine model is shown.
[0027] Fig. 13B An example of a 6-parameter affine model is shown.
[0028] Fig.14 An example of the affine MVF for each sub-block is shown.
[0029] Fig.15A and 15B Examples of a simplified 4-parameter affine model and a simplified 6-parameter affine model are shown respectively.
[0030] Fig.16 An example of motion vector prediction for AF_INTER inheriting affine candidates is shown.
[0031] Fig.17 An example of motion vector prediction of AF_INTER for constructing affine candidates is shown.
[0032] Fig.18A and 18B Example candidates for the AF_MERGE affine motion mode are shown.
[0033] Fig.19 Examples of candidate positions for the affine merge mode are shown.
[0034] Fig. 20 An example of intra picture block copying is shown.
[0035] Fig.21 Example candidate positions for the affine merge mode are shown.
[0036] Fig. 22 The modified merge list construction process is shown.
[0037] Fig.23 An example of merging shared nodes is shown.
[0038] Fig.24 Examples of ternary tree TT and binary tree BT partitions that are not allowed are shown.
[0039] Fig.25 An example of how new motion information can be constructed from neighboring motion information is shown.
[0040] Fig.26 An example of how motion information can be constructed from neighboring motion information is shown.
[0041] Fig. 27 A flow chart of yet another example method for video processing is shown.
[0042] Fig.28 is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described in this document.
[0043] Fig.29 A flow chart of yet another example method for video processing is shown.
[0044] Fig.30 A flow chart of yet another example method for video processing is shown. DETAILED DESCRIPTION
[0045] This document relates to digital video encoding and decoding. The section headings used in this document are only for ease of understanding and do not limit the scope of the disclosed embodiments in any way. In addition, although the terminology from the current working draft of the H.266 standard is used as a reference, the disclosed technology is equally applicable to other video codec technologies.
[0046] 1. Overview
[0047] Video coding methods and techniques are ubiquitous in modern technology due to the growing demand for high-resolution video. Video codecs typically include electronic circuits or software that compress or decompress digital video, and are constantly being improved to provide greater codec efficiency. Video codecs convert uncompressed video to a compressed format and vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bitrate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end latency (delay). The compression format typically conforms to a standard video compression specification, such as the High Efficiency Video Codec (HEVC) standard (also known as H.265 or MPEG-H Part 2), the to-be-completed General Video Codec standard, or other current and / or future video codec standards.
[0048] 2. Preliminary Discussion
[0049] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced H.261 and H.263, ISO / IEC produced MPEG-1 and MPEG-4 Visual, and the two organizations worked together to produce the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / HEVC standards. Starting with H.262, video codec standards are based on a hybrid video codec structure that utilizes temporal prediction and transform codecs. To explore future video codec technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and introduced them into a reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Team (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) was established to work on the VVC standard, which reduces the bit rate by 50% compared to HEVC.
[0050] Embodiments of the disclosed technology can be applied to existing video codec standards (e.g., HEVC, H.265) and future standards to improve compression performance. Section headings are used in this document to improve the readability of the description and do not in any way limit the discussion or embodiments (and / or implementation methods) to the corresponding sections.
[0051] 2.1 Inter-frame prediction in HEVC / H.265
[0052] Each inter-predicted PU has motion parameters for one or two reference picture lists. The motion parameters include motion vectors and reference picture indices. The use of one of the two reference picture lists can also be signaled using inter_pred_idc. Motion vectors can be explicitly encoded as deltas relative to the predictor.
[0053] When a CU is encoded or decoded in skip mode, one PU is associated with the CU and there are no significant residual coefficients, encoded motion vector increments or reference picture indices. A merge mode is specified, whereby the motion parameters of the current PU are obtained from neighboring PUs including spatial and temporal candidates. The merge mode can be applied to any inter-predicted PU, not only for skip mode. An alternative to the merge mode is the explicit transmission of motion parameters, where the motion vector (more precisely, the motion vector difference (MVD) compared to the motion vector predictor), the corresponding reference picture index for each reference picture list, and the reference picture list usage are explicitly signaled for each PU. Such a mode is referred to as advanced motion vector prediction (AMVP) in the present disclosure.
[0054] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is called 'uni-prediction'. Uni-prediction can be used for both P slices and B slices.
[0055] When signaling indicates that two reference picture lists will be used, the PU is generated from two blocks of samples. This is called 'bi-prediction'. Bi-prediction is only available for B slices.
[0056] The following text provides a detailed description of the inter prediction modes specified in HEVC. The description will start with the merge mode.
[0057] 2.1.1 Reference Image List
[0058] In HEVC, the term inter prediction is used to refer to predictions derived from data elements (e.g., sample values or motion vectors) of reference pictures other than the current decoded picture. As in H.264 / AVC, a picture can be predicted from multiple reference pictures. Reference pictures used for inter prediction are organized in one or more reference picture lists. A reference index identifies which reference pictures in the list should be used to generate the prediction signal.
[0059] A single reference picture list (list 0) is used for P slices and two reference picture lists (list 0 and list 1) are used for B slices. It should be noted that the reference pictures included in lists 0 / 1 can be from both past and future pictures in terms of capture / display order.
[0060] 2.1.2Merge Mode
[0061] 2.1.2.1 Merge Mode Candidate Derivation
[0062] When predicting a PU using merge mode, indices pointing to entries in the merge candidate list are parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be summarized according to the following sequence of steps:
[0063] Step 1: Initial candidate derivation
[0064] Step 1.1: Spatial Candidate Derivation
[0065] Step 1.2: Redundancy check of spatial candidates
[0066] οStep 1.3: Time Domain Candidate Derivation
[0067] Step 2: Additional candidate insertions
[0068] οStep 2.1: Create bidirectional prediction candidates
[0069] οStep 2.2: Insert zero motion candidates
[0070] exist Figure 1 These steps are also schematically depicted in . For spatial merge candidate derivation, at most four merge candidates are selected from the candidates located at five different positions. For temporal merge candidate derivation, at most one merge candidate is selected from two candidates. Since the number of candidates for each PU is assumed to be constant at the decoder, additional candidates are generated when the number of candidates obtained from step 1 does not reach the maximum number of merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, the index of the best merge candidate is encoded and decoded using truncated unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a single merge candidate list, which is the same as the merge candidate list of the 2N×2N prediction unit.
[0071] In the following, operations related to the aforementioned steps are described in detail.
[0072] 2.1.2.2. Spatial Candidate Derivation
[0073] In the derivation of spatial merge candidates, Figure 2At most four merge candidates are selected from the candidates at the positions shown in . The derivation order is A1, B1, B0, A0 and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is unavailable (for example, because it belongs to another stripe or slice) or intra-frame coding and decoding is not performed. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates to ensure that candidates with the same motion information are excluded from the list, thereby improving coding and decoding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the candidates at Figure 3 The pairs linked by arrows in , and candidates are added to the list only when the corresponding candidates used for redundancy checking do not have the same motion information.
[0074] 2.1.2.2.1Merge Estimation Region (MER)
[0075] Another source of repeated motion information is a "second PU" associated with a partition different from 2N×2N. As an example, Figure 4 depicts the second PU in the cases of N×2N and 2N×N, respectively. When the current PU is partitioned as N×2N, the candidate at position A1 is not considered for list construction. In fact, adding this candidate will result in two prediction units with the same motion information, which is redundant for having only one PU in the codec unit. Similarly, when the current PU is partitioned as 2N×N, position B1 is not considered.
[0076] FIG. 4 shows example locations of the second PU for N×2N and 2N×N partitions.
[0077] 2.1.2.3 Time Domain Candidate Derivation
[0078] In this step, only one candidate is added to the list. In particular, in the derivation of the temporal merge candidate, the scaled motion vector is derived based on the co-located PU belonging to the picture with the smallest picture order number (POC) difference with the current picture in a given reference picture list. The reference picture list to be used for derivation of the co-located PU is explicitly signaled in the slice header. Figure 5 As shown by the dashed line in , a scaled motion vector for the temporal merge candidate is obtained, which is scaled from the motion vector located in the co-located PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors are obtained, one for reference picture list 0 and the other for reference picture list 1, and are combined to produce a bidirectional prediction merge candidate.
[0079] In the co-located PU (Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, such as Figure 6 As shown in . If the PU at position C0 is not available, not intra-coded or outside the current codec tree unit (CTU, also known as LCU, the largest codec unit) row, position C1 is used. Otherwise, position C0 is used in the derivation of the time domain merge candidate.
[0080] Figure 6 Example candidate positions for the time domain merge candidates C0 and C1 are shown.
[0081] 2.1.2.4 Additional Candidate Insertion
[0082] In addition to spatial and temporal merge candidates, there are two additional types of merge candidates: combined bi-prediction merge candidate and zero merge candidate. Combined bi-prediction merge candidate is generated by utilizing spatial and temporal merge candidates. Combined bi-prediction merge candidate is used only for B slices. Combined bi-prediction candidate is generated by combining the first reference picture list motion parameters of an initial candidate with the second reference picture list motion parameters of another candidate. If the two tuples provide different motion hypotheses, they will form a new bi-prediction candidate. For example, Figure 7 Depicted is the case where two candidates with mvL0 and refIdxL0 or mvL1 and refIdxL1 in the original list (left) are used to generate a combined bi-predictive merge candidate that is added to the final list (right). There are many rules for combining that are considered to generate these additional merge candidates.
[0083] Zero motion candidates are inserted to fill the remaining entries in the merge candidate list and thus reach the Ma*NumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list.
[0084] More specifically, the following steps are performed in sequence until the merge list is full:
[0085] Setting the variable numRef to the number of reference pictures associated with list 0 for a P slice, or the minimum number of reference pictures in the two lists for a B slice;
[0086] Add non-repeating zero-motion candidates:
[0087] For variable i 0...numRef-1, a default motion candidate is added with MV set to (0,0) and the reference picture index set to i for list 0 (if P slice) or both lists (if B slice).
[0088] Repeated zero motion candidates are added with MV set to (0, 0), reference picture index of list 0 set to 0 (if P slice) and reference picture index of both lists set to 0 (if B slice).
[0089] Finally, no redundancy check is performed on these candidates.
[0090] 2.1.3AMVP
[0091] AMVP exploits the spatiotemporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. For each reference picture list, a motion vector candidate list is constructed by first checking the availability of the upper left temporal neighboring PU position, removing redundant candidates and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and transmit the corresponding index indicating the selected candidate. Similar to merge index signaling, the index of the best motion vector candidate is encoded using a truncated unary code. The maximum value to be encoded in this case is 2 ( Figure 8 ). In the following subsections, details about the derivation process of motion vector prediction candidates are provided.
[0092] 2.1.3.1 Derivation of AMVP Candidates
[0093] Figure 8 The derivation process of motion vector prediction candidates is summarized.
[0094] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, the final result is based on the location of Figure 2 The motion vector of each PU at the five different positions shown in derives two motion vector candidates.
[0095] For temporal motion vector candidate derivation, select one motion vector candidate from two candidates derived based on two different co-located positions. After making the first list of spatio-temporal candidates, remove duplicate motion vector candidates in the list. If the number of potential candidates is greater than two, remove from the list motion vector candidates whose reference picture index in their associated reference picture list is greater than 1. If the number of spatio-temporal motion vector candidates is less than two, add an additional zero motion vector candidate to the list.
[0096] 2.1.3.2 Spatial Motion Vector Candidates
[0097] In the derivation of spatial motion vector candidates, the Figure 2 A maximum of two candidates are considered out of the five potential candidates for PU derivation at the positions shown in , which are the same positions as the motion merge positions. The derivation order for the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order for the top side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. For each side, there are therefore four cases that can be used as motion vector candidates, two of which do not require the use of spatial scaling, and two of which use spatial scaling. The four different cases are summarized below.
[0098] No airspace scaling
[0099] –(1) Same reference picture list and same reference picture index (same POC)
[0100] –(2) Different reference picture lists, but same reference picture (same POC)
[0101] Airspace scaling
[0102] –(3) Same reference picture list, but different reference pictures (different POC)
[0103] –(4) Different reference picture lists and different reference pictures (different POC)
[0104] First check the case without spatial scaling, then check spatial scaling. When the POC between the reference picture of the neighboring PU and the reference picture of the current PU is different, spatial scaling is considered regardless of the reference picture list. If all PUs of the left candidate are not available or are not intra-coded, scaling of the above motion vectors is allowed to help derive the left and upper MV candidates in parallel. Otherwise, spatial scaling of the above motion vectors is not allowed.
[0105] Fig. 9 is an illustration of motion vector scaling for spatial motion vector candidates.
[0106] In the spatial domain scaling process, the motion vectors of neighboring PUs are scaled in a similar way to the temporal domain scaling, such as Fig. 9 The main difference is that the reference picture list and index of the current PU are given as input; the actual scaling process is the same as the time domain scaling process.
[0107] 2.1.3.3 Temporal Motion Vector Candidates
[0108] Except for the reference picture index derivation, all the processes used to derive the temporal merge candidate are the same as those used to derive the spatial motion vector candidate (see Figure 6 ). Signaling the reference picture index to the decoder.
[0109] 2.2 Inter-frame prediction method in VVC
[0110] There are several new codec tools for inter prediction improvements, such as Adaptive Motion Vector Difference Resolution (AMVR) for signaling MVD for affine blocks and translational motion blocks, affine prediction mode, Triangular Prediction Mode (TPM), Alternative TMVP (ATMVP), Generalized Bidirectional Prediction (GBI), Bidirectional Optical Flow (BIO), Intra Block Copy (IBC), etc.
[0111] 2.2.1 Adaptive motion vector difference resolution
[0112] In HEVC, the motion vector difference (MVD) (between the PU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples when use_integer_mv_flag in the slice header is equal to 0. In VVC, Locally Adaptive Motion Vector Resolution (LAMVR) is introduced.
[0113] 2.2.1.1 AMVR for Translational Motion Blocks
[0114] In VVC, MVD can be coded and decoded in units of quarter luma samples, integer luma samples, or quad luma samples (i.e., 1 / 4 pixel, 1 pixel, 4 pixels). The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.
[0115] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or quad luma sample MV precision is used.
[0116] When the first MVD resolution flag of a CU is zero or not coded for the CU (meaning that all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or quad luma sample MV precision, the MVP in the AMVP candidate list for the CU is rounded to the corresponding precision.
[0117] 2.2.1.2 AMVR for Affine Motion Blocks
[0118] Similar to the design of the translational motion block (aka normal AMVR mode), the AMVR mode is extended to the affine codec block (aka affine AMVR mode).
[0119] For the affine AMVR mode, three MV / MVD resolutions are supported, namely {1 / 16, 1 / 4, 1} pixels.
[0120] 2.2.2 Triangle prediction model
[0121] The concept of triangular prediction mode (TPM) is to introduce a new triangular partition for motion compensated prediction. As shown in Figure 10, it divides the CU into two triangular prediction units in the diagonal or anti-diagonal direction. Each triangular prediction unit in the CU uses its own unidirectional prediction motion vector and reference frame index derived from a single unidirectional prediction candidate list for inter-frame prediction. After predicting the triangular prediction unit, adaptive weighting is performed on the diagonal edges. Then, the transform and quantization process is applied to the entire CU. It should be noted that this mode is only applied to merge mode (note: skip mode is regarded as a special merge mode).
[0122] Fig. 10A An example of a 135 degree partition type (partitioned from the upper left corner to the lower right corner) is shown.
[0123] Fig. 10B An example of a 45 degree division pattern is shown.
[0124] Figures 10A-10B A diagram showing partitioning of a CU into two triangular prediction units (two partitioning modes) is shown.
[0125] 2.2.2.1 One-way prediction candidate list for TPM
[0126] The unidirectional prediction candidate list, called the TPM motion candidate list, consists of five unidirectional prediction motion vector candidates. Fig.11 As shown in , it is derived from seven neighboring blocks including five spatial neighboring blocks (1 to 5) and two temporal co-located blocks (6 to 7). The motion vectors of the seven neighboring blocks are collected and put into the unidirectional prediction candidate list in the order of unidirectional prediction motion vector, L0 motion vector of bidirectional prediction motion vector, L1 motion vector of bidirectional prediction motion vector, and average motion vector of L0 and L1 motion vector of bidirectional prediction motion vector. If the number of candidates is less than five, the zero motion vector is added to the list. The motion candidates added to this list for TPM are called TPM candidates, and the motion information derived from the spatial / temporal blocks is called conventional motion candidates.
[0127] More specifically, the following steps are involved:
[0128] 1) When adding regular motion candidates from spatial neighboring blocks, use a full cut operation from A1, B1, B0, A0, B2, Col and Col2 (corresponding to Fig.11 Blocks 1-7 in ) obtain regular motion candidates.
[0129] 2) Set the variable numCurrMergeCand = 0
[0130] 3) For each normal motion candidate derived from A1, B1, B0, A0, B2, Col and Col2 (if not cut), and numCurrMergeCand is less than 5, if the normal motion candidate is unidirectional prediction (from list 0 or list 1), it is added directly to the merge list as a TPM candidate, and numCurrMergeCand is increased by 1. Such a TPM candidate is called 'original unidirectional prediction candidate'.
[0131] Apply full cut.
[0132] 4) For each motion candidate derived from A1, B1, B0, A0, B2, Col and Col2 (if not clipped), and numCurrMergeCand is less than 5, if the regular motion candidate is bi-directionally predicted, add the motion information from list 0 as a new TPM candidate to the TPM merge list (i.e., modify to uni-directional prediction from list 0) and increase numCurrMergeCand by 1. Such a TPM candidate is called a 'truncated list 0 prediction candidate'.
[0133] Apply full cut.
[0134] 5) For each motion candidate derived from A1, B1, B0, A0, B2, Col and Col2 (if not clipped), and numCurrMergeCand is less than 5, if the regular motion candidate is bi-directionally predicted, add the motion information from list 1 to the TPM merge list (i.e., modify to uni-directionally predicted from list 1) and increase numCurrMergeCand by 1. Such a TPM candidate is called a 'truncated list 1 prediction candidate'.
[0135] Apply full cut.
[0136] 6) For each motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 (if not clipped), and numCurrMergeCand is less than 5, if the regular motion candidate is bi-predicted,
[0137] – If the slice QP of list 0 reference picture is less than the slice QP of list 1 reference picture, the motion information of list 1 is first scaled to list 0 reference picture, and the average of two MVs (one from the original list 0 and the other is the scaled MV from list 1) is added to the TPM merge list. Such a candidate is called an average unidirectional prediction from list 0 motion candidate, and numCurrMergeCand is increased by 1.
[0138] – Otherwise, first scale the motion information of list 0 to the reference picture of list 1, and add the average of the two MVs (one from the original list 1 and the other is the scaled MV from list 0) to the TPM merge list. Such a TPM candidate is called the average unidirectional prediction from list 1 motion candidate, and increase numCurrMergeCand by 1.
[0139] Apply full cut.
[0140] 7) If numCurrMergeCand is less than 5, add a zero motion vector candidate.
[0141] Fig.11 Example locations of neighboring blocks are shown.
[0142] When a candidate is inserted into the list, if it must be compared with all previously added candidates to see if it is identical to one of them, such a process is called a full cut.
[0143] 2.2.2.2 Weighting process
[0144] After predicting each triangular prediction unit, the following weighting process is applied to the diagonal edges between two triangular prediction units to derive the final prediction for the entire CU. The weighting factor set is defined as follows:
[0145] Weighting factor sets: {7 / 8, 6 / 8, 5 / 8, 4 / 8, 3 / 8, 2 / 8, 1 / 8} and {6 / 8, 4 / 8, 2 / 8} are used for luma and chroma samples respectively.
[0146] It should be noted that the blending area depends on the aspect ratio of the block. For square blocks, as described above, the width of the weighted pixels is 7 for luma and 3 for chroma. For rectangular blocks, when the width is greater than the height, the width will increase by a factor of (W / H). Each factor is repeated (W / H) times before going to the next factor. If the height is greater than the width, the width is the same as in a square block, and they repeat (H / W) rows before moving one pixel position to the right.
[0147] 2.2.2.3 Signaling of Triangular Prediction Mode (TPM)
[0148] A one-bit flag indicating whether to use TPM may be signaled first, and then two partitioning mode indications (as shown in FIG. 10 ) and a merge index selected for each of the two partitions may be further signaled.
[0149] 2.2.2.3.1 Signaling of TPM Flags
[0150] Let W and H denote the width and height of a luma block, respectively. If W*H<64, triangular prediction mode is disabled.
[0151] When encoding or decoding a block in affine mode, triangular prediction mode is also disabled.
[0152] When a block is encoded or decoded in merge mode, a one-bit flag may be signaled to indicate whether triangular prediction mode is enabled or disabled for the block.
[0153] The flags are encoded and decoded with 3 contexts based on the following equations:
[0154] Ctx index = ((Left block L is available && L is encoded and decoded with TPM?) 1:0)
[0155] +((upper block A is available &&A is encoded and decoded with TPM?) 1:0);
[0156] Fig.12 An example of adjacent blocks (A and L) used for context selection in TPM flag encoding and decoding is shown.
[0157] 2.2.3 Affine Motion Compensated Prediction
[0158] In HEVC, only the translation motion model is applied to motion compensation prediction (MCP). In the real world, there are many kinds of motions, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, the simplified affine transformation motion compensation prediction is applied to the 4-parameter affine model and the 6-parameter affine model. Figures 13A-13B As shown in , the affine motion field of a block is described by two control point motion vectors (CPMVs) for a 4-parameter affine model and by 3 CPMVs for a 6-parameter affine model.
[0159] Fig.13A An example of a 4-parameter affine model is shown.
[0160] Fig. 13B An example of a 6-parameter affine model is shown.
[0161] The motion vector field (MVF) of the block is described by the following equations, where the 4-parameter affine model in equation (1) (where the 4 parameters are defined as variables a, b, e, and f) and the 6-parameter affine model in equation (2) (where the 4 parameters are defined as variables a, b, c, d, e, and f) are respectively:
[0162]
[0163]
[0164] Where (mv h 0,mvh 0) is the motion vector of the upper left control point, (mv h 1,mv h 1) is the motion vector of the upper right control point and (mv h 2,mv h 2) is the motion vector of the lower left control point. All three motion vectors are called control point motion vectors (CPMVs). (x, y) represents the coordinates of the representative point relative to the upper left sample point in the current block, and (mv h (x,y),mv v (x,y)) is the motion vector derived for the sample at (x,y). The CP motion vector can be signaled (e.g. in affine AMVP mode) or derived on the fly (e.g. in affine merge mode). w and h are the width and height of the current block. In practice, division is performed by right shifting with an integer operation. In VTM, the representative point is defined as the center position of the sub-block, for example, when the coordinates of the upper left corner of the sub-block relative to the upper left sample in the current block are (xs,ys), the coordinates of the representative point are defined as (xs+2,ys+2). For each sub-block (i.e., 4×4 in VTM), the representative point is used to derive the motion vector for the entire sub-block.
[0165] To further simplify motion compensation prediction, sub-block based affine transformation prediction is applied. To derive the motion vector of each M×N (both M and N are set to 4 in the current VVC) sub-block, as Fig.14 As shown in , the motion vector of the center sample of each sub-block is calculated according to equations (1) and (2) and rounded to 1 / 16 fractional precision. Then a motion compensation interpolation filter for 1 / 16 pixels is applied to generate a prediction for each sub-block using the derived motion vector. The interpolation filter for 1 / 16 pixels is introduced by the affine mode.
[0166] Fig.14 An example of the affine MVF for each sub-block is shown.
[0167] After MCP, the high-precision motion vector of each sub-block is rounded and saved with the same precision as the normal motion vector.
[0168] 2.2.3.1 Signaling of Affine Prediction
[0169] Similar to the translational motion model, due to affine prediction, there are also two modes for signaling auxiliary information. They are AFFINE_INTER and AFFINE_MERGE modes.
[0170] 2.2.3.2AF_INTER mode
[0171] AF_INTER mode can be applied for CUs with width and height both greater than 8. A CU-level affine flag is signaled in the bitstream to indicate whether AF_INTER mode is used.
[0172] In this mode, for each reference picture list (list 0 or list 1), the affine AMVP candidate list is constructed with three types of affine motion predictors in the following order, where each candidate includes the estimated CPMV of the current block. Signaling the best CPMV found on the encoder side (e.g., Fig.17 In addition, the index of the affine AMVP candidate from which the estimated CPMV is derived is further signaled.
[0173] 1) Inheriting affine motion prediction
[0174] The check order is similar to the spatial MVP in the HEVC AMVP list structure. First, the left inherited affine motion predictor is derived from the first block in {A1, A0}, which is affine coded and has the same reference picture as the current block. Second, the above inherited affine motion predictor is derived from the first block in {B1, B0, B2}, which is affine coded and has the same reference picture as the current block. Fig.16 Five blocks A1, A0, B1, B0, B2 are shown in FIG.
[0175] Once a neighboring block is found to be coded in affine mode, the CPMV of the codec covering the neighboring block is used to derive the prediction of the CPMV of the current block. For example, if A1 is coded in non-affine mode and A0 is coded in 4-parameter affine mode, the left-inherited affine MV prediction will be derived from A0. In this case, the CPMV of the CU covering A0 is given by Fig.18B For the upper left CPMV and for the upper right CPMV The estimated CPMV for the current block is derived from the top left (coordinate (x0, y0)), top right (coordinate (x1, y1)) and bottom right (coordinate (x2, y2)) positions of the current block. express.
[0176] 2) Construct affine motion prediction
[0177] The constructed affine motion predictor consists of control point motion vectors (CPMVs) derived from adjacent inter-codec blocks with the same reference picture, such as Fig.17If the current affine motion model is 4-parameter affine, the number of CPMVs is 2, otherwise, if the current affine motion model is 6-parameter affine, the number of CPMVs is 3. The upper left CPMV is derived by the MV at the first block in the group {A, B, C} The first block is inter-coded and has the same reference picture as the current block. The top right CPMV is derived from the MV at the first block in the group {D, E} The first block is inter-coded and has the same reference picture as the current block. The lower left CPMV is derived from the first block in the group {F, G} The first block is inter-coded and has the same reference picture as the current block.
[0178] – If the current affine motion model is a 4-parameter affine, then only and Both, namely and The constructed affine motion prediction amount is inserted into the candidate list only when it is used as the estimated CPMV of the upper left (coordinates (x0, y0)) and upper right (coordinates (x1, y1)) positions of the current block.
[0179] – If the current affine motion model is a 6-parameter affine, only all and Right now and The constructed affine motion prediction quantity is inserted into the candidate list only when it is used as the estimated CPMV of the upper left (coordinates (x0, y0)), upper right (coordinates (x1, y1)) and lower right (coordinates (x2, y2)) positions of the current block.
[0180] No clipping process is applied when inserting the constructed affine motion predictor into the candidate list.
[0181] 3) Ordinary AMVP motion prediction
[0182] Until the number of affine motion predictors reaches a maximum value, the following applies.
[0183] 1) By setting all CPMVs equal to (if available) to derive affine motion predictors.
[0184] 2) By setting all CPMVs equal to (if available) to derive affine motion predictors.
[0185] 3) By setting all CPMVs equal to (if available) to derive affine motion predictors.
[0186] 4) Affine motion predictors are derived by setting all CPMVs equal to HEVC TMVP (if available).
[0187] 5) Derive the affine motion predictor by setting all CPMVs to zero MV.
[0188] It should be noted that It has been derived in constructing affine motion predictors.
[0189] Fig.15A A 4-parameter affine model is shown.
[0190] Fig. 15B A 6-parameter affine model is shown.
[0191] Fig.16 The MVP of AF_INTER for inherited affine candidates is shown.
[0192] Fig.17 The MVP of AF_INTER used to construct the affine candidate is shown.
[0193] In AF_INTER mode, when using 4 / 6 parameter affine mode, 2 / 3 control points are needed, therefore 2 / 3 MVDs need to be encoded and decoded for these control points, as shown in Figure 15. It is suggested to derive the MVs as follows, ie predict mvd1 and mvd2 from mvd0.
[0194]
[0195]
[0196]
[0197] in mvd i and mvd1 are the predicted motion vector, motion vector difference and motion vector of the upper left pixel (i=0), upper right pixel (i=1) or lower left pixel (i=2), respectively. Fig. 15B Note that the addition of two motion vectors (e.g., mvA(xA, yA) and mvB(xB, yB)) is equal to the sum of the two components, i.e., newMV = mvA + mvB, and the two components of newMV are set to (xA + xB) and (yA + yB), respectively.
[0198] 2.2.3.3AF_MERGE Mode
[0199] When a CU is applied in AF_MERGE mode, it obtains the first block encoded and decoded in affine mode from the valid adjacent reconstructed blocks. And the selection order of candidate blocks is from left, top, top right, bottom left to top left, such as Fig.18AAs shown in (in turn represented by A, B, C, D, E). For example, if the adjacent lower left block is encoded and decoded in affine mode, such as Fig.18B As shown in A0 in the figure, the control point (CP) motion vector mv0 of the upper left corner, upper right corner and lower left corner of the adjacent CU / PU containing block A is obtained. N , mv1 N and mv2 N . And based on mv0 N , mv1 N and mv2 N Calculate the motion vector mv0 of the upper left corner / upper right / lower left corner of the current CU / PU C , mv1 C and mv2 C (Only for 6-parameter affine model). It should be noted that in VTM-2.0, the sub-block at the upper left corner (e.g. 4×4 block in VTM) stores mv0, and if the current block is affine encoded or decoded, the sub-block at the upper right corner stores mv1. If the current block is encoded or decoded with a 6-parameter affine model, the sub-block at the lower left corner will store mv2; otherwise (with a 4-parameter affine model), LB stores mv2'. The other sub-blocks store the MV for MC.
[0200] In deriving the CPMV mv0 of the current CU C , mv1 C and mv2 C Afterwards, the MVF of the current CU is generated according to the simplified affine motion model equations (1) and (2). In order to identify whether the current CU is coded or decoded in AF_MERGE mode, an affine flag is signaled in the bitstream when at least one neighboring block is coded or decoded in affine mode.
[0201] Figures 18A-18B Candidate examples for AF_MERGE are shown (18A - five neighboring blocks, 18B - CPMV predictor derivation).
[0202] The affine merge candidate list is constructed by the following steps:
[0203] 1) Insert inheritance affine candidates
[0204] Inherited affine candidates indicate that the candidate is derived from the affine motion model of its valid neighboring affine codec block. At most two inherited affine candidates are derived from the affine motion model of the neighboring block and inserted into the candidate list. For the left prediction, the scanning order is {A0, A1}; for the upper prediction, the scanning order is {B0, B1, B2}.
[0205] 2) Insertion construction affine candidate
[0206] If the number of candidates in the affine merge candidate list is less than MaxNumAffineCand (for example, 5), the constructed affine candidate is inserted into the candidate list. Constructing the affine candidate means constructing the candidate by combining the adjacent motion information of each control point.
[0207] a) First, Fig.19 The motion information of the control point is derived from the specified spatial neighborhood and temporal neighborhood shown in . CPk (k = 1, 2, 3, 4) represents the kth control point. A0, A1, A2, B0, B1, B2 and B3 are the spatial positions (k = 1, 2, 3) used to predict CPk; T is the temporal position used to predict CP4. The coordinates of CP1, CP2, CP3 and CP4 are (0, 0), (W, 0), (H, 0) and (W, H), respectively, where W and H are the width and height of the current block.
[0208] Fig.19 Examples of candidate positions for affine merge mode are shown.
[0209] The motion information for each control point is obtained according to the following priority order:
[0210] – For CP1, the check priority is B2->B3->A2. If available, use B2. Otherwise, if B2 is available, use B3. If neither B2 nor B3 is available, use A2. If none of the three candidates are available, the motion information of CP1 cannot be obtained.
[0211] – For CP2, the check priority is B1->B0.
[0212] – For CP3, the check priority is A1->A0.
[0213] – For CP4, use T.
[0214] b) Secondly, use the combination of control points to construct affine merge candidates.
[0215] I. Motion information of three control points is required to construct a 6-parameter affine candidate. Three control points can be selected from one of the following four combinations ({CP1, CP2, CP4}, {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4}). The combination of {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4} will be converted into a 6-parameter motion model represented by the top left, top right and bottom left control points.
[0216] II. Motion information of two control points is required to construct a 4-parameter affine candidate. Two control points can be selected from one of two combinations ({CP1, CP2}, {CP1, CP3}). These two combinations will be converted into a 4-parameter motion model represented by the top left and top right control points.
[0217] III. Construct combinations of affine candidates and insert them into the candidate list in the following order: {CP1, CP2, CP3}, {CP1, CP2, CP4}, {CP1, CP3, CP4}, {CP2, CP3, CP4}, {CP1, CP2}, {CP1, CP3}
[0218] i. For each combination, check the reference index of list X for each CP, if they are all the same, then the combination has a valid CPMV for list X. If the combination does not have a valid CPMV for both list 0 and list 1, then the combination is marked as invalid. Otherwise, it is valid and the CPMV is put into the sub-block merge list.
[0219] 3) Zero motion vector filling
[0220] If the number of candidates in the affine merge candidate list is less than 5, zero motion vectors with zero reference index are inserted into the candidate list until the list is full.
[0221] More specifically, for the sub-block merge candidate list, MV is set to (0, 0) and the prediction direction is set to the 4-parameter merge candidate of unidirectional prediction (for P slices) and bidirectional prediction (for B slices) from list 0.
[0222] 2.2.4 Current Image Reference
[0223] Intra-block copying (also known as IBC or intra-picture block compensation), also known as current picture reference (CPR), is employed in the HEVC Screen Content Codec extension (SCC). This tool is very effective for the encoding and decoding of screen content videos, since repetitive patterns of text and graphic-rich content often appear in the same picture. Using previously reconstructed blocks with the same or similar patterns as predictors can effectively reduce the prediction error and thus improve the codec efficiency. An example of intra-block compensation is shown in Fig. 20 Shown in.
[0224] Fig. 20 An example of intra picture block copying is shown.
[0225] Similar to the CRP design in HEVC SCC, in VVC, the use of IBC mode is signaled at both the sequence and picture levels. When IBC mode is enabled at the sequence parameter set (SPS), it can be enabled at the picture level. When IBC mode is enabled at the picture level, the currently reconstructed picture will be considered as a reference picture. Therefore, no syntax changes at the block level are required to signal the use of IBC mode on top of the existing VVC inter-frame mode.
[0226] Key Features:
[0227] – It is considered as a special inter mode. Therefore, merge and skip modes can also be used in IBC mode. The merge candidate list structure is unified and contains merge candidates from neighboring positions, which are coded or decoded in IBC mode or HEVC inter mode. Depending on the selected merge index, the current block in merge or skip mode can have different pictures as reference pictures merged into IBC mode codec neighborhood or normal inter mode codec neighborhood.
[0228] – Motion vectors (also called block vectors) for IBC mode are encoded and decoded with integer pixel precision, but are stored in memory with 1 / 16 pixel precision after decoding because quarter pixel precision is required during the interpolation and deblocking stages. When used for motion vector prediction in IBC mode, the stored vector predictor is right-shifted by 4.
[0229] – Search range: It is limited to the current CTU.
[0230] – CPR is not allowed when affine mode / triangle mode / GBI / weighted prediction is enabled.
[0231] 2.2.5 Merge List Design in VVC
[0232] Four different merge list construction processes are supported in VVC:
[0233] 1) Sub-block merge candidate list: It includes ATMVP and affine merge candidates. Affine mode and ATMVP mode share a merge list construction process. Here, ATMVP and affine merge candidates can be added in sequence. The sub-block merge list size is signaled in the slice header and the maximum value is 5.
[0234] 2) Uni-Prediction TPM merge list: For triangular prediction mode, even though the two partitions can select their own merge candidate indexes, they share one merge list construction process for the two partitions. When constructing the merge list, the spatial neighboring blocks and two temporal blocks of the block are checked. The motion information derived from the spatial neighbors and temporal blocks is called regular motion candidates in our IDF. These regular motion candidates are further used to derive multiple TPM candidates. Note that the transform is performed at the entire block level, even though the two partitions may use different motion vectors to generate their own prediction blocks. The uni-prediction TPM merge list size is fixed to 5.
[0235] 3) Regular merge list: For the rest of the codec blocks, a merge list construction process is shared. Here, spatial / temporal / HMVP, paired bidirectional prediction merge candidates and zero motion candidates can be inserted in order. The regular merge list size is signaled in the slice merge header, and the maximum value is 6. MMVD,
[0236] 4) IBC merge list: It is done in a similar way to the regular merge list.
[0237] 2.2.5.1 Sub-block merge candidate list
[0238] It is recommended to put all sub-block related motion candidates into a separate merge list in addition to the regular merge list for non-sub-block merge candidates.
[0239] Sub-block related motion candidates are put into a separate merge list called 'sub-block merge candidate list'.
[0240] In one example, the sub-block merge candidate list includes affine merge candidates and ATMVP candidates, and / or sub-block-based STMVP candidates.
[0241] The sub-block merge candidate list is populated with candidates in the following order:
[0242] a. ATMVP candidate (may be available or not);
[0243] b. Affine merge list (including inherited affine candidates and constructed affine candidates)
[0244] c. Filled with zero MV 4-parameter affine model
[0245] 2.2.5.1.1 ATMVP (also known as sub-block temporal motion vector predictor, SbTMVP)
[0246] The basic idea of ATMVP is to derive multiple sets of temporal motion vector predictions for a block. Each sub-block is assigned a set of predictions. When generating ATMVP merge candidates, the following process is applied:
[0247] a. Figure 2 As shown in , adjacent blocks A1 are checked in sequence to find a block that is first inter-coded but not CPR-coded, denoted as block X;
[0248] b. Initialize TMV=(0,0). If there is an MV (denoted as MV') for block X indicating a co-located reference picture (signaled in the slice header), then set TMV equal to MV'.
[0249] c. Assuming the center point of the current block is (x0, y0), locate the corresponding position of (x0, y0) in the co-located picture as M = (x0 + MV'x, y0 + MV'y). Find the block Z covering M.
[0250] i. If Z performs intra-frame encoding and decoding, ATMVP is not available;
[0251] ii. If Z performs inter-frame coding and decoding, the MVZ_0 and MVZ_1 of the two lists of block Z are scaled to (Reflist0 index 0) and (Reflist 1 index 0) as MVdefault0, MVdefault1, and stored.
[0252] d. For each 8x8 sub-block, assuming its center point is (x0S, y0S), then locate the corresponding position of (x0S, y0S) in the co-located picture as MS = (x0S + MV'x, y0S + MV'y). Find the block ZS covering MS.
[0253] i. If ZS performs intra-frame encoding and decoding, MVdefault0 and MVdefault1 are assigned to the sub-blocks;
[0254] ii. If ZS performs inter-frame coding and decoding, the MVZS_0 and MVZS_1 of the two lists of block ZS are scaled to (Reflist 0 index 0) and (Reflist 1 index 0) and assigned to the sub-blocks;
[0255] 2.2.5.2 Conventional merge list
[0256] Different from the merge list design, in VVC, the history-based motion vector prediction (HMVP) method is adopted.
[0257] In HMVP, the motion information of the previous codec is stored. The motion information of the previous codec block is defined as the HMVP candidate. Multiple HMVP candidates are stored in a table called the HMVP table, and the table is maintained on the fly during the encoding / decoding process. The HMVP table is cleared when encoding / decoding a new slice / LCU row / slice begins. Whenever there is an inter-coded block and non-sub-block, non-TPM mode, the associated motion information is added as the last entry of the table as a new HMVP candidate. The overall encoding and decoding process is Fig.21 Shown in.
[0258] HMVP candidates can be used during the construction of AMVP and merge candidate lists. Fig. 22 A modified merge candidate list construction process is shown (highlighted in blue). When the merge candidate list is not full after inserting the TMVP candidate, the HMVP candidates stored in the HMVP table can be used to fill the merge candidate list. Considering that a block usually has a higher correlation with the nearest neighboring block in terms of motion information, the HMVP candidates in the table are inserted in descending order of the index. The last entry in the table is added to the list first, and the first entry is added last. Similarly, redundancy elimination is applied to the HMVP candidates. Once the total number of available merge candidates reaches the maximum number of merge candidates allowed for signaling, the merge candidate list construction process terminates.
[0259] It should be noted that all spatial / temporal / HMVP candidates should be coded and decoded in non-IBC mode. Otherwise, they are not allowed to be added to the regular merge candidate list.
[0260] The HMVP table contains up to 5 regular motion candidates, and each candidate is unique.
[0261] 2.2.5.3IBC merge list
[0262] When joining IBC, HMVP candidates are also eligible for the IBC merge list.
[0263] More specifically, an additional 5 IBC candidates can be stored. In the current implementation, conventional candidates and IBC candidates are stored in the same HMVP table. However, they are used and updated independently. The first M (M <= 5) candidates are used for the conventional merge / AMVP list; the remaining N candidates (N <= 5) are used for IBC mode. Two counters are maintained to indicate how many conventional motion candidates and how many IBC motion candidates there are in the HMVP table. Therefore, it is equivalent to using 2 HMVP tables, one for conventional merge mode and the other for IBC mode.
[0264] Shares the same process as regular MV merge, but TMVP is not allowed, and zero vector means unavailable because it is invalid. It should be noted that for spatial neighboring blocks, the associated motion information can be added to the IBCmerge list only when it is encoded and decoded in IBC mode. At the same time, for the HMVP part, only the last few HMVP candidates (which are stored IBC motion candidates) can be considered in the IBC merge list.
[0265] 2.2.6 Shared merge list
[0266] In order to reduce decoder complexity and support parallel encoding, it is recommended that all leaf codec units (CUs) of an ancestor node in the CU partition tree share the same merge candidate list to achieve parallel processing of small skip / merge codec CUs. The ancestor node is called a merge shared node. In the case where the merge shared node is a leaf CU, a shared merge candidate list is generated at the merge shared node.
[0267] More specifically, the following may apply:
[0268] - If the block has no more than 32 luma samples and is divided into 2 4x4 sub-blocks, use a shared merge list between very small blocks (e.g., two adjacent 4x4 blocks).
[0269] - If the luma samples of the block are greater than 32, but after partitioning, at least one sub-block is less than the threshold (32), all sub-blocks of the partition share the same merge list (e.g. 16x4 or 4x16 for three partitions or 8x8 for four partitions).
[0270] Such restrictions apply only to regular merge mode.
[0271] 2.2.6.1HMVP table update and use
[0272] Allocate a temporary HMVP table. Before this, when the first CU of a shared area is entered, the original HMVP table must be copied to the temporary HMVP table.
[0273] exist Fig.23 In the example shown, an 8x8 QT is split into four 4x4 CUs. Before executing the first CU (e.g., the top left 4x4 CU), the history list must be copied to the temporary HMVP table (HmvpCandListShared[]), and when each 4x4 inside the shared parent node (merge shared node in the figure) needs a history list for merge mode, it will use the motion candidate in the temporary HMVP table.
[0274] However, after decoding a 4x4 CU, the original HMVP table is updated accordingly.
[0275] Fig.23 An example of merging shared nodes is shown.
[0276] 2.2.6.2 Virtual Pipeline Data Unit (VPDU)
[0277] A Virtual Pipeline Data Unit (VPDU) is defined as a non-overlapping unit in a picture. In a hardware decoder, consecutive VPDUs are processed simultaneously by multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is important to keep the VPDU size small. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, ternary tree (TT) and binary tree (BT) partitioning may lead to an increase in the VPDU size.
[0278] In order to keep the VPDU size to 64x64 luma samples, the following normative partitioning restrictions (with syntax signalling modifications) apply as follows:
[0279] – For CUs with width or height or width and height equal to 128, TT partitioning is not allowed.
[0280] – For 128xN CUs with N≤64 (i.e., width equal to 128 and height less than 128), horizontal BT is not allowed.
[0281] – For Nx128 CUs with N≤64 (ie, height equal to 128 and width less than 128), vertical BT is not allowed.
[0282] Fig.24 An example of a disallowed TT and BT split is shown.
[0283] 3. Examples of Problems Solved by the Embodiments Described in This Document
[0284] The current design of merge lists with HMVP has the following problems:
[0285] 1) The HMVP table used to encode the current CTU only contains the motion information in the previous CTU in the same CTU because it is reset at the beginning of the new CTU new. Therefore, it cannot utilize the motion information from the previous CTU.
[0286] 2) Due to parallelism, the HMVP table is always reset at the beginning of each CTU row.
[0287] 3) Some motion information in the HMVP table may be the same as the spatial motion information that has been added to the merge list.
[0288] 4) HMVP tables are currently used in regular merge lists or IBC merge lists. History-based motion information can be used more extensively in different merge lists.
[0289] 4. Example Techniques and Implementations
[0290] The techniques detailed below should be considered as examples to explain the general concept. These techniques should not be interpreted narrowly. In addition, these techniques can be combined in any appropriate manner.
[0291] 1. Multiple HMVP tables can be stored during the encoding / decoding process, and before encoding / decoding a block, the HMVP table can be initialized according to the selected HMVP table.
[0292] a. In one example, after encoding / decoding a first region (eg, a CTU or VPDU), an associated HMVP table (eg, before encoding / decoding a second region) may be stored in association with the first region.
[0293] b. In one example, the HMVP table used to encode / decode the current region may be initialized based on multiple HMVP tables of neighboring regions.
[0294] c. In one example, the HMVP table used to encode / decode the current region may be initialized based on a HMVP table of a neighboring region.
[0295] d. In one example, the HMVP table of the adjacent region (not limited to only the left) may be a table associated with the left, top, top-left, or top-right region of the current region.
[0296] e. Alternatively, the HMVP table of the adjacent region (not limited to only the left) may be the HMVP table associated with the region above or to the upper right of the current region.
[0297] 2. A new HMVP table for encoding / decoding one region may be generated based on multiple HMVP tables associated with other regions.
[0298] a. Inherited motion information from multiple tables can be combined into a new HMVP table.
[0299] b. In one example, one combining order is to add all motion information from the first table, then add all motion information from the second table, add all motion information from the third table, and so on.
[0300] i. In one example, one combining order is to add all motion information from the first table, then add all motion information from the second table, and so on.
[0301] ii. Alternatively, the combination order is to join one from the first table and take the next from the second table, take the next from the third table, repeat and so on (alternating between the first, second and third tables).
[0302] iii. Alternatively, the combination order is to add one from the first list and take the next from the second list, repeating only with a limited number of lists (interleaving between the first and second lists).
[0303] c. In one example, one new HMVP table will combine the HMVP tables for the left and top right regions.
[0304] d. In one example, one new HMVP table will combine the HMVP tables for the left, top, top-left, and top-right regions.
[0305] 3. Before adding the motion information to the new HMVP table, the motion information in the HMVP table of other regions may optionally be pruned first.
[0306] a. When there is motion information next to the current CTU (i.e., the left column buffer) in the left HMVP table, it can be cut off. When the CU is located at the left boundary of the CTU, the cutting of the HMVP merge / AMVP candidate can be skipped.
[0307] i. In one example, when the motion information stored in any entry of the HMVP table matches any one of the motion information from the left column, it is removed from the HMVP table.
[0308] ii. In one example, when the motion information stored in any entry of the HMVP table is a certain distance away from any motion information from the left column, it is removed from the HMVP table. The distance metric may include, but is not limited to, the L1 norm and the L2 norm. In addition, a threshold for determining whether to perform such a cut may be determined based on a threshold value (e.g., block size or distance). In another example, such a threshold may be signaled in the SPS / PPS / slice group header.
[0309] b. When there is motion information above the current CTU in the upper HMVP table (ie, the upstream buffer), it can be cut off. When the CU is located at the upper boundary of the CTU, the cutting of the HMVP merge / AMVP candidate can be skipped.
[0310] 4. Tectonic movement information from two different tables can be added to the new HMVP table.
[0311] a. Each motion information in the HMVP table has position information, such as x and y offset from the upper left corner of the current CTU, and a flag indicating whether it is the motion information of the current CTU. Alternatively, there is a CTU-level syntax element that indicates that the motion information from the upper or left CTU should be used.
[0312] b. Generate new motion information using one motion information from the left CTU and another motion information from the upper CTU available in the HMVP table.
[0313] i. New motion information considering the upper left corner can be derived based on the distance between the two motion information in the HMVP table. An example is as follows. Assume that the motion information of the blue CU in the left and upper CTU is in the HMVP table, as follows.
[0314] Fig.25 An example of how new motion information can be constructed from neighboring motion information is shown.
[0315] DiffMVx=TopMVx-LeftMVx
[0316] DiffMVy=TopMVy-LeftMVy
[0317]
[0318]
[0319] 5. Considering its location, one variant for constructing motion information can be considered as a candidate for the merge list in each CU.
[0320] a. Two motion information can be selected with any combination of different neighboring HMVP tables including the current HMVP table. The following are some examples, and are not limited to those combinations.
[0321] i. The first example is one from the left HMVP table, the other is from the upper HMVP table.
[0322] ii. The second example is one from the left HMVP table and the other is from the current HMVP table.
[0323] iii. The third example is one from the previous HMVP table and the other is from the current HMVP table.
[0324] iv. The fourth examples are all from the current HMVP table.
[0325] b. The structural motion information in the CU can be used as a candidate for various merge lists that are not currently used. One example is the triangle merge list.
[0326] c. The following is an example to construct new motion information using two motion information from the left and top HMVP tables. The distance metric may include but is not limited to L1 norm and L2 norm. DiffMVx and DiffMVy are the same as in 5.b.
[0327] Fig.26 An example of how motion information can be constructed from neighboring motion information is shown.
[0328] Xdistance=CTUSIZE-xl+xc
[0329] Ydistance=CTUSIZE-yt+yc
[0330]
[0331]
[0332] 6. When sub-CTU level parallel processing is available, such as VPDU or parallel processing size (i.e., predefined area or any size defined in SPS / PPS / slice group header), the above techniques can be applied to each sub-CTU row or parallel processing size.
[0333] a. In one example, the HMVP table is reset at the beginning of each sub-CTU row within a slice group.
[0334] b. In one example, the inherited HMVP tables from the top-left, top-right sub-CTUs should be available to construct a new HMVP table for the current sub-CTU.
[0335] c. In one example, only the inherited HMVP tables from the top-left and top sub-CTU regions can be used to construct a new HMVP table for the current sub-CTU region.
[0336] d. In one example, any combination of HMVP tables from the left, upper-left, upper, and upper-right sub-CTU regions may be used to construct a new HMVP table for the current sub-CTU region.
[0337] 7. To improve the parallel processing or parallel processing size in each VPDU (predefined area or size defined in SPS / PPS / slice group header), the HMVP table can be generated by any of the above methods and used for the current parallel processing size without updating.
[0338] 8. The HMVP table associated with a region (e.g., left CTU, left CTU, sub-CTU, or sub-CTU or parallel processing size) can use memory compression (e.g., mantissa + exponent) to save bandwidth reading from DDR memory.
[0339] 5. Example Implementation of Disclosed Technology
[0340] Fig. 27 2700 is a flow chart illustrating an example method of video processing. The method includes initializing (2702) a motion vector predictor (MVP) table from a plurality of stored MVP tables during a conversion between a video region and a bitstream representation of the video region, wherein each of the plurality of MVPs includes motion information previously considered during the conversion; and performing (2704) the conversion using an MVP table selected from the plurality of stored MVP tables.
[0341] Fig.28 is a block diagram of a video processing device 2800. The device 2800 may be used to implement one or more of the methods described herein. The device 2800 may be embodied in a smartphone, a tablet, a computer, an Internet of Things (IoT) receiver, etc. The device 2800 may include one or more processors 2802, one or more memories 2804, and video processing hardware 2806. The processor(s) 2802 may be configured to implement one or more of the methods described herein. The memory(s) 2804 may be used to store data and code used to implement the methods and techniques described herein. The video processing hardware 2806 may be used to implement some of the techniques described in this document in hardware circuits.
[0342] The following clause-based description provides additional embodiments.
[0343] 1. A method for video processing, comprising: initializing a motion vector predictor (MVP) table from a plurality of stored MVP tables during a conversion between a video region and a bitstream representation of the video region, wherein each of the plurality of MVPs includes motion information previously considered during the conversion; and performing the conversion using an MVP table selected from the plurality of stored MVP tables.
[0344] 2. The method of clause 1, further comprising: associating the MVP table with the video region; and storing the MVP table for subsequent use during conversion.
[0345] 3. The method of any of clauses 1-2, wherein initializing the MVP table comprises initializing the MVP table using one or more MVP tables of adjacent video regions.
[0346] 4. The method of clause 3, wherein the adjacent video region comprises a left region of the video region.
[0347] 5. The method of clause 3, wherein the adjacent video region comprises an upper left region of the video region.
[0348] 6. The method of clause 3, wherein the adjacent video region comprises an upper right region of the video region.
[0349] 7. The method of clause 3, wherein the adjacent video region comprises an upper region of the video region.
[0350] 8. The method of any one of clauses 1 to 7, wherein the MVP table is initialized as a combination of some of the plurality of MVP tables.
[0351] 9. The method of clause 8, wherein the MVP table is initialized using inherited motion information from some of the plurality of MVP tables.
[0352] 10. A method according to any one of clauses 8 or 9, wherein the MVP table is generated by joining from some of the plurality of MVP tables in a predetermined order.
[0353] 11. The method of clause 10, wherein the predetermined order specifies adding all motion information of a first MVP table, then adding all motion information of a second table, until all MVP information from some of the plurality of tables is added.
[0354] 12. A method according to clause 10, wherein the predetermined order specifies adding all first motion information from some of the multiple MVP tables, then adding all second motion information from some of the multiple MVP tables, until all motion information from some of the multiple MVP tables is added.
[0355] 13. The method of clause 1, wherein the initializing comprises selectively clipping motion information from the plurality of MVP tables based on a clipping criterion.
[0356] 14. The method of clause 13, wherein the cropping criterion specifies motion information of a next codec tree unit from which the video region is cropped.
[0357] 15. The method of clause 13, wherein the cropping criteria specifies skipping of merge and replacement MVP candidates for the video region that satisfies the skipping criteria at a left boundary or a top boundary of a coding tree unit of the video.
[0358] 16. The method of clause 15, wherein the skip criteria specifies a distance to indicate how far from the current video region.
[0359] 17. A method according to any of clauses 13-16, wherein the cropping criterion specifies skipping the addition of new motion information to the MVP table if the new motion information matches the motion information of an entry from the left column of the video region.
[0360] 18. A method according to any of clauses 13-17, wherein the cropping criterion specifies to crop motion information in an upper MVP table corresponding to motion information above a current codec tree unit of the video region.
[0361] 19. A method for video processing, comprising: during a conversion between a video region and a bitstream representation of the video region, initializing one or more entries of a motion vector predictor (MVP) table from one or more stored MVP tables, wherein each MVP table includes motion information previously considered during the conversion, a flag indicating that the motion information is inherited from a previous MVP table or added as new motion information to the current region; and performing the conversion using an MVP table selected from a plurality of stored MVP tables.
[0362] 20. The method according to clause 19, further comprising, after the converting, storing one or more entries of the MVP table by associating the MVP table with the video region, wherein the storing comprises storing position information of each motion information in the MVP table.
[0363] 21. The method of clause 20, wherein the position information comprises an (x, y) position of an upper left corner of a codec tree unit of the video region.
[0364] 22. The method of clause 19, wherein the constructing comprises deriving new motion information for an entry of the MVP table using first motion information from a left codec tree unit to the left of the video region and second motion information from an upper codec tree unit above the video region.
[0365] 23. The method of clause 19, wherein the constructing comprises deriving new motion information for an entry of the MVP table using first motion information from a left codec tree unit on the left side of the video region and an entry from an MVP table for the video region.
[0366] 24. A method according to any of clauses 19-23, wherein the new motion information is derived for constructing an MVP table at a codec tree unit of the video region.
[0367] 25. A method according to any of clauses 19-23, wherein new motion information is derived for constructing an MVP table at a sub-codec tree unit of the video region.
[0368] 26. A method according to any of clauses 1 to 25, wherein the video region corresponds to a sub-codec tree unit region, and wherein the conversion comprises initializing the MVP table in parallel with other MVP tables of other video regions in the codec tree unit of the video region.
[0369] 27. The method of any of clauses 1 or 19, wherein the MVP table is reset at the beginning of each sub-codec tree unit row in a slice group of the video.
[0370] 28. The method of any one of clauses 1 to 27, further comprising: after the converting, updating the MVP table.
[0371] 29. The method of any one of clauses 1 to 27, wherein since the video region is a sub-codec unit level region, the MVP table is used and used in a virtual data pipeline unit without updating the MVP table.
[0372] 30. A method as claimed in any one of clauses 1 to 29, wherein the video region comprises a video block.
[0373] 31. A method according to any of clauses 1 to 30, wherein the converting comprises generating the bitstream representation from the video region or generating the video region from the bitstream representation.
[0374] 32. An apparatus comprising a processor configured to implement the method of any one of clauses 1 to 31.
[0375] 33. A computer-readable medium having stored thereon a program code executable by a processor for implementing the method of any one of clauses 1 to 31.
[0376] From the foregoing, it will be appreciated that specific embodiments of the presently disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without departing from the scope of the invention. Accordingly, the presently disclosed technology is not to be limited, except as set forth in the appended claims.
[0377] Fig.29 is a flow chart of an example method 2900 for video processing. The method 2900 includes, for a first region of a video and a conversion between a bitstream representation of the first region, initializing (2902) a first historical motion vector predictor (HMVP) table associated with the first region based on at least one of a plurality of HMVP tables stored, the plurality of HMVP tables being used to store motion information associated with a region encoded prior to the first region; and performing (2904) the conversion by using the initialized first HMVP table including at least one motion candidate.
[0378] In some examples, the plurality of HMVP tables includes one or more HMVP tables associated with neighboring regions of the first region.
[0379] In some examples, the first HMVP table is initialized based on a plurality of HMVP tables associated with neighboring regions of the first region.
[0380] In some examples, the first HMVP table is initialized based on an HMVP table associated with a neighboring region of the first region.
[0381] In some examples, the multiple HMVP tables associated with neighboring regions of the first region include at least one of an HMVP table associated with the left region, an HMVP table associated with the upper region, an HMVP table associated with the upper left region, and an HMVP table associated with the upper right region of the first region.
[0382] In some examples, the plurality of HMVP tables associated with neighboring regions of the first region include an HMVP table associated with an upper region and / or an HMVP table associated with an upper-right region of the first region.
[0383] In some examples, the method further includes associating the first HMVP table with the first region; and storing the first HMVP table for subsequent use after conversion of the first region.
[0384] In some examples, the region is a codec tree unit (CTU) or a virtual pipeline data unit (VPDU).
[0385] Fig.30 is a flow chart of an example method 3000 for video processing. The method 3000 includes, for a first region of a video and a conversion between a bitstream representation of the first region, generating (3002) a new historical motion vector predictor (HMVP) table associated with the first region based on a plurality of HMVP tables associated with the other regions, the plurality of HMVP tables being used to store motion information associated with the other regions encoded prior to the first region; and performing (3004) the conversion by using the generated new HMVP table including at least one motion candidate.
[0386] In some examples, the region is a codec tree unit (CTU) or a virtual pipeline data unit (VPDU).
[0387] In some examples, inherited motion information from multiple HMVP tables is combined into a new HMVP table.
[0388] In some examples, the new HMVP table is generated by adding all motion information from a predetermined number of HMVP tables among the plurality of HMVP tables in a predetermined order.
[0389] In some examples, the predetermined order is to add all motion information from a first HMVP table, then add all motion information from a second HMVP table, until all motion information from a predetermined number of HMVP tables are added in sequence.
[0390] In some examples, the predetermined number is three.
[0391] In some examples, the predetermined order is to add first motion information from all predetermined number of HMVP tables, then add second motion information from all predetermined number of HMVP tables, until all motion information from the predetermined number of HMVP tables are interleaved and added.
[0392] In some examples, the predetermined number is two or three.
[0393] In some examples, the first HMVP table is generated by combining a plurality of HMVP tables associated with neighboring regions of the first region.
[0394] In some examples, the plurality of HMVP tables associated with neighboring regions of the first region include an HMVP table associated with the left region, an HMVP table associated with the top region, an HMVP table associated with the top left region, and an HMVP table associated with the top right region of the first region.
[0395] In some examples, the plurality of HMVP tables associated with neighboring regions of the first region include an HMVP table associated with an upper region and an HMVP table associated with an upper-right region of the first region.
[0396] In some examples, before adding the motion information in the multiple HMVP tables to the new HMVP table, the motion information in the multiple HMVP tables associated with other regions is pruned based on a pruning criterion.
[0397] In some examples, when the region is a CTU and motion information from a left column buffer of a first CTU is present in a left HMVP table associated with a CTU on the left side of the first CTU, the motion information is pruned from the left HMVP table, where the left column buffer of the first CTU is the rightmost column of a left CTU of the first CTU.
[0398] In some examples, when the CU is located at the left boundary of a CTU, HMVP merge and / or clipping of AMVP candidates is skipped.
[0399] In some examples, when the motion information stored in any entry of the left HMVP table matches any of the motion information from the left column buffer of the first CTU, the motion information is removed from the left HMVP table.
[0400] In some examples, when motion information stored in any entry of the left HMVP table is within a certain distance from any of the motion information from the left column buffer of the first CTU, the motion information is removed from the left HMVP table.
[0401] In some examples, the distance metric includes at least one of an L1 norm and an L2 norm, wherein the L1 norm is calculated as the sum of the absolute values of the vectors and the L2 norm is calculated as the square root of the sum of the squared vector values.
[0402] In some examples, determining whether clipping occurs is based on a threshold, the threshold being associated with a size or distance of the first region.
[0403] In some examples, the threshold is signaled in at least one of a sequence parameter set (SPS), a picture parameter set (PPS), and a slice group header.
[0404] In some examples, when the region is a CTU and motion information from an upstream buffer of a first CTU is present in an upper HMVP table associated with a CTU above the first CTU, the motion information is pruned from the upper HMVP table, where the upstream buffer of the first CTU is the lowest line of the CTU above the first CTU.
[0405] In some examples, when the CU is located at the upper boundary of a CTU, HMVP merge and / or clipping of AMVP candidates is skipped.
[0406] In some examples, constructed motion information constructed from two motion information from two different HMVP tables of the plurality of HMVP tables is added to a new HMVP table.
[0407] In some examples, each motion information in the HMVP table includes position information, which includes a horizontal offset and a vertical offset relative to the upper left corner of the current CTU, and a flag indicating whether it is the motion information of the current CTU.
[0408] In some examples, a CTU level syntax element is provided to indicate that motion information from the above or left CTU will be used.
[0409] In some examples, the motion information is constructed from motion information from an upper HMVP table associated with an upper region of the first region and motion information from a left HMVP table associated with a left region of the first region.
[0410] In some examples, the constructed motion information is constructed based on the difference (DiffMV) between the motion information from the top HMVP table (TopMV) and the motion information from the left HMVP table (LefMV).
[0411] In some examples, the difference (DiffMV) is calculated as follows:
[0412] DiffMVx=TopMVx-LeftMVx
[0413] DiffMVy=TopMVy-LeftMVy
[0414] where DiffMVx is the horizontal component of the difference DiffMV, TopMVx is the horizontal component of the motion information TopMV from the top HMVP table, and LefMVx is the horizontal component of the motion information LefMV from the left HMVP table,
[0415] DiffMVy is the vertical component of the difference DiffMV, TopMVy is the vertical component of the motion information TopMV from the top HMVP table, and fMVy is the vertical component of the motion information LefMV from the left HMVP table.
[0416] In some examples, the construction motion information MV is calculated as follows:
[0417]
[0418]
[0419] Wherein MVx is the horizontal component of MV, TopXOffset is the horizontal offset of the motion information TopMV, and LeftXOffset is the horizontal offset of the motion information LefMV, MVy is the horizontal component of MV, TopYOffset is the vertical offset of the motion information TopMV, and LeftYOffset is the vertical offset of the motion information LefMV, and CTUSIZE is the width and / or height of the current CTU.
[0420] In some examples, constructed motion information constructed from two motion information from one or more HMVP tables in a plurality of HMVP tables is used as a candidate for a merge list in a codec unit (CU) of the first region.
[0421] In some examples, the two motion information are selected from any combination of neighboring HMVP tables including the current HMVP table.
[0422] In some examples, one motion information comes from the left HMVP table and the other motion information comes from the top HMVP table.
[0423] In some examples, one motion information comes from the left HMVP table and the other motion information comes from the current HMVP table.
[0424] In some examples, one motion information comes from a previous HMVP table and the other motion information comes from a current HMVP table.
[0425] In some examples, both motion information comes from the current HMVP table.
[0426] In some examples, the construction motion information in the CU is used as a candidate for various merge lists that are not currently used.
[0427] In some examples, the various merge lists include a triangular merge list.
[0428] In some examples, the constructed motion information is constructed based on the difference (DiffMV) between the motion information from the top HMVP table (TopMV) and the motion information from the left HMVP table (LefMV) and the distance between the two motion information and the current CU.
[0429] In some examples, the difference (DiffMV) is calculated as follows:
[0430] DiffMVx=TopMVx-LeftMVx
[0431] DiffMVy=TopMVy-LeftMVy
[0432] where DiffMVx is the horizontal component of the difference DiffMV, TopMVx is the horizontal component of the motion information TopMV from the top HMVP table, and fMVx is the horizontal component of the motion information LefMV from the left HMVP table,
[0433] DiffMVy is the vertical component of the difference DiffMV, TopMVy is the vertical component of the motion information TopMV from the top HMVP table, and fMVy is the vertical component of the motion information LefMV from the left HMVP table, and the distance is calculated as follows:
[0434] Xdistance=CTUSIZE-xl+xc
[0435] Ydistance=CTUSIZE-yt+yc
[0436] Where Xdistance is the distance between the motion information LeftMV from the left HMVP table and the current CU, xl is the horizontal distance between the motion information in the CU associated with LeftMV and the current CU, xc is the horizontal offset of the motion information in the current CU, Ydistance is the distance between the motion information TopMV from the top HMVP table and the current CU, yt is the vertical distance between the motion information in the CU associated with TopMV and the current CU, yc is the vertical offset of the motion information in the current CU, and TUSIZE is the width and / or height of the current CTU.
[0437] In some examples, the construction motion information MV is calculated as follows:
[0438]
[0439]
[0440] Wherein MVx is the horizontal component of MV, TopXOffset is the horizontal offset of the motion information TopMV, and LeftXOffset is the horizontal offset of the motion information LefMV, MVy is the horizontal component of MV, TopYOffset is the vertical offset of the motion information TopMV, and LeftYOffset is the vertical offset of the motion information LefMV, and CTUSIZE is the width and / or height of the current CTU.
[0441] In some examples, when sub-CTU level parallel processing is enabled, the region includes a virtual pipeline data unit (VPDU) or a parallel processing size defined in at least one of an SPS, a PPS, and a slice group header.
[0442] In some examples, the HMVP table is reset at the beginning of each sub-CTU row within a slice group.
[0443] In some examples, an inherited HMVP table from one or more of the top-left sub-CTU, the top sub-CTU, and the top-right sub-CTU may be used to construct a new HMVP table for the current sub-CTU.
[0444] In some examples, only the inherited HMVP tables from the top-left sub-CTU and the top sub-CTU may be used to construct a new HMVP table for the current sub-CTU.
[0445] In some examples, any combination of HMVP tables from the left sub-CTU, the upper-left sub-CTU, the upper sub-CTU, and the upper-right sub-CTU may be used to construct a new HMVP table for the current sub-CTU.
[0446] In some examples, a virtual pipeline data unit (VPDU) or parallel processing size defined in at least one of an SPS, a PPS, and a slice group header uses a generated HMVP table in parallel processing without updating.
[0447] In some examples, the HMVP table associated with a region uses memory compression to conserve bandwidth reading from the DDR memory.
[0448] In some examples, memory compression includes the mantissa and the exponent.
[0449] In some examples, the transform generates a first region of the video from the bitstream representation.
[0450] In some examples, the transform generates a bitstream representation from the first region of the video.
[0451] The implementation of the subject matter and functional operations described in this patent document can be implemented in various systems, digital electronic circuits or computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more thereof. The implementation of the subject matter described in this specification can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded and decoded on a tangible and non-transitory computer-readable medium to be executed by a data processing device or to control the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, a material composition that affects a machine-readable propagation signal, or a combination of one or more thereof. The term "data processing unit" or "data processing device" covers all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof.
[0452] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language (including compiled or interpreted languages) and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to be executed on one or more computers located at one site or distributed across multiple sites and interconnected by a communications network.
[0453] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and the apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
[0454] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively connected to one or more large-capacity storage devices to receive data from or transmit data to them, or both. However, a computer does not necessarily have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices. The processor and memory can be supplemented by, or incorporated into, a dedicated logic circuit.
[0455] Intended to combine the instructions and the Figure 1 The foregoing are to be considered as exemplary only, where exemplary means example. As used herein, the use of "or" is intended to include "and / or" unless the context clearly indicates otherwise.
[0456] Although this patent document contains many details, they should not be interpreted as limitations on the scope of any invention or claim, but rather as descriptions of features for specific embodiments of specific inventions. Some features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments, or in any suitable subcombination. Moreover, although features may be described above as working in some combinations, or even initially required to be so, in some cases, one or more features in the claim combination may be removed from the combination, and the claim combination may involve subcombinations or variations of subcombinations.
[0457] Similarly, although operations are described in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order or order shown, or that all described operations be performed, in order to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.
[0458] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and shown in this patent document.
Claims
1. A method for processing a video, comprising: For conversion between a first region of a video and a bitstream representation of the first region, initializing a first HMVP table associated with the first region based on a plurality of stored historical motion vector predictor HMVP tables, the plurality of HMVP tables being used to store motion information associated with a region encoded and decoded before the first region; as well as performing the conversion by using an initialized first HMVP table including at least one motion candidate; The plurality of HMVP tables include a plurality of HMVP tables associated with neighboring regions of the first region, and the first HMVP table is initialized based on the plurality of HMVP tables associated with neighboring regions of the first region.
2. The method of claim 1 , wherein the plurality of HMVP tables associated with neighboring regions of the first region include an HMVP table associated with a left region, an HMVP table associated with an upper region, an HMVP table associated with an upper-left region, and an HMVP table associated with an upper-right region of the first region. 3 . The method of claim 1 , wherein the plurality of HMVP tables associated with neighboring regions of the first region include an HMVP table associated with an upper region and an HMVP table associated with an upper-right region of the first region.
4. The method according to claim 1, further comprising: associating the first HMVP table with the first region; as well as The first HMVP table is stored for subsequent use after conversion of the first region. 5 . The method of claim 1 , wherein the region is a codec tree unit (CTU) or a virtual pipeline data unit (VPDU).
6. The method of any one of claims 1-5, wherein the converting generates a first region of the video from the bitstream.
7. The method of any one of claims 1-5, wherein the converting generates the bitstream from a first region of the video.
8. A method for processing a video, comprising: For conversion between a first region of a video and a bitstream representation of the first region, generating a new HMVP table associated with the first region based on a plurality of historical motion vector predictor HMVP tables associated with other regions, the plurality of HMVP tables being used to store motion information associated with adjacent regions encoded and decoded before the first region; as well as The conversion is performed by using a generated new HMVP table including at least one motion candidate.
9. The method of claim 8, wherein the region is a codec tree unit (CTU) or a virtual pipeline data unit (VPDU).
10. The method of claim 8, wherein inherited motion information from the plurality of HMVP tables is combined into a new HMVP table.
11. The method of claim 8, wherein the new HMVP table is generated by adding all motion information from a predetermined number of HMVP tables among the plurality of HMVP tables in a predetermined order.
12. The method according to claim 11, wherein the predetermined order is to add all motion information from a first HMVP table, then add all motion information from a second HMVP table, until all motion information from the predetermined number of HMVP tables are added in sequence. The method of claim 12 , wherein the predetermined number is three.
14. The method of claim 11, wherein the predetermined order is to add first motion information from all predetermined number of HMVP tables, then add second motion information from all predetermined number of HMVP tables, until all motion information from the predetermined number of HMVP tables are interleaved and added. The method according to claim 14 , wherein the predetermined number is 2 or 3.
16. The method of claim 12, wherein the first HMVP table is generated by combining a plurality of HMVP tables associated with neighboring regions of the first region.
17. The method of claim 16, wherein the plurality of HMVP tables associated with neighboring regions of the first region include an HMVP table associated with a left region, an HMVP table associated with an upper region, an HMVP table associated with an upper-left region, and an HMVP table associated with an upper-right region of the first region.
18. The method of claim 16, wherein the plurality of HMVP tables associated with neighboring regions of the first region include an HMVP table associated with an upper region and an HMVP table associated with an upper-right region of the first region.
19. The method of claim 8, wherein before adding the motion information in the plurality of HMVP tables to the new HMVP table, the motion information in the plurality of HMVP tables associated with the neighboring regions is pruned based on a pruning criterion.
20. The method of claim 19, wherein when the region is a CTU and motion information from a left column buffer of a first CTU is present in a left HMVP table associated with a CTU on the left side of the first CTU, the motion information is pruned from the left HMVP table, wherein the left column buffer of the first CTU is a rightmost column of a left CTU of the first CTU.
21. The method of claim 20, wherein when the CU is located at the left boundary of the CTU, HMVP merge and / or AMVP candidate clipping is skipped.
22. The method of claim 20, wherein when motion information stored in any entry of a left HMVP table matches any of the motion information from a left column buffer of the first CTU, the motion information is removed from the left HMVP table.
23. The method of claim 20, wherein when motion information stored in any entry of the left HMVP table is within a certain distance from any of the motion information from the left column buffer of the first CTU, the motion information is removed from the left HMVP table.
24. The method of claim 23, wherein the distance metric comprises at least one of an L1 norm and an L2 norm, wherein the L1 norm is calculated as the sum of the absolute values of the vectors and the L2 norm is calculated as the square root of the sum of the squared vector values.
25. The method of claim 24, wherein determining whether clipping occurs is based on a threshold value, the threshold value being associated with a size of the first area or the distance.
26. The method of claim 25, wherein the threshold is signaled in at least one of a sequence parameter set (SPS), a picture parameter set (PPS), and a slice group header.
27. The method of claim 19, wherein when the region is a CTU and motion information from an upstream buffer of a first CTU exists in an upper HMVP table associated with a CTU above the first CTU, the motion information is pruned from the upper HMVP table, wherein the upstream buffer of the first CTU is the lowest line of a CTU above the first CTU.
28. The method of claim 27, wherein when the CU is located at an upper boundary of the CTU, HMVP merge and / or AMVP candidate clipping is skipped.
29. The method of claim 8, wherein constructed motion information constructed from two motion information from two different HMVP tables of the plurality of HMVP tables is added to a new HMVP table.
30. The method according to claim 29, wherein each motion information in the HMVP table includes position information, the position information including a horizontal offset and a vertical offset relative to the upper left corner of the current CTU, and a flag indicating whether it is the motion information of the current CTU.
31. The method of claim 29, wherein a CTU level syntax element is provided to indicate that motion information from an above or left CTU is to be used.
32. The method of claim 29, wherein the constructed motion information is constructed from motion information from an upper HMVP table associated with an upper region of the first region and motion information from a left HMVP table associated with a left region of the first region.
33. The method of claim 32, wherein the constructed motion information is constructed based on a difference (DiffMV) between motion information from a top HMVP table (TopMV) and motion information from a left HMVP table (LefMV).
34. The method of claim 33, wherein the difference (DiffMV) is calculated as follows: DiffMVx=TopMVx-LeftMVx DiffMVy=TopMVy-LeftMVy where DiffMVx is the horizontal component of the difference DiffMV, TopMVx is the horizontal component of the motion information TopMV from the top HMVP table, and LefMVx is the horizontal component of the motion information LefMV from the left HMVP table, DiffMVy is the vertical component of the difference DiffMV, TopMVy is the vertical component of the motion information TopMV from the top HMVP table, and fMVy is the vertical component of the motion information LefMV from the left HMVP table.
35. The method according to claim 34, wherein the tectonic movement information MV is calculated as follows: Wherein MVx is the horizontal component of MV, TopXOffset is the horizontal offset of the motion information TopMV, and LeftXOffset is the horizontal offset of the motion information LefMV, MVy is the horizontal component of MV, TopYOffset is the vertical offset of the motion information TopMV, and LeftYOffset is the vertical offset of the motion information LefMV, and CTUSIZE is the width and / or height of the current CTU. 36 . The method of claim 8 , wherein constructed motion information constructed from two motion information from a plurality of HMVP tables among the plurality of HMVP tables is used as a candidate for a merge list in a codec unit (CU) of the first region.
37. The method of claim 36, wherein the two motion information are selected from any combination of neighboring HMVP tables including a current HMVP table.
38. The method of claim 37, wherein one motion information is from a left HMVP table and the other motion information is from an upper HMVP table.
39. The method of claim 37, wherein one motion information is from a left HMVP table and the other motion information is from a current HMVP table.
40. The method of claim 37, wherein one motion information is from a previous HMVP table and the other motion information is from a current HMVP table. The method according to claim 37 , wherein the two motion information are both from a current HMVP table.
42. The method of claim 36, wherein construction motion information in the CU is used as a candidate for various merge lists that are not currently in use.
43. The method of claim 42, wherein the various merge lists include triangular merge lists.
44. The method of claim 38, wherein the constructed motion information is constructed based on a difference (DiffMV) between motion information (TopMV) from a top HMVP table and motion information (LefMV) from a left HMVP table and a distance between the two motion information and a current CU.
45. The method of claim 44, wherein the difference (DiffMV) is calculated as follows: DiffMVx=TopMVx-LeftMVx DiffMVy=TopMVy-LeftMVy where DiffMVx is the horizontal component of the difference DiffMV, TopMVx is the horizontal component of the motion information TopMV from the top HMVP table, and fMVx is the horizontal component of the motion information LefMV from the left HMVP table, DiffMVy is the vertical component of the difference DiffMV, TopMVy is the vertical component of the motion information TopMV from the top HMVP table, and fMVy is the vertical component of the motion information LefMV from the left HMVP table, And the distance is calculated as follows: Xdistance=CTUSIZE-xl+xc Ydistance=CTUSIZE-yt+yc Where Xdistance is the distance between the motion information LeftMV from the left HMVP table and the current CU, xl is the horizontal distance between the motion information in the CU associated with LeftMV and the current CU, xc is the horizontal offset of the motion information in the current CU, Ydistance is the distance between the motion information TopMV from the top HMVP table and the current CU, yt is the vertical distance between the motion information in the CU associated with TopMV and the current CU, yc is the vertical offset of the motion information in the current CU, and CTUSIZE is the width and / or height of the current CTU.
46. The method according to claim 45, wherein the tectonic movement information MV is calculated as follows: Wherein MVx is the horizontal component of MV, TopXOffset is the horizontal offset of the motion information TopMV, and LeftXOffset is the horizontal offset of the motion information LefMV, MVy is the horizontal component of MV, TopYOffset is the vertical offset of the motion information TopMV, and LeftYOffset is the vertical offset of the motion information LefMV, and CTUSIZE is the width and / or height of the current CTU.
47. The method of claim 8, wherein when sub-CTU level parallel processing is enabled, the region comprises a virtual pipeline data unit (VPDU) or a parallel processing size defined in at least one of an SPS, a PPS, and a slice group header.
48. The method of claim 47, wherein the HMVP table is reset at the beginning of each sub-CTU row within a slice group.
49. The method of claim 47, wherein inherited HMVP tables from a plurality of a top-left sub-CTU, a top sub-CTU, and a top-right sub-CTU may be used to construct a new HMVP table for a current sub-CTU.
50. The method of claim 47, wherein only inherited HMVP tables from an upper-left sub-CTU and an upper sub-CTU may be used to construct a new HMVP table for a current sub-CTU.
51. The method of claim 47, wherein any combination of HMVP tables from a left sub-CTU, an upper-left sub-CTU, an upper sub-CTU, and an upper-right sub-CTU can be used to construct a new HMVP table for a current sub-CTU.
52. The method of claim 8, wherein a virtual pipeline data unit (VPDU) or a parallel processing size defined in at least one of an SPS, a PPS, and a slice group header uses the generated HMVP table in the parallel processing without updating.
53. The method of claim 52, wherein the HMVP table associated with the region uses memory compression to conserve bandwidth reading from DDR memory.
54. The method of claim 53, wherein the memory compression includes a mantissa and an exponent.
55. The method of any of claims 8-54, wherein the converting generates a first region of the video from the bitstream.
56. The method of any one of claims 9-54, wherein the converting generates the bitstream from a first region of the video.
57. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of any one of claims 1 to 56.
58. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising a program code, which when executed by a processor performs the method according to any one of claims 1 to 56.
