Flag indication method in intra-block copy mode
By applying the merge mode of intra-block copying and motion vector difference in video encoding and decoding, combining inter-intra prediction, the encoding and decoding process of video blocks is optimized, and motion vector derivation in the inter-prediction mode in the prior art is solved. and signaling notification complexity and inefficiency improve the efficiency of video encoding and codec and bandwidth usage efficiency.
Patent Information
- Application Number
- CN201980077195.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-11-29
- Filing Date
- 2019-11-29
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-11-29
AI Technical Summary
When existing video encoding and decoding technologies deal with high-resolution video, there is still room for improvement in the encoding and decoding efficiency and bandwidth usage efficiency. Especially in the inter-frame prediction mode, motion vector derivation and signaling notification have complexity and inefficiency. .
A method is proposed to optimize the encoding and decoding process of video blocks by applying intra-block copying (IBC) mode and merge (MMVD) mode of motion vector difference in video encoding and decoding, combining inter-intra prediction mode, and using The reference pictures of the current video block are predicted and converted, reducing the use of unnecessary signaling notifications and prediction modes.
It improves the efficiency of video encoding and decoding and bandwidth usage efficiency, reduces the computational complexity and decoding time, and enhances the processing capability of high-resolution videos.
Smart Images

Figure CN113170167B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority and the benefits of International Patent Application No. PCT / CN2018 / 118167 filed on November 29, 2018. The entire disclosure of International Patent Application No. PCT / CN2018 / 118167 is incorporated by reference as part of the disclosure of this application. Technical Field
[0003] This patent document relates to video encoding and decoding technology, devices and systems. Background Art
[0004] Despite advances in video compression, digital video still accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, bandwidth requirements for digital video usage are expected to continue to grow. Summary of the invention
[0005] Devices, systems and methods are described that relate to digital video coding and, in particular, to motion vector derivation and signaling for intra-block copy (IBC) in video and image coding. The methods are applicable to both existing video coding standards (e.g., High Efficiency Video Coding (HEVC)) and future video coding standards, or video codecs.
[0006] In one representative aspect, the disclosed technology may be used to provide a method of video processing. The method includes: determining that an intra-block copy (IBC) mode applies to a current video block of a video, wherein in the IBC mode, at least one reference picture used by the current video block is a current picture in which the current video block is located; making a decision about disabling a specific codec mode for the current block; and based on the decision, performing a conversion between the current video block and the bitstream representation, wherein the specific codec mode uses motion vectors and non-current pictures to derive a prediction of the video block.
[0007] In another representative aspect, the disclosed technology can be used to provide a method of video processing. The method includes: determining that a current video block is encoded using a specific codec mode; based on the determination, making a decision about disabling an intra-block copy (IBC) mode of the current block, wherein in the IBC mode, at least one reference picture used by the current video block is a current picture in which the current block is located; based on the decision, performing a conversion between the current block and the bitstream representation, wherein the specific codec mode uses a motion vector and a non-current picture to derive a prediction of the current block.
[0008] In another representative aspect, the disclosed technology can be used to provide a method of video processing, the method comprising: determining an affine mode of a current video block, wherein whether the current video block inherits the affine mode from a neighboring block depends on a reference list; and based on the determination, converting between the current video block of a video and a bitstream representation of the video.
[0009] In another representative aspect, the disclosed technology can be used to provide a method of video processing. The method includes: performing a conversion between a current video block of a video and a bitstream representation of the video, wherein during the conversion, an intra-block copy (IBC) mode and a merge with motion vector difference (MMVD) mode are used, and wherein in the IBC mode, at least one reference picture used by the current video block is the current picture in which the current video block is located, and in the MMVD mode, a motion vector of the video block is derived based on a merge motion candidate list and further refined by at least one motion vector difference.
[0010] In another representative aspect, the disclosed technology can be used to provide a method of video processing. The method includes: performing a conversion between a current video block of a video and a bitstream representation of the video, wherein during the conversion, an intra block copy (IBC) mode and an inter-intra prediction mode are used, and wherein in the IBC mode, at least one reference picture used by the current video block is the current picture in which the current video block is located, and in the inter-intra prediction mode, a prediction signal for the current video block is generated based on at least an intra prediction signal and an inter prediction signal.
[0011] In another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: during a conversion between a current video block and a bitstream representation of the current video block, determining decoded information of at least one encoding and decoding method different from an IBC mode, wherein in the IBC mode, at least one reference picture used by the current video block is the current picture in which the current video block is located; based on the decoded information, determining whether to skip signaling of an IBC flag of the current video block in the bitstream representation of the current video block, wherein the IBC flag is related to the IBC mode; and performing the conversion based on the determination.
[0012] In another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: during conversion between a current video block and a bitstream representation of the current video block, wherein the current video block includes a luma component encoded and decoded using a first codec structure tree and a chroma component encoded and decoded using a second codec structure tree different from the first codec structure tree, deriving first motion vector information of a chroma block from second motion vector information of one or more corresponding blocks of the luma component of the current video block; and performing the conversion based on the first motion vector information and the second vector information.
[0013] In another representative aspect, the disclosed technology can be used to provide a method of video processing. The method includes: determining whether to apply an intra-frame block copy (IBC) mode to a current video block, in which at least one reference picture used by the current video block is the same as the picture in which the current video block is located; signaling whether the IBC mode is applied to the current video block; based on determining that the IBC mode is applied to the current video block, avoiding signaling a flag of a group of prediction modes; and performing a conversion of the current video block based on the application of the IBC mode.
[0014] In another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: parsing a bitstream to determine whether an intra-block copy (IBC) mode is applied to the current video block, in which at least one reference picture used by the current video block is the same as the picture in which the current video block is located; based on the application of the IBC mode, performing a conversion of the current video block, wherein based on determining that the CPR is applied to the current video block, a flag of a group of prediction modes is excluded from the bitstream.
[0015] In another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: in the process of obtaining motion information of a current video block, during conversion between the current video block and a bitstream representation of the current video block, wherein the motion information of the current video block is based on at least one affine model of at least one neighboring block of the current video block; determining whether the neighboring block of the current video block uses an intra-frame block copy (IBC) mode, in which at least one reference picture used by the neighboring block is the same as the picture in which the neighboring block is located; based on determining that the neighboring block uses the IBC mode, prohibiting the derivation of affine candidates from the neighboring block for the current video block; and performing the conversion based on the motion information of the current video block.
[0016] In another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: during conversion between a current video block and a bitstream representation of the current video block, dividing the current video block into sub-blocks based on a sub-block-based temporal motion vector prediction (SbTMVP) mode; identifying a corresponding block for the sub-block in a motion source picture with a temporal vector based on motion information of spatial neighboring blocks of the current video block; generating a motion vector and a reference index of the sub-block from the corresponding block of the sub-block based on mode information of the codec of the corresponding block; and performing the conversion based on the generated motion vector and reference index.
[0017] In another representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes: during conversion between a current video block and a bitstream representation of the current video block, dividing the current video block into sub-blocks based on a sub-block-based temporal motion vector prediction (SbTMVP) mode; identifying a corresponding block for the sub-block in a motion source picture with a temporal vector based on mode information of encoding and decoding of spatial neighboring blocks of the current video block; generating a motion vector and a reference index of the sub-block from the corresponding block of the sub-block; and performing the conversion based on the generated motion vector and reference index.
[0018] In yet another representative aspect, the above method is implemented in the form of processor executable code and stored in a computer readable program medium.
[0019] In yet another representative aspect, a device configured or operable to perform the above method is disclosed. The device may include a processor programmed to implement the method.
[0020] In yet another representative aspect, a video decoder device may implement a method as described herein.
[0021] The above-described aspects and features as well as other aspects and features of the disclosed technology are described in more detail in the drawings, the description and the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 An example of constructing a Merge candidate list is shown.
[0023] Figure 2 Examples of locations of spatial candidates are shown.
[0024] Figure 3 An example of candidate pairs that are subject to redundancy checking for spatial-domain Merge candidates is shown.
[0025] Figure 4A and Figure 4B An example of the position of the second prediction unit PU based on the size and shape of the current block is shown.
[0026] Figure 5 An example of motion vector scaling for temporal merge candidates is shown.
[0027] Figure 6 An example of candidate positions for time-domain Merge candidates is shown.
[0028] Figure 7 An example of generating a combined bi-predictive Merge candidate is shown.
[0029] Figure 8 An example of constructing motion vector prediction candidates is shown.
[0030] Fig. 9 An example of motion vector scaling for spatial motion vector candidates is shown.
[0031] Fig.10 An example of motion prediction using an alternative temporal motion vector prediction (ATMVP) algorithm for a codec unit (CU) is shown.
[0032] Fig.11 An example of a codec unit (CU) with sub-blocks and neighboring blocks used by a spatial-temporal motion vector prediction (STMVP) algorithm is shown.
[0033] Fig.12 An example flow chart for encoding using different MV precisions is shown.
[0034] Fig.13A and 13B Examples of a simplified 4-parameter affine model and a simplified 6-parameter affine model are shown, respectively.
[0035] Fig.14 An example of an affine motion vector field (MVF) for each sub-block is shown.
[0036] Fig.15A and 15B Examples of 4-parameter and 6-parameter affine models are shown, respectively.
[0037] Fig.16 An example of motion vector prediction of AF_INTER for an inherited affine candidate is shown.
[0038] Fig.17 An example of motion vector prediction of AF_INTER for a constructed affine candidate is shown.
[0039] Fig.18A and 18B Example candidates for the AF_MERGE affine motion mode are shown.
[0040] Fig.19 Examples of candidate positions for the affine merge mode are shown.
[0041] Fig. 20 An example of intra-picture block copy is shown.
[0042] Fig.21 Examples of valid corresponding areas in collocated pictures are shown.
[0043] Fig. 22 An example of a codec flow for a history-based motion vector prediction (HMVP) candidate is shown.
[0044] Fig.23 An example of a modified merge list construction process is shown.
[0045] Fig.24 A flow chart of an example method for video processing is shown.
[0046] Fig.25 A flow chart of another example method for video processing is shown.
[0047] Fig.26 A flow chart of yet another example method for video processing is shown.
[0048] Fig. 27 A flow chart of yet another example method for video processing is shown.
[0049] Fig.28 is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques as described in this document.
[0050] Fig.29 A flow chart of yet another example method for video processing is shown.
[0051] Fig.30 A flow chart of yet another example method for video processing is shown.
[0052] Fig.31 A flow chart of yet another example method for video processing is shown.
[0053] Fig.32 A flow chart of yet another example method for video processing is shown.
[0054] Fig.33 A flow chart of yet another example method for video processing is shown.
[0055] Fig.34 A flow chart of yet another example method for video processing is shown.
[0056] Fig.35A flow chart of yet another example method for video processing is shown.
[0057] Fig.36 A flow chart of yet another example method for video processing is shown.
[0058] Fig.37 A flow chart of yet another example method for video processing is shown.
[0059] Fig.38 A flow chart of yet another example method for video processing is shown.
[0060] Fig.39 A flow chart of yet another example method for video processing is shown.
[0061] Fig.40 A flow chart of yet another example method for video processing is shown. DETAILED DESCRIPTION
[0062] Due to the increasing demand for higher resolution video, video coding methods and techniques are ubiquitous in modern technology. Video codecs typically include electronic circuits or software that compress or decompress digital video, and are constantly being improved to provide higher coding efficiency. Video codecs convert uncompressed video to a compressed format and vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end delay (latency). The compression format typically conforms to a standard video compression specification, such as the High Efficiency Video Codec (HEVC) standard (also known as H.265 or MPEG-H Part 2), a general video codec standard to be finalized, or other current and / or future video codec standards.
[0063] Embodiments of the disclosed technology can be applied to existing video codec standards (e.g., HEVC, H.265) and future standards to improve compression performance. Section headings are used in this document to improve the readability of the description and do not in any way limit the discussion or embodiments (and / or implementations) to only the corresponding section.
[0064] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa.
[0065] 1. Example of inter-frame prediction in HEVC / H.265
[0066] Video codec standards have improved significantly over the years and now partially provide high codec efficiency and support for higher resolutions. The latest standards such as HEVC and H.265 are based on a hybrid video codec structure where temporal prediction plus transform coding is used.
[0067] 1.1 Example of prediction mode
[0068] Each inter-predicted PU (prediction unit) has motion parameters for one or two reference picture lists. In some embodiments, the motion parameters include motion vectors and reference picture indices. In other embodiments, the use of one of the two reference picture lists may also be signaled via inter_pred_idc. In other embodiments, the motion vectors may be explicitly encoded and decoded as deltas relative to the predictor.
[0069] When a CU is coded in skip mode, a PU is associated with the CU and there are no significant residual coefficients, no coded motion vector increments or reference picture indices. Merge mode is defined as obtaining the motion parameters of the current PU from neighboring (multiple) PUs, including spatial and temporal candidates. Merge mode can be applied to any inter-predicted PU, not just for skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where the motion vector, for each reference picture list and the corresponding reference picture index used by the reference picture list are explicitly signaled for each PU.
[0070] When signaling indicates that one of the two reference picture lists will be used, a PU is generated from the samples of a block. This is called "unidirectional prediction". Unidirectional prediction can be used for both P slices and B slices.
[0071] When signaling indicates that both reference picture lists are to be used, a PU is generated from samples of both blocks. This is called "bi-prediction". Bi-prediction is only available for B slices.
[0072] Reference Image List
[0073] In HEVC, the term inter prediction is used to refer to predictions derived from data elements (e.g., sample values or motion vectors) of reference pictures other than the currently decoded picture. As in H.264 / AVC, a picture can be predicted from multiple reference pictures. Reference pictures used for inter prediction are organized in one or more reference picture lists. A reference picture index identifies which reference pictures in the list should be used to create a prediction signal.
[0074] A single reference picture list, List 0, is used for P slices, and two reference picture lists, List 0 and List 1, are used for B slices. It should be noted that the reference pictures contained in List 0 / 1 can be pictures from the past and the future in terms of capture / display order.
[0075] 1.1.1 Example of building a candidate for Merge mode
[0076] When predicting a PU using Merge mode, the index pointing to the entry in the merge candidate list is parsed from the bitstream and used to retrieve the motion information. The construction of this list can be summarized as follows:
[0077] Step 1: Initial candidate derivation
[0078] Step 1.1: Spatial Candidate Derivation
[0079] Step 1.2: Redundancy check of spatial candidates
[0080] Step 1.3: Time Domain Candidate Derivation
[0081] Step 2: Additional candidate insertion
[0082] Step 2.1: Create bidirectional prediction candidates
[0083] Step 2.2: Insert zero motion candidates
[0084] exist Figure 1 The construction of the merge candidate list based on the order of the steps summarized above is shown in . For spatial merge candidate derivation, at most four merge candidates are selected from the candidates located at five different positions. For temporal merge candidate derivation, at most one merge candidate is selected from two candidates. Since the number of candidates for each PU is assumed to be constant at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, the index of the best merge candidate is encoded using truncated unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a single merge candidate list, which is the same as the merge candidate list of the 2N×2N prediction unit.
[0085] 1.1.2 Constructing spatial merge candidates
[0086] In the derivation of spatial merge candidates, Figure 2Up to four merge candidates are selected from the candidates in the positions depicted in . The order of derivation is A1, B1, B0, A0, and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is not available (for example, because the PU belongs to another slice or tile) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency.
[0087] In order to reduce computational complexity, not all possible candidate pairs are considered in the proposed redundancy check. Instead, only pairs with Figure 3 The arrows in link pairs, and only if the corresponding candidate for redundancy check has different motion information, then the candidate is added to the list. Another source of duplicate motion information is a "second PU" associated with a partition different from 2N×2N. As an example, Figure 4A and 4B The second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In some embodiments, adding this candidate may result in two prediction units with the same motion information, which is redundant for having only one PU in the codec unit. Similarly, when the current PU is partitioned into 2N×N, position B1 is not considered.
[0088] 1.1.3 Constructing time domain merge candidates
[0089] In this step, only one candidate is added to the list. In particular, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the co-located PU, which belongs to the picture with the smallest picture order count (POC) difference relative to the current picture within a given reference picture list. The reference picture list used for the derivation of the co-located PU is explicitly signaled in the slice header.
[0090] Figure 5An example of the derivation of a scaled motion vector for a temporal merge candidate (shown as a dashed line) is shown, which is scaled from the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal merge candidate is set equal to zero. For B slices, two motion vectors are obtained and combined to produce a bi-directional prediction merge candidate, one for reference picture list 0 and the other for reference picture list 1.
[0091] like Figure 6 As shown, in the collocated PU (Y) belonging to the reference frame, the position for the temporal candidate is selected between candidates C0 and C1. If the PU at position C0 is not available, intra-coded, or outside the current CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merge candidate.
[0092] 1.1.4 Building additional types of merge candidates
[0093] In addition to the spatial-temporal merge candidates, there are two additional types of merge candidates: combined bi-predictive merge candidates and zero merge candidates. Combined bi-predictive merge candidates are generated by utilizing the spatial-temporal merge candidates. Combined bi-predictive merge candidates are only used for B slices. Combined bi-predictive candidates are generated by combining the first reference picture list motion parameters of an initial candidate with the second reference picture list motion parameters of another candidate. If the two tuples provide different motion hypotheses, they will form a new bi-predictive candidate.
[0094] Figure 7 An example of this process is shown where two candidates in the original list (710 on the left) with mvL0 and refIdxL0 or mvL1 and refIdxL1 are used to create a combined bi-predictive merge candidate (720 on the right) that is added to the final list.
[0095] Zero motion candidates are inserted to fill the remaining entries in the merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 for unidirectional prediction and 2 for bidirectional prediction, respectively. In some embodiments, no redundancy check is performed on these candidates.
[0096] In some embodiments, the following steps are performed in sequence until the merge list is full:
[0097] (1) Set the variable numRef to: the number of reference pictures associated with list 0 for P slices, or the minimum number of reference pictures in both lists for B slices;
[0098] (2) Add non-repeating zero motion candidates:
[0099] For variable i being 0...numRef-1, a default motion candidate is added with MV set to (0, 0) and the reference picture index set to i for list 0 (if P slice) or for both lists (if B slices).
[0100] (3) Add repeated zero motion candidates, where MV is set to (0, 0), the reference picture index of list 0 is set to 0 (if it is a P slice), and the reference picture indexes of both lists are set to 0 (if it is a B slice).
[0101] 1.2 Embodiment of Advanced Motion Vector Prediction (AMVP)
[0102] AMVP exploits the spatial-temporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. The motion vector candidate list is constructed by first checking the availability of neighboring PU positions in the time domain to the left and above, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and transmit the corresponding index indicating the selected candidate. Similar to Merge index signaling, unary truncation is used to encode the index of the best motion vector candidate. The maximum value to be encoded in this case is 2 (see Figure 8 ). In the following sections, details about the derivation process of motion vector prediction candidates are provided.
[0103] 1.2.1 Example of deriving AMVP candidates
[0104] Figure 8 The derivation process of motion vector prediction candidates is summarized and can be implemented for each reference picture list with refidx as input.
[0105] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. Figure 2 As shown before, for spatial motion vector candidate derivation, two motion vector candidates are finally derived based on the motion vectors of each PU located at five different positions.
[0106] For temporal motion vector candidate derivation, select one motion vector candidate from two candidates derived based on two different collocated positions. After generating the first list of spatial-temporal candidates, remove duplicate motion vector candidates in the list. If the number of potential candidates is greater than 2, remove motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 from the list. If the number of spatial-temporal motion vector candidates is less than 2, add an additional zero motion vector candidate to the list.
[0107] 1.2.2 Constructing Spatial Motion Vector Candidates
[0108] In the derivation of spatial motion vector candidates, at most two candidates are considered among five potential candidates from the locations such as Figure 2 PUs at the positions shown previously, which are the same as those of the motion Merge. The derivation order for the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order for the upper side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require the use of spatial domain scaling, and two use spatial domain scaling. The four different cases are summarized as follows.
[0109] --No airspace scaling
[0110] (1) Same reference picture list and same reference picture index (same POC)
[0111] (2) Different reference picture lists, but same reference picture index (same POC)
[0112] --Airspace scaling
[0113] (3) Same reference picture list, but different reference picture index (different POC)
[0114] (4) Different reference picture lists and different reference picture indexes (different POCs)
[0115] First check the case where no spatial scaling is done, then check the case where spatial scaling is done. Regardless of the reference picture list, consider spatial scaling when the POC of the reference picture of the neighboring PU is different from the POC of the reference picture of the current PU. If all PUs of the left candidate are unavailable or are intra-coded, then scaling of the upper motion vector is allowed to help parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.
[0116] like Fig. 9As shown in the example in , for the case of spatial scaling, the motion vectors of neighboring PUs are scaled in a similar way to temporal scaling. The difference is that the reference picture list and index of the current PU are given as input; the actual scaling process is the same as the temporal scaling process.
[0117] 1.2.3 Constructing temporal motion vector candidates
[0118] Except for the reference picture index derivation, all the processes for the derivation of temporal merge candidates are the same as those for the derivation of spatial motion vector candidates (e.g. Figure 6 In some embodiments, the reference picture index is signaled to the decoder.
[0119] 2. Example of inter prediction method in Joint Exploration Model (JEM)
[0120] In some embodiments, a reference software called Joint Exploration Model (JEM) is used to explore future video codec technologies. In JEM, sub-block based prediction is used in several codec tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), spatial-temporal motion vector prediction (STMVP), bidirectional optical flow (BIO), frame rate up conversion (FRUC), local adaptive motion vector resolution (LAMVR), overlapped block motion compensation (OBMC), local brightness compensation (LIC), and decoder-side motion vector refinement (DMVR).
[0121] 2.1 Example of motion vector prediction based on sub-CU
[0122] In JEM with quadtree plus binary tree (QTBT), each CU can have at most one set of motion parameters for each prediction direction. In some embodiments, two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU. The optional temporal motion vector prediction (ATMVP) method allows each CU to obtain multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the spatial-temporal motion vector prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using the temporal motion vector prediction and the spatial neighboring motion vector. In some embodiments, in order to retain a more accurate motion field for sub-CU motion prediction, motion compression of the reference frame can be disabled.
[0123] 2.1.1 Example of Alternative Temporal Motion Vector Prediction (ATMVP)
[0124] In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by retrieving multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.
[0125] Fig.10 An example of the ATMVP motion prediction process for CU 1000 is shown. The ATMVP method predicts the motion vector of sub-CU 1001 within CU 1000 in two steps. The first step is to identify the corresponding block 1051 in the reference picture 1050 using the time domain vector. The reference picture 1050 is also called a motion source picture. The second step is to divide the current CU 1000 into sub-CUs 1001 and obtain the motion vector and the reference index of each sub-CU from the block corresponding to each sub-CU.
[0126] In the first step, the reference picture 1050 and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU 1000. In order to avoid repeated scanning process of the neighboring blocks, the first merge candidate in the merge candidate list of the current CU 1000 is used. The first available motion vector and its associated reference picture index are set to the temporal vector and the index of the motion source picture. In this way, the corresponding block can be identified more accurately compared to TMVP, where the corresponding block (sometimes called a collocated block) is always in the lower right or center position relative to the current CU.
[0127] In the second step, the corresponding block of the sub-CU 1051 is identified by the temporal vector in the motion source picture 1050 by adding the temporal vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample point) is used to derive the motion information of the sub-CU. After the motion information of the corresponding N×N block is identified, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, in which motion scaling and other processes are applied. For example, the decoder checks whether the low latency condition is met (for example, the POC of all reference pictures of the current picture is less than the POC of the current picture) and may use the motion vector MVx (for example, the motion vector corresponding to the reference picture list X) to predict the motion vector MVy of each sub-CU (for example, where X is equal to 0 or 1 and Y is equal to 1-X).
[0128] 2.1.2 Example of Spatial-Temporal Motion Vector Prediction (STMVP)
[0129] In the STMVP method, the motion vector of a sub-CU is recursively derived in raster scan order. Fig.11An example of a CU with four sub-blocks and neighboring blocks is shown. Consider an 8×8 CU 1100 including four 4×4 sub-CUs A (1101), B (1102), C (1103) and D (1104). The neighboring 4×4 blocks in the current frame are labeled a (1111), b (1112), c (1113) and d (1114).
[0130] The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighborhood is the N×N block (block c 1113) above sub-CU A (1101). If block c (1113) is not available or is intra-coded, check (from left to right, starting from block c 1113) other N×N blocks above sub-CU A (1101). The second neighborhood is the block on the left side of sub-CU A 1101 (block b 1112). If block b (1112) is not available or is intra-coded, check other blocks on the left side of sub-CU A 1101 (from top to bottom, starting from block b 1112). The motion information obtained from the neighboring blocks for each list is scaled to the first reference frame of the given list. Next, the temporal motion vector prediction (TMVP) of sub-block A1101 is derived by following the same process as the TMVP derivation specified in HEVC. The motion information of the collocated block at block D1104 is retrieved and scaled accordingly. Finally, after extracting and scaling the motion information, all available motion vectors are averaged separately for each reference list. The average motion vector is designated as the motion vector of the current sub-CU.
[0131] 2.1.3 Example of Sub-CU Motion Prediction Mode Signaling
[0132] In some embodiments, sub-CU mode is enabled as an additional merge candidate, and no additional syntax elements are required to signal the mode. Two additional merge candidates are added to merge the candidate list for each CU to represent the ATMVP mode and the STMVP mode. In other embodiments, if the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven merge candidates can be used. The encoding logic of the additional merge candidates is the same as that of the merge candidates in the HM, which means that for each CU in the P or B slice, the two additional merge candidates may require two more RD checks. In some embodiments, for example, JEM, all bins of the merge index are context-coded by CABAC (context-based adaptive binary arithmetic coding and decoding). In other embodiments, for example, HEVC, only the first binary is context-coded and the remaining binary numbers are context-bypass coded.
[0133] 2.2 Example of inter-frame prediction method in VVC
[0134] There are several new codec tools for improving inter prediction, such as Adaptive Motion Vector Difference Resolution (AMVR) for signaling MVD, Affine Prediction Mode, Triangular Prediction Mode (TPM), ATMVP, Generalized Bidirectional Prediction (GBI), Bidirectional Optical Flow (BIO).
[0135] 2.2.1 Example of Codec Block Structure in VVC
[0136] In VVC, a quadtree / binarytree / multitree (QT / BT / TT) structure is used to divide the picture into square or rectangular blocks. In addition to QT / BT / TT, VVC also uses a separate tree (also called a dual codec tree) for I-frames. For separate trees, the codec block structure is signaled separately for the luminance and chrominance components.
[0137] 2.2.2 Example of Adaptive Motion Vector Difference Resolution
[0138] In some embodiments, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the motion vector of the PU and the predicted motion vector) is signaled in units of quarter luma samples. In JEM, Local Adaptive Motion Vector Resolution (LAMVR) is introduced. In JEM, MVD can be coded and decoded in units of quarter luma samples, integer luma samples, or four luma samples. The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled to each CU with at least one non-zero MVD component.
[0139] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or four luma sample MV precision is used.
[0140] When the first MVD resolution flag of a CU is zero, or is not encoded for a CU (meaning that all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When a CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the AMVP candidate list of the CU is rounded to the corresponding precision.
[0141] In the encoder, a CU-level RD check is used to determine which MVD resolution will be used for a CU. In other words, for each MVD resolution, a CU-level RD check is performed three times. In order to speed up the encoder, the following encoding scheme is applied in JEM.
[0142] -- During the RD check of a CU with normal quarter-luminance sample MVD resolution, store the motion information (integer luminance sample precision) of the current CU. For the same CU with integer luminance samples and 4-luminance sample MVD resolution, the stored motion information (after rounding) is used as the starting point for further small-range motion vector refinement during the RD check, so that the time-consuming motion estimation process does not have to be repeated three times.
[0143] -- Conditionally call the RD check of a CU with 4-luminance sample MVD resolution. For a CU, when the RD cost of integer luminance sample MVD resolution is much greater than the RD cost of quarter-luminance sample MVD resolution, skip the RD check of the 4-luminance sample MVD resolution of the CU.
[0144] The encoding process is shown in Fig.12 First, test the 1 / 4 pixel MV, calculate the RD cost, and denote the RD cost as RDCost0. Then test the integer MV and denote the RD cost as RDCost1. If RDCost1 < th * RDCost0 (where th is a positive threshold), then test the 4-pixel MV; otherwise, skip the 4-pixel MV. Basically, when checking the integer or 4-pixel MV, the motion information and RD cost of the 1 / 4 pixel MV are known, etc., which can be reused to accelerate the encoding process of the integer or 4-pixel MV.
[0145] 2.2.3 Examples of Affine Motion Compensation Prediction
[0146] In HEVC, only a translational motion model is applied for motion compensation prediction (MCP). However, the camera and the object can have many kinds of motions, such as zooming in / out, rotation, perspective motion, and / or other irregular motions. In VVC, a simplified affine transform motion compensation prediction with a 4-parameter affine model and a 6-parameter affine model is applied. As Figures 13A-13B shown, the affine motion field of a block is described by two (in the 4-parameter affine model using variables a, b, e, and f) or three (in the 6-parameter affine model using variables a, b, c, d, e, and f) control point motion vectors respectively.
[0147] The motion vector field (MVF) of a block is described by the following equations with a 4-parameter affine model and a 6-parameter affine model respectively:
[0148]
[0149] Here, (mv h 0, mv h 0) is the motion vector of the top-left corner control point (CP), (mv h1, mv h 1) is the motion vector of the right top corner control point, and (mv h 2.mv h 2) is the motion vector of the lower left control point, and (x, y) represents the coordinates of the representative point relative to the upper left sample point in the current block. The CP motion vector can be signaled (such as in affine AMVP mode) or derived on the fly (such as in affine merge mode). w and h are the width and height of the current block. In practice, the division is implemented by rounding and right shifting. In the example, the representative point is defined as the center position of the sub-block. For example, when the coordinates of the upper left corner of the sub-block relative to the upper left sample point in the current block are (xs, ys), the coordinates of the representative point are defined as (xs+2, ys+2). For each sub-block (for example, 4×4), the representative point is used to derive the motion vector of the entire sub-block.
[0150] Fig.14 An example of an affine MVF for each sub-block of block 1400 is shown, where in order to further simplify motion compensated prediction, a sub-block based affine transform prediction is applied. In order to derive the motion vector of each M×N sub-block, the motion vector of the center sample of each sub-block can be calculated according to equations (1) and (2) and rounded to the motion vector fractional accuracy (e.g., 1 / 16 in JEM). Then, a motion compensated interpolation filter can be applied to generate a prediction for each sub-block using the derived motion vector. An interpolation filter of 1 / 16 pixels is introduced through an affine mode. After MCP, the high-precision motion vector of each sub-block is rounded and saved with the same accuracy as the normal motion vector.
[0151] Example of signaling of affine prediction
[0152] Similar to the translational motion model, due to affine prediction, there are also two modes for signaling auxiliary information. They are affine_INTER and affine_MERGE modes.
[0153] 2.2.5 Example of AF_INTER mode
[0154] AF_INTER mode can be applied for CUs with width and height both greater than 8. A CU-level affine flag is signaled in the bitstream to indicate whether AF_INTER mode is used.
[0155] In this mode, for each reference picture list (list 0 or list 1), an affine AMVP candidate list is constructed from three types of affine motion predictors in the following order, where each candidate includes the estimated CPMV of the current block. Fig.17The difference between the best CPMV found by (in mv0 mv1 mv2) and the estimated CPMV is signaled. In addition, the index of the affine AMVP candidate from which the estimated CPMV is derived is further signaled.
[0156] 1) Inherited affine motion predictor
[0157] The checking order is similar to that of the spatial MVP in HEVC AMVP list construction. First, the left inherited affine motion predictor is derived from the first block in {A1, A0} that is affine-coded and has the same reference picture as the current block. Second, the above inherited affine motion predictor is derived from the first block in {B1, B0, B2} that is affine-coded and has the same reference picture as the current block. Fig.16 Five blocks A1, A0, B1, B0, B2 are shown in FIG.
[0158] Once a neighboring block is found to be coded in affine mode, the CPMV of the codec covering the neighboring block is used to derive the predictor of the CPMV of the current block. For example, if A1 is coded in non-affine mode, and A0 is coded in 4-parameter affine mode, the left-inherited affine MV predictor will be derived from A0. In this case, the CPMV of the CU covering A0 is, Fig.18B The upper left CPMV and the upper right CPMV The marked ones are used to derive the estimated CPMV of the current block, using Mark the upper left (coordinate (x0, y0)), upper right (coordinate (x1, y1)), and lower right positions (coordinate (x2, y2)) of the current block.
[0159] 2) Constructed affine motion predictor
[0160] The constructed affine motion predictor includes control point motion vectors (CPMVs) derived from adjacent inter-coded blocks with the same reference picture, such as Fig.17 If the current affine motion model is 4-parameter affine, the number of CPMVs is 2, otherwise, if the current affine motion model is 6-parameter affine, the number of CPMVs is 3. Upper left CPMV Derived from the MV at the first block in the {A, B, C} group that is decoded in inter-frame and has the same reference picture as the current block. Top right CPMV Derived from the MV at the first block in the {D, E} group that is decoded in inter-frame and has the same reference picture as the current block. Bottom-left CPMV Derived by the MV at the first block in the {F, G} group that is decoded with inter-frame coding and has the same reference picture as the current block.
[0161] – If the current affine motion model is 4-parameter affine, then only and
[0162] When both are met, the constructed affine motion predictor is inserted into the candidate list, that is,
[0163] and Used as the upper left (coordinate (x0, y0)) and upper right (coordinate (x1, y1)) of the current block
[0164] The estimated CPMV of the position.
[0165] – If the current affine motion model is 6-parameter affine, then only all
[0166] and , the constructed affine motion predictor will be inserted into the candidate list, that is, Used as the estimated CPMV of the top left (coordinate (x0, y0)), top right (coordinate (x1, y1)), and bottom right (coordinate (x2, y2)) positions of the current block.
[0167] When inserting the constructed affine motion predictor into the candidate list, no pruning process is applied.
[0168] 3) Normal AMVP motion predictor
[0169] The following steps are applied until the number of affine motion predictors reaches a maximum.
[0170] 1) By setting all CPMVs equal to (if available) to derive an affine motion predictor.
[0171] 2) By setting all CPMVs equal to (if available) to derive an affine motion predictor.
[0172] 3) By setting all CPMVs equal to (if available) to derive an affine motion predictor.
[0173] 4) If HEVC TMVP is available, derive the affine motion predictor by setting all CPMVs equal to HEVC TMVP.
[0174] 5) An affine motion predictor is derived by setting all CPMVs to zero MV.
[0175] Note that we have already derived
[0176] In AF_INTER mode, when using 4 or 6-parameter affine mode, 2 or 3 control points are required, so 2 or 3 MVDs need to be encoded for these control points, such as Figures 15A-15B In the existing implementation, MV can be derived as follows, for example, predicting mvd1 and mvd2 from mvd0.
[0177]
[0178] here, mvd i and mv1 are the predicted motion vector, motion vector difference and motion vector of the upper left pixel (i=0), upper right pixel (i=1) or lower left pixel (i=2), respectively. Fig. 15B In some embodiments, adding two motion vectors (e.g., mvA(xA, yA) and mvB(xB, yB)) is equal to adding the two components separately. For example, newMV=mvA+mvB, which means that the two components of newMV are set to (xA+xB) and (yA+yB) respectively.
[0179] 2.2.6 Example of AF_MERGE mode
[0180] When a CU is applied in AF_MERGE mode, it gets the first block coded in affine mode from the valid neighboring reconstructed blocks. And, as Fig.18A As shown in FIG. 1 , the order of selecting candidate blocks is from left, top, top right, bottom left to top left (represented by A, B, C, D, E in sequence). Fig.18B As shown in A0 in FIG, the adjacent lower left block is encoded in affine mode, and the control point (CP) motion vector mv0 of the lower left corner of the upper left corner and the upper right corner of the adjacent CU / PU containing block A is retrieved N , mv1 N and mv2 N . And based on mv0 N , mv1 N and mv2 N Calculate the motion vector mv0 of the upper left corner / upper right / lower left corner of the current CU / PU C , mv1 C and mv2 C(Only for 6-parameter affine model). Note that the sub-block at the upper left corner (e.g. 4×4 block) stores mv0, and if the current block is affine coded, the sub-block at the upper right corner stores mv1. If the current block is coded using the 6-parameter affine model, the sub-block at the lower left corner will store mv2; otherwise (using the 4-parameter affine model), LB stores mv2'. The other sub-blocks store the MV for MC.
[0181] After calculating the CPMV v0 and v1 of the current CU according to the affine motion model in equations (1) and (2), the MVF of the current CU can be generated. In order to identify whether the current CU is encoded in AF_MERGE mode, when at least one neighboring block is encoded in affine mode, an affine flag can be signaled in the bitstream.
[0182] In some embodiments, the affine merge candidate list may be constructed by the following steps:
[0183] 1) Insert inherited affine candidates
[0184] The inherited affine candidates are candidates derived from the affine motion model of the blocks affine-encoded in their effective domain. On a common basis, such as Fig.19 As shown, the scanning order of candidate positions is: A1, B1, B0, A0 and B2.
[0185] After the candidate is derived, a full pruning process is performed to check if the same candidate has been inserted into the list. If there is an identical candidate, the derived candidate is discarded.
[0186] 2) Insert the constructed affine candidate
[0187] If the number of candidates in the affine merge candidate list is less than MaxNumAffineCand (set to 5 in the present disclosure), the constructed affine candidate is inserted into the candidate list. The constructed affine candidate refers to a candidate constructed by combining adjacent motion information of each control point.
[0188] First, from Fig.19 The motion information of the control points is derived from the specified spatial and temporal domains shown. CPk (k = 1, 2, 3, 4) represents the kth control point. A0, A1, A2, B0, B1, B2 and B3 are the spatial locations (k = 1, 2, 3) used to predict CPk; T is the temporal location used to predict CP4.
[0189] The coordinates of CP1, CP2, CP3 and CP4 are (0, 0), (W, 0), (H, 0) and (W, H), respectively, where W and H are the width and height of the current block.
[0190] The motion information for each control point is obtained according to the following priority order:
[0191] For CP1, the priority of checking is B2->B3->A2. If B2 is available, use B2. Otherwise, if B2 is not available, use B3. If both B2 and B3 are not available, use A2. If none of the three candidates are available, the motion information of CP1 cannot be obtained.
[0192] For CP2, the check priority is B1->B0.
[0193] For CP3, the check priority is A1->A0.
[0194] For CP4, use T.
[0195] Second, affine merge candidates are constructed using combinations of control points.
[0196] The motion information of three control points is required to construct a 6-parameter affine candidate. Three control points can be selected from one of the following four combinations: ({CP1, CP2, CP4}, {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4}). The combination of {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4} will be converted into a 6-parameter motion model represented by the top left, top right and bottom left control points.
[0197] The motion information of two control points is required to construct a 4-parameter affine candidate. Two control points can be selected from one of the following six combinations: ({CP1, CP4}, {CP2, CP3}, {CP1, CP2}, {CP2, CP4}, {CP1, CP3}, {CP3, CP4}). The combination {CP1, CP4}, {CP2, CP3}, {CP2, CP4}, {CP1, CP3}, {CP3, CP4} will be converted into a 4-parameter motion model represented by the top left and top right control points.
[0198] Insert the combinations of constructed affine candidates into the candidate list in the following order:
[0199] {CP1, CP2, CP3}, {CP1, CP2, CP4}, {CP1, CP3, CP4}, {CP2, CP3, CP4}, {CP1, CP2}, {CP1, CP3}, {CP2, CP3}, {CP1, CP4}, {CP2, CP4}, {CP3, CP4}
[0200] For a combined reference list X (X is 0 or 1), the most used reference index in the control point is selected as the reference index of list X, and the motion vector pointing to the difference reference picture will be scaled.
[0201] After the candidate is derived, a full pruning process is performed to check if the same candidate has been inserted into the list. If there is an identical candidate, the derived candidate is discarded.
[0202] 3) Fill with zero motion vectors
[0203] If the number of candidates in the affine merge candidate list is less than 5, zero motion vectors with zero reference index are inserted into the candidate list until the list is full.
[0204] In some embodiments, for the sub-block merge candidate list, the MV of the 4-parameter merge candidate is set to (0, 0) and the prediction direction is unidirectional prediction (for P slices) and bidirectional prediction (for B slices) from list 0.
[0205] 2.3 Example of Intra-Block Copy (IBC)
[0206] Intra-block copying (IBC or intra-picture block compensation), also known as current picture reference (CPR), is used in the HEVC Screen Content Codec extension (SCC). This tool is very effective for the encoding and decoding of screen content videos, because repeated patterns in the same picture often appear in text and graphic-rich content. Using previously reconstructed blocks with the same or similar patterns as predictors can effectively reduce the prediction error and thus improve the codec efficiency. An example of intra-block compensation is shown in Fig. 20 Shown in.
[0207] Similar to the design of CRP in HEVC SCC, in VVC, the use of IBC mode is signaled at the sequence and picture levels. When IBC mode is enabled at the sequence parameter set (SPS), it can be enabled at the picture level. When IBC mode is enabled at the picture level, the currently reconstructed picture will be considered as a reference picture. Therefore, no syntax changes are required at the block level on top of the existing VVC inter-frame mode to signal the use of IBC mode.
[0208] Some characteristics of CPR include:
[0209] – It is considered as normal inter mode. Therefore, merge mode and skip mode can also be used in IBC mode. The merge candidate list construction is unified and contains merge candidates from neighboring positions, which are coded in IBC mode or HEVC inter mode. Depending on the selected merge index, the current block in merge mode or skip mode can be merged into the field coded in IBC mode or into the field coded in normal inter mode with different pictures as reference pictures.
[0210] – The block vector prediction and encoding scheme of the IBC mode reuses the scheme used for motion vector prediction and encoding in the HEVC inter-frame mode (AMVP and MVD encoding and decoding).
[0211] – IBC mode motion vectors (also called block vectors) are encoded and decoded with integer pixel precision, but are stored in memory with 1 / 16 pixel precision after decoding because quarter pixel precision is required during the interpolation and deblocking stages. When used for IBC mode motion vector prediction, the stored vector predictor is right-shifted by 4.
[0212] – Search range: limited to the current CTU.
[0213] – CPR is not allowed when Affine Mode / Triangle Mode / GBI / Weighted Prediction is enabled.
[0214] 2.3.1 Coordination with CPR
[0215] In PCT / CN2018 / 089920, several methods for coordinating CPR (also called IBC) and some new encoding and decoding tools are proposed.
[0216] 2.4 Example of merge list design in VVC
[0217] VVC supports three different merge list construction processes:
[0218] (1) Sub-block merge candidate list: It includes ATMVP and affine merge candidates. Both affine mode and ATMVP mode share a merge list construction process. Here, ATMVP and affine merge candidates can be added in sequence. The size of the sub-block merge list is signaled in the stripe header, and the maximum value is 5.
[0219] (2) Unidirectional prediction TPM merge list: For triangular prediction mode, even if the two partitions can choose their own merge candidate indexes, they share a merge list construction process for the two partitions. When building the merge list, the spatial neighboring blocks and two temporal blocks of the block are checked. The motion information derived from the spatial and temporal blocks is called regular motion candidates in our IDF. These regular motion candidates are further used to derive multiple TPM candidates. Note that the transformation is performed at the whole block level, even if the two partitions can use different motion vectors to generate their own prediction blocks. In some embodiments, the unidirectional prediction TPM merge list size is fixed to 5.
[0220] (3) Regular merge list: For the remaining codec blocks, a merge list construction process is shared. Here, spatial / temporal / HMVP, paired combined bidirectional prediction merge candidates, and zero motion candidates can be inserted in order. The size of the regular merge list is signaled in the slice header, and the maximum value is 6.
[0221] Sub-block merge candidate list
[0222] It is proposed to put all sub-block related motion candidates into a separate merge list in addition to the regular merge list of non-sub-block merge candidates.
[0223] Motion candidates associated with sub-blocks are put into a separate merge list called "sub-block merge candidate list".
[0224] In one example, the sub-block merge candidate list includes affine merge candidates, ATMVP candidates, and / or sub-block based STMVP candidates.
[0225] In another example, the ATMVP merge candidate in the normal merge list is moved to the first position of the affine merge list. In this way, all merge candidates in the new list (ie, the sub-block based merge candidate list) are based on the sub-block codec tool.
[0226] In yet another example, in addition to the regular merge candidate list, a special merge candidate list is added, called the sub-block merge candidate list (also called the affine merge candidate list). The sub-block merge candidate list is filled with candidates in the following order:
[0227] (a) ATMVP candidates (may be available or unavailable);
[0228] (b) inherited affine candidates;
[0229] (c) constructed affine candidates;
[0230] (d) MV 4-parameter affine model filled with zeros
[0231] The maximum number of candidates in the sub-block merge candidate list (denoted as ML) is derived as follows:
[0232] (1) If the ATMVP usage flag (e.g., the flag may be named “sps_sbtmvp_enabled_flag”) is on (equal to 1), but the affine usage flag (e.g., the flag may be named “sps_affine_enabled_flag”) is off (equal to 0), then ML is set equal to 1.
[0233] (2) If the ATMVP use flag is off (equal to 0) and the affine use flag is OFF (equal to 0), ML is set to 0. In this case, the sub-block merge candidate list is not used.
[0234] (3) Otherwise (affine use flag is on (equal to 1), ATMVP use flag is on or off), ML is signaled from the encoder to the decoder. Valid ML is 0≤ML≤5.
[0235] When building the sub-block merge candidate list, the ATMVP candidate is checked first. If any of the following conditions is true, the ATMVP candidate is skipped and not put into the sub-block merge candidate list.
[0236] (1) The ATMVP use flag is off;
[0237] (2) Any TMVP usage flag (e.g., in slice-level signaling, the flag may be named “slice_temporal_mvp_enabled_flag”) is off;
[0238] (3) The reference picture with reference index 0 in reference list 0 is the same as the current picture (it is the CPR).
[0239] The ATMVP in the above example is much simpler than in JEM. When generating ATMVP merge candidates, the following process is applied:
[0240] (a) Check in order Figure 2 The adjacent blocks A1, B1, B0, A0 shown are used to find the first inter-coded but not CPR-coded block, marked as block X;
[0241] (b) Initialize TMV=(0,0). If there is an MV for block X (denoted as MVn), then TMV is set equal to MVn with reference to the collocated reference picture (as signaled in the slice header).
[0242] (c) Assuming the center point of the current block is (x0, y0), locate the corresponding position (x0, y0) in the collocated picture as M = (x0 + MV*x, y0 + MV*y). Find the block Z covering M.
[0243] (i) If Z is intra-coded, ATMVP is not applicable;
[0244] (ii) If Z is inter-coded, MVZ_0 and MVZ_1 of the two lists of block Z are scaled to (Reflist0index 0) and (Reflist 1index 1) as MVdefault0, MVdefault1, and stored.
[0245] (d) For each 8×8 sub-block, assume its center point is (x0S, y0S), and then locate the corresponding (x0S, y0S) position in the collocated image as MS = (x0S + MV*x, y0S + MV*y). Find the block ZS covering MS.
[0246] (i) If ZS is intra-coded, MVdefault0 and MVdefault1 are assigned to the sub-block;
[0247] (ii) If the ZS is inter-coded, MVZS_0 and MVZS_1 of the two lists of the block ZS are scaled to (Reflist 0 index 0) and (Reflist 1 index 0) and allocated to the sub-blocks.
[0248] MV clipping and masking in ATMVP : When locating a corresponding position such as M or MS in a collocated picture, it is clipped to be within a predetermined area. The CTU size is S×S, and S=128 in this example. Assuming that the upper left position of the collocated CTU is (xCTU, yCTU), the corresponding position M or MS at (xN, yN) will be clipped to the valid area xCTU≤xN <xCTU+S+4;yCTU≤yN<yCTU+S。 Fig.21 An example of the valid area is shown.
[0249] In addition to clipping, (xN, yN) is also masked as xN = xN & MASK, yN = yN & MASK, where MASK is equal to (2 N -1), and N = 3 to set the lowest 3 bits to 0. Therefore, xN and yN must be multiples of 8 ( stands for the bitwise complement operator).
[0250] Regular merge list
[0251] Different from the merge list design, in VVC, a history-based motion vector prediction (HMVP) method is adopted. In some embodiments, the combined bi-directional prediction merge candidate has been replaced by a pair of bi-directional prediction merge candidates.
[0252] HMVP
[0253] In HMVP, the motion information of the previously coded blocks is stored. The motion information of the previously coded blocks is defined as HMVP candidates. Multiple HMVP candidates are stored in a table called HMVP table, and the table is maintained during the real-time encoding / decoding process. When encoding / decoding of a new slice begins, the HMVP table is cleared. Whenever there is an inter-coded block, the associated motion information is added to the last entry of the table as a new HMVP candidate. The entire encoding and decoding process is as follows Fig. 22 shown.
[0254] HMVP candidates can be used in the construction process of both AMVP and merge candidate lists. Fig.23 A modified merge candidate list construction process is shown (highlighted in gray). When the merge candidate list is not full after the TMVP candidate is inserted, the HMVP candidates stored in the HMVP table can be used to fill the merge candidate list. Considering that a block usually has a higher correlation with the nearest neighboring block in terms of motion information, the HMVP candidates in the table are inserted in descending order of index. The last entry in the table is added to the list first, and the first entry is added last. Similarly, redundancy removal is applied to the HMVP candidates. Once the total number of available merge candidates reaches the maximum number of merge candidates allowed to be signaled, the merge candidate list construction process terminates.
[0255] 2.5 Example of Generalized Bidirectional Prediction (GBi) Improvement
[0256] A generalized bi-prediction improvement (GBi) proposed in one example is provided.
[0257] GBi is proposed in one example. An improved embodiment is provided that improves the gain-complexity tradeoff of GBi, which applies unequal weighting to predictors from L0 and L1 in bidirectional prediction mode. In inter-frame prediction mode, multiple weighted pairs including equal weighted pairs (1 / 2, 1 / 2) are evaluated based on rate-distortion optimization (RDO), and the GBi index of the selected weighted pair is signaled to the decoder. In merge mode, the GBi index is inherited from the adjacent CU. In an embodiment, GBi, the predictor generation in bidirectional prediction mode is shown in the following equation.
[0258] PGBi=(w0*PL0+w1*PL1+RoundingOffsetGBi)>>shiftNumGBi,
[0259] Here, PGBi is the final predictor of GBi. w0 and w1 are the selected GBi weighted pairs and are applied to the predictors of list 0 (L0) and list 1 (L1), respectively. RoundingOffsetGBi and shiftNumGBi are used to normalize the final predictor in GBi. The supported w1 weighting sets are {-1 / 4, 3 / 8, 1 / 2, 5 / 8, 5 / 4}, where the five weights correspond to one equal weighted pair and four unequal weighted pairs, respectively. The mixed gain - that is, the sum of w1 and w0 - is fixed to 1.0. Therefore, the corresponding w0 weighting set is {5 / 4, 5 / 8, 1 / 2, 3 / 8, -1 / 4}. The weighting pair selection is at the CU level.
[0260] For non-low-latency pictures, the weighted set size is reduced from 5 to 3, where the w1 weighted set is {3 / 8, 1 / 2, 5 / 8} and the w0 weighted set is {5 / 8, 1 / 2, 3 / 8}. In the present disclosure, the weighted set size reduction for non-low-latency pictures is applied to GBi and all GBi tests.
[0261] In one embodiment, a combined solution is proposed to further improve GBi performance. Specifically, in an embodiment, the following modifications are applied based on the existing GBi design.
[0262] GBi encoder bug fixes
[0263] In order to reduce GBi encoding time, in the current encoder design, the encoder will store the unidirectional prediction motion vector estimated from the GBi weight equal to 4 / 8 and reuse it for the unidirectional prediction search of other GBi weights. This fast encoding method is applied to both translational motion models and affine motion models. In one example, a 6-parameter affine model and a 4-parameter affine model are adopted. When the GBi weight is equal to 4 / 8, when the encoder stores the unidirectional prediction affine MV, the encoder does not distinguish between the 4-parameter affine model and the 6-parameter affine model. Therefore, after encoding with GBi weight 4 / 8, the 4-parameter affine MV can be covered with the 6-parameter affine MV. For other GBi weights, the stored 6-parameter affine MV can be used for the 4-parameter affine ME, or the stored 4-parameter affine MV can be used for the 6-parameter affine ME. The proposed GBi encoder error repair is to store the 4-parameter affine and 6-parameter affine MV separately. When the GBi weight is equal to 4 / 8, the encoder will store these affine MVs based on the affine model type, and will reuse the corresponding affine MVs for other GBi weights based on the affine model type.
[0264] GBi CU size limit
[0265] In this method, GBi is disabled for small CUs. In inter prediction mode, if bidirectional prediction is used and the CU area is smaller than 128 luma samples, GBi is disabled without signaling.
[0266] Merge mode with GBi
[0267] By Merge mode, the GBi index is not signaled. Instead, it is inherited from the neighboring block it is merged into. When a TMVP candidate is selected, GBi is turned off in that block.
[0268] Affine prediction with GBi
[0269] GBi can be used when the current block is coded with affine prediction. For affine inter mode, the GBi index is signaled. For affine merge mode, the GBi index is inherited from the neighboring block it is merged with. If the constructed affine model is selected, GBi is turned off in this block.
[0270] 2.6 Example of Inter-Intra Prediction (IIP) Mode
[0271] In inter prediction mode, multi-hypothesis prediction combines an intra prediction and a merge index prediction. Such a block is treated as a special inter codec block. In merge CU, when the flag is true, a flag is signaled for merge mode to select intra mode from the intra candidate list. For luma component, the intra candidate list is derived from 4 intra prediction modes, including DC, planar, horizontal and vertical modes, and the size of the intra candidate list can be 3 or 4, depending on the shape of the block. When the CU width is greater than twice the CU height, the horizontal mode is not included in the intra mode list, and when the CU height is greater than twice the CU width, the vertical mode is removed from the intra mode list. One intra prediction mode selected by the intra mode index and one merge index prediction selected by the merge index are combined using a weighted average. For chroma components, DM is always applied without additional signaling.
[0272] The weighting used for combined prediction is described below. Equal weighting will be applied when DC or planar mode is selected or the CB width or height is less than 4. For those CBs with CB width and height greater than or equal to 4, when horizontal / vertical mode is selected, one CB is first divided vertically / horizontally into four equal-area regions. Each weighting set, denoted as (w_intrai, w_interi), where i is 1 to 4, and (w_intra1, w_inter1) = (6, 2), (w_intra2, w_inter2) = (5, 3), (w_intra3, w_inter3) = (3, 5), and (w_intra4, w_inter4) = (2, 6), will be applied to the corresponding region. (w_intra1, w_inter1) is used for the region closest to the reference sample, and (w_intra4, w_inter4) is used for the region farthest from the reference sample. The combined prediction value can then be calculated by adding the two weighted predictions and shifting right by 3 bits. Furthermore, the intra prediction mode of the intra hypothesis used for the predictor may be saved for reference by subsequent neighboring CUs.
[0273] Detailed design in the example
[0274] 1. If all conditions are met, signaling inter_intra_flag (a new context)
[0275] a.Merge but not skip
[0276] b. Non-affine
[0277] c. Non-MMVD
[0278] dW*H<64 and W!=128 and H!=128 (which means 2*8 chroma intra prediction is possible here)
[0279] 2. Inter-frame prediction can be applied to triangular prediction;
[0280] 3. Apply inter-frame prediction to Y, Cb and Cr.
[0281] 4. If inter_intra_flag is on, the intra prediction mode (IPM) will be signaled. IPM must be one of {DC, Planar, Ver, Hor}.
[0282] a. Involving 3 MPMs.
[0283] b. If W>2*H or H>2*W, the IPM must be one of the three MPMs and the MPM flag is not signaled; otherwise, the MPM is signaled (a new context, Why not share the same context with intra-frame? )
[0284] i. If MPM_flag is on, signaling notifies the selected MPM. 0, 10, 11 represent three MPMs (bypass);
[0285] ii. Otherwise, find one in {DC, Planar, Ver, Hor} that is not in the MPM list as the IPM.
[0286] c.MPM list construction (different from Intra MPM list)
[0287] i. Set LeftIPM = AboveIPM = DC;
[0288] ii. If the left neighboring block (A2) is intra-coded or inter-intra-coded,
[0289] leftIPM is set to the IPM of the left neighboring block;
[0290] iii. If the upper neighboring block (B3) is intra-frame coded or inter-frame-intra-frame coded,
[0291] aboveIPM is set to the IPM of the upper neighboring block;
[0292] iv.leftIPM=(leftIPM>DC_IDX)? ((leftIPM≤DIA_IDX)?
[0293] HOR_IDX:VER_IDX):leftIPM;
[0294] v.aboveIPM=(aboveIPM>DC_IDX)?
[0295] ((aboveIPM≤DIA_IDX)?HOR_IDX:VER_IDX):aboveIPM;
[0296] (What about wide-angle mode?)
[0297] vi. If leftIPM == aboveIPM
[0298] 1. If leftIPM is not DC or plane
[0299] a.MPM[0]=leftIPM, MPM[1]=plane, MPM[2]=DC
[0300] 2. Otherwise,
[0301] a.MPM[0]=DC, MPM[1]=Plane, MPM[2]=Ver
[0302] vii. Otherwise
[0303] 1.MPM[0]=leftIPM, MPM[1]=aboveIPM
[0304] 2. If neither leftIPM nor aboveIPM is a plane, then MPM[2]
[0305] = Plane
[0306] 3. Otherwise: MPM[2] = (leftIPM + aboveIPM) < 2? Ver: DC
[0307] viii. If W>2*H
[0308] 1. If MPM[x] == Hor, find the value in {DC, Planar, Ver, Hor}
[0309] If ModeX is in the MPM list but not in the MPM list, set MPM[x] = ModeX;
[0310] ix. If H>2*W
[0311] 1. If MPM[x] == Ver, find the value in {DC, Planar, Ver, Hor}
[0312] ModeX in the list but not in the MPM list, set MPM[x] = ModeX
[0313] 5. It should be understood that the IPM used for inter-intra prediction cannot be used to predict the IPM of intra-coded blocks.
[0314] a. Intra-coded blocks treat inter-intra-coded blocks the same as inter-coded blocks.
[0315] 6. Neighboring sample filtering: As intra prediction:
[0316] a. No filtering for DC / VER / HOR
[0317] b. Planar filtering when W*H > 32;
[0318] c. Keep PDPC
[0319] 7. Weighted sum method
[0320] P(x, y) = Clip((W0(x, y)*P_Intra(x, y) + W1(x, y)*P_Inter(x, y)) >> 3)
[0321] 8. Weights ( Can be improved or simplified )
[0322] a. If mode == DC / Planar || W < 4 || H < 4: W0 = W1 = 4;
[0323] b. If mode == HOR:
[0324] i. If x < W / 4, then W0 = 6, W1 = 2;
[0325] ii. If W / 4 ≤ x < W / 2, then W0 = 5, W1 = 3;
[0326] iii. If W / 2 ≤ x < 3*W / 4, then W0 = 3, W1 = 5;
[0327] iv. If x ≥ 3*W / 4, then W0 = 2, W1 = 6;
[0328] c. If mode == VER:
[0329] i. If y < H / 4, then W0 = 6, W1 = 2;
[0330] ii. If H / 4 ≤ y < H / 2, then W0 = 5, W1 = 3;
[0331] iii. If H / 2 ≤ y < 3*H / 4, then W0 = 3, W1 = 5;
[0332] iv. If y ≥ 3*H / 4, then W0 = 2, W1 = 6.
[0333] 2.7 Examples of the ultimate motion vector representation (UMVE)
[0334] In some embodiments, a final motion vector expression (UMVE) is proposed. The UMVE can be used in skip or merge mode by the proposed motion vector expression method.
[0335] UMVE reuses merge candidates as those included in the regular merge candidate list in VVC. Among the merge candidates, a basic candidate can be selected and further extended by the proposed motion vector expression method.
[0336] UMVE provides a new method to represent motion vector difference (MVD), in which the starting point, motion amplitude and motion direction are used to represent the MVD.
[0337] The proposed technique uses the merge candidate list as is. However, only candidates of the default merge type (MRG_TYPE_DEFAULT_N) are considered for UMVE extension.
[0338] The base candidate index defines the starting point. The base candidate index indicates the best candidate among the candidates in the list as follows.
[0339] Table 1: Basic candidate IDX
[0340] Basic Candidate IDX 0 1 2 3 The Nth MVP 1st MVP Second MVP 3rd MVP 4th MVP
[0341] If the number of basic candidates is equal to 1, the basic candidate IDX is not signaled.
[0342] The distance index is the motion amplitude information. The distance index indicates the predefined distance from the start point information. The predefined distances are as follows:
[0343] Table 2: Distance IDX
[0344]
[0345] The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent four directions, as shown below.
[0346] Table 3: Direction IDX
[0347] Directions to IDX 00 01 10 11 x-axis + – N / A N / A y-axis N / A N / A + –
[0348] The UMVE flag is signaled immediately after the skip flag or merge flag is sent. If the skip or merge flag is true, the UMVE flag is parsed. If the UMVE flag is equal to 1, the UMVE syntax is parsed. However, if it is not 1, the affine flag is parsed. If the affine flag is equal to 1, it is affine mode; however, if it is not equal to 1, the skip / merge index will be parsed for the skip / merge mode.
[0349] Due to UMVE candidates, no additional line buffer is needed. Because the software's skip / merge candidates are directly used as base candidates. Using the input UMVE index, the complement of the MV can be determined before motion compensation. No need to keep a long line buffer for this.
[0350] Under the current general test conditions, the first or second merge candidate in the merge candidate list can be selected as the base candidate.
[0351] UMVE is called merge of MVD (MMVD).
[0352] 3. Disadvantages of existing implementations
[0353] In existing implementations (e.g., current VVC designs), CPR has the following problems:
[0354] (1) The interaction between CPR and affine is still unclear. (For example, for a CPR-encoded block, a merge candidate can be inserted into the sub-block merge list or the regular merge list.)
[0355] (2) The interaction between CPR and general bidirectional prediction is still unclear (e.g., for CPR coded blocks, fixed half-half weight values are always used, so no signaling of weight indices is required).
[0356] (3) The interaction between CPR and MMVD (also known as UMVE) remains unclear.
[0357] (4) The interaction between CPR and inter-intra prediction remains unclear.
[0358] (5) The interaction between ATMVP and CPR remains unclear.
[0359] (6) It is not clear how to apply CPR with dual (separate) codec trees.
[0360] 4. Example Method for Motion Vector Derivation and Signaling of CPR
[0361] Embodiments of the currently disclosed technology overcome the shortcomings of existing implementations, thereby providing video codecs with higher codec efficiency. It is illustrated in the following examples described for various implementations that, based on the disclosed technology, the use of motion vector derivation and signaling for current picture reference in video and image codecs can enhance existing and future video codec standards. The examples of the disclosed technology provided below explain the general concepts and are not intended to be interpreted as limiting. In the examples, the various features described in these examples can be combined unless explicitly stated otherwise. It should be understood that "a block or candidate using CPR" can mean:
[0362] (i) the reference picture used by the block or candidate is a CPR; or
[0363] (ii) At least one reference picture used by a block or candidate is a CPR.
[0364] Example 1. When using affine motion compensation, CPR is not allowed.
[0365] (a) In one example, a conforming bitstream cannot use both CPR and affine motion compensation in a single block.
[0366] (b) In one example, during the parsing process, the block-level affine enable / disable flag may be first signaled. When encoding and decoding a block using affine mode, there is no need to signal whether the reference picture index of the current block is equal to the reference picture index associated with the current picture.
[0367] (c) Alternatively, the reference picture index can be signaled before the block level affine enable / disable flag. When a block uses CPR, the signaling of the affine flag is always skipped.
[0368] (d) Alternatively, the reference picture index may be signaled before the block level affine enable / disable flag. When a block is encoded with bi-directional prediction and only one reference picture is the current reference picture, the affine flag may still be signaled.
[0369] (i) In one example, additionally, integer motion vectors are required to be used for such blocks.
[0370] Example 2. Blocks cannot inherit affine models from neighboring blocks using CPR.
[0371] (a) In one example, neighboring blocks using CPR cannot be used to derive affine merge candidates in the sub-block merge candidate list.
[0372] (i) In one example, neighboring blocks using CPR are considered unavailable during the sub-block merge candidate list construction process.
[0373] (b) In one example, neighboring blocks using CPR cannot be used to derive affine AMVP candidates in the affine AMVP list.
[0374] (i) In one example, neighboring blocks using CPR are treated as unavailable during the affine AMVP candidate list construction process.
[0375] Example 3. Whether a block can inherit an affine model from a neighboring block using CPR may depend on a reference list.
[0376] (a) In one example, assuming that the neighboring block applies bidirectional prediction, the reference picture in reference list X (X can be 0 or 1) is the current picture, and the reference picture in reference list (1-X) is not the current picture, then
[0377] (i) The motion information of the neighboring blocks referring to reference list X cannot be used to derive the affine merge candidates in the sub-block merge candidate list.
[0378] (1) In one example, the motion information of the neighboring blocks of the reference list (1-X) can be used to derive the affine merge candidates in the sub-block merge candidate list.
[0379] (ii) The motion information of the neighboring blocks referring to the reference list X cannot be used to derive the affine AMVP candidates in the affine AMVP list.
[0380] (1) For example, the motion information of the neighboring blocks referring to reference list X cannot be used to derive the affine AMVP candidates in the affine AMVP list for reference list X.
[0381] (iii) The motion information of the neighboring blocks referring to the reference list (1-X) may be used to derive the affine AMVP candidates in the affine AMVP list.
[0382] (1) For example, motion information of neighboring blocks referring to reference list (1-X) may be used to derive affine AMVP candidates in the affine AMVP list for reference list (1-X).
[0383] Example 4. When GBi is applied in a block (GBi weights on two reference pictures are not equal), CPR is not allowed.
[0384] (a) In one example, a conforming bitstream cannot use both CPR and GBi in a single block.
[0385] (b) In one example, if CPR is used in a block, the GBi weight index is not signaled and is inferred to be 0 (equal weighting values on both reference pictures).
[0386] (c) In one example, if unequal weighting in GBi is applied in a block (GBi weighting values on two reference pictures are not equal), the CPR indication is not signaled and is inferred to be 0 (no CPR used).
[0387] Example 5. When MMVD is applied in blocks, CPR is not allowed.
[0388] (a) In one example, a conforming bitstream cannot use both CPR and MMVD in a single block.
[0389] (b) In one example, if CPR is used in a block, the MMVD flag is not signaled and is inferred to be 0 (equal weighting on both reference pictures).
[0390] (c) In one example, if MMVD is applied in a block, the CPR indication is not signaled and is inferred to be 0 (CPR not used).
[0391] Example 6. When MMVD is applied in blocks, CPR is allowed.
[0392] (a) In one example, if the base MV candidate has at least one reference picture that refers to the current picture:
[0393] (i) In one example, the MMVD is based on the MVs that refer to the current picture, omitting the MVs that do not refer to the current picture (if any);
[0394] (ii) In one example, the distance signaled by MMVD can only be an integer.
[0395] (iii) In one example, MMVD is unidirectionally predicted.
[0396] (b) In one example, if the base MV candidate has at least one reference picture that does not reference the current picture:
[0397] (i) In one example, the MMVD is based on MVs that do not reference the current picture, omitting MVs that reference the current picture (if any).
[0398] Example 7. When inter-intra prediction is applied in a block, CPR is not allowed.
[0399] (a) In one example, a conforming bitstream cannot use both CPR and inter-intra prediction in a single block.
[0400] (b) In one example, if CPR is used in a block, the inter-intra prediction flag is not signaled and is inferred to be 0.
[0401] (c) In one example, if inter-intra prediction is applied in a block, the CPR indication is not signaled and is inferred to be 0 (CPR not used).
[0402] Example 8. When inter-intra prediction is applied in a block, CPR is allowed to be used.
[0403] (a) In one example, if a merge candidate for inter-intra prediction has at least one reference picture that refers to the current picture:
[0404] (i) Inter-intra prediction is based on the MV that refers to the current picture, omitting the MV that does not refer to the current picture (if any);
[0405] (ii) In one example, the intra prediction part of the inter-intra prediction may be changed to:
[0406] (1) No boundary filtering is performed;
[0407] (2) Position-determined prediction combination (PDPC);
[0408] (3) Only the following specific modes are allowed:
[0409] ○MPM
[0410] ○DC
[0411] ○Vertical
[0412] ○ Level
[0413] (iii) In one example, the inter prediction part of the inter-intra prediction is unidirectionally predicted.
[0414] (b) In one example, if a merge candidate for inter-intra prediction has at least one reference picture that does not reference the current picture:
[0415] (i) In one example, inter-intra prediction is based on MVs that do not reference the current picture, omitting MVs that reference the current picture (if any).
[0416] Example 9. During the ATMVP process:
[0417] (a) If the neighboring block uses CPR, it is considered unavailable;
[0418] (b) If the blocks in the collocated image use CPR
[0419] (i) It may be considered unavailable during the ATMVP process;
[0420] (ii) It can be considered as available and the MV' of the current block or sub-block can be derived from this block.
[0421] (1) In one example, MV' refers to the current picture and is copied from blocks in the collocated picture.
[0422] Example 10. When CPR is coded as a new mode other than intra and inter modes (instead of being indicated as a specific reference picture in the reference picture list), whether to skip signaling of the CPR mode may depend on decoding information of other coding methods.
[0423] (a) In one example, other coding methods may include, for example, an affine flag, an inter-intra prediction flag, a GBI index indicating unequal weighting, MMVD, ATMVP.
[0424] (b) When any of the other coding methods is applied to a block, signaling of the CPR mode for the block is skipped.
[0425] (c) Alternatively, whether to skip signaling using other coding methods may depend on the decoding information of the CPR mode.
[0426] Example 11 When the luma and chroma components are encoded and decoded in separate codec structure trees:
[0427] (a) In one example, when the chrominance component is encoded and decoded, the MV information of the corresponding block of the luma component (the MV may refer to the current picture) may be used as an MV prediction or merge candidate.
[0428] (i) The corresponding block of the luma component can be at any position in the current chroma block (such as top left, top right, bottom left, bottom right, or in the middle).
[0429] (1) If the color format is not 4:4:4, the current chrominance block should be scaled to find the corresponding luma block.
[0430] (ii) In one example, there may be MV information of several corresponding blocks from the luma component, which are pruned as MV prediction or merge candidates for the current chroma block. One of them can be selected to be signaled to the decoder or derived at the decoder.
[0431] (b) The width / height of the chroma blocks using CPR may have limitations.
[0432] (i) In one example, the width / height of chroma blocks using CPR may be constrained differently than chroma blocks for intra-frame coding.
[0433] (ii) In one example, for intra-coded blocks, width>2&&height>2, but for CPR-coded blocks, width≥2&&height≥2.
[0434] The above examples may be incorporated in the context of the methods described below, such as methods 2400 , 2500 , 2600 , and 2700 , which may be implemented at a video decoder or a video encoder.
[0435] Fig.24 A flowchart of an exemplary method for video processing is shown. The method 2400 includes, at step 2410, performing a conversion of a current video block and a bitstream representation of the current video block according to a first codec rule, the first codec rule specifying the application of a current picture reference (CPR) during encoding or decoding of the current video block, wherein the first codec rule specifies at least one of the following: (a) avoiding the simultaneous use of CPR and affine motion compensation during the conversion process; or (b) disabling, for the conversion, inheriting an affine model of a neighboring block that is encoded or decoded using CPR; or (c) only when the codec condition is met. In the above cases, for conversion, the affine model of the neighboring blocks is inherited; or (d) avoid using CPR and generalized bidirectional prediction (GBi) for conversion at the same time, where GBi uses unequally weighted reference pictures; or (e) avoid using CPR and merge with motion vector difference (MMVD) mode at the same time; or (f) use CPR and MMVD mode at the same time; or (g) avoid using CPR and inter-intra prediction at the same time; or (h) use CPR and inter-intra prediction at the same time; or (i) use the optional temporal motion vector predictor at the same time.
[0436] In some embodiments, and in the context of Example 1, integer precision is used for encoding and decoding rule (a).
[0437] In some embodiments, and in the context of Example 6, the MMVD mode includes unidirectional prediction for codec rule (e) or (f).
[0438] In some embodiments, and in the context of Example 8, the inter-intra prediction includes an intra prediction portion and an inter prediction portion for codec rules (g) and (h). In an example, the intra prediction portion does not include boundary filtering. In another example, the intra prediction portion does not include a position-determined prediction combination. In yet another example, the intra prediction portion includes the most likely mode selection, DC prediction mode, horizontal prediction mode, or vertical prediction mode. In yet another example, the inter prediction portion includes a unidirectional prediction.
[0439] Fig.25 A flowchart of another exemplary method for video processing is shown. The method 2500 includes, at step 2510, determining to use a current picture reference mode to encode and decode a neighboring block of a current video block.
[0440] The method 2500 includes, at step 2520, deriving a merge candidate based on the determination. In some embodiments, and in the context of Example 2, the merge candidate includes a first affine model that is different from a second affine model of the adjacent block. In an example, the merge candidate includes an affine merge candidate, and wherein the merge candidate list includes a sub-block merge candidate list. In another example, the merge candidate includes an affine AMVP candidate, and wherein the merge candidate list includes an affine AMVP candidate list.
[0441] In some embodiments, and in the context of Example 9, the merge candidate may include an optional temporal motion vector prediction (ATMVP) candidate.
[0442] Method 2500 includes, at step 2530, inserting the merge candidate list into the merge candidate list.
[0443] The method 2500 includes, at step 2540, reconstructing the current video block based on the merge candidate list.
[0444] In some embodiments, and in the context of Example 3, the neighboring blocks apply bidirectional prediction, and the method 2500 further includes the following steps: determining an affine model of a merge candidate based on a first reference picture list or a second reference picture list. In an example, the first reference picture list includes a current picture of a current video block, and the affine model is determined based on at least one picture in the second reference picture list. In another example, the merge candidate includes an AMVP candidate, and the merge candidate list includes an affine AMVP candidate list.
[0445] Fig.26 A flowchart of another exemplary method of video processing is shown. The method 2600 includes, at step 2610, generating decoded information by decoding a first codec mode of a current video block, wherein the current video block is encoded in the first codec mode and a second codec mode different from the first codec mode.
[0446] The method 2600 comprises, at step 2620, determining, based on the decoded information, whether to signal the first codec mode or the second codec mode in a bitstream representation of the current video block.
[0447] The method 2600 includes, at step 2630, reconstructing the current video block based on the determination.
[0448] In some embodiments, and in the context of Example 10, the first codec mode includes: an affine codec mode, an inter-intra prediction mode, a generalized bidirectional prediction mode with unequal weights, a merge mode with motion vector difference, or an optional time domain motion vector prediction mode, the second codec mode includes a CPR mode, and the bitstream representation does not include a flag for the CPR mode.
[0449] In some embodiments, and in the context of Example 10, the first codec mode includes: a current picture reference (CPR) mode, the second codec mode includes: an affine codec mode, an inter-intra prediction mode, a generalized bidirectional prediction mode with unequal weights, a merge mode with motion vector difference, or an optional temporal motion vector prediction mode, and the bitstream representation does not include a flag for the second codec mode.
[0450] Fig. 27 A flowchart of another exemplary method of video processing is shown. The method 2700 includes, at step 2710, determining that a current video block includes a luma component and a chroma component, the luma component being encoded and decoded using a first codec structure tree and the chroma component being encoded and decoded using a second codec structure tree different from the first codec structure tree.
[0451] The method 2700 includes, at step 2720, reconstructing the current video block based on the determination. In some embodiments, and in the context of Example 11, the reconstructing includes decoding the chrominance components based on motion vector information derived from one or more blocks of the luma component. In an example, the one or more blocks of the luma component are the upper left position, the upper right position, the lower left position, the lower right position, or the middle position of the current video block. In another example, the method 2700 also includes the following steps: when it is determined that the color format of the chrominance components is different from the 4:4:4 color format, scaling the current video block to determine one or more corresponding blocks of the luma component.
[0452] In some embodiments, and in the context of Example 11, the current video block is coded using a current picture reference (CPR) mode. In an example, a first set of constraints on a height and a width of the current video block is different from a second set of constraints on a height and a width of an intra-coded video block that is different from the current video block.
[0453] 5. Example Implementation of Disclosed Technology
[0454] Fig.282800. The apparatus 2800 may be used to implement one or more methods described herein. The apparatus 2800 may be implemented in a smartphone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 2800 may include one or more processors 2802, one or more memories 2804, and video processing hardware 2806. The processor 2802 may be configured to implement one or more methods described herein (including but not limited to methods 2400, 2500, and 2600). The memory (multiple memories) 2804 may be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 2806 may be used to implement some of the techniques described herein in hardware circuits.
[0455] Fig.29 A flowchart of another exemplary method of video processing is shown. Method 2900 includes, at step 2910, determining that an intra-block copy (IBC) mode applies to a current video block of a video, wherein in the IBC mode, at least one reference picture used by the current video block is a current picture in which the current video block is located; at step 2920, making a decision to disable a specific codec mode for the current block; and at step 2930, based on the decision, converting between the current video block and the bitstream representation, wherein the specific codec mode uses motion vectors and non-current pictures to derive a prediction of the video block.
[0456] Fig.30 A flowchart of another exemplary method of video processing is shown. The method 3000 includes, at step 3010, determining that a current video block is encoded using a specific codec mode; at step 3020, based on the determination, making a decision about disabling an intra-block copy (IBC) mode of the current block, wherein in the IBC mode, at least one reference picture used by the current video block is a current picture in which the current block is located; and at step 3030, based on the decision, performing a conversion between the current block and the bitstream representation, wherein the specific codec mode uses a motion vector and a non-current picture to derive a prediction of the current block.
[0457] Fig.31 A flowchart of another exemplary method of video processing is shown. The method 3100 includes, at step 3110, determining an affine mode of a current video block, wherein whether the current video block inherits the affine mode from a neighboring block depends on a reference list; at step 3120, based on the determination, converting between the current video block of the video and a bitstream representation of the video.
[0458] Fig.32A flowchart of another exemplary method of video processing is shown. The method 3200 includes, at step 3210, performing a conversion between a current video block of a video and a bitstream representation of the video, wherein during the conversion, an intra-block copy (IBC) mode and a merge with motion vector difference (MMVD) mode are used, and wherein in the IBC mode, at least one reference picture used by the current video block is the current picture in which the current video block is located, and in the MMVD mode, a motion vector of the video block is derived based on a merge motion candidate list and further refined by at least one motion vector difference.
[0459] Fig.33 A flowchart of another exemplary method of video processing is shown. The method 3300 includes, at step 3310, performing a conversion between a current video block of a video and a bitstream representation of the video, wherein during the conversion, an intra block copy (IBC) mode and an inter-intra prediction mode are used, and wherein in the IBC mode, at least one reference picture used by the current video block is the current picture in which the current video block is located, and in the inter-intra prediction mode, a prediction signal for the current video block is generated based on at least an intra prediction signal and an inter prediction signal.
[0460] Fig.34 A flowchart of another exemplary method of video processing is shown. The method 3400 includes, at step 3410, during a conversion between a current video block and a bitstream representation of the current video block, determining decoded information of at least one coding method different from an IBC mode, wherein in the IBC mode, at least one reference picture used by the current video block is the current picture in which the current video block is located; at step 3420, based on the decoded information, determining whether to skip signaling of an IBC flag of the current video block in the bitstream representation of the current video block, wherein the IBC flag is related to the IBC mode; and at step 3430, performing the conversion based on the determination.
[0461] Fig.35 A flowchart of another exemplary method of video processing is shown. The method 3500 includes, at step 3510, during conversion between a current video block and a bitstream representation of the current video block, wherein the current video block includes a luma component encoded and decoded using a first codec structure tree and a chroma component encoded and decoded using a second codec structure tree different from the first codec structure tree, deriving first motion vector information of a chroma block from second motion vector information of one or more corresponding blocks of the luma component of the current video block; and at step 3520, performing the conversion based on the first motion vector information and the second vector information.
[0462] Fig.36 A flowchart of another exemplary method of video processing is shown. The method 3600 includes, at step 3610, determining whether an intra-block copy (IBC) mode is applied to a current video block, wherein at least one reference picture used by the current video block is the same as a picture in which the current video block is located; at step 3620, signaling whether the IBC mode is applied to the current video block; at step 3630, based on determining that the IBC mode is applied to the current video block, avoiding signaling a flag of a group of prediction modes; and at step 3640, performing a conversion of the current video block based on the application of the IBC mode.
[0463] Fig.37 A flowchart of another exemplary method of video processing is shown. The method 3700 includes, at step 3710, parsing a bitstream to determine whether an intra-block copy (IBC) mode is applied to the current video block, in which at least one reference picture used by the current video block is the same as the picture in which the current video block is located; at step 3720, based on the application of the IBC mode, performing a conversion of the current video block, wherein a flag of a group of prediction modes is excluded from the bitstream based on determining that the CPR is applied to the current video block.
[0464] Fig.38 A flowchart of another exemplary method of video processing is shown. The method 3800 includes, at step 3810, in the process of obtaining motion information of a current video block, during conversion between the current video block and a bitstream representation of the current video block, wherein the motion information of the current video block is based on at least one affine model of at least one neighboring block of the current video block; at step 3820, determining whether a neighboring block of the current video block uses an intra-frame block copy (IBC) mode, in which at least one reference picture used by the neighboring block is the same as the picture in which the neighboring block is located; at step 3830, based on determining that the neighboring block uses the IBC mode, prohibiting the derivation of affine candidates from the neighboring block for the current video block; and at step 3830, performing the conversion based on the motion information of the current video block.
[0465] Fig.39A flowchart of another exemplary method of video processing is shown. The method 3900 includes, at step 3910, during conversion between a current video block and a bitstream representation of the current video block, dividing the current video block into sub-blocks based on a sub-block-based temporal motion vector prediction (SbTMVP) mode; at step 3920, identifying a corresponding block for the sub-block in a motion source picture with a temporal vector based on motion information of spatial neighboring blocks of the current video block; at step 3930, generating a motion vector and a reference index of the sub-block from the corresponding block of the sub-block based on mode information of the encoding and decoding of the corresponding block; at step 3940, performing the conversion based on the generated motion vector and reference index.
[0466] Fig.40 A flowchart of another exemplary method of video processing is shown. The method 4000 includes, at step 4010, during conversion between a current video block and a bitstream representation of the current video block, dividing the current video block into sub-blocks based on a sub-block-based temporal motion vector prediction (SbTMVP) mode; at step 4020, identifying a corresponding block for the sub-block in a motion source picture with a temporal vector based on mode information of encoding and decoding of spatial neighboring blocks of the current video block; at step 4030, generating a motion vector and a reference index for the sub-block from the corresponding block of the sub-block; and at step 4040, performing the conversion based on the generated motion vector and reference index.
[0467] In some embodiments, the video encoding and decoding method can be used in Fig.28 The device is implemented on the described hardware platform.
[0468] Some embodiments may be described using the following examples.
[0469] 1.1. A method for video processing, comprising: determining that an intra block copy (IBC) mode applies to a current video block of a video, wherein in the IBC mode, at least one reference picture used by the current video block is a current picture in which the current video block is located; making a decision about disabling a specific codec mode for the current block; based on the decision, performing a conversion between the current video block and the bitstream representation; wherein the specific codec mode uses motion vectors and non-current pictures to derive a prediction of the video block.
[0470] 1.2. The method of example 1.1, wherein the converting comprises encoding the current video block into the bitstream representation.
[0471] 1.3. The method of Example 1.1, wherein the converting comprises decoding a bitstream representation of the current video block to generate pixel values of the current video block.
[0472] 1.4. The method of any of Examples 1.1-1.3, wherein the specific codec mode comprises an affine motion compensation mode, wherein the affine motion compensation mode uses at least one control point motion vector.
[0473] 1.5. The method of Example 1.4, wherein a reference picture index is signaled and signaling of an affine flag is skipped in response to the current video block using the IBC mode.
[0474] 1.6. As described in Example 1.4, the method also includes: before signaling the affine flag, signaling the reference picture index and determining whether the current video block uses a bidirectional prediction mode, wherein if the current video block uses a bidirectional prediction mode and only one reference picture is the current reference picture, the affine flag is signaled; and if the current video block uses a bidirectional prediction mode and both reference pictures are the current reference picture, the affine flag is not signaled.
[0475] 1.7. The method of Example 1.6, wherein integer precision is used for the motion vector of the current video block.
[0476] 1.8. The method of any of Examples 1.1-1.7, wherein the specific codec mode comprises a GBi mode, wherein in the GBi mode, different weighting values are associated with different reference pictures.
[0477] 1.9. The method of Example 1.8, wherein the GBi weight index is not signaled in response to the IBC mode used in the current block.
[0478] 1.10 The method of Example 1.8, wherein the GBi weight index is inferred to be 0.
[0479] 1.11. A method as described in any of Examples 1.1-1.10, wherein the specific coding mode includes a merge with motion vector difference (MMVD) mode, wherein in the MMVD mode, the motion vector of the video block is derived based on a merge motion candidate list and further refined by at least one motion vector difference.
[0480] 1.12. The method of Example 1.11, wherein the MMVD flag is not signaled in response to using IBC mode in the current video block.
[0481] 1.13. The method of Example 1.12, wherein the MMVD flag is inferred to be 0.
[0482] 1.14. In the method described in any one of Examples 1.1-1.13, the specific coding mode includes a combined inter-intra prediction (CIIP) mode, wherein in the CIIP mode, a prediction signal for the current video block is generated based at least on an intra prediction signal and an inter prediction signal.
[0483] 1.15. The method of Example 1.14, wherein in response to using the IBC mode in the current block, a CIIP flag of the current video block is not signaled.
[0484] 1.16. The method of Example 1.15, wherein the CIIP flag is inferred to be 0.
[0485] 1.17. A method for video processing, comprising: determining that a current video block is encoded using a specific codec mode; based on the determination, making a decision to disable an intra-block copy (IBC) mode for the current block, wherein in the IBC mode, at least one reference picture used by the current video block is a current picture where the current block is located; based on the decision, performing a conversion between the current block and the bitstream representation; wherein the specific codec mode uses a motion vector and a non-current picture to derive a prediction for the current block.
[0486] 1.18. The method of Example 1.17, wherein the converting comprises encoding the current video block into the bitstream representation.
[0487] 1.19. The method of Example 1.17, wherein the converting comprises decoding a bitstream representation of the current video block to generate pixel values of the current video block.
[0488] 1.20. The method of any of Examples 1.1-1.3, wherein the particular codec mode comprises an affine motion compensation mode, wherein the affine motion compensation uses at least one control point motion vector.
[0489] 1.21. A method as described in Example 1.20, wherein the block-level affine flag is signaled in the bitstream to indicate whether the affine motion compensation mode is enabled for the current block, and if the affine flag indicates that the affine motion compensation mode is enabled for the current video block, whether the IBC mode is applied is not signaled.
[0490] 1.22. The method of any of Examples 1.17-1.19, wherein the specific codec mode comprises a GBi mode, wherein in the GBi mode, different weighting values are associated with different reference pictures.
[0491] 1.23. The method of Example 1.22, wherein in response to applying the IBC mode, no IBC indication is signaled.
[0492] 1.24. The method of example 1.23, wherein the IBC indication is inferred to be zero.
[0493] 1.25. A method as described in any one of Examples 1.17-1.24, wherein the specific coding mode includes a merge with motion vector difference (MMVD) mode, wherein in the MMVD mode the motion vector of the current block is derived based on a merge motion candidate list and is further refined by at least one motion vector difference.
[0494] 1.26. The method of Example 1.25, wherein in response to applying the MMVD mode in the current video block, no IBC indication is signaled.
[0495] 1.27. The method of Example 1.26, wherein the IBC indication is inferred to be zero.
[0496] 1.28. In the method described in any one of Examples 1.17-1.27, the specific coding mode includes a combined inter-intra prediction (CIIP) mode, wherein in the CIIP mode, a prediction signal for the current video block is generated based at least on an intra prediction signal and an inter prediction signal.
[0497] 1.29. The method of Example 1.28, wherein in response to applying the inter-intra prediction mode, an IBC flag of the current video block is not signaled.
[0498] 1.30. The method of Example 1.29, wherein the IBC flag is inferred to be 0.
[0499] 1.31. A method for video processing, comprising: determining an affine mode of a current video block, wherein whether the current video block inherits the affine mode from a neighboring block depends on a reference list; based on the determination, performing a conversion between the current video block of a video and a bitstream representation of the video.
[0500] 1.32 A method as described in Example 1.31, wherein the neighboring blocks apply bidirectional prediction, the reference picture in reference list X is the current picture, and the reference picture in reference list (1-X) is not the current picture, where X is 0 or 1.
[0501] 1.33. The method of Example 1.32, wherein the motion information of the neighboring blocks referring to the reference list X is not used to derive the affine merge candidates in the sub-block merge candidate list.
[0502] 1.34. A method as described in Example 1.32 or 1.33, wherein the motion information of the neighboring blocks referenced in the reference list (1-X) is used to derive the affine merge candidates in the sub-block merge candidate list.
[0503] 1.35. The method of Example 1.32, wherein the motion information of the neighboring blocks referring to the reference list X is not used to derive the affine AMVP candidates in the affine AMVP list.
[0504] 1.36. The method of Example 1.35, wherein the motion information of the neighboring blocks referring to the reference list X is not used to derive the affine AMVP candidate in the affine AMVP list of the reference list X.
[0505] 1.37. The method of Example 1.32, wherein motion information of the neighboring blocks referring to the reference list (1-X) is used to derive affine AMVP candidates in an affine AMVP list.
[0506] 1.38. The method of Example 1.37, wherein the motion information of the neighboring blocks referring to the reference list (1-X) is used to derive the affine AMVP candidate in the affine AMVP list of the reference list (1-X).
[0507] 1.39. A method for video processing, comprising: performing a conversion between a current video block of a video and a bitstream representation of the video; wherein during the conversion, an intra-block copy (IBC) mode and a merge with motion vector difference (MMVD) mode are used, wherein in the IBC mode, at least one reference picture used by the current video block is the current picture where the current video block is located, and in the MMVD mode, a motion vector of the video block is derived based on a merge motion candidate list and further refined by at least one motion vector difference.
[0508] 1.40. The method of Example 1.39, wherein the MMVD mode uses a base MV candidate having at least one reference picture that references a current picture.
[0509] 1.41. The method of Example 1.40, wherein the MMVD is based on at least one MV that references the current picture, and MVs that do not reference the current picture are omitted if any.
[0510] 1.42. The method of Example 1.41 wherein the distance signaled by the MMVD is an integer.
[0511] 1.43. The method of Example 1.41, wherein the MMVD is unidirectionally predicted.
[0512] 1.44. A method as described in Example 1.39, wherein if the MMVD mode uses a basic MV candidate with at least one reference picture that does not reference the current picture, the MMVD is based on at least one MV that does not reference the current picture, and if there is an MV that references the current picture, it is omitted.
[0513] 1.45. A method for video processing, comprising: performing a conversion between a current video block of a video and a bitstream representation of the video; wherein during the conversion, an intra block copy (IBC) mode and an inter-intra prediction mode are used, wherein in the IBC mode, at least one reference picture used by the current video block is the current picture where the current video block is located, and in the inter-intra prediction mode, a prediction signal for the current video block is generated based on at least an intra prediction signal and an inter prediction signal.
[0514] 1.46. The method of Example 1.45, wherein a merge candidate to be inter-intra predicted has at least one reference picture that refers to the current picture.
[0515] 1.47. The method of Example 1.46, wherein the inter-intra prediction is based on at least one MV that references the current picture, and MVs that do not reference the current picture are omitted if any.
[0516] 1.48. The method of Example 1.45, wherein the intra prediction portion of the inter-intra prediction mode does not include boundary filtering.
[0517] 1.49. The method of Example 1.45, wherein the intra prediction portion of the inter-intra prediction mode does not include position determined prediction combining (PDPC).
[0518] 1.50. The method of Example 1.45, wherein the intra prediction portion of the inter-intra prediction mode comprises at least one of: a most probable mode selection (MPM), a DC prediction mode, a horizontal prediction mode, or a vertical prediction mode.
[0519] 1.51. The method of Example 1.45, wherein the inter prediction portion of the inter-intra prediction mode is uni-directional prediction.
[0520] 1.52. A method as described in any one of Examples 1.45-1.51, wherein if a merge candidate to be inter-intra predicted has at least one reference picture that does not refer to the current picture, inter-intra prediction is performed based on at least one MV that does not refer to the current picture, and if the MV that refers to the current picture exists, it is omitted.
[0521] 1.53. A method for video processing, comprising: during a conversion between a current video block and a bitstream representation of the current video block, determining decoded information of at least one encoding and decoding method different from an IBC mode, wherein in the IBC mode, at least one reference picture used by the current video block is the current picture in which the current video block is located; based on the decoded information, determining whether to skip signaling notification of an IBC flag of the current video block in the bitstream representation of the current video block, wherein the IBC flag is related to the IBC mode; and performing the conversion based on the determination.
[0522] 1.54. The method of Example 1.53, wherein the converting comprises encoding the current video block into the bitstream representation.
[0523] 1.55. The method of Example 1.53, wherein the converting comprises decoding a bitstream representation of the current video block into the current video block to generate pixel values for the current video block.
[0524] 1.56. The method as described in any one of Examples 1.53-1.55 further includes: if it is determined in the determining step not to skip the signaling notification of the IBC flag, then signaling the IBC flag of the current video block in the bitstream representation; otherwise, avoiding signaling the IBC flag of the current video block in the bitstream representation.
[0525] 1.57. A method as described in any one of Examples 1.53-1.56, wherein when the decoding information of the coding and decoding method includes at least one of the following: an affine flag, an inter-frame-intra prediction flag, a generalized bidirectional prediction (GBi) index, merge with motion vector difference (MMVD) information, and optional time domain motion vector prediction (ATMVP) information, it is determined to skip the signaling notification of the IBC flag.
[0526] 1.58. A method for video processing, comprising: during conversion between a current video block and a bitstream representation of the current video block, wherein the current video block includes a luminance component encoded and decoded using a first codec structure tree, and a chrominance component encoded and decoded using a second codec structure tree different from the first codec structure tree, deriving first motion vector information of a chrominance block from second motion vector information of one or more corresponding blocks of the luminance component of the current video block; and performing the conversion based on the first motion vector information and the second vector information.
[0527] 1.59. The method of Example 1.58, wherein the converting comprises encoding the current video block into the bitstream representation.
[0528] 1.60. The method of Example 1.58, wherein the converting comprises decoding a bitstream representation of the current video block into the current video block to generate pixel values for the current video block.
[0529] 1.61. The method of Example 1.58, wherein the first motion vector information is used as an MV prediction or merge candidate to derive the second motion vector information.
[0530] 1.62. A method as described in any of Examples 1.58-1.61, wherein the one or more corresponding blocks of the luma component are located at the upper left position, upper right position, lower left position, lower right position and / or center position of the current video block.
[0531] 1.63. The method described in Example 1.62 also includes: determining whether the color format of the chrominance component is different from the color format of 4:4:4; and when it is determined that the color format of the chrominance component is different from the color format of 4:4:4, scaling the current video block to determine one or more corresponding blocks of the luminance component.
[0532] 1.64. The method of any of Examples 1.58-1.60, wherein the current video block is encoded using an intra block copy (IBC) mode.
[0533] 1.65. The method of Example 1.64, wherein the first set of constraints on the height and width of the current video block is different than a second set of constraints on the height and width of an intra-coded video block other than the current video block.
[0534] 1.66. A method as described in Example 1.65, wherein when the current video block is encoded and decoded using the intra block copy mode, the width of the current video block is greater than 2 and the height is greater than 2, and when the current video block is encoded and decoded using the IBC mode, the width of the current video block is greater than or equal to 2 and the height is greater than or equal to 2.
[0535] 1.67. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein when the instructions are executed by the processor, the processor is caused to implement the method as described in any one of Examples 1.1 to 1.66.
[0536] 1.68. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for executing the method of any one of Examples 1.1 to 1.66.
[0537] 2.1. A method for video processing, comprising: determining whether an intra-frame block copy (IBC) mode is applied to a current video block, in which at least one reference picture used by the current video block is the same as a picture in which the current video block is located; signaling whether the IBC mode is applied to the current video block; based on determining that the IBC mode is applied to the current video block, avoiding signaling a flag of a group of prediction modes; and performing a conversion of the current video block based on the application of the IBC mode.
[0538] 2.2. The method of Example 2.1, wherein the converting comprises encoding the current video block into the bitstream representation.
[0539] 2.3. The method of example 2.1 or 2.2, wherein the set of prediction modes includes an affine mode, wherein the affine mode uses at least one control point motion vector.
[0540] 2.4. A method as described in Example 2.3, wherein determining whether to apply the IBC mode to the current video block is performed before signaling an affine flag; and the method further includes: if it is determined that the IBC mode is applied to the current video block, avoiding signaling the affine flag; and otherwise, signaling the affine flag.
[0541] 2.5. The method of example 2.4, wherein the affine flag is inferred to be 0 when the affine flag is not signaled.
[0542] 2.6. The method of any of Examples 2.1-2.5, wherein the set of prediction modes comprises a generalized bi-prediction (GBi) mode, wherein in the GBi mode, different weighting values are associated with different reference pictures.
[0543] 2.7. A method as described in Example 2.6, wherein determining whether to apply the IBC mode to the current video block occurs before signaling the GBi index; and the method further includes: if it is determined that the IBC mode is applied to the current video block, avoiding signaling the GBi weighted index, and otherwise, signaling the GBi weighted index.
[0544] 2.8. The method of Example 2.7, wherein when the GBi weight index is not signaled, the weight value of the reference picture used by the GBi is inferred to be zero.
[0545] 2.9. A method for video processing, comprising: parsing a bitstream to determine whether an intra-block copy (IBC) mode is applied to the current video block, in which at least one reference picture used by the current video block is the same as the picture in which the current video block is located; based on the application of the IBC mode, performing a conversion of the current video block, wherein a flag of a group of prediction modes is excluded from the bitstream based on determining that the CPR is applied to the current video block.
[0546] 2.10. The method of Example 2.9, wherein the converting comprises decoding a bitstream representation of the current video block into the current video block to generate pixel values of the current video block.
[0547] 2.11. The method of example 2.9 or 2.10, wherein the flag of the prediction mode is inferred to be 0.
[0548] 2.12. A method as described in any one of Examples 2.9-2.11, wherein the group of prediction modes includes at least one of the following: an affine mode, in which at least one control point motion vector is used, an inter-intra prediction mode, in which a prediction signal for the current video block is generated based on at least an intra prediction signal and an inter prediction signal, a generalized bi-predictive prediction (GBi) mode with unequal weighting, a merge with motion vector difference (MMVD) mode, in which the motion vector of the video block is derived based on a merge motion candidate list and is further refined by at least one motion vector difference, and an optional temporal motion vector prediction (ATMVP) mode, in which motion information of a sub-block of the current video block is based on a corresponding block determined by motion information of spatially neighboring blocks of the current video block.
[0549] 2.13. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein when the instructions are executed by the processor, the processor is caused to implement the method described in any one of Examples 2.1 to 2.12.
[0550] 2.14. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for executing the method of any one of Examples 2.1 to 2.12.
[0551] 2.15. A video decoding device comprising a processor configured to implement the method described in any one of Examples 2.1 to 2.12.
[0552] 2.16. A video encoding apparatus comprising a processor configured to implement the method of any one of Examples 2.1 to 2.12.
[0553] 3.1. A method for video processing, comprising: in a process of obtaining motion information of a current video block, during a conversion between the current video block and a bitstream representation of the current video block, wherein the motion information of the current video block is based on at least one affine model of at least one neighboring block of the current video block, determining whether the neighboring block of the current video block uses an intra-frame block copy (IBC) mode, in which at least one reference picture used by the neighboring block is the same as the picture in which the neighboring block is located; based on determining that the neighboring block uses the IBC mode, prohibiting the derivation of affine candidates from the neighboring block for the current video block; and performing the conversion based on the motion information of the current video block.
[0554] 3.2. The method of Example 3.1, wherein the converting comprises encoding the current video block into the bitstream representation.
[0555] 3.3. The method of Example 3.1, wherein the converting comprises decoding a bitstream representation of the current video block to generate pixel values for the current video block.
[0556] 3.4. The method of any of Examples 3.1-3.3, wherein the method further comprises: based on determining that the neighboring block does not use the IBC mode, deriving an affine candidate for the current video block from the neighboring block.
[0557] 3.5. The method of any of Examples 3.1-3.4, wherein the affine candidate comprises an affine merge candidate, and wherein the affine candidate is in a sub-block merge candidate list.
[0558] 3.6. The method of Example 3.5, wherein the neighboring block is considered unavailable based on determining that the neighboring block uses the IBC mode.
[0559] 3.7. The method of any of Examples 3.1-3.4, wherein the affine candidate comprises an affine advanced motion vector prediction (AMVP) candidate, and wherein the affine candidate is in an affine AMVP list.
[0560] 3.8. The method of Example 3.7, wherein the neighboring block is considered unavailable based on determining that the neighboring block uses the IBC mode.
[0561] 3.9. An apparatus in a video system, comprising a processor and a non-volatile memory having instructions thereon, wherein when the instructions are executed by the processor, the processor is caused to implement a method as described in any one of Examples 3.1 to 3.8.
[0562] 3.10. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for executing the method of any one of Examples 3.1 to 3.8.
[0563] 3.11. A video decoding apparatus comprising a processor configured to implement the method as described in one or more of Examples 3.1 to 3.8.
[0564] 3.12. A video encoding apparatus comprising a processor configured to implement the method of one or more of Examples 3.1 to 3.8.
[0565] 4.1. A method for video processing, comprising: during conversion between a current video block and a bitstream representation of the current video block, dividing the current video block into sub-blocks based on a sub-block-based temporal motion vector prediction (SbTMVP) mode; identifying a corresponding block for the sub-block in a motion source picture with a temporal vector based on motion information of spatially adjacent blocks of the current video block; generating a motion vector and a reference index of the sub-block from the corresponding block of the sub-block based on encoding and decoding mode information of the corresponding block; and performing the conversion based on the generated motion vector and reference index.
[0566] 4.2. The method of example 4.1, wherein the motion source picture is a collocated reference picture of the current video block.
[0567] 4.3. A method as described in any one of Examples 4.1 to 4.2, wherein the corresponding block is encoded and decoded in an intra block copy (IBC) mode, and the reference block of the corresponding block includes at least one pixel in the picture where the corresponding block is located.
[0568] 4.4. The method of Example 4.3 further includes treating the corresponding block as unavailable during the conversion.
[0569] 4.5. The method of Example 4.4 further comprises: avoiding deriving the motion vector and the reference picture of the sub-block from the corresponding block.
[0570] 4.6. The method of Example 4.5 further comprises: assigning default motion information to the current video block.
[0571] 4.7. The method of Example 4.3, wherein the corresponding block is considered available and the motion vector and the reference picture of the sub-block are derived from the corresponding block.
[0572] 4.8. The method of Example 4.7, wherein the motion vector of the sub-block is copied from a motion vector associated with the corresponding block.
[0573] 4.9. The method of Example 4.1, wherein if the spatial neighboring block uses the IBC mode, the spatial neighboring block is considered unavailable, wherein the reference block of the spatial neighboring block includes at least one pixel in the picture where the spatial block is located.
[0574] 4.10. A method for video processing, comprising: during the conversion between a current video block and a bitstream representation of the current video block, dividing the current video block into sub-blocks based on a sub-block-based temporal motion vector prediction (SbTMVP) mode; identifying a corresponding block for the sub-block in a motion source picture with a temporal vector based on encoding and decoding mode information of spatially adjacent blocks of the current video block; generating a motion vector and a reference index for the sub-block from the corresponding block of the sub-block; and performing the conversion based on the generated motion vector and reference index.
[0575] 4.11. The method of Example 4.10, wherein the motion source picture is a collocated reference picture of the current video block.
[0576] 4.12. A method as described in any one of Examples 4.10 to 4.11, wherein the spatial neighboring block is encoded and decoded using an intra block copy (IBC) mode, and wherein the reference block of the spatial neighboring block includes at least one pixel in the picture where the spatial neighboring block is located.
[0577] 4.13. The method of Example 4.12 further comprises treating the spatially neighboring blocks as unavailable during the conversion.
[0578] 4.14. The method as described in Example 4.13 also includes: avoiding identifying the corresponding block based on the motion vector of the spatial neighboring block.
[0579] 4.15. The method of Example 4.14 further comprises: assigning default motion information to identify the corresponding block.
[0580] 4.16. The method of any of Examples 4.1 to 4.15, wherein the converting comprises encoding the current video block into the bitstream representation.
[0581] 4.17. The method of any of Examples 4.1 to 4.15, wherein the converting comprises decoding a bitstream representation of the current video block to generate pixel values for the current video block.
[0582] 4.18. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein when the instructions are executed by the processor, the processor is caused to implement the method as described in any one of Examples 4.1 to 4.15.
[0583] 4.19. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method of any of Examples 4.1 to 4.15.
[0584] 4.20. A video decoding apparatus comprising a processor configured to implement a method as described in one or more of Examples 4.1 to 4.15.
[0585] 4.21. A video encoding apparatus comprising a processor configured to implement a method as described in one or more of Examples 4.1 to 4.15.
[0586] It will be appreciated from the foregoing that specific embodiments of the disclosed technology have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of the invention. Accordingly, the disclosed technology is not to be limited except as set forth in the appended claims.
[0587] The implementation of the subject matter and functional operations described in this patent document can be implemented in various systems, implemented in digital electronic circuits, or implemented in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or implemented in a combination of one or more of them. The implementation of the subject matter described in this specification can be implemented as one or more computer program products, that is, one or more computer program instruction modules encoded on a tangible and non-transitory computer-readable medium, for being executed by a data processing device or controlling the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material combination that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing unit" or "data processing device" covers all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0588] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language file), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on one computer or on multiple computers located on one site or distributed on multiple sites and interconnected by a communication network.
[0589] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits).
[0590] For example, processors suitable for executing computer programs include general and special microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to one or more large-capacity storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, to receive data from the one or more large-capacity storage devices, or to transfer data to the one or more large-capacity storage devices, or to both receive and transfer data. However, a computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, for example, including semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.
[0591] Intended to combine the instructions and the Figure 1 The foregoing are to be considered exemplary only, where exemplary means example. As used herein, the use of "or" is intended to include "and / or" unless the context clearly indicates otherwise.
[0592] Although this patent document contains many details, these details should not be interpreted as limitations on any invention or the scope of protection that can be claimed, but rather as descriptions of features specific to a particular embodiment of a particular invention. In this patent document, certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although the above features may be described as working in certain combinations and even initially claimed as such, in some cases, one or more features from the combination may be removed from the claimed combination, and the claimed combination may point to a sub-combination or a variation of the sub-combination.
[0593] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, in order to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.
[0594] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.
Claims
1. A method for video processing, include: Determine whether to apply an intra block copy (IBC) mode to a current video block, in which at least one reference picture used by the current video block is the same as a picture in which the current video block is located; Signaling whether the IBC mode is applied to the current video block; Based on determining that the IBC mode applies to the current video block, avoiding signaling a flag of a group of prediction modes; as well as Based on the application of the IBC mode, converting the current video block, wherein the group of prediction modes comprises an affine mode, wherein the affine mode uses at least one control point motion vector, wherein determining whether to apply the IBC mode to the current video block is performed before signaling an affine flag, and the method further comprises: if it is determined that the IBC mode is applied to the current video block, avoiding signaling the affine flag; and otherwise, signaling the affine flag, The group of prediction modes includes a generalized bidirectional prediction GBi mode, wherein in the GBi mode, different weighting values are associated with different reference pictures, and Wherein, determining whether to apply the IBC mode to the current video block occurs before signaling the GBi index; and the method further includes: If it is determined that the IBC mode applies to the current video block, avoiding signaling a GBi weight index, and Otherwise, the GBi weighted index is signaled.
2. The method of claim 1, wherein the converting comprises encoding the current video block into a bitstream.
3. The method of claim 1, wherein the converting comprises decoding the current video block from a bitstream.
4. The method of claim 1, wherein the affine flag is inferred to be 0 when the affine flag is not signaled. 5 . The method of claim 1 , wherein when the GBi weight index is not signaled, the weight value of the reference picture used by the GBi is inferred to be 0.
6. A method for video processing, include: Parsing a bitstream to determine whether an intra block copy (IBC) mode is applied to a current video block, in which at least one reference picture used by the current video block is the same as a picture in which the current video block is located; Based on the application of the IBC mode, performing a conversion of the current video block, wherein based on determining to apply the IBC mode to the current video block, excluding a flag of a group of prediction modes from the bitstream, wherein the group of prediction modes comprises an affine mode, wherein the affine mode uses at least one control point motion vector, wherein determining whether to apply the IBC mode to the current video block is performed before signaling an affine flag, and the method further comprises: if it is determined that the IBC mode is applied to the current video block, avoiding signaling the affine flag; and otherwise, signaling the affine flag, The group of prediction modes includes a generalized bidirectional prediction GBi mode, wherein in the GBi mode, different weighting values are associated with different reference pictures, and Wherein, determining whether to apply the IBC mode to the current video block occurs before signaling the GBi index; and the method further includes: If it is determined that the IBC mode applies to the current video block, avoiding signaling a GBi weight index, and Otherwise, the GBi weighted index is signaled.
7. The method of claim 6, wherein the converting comprises decoding a bitstream of the current video block into the current video block to generate pixel values of the current video block. The method of claim 6 , wherein the flag of the prediction mode is inferred to be 0.
9. The method according to any one of claims 6 to 8, wherein the group of prediction modes further comprises at least one of the following: an inter-intra prediction mode, wherein a prediction signal for the current video block is generated based at least on an intra prediction signal and an inter prediction signal, merge MMVD mode of motion vector differences, where the motion vector of the video block is derived based on the merge motion candidate list and further refined by at least one motion vector difference, and An optional temporal motion vector prediction ATMVP mode, wherein the motion information of a sub-block of the current video block is based on a corresponding block determined by motion information of a spatial neighboring block of the current video block.
10. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein when the instructions are executed by the processor, the processor is caused to implement the method according to any one of claims 1 to 9.
11. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.
12. A video decoding device, comprising a processor configured to implement the method according to any one of claims 1-9.
13. A video encoding apparatus, comprising a processor configured to implement the method according to any one of claims 1-9.
Citation Information
Patent Citations
Intra BC and inter unification
CN106797476A
Data encoding and decoding
US20160241852A1
Methods and systems of restricting BI-prediction in video coding
US20170034526A1
Method of Intra Block Copy with Flipping for Image and Video Coding
US20170195677A1
Affine motion prediction for video coding
US20170332095A1
