Video sub-picture encoding and decoding

Abstract

A method for visual media processing comprises: converting between a video picture contained in visual media data and a bitstream of the visual media data using sub-pictures, determining that a rule related to one or more sub-pictures is satisfied by the conversion; and performing the conversion according to the rule constrained by the rule, wherein the rule specifies that the size of the sub-picture in the video picture is an integer multiple of the size of a coding tree unit associated with the video picture.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of International Patent Application No. PCT / CN2019 / 107159, filed on September 22, 2019, in a timely manner under applicable patent law and / or under the rules of the Paris Convention. The entire disclosure of the foregoing application is incorporated by reference as a part of the disclosure of this application for all purposes under U.S. patent law. Technical Field

[0003] This patent document relates to image and video encoding and decoding technologies, devices and systems. Background Art

[0004] Digital video consumes the largest amount of bandwidth on the Internet and other digital communication networks.As the number of connected user devices capable of receiving and displaying video increases, it is expected that the bandwidth demand for digital video usage will continue to grow. Summary of the Invention

[0005] The techniques of this disclosure may be used by video decoder or encoder embodiments during video decoding or encoding using alternative temporal motion vector prediction.

[0006] In one exemplary aspect, a video processing method is disclosed. The method includes determining, for a current video block, a motion vector used in a sub-block based motion vector prediction (sbTMVP) process to locate a corresponding block in a collocated picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated based on a scaling operation; and performing conversion between the current video block and a bitstream representation of visual media data based on the motion vector.

[0007] In another exemplary aspect, another video processing method is disclosed, comprising determining, for a current video block, a motion vector used in a sub-block based temporal motion vector prediction (sbTMVP) process to locate a corresponding block in a collocated picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated with respect to a center point of the current video block; modifying the center point of the current video block by applying one or more operations; and performing conversion between the current video block and a bitstream representation of visual media data based on using the center point modified by applying the one or more operations.

[0008] In yet another exemplary aspect, another video processing method is disclosed. The method includes determining, for a current video block, a motion vector used in a sub-block-based temporal motion vector prediction (sbTMVP) process to locate a corresponding block in a collocated picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated with respect to points in the corresponding block in the collocated picture; modifying the points in the corresponding block in the collocated picture by applying one or more operations; and performing conversion between the current video block and a bitstream representation of visual media data based on using the points in the corresponding block in the collocated picture modified by applying the one or more operations.

[0009] In yet another aspect, another method of video processing is disclosed. The method includes: determining, for converting between a video picture included in visual media data and a bitstream representation of the visual media data using sub-pictures, that a rule related to one or more sub-pictures is satisfied by the conversion; and performing the conversion according to a constrained rule, wherein the rule specifies that a size of the sub-picture in the video picture is an integer multiple of a size of a codec tree unit associated with the video picture.

[0010] In yet another exemplary aspect, another method of video processing is disclosed. The method includes: determining, for a conversion between a video picture included in visual media data and a bitstream representation of the visual media data using sub-pictures, that a rule related to one or more sub-pictures is satisfied by the conversion; and performing the conversion according to a constrained rule, wherein the rule specifies that all sub-pictures in the video picture do not overlap and that all sub-pictures in the video picture together cover the entire video picture.

[0011] In yet another exemplary aspect, another method of video processing is disclosed, comprising: determining, during conversion between a video unit of visual media and a bitstream representation of the visual media data, whether to use a reference picture resampling (RPR) technique; and performing the conversion based on the determination, wherein a flag corresponding to the determination is included in the bitstream representation at a sequence parameter set level.

[0012] In yet another exemplary aspect, another method of video processing is disclosed. The method includes selecting, during a motion compensation process, an interpolation filter to derive a prediction block for a current block of visual media data based on satisfying a condition, wherein the condition is based at least in part on determining that a resolution of a reference picture is different from a resolution of the current picture and / or a dimension of a window associated with the reference picture is different from a dimension of a window associated with the current picture; and performing a conversion between the current block of visual media data and a bitstream representation of the current block.

[0013] In yet another exemplary aspect, another method of video processing is disclosed. The method includes: converting between a current block of visual media data and a bitstream representation of the visual media data, determining that the current block is a combined inter-frame intra prediction (CIIP) block, wherein an intra prediction block of the CIIP block is generated using a transform unit (TU) size, wherein a final prediction of the current block in the combined inter-frame intra prediction (CIIP) block is based on a weighted sum of an inter prediction of the current block and an intra prediction of the current block; and performing the conversion based on the determination.

[0014] In yet another exemplary aspect, a video encoder device is disclosed. The video encoder device includes a processor configured to implement the method described herein.

[0015] In yet another exemplary aspect, a video decoder device is disclosed. The video decoder device includes a processor configured to implement the method described herein.

[0016] In yet another aspect, a computer-readable medium is disclosed having code stored thereon that, when executed by a processor, causes the processor to implement the methods described in this document.

[0017] These and other aspects are described in this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an example of the derivation process for building a Merge candidate list.

[0019] Figure 2 Exemplary locations of spatial merge candidates are shown.

[0020] Figure 3 An example of candidate pairs considered for redundancy checking of spatial merge candidates is shown.

[0021] Figures 4A-4B Exemplary locations of the second PU for Nx2N and 2NxN partitions are shown.

[0022] Figure 5 FIG. 4 is an exemplary illustration of motion vector scaling of a temporal merge candidate.

[0023] Figure 6 Exemplary candidate positions of the time-domain Merge candidate, C0 and C1, are shown.

[0024] Figure 7 An example of combining bi-predictive Merge candidates is shown.

[0025] Figure 8 An exemplary derivation process of motion vector prediction candidates is shown.

[0026] Figure 9 is an exemplary illustration of motion vector scaling of spatial motion vector candidates.

[0027] Figure 10 An example of alternative temporal motion vector prediction (ATMVP) motion prediction for a CU is shown.

[0028] Figure 11 An example of one CU with four sub-blocks (AD) and its neighboring blocks (a–d) is shown.

[0029] Figure 12 Flowchart of an example of encoding with different MV precisions

[0030] Figures 13A-13B Shown are a 135 degree partition type (partitioned from the top left corner to the bottom right corner) and a 45 degree partition mode. Illustration of the partitioning of a CU into two triangular prediction units (two partition modes).

[0031] Figure 14 An example of the positions of adjacent blocks is shown.

[0032] Figure 15 Examples of top and left blocks of a video block are shown.

[0033] Figures 16A-16B Examples of 2 control point motion vectors (CPMV) and 3 CPMV are shown.

[0034] Figure 17 An example of the affine MVF for each sub-block is shown.

[0035] Figures 18A-18B Examples of 4 and 6 parameter affine models are shown.

[0036] Figure 19 It is the MVP of AF_INTER that inherits the affine candidate.

[0037] Figure 20 An example of building an affine motion predictor in AF_INTER is shown.

[0038] Figures 21A-21B An example of control point motion vectors in affine codec in AF_MERGE is shown.

[0039] Figure 22 An example of candidate positions for the affine merge mode is shown.

[0040] Figure 23 An example of an intra picture block copy operation is shown.

[0041] Figure 24 An example of valid corresponding areas in a co-located picture is shown.

[0042] Figure 25 An exemplary flow chart for history-based motion vector prediction is shown.

[0043] Figure 26 Shows the modified Merge list building process.

[0044] Figure 27 An exemplary embodiment of proposing a valid region when a current block is within a basic region is shown.

[0045] Figure 28 An exemplary embodiment of a valid area when the current block is not within the base area is shown.

[0046] Figure 29 is a block diagram of an example of a video processing device.

[0047] Figure 30 is a flowchart of an example of a video processing method.

[0048] Figure 31A An example of the location of the identification of default motion information in the current standard is shown.

[0049] Figure 31B An example of the location of the identification of the default motion information in the proposed standard is shown.

[0050] Figure 32 is a block diagram of an exemplary video processing system in which the disclosed technology may be implemented.

[0051] Figure 33 is a flowchart of an example of a visual media processing method.

[0052] Figure 34 is a flowchart of an example of a visual media processing method.

[0053] Figure 35 is a flowchart of an example of a visual media processing method.

[0054] Figure 36 is a flowchart of an example of a visual media processing method.

[0055] Figure 37 is a flowchart of an example of a visual media processing method.

[0056] Figure 38 is a flowchart of an example of a visual media processing method.

[0057] Figure 39 is a flowchart of an example of a visual media processing method.

[0058] Figure 40 is a flowchart of an example of a visual media processing method.

[0059] Figure 41 is a block diagram illustrating an exemplary video coding system.

[0060] Figure 42 is a block diagram illustrating an encoder according to some embodiments of the present disclosure.

[0061] Figure 43 is a block diagram illustrating a decoder according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0062] This document provides various techniques that can be used by a decoder of a video bitstream to improve the quality of a decompressed or decoded digital video or image. Additionally, a video encoder can also implement these techniques during the encoding process to reconstruct decoded frames for further encoding.

[0063] The section headings are used in this document to facilitate understanding and do not limit the embodiments and techniques to the corresponding sections. Therefore, embodiments from one section can be combined with embodiments from other sections.

[0064] 1. Overview

[0065] This patent document relates to video codec technology. Specifically, it relates to motion vector coding in video codecs. It can be applied to existing video codec standards such as HEVC or to a standard to be finalized (universal video codec). It can also be applied to future video codec standards or video encoders.

[0066] 2. Introduction

[0067] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced the H.261 and H.263 standards, ISO / IEC produced the MPEG-1 and MPEG-4 Visual standards, and the two organizations jointly produced the H.262 / MPEG-2 Video standard, the H.264 / MPEG-4 Advanced Video Coding (AVC) standard, and the H.265 / HEVC standard. Starting with H.262, video codec standards have been based on a hybrid video codec architecture that utilizes temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, JVET has adopted many new methods and incorporated them into reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Team (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1SC29 / WG11 (MPEG) was established to work on the VVC standard, which reduces the bit rate by 50% compared to HEVC.

[0068] The latest version of the VVC draft, Versatile Video Codec (Draft 3), can be found at: http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 12_Macao / wg11 / JVET-L1001-v2.zip.

[0069] The latest reference software for VVC, named VTM, can be found at: https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / tags / VTM-3.0rc1

[0070] 2.1 Inter-frame prediction in HEVC / H.265

[0071] Each inter-prediction PU has motion parameters for one or two reference picture lists. The motion parameters include motion vectors and reference picture indices. The use of one of the two reference picture lists can also be signaled using inter_pred_idc. Motion vectors can be explicitly encoded or decoded as deltas relative to the predictor.

[0072] When a CU is encoded using skip mode, a PU is associated with the CU and there are no significant residual coefficients, no coded motion vector increments or reference picture indices. A Merge mode is specified whereby the motion parameters for the current PU are obtained from neighboring PUs, including spatial and temporal candidates. Merge mode can be applied to any inter-predicted PU, not just skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where the motion vector (more precisely, the motion vector difference (MVD) compared to the motion vector predictor), the corresponding reference picture index for each reference picture list, and the reference picture list usage are explicitly signaled for each PU. Such a mode is named advanced motion vector prediction in this disclosure.

[0073] When signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of samples. This is often called "unidirectional prediction." Unidirectional prediction is available for P slices and B slices.

[0074] When signaling indicates that two reference picture lists are to be used, a PU is generated from two sample blocks. This is called "bi-prediction." Bi-prediction is only available for B slices.

[0075] The following provides detailed information on the inter prediction modes specified in HEVC, starting with the Merge mode.

[0076] 2.1.1 Reference Image List

[0077] In HEVC, the term inter prediction is used to refer to predictions derived from data elements (e.g., sample values ​​or motion vectors) of reference pictures other than the currently decoded picture. As in H.264 / AVC, a picture can be predicted from multiple reference pictures. Reference pictures used for inter prediction are organized into one or more reference picture lists. A reference index identifies which reference picture in the list should be used to create the prediction signal.

[0078] A single reference picture list, List 0, is used for P slices, and two reference picture lists, List 0 and List 1, are used for B slices. The reference pictures contained in Lists 0 and 1 can be pictures from the past and the future in terms of capture / display order.

[0079] 2.1.2Merge Mode

[0080] 2.1.2.1 Merge Mode Candidate Derivation

[0081] When using Merge mode to predict a PU, the index to the entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be summarized in the following steps:

[0082] Step 1: Initial candidate derivation

[0083] Step 1.1: Spatial Candidate Derivation

[0084] Step 1.2: Redundancy check of spatial candidates

[0085] Step 1.3: Time Domain Candidate Derivation

[0086] Step 2: Additional candidate insertions

[0087] Step 2.1: Create bidirectional prediction candidates

[0088] Step 2.2: Insert zero motion candidates

[0089] exist Figure 1 These steps are also schematically depicted in . For spatial domain Merge candidate derivation, a maximum of four Merge candidates are selected from candidates located at five different positions. For temporal domain Merge candidate derivation, a maximum of one Merge candidate is selected from two candidates. Since the number of candidates for each PU is assumed to be constant at the decoder, additional candidates are generated when the number of candidates obtained from step 1 does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, binary unary truncation (TU) is used to encode the index of the best Merge candidate. If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0090] Hereinafter, operations associated with the aforementioned steps are described in detail.

[0091] 2.1.2.2 Spatial Candidate Derivation

[0092] In the derivation of spatial domain Merge candidates, select Figure 2There are up to four Merge candidates among the candidates for the positions shown in . The order of derivation is A1, B1, B0, A0 and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is unavailable (for example because it belongs to another strip or slice) or is intra-coded. After the candidate at position A1 is added, the remaining candidates are added and subjected to a redundancy check, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the pairs with Figure 3 The pairs linked by arrows in are considered, and candidates are added to the list only if the corresponding candidates for redundancy checking do not have the same motion information. Another source of duplicate motion information is a "second PU" associated with a partition other than 2Nx2N. As an example, Figures 4A-4B The second PU is shown for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the list construction does not consider the candidate at position A1. In fact, adding this candidate would result in two prediction units with the same motion information, which is redundant for only one PU in the codec unit. Similarly, position B1 is not considered when the current PU is partitioned into 2N×N.

[0093] 2.1.2.3 Time Domain Candidate Derivation

[0094] In this step, only one candidate is added to the list. In particular, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the co-located PU, which belongs to the picture with the smallest POC difference relative to the current picture in the given reference picture list. The reference picture list used for the derivation of the co-located PU is explicitly signaled in the slice header. Figure 5 As shown by the dashed line, the derivation of the scaled motion vector for the temporal merge candidate is obtained, which is scaled from the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors are obtained and combined to produce a bi-predictive merge candidate, one for reference picture list 0 and the other for reference picture list 1.

[0095] Figure 5 It is a diagram of the motion vector scaling of the time domain Merge candidate.

[0096] In the co-located PU(Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, such as Figure 6 If the PU at position C0 is not available, is intra-coded, or is outside the current codec tree unit (CTU, also known as LCU, largest codec unit) row, position C1 is used. Otherwise, position C0 is used in the derivation of the time domain merge candidate.

[0097] Figure 6 Examples of candidate positions of the time-domain Merge candidate, C0 and C1, are shown.

[0098] 2.1.2.4 Additional Candidate Insertion

[0099] In addition to spatial and temporal Merge candidates, there are two additional types of Merge candidates: combined bi-directional prediction Merge candidate and zero Merge candidate. Combined bi-directional prediction Merge candidate is generated by utilizing spatial and temporal Merge candidates. Combined bi-directional prediction Merge candidate is only used for B slices. Combined bi-directional prediction candidate is generated by combining the first reference picture list motion parameters of the initial candidate with the second reference picture list motion parameters of another candidate. If the two tuples provide different motion hypotheses, they will form a new bi-directional prediction candidate. As an example, Figure 7 The following situation is shown, in which two candidates with mvL0 and refIdxL0 or mvL1 and refIdxL1 in the original list (on the left) are used to create a combined bi-predictive Merge candidate, which is added to the final list (on the right). There are many rules about the combinations that are considered to generate these additional Merge candidates.

[0100] Zero motion candidates are inserted to fill the remaining entries of the Merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list.

[0101] More specifically, the following steps are performed in order until the Merge list is full:

[0102] 1. Set the variable numRef to the number of reference pictures associated with list 0 for a P slice, or the minimum number of reference pictures in both lists for a B slice;

[0103] 2. Add non-repeating zero motion candidates:

[0104] For variable i being 0...numRef-1, a default motion candidate is added to list 0 (if P slice) or for both lists (if B slices), with MV set to (0, 0) and reference picture index set to i.

[0105] 3. Add repeated zero motion candidates with MV set to (0, 0), reference picture index of list 0 set to 0 (if P slice) and reference picture index of both lists set to 0 (if B slice).

[0106] Finally, these candidates are not checked redundantly.

[0107] 2.1.3AMVP

[0108] AMVP exploits the spatial-temporal correlation of motion vectors with adjacent PUs, which is used for explicit transmission of motion parameters. For each reference picture list, a motion vector candidate list is constructed by first checking the availability of adjacent PU positions in the time domain to the left and above, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and transmit the corresponding index indicating the selected candidate. Similar to the Merge index signaling, unary truncation is used to encode the index of the best motion vector candidate. The maximum value to be encoded in this case is 2 (see Figure 8 ). In the following sections, details on the derivation process of motion vector prediction candidates are provided.

[0109] 2.1.3.1 Derivation of AMVP Candidates

[0110] Figure 8 The derivation process of motion vector prediction candidates is summarized.

[0111] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. Figure 2 As shown, for spatial motion vector candidate derivation, two motion vector candidates are ultimately derived based on the motion vectors of each PU located at five different positions.

[0112] For temporal motion vector candidate derivation, a motion vector candidate is selected from two candidates derived based on two different co-located positions. After making the first list of spatial-temporal candidates, duplicate motion vector candidates are removed from the list. If the number of potential candidates is greater than two, motion vector candidates whose reference picture index within the associated reference picture list is greater than one are removed from the list. If the number of spatial-temporal motion vector candidates is less than two, an additional zero motion vector candidate is added to the list.

[0113] 2.1.3.2 Spatial Motion Vector Candidates

[0114] In the derivation of spatial motion vector candidates, a maximum of two candidates are considered among five potential candidates, which are selected from the candidate located at Figure 2 The derivation order for the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order for the top side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Therefore, there are four cases for each side that can be used as motion vector candidates, two of which do not require spatial scaling and two of which use spatial scaling. The four different cases are summarized below.

[0115] No airspace scaling

[0116] –(1) Same reference picture list and same reference picture index (same POC)

[0117] –(2) Different reference picture lists, but same reference picture (same POC)

[0118] Airspace scaling

[0119] –(3) Same reference picture list, but different reference pictures (different POC)

[0120] –(4) Different reference picture lists and different reference pictures (different POC)

[0121] First, the no spatial scaling case is checked, followed by spatial scaling. Spatial scaling is considered when the POC has different reference pictures for the neighboring PU and the current PU, regardless of the reference picture list. If all PUs of the left candidate are unavailable or intra-coded, scaling of the upper motion vector is allowed to facilitate the parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0122] In the spatial domain scaling process, the motion vectors of the adjacent PUs are scaled in a similar way to the temporal domain scaling, such as Figure 9 The main difference is that the reference picture list and index of the current PU are given as input; the actual scaling process is the same as the temporal scaling.

[0123] 2.1.3.3 Temporal Motion Vector Candidates

[0124] In addition to the reference picture index derivation, the entire process of deriving the temporal Merge candidate is the same as that of deriving the spatial motion vector candidate (see Figure 6 The reference picture index is signaled to the decoder.

[0125] 2.2 Sub-CU-based motion vector prediction method in JEM

[0126] In JEM with QTBT, each CU can have at most one set of motion parameters for each prediction direction. Two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving motion information for all sub-CUs of the large CU. The alternative temporal motion vector prediction (ATMVP) method allows each CU to retrieve multiple sets of motion information from multiple blocks smaller than the current CU in a co-located reference picture. In the spatio-temporal motion vector prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using the temporal motion vector predictor and the spatial neighboring motion vectors.

[0127] In order to preserve a more accurate motion field for sub-CU motion prediction, motion compression of reference frames is currently disabled.

[0128] Figure 10 An example of ATMVP motion prediction for a CU is shown.

[0129] 2.2.1 Alternative temporal motion vector prediction

[0130] In the alternative temporal motion vector prediction (ATMVP) method, the motion vector temporal motion vector prediction (TMVP) is modified by retrieving multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU. A sub-CU is a square N×N block (N is set to 4 by default).

[0131] ATMVP predicts the motion vectors of sub-CUs within a CU in two steps. The first step is to identify corresponding blocks in a reference picture with so-called temporal vectors. The reference picture is called the motion source picture. The second step is to divide the current CU into sub-CUs and obtain the motion vector and reference index of each sub-CU from the blocks corresponding to each sub-CU.

[0132] In the first step, the reference picture and the corresponding block are determined by the motion information of the spatially adjacent blocks of the current CU. To avoid repeated scanning of adjacent blocks, the first merge candidate in the merge candidate list of the current CU is used. The first available motion vector and its associated reference index are set as the temporal vector and the index to the motion source picture. In this way, in ATMVP, the corresponding block can be identified more accurately than in TMVP, where the corresponding block (sometimes called the co-located block) is always at the right bottom or center position relative to the current CU.

[0133] In the second step, the corresponding block of the sub-CU is identified by the time domain vector in the motion source picture by adding the time domain vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample point) is used to derive the motion information of the sub-CU. After identifying the motion information of the corresponding N×N block, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes apply. For example, the decoder checks whether the low latency condition (i.e., the POC of all reference pictures of the current picture is less than the POC of the current picture) is met, and possibly uses the motion vector MV for each sub-CU x (motion vector corresponding to reference picture list X) to predicted motion vector MV y (where X equals 0 or 1, and Y equals 1-X).

[0134] 2.2.2 Spatial Temporal Motion Vector Prediction (STMVP)

[0135] In this method, the motion vector of a sub-CU is recursively derived following the grid scan order. Figure 11 The concept is illustrated. For example, Figure 11 An 8x8 CU is shown with four 4x4 sub-CUs A, B, C, and D. Adjacent 4x4 blocks in the current frame are labeled a, b, c, and d.

[0136] The motion derivation of sub-CU A begins with identifying its two spatial neighbors. The first neighborhood is the N×N block (block c) above sub-CU A. If block c is not available or is intra-coded, the other N×N blocks above sub-CU A are checked (from left to right, starting from block c). The second neighborhood is the block (block b) to the left of sub-CU A. If block b is not available or is intra-coded, the other blocks to the left of sub-CU A are checked (from top to bottom, starting from block b). For each list, the motion information obtained from the adjacent blocks is scaled to the first reference frame of the given list. Next, the temporal motion vector predictor (TMVP) of sub-block A is derived by following the same process as the TMVP derivation specified in HEVC. The motion information of the co-located block at position D is retrieved and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged separately for each reference list. The average motion vector is assigned as the motion vector of the current sub-CU.

[0137] 2.2.3 Sub-CU Motion Prediction Mode Signaling

[0138] Sub-CU mode is enabled as an additional Merge candidate, and there are no additional syntax elements required to signal the mode. Two additional Merge candidates are added to the Merge candidate list of each CU to represent ATMVP mode and STMVP mode. If the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates are used. The encoding and decoding logic of the additional Merge candidates is the same as the encoding logic of the Merge candidates in HM, which means that for each CU in a P or B slice, the two additional Merge candidates require two more RD checks.

[0139] In JEM, all bins of the Merge index are CABAC context coded, whereas in HEVC, only the first bin is context coded, and the remaining bins are context bypass coded.

[0140] 2.3 Inter-frame prediction method in VVC

[0141] There are several new codec tools for inter prediction improvements, such as Adaptive Motion Vector Difference Resolution (AMVR) for signaling MVD, Affine Prediction Mode, Triangle Prediction Mode (TPM), ATMVP, Normalized Bi-Prediction (GBI), Bidirectional Optical Flow (BIO).

[0142] 2.3.1 Adaptive Motion Vector Difference Resolution

[0143] In HEVC, when use_integer_mv_flag is equal to 0 in the slice header, the motion vector difference (MVD) (between the motion vector and the predicted motion vector of the PU) is signaled in units of quarter luma samples. In VVC, Local Adaptive Motion Vector Resolution (LAMVR) is introduced. In VVC, MVD can be signaled in units of quarter luma samples, integer luma samples, or four luma samples (i.e., 1 The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

[0144] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or quad luma sample MV precision is used.

[0145] When the first MVD resolution flag of the CU is zero or when the CU is not coded / decoded (meaning all MVDs in the CU are zero), a quarter-luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or four-luma sample MV precision, the MVP in the AMVP candidate list of the CU is rounded to the corresponding precision.

[0146] In the encoder, CU-level RD checks are used to determine which MVD resolution is to be used for the CU. That is, three CU-level RD checks are performed for each MVD resolution. To speed up the encoder, the following coding scheme is applied in JEM.

[0147] · During the RD check of a CU with normal quarter-luma sample MVD resolution, the motion information (integer luma sample accuracy) of the current CU is stored. The stored motion information (after rounding) is used as a starting point for a small-range motion vector refinement during the RD check of the same CU with integer luma sample and 4-luma sample MVD resolutions, so that the time-consuming motion estimation process is not repeated three times.

[0148] · The RD check of a CU with 4-luma sample MVD resolution is conditionally invoked. For a CU, when the RD cost of the integer luma sample MVD resolution is much larger than that of the quarter-luma sample MVD resolution, the RD check of the 4-luma sample MVD resolution of the CU is skipped.

[0149] The encoding process is shown in Figure 12 First, the 1 / 4 pixel MV is tested and the RD cost is calculated and designated as RDCost0, then the integer MV is tested and the RD cost is designated as RDCost1. If RDCost1 < th * RDCost0 (where th is a positive value), then the 4 pixel MV is tested; otherwise, the 4 pixel MV is skipped. Basically, when checking the integer or 4 pixel MV, the motion information and RD cost, etc. for the 1 / 4 pixel MV are known and can be reused to accelerate the encoding process of the integer or 4 pixel MV.

[0150] 2.3.2 Triangular Prediction Mode

[0151] The concept of the Triangular Prediction Mode (TPM) is to introduce a new triangular partitioning for motion compensation prediction. As Figures 13A-13BAs shown, it divides the CU into two triangular prediction units in the diagonal or anti-diagonal direction. Each triangular prediction unit in the CU is inter-predicted using its own unidirectional prediction motion vector and reference frame index, which is derived from a single unidirectional prediction candidate list. An adaptive weighting process is applied to the diagonal edges after predicting the triangular prediction unit. Then, the transformation and quantization process are applied to the entire CU. This mode is only applied to Merge mode (Skip mode is treated as a special Merge mode).

[0152] Figures 13A-13B is a diagram of dividing the CU into two triangular prediction units (two division modes). For example, Figure 13A An example of a 135 degree split type (divided from the top left corner to the bottom right corner) is shown, and Figure 13B An example of a 45-degree division pattern is shown.

[0153] 2.3.2.1 TPM’s One-Way Prediction Candidate List

[0154] The unidirectional prediction candidate list (called TPM motion candidate list) consists of five unidirectional prediction motion vector candidates. It is derived from seven adjacent blocks including five spatial adjacent blocks (1 to 5) and two temporal co-located blocks (6 to 7), as shown in Figure 14 As shown in the figure, the motion vectors of seven adjacent blocks are collected and placed into a unidirectional prediction candidate list in the following order: unidirectional prediction motion vector, L0 motion vector of bidirectional prediction motion vector, L1 motion vector of bidirectional prediction motion vector, and the average motion vector of L0 and L1 motion vectors of bidirectional prediction motion vector. If the number of candidates is less than five, a zero motion vector is added to the list. Motion candidates added to this list for TPM are called TPM candidates, while motion information derived from spatial / temporal blocks is called regular motion candidates.

[0155] More specifically, the following steps are involved:

[0156] 1) When adding regular motion candidates from spatially adjacent blocks, the full pruning operation From A1, B1, B0, A0, B2, Col and Col2 (corresponding to Figure 14 Blocks 1-7 in ) obtain regular motion candidates.

[0157] 2) Set the variable numCurrMergeCand = 0

[0158] 3) For each normal motion candidate derived from A1, B1, B0, A0, B2, Col and Col2, if not pruned and numCurrMergeCand is less than 5, if the normal motion candidate is unidirectionally predicted (from list 0 or list 1), it is directly added to the Merge list as a TPM candidate and numCurrMergeCand is increased by 1. Such a TPM candidate is named

[0159] 'Original unidirectional prediction candidate'.

[0160] Apply Full Trim .

[0161] 4) For each motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2, if not pruned and numCurrMergeCand is less than 5, if the regular motion candidate is bidirectionally predicted, the motion information from List 0 is added to the TPM Merge list (i.e., modified to be unidirectionally predicted from List 0) as a new TPM candidate and numCurrMergeCand is incremented by 1. Such a TPM candidate is named 'truncated List 0 prediction candidate'.

[0162] Apply a full trim.

[0163] 5) For each motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2, if not pruned and numCurrMergeCand is less than 5, if the regular motion candidate is bi-directionally predicted, the motion information from List 1 is added to the TPM Merge list (i.e., modified to be uni-directionally predicted from List 1), and numCurrMergeCand is incremented by 1. Such a TPM candidate is named 'truncated List 1 prediction candidate'.

[0164] Apply a full trim.

[0165] 6) For each motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2, if not pruned and numCurrMergeCand is less than 5, if the regular motion candidate is bidirectionally predicted,

[0166] If the slice QP of the list 0 reference picture is less than the slice QP of the list 1 reference picture, the motion information of list 1 is first scaled to the list 0 reference picture, and the average of the two MVs (one from the original list 0 and the other is the scaled MV from list 1) is added to the TPM Merge list, such a candidate is called an average unidirectional prediction from list 0 motion candidate, and numCurrMergeCand is increased by 1.

[0167] Otherwise, the motion information of list 0 is first scaled to the list 1 reference picture, and the average of the two MVs (one from the original list 1 and the other is the scaled MV from list 0) is added to the TPM Merge list, such a TPM candidate is called the average unidirectional prediction from list 1 motion candidate, and numCurrMergeCand is increased by 1.

[0168] Apply a full trim.

[0169] 7) If numCurrMergeCand is less than 5, add a zero motion vector candidate.

[0170] When a candidate is inserted into the list, if it has to be compared with all previously added candidates to find out if it is identical to one of them, such a process is called full pruning.

[0171] 2.3.2.2 Adaptive Weighting Process

[0172] After predicting each triangle prediction unit, an adaptive weighting process is applied to the diagonal edge between two triangle prediction units to derive the final prediction for the entire CU. Two weighting factor groups are defined as follows:

[0173] · First weighted factor group: {7 / 8, 6 / 8, 4 / 8, 2 / 8, 1 / 8} and {7 / 8, 4 / 8, 1 / 8}

[0174] are used for luminance and chrominance samples respectively;

[0175] · Second weighted factor group: {7 / 8, 6 / 8, 5 / 8, 4 / 8, 3 / 8, 2 / 8, 1 / 8} and {6 / 8,

[0176] 4 / 8, 2 / 8} are used for luma and chroma samples respectively.

[0177] The weighting factor set is selected based on the comparison of the motion vectors of the two triangle prediction units. When the reference pictures of the two triangle prediction units are different or the difference between their motion vectors is greater than 16 pixels, the second weighting factor set is used. Otherwise, the first weighting factor set is used.

[0178] 2.3.2.3 Triangle Prediction Mode (TPM) Signaling

[0179] A one-bit flag indicating whether to use TPM can be first signaled. Figures 13A-13B ), and further signals the selected Merge index for each of the two partitions.

[0180] 2.3.2.3.1TPM Flag Signaling Notification

[0181] If the width and height of the luma block are denoted by W and H respectively, for W*H<64, triangle prediction mode is disabled.

[0182] When a block is encoded or decoded in affine mode, triangle prediction mode is also disabled.

[0183] When a block is coded in Merge mode, a one-bit flag may be signaled to indicate whether the triangle prediction mode is enabled or disabled for the block.

[0184] Flags are encoded and decoded with 3 contexts, based on the following equations:

[0185] Ctx index=((left block L available&&L is coded with TPM?)1:0)+((Above block A available&&Ais coded with TPM?)1:0);

[0186] Figure 15 Shown are examples of contiguous blocks used for context selection in TPM flag encoding and decoding (A and L).

[0187] 2.3.2.3.2 Signaling of the indication of the two partitioning modes (as shown in FIG13 ), and the selected Merge index for each of the two partitions

[0188] Note that the partition mode and the Merge index of the two partitions are jointly coded and decoded. In addition, the Merge index is restricted so that the two partitions cannot use the same reference index. Therefore, there are 2 (partition mode) * N (maximum number of Merge candidates) * (N-1) possibilities, where N is set to 5. One indication is coded and decoded, and the mapping between the partition mode, the two Merge indexes and the coding indication is derived from the array defined below:

[0189] const uint8_t g_TriangleCombination[TRIANGLE_MAX_NUM_CANDS][3]=

[0190] {

[0191] {0,1,0},{1,0,1},{1,0,2},{0,0,1},{0,2,0},

[0192] {1,0,3},{1,0,4},{1,1,0},{0,3,0},{0,4,0},

[0193] {0,0,2},{0,1,2},{1,1,2},{0,0,4},{0,0,3},

[0194] {0,1,3},{0,1,4},{1,1,4},{1,1,3},{1,2,1},

[0195] {1,2,0},{0,2,1},{0,4,3},{1,3,0},{1,3,2},

[0196] {1,3,4},{1,4,0},{1,3,1},{1,2,3},{1,4,1},

[0197] {0,4,1},{0,2,3},{1,4,2},{0,3,2},{1,4,3},

[0198] {0,3,1},{0,2,4},{1,2,4},{0,4,2},{0,3,4}};

[0199] Division mode (45 degrees or 135 degrees) = g_TriangleCombination[signaled indication][0];

[0200] Merge index of candidate A = g_TriangleCombination[signaled indication][1];

[0201] Merge index of candidate B = g_TriangleCombination[signaled indication][2];

[0202] Once two motion candidates A and B are derived, the motion information of the two partitions (PU1 and PU2) can be set from A or B. Whether PU1 uses the motion information of Merge candidate A or B depends on the prediction direction of the two motion candidates. Table 1 shows the relationship between the two motion candidates A and B derived using two partitions.

[0203] Table 1: Derivation of motion information for the segmentation of two derived Merge candidates (A, B)

[0204]

[0205] 2.3.2.3.3 Entropy codec indicated (indicated by merge_triangle_idx)

[0206] merge_triangle_idxIn the range [0, 39] (inclusive), K-order Exponential Golomb (EG) codec is used for binarization of merge_triangle_idx, where K is set to 1.

[0207] K times EG

[0208] To encode larger numbers in fewer bits (at the expense of using more bits to encode smaller numbers), this can be normalized with a non-negative integer parameter k. To encode a non-negative integer x in the k-th Exponential Golomb code:

[0209] 1. Use the above 0-degree Exponential Golomb codec to encode □x / 2 k □, then

[0210] 2. Encode x modulo 2 in binary k

[0211] Table 2: Exponential Golomb k encoding and decoding examples

[0212]

[0213] 2.3.3 Affine Motion Compensated Prediction

[0214] In HEVC, only the translation motion model is applied to motion compensation prediction (MCP). In the real world, there are many kinds of motion, such as scaling, rotation, perspective motion, and other unconventional motions. In VVC, a simplified affine transformation motion compensation prediction with a 4-parameter affine model and a 6-parameter affine model is applied. Figures 16A-16B As shown, the block affine motion field is used for the 4-parameter affine model ( Figure 16A ) is described by two control point motion vectors (CPMV) and for a 6-parameter affine model ( Figure 16B ) is described by 3 CPMVs.

[0215] The motion vector field (MVF) of a block is described by the following equations, using the 4-parameter affine model in equation (1) (where the 4 parameters are defined as variables a, b, e, and f) and the 6-parameter affine model in equation (2) (where the 4 parameters are defined as variables a, b, c, d, e, and f), respectively:

[0216]

[0217] Among them (mv h 0,mv h 0) is the motion vector of the left top corner control point, and (mv h 1, mv h 1) is the motion vector of the right top corner control point, and (mv h 2, mv h2) is the motion vector of the left bottom corner control point, all three motion vectors are called control point motion vectors (CPMV), (x, y) represents the coordinates of the representative point relative to the left top sample point in the current block, and (mv h (x,y),mv v (x, y)) is the motion vector derived for the sample at (x, y). The CP motion vector can be signaled (as in affine AMVP mode) or derived in real time (as in affine Merge mode). w and h are the width and height of the current block. In practice, division is implemented by a right shift with a rounding operation. In VTM, the representative point is defined as the center position of the sub-block, for example, when the coordinates of the left top corner of the sub-block relative to the left top sample in the current block are (xs, ys), the coordinates of the representative point are located as (xs+2, ys+2). For each sub-block (i.e., 4x4 in VTM), the representative point is used to derive the motion vector for the entire sub-block.

[0218] In order to further simplify the motion compensation prediction, the sub-block based affine transformation prediction is applied. In order to derive the motion vector of each M×N (in the current VVC, M and N are both set to 4) sub-block, as Figure 17 The motion vector of the center sample of each sub-block shown is calculated according to equations (1) and (2) and rounded to 1 / 16 fractional accuracy. Then, a 1 / 16 pixel motion compensated interpolation filter is applied to generate a prediction for each sub-block using the derived motion vector. The 1 / 16 pixel interpolation filter is introduced by the affine mode.

[0219] After MCP, the high-accuracy motion vector of each sub-block is rounded and saved with the same accuracy as the normal motion vector.

[0220] 2.3.3.1 Signaling of Affine Prediction

[0221] Similar to the translational motion model, there are two modes to signal the auxiliary information of affine prediction, namely AFFINE_INTER and AFFINE_MERGE.

[0222] 2.3.3.2AF_INTER Mode

[0223] AF_INTER mode may be applied for CUs whose width and height are both greater than 8. An affine flag in the CU level is signaled in the bitstream to indicate whether AF_INTER mode is used.

[0224] In this mode, for each reference picture list (list 0 or list 1), an affine AMVP candidate list is constructed with three types of affine motion predictors in the following order, where each candidate contains the estimated CPMV of the current block. The best CPMV found at the encoder side (such as Figure 20 The difference of mv0, mv1, and mv2 in the mv1 and estimated CPMV is signaled. In addition, the index of the affine AMVP candidate from which the estimated CPMV is derived is also signaled.

[0225] 1) Inherited affine motion predictor

[0226] The checking order is similar to that of spatial MVP in HEVC AMVP list construction. First, the left inherited affine motion predictor is derived from the first block in {A1, A0}, which is affine coded and has the same reference picture as the current block. Second, the above inherited affine motion predictor is derived from the first block in {B1, B0, B2}, which is affine coded and has the same reference picture as the current block. Figure 19 Five blocks A1, A0, B1, B0, B2 are shown in FIG.

[0227] Once it is found that the adjacent block is coded in affine mode, the CPMV of the codec covering the adjacent block is used to derive the predictor of the CPMV of the current block. For example, if A1 is coded in non-affine mode and A0 is coded in 4-parameter affine mode, the left-inherited affine MV predictor will be derived from A0. In this case, the CPMV of the CU covering A0 (in Figure 21B In the figure, the left top CPMV is represented by And the right top CPMV refers to ) is used to derive the estimated CPMV of the current block, with the top left (with coordinates (x0, y0)), top right (with coordinates (x1, y1)) and bottom right (with coordinates (x2, y2)) positions of the current block being denoted as

[0228] 2) Constructed affine motion predictor

[0229] The constructed affine motion predictor consists of control point motion vectors (CPMVs) derived from adjacent inter-frame codec blocks with the same reference picture, as Figure 20 If the current affine motion model is 4-parameter affine, the number of CPMVs is 2, otherwise if the current affine motion model is 6-parameter affine, the number of CPMVs is 3. Top left CPMV Derived from the MV at the first block in group {A, B, C}, which is inter-coded and has the same reference picture as the current block. Derived from the MV at the first block in group {D, E}, which is inter-coded and has the same reference picture as the current block. Left bottom CPMV Derived from the MV at the first block in the group {F, G}, which is inter-coded and has the same reference picture as the current block.

[0230] – If the current affine motion model is 4-parameter affine, only if and are found (i.e., and The constructed affine motion predictor is inserted into the candidate list only when it is used as the estimated CPMV of the left top (with coordinates (x0, y0)) and right top (with coordinates (x1, y1)) positions of the current block.

[0231] – If the current affine motion model is 6-parameter affine, only if and are found (i.e., and The constructed affine motion predictor is inserted into the candidate list only when it is used as the estimated CPMV of the left top (with coordinates (x0, y0) of the current block), right top (with coordinates (x1, y1)) and right bottom (with coordinates (x2, y2)) positions.

[0232] When inserting the constructed affine motion predictor into the candidate list, no pruning process is applied.

[0233] 3) Ordinary AMVP motion predictor

[0234] The following is applied until the number of affine motion predictors reaches a maximum.

[0235] 1) By setting all CPMVs equal to (If available) Derive an affine motion predictor.

[0236] 2) By setting all CPMVs equal to (If available) Derive an affine motion predictor.

[0237] 3) By setting all CPMV equal to (If available) Derive an affine motion predictor.

[0238] 4) Derive an affine motion predictor by setting all CPMVs equal to HEVC TMVP (if available).

[0239] 5) Derive an affine motion predictor by setting all CPMVs to zero MV.

[0240] Notice that It has been derived in the constructed affine motion predictor.

[0241] Figure 18AAn example of a 4-parameter affine model is shown. Figure 18B An example of a 6-parameter affine model is shown.

[0242] Figure 19 An example of MVP of AF_INTER inheriting affine candidates is shown.

[0243] Figure 20 An example of MVP of AF_INTER of constructed affine candidates is shown.

[0244] In AF_INTER mode, when using 4 / 6 parameter affine mode, 2 / 3 control points are required, and therefore 2 / 3 MVDs need to be encoded and decoded for these control points, as shown in Figure 18. In JVET-K0337, it is proposed to derive MVs as follows, i.e., predict mvd1 and mvd2 from mvd0.

[0245]

[0246] in mvd i and mv1 are the predicted motion vector, motion vector difference, and motion vector of the top left pixel (i=0), top right pixel (i=1), or bottom left pixel (i=2), respectively. Figure 18B Note that adding two motion vectors (e.g., mvA(xA, yA) and mvB(xB, yB)) is equal to the sum of the two separate components, i.e., newMV = mvA + mvB, and the two components of newMV are set to (xA + xB) and (yA + yB), respectively.

[0247] 2.3.3.3AF_MERGE Mode

[0248] When a CU is applied in AF_MERGE mode, it reconstructs the blocks from the valid neighborhood to get the first block encoded and decoded in affine mode. And the order of candidate blocks is from left, top, top right, bottom left to top left, as shown in Figure 21A As shown in (in order referred to as A, B, C, D, E). For example, if the neighborhood left bottom block is coded in affine mode (in Figure 21B The control point (CP) motion vector mv0 of the top left corner, top right corner, and bottom left corner of the adjacent CU / PU containing block A is retrieved. N 、mv1 N and or mv2 N . And the motion vector mv0 of the left top corner / right top / left bottom on the current CU / PU C 、mv1 C and mv2 C (It is only used for 6-parameter affine models) is based on mv0 N 、mv1 Nand mv2 N Calculation. In VTM-2.0, if the current block is affine-encoded, the subblock at the top left corner (e.g., a 4×4 block in VTM) stores mv0, and the subblock at the top right corner stores mv1. If the current block is affine-encoded, the subblock at the bottom left corner stores mv2; otherwise (using a 4-parameter affine model), LB stores mv2'. Other subblocks store the MV used for MC.

[0249] In deriving the CPMV mv0 of the current CU C 、mv1 C and mv2 C Afterwards, the MVF of the current CU is generated according to the simplified affine motion model equations (1) and (2). In order to identify whether the current CU is coded or decoded in AF_MERGE mode, the affine flag is signaled in the bitstream when there is at least one neighboring block coded or decoded in affine mode.

[0250] In JVET-L0142 and JVET-L0632, the following steps are used to construct the affine merge candidate list:

[0251] 1) Insert inheritance affine candidate

[0252] Inherited affine candidates are candidates derived from the affine motion model of their valid neighboring affine codec blocks. Up to two inherited affine candidates are derived from the affine motion model of the adjacent block and inserted into the candidate list. For the left predictor, the scan order is {A0, A1}; for the above predictor, the scan order is {B0, B1, B2}.

[0253] 2) Insert the constructed affine candidate

[0254] If the number of candidates in the affine Merge candidate list is less than MaxNumAffineCand (e.g., 5), the constructed affine candidate is inserted into the candidate list. The constructed affine candidate refers to a candidate constructed by combining the neighborhood motion information of each control point.

[0255] a) First, Figure 22 The motion information of the control point is derived using the specified spatial and temporal neighborhoods shown in [1]. CPk (k = 1, 2, 3, 4) represents the kth control point. A0, A1, A2, B0, B1, B2, and B3 are the spatial locations used to predict CPk (k = 1, 2, 3); T is the temporal location used to predict CP4.

[0256] The coordinates of CP1, CP2, CP3 and CP4 are (0, 0), (W, 0), (H, 0) and (W, H), respectively, where W and H are the width and height of the current block.

[0257] Get the motion information for each control point according to the following priority order:

[0258] For CP1, the priority is B2->B3->A2. If B2 is available, it is used. Otherwise, if B2 is available, B3 is used. If neither B2 nor B3 is available, A2 is used. If all three candidates are unavailable, motion information for CP1 cannot be obtained.

[0259] – For CP2, the checking priority is B1->B0.

[0260] – For CP3, the checking priority is A1->A0.

[0261] – For CP4, use T.

[0262] b) Secondly, the combination of control points is used to construct affine merge candidates.

[0263] I. Motion information of three control points is required to construct a 6-parameter affine candidate. Three control points can be selected from one of the following four combinations: ({CP1, CP2, CP4}, {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4}). The combination {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4} will be converted into a 6-parameter motion model represented by the top left, top right, and bottom left control points.

[0264] II. Motion information of two control points is required to construct a 4-parameter affine candidate. Two control points can be selected from one of two combinations: ({CP1, CP2}, {CP1, CP3}). Both combinations will be converted into a 4-parameter motion model represented by the left and right top control points.

[0265] III. The constructed combinations of affine candidates are inserted into the candidate list in the following order:

[0266] {CP1, CP2, CP3}, {CP1, CP2, CP4}, {CP1, CP3, CP4}, {CP2, CP3, CP4}, {CP1, CP2}, {CP1, CP3}

[0267] i. For each combination, check the reference index of List X for each CP. If they are all the same, then the combination has a valid CPMV for List X. If the combination does not have a valid CPMV for both List 0 and List 1, then the combination is marked as invalid. Otherwise, it is valid, and the CPMV is placed in the sub-block Merge list.

[0268] 3) Fill with zero motion vectors

[0269] If the number of candidates in the Affine Merge candidate list is less than 5, a zero motion vector with a zero reference index is inserted into the candidate list until the list is full.

[0270] More specifically, for the sub-block Merge candidate list, there are 4-parameter Merge candidates with MV set to (0, 0) and prediction directions set to unidirectional prediction (for P slices) and bidirectional prediction (for B slices) from list 0.

[0271] 2.3.4 Current Image Reference

[0272] Intra Block Copying (also known as IBC, or Intra Picture Block Compensation) (also known as Current Picture Reference (CPR)) is used in the HEVC Screen Content Codec Extension (SCC). This tool is very efficient for encoding and decoding screen content videos, because repetitive patterns in text and graphic-rich content frequently occur within the same picture. Using previously reconstructed blocks with equal or similar patterns as predictors can effectively reduce prediction errors and thus improve codec efficiency. An example of intra block compensation is shown in Figure 23 Middle picture.

[0273] Similar to the design of CRP in HEVC SCC, in VVC, the use of IBC mode is signaled at the sequence and picture levels. When IBC mode is enabled in the sequence parameter set (SPS), it can be enabled at the picture level. When IBC mode is enabled at the picture level, the currently reconstructed picture is treated as a reference picture. Therefore, no block-level syntax changes are required to signal the use of IBC mode based on the existing VVC inter-frame mode.

[0274] Key Features:

[0275] – It is considered as normal inter-frame mode. Therefore, Merge mode and Skip mode also affect IBC

[0276] Mode available. Merge candidate list construction is unified, including from IBC mode or HEVC

[0277] Merge candidates for neighboring locations coded in inter-frame mode. Depending on the selected merge index, the current block in merge mode or skip mode can be merged into a neighborhood coded in IBC mode or a neighborhood coded in other normal inter-frame modes with different pictures as reference pictures.

[0278] – The block vector prediction and codec scheme of IBC mode reuses the scheme used for motion vector prediction and the codec in HEVC inter mode (AMVP and MVD codecs).

[0279] – Motion vectors (also called block vectors) in IBC mode are encoded and decoded with integer pixel precision, but after decoding are stored in memory with 1 / 16 pixel precision because quarter pixel precision is used in the interpolation and deblocking stages. When used in motion vector prediction for IBC mode, the stored vector predictor is right-shifted by 4.

[0280] – Search range: It is limited to within the current CTU.

[0281] – CPR is disabled when affine mode / triangle mode / GBI / weighted prediction is enabled.

[0282] 2.3.5 Merge List Design in VVC

[0283] There are three different Merge list construction processes supported in VVC:

[0284] 1) Sub-block Merge Candidate List: This list contains both ATMVP and affine merge candidates. The merge list construction process is shared by both affine and ATMVP modes. ATMVP and affine merge candidates can be added sequentially. The sub-block merge list size is signaled in the slice header and has a maximum value of 5.

[0285] 2) Unidirectional Prediction TPM Merge List: For triangle prediction mode, a merge list construction process is shared for both partitions, even though the two partitions can select their own merge candidate indices. When building the merge list, the spatial neighbors of the block and the two temporal blocks are checked. The motion information derived from the spatial neighbors and the temporal blocks is called regular motion candidates in our IDF. These regular motion candidates are further used to derive multiple TPM candidates. Note that the transformation is performed at the whole block level, even though the two partitions can use different motion vectors to generate their own prediction blocks.

[0286] The one-way prediction TPM Merge list size is fixed to 5.

[0287] 3) Regular Merge List: For the remaining codec blocks, a single Merge list construction process is shared. Here, spatial / temporal / HMVP, pairwise combined bi-prediction Merge candidates, and zero motion candidates can be inserted in sequence. The regular Merge list size is signaled in the slice header and has a maximum value of 6.

[0288] 2.3.5.1 Sub-block Merge Candidate List

[0289] It is proposed to put all sub-blocks related to motion candidates into a separate Merge list in addition to the regular Merge list for non-sub-block Merge candidates.

[0290] The motion candidates associated with the sub-blocks are placed into a separate Merge list named 'Sub-block Merge Candidate List'.

[0291] In one example, the sub-block Merge candidate list includes affine Merge candidates, and ATMVP candidates, and / or sub-block-based STMVP candidates.

[0292] 2.3.5.1.1JVET-L0278

[0293] In this contribution, the ATMVP Merge candidate in the normal Merge list is moved to the first position of the affine Merge list, so that all Merge candidates in the new list (i.e., the sub-block-based Merge candidate list) are based on sub-block codec tools.

[0294] 2.3.5.1.2 ATMVP in VTM-3.0

[0295] In VTM-3.0, in addition to the regular merge candidate list, a special merge candidate list known as the sub-block merge candidate list (also known as the affine merge candidate list) is added. The sub-block merge candidate list is filled with candidates in the following order:

[0296] a. ATMVP candidate (may be available or not);

[0297] b. Inheritance affine candidate;

[0298] c. Construct affine candidates;

[0299] d. Filling as zero MV 4-parameter affine model

[0300] The maximum number of candidates in the sub-block Merge candidate list (referred to as ML) is derived as follows:

[0301] 1) If ATMVP uses a logo (e.g. the logo can be named

[0302] “sps_sbtmvp_enabled_flag”) is turned on (equal to 1), but the affine usage flag (for example, the flag can be named “sps_affine_enabled_flag”) is turned off (equal to 0), then ML is set to equal to 1.

[0303] 2) If the ATMVP use flag is off (equal to 0) and the affine use flag is off (equal to 0), then ML is set to be equal to 0. In this case, the sub-block Merge candidate list is not used.

[0304] 3) Otherwise (affine use flag is on (equal to 1), ATMVP use flag is on or off), ML is signaled from the encoder to the decoder. Valid ML is 0<=ML<=5.

[0305] When building the sub-block Merge candidate list, the ATMVP candidate is checked first. If any of the following conditions is true, the ATMVP candidate is skipped and not placed in the sub-block Merge candidate list.

[0306] 1) ATMVP use flag is turned off;

[0307] 2) Any TMVP usage flag (e.g. when signaled at the slice level, the flag may be named “slice_temporal_mvp_enabled_flag”) is turned off;

[0308] 3) The reference picture with reference index 0 in reference list 0 is the same as the current picture (which is CPR)

[0309] ATMVP in VTM-3.0 is much simpler than in JEM. When generating ATMVP Merge candidates, the following process is applied:

[0310] a. Check adjacent blocks A1, B1, B0, A0 in order, such as Figure 22 As shown, the first inter-frame codec but not CPR-coded block is found, denoted as block X;

[0311] b. Initialize TMV=(0,0). If there is an MV (denoted as MV') for block X, referring to a collocated reference picture (as signaled in the slice header), then TMV is set equal to MV'.

[0312] c. Assuming the center point of the current block is (x0, y0), locate the corresponding position of (x0, y0) in the co-located picture as M = (x0 + MV'x, y0 + MV'y). Find the block Z that covers M.

[0313] i. If Z is intra-coded, ATMVP is not available;

[0314] ii. If Z is inter-coded, MVZ_0 and MVZ_1 of the two lists of block Z are scaled to (Ref list 0 index 0) and (Ref list 1 index 0) as MVdefault0, MVdefault1, and stored.

[0315] d. For each 8x8 sub-block, assuming its center point is (x0S, y0S), locate the corresponding position of (x0S, y0S) in the co-located picture as MS = (x0S + MV'x, y0S + MV'y). Find the block ZS covering MS.

[0316] i. If ZS is intra-frame coded or decoded, MVdefault0 and MVdefault1 are assigned to the sub-block;

[0317] ii. If ZS is inter-coded, MVZS_0 and MVZS_1 of the two lists of block ZS are scaled to (Ref list 0 index 0) and (Ref list 1 index 0) and distributed to the sub-blocks;

[0318] MV cropping and masking in ATMVP:

[0319] When locating a corresponding position such as M or MS in a co-located picture, it is cropped within a predetermined area. The CTU size is S×S, and S=128 in VTM-3.0. Assuming that the top left position of the co-located CTU is (xCTU, yCTU), the corresponding position M or MS at (xN, yN) will be cropped to the valid area xCTU<=xN <xCTU+S+4;yCTU<=yN<yCTU+S。

[0320] In addition to clipping, (xN, yN) is also masked as xN=xN&MASK, yN=yN&MASK, where MASK is equal to ~(2 N -1), and N=3 to set the lowest 3 bits to 0. Therefore, xN and yN are numbers that are multiples of 8 ("~" represents a bitwise compensation operator).

[0321] Figure 24 An example of valid corresponding areas in a co-located picture is shown.

[0322] 2.3.5.1.3 Syntax Design in the Header

[0323]

[0324]

[0325] 2.3.5.2 Conventional Merge List

[0326] Different from the Merge list design, in VVC, a history-based motion vector prediction (HMVP) method is adopted.

[0327] In HMVP, the motion information of the previously coded blocks is stored. The motion information of the previously coded blocks is defined as HMVP candidates. Multiple HMVP candidates are stored in a table named HMVP, and the table is maintained in real time during the encoding / decoding process. The HMVP table is cleared when a new slice is encoded / decoded. As long as there is an inter-frame coded block, the associated motion information is added to the last entry of the table as a new HMVP candidate. The overall encoding and decoding process is as follows: Figure 25 Shown in.

[0328] HMVP candidates can be used in both AMVP and Merge candidate list construction processes. Figure 26 Illustration of the modified Merge candidate list construction process (using Figure 26 (as shown by the highlighted box in ). When the Merge candidate list is not full after the TMVP candidate is inserted, the HMVP candidates stored in the HMVP table can be used to fill in the Merge candidate list. Taking into account that a block usually has a higher correlation with the closest adjacent block in terms of motion information, the HMVP candidates in the table are inserted in descending order of index. The last entry in the table is added to the list first, and the first entry is added last. Similarly, redundancy removal is applied to the HMVP candidates. Once the total number of available Merge candidates reaches the maximum number of Merge candidates allowed for signaling, the Merge candidate list construction process is terminated.

[0329] 2.4MV rounded up

[0330] In VVC, when MV is right-shifted, it is required to round towards zero. In a formal way, for an MV (MVx, MVy) to be right-shifted by N bits, the result MV' (MVx', MVy') will be derived as:

[0331] MVx'=(MVx+((1<<N)> >1)-(MVx>=0?1:0))>>N;

[0332] MVy'=(MVy+((1<<N)> >1)-(MVy>=0?1:0))>>N;

[0333] 2.5 RPR in JVET-O2001-v14

[0334] ARC, also known as RPR (reference picture resampling), was integrated into JVET-O2001-v14.

[0335] In the case of RPR in JVET-O2001-v14, TMVP is disabled if the collocated picture has a different resolution than the current picture. In addition, BDOF and DMVR are disabled when the reference picture has a different resolution than the current picture.

[0336] To handle normal MC when the reference picture has a different resolution than the current picture, the interpolation segment is defined as follows:

[0337] 8.5.6.3 Fractional Sample Interpolation Process

[0338] 8.5.6.3.1 Overall

[0339] The inputs to this process are:

[0340] – Luma position (xSb, ySb), specifies the top left sample of the current codec sub-block relative to the top left luma sample of the current picture,

[0341] –Variable sbWidth specifies the width of the current codec sub-block.

[0342] –Variable sbHeight specifies the height of the current codec sub-block.

[0343] – motion vector offset mvOffset,

[0344] – Refine the motion vector refMvLX,

[0345] – the selected reference picture sample array refPicLX,

[0346] – Half-sample interpolation filter index hpelIfIdx,

[0347] – bidirectional optical flow flag bdofFlag,

[0348] –Variable cIdx specifies the color component index of the current block.

[0349] The output of this process is:

[0350] – predSamplesLX, an array of (sbWidth+brdExtSize)x(sbHeight+brdExtSize) predicted sample values.

[0351] The prediction block bounds extension size brdExtSize is derived as follows:

[0352] brdExtSize=(bdofFlag||(inter_affine_flag[xSb][ySb]&&sps_affine_prof_enabled_flag))? 2:0 (8-752)

[0353] The variable fRefWidth is set equal to the PicOutputWidthL of the reference picture in luma samples.

[0354] The variable fRefHeight is set equal to the PicOutputHeightL of the reference picture in luma samples.

[0355] The motion vector mvLX is set to (refMvLX-mvOffset).

[0356] – If cIdx is equal to 0, the following applies:

[0357] – The scaling factors and their fixed-point representations are defined as

[0358] hori_scale_fp=((fRefWidth<<14)+(PicOutputWidthL>>1)) / PicOutputWidthL(8-753)

[0359] vert_scale_fp=((fRefHeight<<14)+(PicOutputHeightL>>1)) / PicOutputHeightL (8-754)

[0360] – Let (xIntL, yIntL) be the luma position given in whole sample units, and (xFracL, yFracL) be the offset given in 1 / 16 sample units. These variables are used only in this clause to specify fractional sample positions within the reference sample array refPicLX.

[0361] – The left top coordinate of the boundary block filled with reference samples (xSbInt L ,ySbInt L ) is set to (xSb+(mvLX[0]>>4), ySb+(mvLX[1]>>4)).

[0362] – For each luma sample position (x L =0..sbWidth-1+brdExtSize,y L =0..sbHeight-1+brdExtSize), the corresponding predicted brightness sample value predSamplesLX[x L ][y L ] is derived as follows:

[0363] – Make(refxSb L ,refySb L ) and (refx L , refy L) is the luminance position pointed to by the motion vector (refMvLX[0], refMvLX[1]) given in 1 / 16 sample units. L 、refx L 、refySb L and refy L The derivation is as follows:

[0364] refxSb L =((xSb<<4)+refMvLX[0])*hori_scale_fp (8-755)

[0365] refx L =((Sign(refxSb)*((Abs(refxSb)+128)>>8)+x L *((hori_scale_fp+8)>>4))+32)>>6 (8-756)

[0366] refySb L =((ySb<<4)+refMvLX[1])*vert_scale_fp (8-757)

[0367] refyL=((Sign(refySb)*((Abs(refySb)+128)>>8)+yL*((vert_scale_fp+8)>>4))+32)>>6 (8-758)

[0368] –Variable xInt L 、yInt L 、xFrac L and yFrac L The derivation is as follows:

[0369] xInt L =refx L >>4 (8-759)

[0370] yInt L =refy L >>4 (8-760)

[0371] xFrac L =refx L &15 (8-761)

[0372] yFrac L =refy L &15 (8-762)

[0373] – If bdofFlag is equal to true or (sps_affine_prof_enabled_flag is equal to true and inter_affine_flag[xSb][ySb] is equal to true), and one or more of the following conditions are true, then the luma integer sample retrieval process is called as specified in clause 8.5.6.3.3 with (xInt L +(xFrac L >>3)-1), yInt L +(yFrac L >>3)-1) and refPicLX as input, derive the predicted brightness sample value predSamplesLX[x L ][y L ].

[0374] 1.x L Equal to 0.

[0375] 2.x L Equal to sbWidth+1.

[0376] 3.y L Equal to 0.

[0377] 4.y L Equal to sbHeight+1.

[0378] – Otherwise, the 8-tap interpolation filter process for luma samples is called by invoking the 8-tap interpolation filter process for luma samples as specified in clause 8.5.6.3.2 with (xIntL-(brdExtSize>0?1:0), yIntL-(brdExtSize>0?1:0)), (xFracL, yFracL), (xSbInt L ,ySbInt L ), refPicLX, hpelIfIdx, sbWidth, sbHeight and (xSb, ySb) are used as input to derive the predicted luminance sample value predSamplesLX[xL][yL].

[0379] – Otherwise (cIdx is not equal to 0), the following applies:

[0380] – Let (xIntC, yIntC) be the chroma position given in whole sample units, and (xFracC, yFracC) be the offset given in 1 / 32 sample units. These variables are used only in this clause to specify the overall fractional sample position within the reference sample array refPicLX.

[0381] – The top left coordinate (xSbIntC, ySbIntC) of the reference sample filled boundary block is set to ((xSb / SubWidthC)+(mvLX[0]>>5),(ySb / SubHeightC)+(mvLX[1]>>5)).

[0382] For each chroma sample position (xC=0..sbWidth-1, yC=0..sbHeight-1) within the predicted chroma sample array predSamplesLX, the corresponding predicted chroma sample value predSamplesLX[xC][yC] is derived as follows:

[0383] – Make(refxSb C ,refySb C ) and (refx C , refy C ) is the chroma position pointed to by the motion vector (mvLX[0], mvLX[1]) given in 1 / 32 sample units. The variable refxSb C 、refySb C 、refx C and refy C The derivation is as follows:

[0384] refxSb C =((xSb / SubWidthC<<5)+mvLX[0])*hori_scale_fp

[0385] (8-763)

[0386] refx C =((Sign(refxSb C )*((Abs(refxSb C )+256)>>9)

[0387] +xC*((hori_scale_fp+8)>>4))+16)>>5 (8-764)

[0388] refySb C =((ySb / SubHeightC<<5)+mvLX[1])*vert_scale_fp

[0389] (8-765)

[0390] refy C =((Sign(refySb C )*((Abs(refySb C)+256)>>9)+yC*((vert_scale_fp+8)>>4))+16)>>5 (8-766)

[0391] –Variable xInt C 、yInt C 、xFrac C and yFrac C The derivation is as follows:

[0392] xInt C =refx C >>5 (8-767)

[0393] yInt C =refy C >>5 (8-768)

[0394] xFrac C =refy C &31 (8-769)

[0395] yFrac C =refy C &31 (8-770)

[0396] – Derives the predicted sample values ​​predSamplesLX[xC][yC] by calling the procedure specified in clause 8.5.6.3.4 with (xIntC, yIntC), (xFracC, yFracC), (xSbIntC, ySbIntC), sbWidth, sbHeight and refPicLX as input.

[0397] 8.5.6.3.2 Luma Sample Interpolation Filtering Process

[0398] The inputs to this process are:

[0399] – Luminance position in units of whole samples (xInt L , yInt L ),

[0400] – Luminance position in fractional sample units (xFrac L ,yFrac L ),

[0401] – Luminance position in units of whole samples (xSbInt L ,ySbInt L ), specifies the top left sample of the boundary block filled with reference samples relative to the top left luma sample of the reference picture,

[0402] – Luminance reference sample array refPicLXL ,

[0403] – Half-sample interpolation filter index hpelIfIdx,

[0404] –Variable sbWidth specifies the width of the current sub-block,

[0405] –Variable sbHeight specifies the height of the current sub-block.

[0406] – Luma position (xSb, ySb), specifies the top left sample of the current sub-block relative to the top left luma sample of the current picture,

[0407] The output of this process is the predicted luminance sample value predSampleLX L

[0408] The variables shift1, shift2, and shift3 are derived as follows:

[0409] – The variable shift1 is set equal to Min(4, BitDepth Y -8), the variable shift2 is set equal to 6, and the variable shift3 is set equal to Max(2,14-BitDepth Y ).

[0410] – The variable picW is set equal to pic_width_in_luma_samples, and the variable picH is set equal to pic_height_in_luma_samples.

[0411] Equal to xFrac L or yFrac L Luma interpolation filter coefficient f for each 1 / 16 fractional sample position p L [p]The derivation is as follows:

[0412] – If MotionModelIdc[xSb][ySb] is greater than 0, and sbWidth and sbHeight are both equal to 4, then the brightness interpolation filter coefficient f L [p] is specified in Table 8-12.

[0413] – Otherwise, the brightness interpolation filter coefficient f L [p] Depends on hpelIfIdx and is specified in Table 8-11.

[0414] Luminance position in units of whole samples (xInt i , yInt i ) for i=0..7 is derived as follows:

[0415] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0416] xInt i =Clip3(SubPicLeftBoundaryPos, SubPicRightBoundaryPos, xInt L +i-3)(8-771)

[0417] yInt i =Clip3(SubPicTopBoundaryPos, SubPicBotBoundaryPos, yInt L +i-3)(8-772)

[0418] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0419] xInt i =Clip3(0,picW-1,sps_ref_wraparound_enabled_flag?

[0420] ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY, picW, xInt L +i-3): (8-773)

[0421] xInt L +i-3)

[0422] yInt i =Clip3(0,picH-1,yInt L +i-3) (8-774)

[0423] The luminance position in integer sample units is further modified for i=0..7 as follows:

[0424] xInt i =Clip3(xSbInt L -3,xSbInt L +sbWidth+4,xInt i ) (8-775)

[0425] yInt i =Clip3(ySbInt L -3,ySbInt L+sbHeight+4,yInt i ) (8-776)

[0426] Predicted brightness sample value predSampleLX L is derived as follows:

[0427] – If xFrac L and yFrac L are all equal to 0, predSampleLX L The value of is derived as follows:

[0428] predSampleLX L =refPicLX L [xInt3][yInt3]< <shift3 (8-777)

[0429] – Otherwise, if xFrac L Not equal to 0 and yFrac L If equal to 0, then predSampleLX L The value of is derived as follows:

[0430]

[0431] – Otherwise, if xFrac L Equal to 0 and yFrac L If not equal to 0, then predSampleLX L The value of is derived as follows:

[0432]

[0433] – Otherwise, if xFrac L Not equal to 0 and yFrac L If not equal to 0, then predSampleLX L The value of is derived as follows:

[0434] – The sample array temp[n], where n=0..7, is derived as follows:

[0435]

[0436] –Predicted brightness sample value predSampleLX L is derived as follows:

[0437]

[0438] Table 8-11 – Luma interpolation filter coefficients f for each 1 / 16 fractional sample position p L[p] specification.

[0439]

[0440]

[0441] Table 8-12 – Luma interpolation filter coefficients f for each 1 / 16 fractional sample position p for affine motion mode L [p] specification.

[0442]

[0443] 8.5.6.3.3 Luminance integer sample retrieval process

[0444] The inputs to this process are:

[0445] – Luminance position in units of whole samples (xInt L , yInt L ),

[0446] – Luminance reference sample array refPicLX L ,

[0447] The output of this process is the predicted luminance sample value predSampleLX L

[0448] The variable Offset is set equal to Max(2,14-BitDepth Y ).

[0449] The variable picW is set equal to pic_width_in_luma_samples, and the variable picH is set equal to pic_height_in_luma_samples.

[0450] The luma position (xInt, yInt) in integer sample units is derived as follows:

[0451] xInt=Clip3(0,picW-1,sps_ref_wraparound_enabled_flag?

[0452] (8-782)

[0453] ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY, picW, xInt L ): xInt L )

[0454] yInt=Clip3(0,picH-1,yInt L ) (8-783)

[0455] Predicted brightness sample value predSampleLX L is derived as follows:

[0456] predSampleLX L =refPicLX L [xInt][yInt]< <shift3 (8-784)

[0457] 8.5.6.3.4 Chroma sample interpolation process

[0458] The inputs to this process are:

[0459] – Chroma position in units of integer samples (xInt C , yInt C ),

[0460] – Chroma position in 1 / 32 fractional sample units (xFrac C ,yFrac C ),

[0461] – The chroma position (xSbIntC, ySbIntC) in integer sample units specifies the top left sample of the reference sample filled boundary block relative to the top left chroma sample of the reference picture,

[0462] –Variable sbWidth specifies the width of the current sub-block,

[0463] –Variable sbHeight specifies the height of the current sub-block.

[0464] – Chroma reference sample array refPicLX C .

[0465] The output of this process is the predicted chrominance sample value predSampleLX C

[0466] The variables shift1, shift2, and shift3 are derived as follows:

[0467] – The variable shift1 is set equal to Min(4, BitDepth C -8), the variable shift2 is set equal to 6, and the variable shift3 is set equal to Max(2,14-BitDepth C ).

[0468] –Variable picW Cis set equal to pic_width_in_luma_samples / SubWidthC, and the variable picH C Set equal to pic_height_in_luma_samples / SubHeightC.

[0469] Equal to xFrac C or yFrac C The chroma interpolation filter coefficient f for each 1 / 32 fractional sample position p C [p] is specified in Table 8-13.

[0470] The variable xOffset is set to (sps_ref_wraparound_offset_minus1+1)*MinCbSizeY) / SubWidthC.

[0471] Chroma position in units of integer samples (xInt i , yInt i ) for i = 0..3 is derived as follows:

[0472] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0473] xInt i =Clip3(SubPicLeftBoundaryPos / SubWidthC, SubPicRightBoundaryPos / SubWidthC, xInt L +i) (8-785)

[0474] yInt i =Clip3(SubPicTopBoundaryPos / SubHeightC, SubPicBotBoundaryPos / SubHeightC, yInt L +i) (8-786)

[0475] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0476] xInt i =Clip3(0, picW C -1, sps_ref_wraparound_enabled_flag? ClipH(xOffset,picW C , xInt C+i-1):(8-787)

[0477] xInt C +i-1)

[0478] yInt i =Clip3(0,picH C -1, yInt C +i-1) (8-788)

[0479] Chroma position in units of integer samples (xInt i , yInt i ) is further modified for i=0..3 as follows:

[0480] xInt i =Clip3(xSbIntC-1,xSbIntC+sbWidth+2,xInt i ) (8-789)

[0481] yInt i =Clip3(ySbIntC-1,ySbIntC+sbHeight+2,yInt i ) (8-790)

[0482] Predicted chroma sample value predSampleLX C is derived as follows:

[0483] – If xFrac C and yFrac C If both are equal to 0, then predSampleLX C The value of is derived as follows:

[0484] predSampleLX C =refPicLX C [xInt1][yInt1]< <shift3 (8-791)

[0485] – Otherwise, if xFrac C Not equal to 0 and yFrac C If equal to 0, then predSampleLX C The value of is derived as follows:

[0486]

[0487] – Otherwise, if xFrac C Equal to 0 and yFrac C If not equal to 0, then predSampleLX C The value of is derived as follows:

[0488]

[0489] – Otherwise, if xFrac C Not equal to 0 and yFrac C If not equal to 0, then predSampleLX C The value of is derived as follows:

[0490] – The sample array temp[n], where n=0..3, is derived as follows:

[0491]

[0492] –Predicted chroma sample value predSampleLX C is derived as follows:

[0493]

[0494]

[0495] Table 8-13 – Chroma interpolation filter coefficients f for each 1 / 32 fractional sample position p C [p] specification.

[0496]

[0497]

[0498] 2.6 Sub-images in JVET-O2001-v14

[0499] With the current syntax design of sub-pictures in JVET-O2001-vE, the positions and dimensions of sub-pictures are derived as follows:

[0500]

[0501] subpics_present_flag equal to 1 indicates that sub-picture parameters are currently present in the SPS RBSP syntax. subpics_present_flag equal to 0 indicates that sub-picture parameters are currently not present in the SPS RBSP syntax.

[0502] When the bitstream is the result of a sub-bitstream extraction process and contains only a subset of sub-pictures of the input bitstream to the sub-bitstream extraction process, it may be required to set the value of subpics_present_flag equal to 1 in the RBSP of the SPS.

[0503] max_subpics_minus1 plus 1 specifies the maximum number of sub-pictures that can exist in the CVS. max_subpics_minus1 should be in the range 0 to 254. The value 255 is reserved for future use by ITU-T | ISO / IEC.

[0504] subpic_grid_col_width_minus1 plus 1 specifies the width in units of 4 samples of each element of the sub-picture identifier grid. The length of the syntax element is Ceil(Log2(pic_width_max_in_luma_samples / 4)) bits.

[0505] The variable NumSubPicGridCols is derived as follows:

[0506] NumSubPicGridCols=(pic_width_max_in_luma_samples+subpic_grid_col_width_minus1*4+3) /

[0507] (subpic_grid_col_width_minus1*4+4) (7-5)

[0508] subpic_grid_row_height_minus1 plus 1 specifies the height in units of 4 samples of each element of the sub-picture identifier grid. The length of the syntax element is Ceil(Log2(pic_height_max_in_luma_samples / 4)) bits.

[0509] The variable NumSubPicGridRows is derived as follows:

[0510] NumSubPicGridRows=(pic_height_max_in_luma_samples+subpic_grid_row_height_minus1*4+3) /

[0511] (subpic_grid_row_height_minus1*4+4) (7-6)

[0512] subpic_grid_idx[i][j] specifies the sub-picture index of the grid position (i, j). The length of the syntax element is Ceil(Log2(max_subpics_minus1+1)) bits.

[0513] The variables SubPicTop[subpic_grid_idx[i][j]], SubPicLeft[subpic_grid_idx[i][j]], SubPicWidth[subpic_grid_idx[i][j]], SubPicHeight[subpic_grid_idx[i][j]], and NumSubPics are derived as follows:

[0514]

[0515]

[0516] subpic_treated_as_pic_flag[i] equal to 1 specifies that the i-th sub-picture of each codec picture in the CVS is treated as a picture in the decoding process excluding in-loop filtering operations. subpic_treated_as_pic_flag[i] equal to 0 specifies that the i-th sub-picture of each codec picture in the CVS is not treated as a picture in the decoding process excluding in-loop filtering operations. When not present, the value of subpic_treated_as_pic_flag[i] is inferred to be equal to 0.

[0517] 2.7 CIIP in JVET-O2001-v14

[0518] Combined Inter-Intra Prediction (CIIP) is used in VVC as a special Merge candidate. It can only be enabled for WxH blocks (W<=64 and H<=64).

[0519] 3. Examples of Problems Solved by the Disclosed Technologies and Embodiments

[0520] In the current design of VVC, ATMVP has the following problems:

[0521] 1) Whether the application of ATMVP at the stripe level and at the CU level does not match;

[0522] 2) In the slice header, ATMVP can be enabled even if TMVP is disabled. At the same time, the ATMVP flag is signaled before the TMVP flag.

[0523] 3) Masking is always performed regardless of whether the MV is compressed;

[0524] 4) The effective corresponding area may be too large;

[0525] 5) The derivation of TMV is too complicated;

[0526] 6) ATMVP may not be available in some cases, and a better default MV is desired.

[0527] 7) The MV scaling method in ATMVP may not be efficient;

[0528] 8) ATMVP should consider CPR;

[0529] 9) Even when affine prediction is disabled, default zero affine Merge candidates can be put into the list.

[0530] 10) The current picture is considered a long-term reference picture, while other pictures are considered short-term reference pictures. For both ATMVP and TMVP candidates, the motion information from the temporal blocks in the co-located picture will be scaled to a reference picture with a fixed reference index (i.e., 0 for each reference picture list in the current design). However, when CPR mode is enabled, the current picture is also considered a reference picture and can be added to reference picture list 0 (RefPicList0) with index equal to 0.

[0531] a. For TMVP, if the time domain block is coded in CPR mode and the reference picture of RefPicList0 is a short-term reference picture, the TMVP candidate is set to unavailable.

[0532] b. If the reference picture of RefPicList0 with index equal to 0 is the current picture and the current picture is an intra random access point (IRAP) picture, the ATMVP candidate is set to unavailable.

[0533] c. For the ATMVP sub-block within a block, when the motion information of the sub-block is derived from the time domain block, if the time domain block is encoded and decoded in CPR mode, the default ATMVP candidate (derived from the time domain block identified by the starting TMV and the center position of the current block) is used to fill in the motion information of the sub-block.

[0534] 11) MV is right-shifted to integer precision, but does not follow the rounding rules in VVC.

[0535] 12) The MVs (MVx, MVy) used in ATMVP to locate corresponding blocks in different pictures (e.g., TMV in 2.3.5.1.2) are used directly because they refer to the collocated picture. This is based on the assumption that all pictures have the same resolution. However, when RPR is enabled, different picture resolutions may be used. Similar issues exist for identifying corresponding blocks in the collocated picture for deriving sub-block motion information.

[0536] 13) If a block width or height is greater than 32 and the maximum transform unit size is 32, then for CIIP codec blocks, intra prediction signaling is generated at the codec unit (CU) size; while inter prediction signaling is generated at the transform unit (TU) size (recursively partitioning the current block into multiple 32x32 blocks). Using the CU to derive intra prediction signaling results in lower efficiency.

[0537] There are some problems with the current design. First, if the reference picture with RefPicList0 index equal to 0 is the current picture and the current picture is not an IRAP picture, the ATMVP process is still called, but the ATMVP process cannot locate any available ATMVP candidates because no temporal motion vectors can be scaled to the current picture.

[0538] 4. Examples of Embodiments and Technologies

[0539] The following list of techniques and embodiments should be considered as examples to explain the overall concept. These techniques should not be understood in a narrow sense. In addition, these techniques can be combined in any manner in encoder or decoder embodiments.

[0540] 1. Whether TMVP is allowed and / or whether CPR is used should be considered to decide / parse the maximum number of candidates in the sub-block merge candidate list and / or decide whether ATMVP should be used

[0541] The candidate is added to the candidate list. The maximum number of candidates in the sub-block merge candidate list is referred to as ML. Although some exemplary embodiments are discussed below using alternative temporal motion vector prediction (ATMVP), in alternative embodiments, the techniques disclosed in this document are applicable to sub-block based temporal motion vector prediction (sbTMVP).

[0542] a) In one example, when determining or parsing the maximum number of candidates in the sub-block Merge candidate list, if the ATMVP usage flag is off (equal to 0) or TMVP is disabled, ATMVP is inferred to be not applicable.

[0543] i. In one example, the ATMVP usage flag is turned on (equal to 1) and TMVP is disabled, and the ATMVP candidate is not added to the sub-block Merge candidate list or the ATMVP candidate list.

[0544] ii. In one example, the ATMVP use flag is on (equal to 1) and the TMVP is disabled, and the Affine use flag is off (equal to 0), then ML

[0545] Set to 0, which means that the sub-block merge is not applicable.

[0546] iii. In one example, the ATMVP use flag is on (equal to 1) and TMVP is enabled, and the affine use flag is off (equal to 0), then ML

[0547] is set equal to 1.

[0548] b) In one example, when deciding or parsing the maximum number of candidates in the sub-block Merge candidate list, if the ATMVP usage flag is off (equal to 0) or the co-located reference picture of the current picture is the current picture itself, ATMVP is inferred to be not applicable.

[0549] i. In one example, when the ATMVP usage flag is turned on (equal to 1) and the co-located reference picture of the current picture is the current picture itself, the ATMVP candidate is not added to the sub-block Merge candidate list or the ATMVP candidate list.

[0550] ii. In one example, the ATMVP usage flag is turned on (equal to 1), and the co-located reference picture of the current picture is the current picture itself, and the affine usage flag is turned off (equal to 0), then ML is set to equal to 0, meaning that sub-block Merge is not applicable.

[0551] iii. In one example, the ATMVP usage flag is on (equal to 1), and the co-located reference picture of the current picture is not the current picture itself, and the affine usage flag is off (equal to 0), then ML is set equal to 1.

[0552] c) In one example, when determining or parsing the maximum number of candidates in the sub-block Merge candidate list, if the ATMVP usage flag is off (equal to 0), or the reference picture with reference picture index 0 in reference list 0 is the current picture itself, then ATMVP is inferred to be not applicable.

[0553] i. In one example, the ATMVP usage flag is turned on (equal to 1) and the reference picture co-located with reference picture index 0 in reference list 0 is the current picture itself, the ATMVP candidate is not added to the sub-block Merge candidate list or the ATMVP candidate list.

[0554] ii. In one example, the ATMVP usage flag is turned on (equal to 1), and the reference picture with reference picture index 0 in reference list 0 is the current picture itself, and the affine usage flag is turned off (equal to 0), then ML is set to equal to 0, meaning that sub-block Merge is not applicable.

[0555] iii. In one example, the ATMVP use flag is on (equal to 1) and the reference picture with reference picture index 0 in reference list 0 is not the current picture itself, and the affine use flag is off (equal to 0), then ML is set equal to 1.

[0556] d) In one example, when determining or parsing the maximum number of candidates in the sub-block Merge candidate list, if the ATMVP usage flag is off (equal to 0), or the reference picture with reference picture index 0 in reference list 1 is the current picture itself, then ATMVP is inferred to be not applicable.

[0557] i. In one example, the ATMVP usage flag is turned on (equal to 1) and the reference picture co-located with reference picture index 0 in reference list 1 is the current picture itself, the ATMVP candidate is not added to the sub-block Merge candidate list or the ATMVP candidate list.

[0558] ii. In one example, the ATMVP usage flag is turned on (equal to 1), and the reference picture with reference picture index 0 in reference list 1 is the current picture itself, and the affine usage flag is turned off (equal to 0), then ML is set to equal to 0, meaning that sub-block Merge is not applicable.

[0559] iii. In one example, the ATMVP use flag is on (equal to 1) and the reference picture with reference picture index 0 in reference list 1 is not the current picture itself, and the affine use flag is off (equal to 0), then ML is set equal to 1.

[0560] 2. It is proposed that if TMVP is disabled at the slice / slice / picture level, ATMVP is implicitly disabled and the ATMVP flag is not signaled.

[0561] a) In one example, the ATMVP flag is signaled in the slice header / slice header / PPS after the TMVP flag.

[0562] b) In one example, the ATMVP or / and TMVP flag may not be signaled in the slice header / slice header / PPS, and only signaled in the SPS header.

[0563] 3. Whether and how to mask the corresponding position in ATMVP depends on whether and how to compress MV. Assume that (xN, yN) is the corresponding position calculated using the coordinates of the current block / subblock and the starting motion vector (e.g., TMV) in the co-located picture

[0564] a) In one example, if MV is not required to be compressed (for example, sps_disable_motioncompression signaled in SPS is 1), then (xN, yN)

[0565] Not masked; otherwise, (MV can be compressed) (xN, yN) is masked as xN = xN & MASK, yN = yN & MASK, where MASK is equal to ~ (2 M -1), and M may be an integer such as 3 or 4.

[0566] b) Assume that the MV compression method used for MV storage results in each 2 K x 2 K The blocks share the same motion information, and the mask in the ATMVP process is defined as ~(2 M -1). It is proposed that K may not be equal to M, for example, M=K+1.

[0567] c) The MASKs used in ATMVP and TMVP may be the same, or they may be different.

[0568] 4. In one example, the MV compression method can be flexible.

[0569] a) In one example, the MV compression method can be selected between no compression, 8×8 compression (M=3 in point 3.a), or 16×16 compression (M=4 in point 3.a)

[0570] b) In one example, the MV compression method can be signaled in the VPS / SPS / PPS / slice header / slice group header.

[0571] c) In one example, the MV compression method can be used in different standard profiles

[0572] / level / tier is set differently.

[0573] 5. The effective corresponding area in ATMVP can be adaptive;

[0574] a) For example, the valid corresponding area may depend on the width and height of the current block;

[0575] b) For example, the valid corresponding area may depend on the MV compression method;

[0576] i. In one example, if the MV compression method is not used, the effective corresponding area is smaller; if the MV compression method is used, the effective corresponding area is larger.

[0577] 6. The valid corresponding area in ATMVP can be based on a basic area with a size of M×N that is smaller than the CTU area. For example, the CTU size in VTM-3.0 is 128×128, and the basic area size can be 64×64. Assume that the width and height of the current block are W and H.

[0578] a) In one example, if W<=M and H<=N, meaning the current block is within the base region, then the valid corresponding regions in ATMVP are the extensions in the co-located base region and the co-located picture. Figure 27 An example is shown.

[0579] i. For example, assuming the top left position of the co-located basic area is (xBR, yBR), the corresponding position at (xN, yN) will be cropped to the valid area xBR <=

[0580] xN <xBR+M+4;yBR<=yN<yBR+N。

[0581] Figure 27 An exemplary embodiment of proposing a valid region when a current block is within a base region (BR) is shown.

[0582] Figure 28 An exemplary embodiment of a valid area when the current block is not within the basic area is shown.

[0583] b) In one example, if W>M and H>N, meaning the current block is not within the basic region, the current block is divided into several parts. Each part has a separate valid corresponding region in ATMVP. For position A in the current block, its corresponding position B in the co-located block should be within the valid corresponding region of the part where position A is located.

[0584] i. For example, the current block is divided into non-overlapping basic regions. The valid corresponding regions of a basic region are its co-located basic region and its extension in the co-located picture. Figure 28 An example is shown.

[0585] 1. For example, assume that position A in the current block is in a basic region R. The co-located basic region of R in the co-located picture is referred to as CR. The corresponding position of A in the co-located block is position B. The top left position of CR is (xCR, yCR), then position B at (xN, yN) will be cropped to the valid region xCR <= xN <xCR+M+4;

[0586] yCR<=yN <yCR+N。

[0587] 7. It is proposed that the motion vector used to locate the corresponding block in different pictures in ATMVP (for example, TMV in 2.3.5.1.2) can be derived as:

[0588] a) In one example, the TMV is always set equal to a default MV, such as (0, 0).

[0589] i. In one example, the default MV is in VPS / SPS / PPS / strip header /

[0590] Signaled in the slice group header / CTU / CU.

[0591] b) In one example, the TMV is configured to be stored in

[0592] An MV in the HMVP table;

[0593] i. If the HMVP list is empty, the TMV is set equal to the default MV, such as (0, 0)

[0594] ii. Otherwise (HMVP list is not empty),

[0595] 1. TMV can be set equal to the first element stored in the HMVP table;

[0596] 2. Alternatively, TMV can be set equal to the last element stored in the HMVP table;

[0597] 3. Alternatively, the TMV may simply be set equal to a specific MV stored in the HMVP table;

[0598] a. In one example, a particular MV references reference list 0.

[0599] b. In one example, a specific MV refers to Reference List 1.

[0600] c. In one example, a particular MV references a particular reference picture in reference list 0, such as the reference picture with index 0.

[0601] d. In one example, a specific MV references a specific reference picture in reference list 1, such as the reference picture with index 0.

[0602] e. In one example, a specific MV references a co-located picture.

[0603] 4. Alternatively, if a specific MV stored in the HMVP table

[0604] If it cannot be found (e.g., as mentioned in point 3.), the TMV can be set equal to the default MV;

[0605] a. In one example, only search is performed on the HMVP

[0606] The first element in the table to find a specific MV.

[0607] b. In one example, only search is performed on the HMVP

[0608] The last element in the table to find a specific MV.

[0609] c. In one example, some or all elements in the HMVP table are stored to find a specific MV.

[0610] 5. Alternatively, furthermore, the TMV obtained from the HMVP cannot reference the current picture itself.

[0611] 6. Alternatively, furthermore, the TMV table obtained from the HMVP may be scaled to the collocated picture if it does not refer to the collocated picture.

[0612] c) In one example, the TMV is set to an MV of a specific adjacent block without reference to other adjacent blocks.

[0613] i. A specific adjacent block can be Figure 22 Blocks A0, A1, B0, B1, B2 in.

[0614] ii. The TMV may be set equal to the default MV if:

[0615] 1. The specific adjacent block does not exist;

[0616] 2. The specific adjacent block is not inter-frame coded;

[0617] iii. TMV can only be set equal to a specific MV stored in a specific adjacent block;

[0618] 1. In one example, a particular MV references reference list 0.

[0619] 2. In one example, a specific MV refers to Reference List 1.

[0620] 3. In one example, a specific MV references a specific reference picture in reference list 0, such as the reference picture with index 0.

[0621] 4. In one example, a specific MV references a specific reference picture in reference list 1, such as the reference picture with index 0.

[0622] 5. In one example, a specific MV references a co-located picture.

[0623] 6. If a specific MV stored in a specific contiguous block cannot be found, the TMV can be set equal to the default MV;

[0624] iv. The TMV obtained from a specific neighboring block can be scaled to the co-located picture if it does not refer to the co-located picture.

[0625] v. The TMV obtained from a specific neighboring block cannot refer to the current picture itself.

[0626] 8. MVdefault0 and MVdefault1 used in ATMVP as disclosed in 2.3.5.1.2 can be derived as

[0627] a) In one example, MVdefault0 and MVdefault1 are set to (0,0);

[0628] b) In one example, MVdefaultX (X=0 or 1) is derived from HMVP,

[0629] i. If the HMVP list is empty, MVdefaultX is set equal to a predetermined default MV, such as (0, 0).

[0630] 1. The default MV can be set in VPS / SPS / PPS / Strip Header /

[0631] Signaled in the slice group header / CTU / CU.

[0632] ii. Otherwise (HMVP list is not empty),

[0633] 1.MVdefaultX can be set equal to the first element stored in the HMVP table;

[0634] 2.MVdefaultX can be set equal to the last element stored in the HMVP table;

[0635] 3.MVdefaultX can only be set equal to a specific MV stored in the HMVP table;

[0636] a. In one example, a particular MV references a reference list X.

[0637] b. In one example, a particular MV references a particular reference picture in reference list X, such as the reference picture with index 0.

[0638] 4. If a specific MV stored in the HMVP table cannot be found, MVdefaultX can be set equal to a predetermined default MV;

[0639] a. In one example, only the first element stored in the HMVP table is searched.

[0640] b. In one example, only the last element stored in the HMVP table is searched.

[0641] c. In one example, some or all elements stored in the HMVP table are searched.

[0642] 5. The MVdefaultX obtained from the HMVP table can be scaled to the co-located picture if it does not refer to the co-located picture.

[0643] 6. The MVdefaultX obtained from HMVP cannot refer to the current picture itself.

[0644] c) In one example, MVdefaultX (X=0 or 1) is derived from neighboring blocks.

[0645] i. Adjacent blocks can contain blocks Figure 22 A0, A1, B0, B1, B2 in .

[0646] 1. For example, only one of these blocks is used to derive MVdefaultX.

[0647] 2. Alternatively, some or all of these blocks are used to derive MVdefaultX.

[0648] a. Check these blocks in order until a valid MVdefaultX is found.

[0649] 3. If no valid MVdefaultX is found from the selected neighboring block or blocks, then MVdefaultX is set equal to a predetermined default MV, such as (0, 0).

[0650] a. Can be used in VPS / SPS / PPS / strip header / slice group header

[0651] / CTU / CU signaling notification pre-determines the default MV.

[0652] ii. No valid MVdefaultX is found from a particular adjacent block if:

[0653] 1. The specific adjacent block does not exist;

[0654] 2. The specific adjacent block is not inter-frame coded;

[0655] iii.MVdefaultX can only be set equal to a specific MV stored in a specific adjacent block;

[0656] 1. In one example, a particular MV references a reference list X.

[0657] 2. In one example, a specific MV references a specific reference picture in reference list X, such as the reference picture with index 0

[0658] iv. The MVdefaultX obtained from a specific neighboring block may be scaled to a specific reference picture, such as the reference picture with index 0 in reference list X.

[0659] v. The MVdefaultX obtained from a specific neighboring block cannot refer to the current picture itself.

[0660] 9. For sub-block or non-sub-block ATMVP candidates, if the temporal block of the sub-block / whole block in the co-located picture is coded in CPR mode, the default motion candidate may be used instead.

[0661] a) In one example, the default motion candidate may be defined as a motion candidate associated with the center position of the current block (eg, MVdefault0 and / or MVdefault1 used in ATMVP as disclosed in 2.3.5.1.2).

[0662] b) In one example, the default motion candidate may be defined as a (0, 0) motion vector and reference picture index equal to 0 (if available) for both reference picture lists.

[0663] 10. It is proposed that the default motion information of the ATMVP process (e.g., MVdefault0 and / or MVdefault1 used in ATMVP as disclosed in 2.3.5.1.2) can be derived based on the position used in the sub-block motion information derivation process. Using the proposed method, no further motion information needs to be derived for the sub-block because the default motion information will be directly assigned.

[0664] a) In one example, instead of using the center position of the current block, the center position of a sub-block (eg, the center sub-block) within the current block may be used.

[0665] b) If Figure 31B An example is shown. Figure 31A An example of where default motion information is identified in the current standard is shown.

[0666] 11. The following methods are always available for proposing ATMVP candidates:

[0667] a) Assume the center point of the current block is (x0, y0). Then, the corresponding position of (x0, y0) in the collocated picture is M = (x0 + MV'x, y0 + MV'y). Find the block Z that covers M. If Z is intra-coded, derive MVdefault0 and MVdefault1 using some of the methods proposed in Project 6.

[0668] b) Alternatively, without locating block Z to get motion information, some of the methods proposed in item 8 are directly applied to get MVdefault0 and MVdefault1.

[0669] c) Alternatively, the default motion candidate used in the ATMVP process is always available. If it is set to be unavailable based on the current design (e.g., the time domain block is intra-coded), other motion vectors can be used as the default motion candidate instead.

[0670] i. In one example, an international application may be applied

[0671] The scheme in PCT / CN2018 / 124639, which is incorporated herein by reference.

[0672] d) Alternatively, furthermore, whether an ATMVP candidate is always available depends on other high-level syntax information.

[0673] i. In one example, ATMVP candidates may always be set to available only when the ATMVP enabled flag at the slice / slice / picture header or other video unit is set to true (inferred as true).

[0674] ii. In one example, the above method may be applied only when the ATMVP enabled flag in the slice header / picture header or other video unit is set to true and the current picture is not an IRAP picture and the current picture is not inserted into RefPicList0 with reference index equal to 0.

[0675] e) A fixed index or a fixed index group is assigned to the ATMVP candidate. When the ATMVP candidate is always unavailable, the fixed index / group index can be inferred to other kinds of motion candidates (such as affine candidates).

[0676] 12. It is proposed that whether a zero-motion affine merge candidate is placed in the sub-block merge candidate list should depend on whether affine prediction is enabled.

[0677] a) For example, if affine is disabled using the flag (sps_affine_enabled_flag

[0678] is equal to 0), the zero-motion affine Merge candidate is not placed in the sub-block Merge candidate list.

[0679] b) Alternatively, furthermore, a default motion vector candidate is added as a non-affine candidate instead.

[0680] 13. It is proposed that non-affine filling candidates can be placed in the sub-block Merge candidate list.

[0681] a) If the sub-block Merge candidate list is not satisfied, zero-motion non-affine filling candidates can be filled.

[0682] b) When such a filling candidate is selected, the affine_flag of the current block should be set to 0.

[0683] c) Alternatively, if the sub-block Merge candidate list is not satisfied and the affine usage flag is off, then a zero-motion non-affine fill candidate is placed into the sub-block Merge candidate list.

[0684] 14. Assume that MV0 and MV1 represent the MVs in reference list 0 and reference list 1 covering the corresponding block (for example, MV0 and MV1 can be MVZ_0 and MVZ_1 or MVZS_0 and MVZS_1 described in Section 2.3.5.1.2). MV0' and MV1' represent the MVs in reference list 0 and reference list 1 to be derived for the current block or sub-block. Then MV0' and MV1' should be derived by scaling.

[0685] a) MV0, if the co-located picture is in reference list 1;

[0686] b) MV1, if the co-located picture is in reference list 0.

[0687] 15. When the current picture is considered as a reference picture with an index set to M (e.g., 0) in reference picture list X (PicRefListX, e.g., X=0), the ATMVP and / or TMVP enable / disable flags may be inferred as false for a slice / slice or other type of video unit. Here, M may be equal to the target reference picture index, whose motion information of the temporal blocks should be scaled to PicRefListX during the ATMVP / TMVP process.

[0688] a) Alternatively, furthermore, the above method is applicable only when the current picture is an intra random access point (IRAP) picture.

[0689] b) In one example, when the current picture is considered as a reference picture with an index set to M (e.g., 0) in PicRefListX and / or the current picture is considered as a reference picture with an index set to N (e.g., 0) in PicRefListY, the ATMVP and / or TMVP enable / disable flags may be inferred to be false. The variables M and N represent the target reference picture indices used in the TMVP or ATMVP process.

[0690] c) For the ATMVP process, the constraint confirmation bitstream shall follow the following rule: the co-located picture from which the motion information of the current block is derived shall not be the current picture.

[0691] d) Alternatively, when the above condition is true, the ATMVP or TMVP process is not called.

[0692] 16. It is proposed that if the reference picture with an index set to M (eg, 0) in the reference picture list X (PicRefListX, eg, X=0) of the current block is the current picture, ATMVP may still be enabled for the block.

[0693] a) In one example, the motion information of all sub-blocks points to the current picture.

[0694] b) In one example, when obtaining motion information of a sub-block from a time-domain block, the time-domain block applies at least one reference picture codec pointing to the current picture of the time-domain block.

[0695] c) In one example, when obtaining the motion information of a sub-block from a temporal block, no scaling operation is applied.

[0696] 17. The encoding and decoding methods of sub-block merge indexes are consistent regardless of whether ATMVP is used.

[0697] a) In one example, for the first L bins, they are context-coded. For the remaining bins, they are bypass-coded. In one example, L is set to 1.

[0698] b) Alternatively, for all bins, they are context encoded.

[0699] 18. The MVs (MVx, MVy) used in ATMVP to locate corresponding blocks of different pictures (e.g., TMVs in 2.3.5.1.2) can be right-shifted to integer precision (denoted as (MVx', MVy') using the same rounding method as in the MV scaling process.

[0700] a) Alternatively, the MVs used in ATMVP to locate corresponding blocks in different pictures (eg, TMVs in 2.3.5.1.2) may be right-shifted to integer precision using the same rounding method as in the MV averaging process.

[0701] b) Alternatively, the MV used in ATMVP to locate corresponding blocks in different pictures (e.g., TMV in 2.3.5.1.2) can be right-shifted to integer precision with the same rounding method as in the Adaptive MV Resolution (AMVR) process.

[0702] 19. The MVs (MVx, MVy) used in ATMVP to locate corresponding blocks in different pictures (eg, TMVs in 2.3.5.1.2) may be right-shifted to integer precision (denoted as (MVx', MVy')) by rounding towards zero.

[0703] a) For example, MVx'=(MVx+((1<<N)> >1)-(MVx>=0?1:0))>>N; N is an integer representing the MV resolution, for example, N=4.

[0704] i. For example, MVx′=(MVx+(MVx>=0?7:8))>>4.

[0705] b) For example, MVy'=(MVy+((1<<N)> >1)-(MVy>=0?1:0))>>N; N is an integer representing the MV resolution, for example, N=4.

[0706] i. For example, MVy′=(MVy+(MVy>=0?7:8))>>4.

[0707] 20. In one example, the MVs (MVx, MVy) in points 18 and 19 are used to locate the corresponding blocks to derive the default motion information used in ATMVP, such as using the center position and shifted MV of the sub-block, or using the top left position and shifted MV of the current block.

[0708] a) In one example, the MVs (MVx, MVy) are used to locate the corresponding block during the ATMVP process to derive motion information of the sub-block in the current block, such as using the center position of the sub-block and the shifted MV.

[0709] 21. The methods proposed in points 18, 19 and 20 can be applied to other coding tools associated with locating a reference block in a different picture or the current picture with a motion vector.

[0710] 22. The MVs (MVx, MVy) used in ATMVP to locate corresponding blocks in different pictures (e.g., TMV in 2.3.5.1.2) can be scaled even if they point to co-located pictures.

[0711] a) In one example, if the width and / or height of the collocated picture (or the consistency window therein) is different from the current picture (or the consistency window therein), the MV may be scaled.

[0712] b) Assume that the width and height of the (consistency window) of the collocated picture are denoted as W1 and H1, respectively. Then the width and height of (the consistency window of) the current picture are denoted as W2 and H2, respectively. Then MV (MVx, MVy) can be scaled as MVx'=MVx*W1 / W2 and MVy'=MVy*H1 / H2.

[0713] 23. The center point of the current block used to derive motion information in the ATMVP process (such as position (x0, y0) in 2.3.5.1.2) can be further modified by scaling and / or adding an offset.

[0714] a) In one example, if the width and / or height of the collocated picture (or the consistency window therein) is different from the current picture (or the consistency window therein), the center point may be further modified.

[0715] b) Assume that the top left position of the consistency window in the co-located picture is denoted by X1 and Y1. The top left position of the consistency window defined in the current picture is denoted by X2 and Y2. The width and height of (the consistency window of) the co-located picture are denoted by W1 and H1, respectively. The width and height of (the consistency window of) the current picture are denoted by W2 and H2, respectively. Then (x0, y0) can be modified as x0' = (x0-X2)*W1 / W2+X1 and y0' = (y0-Y2)*H1 / H2+Y1.

[0716] i. Alternatively, x0′=x0*W1 / W2, y0′=y0*H1 / H2.

[0717] 24. The corresponding position used to derive motion information in the ATMVP process (such as position M in 2.3.5.1.2) can be further modified by scaling and / or adding an offset

[0718] a) In one example, if the width and / or height of the co-located picture (or the consistency window therein) is different from that of the current picture (or the consistency window therein), the corresponding positions may be further modified.

[0719] b) Assume that the top left position of the consistency window in the collocated picture is denoted by X1 and Y1. The top left position of the consistency window defined in the current picture is denoted by X2 and Y2. The width and height of (the consistency window of) the collocated picture are denoted by W1 and H1, respectively. The width and height of (the consistency window of) the current picture are denoted by W2 and H2, respectively. Then M(x, y) can be modified as x' = (x - X2) * W1 / W2 + X1 and y' = (y - Y2) * H1 / H2 + Y1.

[0720] i. Alternatively, x'=x*W1 / W2, y'=y*H1 / H2.

[0721] Sub-image related

[0722] 25. In one example, if positions (i, j) and (i, j-1) belong to different sub-pictures, the width of the sub-picture S ending at the (j-1) column can be set equal to j minus the leftmost column of the sub-picture S.

[0723] a) Examples based on JVET-O2001-vE are emphasized below in bold italics.

[0724]

[0725]

[0726] 26. In one example, the height of a sub-picture S that ends at (NumSubPicGridRows-1) rows can be set to (NumSubPicGridRows-1) minus the top row of the sub-picture S plus one.

[0727] a) An embodiment based on JVET-O2001-vE is highlighted below.

[0728]

[0729]

[0730] 27. In one example, the width of a sub-picture S that ends at (NumSubPicGridColumns-1) columns can be set to (NumSubPicGridColumns-1) minus the leftmost column of the sub-picture S and then plus 1.

[0731] a) An embodiment based on JVET-O2001-vE is highlighted below.

[0732]

[0733]

[0734] 28. The sub-picture grid can be an integer multiple of the CTU size.

[0735] a) The embodiment based on JVET-O2001-vE is emphasized below

[0736] subpic_grid_col_width_minus1 plus 1 specifies the width of each element of the sub-picture identifier grid in units of CtbSizeY. The length of the syntax element is Ceil(Log2(pic_width_max_in_luma_samples / CtbSizeY)) bits.

[0737] The variable NumSubPicGridCols is derived as follows:

[0738] NumSubPicGridCols=(pic_width_max_in_luma_samples+subpic_grid_col_width_minus1*CtbSi zeY+CtbSizeY-1) /

[0739] (subpic_grid_col_width_minus1*CtbSizeY+CtbSizeY)(7-5)

[0740] subpic_grid_row_height_minus1 plus 1 specifies the height of each element of the sub-picture identifier grid in units of 4 samples. The length of the syntax element is Ceil(Log2(pic_height_max_in_luma_samples / CtbSizeY)) bits.

[0741] The variable NumSubPicGridRows is derived as follows:

[0742] NumSubPicGridRows = (pic_height_max_in_luma_samples + subpic_grid_row_height_minus1*Ctb SizeY + CtbSizeY - 1) /

[0743] (subpic_grid_row_height_minus1*CtbSizeY+CtbSizeY)(7-6)

[0744] 29. Add consistency constraints to ensure that sub-pictures do not overlap each other and that all sub-pictures can cover the entire picture.

[0745] a) The embodiment based on JVET-O2001-vE is indicated as follows.

[0746] If all of the following conditions are satisfied, any subpic_grid_idx[i][j] must be equal to idx:

[0747] i >= SubPicTop[idx] and i < SubPicTop[idx] + SubPicHeight[idx]. j >= SubPicLeft[idx] and j < SubPicLeft[idx] + SubPicWidth[idx]. If not all of the following conditions are satisfied, any subpic_grid_idx[i][j] must be different from idx: i >= SubPicTop[idx] and i < SubPicTop[idx] + SubPicHeight[idx]. j >= SubPicLeft[idx] and j < SubPicLeft[idx] + SubPicWidth[idx].

[0748] RPR related

[0749] 30. Signaling a syntax element (such as a flag) denoted as RPR_flag to indicate whether RPR can be used in a video unit (eg, sequence). RPR_flag can be signaled in SPS, VPS, or DPS.

[0750] a) In one example, if RPR is not signaled to be used (e.g. RPR_flag is 0), then the width of the signaled image in the picture parameter set (PPS) is

[0751] The maximum width / height may be the same as the maximum height signaled in the sequence parameter set (SPS).

[0752] b) In one example, if it is signaled that RPR is not used (eg RPR_flag is 0), then the overall width / height is not signaled in the PPS and is inferred to be the maximum width / maximum height signaled in the SPS.

[0753] c) In one example, if the signaling indicates that RPR is not to be used (eg, RPR_flag is 0), the consistency window information is not used during decoding. Otherwise (the signaling indicates that RPR is to be used), the consistency window information may be used during decoding.

[0754] 31. It is proposed that the interpolation filter used to derive the prediction block of the current block in the motion compensation process can be selected depending on whether the resolution of the reference picture is different from the current picture, or whether the width and / or height of the reference picture is larger than the current picture.

[0755] a. In one example, an interpolation filter with fewer taps may be applied when condition A is satisfied, where condition A depends on the dimensions of the current picture and / or the reference picture.

[0756] i. In one example, condition A is that the resolution of the reference picture is different from that of the current picture.

[0757] ii. In one example, condition A is that the width and / or height of the reference picture is larger than that of the current picture.

[0758] iii. In one example, condition A is W1>a*W2 and / or H1>

[0759] b*H2, where (W1, H1) represents the width and height of the reference picture, and

[0760] (W2, H2) represents the width and height of the current image, and a and b are two factors, for example, a=b=1.5.

[0761] iv. In one example, condition A may also depend on whether bidirectional prediction is used.

[0762] 1) Condition A is satisfied only when bidirectional prediction is used for the current block.

[0763] v. In one example, condition A may depend on M and N, where M

[0764] and N represent the width and height of the current block.

[0765] 1) For example, condition A is satisfied only when M*N<=T, where T is an integer such as 64.

[0766] 2) For example, condition A is satisfied only when M<=T1 or N<=T2, where T1 and T2 are integers, for example, T1=T2=4.

[0767] 3) For example, condition A is satisfied only when M<=T1 and N<=T2, where T1 and T2 are integers, for example, T1=T2=4.

[0768] 4) For example, condition A is satisfied only when M*N<=T, or M<=T1, or N<=T2, where T, T1, and T2 are integers, such as T=64.

[0769] T1=T2=4.

[0770] 5) In one example, the less than condition in the above sub-point can be replaced with greater than.

[0771] vi. In one example, 1-tap filtering is applied. In other words, integer pixels without filtering are output as the interpolation result.

[0772] vii. In one example, bilinear filtering is applied when the resolution of the reference picture is different from the current picture.

[0773] viii. In one example, when the resolution of the reference picture is different from that of the current picture, or the width and / or height of the reference picture is larger than that of the current picture, 4-tap filtering or 6-tap filtering is applied.

[0774] 1) 6-tap filtering can also be used for affine motion compensation.

[0775] 2) 4-tap filtering can also be used for interpolation of chroma samples.

[0776] b. Whether and / or how to apply the method disclosed in point 31 may depend on the color components.

[0777] i. For example, the method is applied only to the luma component.

[0778] c. Whether and / or how to apply the method disclosed in point 31 may depend on the interpolation filtering direction.

[0779] i. For example, the method is only applied to horizontal filtering.

[0780] ii. For example, the method is only applied to vertical filtering.

[0781] CIIP related

[0782] 32. The intra prediction signaling used in the CIIP process can be performed at the TU level rather than the CU level (for example, using reference samples outside the TU rather than the CU).

[0783] a) In one example, if the CU width or height is larger than the maximum transform unit size, the CU may be divided into multiple TUs, and intra / inter prediction may be generated for each TU, for example, using reference samples outside the TU.

[0784] b) In one example, if the maximum transform size K is less than 64 (such as K

[0785] =32), the intra prediction used in CIIP is performed recursively, as in the normal intra coding block.

[0786] c) For example, a KM×KN CIIP codec block (M and N are integers) is divided into MN K×K blocks, and intra prediction is performed on each K×K block.

[0787] The intra prediction of a K×K block being coded / decoded may depend on the reconstructed samples of a previously coded / decoded K×K block.

[0788] 5. Additional Exemplary Embodiments

[0789] 5.1 Example #1: Example of Syntax Design in SPS / PPS / Slice Header / Slice Group Header The changes compared to the VTM3.0.1rc1 reference software are highlighted in large bold as follows:

[0790]

[0791]

[0792] 5.2 Example #2: Example of Syntax Design in SPS / PPS / Slice Header / Slice Group Header 7.3.2.1 Sequence Parameter Set RBSP Syntax

[0793]

[0794]

[0795] sps_sbtmvp_enabled_flag equal to 1 specifies that the subblock based temporal motion vector predictor can be used in the CVS when decoding all slices of pictures with slice_type not equal to 1. sps_sbtmvp_enabled_flag equal to 0 specifies that the subblock based temporal motion vector predictor is not used in the CVS. When sps_sbtmvp_enabled_flag is not present, it is inferred to be equal to 0.

[0796] five_minus_max_num_subblock_merge_cand specifies 5 minus the maximum number of subblock-based merge motion vector prediction (MVP) candidates supported in a slice. When five_minus_max_num_subblock_merge_cand is not present, it is inferred to be equal to 5-sps_sbtmvp_enabled_flag. The maximum number of subblock-based merge MVP candidates, MaxNumSubblockMergeCand, is derived as follows:

[0797] MaxNumSubblockMergeCand=5-five_minus_max_num_subblock_merge_cand(7-45)

[0798] The value of MaxNumSubblockMergeCand should be in the range of 0 to 5 (inclusive).

[0799] 8.3.4.2 Derivation of Motion Vectors and Reference Indices in Sub-Block Merge Mode

[0800] The inputs to this process are:

[0801] ..[No changes to the current VVC draft specification].

[0802] The output of this process is:

[0803] ...[No changes to the current VVC draft specification].

[0804] The variables numSbX, numSbY and subblock Merge candidate list subblockMergeCandList are derived through the following sequential steps:

[0805] When sps_sbtmvp_enabled_flag is equal to 1 and (the current picture is an IRAP and index 0 of reference picture list 0 is the current picture) is not true, the following applies:

[0806] Call the process of deriving Merge candidates from adjacent codec units as specified in clause 8.3.2.3, with the luma codec block position (xCb, yCb), luma codec block width cbWidth, luma codec block height cbHeight and luma codec block width as input, and the output is the availability flags availableFlagA0, availableFlagA1, availableFlagB0, availableFlagB1 and availableFlagB2 lagB2, reference indices refIdxLXA0, refIdxLXA1, refIdxLXB0, refIdxLXB1 and refIdxLXB2, prediction list usage flags predFlagLXA0, predFlagLXA1, predFlagLXB0, predFlagLXB1 and predFlagLXB2, and motion vectors mvLXA0, mvLXA1, mvLXB0, mvLXB1 and mvLXB2, where X is 0 or 1.

[0807] Call the derivation process of the sub-block based temporal Merge candidate as specified in clause 8.3.4.3 with the luma position (xCb, yCb), luma codec block width cbWidth, luma codec block height cbHeight, availability flags availableFlagA0, availableFlagA1, availableFlagB0, availableFlagB1, reference indices refIdxLXA0, refIdxLXA1, refIdxLXB0, refIdxLXB1, prediction list usage flags predFlagLXA0, predFlagLXA1, predFlagLXB0, pr It takes edFlagLXB1 and motion vectors mvLXA0, mvLXA1, mvLXB0, mvLXB1 as input, and outputs the availability flag availableFlagSbCol, the number of luminance codec sub-blocks in the horizontal direction numSbX and the number in the vertical direction numSbY, the reference index refIdxLXSbCol, the luminance motion vector mvLXSbCol[xSbIdx][ySbIdx] and the prediction list usage flag predFlagLXSbCol[xSbIdx][ySbIdx], where xSbIdx=0..numSbX-1, ySbIdx=0..numSbY-1, and X is 0 or 1.

[0808] When sps_affine_enabled_flag is equal to 1, the sample positions (xNbA0, yNbA0), (xNbA1, yNbA1), (xNbA2, yNbA2), (xNbB0, yNbB0), (xNbB1, yNbB1), (xNbB2, yNbB2), (xNbB3, yNbB3) and the variables numSbX and numSbY are derived as follows:

[0809] [No changes to the current VVC draft specification].

[0810] 5.3 Example #3MV Rounding

[0811] The syntax changes are based on JVET-O2001-vE.

[0812] 8.5.5.3 Derivation of Sub-Block-Based Time-Domain Merge Candidates

[0813] …

[0814] The position (xColSb, yColSb) of the collocated sub-block within ColPic is derived as follows.

[0815] 1. The following apply:

[0816] yColSb=Clip3(yCtb,Min(CurPicHeightInSamplesY-1,yCtb+(1<<CtbLog2SizeY)-1),ySb+((tempMv[1]+8–(tempMv[1]> =0))>>4))

[0817] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0818] xColSb=Clip3(xCtb,Min(SubPicRightBoundaryPos,xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+8+(tempMV[0]> =0))>>4))

[0819] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0820] xColSb=Clip3(xCtb,Min(CurPicWidthInSamplesY-1,xCtb+(1<<CtbLog2SizeY)+3),xSb+(tempMv[0]+8+(tempMV[0]> =0))>>4))

[0821] …

[0822] 8.5.5.4 Derivation of Sub-Block-Based Temporal Merge Basic Motion Data

[0823] …

[0824] The position (xColCb, yColCb) of the co-located block within ColPic is derived as follows.

[0825] – The following apply:

[0826] yColCb=Clip3(yCtb,Min(CurPicHeightInSamplesY-1,yCtb+(1<<CtbLog2SizeY)-1),yColCtrCb+((tempMv[1]+8–(tempMv[1]> =0))>>4))

[0827] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0828] xColCb=Clip3(xCtb,Min(SubPicRightBoundaryPos,xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+8+(tempMV[0]> =0))>>4))

[0829] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0, the following applies:

[0830] xColCb=Clip3(xCtb,Min(CurPicWidthInSamplesY-1,xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+8+(tempMV[0]> =0))>>4))

[0831] 5.3 Example #3: Example of MV rounding.

[0832] The syntax changes are based on JVET-O2001-vE.

[0833] 8.5.5.3 Derivation of Sub-Block-Based Time-Domain Merge Candidates

[0834] …

[0835] The position (xColSb, yColSb) of the collocated sub-block within ColPic is derived as follows.

[0836] 1. The following apply:

[0837] -yColSb=Clip3(yCtb,Min(CurPicHeightInSamplesY-1,yCtb+(1<<CtbLog2SizeY)-1),ySb+((tempMv[1]+8–(tempMv[1]> =0))>>4))

[0838] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0839] -xColSb=Clip3(xCtb,Min(SubPicRightBoundaryPos,xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+8+(tempMV[0]> =0))>>4))

[0840] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0841] -xColSb=Clip3(xCtb,Min(CurPicWidthInSamplesY-1,xCtb+(1<<CtbLog2SizeY)+3),xSb+(tempMv[0]+8+(tempMV[0]> =0))>>4))

[0842] …

[0843] 8.5.5.4 Derivation of Sub-Block-Based Temporal Merge Basic Motion Data

[0844] …

[0845] The position (xColCb, yColCb) of the co-located block within ColPic is derived as follows.

[0846] – The following apply:

[0847] -yColCb=Clip3(yCtb,Min(CurPicHeightInSamplesY-1,yCtb+(1<<CtbLog2SizeY)-1),yColCtrCb+((tempMv[1]+8–(tempMv[1]> =0))>>4))

[0848] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0849] -xColCb=Clip3(xCtb,Min(SubPicRightBoundaryPos,xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+8+(tempMV[0]> =0))>>4))

[0850] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0, the following applies:

[0851] -xColCb=Clip3(xCtb,Min(CurPicWidthInSamplesY-1,xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+8+(tempMV[0]> =0))>>4))

[0852] 5.4 Example #4: Second example of MV rounding.

[0853] 8.5.5.3 Derivation of Sub-Block-Based Time-Domain Merge Candidates

[0854] The inputs to this process are:

[0855] – the luma position (xCb, yCb) of the top left sample of the current luma codec block relative to the top left luma sample of the current picture,

[0856] – The variable cbWidth specifies the width of the current codec block in luminance samples.

[0857] – The variable cbHeight specifies the height of the current codec block in luma samples.

[0858] – The availability flag of the adjacent codec unit availableFlagA1,

[0859] – The reference index of the adjacent codec unit refIdxLXA1, where X is 0 or 1,

[0860] – The prediction list of the adjacent codec unit uses the flag predFlagLXA1, where X is 0 or 1,

[0861] – The motion vector mvLXA1 of the adjacent codec unit with 1 / 16 fractional sample accuracy, where X is 0 or 1.

[0862] The output of this process is:

[0863] – Availability flag availableFlagSbCol,

[0864] – The number of luma codec sub-blocks in the horizontal direction numSbX and the number in the vertical direction numSbY,

[0865] – reference indexes refIdxL0SbCol and refIdxL1SbCol,

[0866] – Luma motion vectors mvL0SbCol[xSbIdx][ySbIdx] and mvL1SbCol[xSbIdx][ySbIdx] with 1 / 16 fractional sample accuracy, where xSbIdx=0..numSbX-1, ySbIdx=0..numSbY-1,

[0867] – The prediction list uses flags predFlagL0SbCol[xSbIdx][ySbIdx] and predFlagL1SbCol[xSbIdx][ySbIdx], where xSbIdx = 0..numSbX-1, ySbIdx = 0..numSbY-1.

[0868] The availability flag availableFlagSbCol is derived as follows.

[0869] – availableFlagSbCol is set equal to 0 if one or more of the following conditions are true.

[0870] –slice_temporal_mvp_enabled_flag is equal to 0.

[0871] –sps_sbtmvp_enabled_flag is equal to 0.

[0872] –cbWidth is less than 8.

[0873] –cbHeight is less than 8.

[0874] – Otherwise, apply the following sequential steps:

[0875] 1. The position of the top left sample point (xCtb, yCtb) of the luma codec tree block containing the current codec block and the position of the bottom right center sample point (xCtr, yCtr) of the current luma codec block are derived as follows:

[0876] -xCtb=(xCb>>CtuLog2Size)< <CtuLog2Size (8-542)

[0877] -yCtb=(yCb>>CtuLog2Size)< <CtuLog2Size (8-543)

[0878] -xCtr=xCb+(cbWidth / 2) (8-544)

[0879] -yCtr=yCb+(cbHeight / 2) (8-545)

[0880] 2. The luma position (xColCtrCb, yColCtrCb) is set equal to the top left sample of the collocated luma codec block covering the position given by (xCtr, yCtr) within ColPic relative to the top left luma sample of the collocated picture specified by ColPic.

[0881] 3. Invoke the derivation process of the sub-block based temporal Merge basic motion data as specified in clause 8.5.5.4, with position (xCtb, yCtb), position (xColCtrCb, yColCtrCb), availability flag availableFlagA1 and prediction list usage flag predFlagLXA1, and reference index refIdxLXA1, and motion vector mvLXA1, where X is 0 and 1, as input, and motion vector ctrMvLX and prediction list usage flag ctrPredFlagLX of the collocated block, where X is 0 and 1, and temporal motion vector tempMv as output.

[0882] 4. The variable availableFlagSbCol is derived as follows:

[0883] – If both ctrPredFlagL0 and ctrPredFlagL1 are equal to 0, availableFlagSbCol is set equal to 0.

[0884] – Otherwise, availableFlagSbCol is set equal to 1.

[0885] When availableFlagSbCol is equal to 1, the following applies:

[0886] – The variables numSbX, numSbY, sbWidth, sbHeight, and refIdxLXSbCol are derived as follows:

[0887] -numSbX=cbWidth>>3 (8-546)

[0888] -numSbY=cbHeight>>3 (8-547)

[0889] -sbWidth=cbWidth / numSbX (8-548)

[0890] -sbHeight=cbHeight / numSbY (8-549)

[0891] -refIdxLXSbCol=0 (8-550)

[0892] – For xSbIdx = 0..numSbX–1 and ySbIdx = 0..numSbY-1, the motion vector mvLXSbCol[xSbIdx][ySbIdx] and the prediction list are derived using the flag predFlagLXSbCol[xSbIdx][ySbIdx] as follows:

[0893] – Luma position (xSb, ySb), specifies the top left sample of the current codec sub-block relative to the top left luma sample of the current picture, and is derived as follows:

[0894] -xSb=xCb+xSbIdx*sbWidth+sbWidth / 2 (8-551)

[0895] -ySb=yCb+ySbIdx*sbHeight+sbHeight / 2 (8-552)

[0896] The position (xColSb, yColSb) of the collocated sub-block within ColPic is derived as follows.

[0897] 1. The following apply:

[0898] -yColSb=Clip3(yCtb,Min(CurPicHeightInSamplesY-1,yCtb+(1<<CtbLog2SizeY)-1),ySb+((tempMv[1]+8–(tempMv[1]> =0? 1:0))>>4))

[0899] (8-553)

[0900] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0901] -xColSb=Clip3(xCtb,Min(SubPicRightBoundaryPos,xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+8–(tempMv[0]> =0? 1:0))>>4))

[0902] (8-554)

[0903] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0904] -xColSb=Clip3(xCtb,Min(CurPicWidthInSamplesY-1,xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+8–(tempMv[0]> =0? 1:0))>>4))

[0905] (8-555)

[0906] – The variable currCb specifies the luma codec block that covers the current codec sub-block within the current picture.

[0907] – The variable colCb specifies the luma codec block covering the modified position within ColPic given by ((xColSb>>3)<<3, (yColSb>>3)<<3).

[0908] The luma position (xColCb, yColCb) is set equal to the top left sample of the collocated luma codec block specified by colCb relative to the top left luma sample of the collocated picture specified by ColPic.

[0909] – Calls the derivation process of the co-located motion vector as specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL0 set equal to 0 and sbFlag set equal to 1, and outputs the motion vector mvL0SbCol[xSbIdx][ySbIdx] assigned to the sub-block and availableFlagL0SbCol.

[0910] – Calls the derivation process of the co-located motion vector as specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL1 set equal to 0 and sbFlag set equal to 1, and outputs the motion vector mvL1SbCol[xSbIdx][ySbIdx] assigned to the sub-block and availableFlagL1SbCol.

[0911] – When availableFlagL0SbCol and availableFlagL1SbCol are both equal to 0, the following applies for X being 0 and 1:

[0912] -mvLXSbCol[xSbIdx][ySbIdx]=ctrMvLX(8-556)

[0913] -predFlagLXSbCol[xSbIdx][ySbIdx]=ctrPredFlagLX (8-557)

[0914] 8.5.5.4 Derivation of Sub-Block-Based Temporal Merge Basic Motion Data The input to this process is:

[0915] – the position of the top left sample of the luma codec tree block containing the current codec block (xCtb, yCtb),

[0916] – The position of the top left sample of the collocated luma codec block covering the bottom right center sample (xColCtrCb, yColCtrCb).

[0917] – The availability flag of the adjacent codec unit availableFlagA1,

[0918] – the reference index of the adjacent codec unit refIdxLXA1,

[0919] – The prediction list of the adjacent codec unit uses the flag predFlagLXA1,

[0920] – Motion vector mvLXA1 of the adjacent codec unit with 1 / 16 fractional sample accuracy.

[0921] The output of this process is:

[0922] – motion vectors ctrMvL0 and ctrMvL1,

[0923] – The prediction list uses flags ctrPredFlagL0 and ctrPredFlagL1,

[0924] – Temporal motion vector tempMv.

[0925] The variable tempMv is set as follows:

[0926] -tempMv[0]=0 (8-558)

[0927] -tempMv[1]=0 (8-559)

[0928] The variable currPic specifies the current picture.

[0929] When availableFlagA1 is equal to true, then the following applies:

[0930] – tempMv is set equal to mvL0A1 if all of the following conditions are true:

[0931] –predFlagL0A1 is equal to 1,

[0932] –DiffPicOrderCnt(ColPic, RefPicList[0][refIdxL0A1]) is equal to 0,

[0933] – Otherwise, if all of the following conditions are true, tempMv is set equal to mvL1A1:

[0934] –slice_type is equal to B,

[0935] –predFlagL1A1 is equal to 1,

[0936] – DiffPicOrderCnt(ColPic, RefPicList[1][refIdxL1A1]) is equal to 0. The position (xColCb, yColCb) of the collocated block within ColPic is derived as follows.

[0937] – The following apply:

[0938] -yColCb=Clip3(yCtb,Min(CurPicHeightInSamplesY-1,yCtb+(1<<CtbLog2SizeY)-1),yColCtrCb+((tempMv[1]+8-(tempMv[1]> =0? 1:0))>>4))

[0939] (8-560)

[0940] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0941] -xColCb=Clip3(xCtb,Min(SubPicRightBoundaryPos,xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+8–(tempMv[0]> =0? 1:0))>>4))

[0942] (8-561)

[0943] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to o), the following applies:

[0944] -xColCb=Clip3(xCtb,Min(CurPicWidthInSamplesY-1,xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+8–(tempMv[0]> =0? 1:0))>>4))

[0945] (8-562)

[0946] The array colPredMode is set equal to the prediction mode array CuPredMode[0] for the collocated picture specified by ColPic.

[0947] The motion vectors ctrMvL0 and ctrMvL1, and the prediction list are derived using the flags ctrPredFlagL0 and ctrPredFlagL1 as follows:

[0948] – If colPredMode[xColCb][yColCb] is equal to MODE_INTER, the following applies:

[0949] –The variable currCb specifies the luma codec block covering (xCtrCb, yCtrCb) within the current picture.

[0950] – The variable colCb specifies the luma codec block covering the modified position given by ((xColCb>>3)<<3, (yColCb>>3)<<3) within ColPic.

[0951] The luma position (xColCb, yColCb) is set equal to the top left sample of the collocated luma codec block specified by colCb relative to the top left luma sample of the collocated picture specified by ColPic.

[0952] – Invoke the derivation process of the co-located motion vector specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL0 set equal to 0 and sbFlag set equal to 1, and the output is assigned to ctrMvL0 and ctrPredFlagL0.

[0953] – Invoke the derivation process of the co-located motion vector specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL1 set equal to 0 and sbFlag set equal to 1, and the output is assigned to ctrMvL1 and ctrPredFlagL1.

[0954] – Otherwise, the following applies:

[0955] -ctrPredFlagL0=0 (8-563)

[0956] -ctrPredFlagL1=0 (8-564)

[0957] 5.5 Example #5: A third example of MV rounding.

[0958] 8.5.5.3 Derivation of Sub-Block-Based Time-Domain Merge Candidates

[0959] The inputs to this process are:

[0960] – the luma position (xCb, yCb) of the top left sample of the current luma codec block relative to the top left luma sample of the current picture,

[0961] – The variable cbWidth specifies the width of the current codec block in luminance samples.

[0962] – The variable cbHeight specifies the height of the current codec block in luma samples.

[0963] – The availability flag of the adjacent codec unit availableFlagA1,

[0964] – The reference index of the adjacent codec unit refIdxLXA1, where X is 0 or 1,

[0965] – The prediction list of the adjacent codec unit uses the flag predFlagLXA1, where X is 0 or 1,

[0966] – The motion vector mvLXA1 of the adjacent codec unit with 1 / 16 fractional sample accuracy, where X is 0 or 1.

[0967] The output of this process is:

[0968] – Availability flag availableFlagSbCol,

[0969] – The number of luma codec sub-blocks in the horizontal direction numSbX and the number in the vertical direction numSbY,

[0970] – reference indexes refIdxL0SbCol and refIdxL1SbCol,

[0971] – Luma motion vectors mvL0SbCol[xSbIdx][ySbIdx] and mvL1SbCol[xSbIdx][ySbIdx] with 1 / 16 fractional sample accuracy, where xSbIdx=0..numSbX-1, ySbIdx=0..numSbY-1,

[0972] – The prediction list uses flags predFlagL0SbCol[xSbIdx][ySbIdx] and predFlagL1SbCol[xSbIdx][ySbIdx], where xSbIdx = 0..numSbX-1, ySbIdx = 0..numSbY-1.

[0973] The availability flag availableFlagSbCol is derived as follows.

[0974] – availableFlagSbCol is set equal to 0 if one or more of the following conditions are true.

[0975] –slice_temporal_mvp_enabled_flag is equal to 0.

[0976] –sps_sbtmvp_enabled_flag is equal to 0.

[0977] –cbWidth is less than 8.

[0978] –cbHeight is less than 8.

[0979] – Otherwise, apply the following sequential steps:

[0980] 1. The position of the top left sample point (xCtb, yCtb) of the luma codec tree block containing the current codec block and the position of the bottom right center sample point (xCtr, yCtr) of the current luma codec block are derived as follows:

[0981] -xCtb=(xCb>>CtuLog2Size)< <CtuLog2Size (8-542)

[0982] -yCtb=(yCb>>CtuLog2Size)< <CtuLog2Size (8-543)

[0983] -xCtr=xCb+(cbWidth / 2) (8-544)

[0984] -yCtr=yCb+(cbHeight / 2) (8-545)

[0985] 2. The luma position (xColCtrCb, yColCtrCb) is set equal to the top left sample of the collocated luma codec block covering the position given by (xCtr, yCtr) within ColPic relative to the top left luma sample of the collocated picture specified by ColPic.

[0986] 3. Invoke the derivation process of the sub-block based temporal Merge basic motion data as specified in clause 8.5.5.4, with the position (xCtb, yCtb), the position (xColCtrCb, yColCtrCb), the availability flag availableFlagA1, and the prediction list usage flag predFlagLXA1, and the reference index refIdxLXA1, and the motion vector mvLXA1, where X is 0 and 1, as input, and the motion vector ctrMvLX and the prediction list usage flag ctrPredFlagLX of the collocated block, where X is 0 and 1, and the temporal motion vector tempMv as output.

[0987] 4. The variable availableFlagSbCol is derived as follows:

[0988] – If both ctrPredFlagL0 and ctrPredFlagL1 are equal to 0, availableFlagSbCol is set equal to 0.

[0989] – Otherwise, availableFlagSbCol is set equal to 1.

[0990] When availableFlagSbCol is equal to 1, the following applies:

[0991] – The variables numSbX, numSbY, sbWidth, sbHeight, and refIdxLXSbCol are derived as follows:

[0992] -numSbX=cbWidth>>3 (8-546)

[0993] -numSbY=cbHeight>>3 (8-547)

[0994] -sbWidth=cbWidth / numSbX (8-548)

[0995] -sbHeight=cbHeight / numSbY (8-549)

[0996] -refIdxLXSbCol=0 (8-550)

[0997] – For xSbIdx = 0..numSbX–1 and ySbIdx = 0..numSbY-1, the motion vector mvLXSbCol[xSbIdx][ySbIdx] and the prediction list are derived using the flag predFlagLXSbCol[xSbIdx][ySbIdx] as follows:

[0998] – Luma position (xSb, ySb), specifies the top left sample of the current codec sub-block relative to the top left luma sample of the current picture, and is derived as follows:

[0999] -xSb=xCb+xSbIdx*sbWidth+sbWidth / 2 (8-551)

[1000] -ySb=yCb+ySbIdx*sbHeight+sbHeight / 2 (8-552)

[1001] The position (xColSb, yColSb) of the collocated sub-block within ColPic is derived as follows.

[1002] 1. The following apply:

[1003] -yColSb=Clip3(yCtb,Min(CurPicHeightInSamplesY-1,yCtb+(1<<CtbLog2SizeY)-1),ySb+((tempMv[1]+(tempMv[1]> =0?7:8))>>4)) (8-553)

[1004] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[1005] -xColSb=Clip3(xCtb,Min(SubPicRightBoundaryPos,xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+(tempMv[0]> =0?7:8))>>4)) (8-554)

[1006] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[1007] -xColSb=Clip3(xCtb,Min(CurPicWidthInSamplesY-1,xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+(tempMv[0]> =0?7:8))>>4)) (8-555)

[1008] – The variable currCb specifies the luma codec block that covers the current codec sub-block within the current picture.

[1009] – The variable colCb specifies the luma codec block covering the modified position within ColPic given by ((xColSb>>3)<<3, (yColSb>>3)<<3).

[1010] The luma position (xColCb, yColCb) is set equal to the top left sample of the collocated luma codec block specified by colCb relative to the top left luma sample of the collocated picture specified by ColPic.

[1011] – Calls the derivation process of the co-located motion vector as specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL0 set equal to 0 and sbFlag set equal to 1, and outputs the motion vector mvL0SbCol[xSbIdx][ySbIdx] assigned to the sub-block and availableFlagL0SbCol.

[1012] – Calls the derivation process of the co-located motion vector as specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL1 set equal to 0 and sbFlag set equal to 1, and outputs the motion vector mvL1SbCol[xSbIdx][ySbIdx] assigned to the sub-block and availableFlagL1SbCol.

[1013] – When availableFlagL0SbCol and availableFlagL1SbCol are both equal to 0, the following applies for X being 0 and 1:

[1014] -mvLXSbCol[xSbIdx][ySbIdx]=ctrMvLX(8-556)

[1015] -predFlagLXSbCol[xSbIdx][ySbIdx]=ctrPredFlagLX (8-557)

[1016] 8.5.5.4 Derivation of Sub-Block-Based Temporal Merge Basic Motion Data

[1017] The inputs to this process are:

[1018] – the position of the top left sample of the luma codec tree block containing the current codec block (xCtb, yCtb),

[1019] – The position of the top left sample of the collocated luma codec block covering the bottom right center sample (xColCtrCb, yColCtrCb).

[1020] – The availability flag of the adjacent codec unit availableFlagA1,

[1021] – the reference index of the adjacent codec unit refIdxLXA1,

[1022] – The prediction list of the adjacent codec unit uses the flag predFlagLXA1,

[1023] – Motion vector mvLXA1 of the adjacent codec unit with 1 / 16 fractional sample accuracy.

[1024] The output of this process is:

[1025] – motion vectors ctrMvL0 and ctrMvL1,

[1026] – The prediction list uses flags ctrPredFlagL0 and ctrPredFlagL1,

[1027] – Temporal motion vector tempMv.

[1028] The variable tempMv is set as follows:

[1029] -tempMv[0]=0 (8-558)

[1030] -tempMv[1]=0 (8-559)

[1031] The variable currPic specifies the current picture.

[1032] When availableFlagA1 is equal to true, then the following applies:

[1033] – tempMv is set equal to mvL0A1 if all of the following conditions are true:

[1034] –predFlagL0A1 is equal to 1,

[1035] –DiffPicOrderCnt(ColPic, RefPicList[0][refIdxL0A1]) is equal to 0,

[1036] – Otherwise, if all of the following conditions are true, tempMv is set equal to mvL1A1:

[1037] –slice_type is equal to B,

[1038] –predFlagL1A1 is equal to 1,

[1039] – DiffPicOrderCnt(ColPic, RefPicList[1][refIdxL1A1]) is equal to 0. The position (xColCb, yColCb) of the collocated block within ColPic is derived as follows.

[1040] – The following apply:

[1041] -yColCb=Clip3(yCtb,Min(CurPicHeightInSamplesY-1,yCtb+(1<<CtbLog2SizeY)-1),yColCtrCb+((tempMv[1]+(tempMv[1]> =0?7:8))>>4))

[1042] (8-560)

[1043] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[1044] -xColCb=Clip3(xCtb,Min(SubPicRightBoundaryPos,xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+(tempMv[0]> =0?7:8))>>4))

[1045] (8-561)

[1046] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to o), the following applies:

[1047] -xColCb=Clip3(xCtb,Min(CurPicWidthInSamplesY-1,xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+(tempMv[0]> =0?7:8))>>4))

[1048] (8-562)

[1049] The array colPredMode is set equal to the prediction mode array CuPredMode[0] for the collocated picture specified by ColPic.

[1050] The motion vectors ctrMvL0 and ctrMvL1, and the prediction list are derived using the flags ctrPredFlagL0 and ctrPredFlagL1 as follows:

[1051] – If colPredMode[xColCb][yColCb] is equal to MODE_INTER, the following applies:

[1052] –The variable currCb specifies the luma codec block covering (xCtrCb, yCtrCb) within the current picture.

[1053] – The variable colCb specifies the luma codec block covering the modified position given by ((xColCb>>3)<<3, (yColCb>>3)<<3) within ColPic.

[1054] The luma position (xColCb, yColCb) is set equal to the top left sample of the collocated luma codec block specified by colCb relative to the top left luma sample of the collocated picture specified by ColPic.

[1055] – Invoke the derivation process of the co-located motion vector specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL0 set equal to 0 and sbFlag set equal to 1, and the output is assigned to ctrMvL0 and ctrPredFlagL0.

[1056] – Invoke the derivation process of the co-located motion vector specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL1 set equal to 0 and sbFlag set equal to 1, and the output is assigned to ctrMvL1 and ctrPredFlagL1.

[1057] – Otherwise, the following applies:

[1058] -ctrPredFlagL0=0 (8-563)

[1059] -ctrPredFlagL1=0 (8-564)

[1060] 8.5.6.3 Fractional Sample Interpolation Process

[1061] 8.5.6.3.1 Overall

[1062] The inputs to this process are:

[1063] – Luma position (xSb, ySb), specifies the top left sample of the current codec sub-block relative to the top left luminance sample of the current picture,

[1064] –Variable sbWidth specifies the width of the current codec sub-block.

[1065] –Variable sbHeight specifies the height of the current codec sub-block.

[1066] – motion vector offset mvOffset,

[1067] – Refine the motion vector refMvLX,

[1068] – the selected reference picture sample array refPicLX,

[1069] – Half-sample interpolation filter index hpelIfIdx,

[1070] – bidirectional optical flow flag bdofFlag,

[1071] –Variable cIdx specifies the color component index of the current block.

[1072] The output of this process is:

[1073] – predSamplesLX, an array of (sbWidth+brdExtSize)x(sbHeight+brdExtSize) predicted sample values.

[1074] The prediction block bounds extension size brdExtSize is derived as follows:

[1075] -brdExtSize=(bdofFlag||(inter_affine_flag[xSb][ySb]&&sps_affine_prof_enabled_flag))? 2:0 (8-752)

[1076] The variable fRefWidth is set equal to the PicOutputWidthL of the reference picture in luma samples.

[1077] The variable fRefHeight is set equal to the PicOutputHeightL of the reference picture in luma samples.

[1078] The motion vector mvLX is set to (refMvLX-mvOffset).

[1079] – If cIdx is equal to 0, the following applies:

[1080] – The scaling factors and their fixed-point representations are defined as

[1081] -hori_scale_fp=((fRefWidth<<14)+(PicOutputWidthL>>1)) / PicOutputWidthL (8-753)

[1082] -vert_scale_fp=((fRefHeight<<14)+(PicOutputHeightL>>1)) / PicOutputHeightL (8-754)

[1083] – Let (xIntL, yIntL) be the luma position given in whole sample units, and (xFracL, yFracL) be the offset given in 1 / 16 sample units. These variables are used only in this clause to specify fractional sample positions within the reference sample array refPicLX.

[1084] – The left top coordinate of the boundary block filled with reference samples (xSbInt L ,ySbInt L ) is set to (xSb+(mvLX[0]>>4), ySb+(mvLX[1]>>4)).

[1085] – For each luma sample position (x L =0..sbWidth-1+brdExtSize,y L =0..sbHeight-1+brdExtSize), the corresponding predicted brightness sample value predSamplesLX[x L ][y L ] is derived as follows:

[1086] – Make(refxSb L ,refySb L ) and (refxL , refy L ) is the luminance position pointed to by the motion vector (refMvLX[0], refMvLX[1]) given in 1 / 16 sample units. L 、refx L 、refySb L and refy L The derivation is as follows:

[1087] -refxSb L =((xSb<<4)+refMvLX[0])*hori_scale_fp

[1088] (8-755)

[1089] -refx L =((Sign(refxSb)*((Abs(refxSb)+128)>>8)+x L *((hori_scale_fp+8)>>4))+32)>>6 (8-756)

[1090] -refySb L =((ySb<<4)+refMvLX[1])*vert_scale_fp

[1091] (8-757)

[1092] -refyL=((Sign(refySb)*((Abs(refySb)+128)>>8)+yL*

[1093] ((vert_scale_fp+8)>>4))+32)>>6(8-758)

[1094] –Variable xInt L 、yInt L 、xFrac L and yFrac L The derivation is as follows:

[1095] -xInt L =refx L >>4 (8-759)

[1096] -yInt L =refy L >>4 (8-760)

[1097] -xFrac L =refx L &15 (8-761)

[1098] -yFrac L =refy L &15 (8-762)

[1099] – Using6TapFlag is set to 1 if all of the following conditions are met:

[1100] –cbWidth[0][xSb][ySb]<=4||cbHeight[0][xSb][ySb]<=4||cbWidth[0][xSb][ySb]*cbHeight[0][xSb][ySb]<=64.

[1101] –PredFlagL0[xSb][ySb]==1&&PredFlagL1[xSb][ySb]==1.

[1102] – If bdofFlag is equal to true or (sps_affine_prof_enabled_flag is equal to true and inter_affine_flag[xSb][ySb] is equal to true), and one or more of the following conditions are true, then the luma integer sample retrieval process is called as specified in clause 8.5.6.3.3 with (xInt L +(xFrac L >>3)-1), yInt L +(yFrac L >>3)-1) and refPicLX as input, derive the predicted brightness sample value predSamplesLX[x L ][y L ].

[1103] 1.x L Equal to 0.

[1104] 2.x L Equal to sbWidth+1.

[1105] 3.y L Equal to 0.

[1106] 4.y L Equal to sbHeight+1.

[1107] – Otherwise, the 8-tap interpolation filter process for luma samples is called by invoking the 8-tap interpolation filter process for luma samples as specified in clause 8.5.6.3.2 with (xIntL-(brdExtSize>0?1:0), yIntL-(brdExtSize>0?1:0)), (xFracL, yFracL), (xSbInt L,ySbInt L ), refPicLX, hpelIfIdx, sbWidth, sbHeight and (xSb, ySb) and using6TapFlag are used as input to derive the predicted luminance sample value predSamplesLX[xL][yL].

[1108] – Otherwise (cIdx is not equal to 0), the following applies:

[1109] – Let (xIntC, yIntC) be the chroma position given in integer sample units, and (xFracC, yFracC) be the offset given in 1 / 32 sample units. These variables are used only in this clause to specify the overall fractional sample position within the reference sample array refPicLX.

[1110] – The top left coordinate (xSbIntC, ySbIntC) of the reference sample filled boundary block is set to ((xSb / SubWidthC)+(mvLX[0]>>5),(ySb / SubHeightC)+(mvLX[1]>>5)).

[1111] For each chroma sample position (xC=0..sbWidth-1, yC=0..sbHeight-1) within the predicted chroma sample array predSamplesLX, the corresponding predicted chroma sample value predSamplesLX[xC][yC] is derived as follows:

[1112] – Make(refxSb C ,refySb C ) and (refx C , refy C ) is the chroma position pointed to by the motion vector (mvLX[0], mvLX[1]) given in 1 / 32 sample units. The variable refxSb C 、refySb C 、refx C and refy C The derivation is as follows:

[1113] -refxSb C =((xSb / SubWidthC<<5)+mvLX[0])*hori_scale_fp (8-763)

[1114] -refx C =((Sign(refxSb C )*((Abs(refxSb C)+256)>>9)+xC*((hori_scale_fp+8)>>4))+16)>>5 (8-764)

[1115] -refySb C =((ySb / SubHeightC<<5)+mvLX[1])*vert_scale_fp (8-765)

[1116] -refy C =((Sign(refySb C )*((Abs(refySb C )+256)>>9)+yC*((vert_scale_fp+8)>>4))+16)>>5 (8-766)

[1117] –Variable xInt C 、yInt C 、xFrac C and yFrac C The derivation is as follows:

[1118] -xInt C =refx C >>5 (8-767)

[1119] -yInt C =refy C >>5 (8-768)

[1120] -xFrac C =refy C &31 (8-769)

[1121] -yFrac C =refy C &31 (8-770)

[1122] – Derives the predicted sample values ​​predSamplesLX[xC][yC] by calling the procedure specified in clause 8.5.6.3.4 with (xIntC, yIntC), (xFracC, yFracC), (xSbIntC, ySbIntC), sbWidth, sbHeight and refPicLX as input.

[1123] 8.5.6.3.2 Luma Sample Interpolation Filtering Process

[1124] The inputs to this process are:

[1125] – Luminance position in units of whole samples (xInt L , yIntL ),

[1126] – Luma position in fractional sample units (xFrac L ,yFrac L ),

[1127] – Luminance position in units of whole samples (xSbInt L ,ySbInt L ), specifies the top left sample of the boundary block filled with reference samples relative to the top left luma sample of the reference picture,

[1128] – Luminance reference sample array refPicLX L ,

[1129] – Half-sample interpolation filter index hpelIfIdx,

[1130] –Variable sbWidth specifies the width of the current sub-block,

[1131] –Variable sbHeight specifies the height of the current sub-block.

[1132] – Luma position (xSb, ySb), specifies the top left sample of the current sub-block relative to the top left luma sample of the current picture,

[1133] – Flag using6TapFlag, specifies whether to use 6-tap interpolation filtering.

[1134] The output of this process is the predicted luminance sample value predSampleLX L

[1135] The variables shift1, shift2, and shift3 are derived as follows:

[1136] – The variable shift1 is set equal to Min(4, BitDepth Y -8), the variable shift2 is set equal to 6, and the variable shift3 is set equal to Max(2,14-BitDepth Y ).

[1137] – The variable picW is set equal to pic_width_in_luma_samples, and the variable picH is set equal to pic_height_in_luma_samples.

[1138] Luminance interpolation filter coefficient f L [p] For each 1 / 16 fractional sample position p is equal to xFrac L or yFrac L The derivation is as follows:

[1139] – If at least one of the following conditions is met, the luminance interpolation filter coefficient f L [p] is specified in Table 8-12.

[1140] –MotionModelIdc[xSb][ySb] is greater than 0, and both sbWidth and sbHeight are equal to 4,

[1141] –using6TapFlag is equal to 1.

[1142] – Otherwise, the brightness interpolation filter coefficient f L [p] is specified in Table 8-11, depending on hpelIfIdx. Luma position in integer sample units (xInt i , yInt i ) for i=0..7 is derived as follows:

[1143] – If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[1144] -xInt i =Clip3(SubPicLeftBoundaryPos, SubPicRightBoundaryPos, xInt L +i-3)(8-771)

[1145] -yInt i =Clip3(SubPicTopBoundaryPos, SubPicBotBoundaryPos, yInt L +i-3)(8-772)

[1146] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[1147] -xInt i =Clip3(0,picW-1,sps_ref_wraparound_enabled_flag?ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY,picW,xInt L +i-3): (8-773)

[1148] xInt L +i-3)

[1149] -yInt i =Clip3(0,picH-1,yInt L +i-3) (8-774)

[1150] The luminance position in integer sample units is further modified for i=0..7 as follows:

[1151] -xInt i =Clip3(xSbInt L -3,xSbInt L +sbWidth+4,xInt i )

[1152] (8-775)

[1153] -yInt i =Clip3(ySbInt L -3,ySbInt L +sbHeight+4,yInt i )

[1154] (8-776)

[1155] Predicted brightness sample value predSampleLX L is derived as follows:

[1156] – If xFrac L and yFrac L If both are equal to 0, then predSampleLX L The value of is derived as follows:

[1157] -predSampleLX L =refPicLX L [xInt3][yInt3]< <shift3 (8-777)

[1158] – Otherwise, if xFrac L Not equal to 0 and yFrac L If equal to 0, then predSampleLX L The value of is derived as follows:

[1159]

[1160] – Otherwise, if xFrac L Equal to 0 and yFrac L If not equal to 0, then predSampleLX L The value of is derived as follows:

[1161]

[1162] – Otherwise, if xFrac L Not equal to 0 and yFrac L If not equal to 0, then predSampleLX L The value of is derived as follows:

[1163] – The sample array temp[n], where n=0..7, is derived as follows:

[1164]

[1165] –Predicted brightness sample value predSampleLX L is derived as follows:

[1166] -

[1167] Table 8-11 – Luma interpolation filter coefficients f for each 1 / 16 fractional sample position p L [p] specification.

[1168]

[1169] Table 8-12 – Luma interpolation filter coefficients f for each 1 / 16 fractional sample position p for affine motion mode L [p] specification.

[1170]

[1171] Figure 29 is a block diagram of a video processing device 2600. Device 2600 can be used to implement one or more of the methods described herein. Device 2600 can be implemented as a smartphone, tablet, computer, Internet of Things (IoT) receiver, and the like. Device 2600 can include one or more processors 2602, one or more memories 2604, and video processing hardware 2606. Processor(s) 2602 can be configured to implement one or more of the methods described herein. Memory(s) 2604 can be used to store data and code to implement the methods and techniques described herein. Video processing hardware 2606 can be used to implement some of the techniques described in this document in hardware circuitry.

[1172] Figure 303000 is a flowchart of an exemplary method of video processing. The method includes determining (3002) a maximum number of candidate MLs in a sub-block Merge candidate list or enabling use of an alternative temporal motion vector prediction (ATMVP) candidate based on a state of use of temporal motion vector prediction (TMVP) or current picture reference codec (CPR) of the current video block for a current video block, wherein the state is one of enabled or disabled, and performing (3004) a conversion between the current video block and a bitstream representation of the current video block based on a result of the determination.

[1173] It should be understood that several technologies have been disclosed that will benefit video encoder and decoder embodiments integrated into video processing devices, such as smartphones, laptops, desktop computers, and similar devices, by allowing the use of ATMVP codec tools in encoding or decoding video or images. Various embodiments and technologies can use the following clause-based description.

[1174] 1. A method of video processing, comprising: determining, for a current video block, a maximum number of candidate MLs in a sub-block Merge candidate list or enabling use of an alternative temporal motion vector prediction (ATMVP) candidate based on a state of use of temporal motion vector prediction (TMVP) or a state of current picture reference codec (CPR) of the current video block, wherein the state is one of enabled or disabled, and performing conversion between the current video block and a bitstream representation of the current video block based on a result of the determination.

[1175] 2. The method of clause 1, wherein enabling usage of an ATMVP candidate is further based on a status of ATMVP usage for the current video block.

[1176] 3. The method of any of clauses 1-2, wherein determining the ML comprises: determining that ATMVP and TMVP are disabled; and determining the ML by excluding ATMVP candidates from a sub-block Merge candidate list.

[1177] 4. The method of any of clauses 1-2, wherein determining the ML comprises:

[1178] Determining that ATMVP is enabled for the current video block, TMVP is disabled for the current video block, and affine codec is disabled for the current video block; and

[1179] Based on this determination, ML is set to zero.

[1180] 5. The method of any of clauses 1-2, wherein determining the ML comprises:

[1181] Determining that ATMVP is enabled for the current video block, TMVP is enabled for the current video block, and affine codec is disabled for the current video block; and

[1182] Based on the determination, ML is set to 1.

[1183] 6. A method according to any of clauses 1-2, wherein determining the ML is based on a status of ATMVP usage for the current block and whether a collocated reference picture of the current picture of the current video block is the current picture.

[1184] 7. The method of any of clauses 1-2, wherein determining the ML is based on a status of the ATMVP and whether the reference picture with reference index 0 in the reference picture list of the current picture of the current video block is the current picture.

[1185] 8. The method of clause 7, wherein the reference picture list is List 0 or List 1.

[1186] Additional description of clauses 1-8 can be found in items 1-2 listed in Section 4.

[1187] 9. A video processing method, comprising:

[1188] Determining the relationship between a temporal motion vector prediction (TMVP) codec and an alternative TMVP (ATMVP) codec for converting between a bitstream representing a sequence of video pictures using block-based coding, where blocks of the video pictures are organized into slices or slices, and pixel values ​​of the video pictures; and

[1189] Based on the determination, conversion is performed;

[1190] where the relationship specifies applicability, such as one or more of the following:

[1191] (a) In case TMVP is not allowed at slice or picture level, then ATMVP is disabled at slice or picture level;

[1192] (b) In case TMVP is signaled at the slice, slice or picture level, ATMVP is signaled after TMVP at the slice, slice or picture level;

[1193] (c) The relationship is specified only at the sequence level and is not included at the slice or tile or picture level.

[1194] 10. A method according to clause 9, wherein the relationship is signalled in a single bit in the bitstream representation.

[1195] Additional description of clauses 9-10 can be found in items 3-4 listed in Section 4.

[1196] 11. A video encoding and decoding method, comprising:

[1197] For conversion between a video region and a bitstream representation, determining an amount of a mask to be applied to the location of coordinates of a current block or current sub-block of a video region for temporal motion vector prediction (TMVP) or advanced temporal motion vector prediction (ATMVP) codecs; and

[1198] By applying a mask based on this amount, conversion between a video region and a corresponding bitstream representation is performed.

[1199] 12. A method according to clause 11, wherein the amount of the mask is signalled in the bitstream representation.

[1200] 13. A method according to clauses 11-12, wherein is set to no mask in case an indication in a signalling parameter set of the bitstream representation indicates motion compression as disabled.

[1201] 14. A method according to any of clauses 11-13, wherein the mask is applied by a bitwise AND operation using an integer MASK having a value equal to ˜(2M−1), where M is an integer and “˜” denotes a bitwise compensation operator.

[1202] 15. A method according to any of clauses 11-14, wherein applying the mask comprises applying the mask resulting in 2K x 2K blocks of video regions sharing the same motion information, where K and M are different integers.

[1203] 16. The method according to any of clauses 11-15, wherein M=3 or M=4.

[1204] 17. A method according to any of clauses 11-16, wherein M is signalled in a bitstream representation.

[1205] Additional description of clauses 11-17 can be found in items 3-4 listed in Section 4.

[1206] 18. A video processing method, comprising:

[1207] During conversion between a current video region and a bitstream representation, determining a valid corresponding region for an alternative temporal motion vector prediction (ATMVP) of the current video region, and

[1208] Performing the conversion based on the valid corresponding area,

[1209] Where determination is based on one or more of the following:

[1210] (a) Adaptive determination based on the characteristics of the current video region;

[1211] (b) a basic region, having size MxN pixels, where M and N are integers;

[1212] (c) determining a motion vector for locating a valid corresponding area, wherein determining the motion vector is based on a rule; or

[1213] (d) Determine the default motion vector for ATMVP codec.

[1214] 19. The method of clause 18, wherein the current video region being a valid corresponding region is a function of: (a) a width or height of the current video region, or (b) a motion vector compression tool applied to the current video region.

[1215] 20. A method according to any of clauses 18-19, wherein the current video region corresponds to a basic region of size MxN which is smaller than a Codec Tree Unit (CTU) region.

[1216] 21. The method of clause 20, wherein the current video block is within a base region, and wherein the valid corresponding region is a co-located base region.

[1217] 22. The method of clause 20, wherein the current video block is within the base region, and wherein the valid corresponding region at least partially does not overlap with the co-located base region.

[1218] 23. A method according to clause 18, wherein the valid corresponding region is in a different picture than the current picture containing the current video region.

[1219] 24. The method of clause 23, wherein the current video region is a current video block, and the temporal motion vector (TMV) corresponding to the current video block is (1) set to a default value, or (2) set to a motion vector in a history-based motion vector predictor table.

[1220] 25. The method of clause 23, wherein determining a default motion vector for the ATMVP codec comprises (1) setting the default motion vector to a value of (0,0), or (2) deriving the default motion vector from a history-based motion vector predictor table, or (3) deriving the default motion vector from a neighboring block.

[1221] Additional description of clauses 18-25 can be found in items 7-10 listed in Section 4.

[1222] 26. A video processing method, comprising:

[1223] Generating a candidate list or motion vectors for a video region using rules; and

[1224] The candidate list performs conversions between video regions and their bitstream representations, where the rules include one or more of the following:

[1225] (a) Rule 1: Candidates corresponding to the alternative temporal motion vector prediction (ATMVP) mode are always considered during conversion;

[1226] (b) Rule 2: The enabled state of the affine mode of the video region is used when determining the availability of the zero-motion affine merge candidate in the sub-block merge candidate list of the sub-block of the video region;

[1227] (c) Rule 3: Use non-affine filling candidates for the sub-block Merge candidate list of the sub-block of the video region; or

[1228] (d) Rule 4: The motion vector of the video region is derived from the motion vectors of the blocks covering the corresponding positions according to the alternative temporal motion vector predictor method.

[1229] 27. The method of clause 26, wherein the video region is a current video block, and rule 1 further specifies determining the ATMVP default motion vector by determining a block at a corresponding position that covers the corresponding position of a center point of the current video block.

[1230] 28. The method of clause 26, wherein rule 2 further specifies avoiding adding a zero-motion affine merge candidate to the sub-block merge candidate list if the affine mode enable state indicates that affine mode is off for the current video block.

[1231] Additional description of clauses 26-28 can be found in items 11-12 listed in Section 4.

[1232] 29. The method of any of clauses 1-28, wherein converting comprises generating a bitstream from the current video block or the current video region.

[1233] 30. A method according to any of clauses 1-28, wherein converting comprises generating pixel values ​​of the current video block or current video region from a bitstream representation.

[1234] 31. The method of any one of clauses 1-28, further comprising determining that a time domain block in a co-located picture associated with the current video block is coded in CPR mode, and wherein a default motion candidate is used instead of an ATMVP candidate based on determining that the time domain block is coded in CPR mode.

[1235] 32. The method of clause 31, wherein the default motion candidate indicates a center position of the current video block.

[1236] 33. The method of clause 31, wherein the default motion candidate is a (0,0) motion vector and the reference picture index is equal to zero for both reference picture lists.

[1237] 34. The method of any of clauses 1-28, further comprising determining a position of a current video block, wherein the used default motion information of the ATMVP candidate is based on the determination of the position of the current video block.

[1238] 35. The method of clause 34, wherein the position is based on a position of a sub-block of the current video block.

[1239] 36. The method of any of clauses 1-28, wherein use of ATMVP candidates is based on a flag provided by a slice, slice, or picture header indicating use of ATMVP candidates.

[1240] 37. The method of clause 36, wherein the current picture containing the current video block is not an intra random access point (IRAP) picture, and wherein the current picture is not inserted into RefPicList0 with a reference index equal to zero.

[1241] 38. The method of any of clauses 1-28, wherein ATMVP or TMVP is enabled or disabled based on a flag being inferred as false for a slice or a slice based on a current picture and a reference picture list X that includes the current video block, the current picture being a reference picture with an index set to M.

[1242] 39. The method of any one of clauses 1-28, further comprising determining a reference picture having an index set to M in a reference picture list X of the current video block as the current picture, and wherein ATMVP is enabled based on determining that the reference picture having an index set to M in a reference picture list X of the current video block is the current picture.

[1243] 40. The method of clause 39, wherein the sub-block motion information is defined to refer to the current picture.

[1244] 41. The method of clause 39, further comprising determining that the sub-block motion information is from a temporal block, wherein the temporal block is encoded using a reference picture that points to a current picture of the temporal block.

[1245] 42. The method of clause 41, wherein the sub-block motion information is not scaled.

[1246] 43. The method of any of clauses 1-28, wherein converting comprises aligning sub-block Merge indices for encoding or decoding.

[1247] 44. The method of clause 43, wherein the first number of bins is context coded and the second number of bins is bypass coded, the second number of bins not being context coded.

[1248] 45. The method of clause 43, wherein the binary number is context-encoded.

[1249] 46. ​​A method for video processing, comprising: determining, for a current video block, a motion vector used to locate corresponding blocks in different pictures in an alternative temporal motion vector prediction (ATMVP) process for conversion between the current video block and a codec representation of the current video block based on a right shift and rounding process, and performing conversion between the current video block and the codec representation of the current video block based on a result of the determination.

[1250] 47. The method of clause 46, wherein the right shift rounding process is the same as the right shift rounding process used for motion vector scaling during conversion.

[1251] 48. The method of clause 46, wherein the right shift rounding process results in rounding to an integer value by rounding towards zero.

[1252] 49. The method of any of clauses 46-48, wherein the motion vectors used in the ATMVP process are used to derive default motion information.

[1253] The following items provide some example aspects of the technology described in the items in the previous section (e.g., items 24-29).

[1254] 50. A method of video processing, comprising: converting between a video picture of a video using sub-pictures and a codec representation of a video segment, determining that a rule of constraints related to sub-blocks is satisfied by the conversion; and performing the conversion according to the rule of constraints.

[1255] 51. A method according to clause 50, wherein the rule of the constraint specifies that the width of the sub-image S ending at the (j-1) column is set to be equal to j minus the leftmost column of the sub-image S, since positions (i, j) and (i, j-1) belong to different sub-images within the video image.

[1256] 52. The method of any of clauses 50-51, wherein the rules of the constraint specify that the height of a sub-picture S ending at a (NumSubPicGridRows-1) row is set to (NumSubPicGridRows-1) minus the topmost row of the sub-picture S plus one.

[1257] 53. The method of any of clauses 50-52, wherein the rules of the constraint specify that the width of a sub-picture S ending at (NumSubPicGridColumns-1) columns is set to (NumSubPicGridColumns-1) minus the leftmost column of the sub-picture S plus 1.

[1258] 54. The method of any of clauses 50-53, wherein the rules of the constraint specify that the size grid of sub-pictures in a video picture is an integer multiple of the codec tree unit size used during conversion.

[1259] 55. The method of any of clauses 50-54, wherein the rules of the constraint specify that all sub-pictures in a video picture do not overlap and together cover the entire video picture.

[1260] The following items provide some example aspects of the technology described in the items in the previous section (e.g., items 30-34).

[1261] 56. A method of video processing, comprising: for converting between a video unit of a video and a codec representation of the video, determining whether a reference picture resampling (RPR) mode is used during the conversion, and performing the conversion based on the determination.

[1262] 57. A method as described in clause 56, wherein a flag in the codec representation indicates use of RPR mode.

[1263] 58. The method of any of clauses 56-57, wherein the RPR mode is indicated at a sequence level or a video parameter set level.

[1264] 59. A method as described in any of clauses 56-59, wherein the RPR mode includes: during conversion, based on whether the resolution of the reference picture is different from the current picture, or whether the width and / or height of the reference picture is larger than the current picture, using interpolation filtering during the motion compensation process to derive a prediction block for the current block of the video unit.

[1265] 60. A method according to any of clauses 1 to 59, wherein converting comprises generating a codec representation from pixel values ​​of the current video block.

[1266] 61. The method of any of clauses 1 to 59, wherein converting comprises pixel values ​​of the current video block from a codec representation.

[1267] 62. A video encoder device comprising a processor configured to implement the method of any of clauses 1 to 61.

[1268] 63. A video decoder device comprising a processor configured to implement the method of any of clauses 1 to 61.

[1269] 64. A computer-readable medium having code stored thereon which, when executed by a processor, causes the processor to carry out the method of any one of clauses 1 to 61.

[1270] Figure 32is a block diagram illustrating an exemplary video processing system 3200 in which the various techniques described herein may be implemented. Various implementations may include some or all of the components of system 3200. System 3200 may include an input 3202 for receiving video content. The video content may be received in a raw or uncompressed format, for example, 8 or 10 bit multi-component pixel values, or may be received in a compressed or encoded format. Input 3202 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces, such as Ethernet, a passive optical network (PON), etc., and wireless interfaces, such as Wi-Fi or a cellular interface.

[1271] System 3200 may include a codec component 3204, which may implement the various codecs or encoding methods described herein. Codec component 3204 may reduce the average bit rate of the video from input 3202 to the output of codec component 3204 to produce a codec representation of the video. Codec techniques are therefore sometimes referred to as video compression or transcoding techniques. The output of codec component 3204 may be stored or transmitted via a connected communication, as represented by component 3206. The stored or communicated bitstream (or codec) representation of the video received at input 3202 may be used by component 3208 to generate pixel values ​​or displayable video that is sent to display interface 3210. The process of generating a user-viewable video from the bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that the codec tools or operations are used by the encoder, and the corresponding decoding tools or operations that reverse the results of the codec are performed by the decoder.

[1272] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), DisplayPort, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The technology described in this document may be implemented in various electronic devices, such as smartphones, laptops, smart phones, or other devices capable of digital data processing and / or video display.

[1273] Some embodiments of the disclosed technology are discussed in a clause-based format.

[1274] Some exemplary embodiments of the technology described in Section 4, Item 22 include:

[1275] A1. A method of visual media processing (e.g., Figure 33), comprising: determining (3302) a motion vector for use in a sub-block based motion vector prediction (sbTMVP) process for a current video block to locate a corresponding block in a co-located picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated based on a scaling operation; and performing (3304) conversion between the current video block and the bitstream representation of the visual media data based on use of the motion vector.

[1276] A2. The method of clause A1, wherein the scaling operation comprises at least one of: a right shift operation, a left shift operation, and / or a rounding operation applied to the motion vector used in the sbTMVP process.

[1277] A3. The method of clause A2, wherein the right shift operation causes the motion vector to be rounded to an integer value by rounding towards zero.

[1278] A4. The method of clause A1, further comprising:

[1279] Upon determining that the dimensions of the collocated picture or the dimensions of the window associated with the collocated picture are different from the dimensions of the current picture associated with the current block or the dimensions of the window associated with the current picture, a scaling operation is applied to the motion vector used in the sbTMVP process.

[1280] A5. The method of clause A4, wherein the result of the scaling operation on the motion vector used in the sbTMVP process is expressed as:

[1281] MVx'=MVx*W1 / W2 and MVy'=MVy*H1 / H2, where MVx, MVy are motion vectors locating a block corresponding to the current video block, where MVx', MVy' are scaled motion vectors of the current video block, where the dimensions of the collocated picture or the dimensions of the window associated with the collocated picture are referred to as W1, H1, and where the dimensions of the current picture or the dimensions of the window associated with the current picture are referred to as W2, H2, respectively.

[1282] Some exemplary embodiments 23 of the technology described in the items of Section 4 include:

[1283] B1. A method of visual media processing (e.g., Figure 34 3400), comprising:

[1284] determining (3402) for a current video block a motion vector used in a sub-block based temporal motion vector prediction (sbTMVP) process to locate a corresponding block in a collocated picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated with respect to a center point of the current video block;

[1285] modifying (3406) the center point of the current video block by applying one or more operations; and

[1286] Converting between the current video block and a bitstream representation of the visual media data is performed (3408) based on using the center point modified by applying the one or more operations.

[1287] B2. The method of clause B1, wherein the one or more operations include a scaling operation and / or an operation corresponding to adding an offset to the center point.

[1288] B3. The method according to clause B1, further comprising:

[1289] Upon determining that the dimensions of the collocated picture or the dimensions of the window associated with the collocated picture are different from the dimensions of the current picture associated with the current block or the dimensions of the window associated with the current picture, additional modification of the center point of the current video block is performed.

[1290] B4. The method of clause B3, wherein the additional modification of the center point of the current video block is expressed as:

[1291] x0′=(x0-X2)*W1 / W2+X1 and y0′=(y0-Y2)*H1 / H2+Y1,

[1292] wherein the center point of the current video block is (x0, y0), and the coordinates of the center point of the current video block after the additional modification are (x0', y0'), wherein the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are referred to as W1, H1, and wherein the dimensions of the current picture or the dimensions of the window associated with the current picture are referred to as W2, H2, respectively, the top left position of the window associated with the current picture is referred to as (X2, Y2), and the top left position of the window associated with the co-located picture is referred to as (X1, Y1).

[1293] B5. The method of clause B3, wherein the additional modification of the center point of the current video block is expressed as:

[1294] x0′=(x0-X2)*W1 / W2 and y0′=(y0-Y2)*H1 / H2,

[1295] wherein the center point of the current video block is (x0, y0), and the coordinates of the center point of the current video block after the additional modification are (x0', y0'), wherein the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are referred to as W1, H1, the left top position of the window associated with the current picture is referred to as (X2, Y2), and wherein the dimensions of the current picture or the dimensions of the window associated with the current picture are referred to as W2, H2, respectively.

[1296] Some exemplary embodiments 24 of the technology described in the items of Section 4 include:

[1297] C1. A method of visual media processing (e.g., Figure 35 3500), comprising:

[1298] determining (3502) for a current video block a motion vector used in a sub-block based temporal motion vector prediction (sbTMVP) process to locate a corresponding block in a collocated picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated with respect to a point in the corresponding block in the collocated picture;

[1299] modifying (3504) the point in the corresponding block in the collocated picture by applying one or more operations; and

[1300] Converting between the current video block and a bitstream representation of the visual media data is performed (3506) based on using the point in the corresponding block in the collocated picture modified by applying the one or more operations.

[1301] C2. The method of clause C1, wherein the one or more operations include a scaling operation and / or an operation corresponding to adding an offset to the point in the corresponding block in the co-located picture.

[1302] C3. The method according to clause C1, further comprising:

[1303] Upon determining that the dimensions of the collocated picture or the dimensions of the window associated with the collocated picture are different from the dimensions of the current picture associated with the current video block or the dimensions of the window associated with the current picture, additional modification of the point in the corresponding block in the collocated picture is performed.

[1304] C4. The method of clause C3, wherein the additional modification of the point in the corresponding block in the co-located picture is expressed as:

[1305] x'=(x-X2)*W1 / W2+X1 and y'=(y-Y2)*H1 / H2+Y1,

[1306] wherein the point in the corresponding block in the co-located picture is (x, y), and the coordinates of the point in the corresponding block in the co-located picture after the additional modification are (x', y'), wherein the width and height of the window associated with the current picture are referred to as W2 and H2, respectively, the width and height of the window associated with the co-located picture are referred to as W1 and H1, respectively, the top left position of the window associated with the current picture is referred to as (X2, Y2), and the top left position of the window associated with the co-located picture is referred to as (X1, Y1).

[1307] C5. The method of clause C3, wherein the additional modification of the point in the corresponding block in the co-located picture is expressed as:

[1308] x'=(x-X2)*W1 / W2 and y'=(y-Y2)*H1 / H2, wherein the point in the corresponding block in the co-located picture is (x, y), and the coordinates of the point in the corresponding block in the co-located picture after the additional modification are (x', y'), wherein the width and height of the window associated with the current picture are referred to as W2 and H2, respectively, the width and height of the window associated with the co-located picture are referred to as W1 and H1, respectively, and the top left position of the window associated with the current picture is referred to as (X2, Y2).

[1309] Some exemplary embodiments 28 of the technology described in the items of Section 4 include:

[1310] D1. A method of visual media processing (e.g., Figure 36 ), comprising: for converting between a video picture contained in visual media data and a bitstream representation of the visual media data using sub-pictures, determining (3602) that a rule relating to one or more sub-pictures is satisfied by the conversion; and performing (3604) the conversion in accordance with the constrained rule, wherein the rule specifies that the size of the sub-pictures in the video picture is an integer multiple of the size of a codec tree unit associated with the video picture.

[1311] D2. The method of clause D1, wherein the rule specifies that the bitstream representation includes a variable corresponding to a width of the sub-picture.

[1312] D3. The method of clause D1, wherein the rule specifies that the bitstream representation includes a variable corresponding to a height of the sub-picture.

[1313] D4. A method according to any one or more of clauses D2-D3, wherein the variable corresponding to the height of the sub-picture and / or the variable corresponding to the width of the sub-picture are expressed in units of luma codec tree unit size.

[1314] D5. A method according to any one or more of clauses D2 and D4, wherein the variable corresponding to the width of the sub-picture is equal to the difference between the width of the sub-picture in units of luma codec treeblock size and 1.

[1315] D6. A method according to any one or more of clauses D3 and D4, wherein the variable corresponding to the height of the sub-picture is equal to the difference between the height of the sub-picture in units of luma codec tree unit size and 1.

[1316] D7. The method of clause D4, wherein the luma codec tree unit size is defined as the width or height of a luma codec treeblock array.

[1317] D8. The method of clause D4, wherein the luma codec tree unit size of a codec tree unit is defined as a variable CtbSizeY expressed as:

[1318] CtbSizeY=(1<<(sps_log2_ctu_size_minus5+5)), where sps_log2_ctu_size_minus5 refers to the size of the syntax element.

[1319] D9. The method of clause D8, wherein the variable corresponding to the width of the sub-picture is calculated as the sub-picture width multiplied by the width of CtbSizeY.

[1320] D10. The method of clause D8, wherein the variable corresponding to the height of the sub-picture is calculated as the sub-picture width multiplied by the height of CtbSizeY.

[1321] D11. The method of clause D1, wherein the rule specifies that a value of a flag indicates use of a sub-picture during the conversion.

[1322] D12. The method of clause D11, wherein the flag is a Boolean value.

[1323] D13. A method according to clause D12, wherein the flag, when taking the value 1, indicates that sub-pictures are used, and the flag, when taking the value 0, indicates that sub-pictures are not used.

[1324] Some exemplary embodiments 29 of the technology described in the items of Section 4 include:

[1325] E1. A method of video processing (eg, Figure 37 ), comprising: for a conversion between a video picture contained in visual media data and a bitstream representation of the visual media data using sub-pictures, determining (3702) that a rule relating to one or more sub-pictures is satisfied by the conversion; and performing (3704) the conversion in accordance with the rule constrained by the constraint, wherein the rule specifies that all sub-pictures in the video picture are non-overlapping and that all sub-pictures in the video picture together cover the entire video picture.

[1326] Some exemplary embodiments 30 of the technology described in the items of Section 4 include:

[1327] F1. A method for visual media processing (e.g., Figure 38 ), comprising: determining (3802) whether to use a reference picture resampling (RPR) technique during a conversion between a video unit of a visual media and a bitstream representation of the visual media data; and performing (3804) the conversion based on the determination, wherein a flag corresponding to the determination is included in the bitstream representation at a sequence parameter set level.

[1328] F2. The method of clause F1, wherein the flag indicates use of the RPR technique, and wherein the flag takes a Boolean value.

[1329] F3. A method according to clause F2, wherein the flag equal to 1 specifies that the spatial resolution of the codec picture with respect to the sequence parameter set is inferred to be changeable, and the flag equal to 0 specifies that the spatial resolution of the codec picture with respect to the sequence parameter set is inferred to be unchanged.

[1330] F4. A method according to clause F2, wherein the flag equal to 1 specifies that the spatial resolution of the picture is changeable within the codec layer video sequence referenced by the sequence parameter set, and the flag equal to 0 specifies that the spatial resolution of the picture is not changed within the codec layer video sequence referenced by the sequence parameter set.

[1331] F5. A method according to any one or more of clauses F1-F4, wherein whether window information for a picture associated with the video unit is included in the bitstream representation is independent of whether the RPR technique is used.

[1332] F6. A method according to any one or more of clauses F1-F4, wherein the height and / or width of the video unit indicated in the picture parameter set (PPS) is inferred to be the same as the height and / or width of the video unit in another picture parameter set (PPS) associated with the same sequence parameter set.

[1333] F7. A method according to any one or more of clauses F1-F6, wherein the bitstream representation is configured to contain window information for pictures associated with the video unit regardless of the value of the flag.

[1334] F8. A method according to any one or more of clauses F1-F6, wherein the bitstream representation is configured to include window information for a picture associated with the video unit, and wherein the flag indicates whether the window information is used during the conversion.

[1335] F9. A method according to any one or more of clauses F1-F4, wherein the height and / or width of the video unit indicated in the picture parameter set (PPS) is inferred to be the same as the height and / or width of the video unit in another picture parameter set (PPS) associated with the same sequence parameter set.

[1336] F10. A method according to any one or more of clauses F-F4, further comprising:

[1337] Upon determining not to use the RPR technique during the conversion, the height and / or width of the video unit indicated in the picture parameter set (PPS) is inferred as a default value for the height and / or a default value for the width of the video unit.

[1338] F11. A method according to clause F10, wherein the default value for the height and / or the default value for the width of the video unit is inferred to be a maximum height and / or maximum width of the video unit indicated in a sequence parameter set (SPS) associated with the video unit.

[1339] Some exemplary embodiments 31 of the technology described in the items of Section 4 include:

[1340] G1. A method of visual media processing (e.g., Figure 39 ), comprising: selecting (3902) an interpolation filter to derive a prediction block for a current block of visual media data during a motion compensation process based on satisfying a condition, wherein the condition is based at least in part on determining that a resolution of a reference picture is different from a resolution of the current picture and / or a dimension of a window associated with the reference picture is different from a dimension of a window associated with the current picture; and performing (3904) a conversion between the current block of visual media data and a bitstream representation of the current block.

[1341] G2. The method of clause G1, further comprising:

[1342] Upon determining that the resolution of the reference picture is different from the resolution of the current picture or the dimensions of the window associated with the reference picture are different from the dimensions of the window associated with the current picture, use an interpolation filter that is different from another filter used when the reference picture has the same resolution as the current picture.

[1343] G3. A method according to clause G1, wherein the condition is based on determining W1>a*W2 and / or H1>b*H2, where (W1, H1) represents the width and height of the reference image or the window associated with the reference image, and (W2, H2) represents the width and height of the current image or the window associated with the reference image, and a and b are scaling factors.

[1344] G4. A method according to clause G3, wherein a=b=1.5.

[1345] G5. A method as described in clause G1, wherein the condition is further based on the height and / or width of the current block.

[1346] G6. The method of clause G5, wherein the height and / or width of the current block individually fulfill one or more threshold conditions.

[1347] G7. A method according to clause G5, wherein a mathematical combination of the height and / or width of the current block implements one or more threshold conditions.

[1348] G8. A method according to clause G6, wherein achieving the one or more threshold conditions includes the height and / or width of the current block exceeding the one or more threshold conditions.

[1349] G9. A method according to clause G6, wherein achieving the one or more threshold conditions includes the height and / or width of the current block missing the one or more threshold conditions.

[1350] G10. The method of clause G7, wherein achieving the one or more threshold conditions comprises a mathematical combination of a height and / or a width of the current block exceeding the one or more threshold conditions.

[1351] G11. The method of clause G7, wherein achieving the one or more threshold conditions comprises a mathematical combination of a height and / or a width of the current block missing the one or more threshold conditions.

[1352] G12. A method as described in any one or more of clauses G1-G11, wherein the selected interpolation filter is applied exclusively to chroma color components of the current video block.

[1353] G13. A method as described in any one or more of clauses G1-G11, wherein the selected interpolation filter is applied exclusively to the luma color component of the current video block.

[1354] G14. A method according to any one or more of clauses G1-G11, wherein the selected interpolation filtering is applied exclusively in the horizontal filtering direction.

[1355] G15. A method according to any one or more of clauses G1-G11, wherein the selected interpolation filtering is applied exclusively to the vertical filtering direction.

[1356] G16. A method according to any one or more of clauses G1-G11, wherein single-tap filtering is selected as the interpolation filtering, so that the result of applying the interpolation filtering is equivalent to replicating integer samples.

[1357] G17. A method according to any one or more of clauses G1-G11, wherein 4-tap interpolation filtering or 6-tap interpolation filtering is selected when the condition is met.

[1358] G18. A method according to any one or more of clauses G1-G11, wherein bilinear filtering is selected as the interpolation filtering when the condition is met.

[1359] Some exemplary embodiments 32 of the technology described in the items of Section 4 include:

[1360] H1. A method for visual media processing (e.g., Figure 40 ), comprising: converting between a current block of visual media data and a bitstream representation of the visual media data, determining (4002) that the current block is a combined inter-intra prediction (CIIP) block, wherein an intra prediction block of the CIIP block is generated using a size of a transform unit (TU), wherein, in the combined inter-intra prediction (CIIP) block, a final prediction of the current block is based on a weighted sum of an inter prediction of the current block and an intra prediction of the current block; and performing (4004) the converting based on the determination.

[1361] H2. The method of clause H1, further comprising:

[1362] Upon determining that the dimension of the codec unit associated with the current block exceeds the maximum size of a transform unit, splitting the codec unit into a plurality of transform units; and

[1363] An intra prediction block and an inter prediction block are generated for each transformation unit included in the plurality of transformation units.

[1364] H3. The method according to clause H2, further comprising:

[1365] Upon determining that a maximum size of a transform unit is smaller than a threshold and a dimension of the codec unit associated with the current block exceeds the maximum size of the transform unit, recursively splitting the codec unit into a plurality of transform units; and

[1366] An intra prediction block and an inter prediction block are generated for each transformation unit included in the plurality of transformation units.

[1367] H4. The method of clause H3, wherein a first intra-frame prediction block is based on reconstructed samples of a second intra-frame prediction block, wherein the first intra-frame prediction block and the second intra-frame prediction block are generated based on partitioning the current block.

[1368] I1. A method as described in any one or more of clauses A1 to H4, wherein the converting includes generating the bitstream representation from the current video block.

[1369] I2. A method as recited in any one or more of clauses A1 to H4, wherein the converting comprises generating samples for the current video block from the bitstream representation.

[1370] I3. A video encoder device comprising a processor configured to implement the method according to any one or more of clauses A1 to H4.

[1371] I4. A video decoder device comprising a processor configured to implement the method according to any one or more of clauses A1 to H4.

[1372] I5. A computer-readable medium having code stored thereon which, when executed by a processor, causes the processor to implement the method according to any one or more of clauses A1 to H4.

[1373] I6. A computer-readable storage medium storing a bitstream representation generated according to the method described in any one or more of clauses A1 to H4.

[1374] Figure 41 is a block diagram illustrating an exemplary video coding system 100 , which may utilize the techniques of this disclosure.

[1375] like Figure 41 As shown, the video coding system 100 may include a source device 110 and a destination device 120. The source device 110 generates encoded video data and may be referred to as a video encoding device. The destination device 120 may decode the encoded video data generated by the source device 110 and may be referred to as a video decoding device.

[1376] Source device 110 may include a video source 112 , a video encoder 114 , and an input / output (I / O) interface 116 .

[1377] Video source 112 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of such sources. The video data may include one or more pictures. Video encoder 114 encodes the video data from video source 112 to generate a bitstream. The bitstream may include a sequence of bits forming a codec representation of the video. The bitstream may include codec pictures and associated data. A codec picture is a codec representation of a picture. The associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. I / O interface 116 may include a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be transmitted directly to destination device 120 via network 130a via I / O interface 116. The encoded video data may also be stored on storage medium / server 130b for access by destination device 120.

[1378] Destination device 120 may include an I / O interface 126 , a video decoder 124 , and a display device 122 .

[1379] I / O interface 126 may include a receiver and / or a modem. I / O interface 126 may obtain encoded video data from source device 110 or storage medium / server 130b. Video decoder 124 may decode the encoded video data. Display device 122 may display the decoded video data to a user. Display device 122 may be integrated with destination device 120 or may be external to destination device 120, with destination device 120 configured to interface with an external display device.

[1380] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other current and / or other standards.

[1381] Figure 42 is a block diagram illustrating an example of a video encoder 200, which may be Figure 41 The video encoder 114 in the system 100 is shown in FIG.

[1382] Video encoder 200 may be configured to perform any or all of the techniques of this disclosure. Figure 42 In the example of FIG, the video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 200. In some embodiments, the processor can be configured to perform any or all of the techniques described in this disclosure.

[1383] The functional components of the video encoder 200 may include a segmentation unit 201, a prediction unit 202 which may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214.

[1384] In other examples, the video encoder 200 may include more, fewer, or different functional components. In an example, the prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode where at least one reference picture is the picture in which the current video block is located.

[1385] Additionally, some components such as the motion estimation unit 204 and the motion compensation unit 205 may be highly integrated, but in Figure 5 are shown separately in the examples for explanation purposes.

[1386] The segmentation unit 201 may segment a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[1387] The mode selection unit 203 may select one of the coding modes (intra or inter) based on the error result, for example, and provide the resulting intra or inter coded block to the residual generation unit 207 to generate residual block data and to the reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 203 may select a combined intra and inter prediction (CIIP) mode, where the prediction is based on inter prediction signaling and intra prediction signaling. The mode selection unit 203 may also select the resolution of the motion vector of the block (e.g., sub-pixel or integer pixel precision) in the case of inter prediction.

[1388] To perform inter-frame prediction on the current video block, motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from buffer 213. Motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from buffer 213 other than the picture associated with the current video block.

[1389] Motion estimation unit 204 and motion compensation unit 205 may operate differently on the current video block, eg, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[1390] In some embodiments, the motion estimation unit 204 may perform unidirectional prediction on the current video block and search for a reference video block for the current video block in the reference pictures in list 0 or list 1. The motion estimation unit 204 may then generate a reference index indicating the reference picture in list 0 or list 1 containing the reference video block and a motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. The motion compensation unit 205 may generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[1391] In other examples, the motion estimation unit 204 may perform bidirectional prediction for the current video block. The motion estimation unit 204 may search for a reference video block for the current video block in the reference pictures in list 0 and may also search for another reference video block for the current video block in the reference pictures in list 1. The motion estimation unit 204 may then generate reference indexes indicating the reference pictures in list 0 and list 1 containing the reference video blocks and a motion vector indicating the spatial displacement between the reference video blocks and the current video block. The motion estimation unit 204 may output the reference index and motion vector for the current video block as motion information for the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information for the current video block.

[1392] In some embodiments, motion estimation unit 204 may output a complete set of motion information to the decoder's decoding process.

[1393] In some embodiments, motion estimation unit 204 may not output a complete set of motion information for the current video block. Instead, motion estimation unit 204 may reference motion information of another video block to signal the motion information of the current video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of an adjacent video block.

[1394] In one example, motion estimation unit 204 may indicate a value in a syntax structure associated with the current video block that indicates to video decoder 300 that the current video block has the same motion information as another video block.

[1395] In another example, the motion estimation unit 204 can identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[1396] As discussed above, the video encoder 200 may predictively signal motion vectors.Two examples of predictive signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and Merge mode signaling.

[1397] The intra-frame prediction unit 206 can perform intra-frame prediction on the current video block. When the intra-frame prediction unit 206 performs intra-frame prediction on the current video block, the intra-frame prediction unit 206 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block can include a predicted video block and various syntax elements.

[1398] The residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the predicted video block(s) of the current video block from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of samples in the current video block.

[1399] In other examples, there may be no residual data for the current video block (eg, in skip mode), and the residual generation unit 207 may not perform the subtraction operation.

[1400] Transform processing unit 208 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[1401] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[1402] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current block for storage in the buffer 213.

[1403] After the reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[1404] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream containing the entropy-coded data.

[1405] Figure 43 is a block diagram illustrating an example of a video decoder 300, which may be Figure 41 The video decoder 114 in the system 100 is shown in FIG.

[1406] Video decoder 300 can be configured to perform any or all of the techniques of this disclosure. Figure 43 In the example of FIG, video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of video decoder 300. In some embodiments, a processor can be configured to perform any or all of the techniques described in this disclosure.

[1407] exist Figure 43 In the example of FIG, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306, as well as a buffer 307. In some embodiments, the video decoder 300 can perform a decoding pass that is generally inverse to the process described with respect to the video encoder 200 (e.g., Figure 42 ) is passed through the encoding described.

[1408] The entropy decoding unit 301 can retrieve a coded bitstream. The coded bitstream can include entropy-encoded video data (e.g., coded blocks of video data). The entropy decoding unit 301 can decode the entropy-encoded video data, and the motion compensation unit 302 can determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information from the entropy-decoded video data. The motion compensation unit 302 can determine such information, for example, by performing AMVP and Merge modes.

[1409] The motion compensation unit 302 may generate a motion compensated block, possibly interpolated based on an interpolation filter. An identifier of an interpolation filter to be used with sub-pixel precision may be included in a syntax element.

[1410] Motion compensation unit 302 may use interpolation filtering to calculate interpolated values ​​for sub-integer pixels of a reference block, as used by video encoder 20 during encoding of the video block. Motion compensation unit 302 may determine the interpolation filtering used by video encoder 200 based on received syntax information and use the interpolation filtering to produce a predictive block.

[1411] The motion compensation unit 302 may use some syntax information to determine the size of blocks used to encode frame(s) and / or slice(s) of the coded video sequence, partitioning information describing how each macroblock of a picture of the coded video sequence is partitioned, a mode indicating how each partition is to be encoded, one or more reference frames (and reference frame lists) for each inter-coded block, and other information for decoding the coded video sequence.

[1412] The intra prediction unit 303 can use, for example, an intra prediction mode received in the bitstream to form a prediction block from spatially adjacent blocks. The inverse quantization unit 303 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[1413] The reconstruction unit 306 can add the residual block to the corresponding prediction block generated by the motion compensation unit 202 or the intra prediction unit 303 to form a decoded block. If desired, a deblocking filter can also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in the buffer 307, which provides a reference block for subsequent motion compensation.

[1414] In this document, the terms "video processing" or "visual media processing" may refer to video encoding, video decoding, video compression or video decompression. For example, a video compression algorithm may be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. The bitstream representation of the current video block may, for example, correspond to bits that are co-located within the bitstream or distributed at different locations in the bitstream, as defined by the syntax. For example, a macroblock may be encoded based on error residual values ​​from a transform and a codec and also using bits in the header and other fields in the bitstream. Furthermore, as described in the above solution, during conversion, the decoder may parse the bitstream based on a determination with the knowledge that some fields may or may not be present. Similarly, the encoder may determine that certain syntax fields are to be included or not included and generate the codec representation accordingly by including or excluding syntax fields from the codec representation.

[1415] The disclosed and other solutions, examples, embodiments, modules, and functional operations described herein can be implemented in digital electronic circuitry or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or any combination thereof. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by a data processing apparatus or to control the operation of the data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that implements a machine-readable propagated signal, or any combination thereof. The term "data processing apparatus" encompasses all devices, apparatus, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or any combination thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device.

[1416] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language (including compiled or interpreted languages) and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[1417] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[1418] For example, processors suitable for executing a computer program include general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, receive data from or transfer data to one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated into, special-purpose logic circuitry.

[1419] Although this patent document includes many details, these details should not be interpreted as limitations on any subject matter or the scope of the claims, but rather as descriptions of features specific to particular embodiments of a particular technology. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Moreover, although features may be described above as functioning in certain combinations and even initially claimed as such, in some cases one or more features in the claimed combination may be excluded from the combination, and the claimed combination may be used in subcombinations or variations of the subcombinations.

[1420] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[1421] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method for visual media processing, comprising: For converting between a video picture contained in visual media data and a bitstream of the visual media data using sub-pictures, determining that a rule associated with one or more sub-pictures is satisfied by the conversion; and performing the conversion according to the rule of the constraint, wherein the rule specifies that the size of the sub-picture in the video picture is an integer multiple of the size of the coding tree unit associated with the video picture; wherein the variable corresponding to the height of the sub-picture and / or the variable corresponding to the width of the sub-picture is expressed in units of luma coding tree unit size; Among them, the variable corresponding to the width of the sub-picture is equal to the difference between the width of the sub-picture in units of the brightness coding tree unit size and 1, and the variable corresponding to the height of the sub-picture is equal to the difference between the height of the sub-picture in units of the brightness coding tree unit size and 1.

2. The method of claim 1, wherein the rule specifies that the bitstream includes the variable corresponding to the width of the sub-picture. 3 . The method of claim 1 , wherein the rule specifies that the bitstream includes the variable corresponding to the height of the sub-picture. 4 . The method of claim 1 , wherein the luma coding tree unit size is defined as a width or a height of a luma coding tree block array.

5. The method of claim 1 , wherein the luma coding tree unit size of a coding tree unit is defined as a variable CtbSizeY, expressed as: CtbSizeY=(1<<(sps_log2_ctu_size_minus5+5)), where sps_log2_ctu_size_minus5 refers to the size of the syntax element. 6 . The method of claim 5 , wherein the variable corresponding to the width of the sub-picture is calculated as the width of the sub-picture multiplied by the width of CtbSizeY. 7 . The method of claim 5 , wherein the variable corresponding to the height of the sub-picture is calculated as the height of the sub-picture multiplied by the height of CtbSizeY.

8. The method of claim 1, wherein the rule specifies a value of a flag that indicates use of the sub-picture during the conversion. The method of claim 8 , wherein the flag is a Boolean value.

10. The method according to claim 9, wherein: When the flag takes a value of 1, it indicates that the sub-picture is used, and when the flag takes a value of 0, it indicates that the sub-picture is not used.

11. The method according to any one of claims 1 to 10, wherein the converting comprises generating the bitstream from the video pictures.

12. The method of any one of claims 1 to 10, wherein the converting comprises generating the video pictures from the bitstream.

13. A video processing method, comprising: For a conversion between a video picture contained in visual media data and a bitstream of the visual media data using sub-pictures, determining that a rule associated with one or more sub-pictures is satisfied by the conversion; and performing the conversion according to the constraint rule, wherein the rule specifies that all sub-pictures in the video picture are non-overlapping and all sub-pictures in the video picture together cover the entire video picture; wherein the variable corresponding to the height of the sub-picture and / or the variable corresponding to the width of the sub-picture is expressed in units of luma coding tree unit size; Among them, the variable corresponding to the width of the sub-picture is equal to the difference between the width of the sub-picture in units of the brightness coding tree unit size and 1, and the variable corresponding to the height of the sub-picture is equal to the difference between the height of the sub-picture in units of the brightness coding tree unit and 1.

14. A video encoder device comprising a processor configured to implement the method according to any one of claims 1 to 13.

15. A video decoder device comprising a processor configured to implement the method according to any one of claims 1 to 13.

16. A computer-readable medium having code stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 13.

17. A method for storing a bitstream of a video, comprising: For visual media data using sub-pictures, determining that a rule associated with one or more sub-pictures is satisfied; generating the bitstream according to the rule of the constraint, wherein the rule specifies that a size of a sub-picture in the video picture is an integer multiple of a coding tree unit size associated with the video picture; storing the bitstream in a non-transitory computer-readable recording medium; as well as wherein the variable corresponding to the height of the sub-picture and / or the variable corresponding to the width of the sub-picture is expressed in units of luma coding tree unit size; Among them, the variable corresponding to the width of the sub-picture is equal to the difference between the width of the sub-picture in units of the brightness coding tree unit size and 1, and the variable corresponding to the height of the sub-picture is equal to the difference between the height of the sub-picture in units of the brightness coding tree unit size and 1.

Images