Scaling method based on sub-block inter-frame prediction

Abstract

A method of visual media processing, comprising: determining, for a current video block, a motion vector used in a sub-block based motion vector prediction (sbTMVP) process to locate a corresponding block in a co-located picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated based on a scaling operation; and performing conversion between the current video block and a bitstream representation of visual media data based on using the motion vector.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application timely claims priority to and the benefit of International Patent Application No. PCT / CN2019 / 107159 filed on September 22, 2019, under applicable patent law and / or under the rules of the Paris Convention. The entire disclosure of the foregoing application is incorporated by reference as part of the disclosure of this application for all purposes under U.S. patent law. Technical Field

[0003] This patent document relates to image and video encoding and decoding technology, devices and systems. Background Art

[0004] Digital video occupies the largest amount of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, it is expected that the bandwidth requirements used by digital video will continue to grow. Summary of the invention

[0005] The techniques of this disclosure may be used by video decoder or encoder embodiments during video decoding or encoding using alternative temporal motion vector prediction.

[0006] In one exemplary aspect, a video processing method is disclosed. The method includes determining, for a current video block, a motion vector used in a sub-block based motion vector prediction (sbTMVP) process to locate a corresponding block in a co-located picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated according to a scaling operation; and performing conversion between the current video block and a bitstream representation of visual media data based on using the motion vector.

[0007] In another exemplary aspect, another video processing method is disclosed. The method includes determining, for a current video block, a motion vector used in a sub-block based temporal motion vector prediction (sbTMVP) process to locate a corresponding block in a co-located picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated with respect to a center point of the current video block; modifying the center point of the current video block by applying one or more operations; and performing conversion between the current video block and a bitstream representation of visual media data based on using the center point modified by applying the one or more operations.

[0008] In yet another exemplary aspect, another video processing method is disclosed. The method includes determining, for a current video block, a motion vector used in a sub-block based temporal motion vector prediction (sbTMVP) process to locate a corresponding block in a co-located picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated with respect to a point in the corresponding block in the co-located picture; modifying the point in the corresponding block in the co-located picture by applying one or more operations; and performing conversion between the current video block and the bitstream representation of the visual media data based on using the point in the corresponding block in the co-located picture modified by applying the one or more operations.

[0009] In yet another aspect, another method of video processing is disclosed. The method includes: for converting between a video picture included in visual media data and a bitstream representation of the visual media data using sub-pictures, determining that a rule related to one or more sub-pictures is satisfied by the conversion; and performing the conversion according to a constrained rule, wherein the rule specifies that a size of the sub-picture in the video picture is an integer multiple of a codec tree unit size associated with the video picture.

[0010] In yet another exemplary aspect, another method of video processing is disclosed. The method includes: for a conversion between a video picture included in visual media data and a bitstream representation of the visual media data using sub-pictures, determining that a rule related to one or more sub-pictures is satisfied by the conversion; and performing the conversion according to a constrained rule, wherein the rule specifies that all sub-pictures in the video picture do not overlap and all sub-pictures in the video picture together cover the entire video picture.

[0011] In yet another exemplary aspect, another method of video processing is disclosed. The method includes: determining whether to use a reference picture resampling (RPR) technique during conversion between a video unit of visual media and a bitstream representation of visual media data; performing the conversion based on the determination, wherein a flag corresponding to the determination is included in the bitstream representation at a sequence parameter set level.

[0012] In yet another exemplary aspect, another method of video processing is disclosed. The method includes selecting an interpolation filter during a motion compensation process to derive a prediction block for a current block of visual media data based on satisfying a condition, wherein the condition is based at least in part on determining that a resolution of a reference picture is different from a resolution of a current picture and / or a dimension of a window associated with the reference picture is different from a dimension of a window associated with the current picture; and performing a conversion between the current block of visual media data and a bitstream representation of the current block.

[0013] In yet another exemplary aspect, another method of video processing is disclosed. The method includes: converting between a current block of visual media data and a bitstream representation of the visual media data, determining that the current block is a combined inter-frame intra-frame prediction (CIIP) block, wherein the intra-frame prediction block of the CIIP block is generated using the size of a transform unit (TU), wherein in the combined inter-frame intra-frame prediction (CIIP) block, a final prediction of the current block is based on a weighted sum of an inter-frame prediction of the current block and an intra-frame prediction of the current block; and performing the conversion based on the determination.

[0014] In yet another exemplary aspect, a video encoder device is disclosed. The video encoder device includes a processor configured to implement the method described herein.

[0015] In yet another exemplary aspect, a video decoder device is disclosed. The video decoder device comprises a processor configured to implement the method described herein.

[0016] In yet another aspect, a computer readable medium is disclosed having code stored thereon. When the code is executed by a processor, the processor is caused to implement the methods described in this document.

[0017] These and other aspects are described in this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is an example of the derivation process for Merge candidate list construction.

[0019] Figure 2 Exemplary locations of spatial merge candidates are shown.

[0020] Figure 3 An example of candidate pairs considered for redundancy checking of spatial merge candidates is shown.

[0021] Figure 4A-4B Exemplary locations of the second PU for Nx2N and 2NxN partitions are shown.

[0022] Figure 5 FIG. 4 is an exemplary illustration of motion vector scaling of a temporal merge candidate.

[0023] Figure 6 Exemplary candidate positions of the time-domain Merge candidate, C0 and C1, are shown.

[0024] Figure 7 An example of combining bi-predictive Merge candidates is shown.

[0025] Figure 8 An exemplary derivation process of motion vector prediction candidates is shown.

[0026] Fig. 9 is an exemplary illustration of motion vector scaling of spatial motion vector candidates.

[0027] Fig.10 An example of alternative temporal motion vector prediction (ATMVP) motion prediction for a CU is shown.

[0028] Fig.11 An example of one CU having four sub-blocks (AD) and its neighboring blocks (ad) is shown.

[0029] Fig.12 Flowchart of an example of encoding with different MV precisions

[0030] Figures 13A-13B A 135 degree partition type (partitioned from the top left corner to the bottom right corner) and a 45 degree partition mode are shown. Illustration of partitioning a CU into two triangular prediction units (two partition modes).

[0031] Fig.14 An example of the positions of adjacent blocks is shown.

[0032] Fig.15 Examples of top and left blocks of video blocks are shown.

[0033] Figures 16A-16B Examples of 2 control point motion vector (CPMV) and 3 CPMV are shown.

[0034] Fig.17 An example of the affine MVF for each sub-block is shown.

[0035] Figures 18A-18B Examples of 4 and 6 parameter affine models are shown.

[0036] Fig.19 It is the MVP of AF_INTER that inherits the affine candidate.

[0037] Fig. 20 An example of building an affine motion predictor in AF_INTER is shown.

[0038] Figures 21A-21B An example of control point motion vectors in affine codec in AF_MERGE is shown.

[0039] Fig. 22 An example of candidate positions for the affine merge mode is shown.

[0040] Fig.23 An example of an intra picture block copy operation is shown.

[0041] Fig.24 An example of valid corresponding areas in a co-located picture is shown.

[0042] Fig.25 An exemplary flow chart for history-based motion vector prediction is shown.

[0043] Fig.26 The modified Merge list building process is shown.

[0044] Fig. 27 An exemplary embodiment of proposing a valid region when a current block is within a basic region is shown.

[0045] Fig.28 An exemplary embodiment of a valid area when the current block is not within the basic area is shown.

[0046] Fig.29 is a block diagram of an example of a video processing device.

[0047] Fig.30 is a flow chart of an example of a video processing method.

[0048] Fig.31A An example of the location of the identification of default motion information in the current standard is shown.

[0049] Fig.31B An example of the location of an identification of default motion information in the proposed standard is shown.

[0050] Fig.32 is a block diagram of an exemplary video processing system in which the disclosed technology may be implemented.

[0051] Fig.33 is a flow chart of an example of a visual media processing method.

[0052] Fig.34 is a flow chart of an example of a visual media processing method.

[0053] Fig.35 is a flow chart of an example of a visual media processing method.

[0054] Fig.36 is a flow chart of an example of a visual media processing method.

[0055] Fig.37 is a flow chart of an example of a visual media processing method.

[0056] Fig.38 is a flow chart of an example of a visual media processing method.

[0057] Fig.39 is a flow chart of an example of a visual media processing method.

[0058] Fig.40 is a flow chart of an example of a visual media processing method.

[0059] Fig.41 is a block diagram illustrating an exemplary video coding system.

[0060] Fig.42 is a block diagram illustrating an encoder according to some embodiments of the present disclosure.

[0061] Fig.43 is a block diagram illustrating a decoder according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0062] This document provides various techniques that can be used by a decoder of a video bitstream to improve the quality of a decompressed or decoded digital video or image. In addition, a video encoder can also implement these techniques during the encoding process to reconstruct decoded frames for further encoding.

[0063] The section headings are used in this document to facilitate understanding and do not limit the embodiments and techniques to the corresponding sections. Therefore, embodiments from one section can be combined with embodiments from other sections.

[0064] 1. Overview

[0065] This patent document relates to video coding technology. Specifically, it relates to motion vector coding in video coding. It can be applied to existing video coding standards such as HEVC or standards to be finalized (universal video coding). It can also be applied to future video coding standards or video encoders.

[0066] 2. Introduction

[0067] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced the H.261 and H.263 standards, ISO / IEC produced the MPEG-1 and MPEG-4 Visual standards, and the two organizations jointly produced the H.262 / MPEG-2 Video standard and the H.264 / MPEG-4 Advanced Video Coding (AVC) standard and the H.265 / HEVC standard. Starting with H.262, video codec standards are based on a hybrid video codec structure that utilizes temporal prediction plus transform coding. In order to explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, JVET has adopted many new methods and incorporated them into reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Team (JVET) between VCEG (Q6 / 16) and ISO / IEC JTC1SC29 / WG11 (MPEG) was established to work on the VVC standard, which reduces the bit rate by 50% compared to HEVC.

[0068] The latest version of the VVC draft, Versatile Video Codec (Draft 3), can be found at: http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 12_Macao / wg11 / JVET-L1001-v2.zip.

[0069] The latest reference software for VVC named VTM can be found at: https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / tags / VTM-3.0rc1

[0070] 2.1 Inter-frame prediction in HEVC / H.265

[0071] Each inter-prediction PU has motion parameters for one or two reference picture lists. The motion parameters contain motion vectors and reference picture indices. The use of one of the two reference picture lists can also be signaled using inter_pred_idc. Motion vectors can be explicitly encoded as deltas relative to the predictor.

[0072] When a CU is encoded using skip mode, a PU is associated with the CU and there are no significant residual coefficients, no coded motion vector increments or reference picture indices. A Merge mode is specified, whereby motion parameters for the current PU are obtained from neighboring PUs, including spatial and temporal candidates. Merge mode can be applied to any inter-predicted PU, not just skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where the motion vector (more precisely, the motion vector difference (MVD) compared to the motion vector predictor), the corresponding reference picture index for each reference picture list, and the reference picture list usage are explicitly signaled for each PU. Such a mode is named advanced motion vector prediction in the present disclosure.

[0073] When signaling indicates that one of the two reference picture lists is to be used, the PU is generated from a block of samples. This is usually called "unidirectional prediction". Unidirectional prediction is available for P slices and B slices.

[0074] When signaling indicates that two reference picture lists are to be used, the PU is generated from two sample blocks. This is called "bi-prediction". Bi-prediction is only available for B slices.

[0075] The following provides detailed information on the inter prediction modes specified in HEVC. The description will start with the Merge mode.

[0076] 2.1.1 Reference Image List

[0077] In HEVC, the term inter prediction is used to refer to predictions derived from data elements (e.g., sample values ​​or motion vectors) of reference pictures other than the currently decoded picture. As in H.264 / AVC, a picture can be predicted from multiple reference pictures. Reference pictures used for inter prediction are organized in one or more reference picture lists. A reference index identifies which reference pictures in the list should be used to create a prediction signal.

[0078] A single reference picture list, List 0, is used for P slices, and two reference picture lists, List 0 and List 1, are used for B slices. The reference pictures contained in List 0 / 1 can be pictures from the past and the future in terms of capture / display order.

[0079] 2.1.2 Merge mode

[0080] 2.1.2.1 Merge Mode Candidate Derivation

[0081] When using Merge mode to predict a PU, an index pointing to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list is specified in the HEVC standard and can be summarized in the following order:

[0082] Step 1: Initial candidate derivation

[0083] Step 1.1: Spatial Candidate Derivation

[0084] Step 1.2: Redundancy check of spatial candidates

[0085] Step 1.3: Time Domain Candidate Derivation

[0086] Step 2: Additional candidate insertions

[0087] Step 2.1: Create bidirectional prediction candidates

[0088] Step 2.2: Insert zero motion candidates

[0089] exist Figure 1 These steps are also schematically depicted in . For spatial Merge candidate derivation, up to four Merge candidates are selected from candidates located at five different positions. For temporal Merge candidate derivation, at most one Merge candidate is selected from two candidates. Since the number of candidates for each PU is assumed to be constant at the decoder, when the number of candidates obtained from step 1 does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header, additional candidates are generated. Since the number of candidates is constant, binary unary truncation (TU) is used to encode the index of the best Merge candidate. If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

[0090] Hereinafter, operations associated with the aforementioned steps are described in detail.

[0091] 2.1.2.2 Spatial Candidate Derivation

[0092] In the derivation of spatial domain merge candidates, select Figure 2There are up to four Merge candidates among the candidates for the positions illustrated in . The order of derivation is A1, B1, B0, A0 and B2. Position B2 is considered only when any PU at position A1, B1, B0, A0 is unavailable (for example, because it belongs to another stripe or slice) or is intra-coded. After the candidate at position A1 is added, the remaining candidates are added and subjected to a redundancy check, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only pairs with Figure 3 The pairs linked by arrows in are considered, and candidates are added to the list only if the corresponding candidates for redundancy checking do not have the same motion information. Another source of duplicate motion information is a "second PU" associated with a partition different from 2Nx2N. As an example, Figure 4A-4B The second PU for the N×2N and 2N×N cases is illustrated, respectively. When the current PU is partitioned into N×2N, the list construction does not consider the candidate at position A1. In fact, adding this candidate will result in two prediction units with the same motion information, which is redundant for only one PU in the codec unit. Similarly, position B1 is not considered when the current PU is partitioned into 2N×N.

[0093] 2.1.2.3 Time Domain Candidate Derivation

[0094] In this step, only one candidate is added to the list. In particular, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the co-located PU, which belongs to the picture with the smallest POC difference relative to the current picture in a given reference picture list. The reference picture list used for the derivation of the co-located PU is explicitly signaled in the slice header. Figure 5 As shown in the dashed line, a derivation of a scaled motion vector for a temporal Merge candidate is obtained, which is scaled from the motion vector of the co-located PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal Merge candidate is set equal to zero. The actual implementation of the scaling process is described in the HEVC specification. For B slices, two motion vectors are obtained and combined to produce a bidirectional prediction Merge candidate, one of which is used for reference picture list 0 (list 0) and the other for reference picture list 1 (list 1).

[0095] Figure 5 It is a diagram of the motion vector scaling of the temporal merge candidate.

[0096] In the co-located PU (Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C0 and C1, such as Figure 6 As shown. If the PU at position C0 is not available, is intra-coded, or is outside the current codec tree unit (CTU, also known as LCU, largest codec unit) row, position C1 is used. Otherwise, position C0 is used in the derivation of the time domain Merge candidate.

[0097] Figure 6 Examples of candidate positions of the time-domain Merge candidate, C0 and C1, are shown.

[0098] 2.1.2.4 Additional candidate insertion

[0099] In addition to spatial and temporal Merge candidates, there are two additional types of Merge candidates: combined bidirectional prediction Merge candidates and zero Merge candidates. Combined bidirectional prediction Merge candidates are generated by utilizing spatial and temporal Merge candidates. Combined bidirectional prediction Merge candidates are only used for B slices. Combined bidirectional prediction candidates are generated by combining the first reference picture list motion parameters of the initial candidate with the second reference picture list motion parameters of another candidate. If the two tuples provide different motion hypotheses, they will form a new bidirectional prediction candidate. As an example, Figure 7 The following situation is shown, where two candidates with mvL0 and refIdxL0 or mvL1 and refIdxL1 in the original list (on the left) are used to create a combined bi-predictive Merge candidate, which is added to the final list (on the right). There are many rules about the combinations that are considered to generate these additional Merge candidates.

[0100] Zero motion candidates are inserted to fill the remaining entries of the Merge candidate list and thus reach the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero motion candidate is added to the list.

[0101] More specifically, the following steps are performed in order until the Merge list is full:

[0102] 1. Set the variable numRef to the number of reference pictures associated with list 0 for a P slice, or the minimum number of reference pictures in both lists for a B slice;

[0103] 2. Add non-repeating zero motion candidates:

[0104] For variable i being 0...numRef-1, a default motion candidate is added to list 0 (if P slice) or for both lists (if B slices), with MV set to (0, 0) and reference picture index set to i.

[0105] 3. Add repeated zero motion candidates with MV set to (0, 0), reference picture index of list 0 set to 0 (if P slice) and reference picture index of both lists set to 0 (if B slice).

[0106] Finally, these candidates are not checked redundantly.

[0107] 2.1.3 AMVP

[0108] AMVP exploits the spatial-temporal correlation of motion vectors with adjacent PUs, which is used for explicit transmission of motion parameters. For each reference picture list, a motion vector candidate list is constructed by first checking the availability of adjacent PU positions in the time domain to the left and above, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and transmit the corresponding index indicating the selected candidate. Similar to the Merge index signaling notification, unary truncation is used to encode the index of the best motion vector candidate. The maximum value to be encoded in this case is 2 (see Figure 8 ). In the following sections, details about the derivation process of motion vector prediction candidates are provided.

[0109] 2.1.3.1 Derivation of AMVP Candidates

[0110] Figure 8 The derivation process of motion vector prediction candidates is summarized.

[0111] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. Figure 2 As shown, for spatial motion vector candidate derivation, two motion vector candidates are finally derived based on the motion vectors of each PU located at five different positions.

[0112] For temporal motion vector candidate derivation, select one motion vector candidate from two candidates derived based on two different co-located positions. After making the first list of spatial-temporal candidates, remove duplicate motion vector candidates in the list. If the number of potential candidates is greater than 2, remove motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 from the list. If the number of spatial-temporal motion vector candidates is less than 2, add an additional zero motion vector candidate to the list.

[0113] 2.1.3.2 Spatial Motion Vector Candidates

[0114] In the derivation of spatial motion vector candidates, at most two candidates are considered among five potential candidates, the five potential candidates are selected from the candidate located at Figure 2 The order of derivation of the left side of the current PU is defined as A0, A1 and scaled A0, scaled A1. The order of derivation of the upper side of the current PU is defined as B0, B1, B2, scaled B0, scaled B1, scaled B2. Therefore, there are four cases for each side that can be used as motion vector candidates, two of which do not require spatial scaling and two use spatial scaling. The four different cases are summarized as follows.

[0115] No airspace scaling

[0116] -(1) Same reference picture list and same reference picture index (same POC)

[0117] -(2) Different reference picture lists, but same reference picture (same POC)

[0118] Airspace scaling

[0119] -(3) Same reference picture list, but different reference pictures (different POC)

[0120] -(4) Different reference picture lists and different reference pictures (different POC)

[0121] First check for no spatial scaling, then spatial scaling. Spatial scaling is considered when the POC has different reference pictures for the neighboring PU and the current PU, regardless of the reference picture list. If all PUs of the left candidate are not available or are intra-coded, scaling of the upper motion vector is allowed to help derive the left and upper MV candidates in parallel. Otherwise, spatial scaling is not allowed for the upper motion vector.

[0122] In the spatial domain scaling process, the motion vectors of neighboring PUs are scaled in a similar way to the temporal domain scaling, such as Fig. 9 The main difference is that the reference picture list and index of the current PU are given as input; the actual scaling process is the same as the temporal scaling.

[0123] 2.1.3.3 Temporal Motion Vector Candidates

[0124] In addition to the reference picture index derivation, the entire process of deriving the temporal Merge candidate is similar to the process of deriving the spatial motion vector candidate (see Figure 6 The reference picture index is signaled to the decoder.

[0125] 2.2 Sub-CU-based motion vector prediction method in JEM

[0126] In JEM with QTBT, each CU can have at most one set of motion parameters for each prediction direction. Two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving motion information of all sub-CUs of the large CU. The alternative temporal motion vector prediction (ATMVP) method allows each CU to retrieve multiple sets of motion information from multiple blocks smaller than the current CU in a co-located reference picture. In the spatial-temporal motion vector prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using a temporal motion vector predictor and a spatial neighboring motion vector.

[0127] In order to preserve a more accurate motion field for sub-CU motion prediction, motion compression of reference frames is currently disabled.

[0128] Fig.10 An example of ATMVP motion prediction for a CU is shown.

[0129] 2.2.1 Alternative temporal motion vector prediction

[0130] In the alternative temporal motion vector prediction (ATMVP) method, the motion vectors are modified by retrieving multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU. A sub-CU is a square N×N block (N is set to 4 by default).

[0131] ATMVP predicts the motion vectors of sub-CUs within a CU in two steps. The first step is to identify the corresponding blocks of the reference picture with the so-called time domain vectors. The reference picture is called the motion source picture. The second step is to divide the current CU into sub-CUs and obtain the motion vectors and the reference index of each sub-CU from the blocks corresponding to each sub-CU.

[0132] In the first step, the reference picture and the corresponding block are determined by the motion information of the spatial adjacent blocks of the current CU. To avoid repeated scanning of adjacent blocks, the first Merge candidate in the Merge candidate list of the current CU is used. The first available motion vector and its associated reference index are set as the temporal vector and the index to the motion source picture. In this way, in ATMVP, the corresponding block can be identified more accurately than in TMVP, where the corresponding block (sometimes called the co-located block) is always at the right bottom or center position relative to the current CU.

[0133] In the second step, the corresponding block of the sub-CU is identified by the time domain vector in the motion source picture by adding the time domain vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample point) is used to derive the motion information of the sub-CU. After identifying the motion information of the corresponding N×N block, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes apply. For example, the decoder checks whether the low latency condition (that is, the POC of all reference pictures of the current picture is less than the POC of the current picture) is met, and possibly uses the motion vector MV for each sub-CU. x (corresponding to the motion vector of reference picture list X) to the predicted motion vector MV y (where X equals 0 or 1, and Y equals 1-X).

[0134] 2.2.2 Space-Temporal Motion Vector Prediction (STMVP)

[0135] In this method, the motion vector of a sub-CU is recursively derived following a grid scan order. Fig.11 The concept is illustrated in Figure 1. For example, Fig.11 An 8x8 CU is shown with four 4x4 sub-CUs A, B, C, and D. Adjacent 4x4 blocks in the current frame are labeled a, b, c, and d.

[0136] The motion derivation of sub-CU A begins with identifying its two spatial neighbors. The first neighborhood is the N×N block (block c) above sub-CU A. If block c is not available or is intra-coded, the other N×N blocks above sub-CU A are checked (from left to right, starting from block c). The second neighborhood is the block (block b) on the left side of sub-CU A. If block b is not available or is intra-coded, the other blocks on the left side of sub-CUA are checked (from top to bottom, starting from block b). For each list, the motion information obtained from the adjacent blocks is scaled to the first reference frame of the given list. Next, the temporal motion vector predictor (TMVP) of sub-block A is derived by following the same process as the TMVP derivation specified in HEVC. The motion information of the co-located block at position D is retrieved and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged separately for each reference list. The average motion vector is assigned as the motion vector of the current sub-CU.

[0137] 2.2.3 Sub-CU motion prediction mode signaling notification

[0138] Sub-CU mode is enabled as an additional Merge candidate, and there are no additional syntax elements required to signal the mode. Two additional Merge candidates are added to the Merge candidate list of each CU to represent ATMVP mode and STMVP mode. If the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates are used. The encoding and decoding logic of the additional Merge candidates is the same as the encoding logic of the Merge candidates in HM, which means that for each CU in a P or B slice, the two additional Merge candidates require two more RD checks.

[0139] In JEM, all bins of the Merge index are CABAC context-coded, whereas in HEVC, only the first bin is context-coded and the remaining bins are context-bypass-coded.

[0140] 2.3 Inter-frame prediction method in VVC

[0141] There are several new codec tools for inter prediction improvements, such as Adaptive Motion Vector Difference Resolution (AMVR) for signaling MVD, Affine Prediction Mode, Triangle Prediction Mode (TPM), ATMVP, Normalized Bi-Prediction (GBI), Bidirectional Optical Flow (BIO).

[0142] 2.3.1 Adaptive motion vector difference resolution

[0143] In HEVC, when use_integer_mv_flag is equal to 0 in the slice header, the motion vector difference (MVD) (between the motion vector and the predicted motion vector of the PU) is signaled in units of quarter luma samples. In VVC, local adaptive motion vector resolution (LAMVR) is introduced. In VVC, MVD can be encoded and decoded in units of quarter luma samples, integer luma samples, or four luma samples (i.e., 1 / 4-pixel, 1-pixel, 4-pixel). The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

[0144] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or quad luma sample MV precision is used.

[0145] When the first MVD resolution flag of a CU is zero, or the CU is not encoded or decoded (meaning that all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the CU's AMVP candidate list is rounded to the corresponding precision.

[0146] In the encoder, a CU-level RD check is used to determine which MVD resolution is to be used for a CU. That is, three CU-level RD checks are performed for each MVD resolution. To speed up the encoder, the following encoding scheme is applied in JEM.

[0147] During RD check of a CU with normal quarter luma sample MVD resolution, the motion information (integer luma sample accuracy) of the current CU is stored. The stored motion information (after rounding) is used as a starting point for small-range motion vector refinement during further RD check of the same CU with integer luma sample and 4 luma sample MVD resolution, so that the time-consuming motion estimation process is not repeated three times.

[0148] · RD check for CUs with 4 luma samples MVD resolution is conditionally called. For a CU, when the RD cost integer luma sample MVD resolution is much larger than a quarter luma sample MVD resolution, the RD check for the CU's 4 luma sample MVD resolution is skipped.

[0149] The encoding process Fig.12 First, the 1 / 4-pixel MV is tested, and the RD cost is calculated and referred to as RDCost0, then the integer MV is tested, and the RD cost is referred to as RDCost1. If RDCost1<th*RDCost0 (where th is a positive value), the 4-pixel MV is tested; otherwise, the 4-pixel MV is skipped. Basically, when checking the integer or 4-pixel MV, the motion information and RD cost, etc. are known for the 1 / 4-pixel MV, which can be reused to speed up the encoding process of the integer or 4-pixel MV.

[0150] 2.3.2 Triangle Prediction Mode

[0151] The concept of triangle prediction mode (TPM) is to introduce a new triangle partitioning for motion compensated prediction. Figures 13A-13BAs shown, it divides the CU into two triangular prediction units in the diagonal or anti-diagonal direction. Each triangular prediction unit in the CU is inter-predicted using its own unidirectional prediction motion vector and reference frame index, which is derived from a single unidirectional prediction candidate list. An adaptive weighting process is applied to the diagonal edges after predicting the triangular prediction unit. Then, the transform and quantization process is applied to the entire CU. This mode is only applied to Merge mode (skip mode is regarded as a special Merge mode).

[0152] Figures 13A-13B is a diagram of dividing a CU into two triangular prediction units (two division modes). For example, Fig.13A An example of a 135 degree segmentation type (divided from the top left corner to the bottom right corner) is shown, and Fig. 13B An example of a 45 degree division pattern is shown.

[0153] 2.3.2.1 TPM’s One-Way Prediction Candidate List

[0154] The unidirectional prediction candidate list (named TPM motion candidate list) consists of five unidirectional prediction motion vector candidates. It is derived from seven neighboring blocks including five spatial neighboring blocks (1 to 5) and two temporal co-located blocks (6 to 7), as shown in Fig.14 As shown. The motion vectors of the seven adjacent blocks are collected and put into the unidirectional prediction candidate list according to the following order: unidirectional prediction motion vector, L0 motion vector of bidirectional prediction motion vector, L1 motion vector of bidirectional prediction motion vector, and average motion vector of L0 and L1 motion vector of bidirectional prediction motion vector. If the number of candidates is less than five, zero motion vector is added to the list. The motion candidates added to the list for TPM are called TPM candidates, and the motion information derived from the spatial / temporal blocks is called conventional motion candidates.

[0155] More specifically, the following steps are involved:

[0156] 1) When adding regular motion candidates from spatially adjacent blocks, the full pruning operation From A1, B1, B0, A0, B2, Col and Col2 (corresponding to Fig.14 Blocks 1-7 in ) obtain regular motion candidates.

[0157] 2) Set the variable numCurrMergeCand = 0

[0158] 3) For each normal motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2, if not pruned and numCurrMergeCand is less than 5, if the normal motion candidate is unidirectionally predicted (from list 0 or list 1), it is directly added to the Merge list as a TPM candidate, and numCurrMergeCand is increased by 1. Such a TPM candidate is named

[0159] 'Original unidirectional prediction candidate'.

[0160] Apply Full Trim .

[0161] 4) For each motion candidate derived from A1, B1, B0, A0, B2, Col and Col2, if not pruned and numCurrMergeCand is less than 5, if the regular motion candidate is bidirectionally predicted, the motion information from list 0 is added to the TPMMerge list (i.e., modified to be unidirectionally predicted from list 0) as a new TPM candidate and numCurrMergeCand is increased by 1. Such a TPM candidate is named 'truncated List0 prediction candidate'.

[0162] Apply a full trim.

[0163] 5) For each motion candidate derived from A1, B1, B0, A0, B2, Col and Col2, if not pruned and numCurrMergeCand is less than 5, if the regular motion candidate is bi-directionally predicted, the motion information from list 1 is added to the TPMMerge list (i.e., modified to be uni-directionally predicted from list 1), and numCurrMergeCand is increased by 1. Such a TPM candidate is named 'truncated list 1 prediction candidate'.

[0164] Apply a full trim.

[0165] 6) For each motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2, if not pruning and numCurrMergeCand is less than 5, if the regular motion candidate is bidirectionally predicted,

[0166] - If the slice QP of list 0 reference picture is less than the slice QP of list 1 reference picture, the motion information of list 1 is first scaled to list 0 reference picture, and the average of two MVs (one from original list 0 and the other is the scaled MV from list 1) is added to the TPM Merge list, such candidate is called average unidirectional prediction from list 0 motion candidate, and numCurrMergeCand is increased by 1.

[0167] Otherwise, the motion information of list 0 is first scaled to the list 1 reference picture, and the average of the two MVs (one from the original list 1 and the other is the scaled MV from list 0) is added to the TPM Merge list, such a TPM candidate is called the average unidirectional prediction from list 1 motion candidate, and numCurrMergeCand is increased by 1.

[0168] Apply a full trim.

[0169] 7) If numCurrMergeCand is less than 5, add a zero motion vector candidate.

[0170] When inserting a candidate into the list, if it has to be compared with all previously added candidates to find out if it is identical to one of them, such a process is called full pruning.

[0171] 2.3.2.2 Adaptive Weighting Process

[0172] After predicting each triangle prediction unit, an adaptive weighting process is applied to the diagonal edges between two triangle prediction units to derive the final prediction for the entire CU. Two weighting factor groups are defined as follows:

[0173] · The first weighting factor set: {7 / 8, 6 / 8, 4 / 8, 2 / 8, 1 / 8} and {7 / 8, 4 / 8, 1 / 8} are used for luma and chroma samples respectively;

[0174] · The second weighting factor set: {7 / 8, 6 / 8, 5 / 8, 4 / 8, 3 / 8, 2 / 8, 1 / 8} and {6 / 8, 4 / 8, 2 / 8} are used for luma and chroma samples respectively.

[0175] The weighting factor set is selected based on the comparison of the motion vectors of the two triangle prediction units. When the reference pictures of the two triangle prediction units are different from each other or the difference of their motion vectors is greater than 16 pixels, the second weighting factor set is used. Otherwise, the first weighting factor set is used.

[0176] 2.3.2.3 Signaling of Triangle Prediction Mode (TPM)

[0177] A one-bit flag indicating whether to use TPM may be signaled first. Figures 13A-13B ) and further signaling the selected Merge index for each of the two partitions.

[0178] 2.3.2.3.1 TPM Flag Signaling Notification

[0179] If the width and height of the luma block are denoted by W and H respectively, for W*H<64, the triangle prediction mode is disabled.

[0180] When a block is encoded or decoded in affine mode, triangle prediction mode is also disabled.

[0181] When a block is encoded and decoded in Merge mode, a one-bit flag may be signaled to indicate whether the triangle prediction mode is enabled or disabled for the block.

[0182] The flag is encoded and decoded with 3 contexts, based on the following equations:

[0183] Ctx index=((leftblockLavailable&&L is codedwithTPM?)1:0)+((Aboveblock A available&&A is coded with TPM?)1:0);

[0184] Fig.15 Examples of contiguous blocks for context selection in TPM flag encoding and decoding are shown (A and L).

[0185] 2.3.2.3.2 Signaling of two division modes (e.g. Figures 13A-13B ), and the selected Merge index for each of the two splits

[0186] Note that the partition mode and the Merge index of the two partitions are jointly coded. In addition, the Merge index is restricted so that the two partitions cannot use the same reference index. Therefore, there are 2 (partition mode) * N (maximum number of Merge candidates) * (N-1) possibilities, where N is set to 5. Codec an indication, and the mapping between the partition mode, the two Merge indexes and the codec indication is derived from the array defined below:

[0187] const uint8_t g_TriangleCombination[TRIANGLE_MAX_NUM_CANDS][3]={{0,1,0},{1,0,1},{1,0,2},{0,0,1},{0,2,0},{1 ,0,3},{1,0,4},{1,1,0},{0,3,0},{0,4,0},{0,0,2},{0,1,2},{1,1,2},{0,0,4},{0,0,3},{0,1,3},{0,1,4} , {1, 1, 4}, {1, 1, 3}, {1, 2, 1}, {1, 2, 0}, {0, 2, 1}, {0, 4, 3}, {1, 3, 0}, {1, 3, 2}, {1, 3, 4}, {1, 4, 0}, {1, 3, 1}, {1, 2, 3}, {1, 4, 1}, {0, 4, 1}, {0, 2, 3}, {1, 4, 2}, {0, 3, 2}, {1, 4, 3}, {0, 3, 1}, {0, 2, 4}, {1, 2, 4}, {0, 4, 2}, {0, 3, 4}};

[0188] Division mode (45 degrees or 135 degrees) = g_TriangleCombination [signaled indication] [0];

[0189] Merge index of candidate A = g_TriangleCombination[signaled indication][1];

[0190] Merge index of candidate B = g_TriangleCombination[signaled indication][2];

[0191] Once two motion candidates A and B are derived, the motion information of the two partitions (PU1 and PU2) can be set from A or B. Whether PU1 uses the motion information of Merge candidate A or B depends on the prediction direction of the two motion candidates. Table 1 shows the relationship between the two motion candidates A and B derived using two partitions.

[0192] Table 1: Derivation of motion information for segmentation from two derived Merge candidates (A, B)

[0193]

[0194] 2.3.2.3.3 Indicated entropy codec (indicated by merge_triangle_idx)

[0195] merge_triangle_idxIn the range [0, 39] (inclusive), K-order Exponential Golomb (EG) codec is used for binarization of merge_triangle_idx, where K is set to 1.

[0196] K times EG

[0197] To encode larger numbers in fewer bits (at the expense of using more bits to encode smaller numbers), this can be normalized with a non-negative integer parameter k. To encode a non-negative integer x in the k-th EGOLomb coding:

[0198] 1. Encode using the above 0-order Exponential Columbus codec but

[0199] 2. Encode x modulo 2k in binary

[0200] Table 2: Exponential Golomb k encoding and decoding example

[0201]

[0202] 2.3.3 Affine Motion Compensated Prediction

[0203] In HEVC, only the translation motion model is applied to motion compensation prediction (MCP). In the real world, there are many kinds of motion, such as scaling, rotation, perspective motion, and other unconventional motions. In VVC, a simplified affine transformation motion compensation prediction with a 4-parameter affine model and a 6-parameter affine model is applied. Figures 16A-16B As shown, the block affine motion field for the 4-parameter affine model ( Fig.16A ) is described by two control point motion vectors (CPMV) and for the 6-parameter affine model ( Fig. 16B ) is described by 3 CPMVs.

[0204] The motion vector field (MVF) of a block is described by the following equations, using the 4-parameter affine model in equation (1) (where the 4 parameters are defined as variables a, b, e, and f) and the 6-parameter affine model in equation (2) (where the 4 parameters are defined as variables a, b, c, d, e, and f), respectively:

[0205]

[0206]

[0207] Where (mv h 0,mv h 0) is the motion vector of the left top corner control point, and (mv h 1, mv h 1) is the motion vector of the right top corner control point, and (mv h2.mv h 2) is the motion vector of the left bottom corner control point, all three motion vectors are called control point motion vectors (CPMV), (x, y) represents the coordinates of the representative point relative to the left top sample point in the current block, and (mv h (x,y),mv v (x, y)) is the motion vector derived for the sample at (x, y). The CP motion vector can be signaled (as in affine AMVP mode) or derived in real time (as in affine Merge mode). w and h are the width and height of the current block. In practice, the division is implemented by a right shift with a rounding operation. In VTM, the representative point is defined as the center position of the sub-block, for example, when the coordinates of the left top corner of the sub-block relative to the left top sample in the current block are (xs, ys), the coordinates of the representative point are located as (xs+2, ys+2). For each sub-block (i.e., 4x4 in VTM), the representative point is used to derive the motion vector for the entire sub-block.

[0208] To further simplify motion compensation prediction, sub-block based affine transformation prediction is applied. To derive the motion vector of each M×N (in the current VVC, both M and N are set to 4) sub-block, as Fig.17 The motion vector of the center sample of each sub-block shown is calculated according to equations (1) and (2) and rounded to 1 / 16 fractional accuracy. Then, a 1 / 16 pixel motion compensated interpolation filter is applied to generate a prediction of each sub-block using the derived motion vector. The 1 / 16 pixel interpolation filter is introduced by the affine mode.

[0209] After MCP, the high-accuracy motion vector of each sub-block is rounded and saved to the same accuracy as the normal motion vector.

[0210] 2.3.3.1 Signaling of Affine Prediction

[0211] Similar to the translational motion model, there are also two modes to signal the auxiliary information of affine prediction. They are AFFINE_INTER and AFFINE_MERGE modes.

[0212] 2.3.3.2 AF_INTER mode

[0213] AF_INTER mode may be applied for CUs whose width and height are both greater than 8. An affine flag in the CU level is signaled in the bitstream to indicate whether AF_INTER mode is used.

[0214] In this mode, for each reference picture list (list 0 or list 1), an affine AMVP candidate list is constructed with three types of affine motion predictors in the following order, where each candidate contains the estimated CPMV of the current block. The best CPMV found at the encoder side (such as Fig. 20 The difference of mv0 mv1 mv2) in mv0 mv1 and the estimated CPMV are signaled. In addition, the index of the affine AMVP candidate from which the estimated CPMV is derived is also signaled.

[0215] 1) Inherited affine motion predictor

[0216] The checking order is similar to that of spatial MVP in HEVC AMVP list construction. First, the left inherited affine motion predictor is derived from the first block in {A1, A0}, which is affine coded and has the same reference picture as the current block. Second, the above inherited affine motion predictor is derived from the first block in {B1, B0, B2}, which is affine coded and has the same reference picture as the current block. Fig.19 Five blocks A1, A0, B1, B0, B2 are shown in FIG.

[0217] Once it is found that the neighboring block is coded in affine mode, the CPMV of the codec covering the neighboring block is used to derive the predictor of the CPMV of the current block. For example, if A1 is coded in non-affine mode and A0 is coded in 4-parameter affine mode, the left-inherited affine MV predictor will be derived from A0. In this case, the CPMV of the CU covering A0 (in Fig. 21B In the figure, the left top CPMV is represented by And the right top CPMV is represented by ) is used to derive the estimated CPMV of the current block, and the top left (with coordinates (x0, y0)), top right (with coordinates (x1, y1)) and bottom right (with coordinates (x2, y2)) positions of the current block are denoted as

[0218] 2) Constructed affine motion predictor

[0219] The constructed affine motion predictor consists of control point motion vectors (CPMVs) derived from adjacent inter-frame codec blocks with the same reference picture, such as Fig. 20 If the current affine motion model is 4-parameter affine, the number of CPMVs is 2, otherwise if the current affine motion model is 6-parameter affine, the number of CPMVs is 3. Top left CPMV Derived from the MV at the first block in the group {A, B, C}, which is inter-coded and has the same reference picture as the current block. Derived from the MV at the first block in group {D, E}, which is inter-coded and has the same reference picture as the current block. Left bottom CPMV Derived from the MV at the first block in the group {F, G}, which is inter-coded and has the same reference picture as the current block.

[0220] - If the current affine motion model is 4-parameter affine, then only if and are found (i.e., and The constructed affine motion predictor is inserted into the candidate list only when it is used as the estimated CPMV of the left top (with coordinates (x0, y0)) and right top (with coordinates (x1, y1)) positions of the current block.

[0221] - If the current affine motion model is 6-parameter affine, only if and are found (i.e., and The constructed affine motion predictors are inserted into the candidate list only when they are used as the estimated CPMVs for the left top (with coordinates (x0, y0) of the current block), right top (with coordinates (x1, y1)) and right bottom (with coordinates (x2, y2)) positions.

[0222] When inserting the constructed affine motion predictor into the candidate list, no pruning process is applied.

[0223] 3) Ordinary AMVP motion predictor

[0224] The following is applied until the number of affine motion predictors reaches a maximum value.

[0225] 1) By setting all CPMV equal to Derive an affine motion predictor (if available).

[0226] 2) By setting all CPMV equal to Derive an affine motion predictor (if available).

[0227] 3) By setting all CPMV equal to Derive an affine motion predictor (if available).

[0228] 4) Derive an affine motion predictor by setting all CPMVs equal to HEVC TMVP (if available).

[0229] 5) An affine motion predictor is derived by setting all CPMVs to zero MV.

[0230] Notice that It has been derived in the constructed affine motion predictor.

[0231] Fig.18A An example of a 4-parameter affine model is shown. Fig.18B An example of a 6-parameter affine model is shown.

[0232] Fig.19 An example of MVP of AF_INTER inheriting the affine candidate is shown.

[0233] Fig. 20 An example of MVP of AF_INTER of constructed affine candidates is shown.

[0234] In AF_INTER mode, when using the 4 / 6 parameter affine mode, 2 / 3 control points are required, and therefore 2 / 3 MVDs need to be encoded and decoded for these control points, such as Fig. 20 In JVET-K0337, it is proposed to derive MV as follows, that is, predict mvd1 and mvd2 from mvd0.

[0235]

[0236]

[0237]

[0238] in mvd i and mv1 are the predicted motion vector, motion vector difference, and motion vector of the top left pixel (i=0), the top right pixel (i=1), or the bottom left pixel (i=2), respectively. Fig.18B Note that adding two motion vectors (e.g., mvA(xA, yA) and mvB(xB, yB)) is equal to the sum of two separate components, i.e., newMV=mvA+mvB, and the two components of newMV are set to (xA+xB) and (yA+yB), respectively.

[0239] 2.3.3.3 AF_MERGE Mode

[0240] When a CU is applied in AF_MERGE mode, it reconstructs the blocks from the valid neighborhood to obtain the first block encoded and decoded in affine mode. And the selection order of candidate blocks is from left, top, top right, bottom left to top left, such as Fig.21A As shown in (in order referred to as A, B, C, D, E). For example, if the neighborhood left bottom block is encoded and decoded in affine mode (in Fig. 21BThe control point (CP) motion vector mv0 of the top left corner, top right corner, and bottom left corner of the adjacent CU / PU containing block A is retrieved. N 、mv1 N and or mv2 N . And the motion vector mv0 of the left top corner / right top / left bottom on the current CU / PU C 、mv1 C and mv2 C (It is only used for 6-parameter affine models) is based on mv0 N 、mv1 N and mv2 N Calculation. In VTM-2.0, if the current block is affine coded, the sub-block at the top left corner (e.g., 4×4 block in VTM) stores mv0, and the sub-block at the top right corner stores mv1. If the current block is coded with a 6-parameter affine model, the sub-block at the bottom left corner stores mv2; otherwise (using a 4-parameter affine model), LB stores mv2′. Other sub-blocks store MVs for MC.

[0241] In deriving the CPMV mv0 of the current CU C 、mv1 C and mv2 C Afterwards, the MVF of the current CU is generated according to the simplified affine motion model equations (1) and (2). In order to identify whether the current CU is coded in AF_MERGE mode, an affine flag is signaled in the bitstream when there is at least one neighboring block coded in affine mode.

[0242] In JVET-L0142 and JVET-L0632, the following steps are used to construct the affine merge candidate list:

[0243] 1) Insert inheritance affine candidates

[0244] An inherited affine candidate is a candidate derived from the affine motion model of its valid neighboring affine codec block. Up to two inherited affine candidates are derived from the affine motion model of the adjacent block and inserted into the candidate list. For the left predictor, the scanning order is {A0, A1}; for the upper predictor, the scanning order is {B0, B1, B2}.

[0245] 2) Insert the constructed affine candidate

[0246] If the number of candidates in the affine Merge candidate list is less than MaxNumAffineCand (eg, 5), the constructed affine candidate is inserted into the candidate list. The constructed affine candidate refers to a candidate constructed by combining the neighborhood motion information of each control point.

[0247] a) First, Fig. 22 The motion information of the control point is derived from the specified spatial and temporal neighborhoods shown in . CPk (k = 1, 2, 3, 4) represents the kth control point. A0, A1, A2, B0, B1, B2 and B3 are the spatial locations used to predict CPk (k = 1, 2, 3); T is the temporal location used to predict CP4.

[0248] The coordinates of CP1, CP2, CP3 and CP4 are (0, 0), (W, 0), (H, 0) and (W, H), respectively, where W and H are the width and height of the current block.

[0249] Get the motion information for each control point according to the following priority order:

[0250] - For CP1, the checking priority is B2->B3->A2. B2 is used if available. Otherwise, if B2 is available, B3 is used. If both B2 and B3 are not available, A2 is used. If all three candidates are not available, the motion information of CP1 cannot be obtained.

[0251] -For CP2, the checking priority is B1->B0.

[0252] -For CP3, the inspection priority is A1->A0.

[0253] -For CP4, use T.

[0254] b) Secondly, the combination of control points is used to construct affine merge candidates.

[0255] I. Motion information of three control points is required to construct a 6-parameter affine candidate. Three control points can be selected from one of the following four combinations: ({CP1, CP2, CP4}, {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4}). The combination {CP1, CP2, CP3}, {CP2, CP3, CP4}, {CP1, CP3, CP4} will be converted into a 6-parameter motion model represented by the top left, top right, and bottom left control points.

[0256] II. Motion information of two control points is needed to construct a 4-parameter affine candidate. Two control points can be selected from one of two combinations: ({CP1, CP2}, {CP1, CP3}). Both combinations will be converted into a 4-parameter motion model represented by the left and right top control points.

[0257] III. The combinations of constructed affine candidates are inserted into the candidate list in the following order:

[0258] {CP1, CP2, CP3}, {CP1, CP2, CP4}, {CP1, CP3, CP4}, {CP2, CP3, CP4}, {CP1, CP2}, {CP1, CP3}

[0259] i. For each combination, check the reference index of List X for each CP, if they are all the same, then the combination has a valid CPMV for List X. If the combination does not have a valid CPMV for both List 0 and List 1, then the combination is marked as invalid. Otherwise, it is valid and the CPMV is placed in the Sub-block Merge list.

[0260] 3.) Fill with zero motion vectors

[0261] If the number of candidates in the Affine Merge candidate list is less than 5, a zero motion vector with a zero reference index is inserted into the candidate list until the list is full.

[0262] More specifically, for the sub-block Merge candidate list, there are 4-parameter Merge candidates with MV set to (0, 0) and prediction directions set to unidirectional prediction (for P slices) and bidirectional prediction (for B slices) from list 0.

[0263] 2.3.4 Current Image Reference

[0264] Intra Block Copy (also known as IBC, or Intra Picture Block Compensation), also known as Current Picture Reference (CPR), is used in the HEVC Screen Content Codec Extension (SCC). This tool is very effective for coding screen content videos, since repetitive patterns in text and graphic-rich content frequently occur within the same picture. Using previously reconstructed blocks with equal or similar patterns as predictors can effectively reduce prediction errors and thus improve codec efficiency. An example of intra block compensation is shown in Fig.23 Medium picture.

[0265] Similar to the design of CRP in HEVC SCC, in VVC, the use of IBC mode is signaled at the sequence and picture level. When IBC mode is enabled in the sequence parameter set (SPS), it can be enabled at the picture level. When IBC mode is enabled at the picture level, the currently reconstructed picture is considered as a reference picture. Therefore, no block-level syntax changes are required to signal the use of IBC mode based on the existing VVC inter-frame mode.

[0266] Key Features:

[0267] -It is treated as a normal inter mode. Therefore, Merge mode and Skip mode are also available for IBC mode. The Merge candidate list construction is unified, containing Merge candidates from adjacent positions encoded in IBC mode or HEVC inter mode. Depending on the selected Merge index, the current block in Merge mode or Skip mode can be merged into a neighborhood encoded in IBC mode or a neighborhood encoded in other normal inter modes with different pictures as reference pictures.

[0268] - The block vector prediction and coding scheme of the IBC mode reuses the scheme used for motion vector prediction and the coding in the HEVC inter mode (AMVP and MVD coding).

[0269] - Motion vectors (also called block vectors) for IBC mode are encoded and decoded with integer pixel precision, but after decoding are stored in memory with 1 / 16 pixel precision because quarter pixel precision is used in the interpolation and deblocking stages. When used in motion vector prediction for IBC mode, the stored vector predictor will be right-shifted by 4.

[0270] - Search range: It is limited to within the current CTU.

[0271] - CPR is disabled when Affine Mode / Triangle Mode / GBI / Weighted Prediction is enabled.

[0272] 2.3.5 Merge List Design in VVC

[0273] There are three different Merge list construction processes supported in VVC:

[0274] 1) Sub-block Merge candidate list: It contains ATMVP and affine merge candidates. One merge list construction process is shared for both affine mode and ATMVP mode. Here, ATMVP and affine merge candidates can be added in sequence. The sub-block merge list size is signaled in the slice header and the maximum value is 5.

[0275] 2) Unidirectional Prediction TPM Merge List: For triangle prediction mode, one Merge list construction process for two partitions is shared, even though the two partitions can select their own Merge candidate indexes. When building the Merge list, the spatial neighboring blocks and two temporal blocks of the block are checked. The motion information derived from the spatial neighbors and temporal blocks is called regular motion candidates in our IDF. These regular motion candidates are further used to derive multiple TPM candidates. Note that the transformation is performed at the whole block level, even though the two partitions can use different motion vectors to generate their own prediction blocks.

[0276] The size of the one-way prediction TPM Merge list is fixed to 5.

[0277] 3) Regular Merge List: For the rest of the codec blocks, one Merge list construction process is shared. Here, spatial / temporal / HMVP, paired combined bidirectional prediction Merge candidates and zero motion candidates can be inserted in order. The regular Merge list size is signaled in the slice header and the maximum value is 6.

[0278] 2.3.5.1 Sub-block Merge Candidate List

[0279] It is suggested to put all sub-blocks related to motion candidates into a separate Merge list in addition to the regular Merge list for non-sub-block Merge candidates.

[0280] The sub-block related motion candidates are placed into a separate Merge list named 'Sub-block Merge Candidate List'.

[0281] In one example, the sub-block Merge candidate list includes affine Merge candidates, and ATMVP candidates, and / or sub-block-based STMVP candidates.

[0282] 2.3.5.1.1 JVET-L0278

[0283] In this contribution, the ATMVP Merge candidate in the normal Merge list is moved to the first position of the affine Merge list, so that all Merge candidates in the new list (ie, the sub-block based Merge candidate list) are based on the sub-block codec tool.

[0284] 2.3.5.1.2 ATMVP in VTM-3.0

[0285] In VTM-3.0, in addition to the regular Merge candidate list, a special Merge candidate list known as the sub-block Merge candidate list (also known as the affine Merge candidate list) is added. The sub-block Merge candidate list is filled with candidates in the following order:

[0286] a. ATMVP candidate (may be available or not);

[0287] b. Inheritance affine candidate;

[0288] c. Construct affine candidates;

[0289] d. Filling in as zero MV 4-parameter affine model

[0290] The maximum number of candidates in the sub-block Merge candidate list (referred to as ML) is derived as follows:

[0291] 1) If the ATMVP usage flag (eg, the flag may be named "sps_sbtmvp_enabled_flag") is turned on (equal to 1), but the affine usage flag (eg, the flag may be named "sps_affine_enabled_flag") is turned off (equal to 0), then ML is set equal to 1.

[0292] 2) If the ATMVP use flag is off (equal to 0), and the affine use flag is off (equal to 0), then ML is set equal to 0. In this case, the sub-block Merge candidate list is not used.

[0293] 3) Otherwise (affine use flag is on (equal to 1), ATMVP use flag is on or off), ML is signaled from the encoder to the decoder. Valid ML is 0 <= ML <= 5.

[0294] When constructing the sub-block Merge candidate list, the ATMVP candidate is checked first. If any of the following conditions is true, the ATMVP candidate is skipped and not placed in the sub-block Merge candidate list.

[0295] 1) ATMVP use flag is turned off;

[0296] 2) Any TMVP usage flag (e.g. when signaled at slice level, the flag may be named "slice_temporal_mvp_enabled_flag") is turned off;

[0297] 3) The reference picture with reference index 0 in reference list 0 is the same as the current picture (which is the CPR)

[0298] ATMVP in VTM-3.0 is much simpler than in JEM. When generating ATMVP Merge candidates, the following process is applied:

[0299] a. Check the adjacent blocks A1, B1, B0, A0 in order, such as Fig. 22 As shown, the first inter-frame coded block but not CPR coded block is found, which is referred to as block X;

[0300] b. Initialize TMV=(0,0). If there is an MV (denoted as MV') for block X, referring to a co-located reference picture (as signaled in the slice header), then TMV is set equal to MV'.

[0301] c. Assuming the center point of the current block is (x0, y0), locate the corresponding position of (x0, y0) in the co-located picture as M = (x0 + MV'x, y0 + MV'y). Find the block Z covering M.

[0302] i. If Z is intra-coded, ATMVP is not available;

[0303] ii. If Z is inter-coded, MVZ_0 and MVZ_1 of the two lists of block Z are scaled to (Ref list 0 index 0) and (Ref list 1 index 0) as MVdefault0, MVdefault1, and stored.

[0304] d. For each 8x8 sub-block, assuming its center point is (x0S, y0S), locate the corresponding position of (x0S, y0S) in the co-located picture as MS = (x0S + MV′x, y0S + MV′y). Find the block ZS covering MS.

[0305] i. If ZS is intra-coded, MVdefault0 and MVdefault1 are assigned to the sub-block;

[0306] ii. If ZS is inter-coded, MVZS_0 and MVZS_1 of the two lists of block ZS are scaled to (Ref list 0 index 0) and (Ref list 1 index 0) and allocated to the sub-blocks;

[0307] MV cropping and masking in ATMVP:

[0308] When a corresponding position such as M or MS is located in a co-located picture, it is cropped within a predetermined area. The CTU size is S×S, and S=128 in VTM-3.0. Assuming that the top left position of the co-located CTU is (xCTU, yCTU), the corresponding position M or MS at (xN, yN) will be cropped to the valid area xCTU<=xN<xCTU+S+4; yCTU<=yN<yCTU+S.

[0309] In addition to clipping, (xN, yN) is also masked as xN = xN & MASK, yN = yN & MASK, where MASK is equal to ~ (2 N -1), and N=3 to set the lowest 3 bits to 0. Therefore, xN and yN are numbers that are multiples of 8 ("~" indicates a bitwise compensation operator).

[0310] Fig.24 An example of valid corresponding areas in a co-located picture is shown.

[0311] 2.3.5.1.3 Syntax Design in the Header

[0312]

[0313]

[0314] 2.3.5.2 Conventional Merge List

[0315] Different from the Merge list design, in VVC, a history-based motion vector prediction (HMVP) method is adopted.

[0316] In HMVP, the motion information of the previously coded blocks is stored. The motion information of the previously coded blocks is defined as HMVP candidates. Multiple HMVP candidates are stored in a table named HMVP, and the table is maintained in real time during the encoding / decoding process. The HMVP table is cleared when a new slice is started to be coded / decoded. As long as there is an inter-coded block, the associated motion information is added to the last entry of the table as a new HMVP candidate. The overall coding and decoding process is as follows: Fig.25 Shown in.

[0317] HMVP candidates can be used in both AMVP and Merge candidate list construction processes. Fig.26 Diagram showing the modified Merge candidate list construction process (using Fig.26 ). When the Merge candidate list is not full after the TMVP candidate is inserted, the HMVP candidates stored in the HMVP table can be used to fill in the Merge candidate list. Considering that a block usually has a higher correlation with the closest adjacent block in terms of motion information, the HMVP candidates in the table are inserted in descending order of index. The last entry in the table is added to the list first, and the first entry is added last. Similarly, redundancy removal is applied to the HMVP candidates. Once the total number of available Merge candidates reaches the maximum number of Merge candidates allowed for signaling, the Merge candidate list construction process is terminated.

[0318] 2.4 MV rounding

[0319] In VVC, when MV is right-shifted, it is required to round towards zero. In a formal way, for an MV (MVx, MVy) to be right-shifted by N bits, the result MV′ (MVx′, MVy′) will be derived as:

[0320] MVx′=(MVx+((1<<N)>>1)-(MVx>=0?1:0))>>N;

[0321] MVy′=(MVy+((1<<N)>>1)-(MVy>=0?1:0))>>N;

[0322] 2.5 RPR in JVET-O2001-v14

[0323] ARC, also known as RPR (Reference Picture Resampling) is integrated in JVET-O2001-v14.

[0324] In the case of RPR in JVET-O2001-v14, TMVP is disabled if the co-located picture has a different resolution than the current picture. In addition, BDOF and DMVR are disabled when the reference picture has a different resolution than the current picture.

[0325] To handle normal MC when the reference picture has a different resolution than the current picture, the interpolation segment is defined as follows:

[0326] 8.5.6.3 Fractional Sample Interpolation Process

[0327] 8.5.6.3.1 Overall

[0328] The inputs to this process are:

[0329] - Luma position (xSb, ySb), specifies the top left sample of the current codec sub-block relative to the top left luma sample of the current picture,

[0330] -Variable sbWidth specifies the width of the current codec sub-block.

[0331] -Variable sbHeight specifies the height of the current codec sub-block.

[0332] - motion vector offset mvOffset,

[0333] - refine the motion vector refMvLX,

[0334] - the selected reference picture sample array refPicLX,

[0335] - half-sample interpolation filter index hpelIfIdx,

[0336] - bidirectional optical flow flag bdofFlag,

[0337] -Variable cIdx specifies the color component index of the current block.

[0338] The output of this process is:

[0339] - predSamplesLX, an array of (sbWidth+brdExtSize)x(sbHeight+brdExtSize) predicted sample values.

[0340] The prediction block bounds extension size brdExtSize is derived as follows:

[0341] brdExtSize=(bdofFlag||(inter_affine_flag[xSb][ySb]&&sps_affine_prof_enabled_flag))? 2:0 (8-752)

[0342] The variable fRefWidth is set equal to PicOutputWidthL of the reference picture in luma samples.

[0343] The variable fRefHeight is set equal to the PicOutputHeightL of the reference picture in luma samples.

[0344] The motion vector mvLX is set to (refMvLX-mvOffset).

[0345] - If cIdx is equal to 0, the following applies:

[0346] - The scaling factors and their fixed point representations are defined as

[0347] hori_scale_fp=((fRefWidth<<14)+(PicOutputWidthL>>1)) / PicOutputWidthL (8-753)

[0348] vert_scale_fp=((fRefHeight<<14)+(PicOutputHeightL>>1)) / PicOutputHeightL (8-754)

[0349] - Let (xIntL, yIntL) be the luma position given in integer sample units, and (xFracL, yFracL) be the offset given in 1 / 16 sample units. These variables are used only in this clause to specify fractional sample positions within the reference sample array refPicLX.

[0350] -The left top coordinate of the boundary block filled with reference samples (xSbInt L ,ySbInt L ) is set to (xSb+(mvLX[0]>>4), ySb+(mvLX[1]>>4)).

[0351] - For each luma sample position (x L =0..sbWidth-1+brdExtSize,y L=0..sbHeight-1+brdExtSize), the corresponding predicted brightness sample value predSamplesLX[x L ][y L ] is derived as follows:

[0352] - Make(refxSb L ,refySb L ) and (refx L , refy L ) is the luminance position pointed to by the motion vector (refMvLX[0], refMvLX[1]) given in 1 / 16 sample units. The variable refxSb L ,refx L ,refySb L and refy L The derivation is as follows:

[0353] sbX L =((xSb<<4)+refMvLX[0])*hori_scale_fp (8-755)

[0354] refx L =((Sign(refxSb)*((Abs(refxSb)+128)>>8)+x L *((hori_scale_fp+8)>>4))+32)>>6 (8-756)

[0355] sb L =((ySb<<4)+refMvLX[1])*vert_scale_fp (8-757)

[0356] refyL=((Sign(refySb)*((Abs(refySb)+128)>>8)+yL*((vert_scale_fp+8)>>4))+32)>>6 (8-758)

[0357] -Variable xInt L ,yInt L 、xFrac L and yFrac L The derivation is as follows:

[0358] xInt L =refx L >>4 (8-759)

[0359] yInt L =refy L >>4 (8-760)

[0360] xFrac L =refx L &15 (8-761)

[0361] yFrac L =refy L &15 (8-762)

[0362] - If bdofFlag is equal to true or (sps_affine_prof_enabled_flag is equal to true and inter_affine_flag[xSb][ySb] is equal to true), and one or more of the following conditions are true, then the luma integer sample retrieval process is invoked as specified in clause 8.5.6.3.3 with (xInt L +(xFrac L >>3)-1),yIntL+(yFrac L >>3)-1) and refPicLX as input, derive the predicted brightness sample value predSamplesLX[x L ][y L ].

[0363] 1.x L is equal to 0.

[0364] 2.x L Equal to sbWidth+1.

[0365] 3.y L is equal to 0.

[0366] 4.y L Equal to sbHeight+1.

[0367] Otherwise, derive the predicted luma sample values ​​predSamplesLX[xL][yL] by invoking the luma sample 8-tap interpolation filtering process as specified in clause 8.5.6.3.2, with (xIntL - (brdExtSize > 0? 1:0), yIntL - (brdExtSize > 0? 1:0)), (xFracL, yFracL), (xSbIntL, ySbIntL), refPicLX, hpelIfIdx, sbWidth, sbHeight and (xSb, ySb) as input.

[0368] - Otherwise (cIdx is not equal to 0), the following applies:

[0369] - Let (xIntC, yIntC) be the chroma position given in integer sample units, and (xFracC, yFracC) be the offset given in 1 / 32 sample units. These variables are used only in this clause to specify the overall fractional sample position within the reference sample array refPicLX.

[0370] - The left top coordinate (xSbIntC, ySbIntC) of the reference sample filled boundary block is set to ((xSb / SubWidthC)+(mvLX[0]>>5), (ySb / SubHeightC)+(mvLX[1]>>5)).

[0371] For each chroma sample position (xC=0..sbWidth-1, yC=0..sbHeight-1) within the predicted chroma sample array predSamplesLX, the corresponding predicted chroma sample value predSamplesLX[xC][yC] is derived as follows:

[0372] - Make(refxSb C ,refySb C ) and (refx C , refy C ) is the chroma position pointed to by the motion vector (mvLX[0], mvLX[1]) given in 1 / 32 sample units. The variable refxSb C ,refySb C ,refx C and refy C The derivation is as follows:

[0373] sbX C =((xSb / SubWidthC<<5)+mvLX[0])*hori_scale_fp (8-763)

[0374] refx C =((Sign(refxSb C )*((Abs(refxSb C )+256)>>9)+xC*((hori_scale_fp+8)>>4))+16)>>5(8-764)

[0375] sb C =((ySb / SubHeightC<<5)+mvLX[1])*vert_scale_fp (8-765)

[0376] refy C =((Sign(refySb C)*((Abs(refySb C )+256)>>9)+yC*((vert_scale_fp+8)>>4))+16)>>5 (8-766)

[0377] -Variable xInt C ,yInt C 、xFrac C and yFrac C The derivation is as follows:

[0378] xInt C =refx C >>5 (8-767)

[0379] yInt C =refy C >>5 (8-768)

[0380] xFrac C =refy C &31 (8-769)

[0381] yFrac C =refy C &31 (8-770)

[0382] - derive the predicted sample values ​​predSamplesLX[xC][yC] by calling the procedure specified in clause 8.5.6.3.4 with as input (xIntC, yIntC), (xFracC, yFracC), (xSbIntC, ySbIntC), sbWidth, sbHeight and refPicLX.

[0383] 8.5.6.3.2 Luma Sample Interpolation Filtering Process

[0384] The inputs to this process are:

[0385] - Luminance position in integer sample units (xInt L ,yInt L ),

[0386] - Luma position in fractional sample units (xFrac L ,yFrac L ),

[0387] - Luminance position in integer sample units (xSbInt L ,ySbInt L ), specifies the top left sample of the boundary block filled with reference samples relative to the top left luma sample of the reference picture,

[0388] - Luminance reference sample array refPicLX L ,

[0389] - half-sample interpolation filter index hpelIfIdx,

[0390] -Variable sbWidth, specifies the width of the current sub-block,

[0391] -Variable sbHeight specifies the height of the current sub-block.

[0392] - luma position (xSb, ySb), specifies the top left sample of the current subblock relative to the top left luma sample of the current picture,

[0393] The output of this process is the predicted luminance sample value predSampleLX L

[0394] The variables shift1, shift2, and shift3 are derived as follows:

[0395] - The variable shift1 is set equal to Min(4, BitDepthY-8), the variable shift2 is set equal to 6, and the variable shift3 is set equal to Max(2, 14-BitDepth Y ).

[0396] - The variable picW is set equal to pic_width_in_luma_samples, and the variable picH is set equal to pic_height_in_luma_samples.

[0397] Equal to xFrac L or yFrac L The luminance interpolation filter coefficient f for each 1 / 16 fractional sample position p L [p]The derivation is as follows:

[0398] - If MotionModelIdc[xSb][ySb] is greater than 0, and sbWidth and sbHeight are both equal to 4, then the brightness interpolation filter coefficient f L [p] is specified in Table 8-12.

[0399] - Otherwise, the brightness interpolation filter coefficient f L [p] Depends on hpelIfIdx and is specified in Table 8-11.

[0400] The brightness position in integer sample units (xInt i ,yInt i ) for i=0..7 is derived as follows:

[0401] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0402] xInt i =Clip3(SubPicLeftBoundaryPos,SubPicRightBoundaryPos,xInt L +i-3)(8-771)

[0403] yInt i =Clip3(SubPicTopBoundaryPos, SubPicBotBoundaryPos, yInt L +i-3) (8-772)

[0404] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0405] xInt i =Clip3(0,picW-1,sps_ref_wraparound_enabled_flag?

[0406] ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY, picW, xInt L +i-3): (8-773)

[0407] xInt L +i-3)

[0408] yInt i = Clip3(0, picH-1, yInt L +i-3) (8-774)

[0409] The luminance position in integer sample units is further modified for i=0..7 as follows:

[0410] xInt i =Clip3(xSbInt L -3,xSbInt L +sbWidth+4,xInt i ) (8-775)

[0411] yInt i =Clip3(ySbInt L-3,ySbInt L +sbHeight+4,yInt i ) (8-776)

[0412] Predicted brightness sample value predSampleLX L is derived as follows:

[0413] -If xFrac L and yFrac L are all equal to 0, predSampleLX L The value of is derived as follows:

[0414] predSampleLX L =refPicLX L [xInt3][yInt3]<<shift3 (8-777)

[0415] - Otherwise, if xFrac L Not equal to 0 and yFrac L If equal to 0, predSampleLX L The value of is derived as follows:

[0416]

[0417] - Otherwise, if xFrac L Equal to 0 and yFrac L If not equal to 0, predSampleLX L The value of is derived as follows:

[0418]

[0419] - Otherwise, if xFrac L Not equal to 0 and yFrac L If not equal to 0, predSampleLX L The value of is derived as follows:

[0420] - The sample array temp[n], where n=0..7, is derived as follows:

[0421]

[0422] -Predicted brightness sample value predSampleLX L is derived as follows:

[0423]

[0424] Table 8-11 - Luma interpolation filter coefficients f for each 1 / 16 fractional sample position p L [p] specification.

[0425]

[0426]

[0427] Table 8-12 - Luma interpolation filter coefficients f for each 1 / 16 fractional sample position p in affine motion mode L [p] specification.

[0428]

[0429] 8.5.6.3.3 Luminance integer sample retrieval process

[0430] The inputs to this process are:

[0431] - Luminance position in integer sample units (xInt L ,yInt L ),

[0432] - Luminance reference sample array refPicLX L ,

[0433] The output of this process is the predicted luminance sample value predSampleLX L

[0434] The variable Offset is set equal to Max(2,14-BitDepthY).

[0435] The variable picW is set equal to pic_width_in_luma_samples, and the variable picH is set equal to pic_height_in_luma_samples.

[0436] The luminance position (xInt, yInt) in integer sample units is derived as follows:

[0437] xInt=Clip3(0,picW-1,sps_ref_wraparound_enabled_flag? (8-782)

[0438] ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY, picW, xInt L ): xInt L )

[0439] yInt=Clip3(0,picH-1,yIntL ) (8-783)

[0440] Predicted brightness sample value predSampleLX L is derived as follows:

[0441] predSampleLX L =refPicLX L [xInt][yInt]<<shift3 (8-784)

[0442] 8.5.6.3.4 Chroma sample interpolation process

[0443] The inputs to this process are:

[0444] - Chroma position in integer sampling units (xInt C ,yInt C ),

[0445] - Chroma position in 1 / 32 fractional sample units (xFrac C ,yFrac C ),

[0446] - the chroma position (xSbIntC, ySbIntC) in integer sample units, specifying the top left sample of the reference sample filled boundary block relative to the top left chroma sample of the reference picture,

[0447] -Variable sbWidth, specifies the width of the current sub-block,

[0448] -Variable sbHeight specifies the height of the current sub-block.

[0449] - Chroma reference sample array refPicLX C .

[0450] The output of this process is the predicted chrominance sample value predSampleLX C

[0451] The variables shift1, shift2, and shift3 are derived as follows:

[0452] - The variable shift1 is set equal to Min(4, BitDepth C -8), the variable shift2 is set equal to 6, and the variable shift3 is set equal to Max(2, 14-BitDepth C ).

[0453] -Variable picW Cis set equal to pic_width_in_luma_samples / SubWidthC, and the variable picH C Set equal to pic_height_in_luma_samples / SubHeightC.

[0454] Equal to xFrac C or yFrac C The chroma interpolation filter coefficient f for each 1 / 32 fractional sample position p C [p] is specified in Table 8-13.

[0455] The variable xOffset is set to (sps_ref_wraparound_offset_minus1+1)*MinCbSizeY) / SubWidthC.

[0456] Chroma position in units of integer samples (xInt i ,yInt i ) for i = 0..3 is derived as follows:

[0457] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0458] xInt i =Clip3(SubPicLeftBoundaryPos / SubWidthC, SubPicRightBoundaryPos / SubWidthC, xInt L +i) (8-785)

[0459] yInt i =Clip3(SubPicTopBoundaryPos / SubHeightC, SubPicBotBoundaryPos / SubHeightC, yInt L +i) (8-786)

[0460] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0461]

[0462] yInt i =Clip3(0,picH C -1,yInt C +i-1) (8-788)

[0463] Chroma position in units of integer samples (xInt i ,yInt i ) is further modified for i=0..3 as follows:

[0464] xInt i =Clip3(xSbIntC-1,xSbIntC+sbWidth+2,xInt i ) (8-789)

[0465] yInt i =Clip3(ySbIntC-1,ySbIntC+sbHeight+2,yInt i ) (8-790)

[0466] Predicted chroma sample value predSampleLX C is derived as follows:

[0467] -If xFrac C and yFrac C If both are equal to 0, then predSampleLX C The value of is derived as follows:

[0468] predSampleLX C =refPicLX C [xInt1][yInt1]<<shift3 (8-791)

[0469] - Otherwise, if xFrac C Not equal to 0 and yFrac C If equal to 0, predSampleLX C The value of is derived as follows:

[0470]

[0471] - Otherwise, if xFrac C Equal to 0 and yFrac C If not equal to 0, predSampleLX C The value of is derived as follows:

[0472]

[0473] - Otherwise, if xFrac C Not equal to 0 and yFrac C If not equal to 0, predSampleLX C The value of is derived as follows:

[0474] - The sample array temp[n], where n=0..3, is derived as follows:

[0475]

[0476] -Predicted chroma sample value predSampleLX C is derived as follows:

[0477]

[0478]

[0479] Table 8-13 – Chroma interpolation filter coefficients f for each 1 / 32 fractional sample position p C [p] specification.

[0480]

[0481]

[0482] 2.6 Sub-images in JVET-O2001-v14

[0483] With the current syntax design of sub-pictures in JVET-O2001-vE, the positions and dimensions of sub-pictures are derived as follows:

[0484]

[0485] subpics_present_flag equal to 1 indicates that sub-picture parameters are currently present in the SPS RBSP syntax. subpics_present_flag equal to 0 indicates that sub-picture parameters are not currently present in the SPS RBSP syntax.

[0486] When the bitstream is the result of a sub-bitstream extraction process and contains only a subset of sub-pictures of the input bitstream to the sub-bitstream extraction process, it may be required to set the value of subpics_present_flag equal to 1 in the RBSP of the SPS.

[0487] max_subpics_minus1 plus 1 specifies the maximum number of sub-pictures that can be present in the CVS. max_subpics_minus1 should be in the range 0 to 254. The value 255 is reserved for future use by ITU-T|ISO / IEC.

[0488] subpic_grid_col_width_minus1 plus 1 specifies the width in units of 4 samples of each element of the sub-picture identifier grid. The length of the syntax element is Ceil(Log2(pic_width_max_in_luma_samples / 4)) bits.

[0489] The variable NumSubPicGridCols is derived as follows:

[0490] NumSubPicGridCols=(pic_width_max_in_luma_samples+subpic_grid_col_width_minus1*4+3) /

[0491] (subpic_grid_col_width_minus1*4+4) (7-5)

[0492] subpic_grid_row_height_minus1 plus 1 specifies the height in units of 4 samples of each element of the sub-picture identifier grid. The length of the syntax element is Ceil(Log2(pic_height_max_in_luma_samples / 4)) bits.

[0493] The variable NumSubPicGridRows is derived as follows:

[0494] NumSubPicGridRows=(pic_height_max_in_luma_samples+subpic_grid_row_height_minus1*4+3) /

[0495] (subpic_grid_row_height_minus1*4+4) (7-6)

[0496] subpic_grid_idx[i][j] specifies the sub-picture index of the grid position (i, j). The length of the syntax element is Ceil(Log2(max_subpics_minus1+1)) bits.

[0497] The variables SubPicTop[subpic_grid_idx[i][j]], SubPicLeft[subpic_grid_idx[i][j]], SubPicWidth[subpic_grid_idx[i][j]], SubPicHeight[subpic_grid_idx[i][j]], and NumSubPics are derived as follows:

[0498]

[0499]

[0500] subpic_treated_as_pic_flag[i] equal to 1 specifies that the i-th subpicture of each codec picture in the CVS is treated as a picture in the decoding process excluding in-loop filtering operations. subpic_treated_as_pic_flag[i] equal to 0 specifies that the i-th subpicture of each codec picture in the CVS is not treated as a picture in the decoding process excluding in-loop filtering operations. When not present, the value of subpic_treated_as_pic_flag[i] is inferred to be equal to 0.

[0501] 2.7 CIIP in JVET-O2001-v14

[0502] Combined Inter-Intra Prediction (CIIP) is used in VVC as a special Merge candidate. It can only be enabled for WxH blocks (W<=64 and H<=64).

[0503] 3. Examples of Problems Solved by the Disclosed Technologies and Embodiments

[0504] In the current design of VVC, ATMVP has the following problems:

[0505] 1) Whether the application of ATMVP at the slice level and at the CU level does not match;

[0506] 2) In the stripe header, ATMVP can be enabled even if TMVP is disabled. At the same time, the ATMVP flag is signaled before the TMVP flag.

[0507] 3) Masking is always performed regardless of whether the MV is compressed;

[0508] 4) The effective corresponding area may be too large;

[0509] 5) The derivation of TMV is too complicated;

[0510] 6) ATMVP may not be available in some cases, and a better default MV is desired.

[0511] 7) The MV scaling method in ATMVP may not be efficient;

[0512] 8) ATMVP should take CPR into consideration;

[0513] 9) Even when affine prediction is disabled, default zero affine Merge candidates can be put into the list.

[0514] 10) The current picture is considered as a long-term reference picture, while other pictures are considered as short-term reference pictures. For both ATMVP and TMVP candidates, the motion information from the temporal blocks in the co-located picture will be scaled to the reference picture with a fixed reference index (i.e., 0 for each reference picture list in the current design). However, when CPR mode is enabled, the current picture is also considered as a reference picture, and the current picture can be added to the reference picture list 0 (RefPicList0) with index equal to 0.

[0515] a. For TMVP, if the time domain block is coded in CPR mode and the reference picture of RefPicList0 is a short-term reference picture, the TMVP candidate is set to unavailable.

[0516] b. If the reference picture of RefPicList0 with index equal to 0 is the current picture and the current picture is an intra random access point (IRAP) picture, the ATMVP candidate is set to unavailable.

[0517] c. For the ATMVP sub-block within a block, when the motion information of the sub-block is derived from the time domain block, if the time domain block is encoded and decoded with the CPR mode, the default ATMVP candidate (derived from the time domain block identified by the starting TMV and the center position of the current block) is used to fill in the motion information of the sub-block.

[0518] 11) The MV is right-shifted to integer precision, but does not follow the rounding rules in VVC.

[0519] 12) The MVs (MVx, MVy) used in ATMVP to locate corresponding blocks in different pictures (e.g., TMV in 2.3.5.1.2) are used directly because they point to co-located pictures. This is based on the assumption that all pictures are of the same resolution. However, when RPR is enabled, different picture resolutions may be used. Similar problems exist for the identification of corresponding blocks in co-located pictures for deriving sub-block motion information.

[0520] 13) If a block width or height is greater than 32, and the maximum transform unit size is 32, for CIIP codec blocks, intra prediction signaling is generated in codec unit (CU) size; while inter prediction signaling is generated in transform unit (TU) size (recursively divide the current block into multiple 32x32 blocks). Using CU to derive intra prediction signaling results in lower efficiency.

[0521] There are some problems with the current design. First, if the reference picture of RefPicList0 with index equal to 0 is the current picture and the current picture is not an IRAP picture, the ATMVP process is still called, but the ATMVP process cannot locate any available ATMVP candidates because no temporal motion vector can be scaled to the current picture.

[0522] 4. Examples of Embodiments and Technologies

[0523] The following technical and embodiment itemized list should be regarded as examples to explain the overall concept. These techniques should not be understood in a narrow manner. In addition, these techniques can be combined in encoder or decoder embodiments in any manner.

[0524] 1. Whether TMVP is allowed and / or whether CPR is used should be considered to determine / parse the maximum number of candidates in the sub-block Merge candidate list and / or to determine whether the ATMVP candidate should be added to the candidate list. The maximum number of candidates in the sub-block Merge candidate list is referred to as ML. Although some exemplary embodiments are discussed below using alternative temporal motion vector prediction (ATMVP), in alternative embodiments, the techniques disclosed in this document are applicable to sub-block-based temporal motion vector prediction (sbTMVP).

[0525] a) In one example, when determining or parsing the maximum number of candidates in the sub-block Merge candidate list, if the ATMVP usage flag is turned off (equal to 0) or TMVP is disabled, ATMVP is inferred to be not applicable.

[0526] i. In one example, the ATMVP usage flag is turned on (equal to 1) and TMVP is disabled, and the ATMVP candidate is not added to the Sub-block Merge candidate list or the ATMVP candidate list.

[0527] ii. In one example, the ATMVP usage flag is turned on (equal to 1) and TMVP is disabled, and the Affine usage flag is turned off (equal to 0), then ML is set to equal to 0, meaning that Sub-block Merge is not applicable.

[0528] iii. In one example, the ATMVP usage flag is on (equal to 1) and TMVP is enabled, and the Affine usage flag is off (equal to 0), then ML is set equal to 1.

[0529] b) In one example, when determining or parsing the maximum number of candidates in the sub-block Merge candidate list, if the ATMVP usage flag is off (equal to 0) or the co-located reference picture of the current picture is the current picture itself, ATMVP is inferred to be not applicable.

[0530] i. In one example, when the ATMVP usage flag is turned on (equal to 1) and the co-located reference picture of the current picture is the current picture itself, the ATMVP candidate is not added to the Sub-block Merge candidate list or the ATMVP candidate list.

[0531] ii. In one example, the ATMVP usage flag is turned on (equal to 1), and the co-located reference picture of the current picture is the current picture itself, and the affine usage flag is turned off (equal to 0), then ML is set to equal to 0, meaning that sub-block Merge is not applicable.

[0532] iii. In one example, the ATMVP usage flag is on (equal to 1), and the co-located reference picture of the current picture is not the current picture itself, and the affine usage flag is off (equal to 0), then ML is set equal to 1.

[0533] c) In one example, when determining or parsing the maximum number of candidates in the sub-block Merge candidate list, if the ATMVP usage flag is turned off (equal to 0), or the reference picture with reference picture index 0 in reference list 0 is the current picture itself, ATMVP is inferred to be not applicable.

[0534] i. In one example, the ATMVP usage flag is turned on (equal to 1) and the reference picture co-located with reference picture index 0 in reference list 0 is the current picture itself, and the ATMVP candidate is not added to the Sub-block Merge candidate list or the ATMVP candidate list.

[0535] ii. In one example, the ATMVP usage flag is turned on (equal to 1), and the reference picture with reference picture index 0 in reference list 0 is the current picture itself, and the affine usage flag is turned off (equal to 0), then ML is set to equal to 0, meaning that sub-block Merge is not applicable.

[0536] iii. In one example, the ATMVP use flag is on (equal to 1) and the reference picture with reference picture index 0 in reference list 0 is not the current picture itself, and the affine use flag is off (equal to 0), then ML is set equal to 1.

[0537] d) In one example, when determining or parsing the maximum number of candidates in the sub-block Merge candidate list, if the ATMVP usage flag is turned off (equal to 0), or the reference picture with reference picture index 0 in reference list 1 is the current picture itself, ATMVP is inferred to be not applicable.

[0538] i. In one example, the ATMVP usage flag is turned on (equal to 1) and the reference picture co-located with reference picture index 0 in reference list 1 is the current picture itself, and the ATMVP candidate is not added to the Sub-block Merge candidate list or the ATMVP candidate list.

[0539] ii. In one example, the ATMVP usage flag is turned on (equal to 1), and the reference picture with reference picture index 0 in reference list 1 is the current picture itself, and the affine usage flag is turned off (equal to 0), then ML is set to equal to 0, meaning that sub-block Merge is not applicable.

[0540] iii. In one example, the ATMVP use flag is on (equal to 1) and the reference picture with reference picture index 0 in reference list 1 is not the current picture itself, and the affine use flag is off (equal to 0), then ML is set equal to 1.

[0541] 2. It is proposed that if TMVP is disabled at the slice / slice / picture level, ATMVP is implicitly disabled and the ATMVP flag is not signaled.

[0542] a) In one example, the ATMVP flag is signaled in the slice header / slice header / PPS after the TMVP flag.

[0543] b) In one example, the ATMVP or / and TMVP flag may not be signaled in the slice header / slice header / PPS, and only signaled in the SPS header.

[0544] 3. Whether and how to mask the corresponding position in ATMVP depends on whether and how to compress MV. Assume (xN, yN) is the corresponding position calculated using the coordinates of the current block / subblock and the starting motion vector (e.g., TMV) in the co-located picture

[0545] a) In one example, if MV is not required to be compressed (e.g., sps_disable_motioncompression signaled in SPS is 1), then (xN, yN) is not masked; otherwise, (MV can be compressed) (xN, yN) is masked as xN = xN & MASK, yN = yN & MASK, where MASK is equal to ~ (2 M -1), and M may be an integer such as 3 or 4.

[0546] b) Assume that the MV compression method used for MV storage results in each 2 K x2 K The blocks share the same motion information, and the mask in the ATMVP process is defined as ~(2 M-1). It is proposed that K may not be equal to M, for example, M=K+1.

[0547] c) The MASKs used in ATMVP and TMVP can be the same, or they can be different.

[0548] 4. In one example, the MV compression method may be flexible.

[0549] a) In one example, the MV compression method can be selected between no compression, 8×8 compression (M=3 in point 3.a), or 16×16 compression (M=4 in point 3.a)

[0550] b) In one example, the MV compression method may be signaled in the VPS / SPS / PPS / slice header / slice group header.

[0551] c) In one example, the MV compression method may be set differently in different standard profiles / levels / tiers.

[0552] 5. The effective corresponding area in ATMVP can be adaptive;

[0553] a) For example, the valid corresponding area may depend on the width and height of the current block;

[0554] b) For example, the valid corresponding area may depend on the MV compression method;

[0555] i. In one example, if the MV compression method is not used, the effective corresponding area is smaller; if the MV compression method is used, the effective corresponding area is larger.

[0556] 6. The valid corresponding area in ATMVP can be based on a basic area with a size of M×N smaller than the CTU area. For example, the CTU size in VTM-3.0 is 128×128, and the basic area size can be 64×64. Assume that the width and height of the current block are W and H.

[0557] a) In one example, if W<=M and H<=N, meaning the current block is within the basic region, then the valid corresponding region in ATMVP is the extension in the co-located basic region and the co-located picture. Fig. 27 An example is shown.

[0558] i. For example, assuming that the top left position of the co-located basic area is (xBR, yBR), the corresponding position at (xN, yN) will be cropped to the valid area xBR<=xN<xBR+M+4; yBR<=yN<yBR+N.

[0559] Fig. 27 An exemplary embodiment of proposing a valid region when a current block is within a base region (BR) is shown.

[0560] Fig.28 An exemplary embodiment of a valid area when the current block is not within the basic area is shown.

[0561] b) In one example, if W>M and H>N, meaning that the current block is not within the basic region, the current block is divided into several parts. Each part has a separate valid corresponding region in ATMVP. For position A in the current block, its corresponding position B in the co-located block should be within the valid corresponding region in the part where position A is located.

[0562] i. For example, the current block is divided into non-overlapping basic regions. The valid corresponding regions of a basic region are its co-located basic regions and extensions in co-located pictures. Fig.28 An example is shown.

[0563] 1. For example, assume that position A in the current block is in a basic region R. The co-located basic region of R in the co-located picture is referred to as CR. The corresponding position of A in the co-located block is position B. The top left position of CR is (xCR, yCR), then position B at (xN, yN) will be trimmed to the valid region xCR<=xN<xCR+M+4; yCR<=yN<yCR+N.

[0564] 7. It is proposed that the motion vector used to locate the corresponding blocks in different pictures in ATMVP (for example, TMV in 2.3.5.1.2) can be derived as:

[0565] a) In one example, the TMV is always set equal to a default MV, such as (0, 0).

[0566] i. In one example, the default MV is signaled in VPS / SPS / PPS / slice header / slice group header / CTU / CU.

[0567] b) In one example, TMV is set to an MV stored in the HMVP table using the following method;

[0568] i. If the HMVP list is empty, the TMV is set equal to the default MV, such as (0, 0)

[0569] ii. Otherwise (HMVP list is not empty),

[0570] 1. TMV can be set equal to the first element stored in the HMVP table;

[0571] 2. Alternatively, TMV can be set equal to the last element stored in the HMVP table;

[0572] 3. Alternatively, the TMV may simply be set equal to a specific MV stored in the HMVP table;

[0573] a. In one example, a particular MV references reference list 0.

[0574] b. In one example, a particular MV refers to reference list 1.

[0575] c. In one example, a particular MV references a particular reference picture in reference list 0, such as a reference picture with index 0.

[0576] d. In one example, a specific MV references a specific reference picture in reference list 1, such as a reference picture with index 0.

[0577] e. In one example, a specific MV references a co-located picture.

[0578] 4. Alternatively, if a specific MV stored in the HMVP table (e.g., mentioned in point 3.) cannot be found, the TMV may be set equal to a default MV;

[0579] a. In one example, only the first element stored in the HMVP table is searched to find a specific MV.

[0580] b. In one example, only the last element stored in the HMVP table is searched to find a specific MV.

[0581] c. In one example, some or all elements in the HMVP table are stored to find a specific MV.

[0582] 5. Alternatively, furthermore, the TMV obtained from the HMVP cannot reference the current picture itself.

[0583] 6. Alternatively, furthermore, the TMV table obtained from the HMVP may be scaled to the co-located picture if it does not refer to the co-located picture.

[0584] c) In one example, the TMV is set to an MV of a specific adjacent block without reference to other adjacent blocks.

[0585] i. A specific adjacent block can be Fig. 22 Blocks A0, A1, B0, B1, B2 in.

[0586] ii. The TMV may be set equal to the default MV if:

[0587] 1. The specific adjacent block does not exist;

[0588] 2. The specific adjacent block is not inter-frame coded;

[0589] iii. TMV can only be set equal to a specific MV stored in a specific adjacent block;

[0590] 1. In one example, a particular MV references reference list 0.

[0591] 2. In one example, a specific MV refers to reference list 1.

[0592] 3. In one example, a specific MV references a specific reference picture in reference list 0, such as a reference picture with index 0.

[0593] 4. In one example, a specific MV references a specific reference picture in reference list 1, such as a reference picture with index 0.

[0594] 5. In one example, a specific MV references a co-located picture.

[0595] 6. If a specific MV stored in a specific contiguous block cannot be found, the TMV can be set equal to a default MV;

[0596] iv. The TMV obtained from a specific neighboring block can be scaled to the co-located picture if it does not refer to the co-located picture.

[0597] v. The TMV obtained from a specific neighboring block cannot refer to the current picture itself.

[0598] 8. MVdefault0 and MVdefault1 used in ATMVP as disclosed in 2.3.5.1.2 can be derived as

[0599] a) In one example, MVdefault0 and MVdefault1 are set to (0, 0);

[0600] b) In one example, MVdefaultX (X=0 or 1) is derived from HMVP,

[0601] i. If the HMVP list is empty, MVdefaultX is set equal to a predetermined default MV, such as (0, 0).

[0602] 1. The predetermined default MV may be signaled in the VPS / SPS / PPS / slice header / slice group header / CTU / CU.

[0603] ii. Otherwise (HMVP list is not empty),

[0604] 1.MVdefaultX can be set equal to the first element stored in the HMVP table;

[0605] 2.MVdefaultX can be set equal to the last element stored in the HMVP table;

[0606] 3.MVdefaultX can only be set equal to a specific MV stored in the HMVP table;

[0607] a. In one example, a particular MV references reference list X.

[0608] b. In one example, a particular MV references a particular reference picture in reference list X, such as a reference picture with index 0.

[0609] 4. If a specific MV stored in the HMVP table cannot be found, MVdefaultX can be set equal to a predetermined default MV;

[0610] a. In one example, only the first element stored in the HMVP table is searched.

[0611] b. In one example, only the last element stored in the HMVP table is searched.

[0612] c. In one example, some or all elements stored in the HMVP table are searched.

[0613] 5. The MVdefaultX obtained from the HMVP table can be scaled to the co-located picture if it does not refer to the co-located picture.

[0614] 6. The MVdefaultX obtained from HMVP cannot refer to the current picture itself.

[0615] c) In one example, MVdefaultX (X=0 or 1) is derived from neighboring blocks.

[0616] i. Adjacent blocks can contain blocks Fig. 22 A0, A1, B0, B1, B2 in.

[0617] 1. For example, only one of these blocks is used to derive MVdefaultX.

[0618] 2. Alternatively, some or all of these blocks are used to derive MVdefaultX.

[0619] a. Check these blocks in order until a valid MVdefaultX is found.

[0620] 3. If no valid MVdefaultX is found from the selected neighboring block or blocks, then MVdefaultX is set equal to a predetermined default MV, such as (0, 0).

[0621] a. The predetermined default MV may be signaled in the VPS / SPS / PPS / slice header / slice group header / CTU / CU.

[0622] ii. No valid MVdefaultX is found from a particular adjacent block if:

[0623] 1. The specific adjacent block does not exist;

[0624] 2. The specific adjacent block is not inter-frame coded;

[0625] iii.MVdefaultX can only be set equal to a specific MV stored in a specific adjacent block;

[0626] 1. In one example, a particular MV refers to reference list X.

[0627] 2. In one example, a specific MV references a specific reference picture in reference list X, such as a reference picture with index 0

[0628] iv. MVdefaultX obtained from a specific neighboring block may be scaled to a specific reference picture, such as the reference picture with index 0 in reference list X.

[0629] v. The MVdefaultX obtained from a specific neighboring block cannot refer to the current picture itself.

[0630] 9. For sub-block or non-sub-block ATMVP candidates, if the temporal block of the sub-block / whole block in the co-located picture is coded in CPR mode, the default motion candidate may be used instead.

[0631] a) In one example, the default motion candidate may be defined as a motion candidate associated with the center position of the current block (eg, MVdefault0 and / or MVdefault1 used in ATMVP as disclosed in 2.3.5.1.2).

[0632] b) In one example, the default motion candidate may be defined as a (0, 0) motion vector and reference picture index equal to 0 (if available) for both reference picture lists.

[0633] 10. It is proposed that the default motion information of the ATMVP process (e.g., MVdefault0 and / or MVdefault1 used in the ATMVP as disclosed in 2.3.5.1.2) can be derived based on the position used in the sub-block motion information derivation process. Using the proposed method, no further motion information is required for the sub-block, as the default motion information will be directly assigned.

[0634] a) In one example, instead of using the center position of the current block, the center position of a sub-block (eg, the center sub-block) within the current block may be used.

[0635] b) If Fig.31BAn example is given as shown. Fig.31A An example of where the default motion information is identified in the current standard is shown.

[0636] 11. The following methods are always available for proposing ATMVP candidates:

[0637] a) Assuming the center point of the current block is (x0, y0), the corresponding position of locating (x0, y0) in the co-located picture is M = (x0 + MV′x, y0 + MV′y). Find the block Z covering M. If Z is intra-coded, derive MVdefault0 and MVdefault1 through some methods proposed in Project 6.

[0638] b) Alternatively, without locating block Z to obtain motion information, some of the methods proposed in item 8 are directly applied to obtain MVdefault0 and MVdefault1.

[0639] c) Alternatively, the default motion candidate used in the ATMVP process is always available. If it is set to be unavailable based on the current design (e.g., the temporal block is intra-coded), other motion vectors can be used as the default motion candidate instead.

[0640] i. In one example, the scheme in international application PCT / CN2018 / 124639 can be applied, which is incorporated herein by reference.

[0641] d) Alternatively, furthermore, whether an ATMVP candidate is always available depends on other high-level grammatical information.

[0642] i. In one example, ATMVP candidates may always be set as available only when the ATMVP enabled flag at the slice / slice / picture header or other video unit is set to true (inferred as true).

[0643] ii. In one example, the above method may be applicable only when the ATMVP enabled flag in the slice header / picture header or other video unit is set to true and the current picture is not an IRAP picture and the current picture is not inserted into RefPicList0 with reference index equal to 0.

[0644] e) A fixed index or a fixed index group is assigned to the ATMVP candidate. When the ATMVP candidate is always unavailable, the fixed index / group index can be inferred to other kinds of motion candidates (such as affine candidates).

[0645] 12. It is proposed that whether a zero-motion affine Merge candidate is placed in the sub-block Merge candidate list should depend on whether affine prediction is enabled.

[0646] a) For example, if the affine enabled flag is turned off (sps_affine_enabled_flag is equal to 0), the zero-motion affine Merge candidate is not placed in the sub-block Merge candidate list.

[0647] b) Alternatively, furthermore, a default motion vector candidate is alternatively added as a non-affine candidate.

[0648] 13. It is proposed that non-affine filling candidates can be placed in the sub-block Merge candidate list.

[0649] a) If the sub-block Merge candidate list is not satisfied, zero-motion non-affine filling candidates can be filled.

[0650] b) When such a filling candidate is selected, the affine_flag of the current block should be set to 0.

[0651] c) Alternatively, if the Sub-block Merge candidate list is not satisfied and the affine usage flag is off, the zero-motion non-affine fill candidate is placed in the Sub-block Merge candidate list.

[0652] 14. Assume that MV0 and MV1 represent the MVs in reference list 0 and reference list 1 covering the block at the corresponding position (for example, MV0 and MV1 may be MVZ_0 and MVZ_1 or MVZS_0 and MVZS_1 described in Section 2.3.5.1.2). MV0′ and MV1′ represent the MVs in reference list 0 and reference list 1 to be derived for the current block or subblock. Then MV0′ and MV1′ should be derived by scaling.

[0653] a) MV0, if the co-located picture is in reference list 1;

[0654] b) MV1, if the co-located picture is in reference list 0.

[0655] 15. When the current picture is considered as a reference picture with an index set to M (e.g., 0) in a reference picture list X (PicRefListX, e.g., X=0), the ATMVP and / or TMVP enable / disable flags may be inferred as false for a slice / slice or other type of video unit. Here, M may be equal to the target reference picture index, whose motion information of the temporal blocks should be scaled to PicRefListX during the ATMVP / TMVP process.

[0656] a) Alternatively, furthermore, the above method is only applicable when the current picture is an intra random access point (IRAP) picture.

[0657] b) In one example, the ATMVP and / or TMVP enable / disable flags may be inferred to be false when the current picture is considered as a reference picture with an index in PicRefListX set to M (e.g., 0) and / or the current picture is considered as a reference picture with an index in PicRefListY set to N (e.g., 0). The variables M and N represent the target reference picture indexes used in the TMVP or ATMVP process.

[0658] c) For the ATMVP process, the constraint confirmation bitstream shall follow the following rule: the co-located picture from which the motion information of the current block is derived shall not be the current picture.

[0659] d) Alternatively, when the above condition is true, the ATMVP or TMVP process is not called.

[0660] 16. It is proposed that if a reference picture with an index set to M (eg, 0) in the reference picture list X (PicRefListX, eg, X=0) of the current block is the current picture, ATMVP may still be enabled for the block.

[0661] a) In one example, the motion information of all sub-blocks points to the current picture.

[0662] b) In one example, when obtaining motion information of a sub-block from a time domain block, the time domain block applies at least one reference picture codec pointing to the current picture of the time domain block.

[0663] c) In one example, when obtaining the motion information of a sub-block from a temporal block, no scaling operation is applied.

[0664] 17. The encoding and decoding methods of sub-block Merge index are consistent, regardless of whether ATMVP is used.

[0665] a) In one example, for the first L bins, they are context-coded. For the remaining bins, they are bypass-coded. In one example, L is set to 1.

[0666] b) Alternatively, for all bins, they are context encoded.

[0667] 18. The MV (MVx, MVy) used in ATMVP to locate corresponding blocks of different pictures (e.g., TMV in 2.3.5.1.2) can be right-shifted to integer precision (referred to as (MVx′, MVy′) using the same rounding method as in the MV scaling process.

[0668] a) Alternatively, the MV used in ATMVP to locate corresponding blocks in different pictures (eg, TMV in 2.3.5.1.2) may be right-shifted to integer precision with the same rounding method as in the MV averaging process.

[0669] b) Alternatively, the MV used in ATMVP to locate corresponding blocks in different pictures (eg, TMV in 2.3.5.1.2) may be right-shifted to integer precision with the same rounding method as in the Adaptive MV Resolution (AMVR) process.

[0670] 19. The MV (MVx, MVy) used in ATMVP to locate corresponding blocks in different pictures (e.g., TMV in 2.3.5.1.2) can be right-shifted to integer precision (referred to as (MVx′, MVy′) by rounding towards zero.

[0671] a) For example, MVx′=(MVx+((1<<N)>>1)-(MVx>=0?1:0))>>N; N is an integer representing the MV resolution, for example, N=4.

[0672] i. For example, MVx′=(MVx+(MVx>=0?7:8))>>4.

[0673] b) For example, MVy′=(MVy+((1<<N)>>1)-(MVy>=0?1:0))>>N; N is an integer representing the MV resolution, for example, N=4.

[0674] i. For example, MVy′=(MVy+(MVy>=0?7:8))>>4.

[0675] 20. In one example, the MVs (MVx, MVy) in points 18 and 19 are used to locate the corresponding blocks to derive the default motion information used in ATMVP, such as using the center position and shifted MV of the sub-block, or using the top left position and shifted MV of the current block.

[0676] a) In one example, the MV (MVx, MVy) is used to locate the corresponding block during the ATMVP process to derive motion information of the sub-block in the current block, such as using the center position of the sub-block and the shifted MV.

[0677] 21. The methods proposed in points 18, 19 and 20 can be applied to other coding tools associated with locating a reference block in a different picture or in the current picture having a motion vector.

[0678] 22. The MVs (MVx, MVy) used in ATMVP to locate corresponding blocks in different pictures (eg, TMV in 2.3.5.1.2) can be scaled even if they point to co-located pictures.

[0679] a) In one example, if the width and / or height of the co-located picture (or the consistency window therein) is different from the current picture (or the consistency window therein), the MV may be scaled.

[0680] b) Assume that the width and height of the (consistency window) of the co-located picture are denoted as W1 and H1, respectively. Then the width and height of (the consistency window of) the current picture are denoted as W2 and H2, respectively. Then MV (MVx, MVy) can be scaled as MVx′=MVx*W1 / W2 and MVy′=MVy*H1 / H2.

[0681] 23. The center point of the current block used to derive motion information in the ATMVP process (such as position (x0, y0) in 2.3.5.1.2) can be further modified by scaling and / or adding an offset.

[0682] a) In one example, if the width and / or height of the co-located picture (or the consistency window therein) is different from the current picture (or the consistency window therein), the center point may be further modified.

[0683] b) Assume that the top left position of the consistency window in the co-located picture is referred to as X1 and Y1. The top left position of the consistency window defined in the current picture is referred to as X2 and Y2. The width and height of (the consistency window of) the co-located picture are referred to as W1 and H1, respectively. The width and height of (the consistency window of) the current picture are referred to as W2 and H2, respectively. Then (x0, y0) can be modified as x0′=(x0-X2)*W1 / W2+X1 and y0′=(y0-Y2)*H1 / H2+Y1.

[0684] i. Alternatively, x0′=x0*W1 / W2, y0′=y0*H1 / H2.

[0685] 24. The corresponding position used to derive motion information in the ATMVP process (such as position M in 2.3.5.1.2) can be further modified by scaling and / or adding an offset

[0686] a) In one example, if the width and / or height of the co-located picture (or the consistency window therein) is different from the current picture (or the consistency window therein), the corresponding position may be further modified.

[0687] b) Assume that the top left position of the consistency window in the co-located picture is referred to as X1 and Y1. The top left position of the consistency window defined in the current picture is referred to as X2 and Y2. The width and height of (the consistency window of) the co-located picture are referred to as W1 and H1, respectively. The width and height of (the consistency window of) the current picture are referred to as W2 and H2, respectively. Then M(x, y) can be modified as x′=(x-X2)*W1 / W2+X1 and y′=(y-Y2)*H1 / H2+Y1.

[0688] i. Alternatively, x′=x*W1 / W2, y′=y*H1 / H2.

[0689] Sub-image related

[0690] 25. In one example, if positions (i, j) and (i, j-1) belong to different sub-pictures, the width of the sub-picture S ending at the (j-1) column can be set equal to j minus the leftmost column of the sub-picture S.

[0691] a) Examples based on JVET-O2001-vE are emphasized in bold italics below.

[0692]

[0693]

[0694] 26. In one example, the height of a sub-picture S ending at (NumSubPicGridRows-1) rows may be set to (NumSubPicGridRows-1) minus the topmost row of the sub-picture S plus one.

[0695] a) An embodiment based on JVET-O2001-vE is highlighted below.

[0696]

[0697]

[0698] 27. In one example, the width of a sub-picture S that ends at (NumSubPicGridColumns-1) columns can be set to (NumSubPicGridColumns-1) minus the leftmost column of the sub-picture S plus 1.

[0699] a) An embodiment based on JVET-O2001-vE is highlighted below.

[0700]

[0701]

[0702] 28. The sub-picture grid can be an integer multiple of the CTU size.

[0703] a) The examples based on JVET-O2001-vE are emphasized below

[0704] subpic_grid_col_width_minus1 plus 1 specifies the width of each element of the sub-picture identifier grid in units of CtbSizeY. The length of the syntax element is Ceil(Log2(pic_width_max_in_luma_samples / CtbSizeY)) bits.

[0705] The variable NumSubPicGridCols is derived as follows:

[0706] NumSubPicGridCols=(pic_width_max_in_luma_samples+subpic_grid_col_width_minus1*CtbSizeY+CtbSizeY-1) /

[0707] (subpic_grid_col_width_minus1*CtbSizeY+CtbSizeY)(7-5)

[0708] subpic_grid_row_height_minus1 plus 1 specifies the height of each element of the sub-picture identifier grid in units of 4 samples. The length of the syntax element is Ceil(Log2(pic_height_max_in_luma_samples / CtbSizeY)) bits.

[0709] The variable NumSubPicGridRows is derived as follows:

[0710] NumSubPicGridRows=(pic_height_max_in_luma_samples+subpic_grid_row_height_minus1*CtbSizeY+CtbSizeY-1) /

[0711] (subpic_grid_row_height_minus1*CtbSizeY+CtbSizeY) (7-6)

[0712] 29. Add consistency constraints to ensure that sub-images do not overlap each other and that all sub-images can cover the entire image.

[0713] a) An example based on JVET-O2001-vE is indicated as follows.

[0714] Any subpic_grid_idx[i][j] must be equal to idx if all of the following conditions are met:

[0715] i>=SubPicTop[idx] and i<SubPicTop[idx]+SubPicHeight[idx].

[0716] j>=SubPicLeft[idx] and j<SubPicLeft[idx]+SubPicWidth[idx].

[0717] Any subpic_grid_idx[i][j] must be different from idx if the following conditions are not all met:

[0718] i>=SubPicTop[idx] and i<SubPicTop[idx]+SubPicHeight[idx].

[0719] j>=SubPicLeft[idx] andj<SubPicLeft[idx]+SubPicWidth[idx].

[0720] RPR related

[0721] 30. Signal a syntax element (such as a flag) denoted as RPR_flag to indicate whether RPR can be used in a video unit (eg, sequence). RPR_flag can be signaled in SPS, VPS or DPS.

[0722] a) In one example, if it is signaled that RPR is not used (eg, RPR_flag is 0), the overall signaled width / height in the picture parameter set (PPS) may be the same as the maximum width / maximum height signaled in the sequence parameter set (SPS).

[0723] b) In one example, if it is signaled that RPR is not used (eg RPR_flag is 0), then the overall width / height is not signaled in the PPS and is inferred to be the maximum width / maximum height signaled in the SPS.

[0724] c) In one example, if the signaling indicates that RPR is not to be used (eg, RPR_flag is 0), the consistency window information is not used in the decoding process. Otherwise (the signaling indicates that RPR is to be used), the consistency window information may be used in the decoding process.

[0725] 31. It is proposed that the interpolation filter used to derive the prediction block of the current block in the motion compensation process can be selected depending on whether the resolution of the reference picture is different from the current picture, or whether the width and / or height of the reference picture is larger than the current picture.

[0726] a. In one example, an interpolation filter with fewer taps may be applied when condition A is satisfied, where condition A depends on the dimensions of the current picture and / or the reference picture.

[0727] i. In one example, condition A is that the resolution of the reference picture is different from that of the current picture.

[0728] ii. In one example, condition A is that the width and / or height of the reference picture is larger than that of the current picture.

[0729] iii. In one example, condition A is W1>a*W2 and / or H1>b*H2, where (W1, H1) represents the width and height of the reference picture, and (W2, H2) represents the width and height of the current picture, and a and b are two factors, for example a=b=1.5.

[0730] iv. In one example, condition A may also depend on whether bidirectional prediction is used.

[0731] 1) Condition A is satisfied only when bidirectional prediction is used for the current block.

[0732] v. In one example, condition A may depend on M and N, where M and N represent the width and height of the current block.

[0733] 1) For example, condition A is satisfied only when M*N<=T, where T is an integer such as 64.

[0734] 2) For example, condition A is satisfied only when M<=T1 or N<=T2, where T1 and T2 are integers, such as T1=T2=4.

[0735] 3) For example, condition A is satisfied only when M<=T1 and N<=T2, where T1 and T2 are integers, for example, T1=T2=4.

[0736] 4) For example, condition A is satisfied only when M*N<=T, or M<=T1 or N<=T2, where T, T1 and T2 are integers, for example T=64, T1=T2=4.

[0737] 5) In one example, the less than condition in the above sub-points can be replaced with greater than.

[0738] vi. In one example, 1-tap filtering is applied. In other words, integer pixels without filtering are output as the interpolation result.

[0739] vii. In one example, bilinear filtering is applied when the resolution of the reference picture is different from the current picture.

[0740] viii. In one example, when the resolution of the reference picture is different from that of the current picture, or the width and / or height of the reference picture is larger than that of the current picture, a 4-tap filter or a 6-tap filter is applied.

[0741] 1) 6-tap filtering can also be used for affine motion compensation.

[0742] 2) 4-tap filtering can also be used for interpolation of chroma samples.

[0743] b. Whether and / or how to apply the method disclosed in point 31 may depend on the color components.

[0744] i. For example, the method is applied only to the luma component.

[0745] c. Whether and / or how to apply the method disclosed in point 31 may depend on the interpolation filtering direction.

[0746] i. For example, the method is only applied to horizontal filtering.

[0747] ii. For example, the method is only applied to vertical filtering.

[0748] CIIP related

[0749] 32. The intra prediction signaling used in the CIIP process can be performed at the TU level rather than the CU level (for example, using reference samples outside the TU rather than the CU).

[0750] a) In one example, if the CU width or height is larger than the maximum transform unit size, the CU may be divided into multiple TUs, and intra / inter prediction may be generated for each TU, for example, using reference samples outside the TU.

[0751] b) In one example, if the maximum transform size K is less than 64 (such as K=32), the intra prediction used in CIIP is performed in a recursive manner, as in normal intra coding blocks.

[0752] c) For example, a KM×KN CIIP codec block (M and N are integers) is divided into MN K×K blocks, and intra prediction is performed on each K×K block. The intra prediction of the later coded / decoded K×K block can depend on the reconstructed samples of the previously coded / decoded K×K block.

[0753] 5. Additional Exemplary Embodiments

[0754] 5.1 Example #1: Example of syntax design in SPS / PPS / slice header / slice group header

[0755] The changes compared to the VTM3.0.1rc1 reference software are highlighted in large bold as follows:

[0756]

[0757]

[0758] 5.2 Example #2: Example of syntax design in SPS / PPS / slice header / slice group header

[0759] 7.3.2.1 Sequence Parameter Set RBSP Syntax

[0760]

[0761]

[0762] sps_sbtmvp_enabled_flag equal to 1 specifies that the subblock based temporal motion vector predictor can be used in the CVS in decoding all slices of pictures with slice_type not equal to 1. sps_sbtmvp_enabled_flag equal to 0 specifies that the subblock based temporal motion vector predictor is not used in the CVS. When sps_sbtmvp_enabled_flag is not present, it is inferred to be equal to 0.

[0763] five_minus_max_num_subblock_merge_cand specifies 5 minus the maximum number of subblock-based merge motion vector prediction (MVP) candidates supported in a slice. When five_minus_max_num_subblock_merge_cand is not present, it is inferred to be equal to 5-sps_sbtmvp_enabled_flag. The maximum number of subblock-based merge MVP candidates MaxNumSubblockMergeCand is derived as follows:

[0764] MaxNumSubblockMergeCand=5-five_minus_max_num_subblock_merge_cand (7-45)

[0765] The value of MaxNumSubblockMergeCand should be in the range of 0 to 5 (inclusive).

[0766] 8.3.4.2 Derivation of motion vectors and reference indices in sub-block merge mode

[0767] The inputs to this process are:

[0768] ..[No changes to the current VVC draft specification].

[0769] The output of this process is:

[0770] ...[No changes to the current VVC draft specification].

[0771] The variables numSbX, numSbY and subblock Merge candidate list subblockMergeCandList are derived through the following sequential steps:

[0772] When sps_sbtmvp_enabled_flag is equal to 1 and (the current picture is an IRAP and index 0 of reference picture list 0 is the current picture) is not true, the following applies:

[0773] Invoke the process of deriving Merge candidates from adjacent codec units as specified in clause 8.3.2.3, with luma codec block position (xCb, yCb), luma codec block width cbWidth, luma codec block height cbHeight and luma codec block width as input, and output as availability flags availableFlagA0, availableFlagA1, availableFlagB0, availableFlagB1 and availableFlagA2 lagB2, reference indices refIdxLXA0, refIdxLXA1, refIdxLXB0, refIdxLXB1 and refIdxLXB2, prediction list usage flags predFlagLXA0, predFlagLXA1, predFlagLXB0, predFlagLXB1 and predFlagLXB2, and motion vectors mvLXA0, mvLXA1, mvLXB0, mvLXB1 and mvLXB2, where X is 0 or 1.

[0774] Invoke the derivation process of the sub-block based temporal Merge candidate as specified in clause 8.3.4.3, with luma position (xCb, yCb), luma codec block width cbWidth, luma codec block height cbHeight, availability flags availableFlagA0, availableFlagA1, availableFlagB0, availableFlagB1, reference indices refIdxLXA0, refIdxLXA1, refIdxLXB0, refIdxLXB1, prediction list usage flags predFlagLXA0, predFlagLXA1, predFlagLXB0, pr edFlagLXB1 and motion vectors mvLXA0, mvLXA1, mvLXB0, mvLXB1 are taken as input, and the output is the availability flag availableFlagSbCol, the number of luminance codec sub-blocks in the horizontal direction numSbX and the number in the vertical direction numSbY, the reference index refIdxLXSbCol, the luminance motion vector mvLXSbCol[xSbIdx][ySbIdx] and the prediction list usage flag predFlagLXSbCol[xSbIdx][ySbIdx], where xSbIdx=0..numSbX-1, ySbIdx=0..numSbY-1, and X is 0 or 1.

[0775] When sps_affine_enabled_flag is equal to 1, the sample point locations (xNbA0, yNbA0), (xNbA1, yNbA1), (xNbA2, yNbA2), (xNbB0, yNbB0), (xNbB1, yNbB1), (xNbB2, yNbB2), (xNbB3, yNbB3) and the variables numSbX and numSbY are derived as follows:

[0776] [No changes to the current VVC draft specification].

[0777] 5.3 Example #3 of MV rounding

[0778] The syntax changes are based on JVET-O2001-vE.

[0779] 8.5.5.3 Derivation of Sub-Block-Based Temporal Merge Candidates

[0780] …

[0781] The position (xColSb, yColSb) of the co-located sub-block within ColPic is derived as follows.

[0782] 1. The following apply:

[0783] yColSb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1<<CtbLog2SizeY)-1), ySb+((tempMv[1]+8-(tempMv[1]>=0))>>4))

[0784] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0785] xColSb=Clip3(xCtb, Min(SubPicRightBoundaryPos, xCtb+(1<<CtbLog2SizeY)+3), xSb+((tempMv[0]+8+(tempMV[0]>=0))>>4))

[0786] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0787] xColSb=Clip3(xCtb, Min(CurPicWidthInSamplesY-1, xCtb+(1<<CtbLog2SizeY)+3), xSb+(tempMv[0]+8+(tempMV[0]>=0))>>4))

[0788] …

[0789] 8.5.5.4 Derivation process of sub-block-based temporal merge basic motion data

[0790] …

[0791] The position (xColCb, yColCb) of the co-located block within ColPic is derived as follows.

[0792] - The following apply:

[0793] yColCb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1<<CtbLog2SizeY)-1), yColCtrCb+((tempMv[1]+8-(tempMv[1]>=0))>>4))

[0794] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0795] xColCb=Clip3(xCtb, Min(SubPicRightBoundaryPos, xCtb+(1<<CtbLog2SizeY)+3), xColCtrCb+((tempMv[0]+8+(tempMV[0]>=0))>>4))

[0796] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0, the following applies:

[0797] xColCb=Clip3(xCtb, Min(CurPicWidthInSamplesY-1, xCtb+(1<<CtbLog2SizeY)+3), xColCtrCb+((tempMv[0]+8+(tempMV[0]>=0))>>4))

[0798] 5.3 Example #3: Example of MV rounding.

[0799] The syntax changes are based on JVET-O2001-vE.

[0800] 8.5.5.3 Derivation of Sub-Block-Based Temporal Merge Candidates

[0801] …

[0802] The position (xColSb, yColSb) of the co-located sub-block within ColPic is derived as follows.

[0803] 1. The following apply:

[0804] -yColSb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1<<CtbLog2SizeY)-1), ySb+((tempMv[1]+8-(tempMv[1]>=0))>>4))

[0805] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0806] -xColSb=Clip3(xCtb, Min(SubPicRightBoundaryPos, xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+8+(tempMV[0]>=0))>>4))

[0807] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0808] -xColSb=Clip3(xCtb, Min(CurPicWidthInSamplesY-1, xCtb+(1<<CtbLog2SizeY)+3), xSb+(tempMv[0]+8+(tempMV[0]>=0))>>4))

[0809] …

[0810] 8.5.5.4 Derivation process of sub-block-based temporal merge basic motion data

[0811] …

[0812] The position (xColCb, yColCb) of the co-located block within ColPic is derived as follows.

[0813] - The following apply:

[0814] -yColCb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1<<CtbLog2SizeY)-1), yColCtrCb+((tempMv[1]+8-(tempMv[1]>=0))>>4))

[0815] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0816] -xColCb=Clip3(xCtb, Min(SubPicRightBoundaryPos, xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+8+(tempMV[0]>=0))>>4))

[0817] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0, the following applies:

[0818] -xColCb=Clip3(xCtb, Min(CurPicWidthInSamplesY-1, xCtb+(1<<CtbLog2SizeY)+3), xColCtrCb+((tempMv[0]+8+(tempMV[0]>=0))>>4))

[0819] 5.4 Embodiment #4: A second example of MV rounding.

[0820] 8.5.5.3 Derivation of Sub-Block-Based Temporal Merge Candidates

[0821] The inputs to this process are:

[0822] - the luma position (xCb, yCb) of the top left sample of the current luma codec block relative to the top left luma sample of the current picture,

[0823] - variable cbWidth, specifies the width of the current codec block in luminance samples,

[0824] - The variable cbHeight specifies the height of the current codec block in luma samples.

[0825] - the availability flag availableFlagA1 of the adjacent codec unit,

[0826] - the reference index of the adjacent codec unit, refIdxLXA1, where X is 0 or 1,

[0827] - The prediction list of the adjacent codec unit uses the flag predFlagLXA1, where X is 0 or 1,

[0828] - The motion vector mvLXA1 of the adjacent codec with 1 / 16 fractional sample accuracy, where X is 0 or 1.

[0829] The output of this process is:

[0830] -Availability flag availableFlagSbCol,

[0831] - the number of luma codec sub-blocks in the horizontal direction numSbX and in the vertical direction numSbY,

[0832] - reference indices refIdxL0SbCol and refIdxL1SbCol,

[0833] - Luma motion vectors mvL0SbCol[xSbIdx][ySbIdx] and mvL1SbCol[xSbIdx][ySbIdx] with 1 / 16 fractional sample accuracy, where xSbIdx=0..numSbX-1, ySbIdx=0..numSbY-1,

[0834] - The prediction list uses flags predFlagL0SbCol[xSbIdx][ySbIdx] and predFlagL1SbCol[xSbIdx][ySbIdx], where xSbIdx=0..numSbX-1, ySbIdx=0..numSbY-1.

[0835] The availability flag availableFlagSbCol is derived as follows.

[0836] - availableFlagSbCol is set equal to 0 if one or more of the following conditions are true.

[0837] -slice_temporal_mvp_enabled_flag is equal to 0.

[0838] -sps_sbtmvp_enabled_flag is equal to 0.

[0839] -cbWidth is less than 8.

[0840] -cbHeight is less than 8.

[0841] - Otherwise, the following sequential steps apply:

[0842] 1. The position of the top left sample point (xCtb, yCtb) of the luma codec tree block containing the current codec block and the position of the bottom right center sample point (xCtr, yCtr) of the current luma codec block are derived as follows:

[0843] -xCtb=(xCb>>CtuLog2Size)<<CtuLog2Size (8-542)

[0844] -yCtb=(yCb>>CtuLog2Size)<<CtuLog2Size (8-543)

[0845] -xCtr=xCb+(cbWidth / 2) (8-544)

[0846] -yCtr=yCb+(cbHeight / 2) (8-545)

[0847] 2. The luma position (xColCtrCb, yColCtrCb) is set equal to the top left sample of the co-located luma codec block at the position given by (xCtr, yCtr) within ColPic relative to the top left luma sample of the co-located picture specified by ColPic.

[0848] 3. Call the derivation process of the sub-block based temporal Merge basic motion data as specified in clause 8.5.5.4, with position (xCtb, yCtb), position (xColCtrCb, yColCtrCb), availability flag availableFlagA1 and prediction list usage flag predFlagLXA1, and reference index refIdxLXA1, and motion vector mvLXA1, where X is 0 and 1 as input, and motion vector ctrMvLX and prediction list usage flag ctrPredFlagLX of the co-located block, where X is 0 and 1, and temporal motion vector tempMv as output.

[0849] 4. The variable availableFlagSbCol is derived as follows:

[0850] - If both ctrPredFlagL0 and ctrPredFlagL1 are equal to 0, availableFlagSbCol is set equal to 0.

[0851] - Otherwise, availableFlagSbCol is set equal to 1.

[0852] When availableFlagSbCol is equal to 1, the following applies:

[0853] - The variables numSbX, numSbY, sbWidth, sbHeight and refIdxLXSbCol are derived as follows:

[0854] -numSbX=cbWidth>>3 (8-546)

[0855] -numSbY=cbHeight>>3 (8-547)

[0856] -sbWidth=cbWidth / numSbX (8-548)

[0857] -sbHeight=cbHeight / numSbY (8-549)

[0858] -refIdxLXSbCol=0 (8-550)

[0859] - For xSbIdx=0..numSbX-1 and ySbIdx=0..numSbY-1, the motion vector mvLXSbCol[xSbIdx][ySbIdx] and the prediction list are derived using the flag predFlagLXSbCol[xSbIdx][ySbIdx] as follows:

[0860] - Luma position (xSb, ySb), specifies the top left sample of the current codec sub-block relative to the top left luma sample of the current picture, derived as follows:

[0861] -xSb=xCb+xSbIdx*sbWidth+sbWidth / 2 (8-551)

[0862] -ySb=yCb+ySbIdx*sbHeight+sbHeight / 2 (8-552)

[0863] The position (xColSb, yColSb) of the co-located sub-block within ColPic is derived as follows.

[0864] 1. The following apply:

[0865] -yColSb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1<<CtbLog2SizeY)-1), ySb+((tempMv[1]+8-(tempMv[1]>=0?1:0))>>4)) (8-553)

[0866] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0867] -xColSb=Clip3(xCtb, Min(SubPicRightBoundaryPos, xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+8-(tempMv[0]>=0?1:0))>>4)) (8-554)

[0868] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0869] -xColSb=Clip3(xCtb, Min(CurPicWidthInSamplesY-1, xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+8-(tempMv[0]>=0?1:0))>>4)) (8-555)

[0870] - The variable currCb specifies the luma codec block covering the current codec sub-block within the current picture.

[0871] - The variable colCb specifies the luma codec block covering the modified position within ColPic given by ((xColSb>>3)<<3, (yColSb>>3)<<3).

[0872] - The luma position (xColCb, yColCb) is set equal to the top left sample of the collocated luma codec block specified by colCb relative to the top left luma sample of the collocated picture specified by ColPic.

[0873] - Invoke the derivation process of co-located motion vectors as specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL0 set equal to 0 and sbFlag set equal to 1, and output the motion vector mvL0SbCol[xSbIdx][ySbIdx] assigned to the sub-block and availableFlagL0SbCol.

[0874] - Invoke the derivation process of the co-located motion vector as specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL1 set equal to 0 and sbFlag set equal to 1, and output the motion vector assigned to the sub-block mvL1SbCol[xSbIdx][ySbIdx] and availableFlagL1SbCol.

[0875] - When availableFlagL0SbCol and availableFlagL1SbCol are both equal to 0, the following applies for X being 0 and 1:

[0876] -mvLXSbCol[xSbIdx][ySbIdx]=ctrMvLX (8-556)

[0877] -predFlagLXSbCol[xSbIdx][ySbIdx]=ctrPredFlagLX (8-557)

[0878] 8.5.5.4 Derivation process of temporal merge basic motion data based on sub-blocks The input of this process is:

[0879] - the position of the top left sample of the luma codec treeblock containing the current codec block (xCtb, yCtb),

[0880] - The position of the top left sample of the co-located luma codec block covering the bottom right center sample (xColCtrCb, yColCtrCb).

[0881] - the availability flag availableFlagA1 of the adjacent codec unit,

[0882] - the reference index of the adjacent codec unit refIdxLXA1,

[0883] - The prediction list of the adjacent codec unit uses the flag predFlagLXA1,

[0884] - The motion vector mvLXA1 of the neighboring codec with 1 / 16 fractional sample accuracy.

[0885] The output of this process is:

[0886] - motion vectors ctrMvL0 and ctrMvL1,

[0887] - The prediction list uses the flags ctrPredFlagL0 and ctrPredFlagL1,

[0888] - Temporal motion vector tempMv.

[0889] The variable tempMv is set as follows:

[0890] -tempMv[0]=0 (8-558)

[0891] -tempMv[1]=0 (8-559)

[0892] The variable currPic specifies the current picture.

[0893] When availableFlagA1 is equal to true, then the following applies:

[0894] - tempMv is set equal to mvL0A1 if all of the following conditions are true:

[0895] -predFlagL0A1 is equal to 1,

[0896] -DiffPicOrderCnt(ColPic, RefPicList[0][refIdxL0A1]) is equal to 0,

[0897] - Otherwise, if all of the following conditions are true, tempMv is set equal to mvL1A1:

[0898] -slice_type equals B,

[0899] -predFlagL1A1 is equal to 1,

[0900] - DiffPicOrderCnt(ColPic, RefPicList[1][refIdxL1A1]) is equal to 0. The position (xColCb, yColCb) of the co-located block within ColPic is derived as follows.

[0901] - The following apply:

[0902] -yColCb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1<<CtbLog2SizeY)-1), yColCtrCb+((tempMv[1]+8-(tempMv[1]>=0?1:0))>>4)) (8-560)

[0903] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0904] -xColCb=Clip3(xCtb, Min(SubPicRightBoundaryPos, xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+8-(tempMv[0]>=0?1:0))>>4)) (8-561)

[0905] - Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to o, the following applies:

[0906] -xColCb=Clip3(xCtb, Min(CurPicWidthInSamplesY-1, xCtb+(1<<CtbLog2SizeY)+3), xColCtrCb+((tempMv[0]+8-(tempMv[0]>=0?1:0))>>4)) (8-562)

[0907] The array colPredMode is set equal to the prediction mode array CuPredMode[0] for the collocated picture specified by ColPic.

[0908] The motion vectors ctrMvL0 and ctrMvL1, and the prediction list are derived using the flags ctrPredFlagL0 and ctrPredFlagL1 as follows:

[0909] - If colPredMode[xColCb][yColCb] is equal to MODE_INTER, the following applies:

[0910] - The variable currCb specifies the luma codec block covering (xCtrCb, yCtrCb) within the current picture.

[0911] - The variable colCb specifies the luma codec block covering the modified position within ColPic given by ((xColCb>>3)<<3, (yColCb>>3)<<3).

[0912] - The luma position (xColCb, yColCb) is set equal to the top left sample of the collocated luma codec block specified by colCb relative to the top left luma sample of the collocated picture specified by ColPic.

[0913] - Invoke the derivation process of co-located motion vectors specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL0 set equal to 0 and sbFlag set equal to 1, and the output is assigned to ctrMvL0 and ctrPredFlagL0.

[0914] - Invoke the derivation process of co-located motion vectors specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL1 set equal to 0 and sbFlag set equal to 1, and the output is assigned to ctrMvL1 and ctrPredFlagL1.

[0915] - Otherwise, the following applies:

[0916] -ctrPredFlagL0=0 (8-563)

[0917] -ctrPredFlagL1=0 (8-564)

[0918] 5.5 Embodiment #5: A third example of MV rounding.

[0919] 8.5.5.3 Derivation of Sub-Block-Based Temporal Merge Candidates

[0920] The inputs to this process are:

[0921] - the luma position (xCb, yCb) of the top left sample of the current luma codec block relative to the top left luma sample of the current picture,

[0922] - variable cbWidth, specifies the width of the current codec block in luminance samples,

[0923] - The variable cbHeight specifies the height of the current codec block in luma samples.

[0924] - the availability flag availableFlagA1 of the adjacent codec unit,

[0925] - the reference index of the adjacent codec unit, refIdxLXA1, where X is 0 or 1,

[0926] - The prediction list of the adjacent codec unit uses the flag predFlagLXA1, where X is 0 or 1,

[0927] - The motion vector mvLXA1 of the adjacent codec with 1 / 16 fractional sample accuracy, where X is 0 or 1.

[0928] The output of this process is:

[0929] -Availability flag availableFlagSbCol,

[0930] - the number of luma codec sub-blocks in the horizontal direction numSbX and in the vertical direction numSbY,

[0931] - reference indexes reffIdxL0SbCol and refIdxL1SbCol,

[0932] - Luma motion vectors mvL0SbCol[xSbIdx][ySbIdx] and mvL1SbCol[xSbIdx][ySbIdx] with 1 / 16 fractional sample accuracy, where xSbIdx=0..numSbX-1, ySbIdx=0..numSbY-1,

[0933] - The prediction list uses flags predFlagL0SbCol[xSbIdx][ySbIdx] and predFlagL1SbCol[xSbIdx][ySbIdx], where xSbIdx=0..numSbX-1, ySbIdx=0..numSbY-1.

[0934] The availability flag availableFlagSbCol is derived as follows.

[0935] - availableFlagSbCol is set equal to 0 if one or more of the following conditions are true.

[0936] -slice_temporal_mvp_enabled_flag is equal to 0.

[0937] -sps_sbtmvp_enabled_flag is equal to 0.

[0938] -cbWidth is less than 8.

[0939] -cbHeight is less than 8.

[0940] - Otherwise, the following sequential steps apply:

[0941] 1. The position of the top left sample point (xCtb, yCtb) of the luma codec tree block containing the current codec block and the position of the bottom right center sample point (xCtr, yCtr) of the current luma codec block are derived as follows:

[0942] -xCtb=(xCb>>CtuLog2Size)<<CtuLog2Size (8-542)

[0943] -yCtb=(yCb>>CtuLog2Size)<<CtuLog2Size (8-543)

[0944] -xCtr=xCb+(cbWidth / 2) (8-544)

[0945] -yCtr=yCb+(cbHeight / 2) (8-545)

[0946] 2. The luma position (xColCtrCb, yColCtrCb) is set equal to the top left sample of the co-located luma codec block at the position given by (xCtr, yCtr) within ColPic relative to the top left luma sample of the co-located picture specified by ColPic.

[0947] 3. Invoke the derivation process of the temporal Merge basic motion data based on the sub-block as specified in clause 8.5.5.4, with position (xCtb, yCtb), position (xColCtrCb, yColCtrCb), availability flag availableFlagA1, and prediction list usage flag predFlagLXA1, and reference index refIdxLXA l , and motion vector mvLXA1, where X is 0 and 1, as input, and the motion vector ctrMvLX and the prediction list of the co-located block use flag ctrPredFlagLX, where X is 0 and 1, and the temporal motion vector tempMv as output.

[0948] 4. The variable availableFlagSbCol is derived as follows:

[0949] - If both ctrPredFlagL0 and ctrPredFlagL1 are equal to 0, availableFlagSbCol is set equal to 0.

[0950] - Otherwise, availableFlagSbCol is set equal to 1.

[0951] When availableFlagSbCol is equal to 1, the following applies:

[0952] - The variables numSbX, numSbY, sbWidth, sbHeight and refIdxLXSbCol are derived as follows:

[0953] -numSbX=cbWidth>>3 (8-546)

[0954] -numSbY=cbHeight>>3 (8-547)

[0955] -sbWidth=cbWidth / numSbX (8-548)

[0956] -sbHeight=cbHeight / numSbY (8-549)

[0957] -refIdxLXSbCol=0 (8-550)

[0958] - For xSbIdx=0..numSbX-1 and ySbIdx=0..numSbY-1, the motion vector mvLXSbCol[xSbIdx][ySbIdx] and the prediction list are derived using the flag predFlagLXSbCol[xSbIdx][ySbIdx] as follows:

[0959] - Luma position (xSb, ySb), specifies the top left sample of the current codec sub-block relative to the top left luma sample of the current picture, derived as follows:

[0960] -xSb=xCb+xSbIdx*sbWidth+sbWidth / 2 (8-551)

[0961] -ySb=yCb+ySbIdx*sbHeight+sbHeight / 2 (8-552)

[0962] The position (xColSb, yColSb) of the co-located sub-block within ColPic is derived as follows.

[0963] 1. The following apply:

[0964] -yColSb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1<<CtbLog2SizeY)-1), ySb+((tempMv[1]+(tempMv[1]>=0?7:8))>>4)) (8-553)

[0965] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[0966] -xColSb=Clip3(xCtb, Min(SubPicRightBoundaryPos, xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+(tempMv[0]>=0?7:8))>>4)) (8-554)

[0967] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[0968] -xColSb=Clip3(xCtb, Min(CurPicWidthInSamplesY-1, xCtb+(1<<CtbLog2SizeY)+3),xSb+((tempMv[0]+(tempMv[0]>=0?7:8))>>4)) (8-555)

[0969] - The variable currCb specifies the luma codec block covering the current codec sub-block within the current picture.

[0970] - The variable colCb specifies the luma codec block covering the modified position within ColPic given by ((xColSb>>3)<<3, (yColSb>>3)<<3).

[0971] - The luma position (xColCb, yColCb) is set equal to the top left sample of the collocated luma codec block specified by colCb relative to the top left luma sample of the collocated picture specified by ColPic.

[0972] - Invoke the derivation process of co-located motion vectors as specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL0 set equal to 0 and sbFlag set equal to 1, and output the motion vector mvL0SbCol[xSbIdx][ySbIdx] assigned to the sub-block and availableFlagL0SbCol.

[0973] - Invoke the derivation process of the co-located motion vector as specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL1 set equal to 0 and sbFlag set equal to 1, and output the motion vector assigned to the sub-block mvL1SbCol[xSbIdx][ySbIdx] and availableFlagL1SbCol.

[0974] - When availableFlagL0SbCol and availableFlagL1SbCol are both equal to 0, the following applies for X being 0 and 1:

[0975] -mvLXSbCol[xSbIdx][ySbIdx]=ctrMvLX(8-556)

[0976] -predFlagLXSbCol[xSbIdx][ySbIdx]=ctrPredFlagLX (8-557)

[0977] 8.5.5.4 Derivation process of temporal merge basic motion data based on sub-blocks The input of this process is:

[0978] - the position of the top left sample of the luma codec treeblock containing the current codec block (xCtb, yCtb),

[0979] - The position of the top left sample of the co-located luma codec block covering the bottom right center sample (xColCtrCb, yColCtrCb).

[0980] - the availability flag availableFlagA1 of the adjacent codec unit,

[0981] - the reference index of the adjacent codec unit refIdxLXA1,

[0982] - The prediction list of the adjacent codec unit uses the flag predFlagLXA1,

[0983] - The motion vector mvLXA1 of the neighboring codec with 1 / 16 fractional sample accuracy.

[0984] The output of this process is:

[0985] - motion vectors ctrMvL0 and ctrMvL1,

[0986] - The prediction list uses the flags ctrPredFlagL0 and ctrPredFlagL1,

[0987] - Temporal motion vector tempMv.

[0988] The variable tempMv is set as follows:

[0989] -tempMv[0]=0 (8-558)

[0990] -tempMv[1]=0 (8-559)

[0991] The variable currPic specifies the current picture.

[0992] When availableFlagA1 is equal to true, then the following applies:

[0993] - tempMv is set equal to mvL0A1 if all of the following conditions are true:

[0994] -predFlagL0A1 is equal to 1,

[0995] -DiffPicOrderCnt(ColPic, RefPicList[0][refIdxL0A1]) is equal to 0,

[0996] - Otherwise, if all of the following conditions are true, tempMv is set equal to mvL1A1:

[0997] -slice_type equals B,

[0998] -predFlagL1A1 is equal to 1,

[0999] -DiffPicOrderCnt(ColPic, RefPicList[1][refIdxL1A1]) is equal to 0.

[1000] The position (xColCb, yColCb) of the co-located block within ColPic is derived as follows.

[1001] - The following apply:

[1002] -yColCb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1<<CtbLog2SizeY)-1), yColCtrCb+((tempMv[1]+(tempMv[1]>=0?7:8))>>4)) (8-560)

[1003] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[1004] -xColCb=Clip3(xCtb, Min(SubPicRightBoundaryPos, xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+(tempMv[0]>=0?7:8))>>4)) (8-561)

[1005] - Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to o, the following applies:

[1006] -xColCb=Clip3(xCtb, Min(CurPicWidthInSamplesY-1, xCtb+(1<<CtbLog2SizeY)+3),xColCtrCb+((tempMv[0]+(tempMv[0]>=0?7:8))>>4)) (8-562)

[1007] The array colPredMode is set equal to the prediction mode array CuPredMode[0] for the collocated picture specified by ColPic.

[1008] The motion vectors ctrMvL0 and ctrMvL1, and the prediction list are derived using the flags ctrPredFlagL0 and ctrPredFlagL1 as follows:

[1009] - If colPredMode[xColCb][yColCb] is equal to MODE_INTER, the following applies:

[1010] - The variable currCb specifies the luma codec block covering (xCtrCb, yCtrCb) within the current picture.

[1011] - The variable colCb specifies the luma codec block covering the modified position within ColPic given by ((xColCb>>3)<<3, (yColCb>>3)<<3).

[1012] - The luma position (xColCb, yColCb) is set equal to the top left sample of the collocated luma codec block specified by colCb relative to the top left luma sample of the collocated picture specified by ColPic.

[1013] - Invoke the derivation process of co-located motion vectors specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL0 set equal to 0 and sbFlag set equal to 1, and the output is assigned to ctrMvL0 and ctrPredFlagL0.

[1014] - Invoke the derivation process of co-located motion vectors specified in clause 8.5.2.12, with as input currCb, colCb, (xColCb, yColCb), refIdxL1 set equal to 0 and sbFlag set equal to 1, and the output is assigned to ctrMvL1 and ctrPredFlagL1.

[1015] - Otherwise, the following applies:

[1016] -ctrPredFlagL0=0 (8-563)

[1017] -ctrPredFlagL1=0 (8-564)

[1018] 8.5.6.3 Fractional Sample Interpolation Process

[1019] 8.5.6.3.1 Overall

[1020] The inputs to this process are:

[1021] - Luma position (xSb, ySb), specifies the top left sample of the current codec sub-block relative to the top left luma sample of the current picture,

[1022] -Variable sbWidth specifies the width of the current codec sub-block.

[1023] -Variable sbHeight specifies the height of the current codec sub-block.

[1024] - motion vector offset mvOffset,

[1025] - refine the motion vector refMvLX,

[1026] - the selected reference picture sample array refPicLX,

[1027] - half-sample interpolation filter index hpelIfIdx,

[1028] - bidirectional optical flow flag bdofFlag,

[1029] -Variable cIdx specifies the color component index of the current block.

[1030] The output of this process is:

[1031] - predSamplesLX, an array of (sbWidth+brdExtSize)x(sbHeight+brdExtSize) predicted sample values.

[1032] The prediction block bounds extension size brdExtSize is derived as follows:

[1033] -brdExtSize=(bdofFlag||(inter_affine_flag[xSb][ySb]&&sps_affine_prof_enabled_flag))? 2:0 (8-752)

[1034] The variable fRefWidth is set equal to PicOutputWidthL of the reference picture in luma samples.

[1035] The variable fRefHeight is set equal to the PicOutputHeightL of the reference picture in luma samples.

[1036] The motion vector mvLX is set to (refMvLX-mvOffset).

[1037] - If cIdx is equal to 0, the following applies:

[1038] - The scaling factors and their fixed point representations are defined as

[1039] -hori_scale_fp=((fRefWidth<<14)+(PicOutputWidthL>>1)) / PicOutputWidthL (8-753)

[1040] -vert_scale_fp=((fRefHeight<<14)+(PicOutputHeightL>>1)) / PicOutputHeightL (8-754)

[1041] - Let (xIntL, yIntL) be the luma position given in integer sample units, and (xFracL, yFracL) be the offset given in 1 / 16 sample units. These variables are used only in this clause to specify fractional sample positions within the reference sample array refPicLX.

[1042] -The left top coordinate of the boundary block filled with reference samples (xSbInt L ,ySbInt L ) is set to (xSb+(mvLX[O]>>4), ySb+(mvLX[1]>>4)).

[1043] - For each luma sample position (x L =0..sbWidth-1+brdExtSize,y L =0..sbHeight-1+brdExtSize), the corresponding predicted brightness sample value predSamplesLX[x L ][y L ] is derived as follows:

[1044] - Make(refxSb L ,refySb L ) and (refx L , refy L ) is the luminance position pointed to by the motion vector (refMvLX[0], refMvLX[1]) given in 1 / 16 sample units. The variable refxSb L ,refxL,refySb L and refy L The derivation is as follows:

[1045] -refxSb L =((xSb<<4)+refMvLX[0])*hori_scale_fp (8-755)

[1046] -refxL=((Sign(refxSb)*((Abs(refxSb)+128)>>8)+x L *((hori_scale_fp+8)>>4))+32)>>6 (8-756)

[1047] -refySb L =((ySb<<4)+refMvLX[1])*vert_scale_fp (8-757)

[1048] -refyL=((Sign(refySb)*((Abs(refySb)+128)>>8)+yL*((vert_scale_fp+8)>>4))+32)>>6 (8-758)

[1049] -Variable xIntL ,yInt L 、xFrac L and yFrac L The derivation is as follows:

[1050] -xInt L =refx L >>4 (8-759)

[1051] -yInt L =refy L >>4 (8-760)

[1052] -xFrac L =refx L &15 (8-761)

[1053] -yFrac L =refy L &15 (8-762)

[1054] - using6TapFlag is set to 1 if all of the following conditions are met:

[1055] -cbWidth[0][xSb][ySb]<=4||cbHeight[0][xSb][ySb]<=4||cbWidth[0][xSb][ySb]*cbHeight[0][xSb][ySb]<=64.

[1056] -PredFlagL0[xSb][ySb]==1&&PredFlagL1[xSb][ySb]==1.

[1057] - If bdofFlag is equal to true or (sps_affine_prof_enabled_flag is equal to true and inter_affine_flag[xSb][ySb] is equal to true), and one or more of the following conditions are true, then the luma integer sample retrieval process is called as specified in clause 8.5.6.3.3 with (xIntL+(xFrac L >>3)-1), yInt L +(yFrac L >>3)-1) and refPicLX as input, derive the predicted brightness sample value predSamplesLX[x L ][y L ].

[1058] 1.x L is equal to 0.

[1059] 2.x L Equal to sbWidth+1.

[1060] 3.y L is equal to 0.

[1061] 4.y L Equal to sbHeight+1.

[1062] Otherwise, the 8-tap interpolation filter process for luma samples is called by invoking the 8-tap interpolation filter process for luma samples as specified in clause 8.5.6.3.2 with (xIntL-(brdExtSize>0?1:0), yIntL-(brdExtSize>0?1:0)), (xFracL, yFracL), (xSbInt L ,ySbInt L ), refPicLX, hpelIfIdx, sbWidth, sbHeight and (xSb, ySb) and using6TapFlag are used as input to derive the predicted luminance sample value predSamplesLX[xL][yL].

[1063] - Otherwise (cIdx is not equal to 0), the following applies:

[1064] - Let (xIntC, yIntC) be the chroma position given in integer sample units, and (xFracC, yFracC) be the offset given in 1 / 32 sample units. These variables are used only in this clause to specify the overall fractional sample position within the reference sample array refPicLX.

[1065] - The left top coordinate (xSbIntC, ySbIntC) of the reference sample filled boundary block is set to ((xSb / SubWidthC)+(mvLX[0]>>5), (ySb / SubHeightC)+(mvLX[1]>>5)).

[1066] For each chroma sample position (xC=0..sbWidth-1, yC=0..sbHeight-1) within the predicted chroma sample array predSamplesLX, the corresponding predicted chroma sample value predSamplesLX[xC][yC] is derived as follows:

[1067] - Make(refxSb C ,refySb C ) and (refx C , refy C ) is the chroma position pointed to by the motion vector (mvLX[0], mvLX[1]) given in 1 / 32 sample units. The variable refxSbC ,refySb C ,refx C and refy C The derivation is as follows:

[1068] -refxSb C =((xSb / SubWidthC<<5)+mvLX[0])*hori_scale_fp (8-763)

[1069] -refx C =((Sign(refxSb C )*((Abs(refxSb C )+256)>>9)+xC*((hori_scale_fp+8)>>4))+16)>>5 (8-764)

[1070] -refySb C =((ySb / SubHeightC<<5)+mvLX[1])*vert_scale_fp (8-765)

[1071] -refy C =((Sign(refySb C )*((Abs(refySb C )+256)>>9)+yC*((vert_scale_fp+8)>>4))+16)>>5 (8-766)

[1072] -Variable xInt C ,yInt C 、xFrac C and yFrac C The derivation is as follows:

[1073] -xInt C =refx C >>5 (8-767)

[1074] -yInt C =refy C >>5 (8-768)

[1075] -xFrac C =refy C &31 (8-769)

[1076] -yFrac C =refy C &31 (8-770)

[1077] - by calling the procedure specified in clause 8.5.6.3.4 with (xIntC, yIntC),

[1078] (xFracC, yFracC), (xSbIntC, ySbIntC), sbWidth, sbHeight and refPicLX are used as input to derive the predicted sample value predSamplesLX[xC][yC].

[1079] 8.5.6.3.2 Luma Sample Interpolation Filtering Process

[1080] The inputs to this process are:

[1081] - Luminance position in integer sample units (xInt L ,yInt L ),

[1082] - Luma position in fractional sample units (xFrac L ,yFrac L ),

[1083] - Luminance position in integer sample units (xSbInt L ,ySbInt L ), specifies the top left sample of the boundary block filled with reference samples relative to the top left luma sample of the reference picture,

[1084] - Luminance reference sample array refPicLX L ,

[1085] - half-sample interpolation filter index hpelIfIdx,

[1086] -Variable sbWidth, specifies the width of the current sub-block,

[1087] -Variable sbHeight specifies the height of the current sub-block.

[1088] - luma position (xSb, ySb), specifies the top left sample of the current subblock relative to the top left luma sample of the current picture,

[1089] -Flag using6TapFlag, specifies whether to use 6-tap interpolation filtering.

[1090] The output of this process is the predicted brightness sample value predSampleLXL

[1091] The variables shift1, shift2, and shift3 are derived as follows:

[1092] - The variable shift1 is set equal to Min(4, BitDepthY-8), the variable shift2 is set equal to 6, and the variable shift3 is set equal to Max(2, 14-BitDepth Y ).

[1093] - The variable picW is set equal to pic_width_in_luma_samples, and the variable picH is set equal to pic_height_in_luma_samples.

[1094] Luminance interpolation filter coefficient f L [p] For each 1 / 16 fractional sample position p is equal to xFracL or yFrac L The derivation is as follows:

[1095] If at least one of the following conditions is met, the luminance interpolation filter coefficient f L [p] is specified in Table 8-12.

[1096] -MotionModelIdc[xSb][ySb] is greater than 0, and both sbWidth and sbHeight are equal to 4,

[1097] -using6TapFlag is equal to 1.

[1098] - Otherwise, the brightness interpolation filter coefficient f L [p] Specified in Table 8-11, depending on hpelIfIdx. Luma position in integer sample units (xInt i ,yInt i ) for i=0..7 is derived as follows:

[1099] - If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, the following applies:

[1100] -xInt i =Clip3(SubPicLeftBoundaryPos,SubPicRightBoundaryPos,xInt L +i-3)(8-771)

[1101] -yInt i =Clip3(SubPicTopBoundaryPos, SubPicBotBoundaryPos, yInt L +i-3) (8-772)

[1102] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies:

[1103]

[1104] -yInt i = Clip3(0, picH-1, yInt L +i-3) (8-774)

[1105] The luminance position in integer sample units is further modified for i=0..7 as follows:

[1106] -xInt i =Clip3(xSbInt L -3,xSbInt L +sbWidth+4,xInt i ) (8-775)

[1107] -yInt i =Clip3(ySbInt L -3,ySbInt L +sbHeight+4,yInt i ) (8-776)

[1108] Predicted brightness sample value predSampleLX L is derived as follows:

[1109] -If xFrac L and yFrac L If both are equal to 0, then predSampleLX L The value of is derived as follows:

[1110] -predSampleLX L =refPicLX L [xInt3][yInt3]<<shift3 (8-777)

[1111] - Otherwise, if xFrac L Not equal to 0 and yFrac L If equal to 0, predSampleLX L The value of is derived as follows:

[1112] -

[1113] - Otherwise, if xFrac L Equal to 0 and yFracL If not equal to 0, predSampleLX L The value of is derived as follows:

[1114] -

[1115] - Otherwise, if xFrac L Not equal to 0 and yFrac L If not equal to 0, predSampleLX L The value of is derived as follows:

[1116] - The sample array temp[n], where n=0..7, is derived as follows:

[1117] -

[1118] -Predicted brightness sample value predSampleLX L is derived as follows:

[1119] -

[1120] Table 8-11 - Luma interpolation filter coefficients f for each 1 / 16 fractional sample position p L [p] specification.

[1121]

[1122] Table 8-12 - Luma interpolation filter coefficients f for each 1 / 16 fractional sample position p in affine motion mode L [p] specification.

[1123]

[1124] Fig.29 is a block diagram of a video processing device 2600. Device 2600 may be used to implement one or more of the methods described herein. Device 2600 may be implemented as a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, and the like. Device 2600 may include one or more processors 2602, one or more memories 2604, and video processing hardware 2606. Processor(s) 2602 may be configured to implement one or more methods described in this document. Memory(s) 2604 may be used to store data and code to implement the methods and techniques described herein. Video processing hardware 2606 may be used to implement some of the techniques described in this document in hardware circuits. In some embodiments, video processing hardware 2606 may be partially or completely in processor 2602, for example, a graphics coprocessor.

[1125] Fig.30 3000 is a flowchart of an exemplary method of video processing. The method includes determining (3002) a maximum number of candidate MLs in a sub-block Merge candidate list or enabling the use of an alternative temporal motion vector prediction (ATMVP) candidate for a current video block based on a state of use of a temporal motion vector prediction (TMVP) or a current picture reference codec (CPR) of the current video block, wherein the state is one of enabled or disabled, and performing (3004) a conversion between the current video block and a bitstream representation of the current video block based on a result of the determination.

[1126] It should be understood that several technologies have been disclosed that will benefit video encoder and decoder embodiments integrated into video processing devices, such as smart phones, laptops, desktop computers, and similar devices, by allowing the use of ATMVP codec tools in encoding or decoding video or images. Various embodiments and technologies may use the following clause-based description.

[1127] 1. A method of video processing, comprising: determining, for a current video block, a maximum number of candidate MLs in a sub-block Merge candidate list or enabling the use of an alternative temporal motion vector prediction (ATMVP) candidate based on a state of use of a temporal motion vector prediction (TMVP) or a state of a current picture reference codec (CPR) of the current video block, wherein the state is one of enabled or disabled, and performing conversion between the current video block and a bitstream representation of the current video block based on a result of the determination.

[1128] 2. The method of clause 1, wherein enabling usage of an ATMVP candidate is further based on a status of ATMVP usage of a current video block.

[1129] 3. The method according to any one of claims 1-2, wherein determining the ML comprises: determining that ATMVP and TMVP are disabled; and determining the ML by excluding ATMVP candidates from a sub-block Merge candidate list.

[1130] 4. The method according to any one of claims 1-2, wherein determining the ML comprises:

[1131] Determining that ATMVP is enabled for the current video block, TMVP is disabled for the current video block, and affine codec is disabled for the current video block; and

[1132] Based on this determination, ML is set to zero.

[1133] 5. The method according to any one of claims 1-2, wherein determining the ML comprises:

[1134] Determining that ATMVP is enabled for the current video block, TMVP is enabled for the current video block, and affine codec is disabled for the current video block; and

[1135] Based on this determination, ML is set to 1.

[1136] 6. The method of any one of claims 1-2, wherein determining the ML is based on a state of ATMVP usage of the current block and whether a co-located reference picture of a current picture of the current video block is a current picture.

[1137] 7. The method of any one of claims 1-2, wherein determining the ML is based on a state of the ATMVP and whether a reference picture with a reference index 0 in a reference picture list of a current picture of a current video block is a current picture.

[1138] 8. A method as described in clause 7, wherein the reference picture list is list 0 or list 1.

[1139] Additional description of clauses 1-8 can be found in items 1-2 listed in Section 4.

[1140] 9. A video processing method, comprising:

[1141] Determining the relationship between a temporal motion vector prediction (TMVP) codec and an alternative TMVP (ATMVP) codec for conversion between a bitstream representing a sequence of video pictures and pixel values ​​of the video pictures, the video pictures using block-based coding, wherein the blocks of the video pictures are organized into slices or slices, and

[1142] Based on the determination, conversion is performed;

[1143] where the relationship specifies applicability, such as one or more of the following:

[1144] (a) In case TMVP is not allowed at slice or picture level, then ATMVP is disabled at slice or picture level;

[1145] (b) In case TMVP is signaled at slice, slice or picture level, ATMVP is signaled at slice, slice or picture level after TMVP;

[1146] (c) The relationship is specified only at the sequence level and is not included at the slice or tile or picture level.

[1147] 10. A method according to clause 9, wherein the relationship is signalled in a single bit in the bitstream representation.

[1148] Additional description of clauses 9-10 can be found in items 3-4 listed in Section 4.

[1149] 11. A video encoding and decoding method, comprising:

[1150] For conversion between a video region and a bitstream representation, determining an amount of a mask to be applied to the location of coordinates of a current block or current sub-block of a video region for a temporal motion vector prediction (TMVP) or advanced temporal motion vector prediction (ATMVP) codec; and

[1151] By applying a mask based on this amount, conversion between a video region and a corresponding bitstream representation is performed.

[1152] 12. A method according to clause 11, wherein the amount of the mask is signalled in the bitstream representation.

[1153] 13. The method according to claims 11-12, wherein in case an indication in a signaling parameter set of the bitstream representation indicates motion compression as disabled, it is set to no mask.

[1154] 14. The method according to any one of claims 11-13, wherein the mask is applied by a bitwise AND operation using an integer MASK with a value equal to ~(2M-1), where M is an integer and "~" represents a bitwise compensation operator.

[1155] 15. The method of any of claims 11-14, wherein applying the mask comprises applying a mask that results in 2K x 2K blocks of video regions that share the same motion information, where K and M are different integers.

[1156] 16. The method according to any one of claims 11 to 15, wherein M=3 or M=4.

[1157] 17. The method according to any one of claims 11-16, wherein M is signaled in a bitstream representation.

[1158] Additional description of clauses 11-17 can be found in items 3-4 listed in Section 4.

[1159] 18. A method for video processing, comprising:

[1160] During conversion between a current video region and a bitstream representation, determining a valid corresponding region for an alternative temporal motion vector prediction (ATMVP) of the current video region, and

[1161] The conversion is performed based on the effective corresponding area,

[1162] Where determination is based on one or more of the following:

[1163] (a) Adaptive determination based on the characteristics of the current video region;

[1164] (b) a basic region, having size MxN pixels, where M and N are integers;

[1165] (c) determining a motion vector for locating a valid corresponding area, wherein determining the motion vector is based on a rule; or

[1166] (d) Determine the default motion vector for ATMVP codec.

[1167] 19. A method according to clause 18, wherein the current video region is a valid corresponding region is a function of: (a) the width or height of the current video region, or (b) a motion vector compression tool applied to the current video region.

[1168] 20. The method according to any one of claims 18-19, wherein the current video region corresponds to a basic region of size MxN which is smaller than a codec tree unit (CTU) region.

[1169] 21. The method of clause 20, wherein the current video block is within a base region, and wherein the valid corresponding region is a co-located base region.

[1170] 22. The method of clause 20, wherein the current video block is within the base region, and wherein the valid corresponding region at least partially does not overlap with the co-located base region.

[1171] 23. A method according to clause 18, wherein the valid corresponding area is in a picture different from the current picture containing the current video area.

[1172] 24. The method of clause 23, wherein the current video region is a current video block, and a temporal motion vector (TMV) corresponding to the current video block is (1) set to a default value, or (2) set to a motion vector in a history-based motion vector predictor table.

[1173] 25. A method according to clause 23, wherein determining a default motion vector for ATMVP encoding comprises (1) setting the default motion vector to a value of (0, 0), or (2) deriving the default motion vector from a history-based motion vector predictor table, or (3) deriving the default motion vector from a neighboring block.

[1174] Additional description of clauses 18-25 can be found in items 7-10 listed in Section 4.

[1175] 26. A method for video processing, comprising:

[1176] Generate a candidate list or motion vectors for a video region using rules; and

[1177] The candidate list performs conversions between video regions and bitstream representations of video regions, where the rules include one or more of the following:

[1178] (a) Rule 1: Candidates corresponding to the alternative temporal motion vector prediction (ATMVP) mode are always considered during conversion;

[1179] (b) Rule 2: The enabled state of the affine mode of the video region is used when determining the availability of the zero-motion affine merge candidate in the sub-block merge candidate list of the sub-block of the video region;

[1180] (c) Rule 3: Use non-affine filling candidates for the sub-block Merge candidate list of the sub-blocks of the video region; or

[1181] (d) Rule 4: The motion vector of the video region is derived from the motion vectors of the blocks covering the corresponding positions according to the alternative temporal motion vector predictor method.

[1182] 27. The method of clause 26, wherein the video region is a current video block, and rule 1 further specifies that the ATMVP default motion vector is determined by determining a block at a corresponding position that covers the corresponding position of a center point of the current video block.

[1183] 28. The method of clause 26, wherein rule 2 further specifies avoiding adding a zero-motion affine merge candidate to the sub-block merge candidate list if the affine mode enable state indicates that affine mode is off for the current video block.

[1184] Additional description of clauses 26-28 can be found in items 11-12 listed in Section 4.

[1185] 29. The method of any one of claims 1-28, wherein converting comprises generating a bitstream from a current video block or a current video region.

[1186] 30. The method of any one of claims 1-28, wherein converting comprises generating pixel values ​​of a current video block or a current video region from a bitstream representation.

[1187] 31. The method of any of clauses 1-28, further comprising: determining that a time domain block in a co-located picture associated with the current video block is encoded in CPR mode, and wherein a default motion candidate is used instead of an ATMVP candidate based on determining that the time domain block is encoded in CPR mode.

[1188] 32. The method of clause 31, wherein the default motion candidate indicates a center position of the current video block.

[1189] 33. The method of clause 31, wherein the default motion candidate is a (0, 0) motion vector and the reference picture index is equal to zero for both reference picture lists.

[1190] 34. The method of any of clauses 1-28, further comprising: determining a position of a current video block, wherein the used default motion information of the ATMVP candidate is based on the determination of the position of the current video block.

[1191] 35. The method of clause 34, wherein the position is based on a position of a sub-block of the current video block.

[1192] 36. The method of any of clauses 1-28, wherein usage of ATMVP candidates is based on a flag provided by a slice, slice, or picture header indicating usage of ATMVP candidates.

[1193] 37. The method of clause 36, wherein the current picture containing the current video block is not an intra random access point (IRAP) picture, and wherein the current picture is not inserted into RefPicList0 with a reference index equal to zero.

[1194] 38. A method as described in any of clauses 1-28, wherein ATMVP or TMVP is enabled or disabled based on a flag that is inferred as false for a slice or slice based on a current picture and a reference picture list X that includes the current video block, the current picture being a reference picture with an index set to M.

[1195] 39. The method as described in any of clauses 1-28 also includes: determining a reference picture with an index set to M in a reference picture list X of the current video block as the current picture, and wherein ATMVP is enabled based on determining that the reference picture with an index set to M in a reference picture list X of the current video block is the current picture.

[1196] 40. A method according to clause 39, wherein the sub-block motion information is defined to point to the current picture.

[1197] 41. The method of clause 39, further comprising: determining that the sub-block motion information is from a temporal block, wherein the temporal block is encoded using a reference picture pointing to a current picture of the temporal block.

[1198] 42. A method according to clause 41, wherein the sub-block motion information is not scaled.

[1199] 43. The method of any of clauses 1-28, wherein converting comprises aligning a sub-block Merge index for encoding or decoding.

[1200] 44. A method according to clause 43, wherein the first number of bins are context-coded and the second number of bins are bypass-coded, the second number of bins not being context-coded.

[1201] 45. A method according to clause 43, wherein the binary number is context encoded.

[1202] 46. ​​A method for video processing, comprising: determining, for a current video block, a motion vector used to locate a corresponding block in a different picture in an alternative temporal motion vector prediction (ATMVP) process for a conversion between the current video block and a codec representation of the current video block based on a right shift and integer process, and performing a conversion between the current video block and the codec representation of the current video block based on a result of the determination.

[1203] 47. The method of clause 46, wherein the right shift rounding process is the same as the right shift rounding process used for motion vector scaling during conversion.

[1204] 48. The method of clause 46, wherein the right shift rounding process results in rounding to an integer value by rounding towards zero.

[1205] 49. A method as described in any of clauses 46-48, wherein the motion vectors used in the ATMVP process are used to derive default motion information.

[1206] The following items provide some exemplary aspects of the techniques described in the items in the previous sections (e.g., items 24-29).

[1207] 50. A method of video processing, comprising: converting between a video picture of a video using sub-pictures and a codec representation of a video segment, determining that a rule of a constraint related to a sub-block is satisfied by the conversion; and performing the conversion according to the rule of the constraint.

[1208] 51. A method according to clause 50, wherein the constraint rule specifies that the width of the sub-image S ending at the (j-1) column is set equal to j minus the leftmost column of the sub-image S, since positions (i, j) and (i, j-1) belong to different sub-images within the video image.

[1209] 52. A method as described in any of clauses 50-51, wherein the rules of the constraint specify that the height of a sub-picture S ending at (NumSubPicGridRows-1) rows is set to (NumSubPicGridRows-1) minus the topmost row of sub-picture S plus one.

[1210] 53. A method as described in any of clauses 50-52, wherein the rules of the constraint specify that the width of a sub-picture S ending at (NumSubPicGridColumns-1) columns is set to (NumSubPicGridColumns-1) minus the leftmost column of the sub-picture S plus 1.

[1211] 54. The method of any of clauses 50-53, wherein the rules of the constraint specify that the size grid of sub-pictures in a video picture is an integer multiple of a codec tree unit size used during conversion.

[1212] 55. The method of any of clauses 50-54, wherein the rules of the constraint specify that all sub-pictures in a video picture do not overlap and together cover the entire video picture.

[1213] The following items provide some exemplary aspects of the techniques described in the items in the previous sections (e.g., items 30-34).

[1214] 56. A method of video processing, comprising: for conversion between a video unit of a video and a codec representation of the video, determining whether a reference picture resampling (RPR) mode is used during the conversion, and performing the conversion based on the determination.

[1215] 57. A method as described in clause 56, wherein a flag in the codec representation indicates the use of RPR mode.

[1216] 58. A method as described in any of clauses 56-57, wherein the RPR mode is indicated at a sequence level or a video parameter set level.

[1217] 59. A method as described in any of clauses 56-59, wherein the RPR mode includes: during conversion, based on whether the resolution of the reference picture is different from the current picture, or whether the width and / or height of the reference picture is larger than the current picture, using interpolation filtering during the motion compensation process to derive a prediction block for the current block of the video unit.

[1218] 60. A method according to any of clauses 1 to 59, wherein converting comprises generating a codec representation from pixel values ​​of the current video block.

[1219] 61. A method as described in any of clauses 1 to 59, wherein converting comprises pixel values ​​of the current video block from a codec representation.

[1220] 62. A video encoder device comprising a processor configured to implement the method of any one of clauses 1 to 61.

[1221] 63. A video decoder device comprising a processor configured to implement the method of any of clauses 1 to 61.

[1222] 64. A computer readable medium having code stored thereon which, when executed by a processor, causes the processor to implement the method according to any one of clauses 1 to 61.

[1223] Fig.32 3200 is a block diagram illustrating an exemplary video processing system 3200 in which the various techniques described herein may be implemented. Various implementations may include some or all of the components of system 3200. System 3200 may include an input 3202 to receive video content. The video content may be received in a raw or uncompressed format, e.g., 8 or 10 bit multi-component pixel values, or may be received in a compressed or encoded format. Input 3202 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces, such as Ethernet, a passive optical network (PON), etc., and wireless interfaces, such as Wi-Fi or a cellular interface.

[1224] System 3200 may include a codec component 3204, which may implement various codecs or coding methods described in this document. Codec component 3204 may reduce the average bit rate of a video from input 3202 to the output of codec component 3204 to generate a codec representation of the video. Codec technology is therefore sometimes referred to as video compression or transcoding codec technology. The output of codec component 3204 may be stored or transmitted via a connected communication, as represented by component 3206. The bitstream (or codec) representation of the storage or communication of the video received at input 3202 may be used by component 3208 to generate pixel values ​​or displayable video that are sent to display interface 3210. The process of generating a user-viewable video from a bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that codec tools or operations are used in encoders, and the corresponding decoding tools or operations of the results of the reverse codecs will be performed by decoders.

[1225] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Displayport, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The techniques described in this document may be implemented in various electronic devices, such as smart phones, laptop computers, smartphones, or other devices capable of digital data processing and / or video display.

[1226] Some embodiments of the disclosed technology are discussed in a clause-based format.

[1227] Some exemplary embodiments of the technology described in Item 22 of Section 4 include:

[1228] A1. A method of visual media processing (e.g., Fig.33 The method 3300 shown in FIG. 3 comprises: determining (3302) a motion vector used in a sub-block based motion vector prediction (sbTMVP) process for a current video block to locate a corresponding block in a co-located picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated based on a scaling operation; and performing (3304) conversion between the current video block and a bitstream representation of the visual media data based on the use of the motion vector.

[1229] A2. A method according to clause A1, wherein the scaling operation includes at least one of the following: a right shift operation, a left shift operation and / or a rounding operation applied to the motion vector used in the sbTMVP process.

[1230] A3. The method of clause A2, wherein the right shift operation causes the motion vector to be rounded to an integer value by rounding towards zero.

[1231] A4. The method according to clause A1, further comprising:

[1232] Upon determining that the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are different from the dimensions of the current picture associated with the current block or the dimensions of the window associated with the current picture, a scaling operation is applied to the motion vector used in the sbTMVP process.

[1233] A5. A method according to clause A4, wherein the result of the scaling operation on the motion vector used in the sbTMVP process is expressed as:

[1234] MVx′=MVx*W1 / W2 and MVy′=MVy*H1 / H2, where MVx, MVy are

[1235] Locate a motion vector of a block corresponding to the current video block, wherein MVx′, MVy′ are scaled motion vectors of the current video block, wherein dimensions of the co-located picture or dimensions of the window associated with the co-located picture are referred to as W1, H1, and wherein dimensions of the current picture or dimensions of the window associated with the current picture are referred to as W2, H2, respectively.

[1236] Some exemplary embodiments of the technology described in the items of Section 4 23 include:

[1237] B1. A method of visual media processing (e.g., Fig.34 3400), comprising:

[1238] determining (3402) for a current video block a motion vector used in a sub-block based temporal motion vector prediction (sbTMVP) process to locate a corresponding block in a co-located picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated with respect to a center point of the current video block;

[1239] modifying (3406) the center point of the current video block by applying one or more operations; and

[1240] Based on using the center point modified by applying the one or more operations, converting (3408) between the current video block and a bitstream representation of the visual media data.

[1241] B2. A method according to clause B1, wherein the one or more operations include a scaling operation and / or an operation corresponding to adding an offset to the center point.

[1242] B3. The method according to clause B1, further comprising:

[1243] Upon determining that the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are different than the dimensions of the current picture associated with the current block or the dimensions of the window associated with the current picture, an additional modification of the center point of the current video block is performed.

[1244] B4. A method according to clause B3, wherein the additional modification of the center point of the current video block is expressed as:

[1245] x0′=(x0-X2)*W1 / W2+X1 and y0′=(y0-Y2)*H1 / H2+Y1,

[1246] wherein the center point of the current video block is (x0, y0), and the coordinates of the center point of the current video block after the additional modification are (x0′, y0′), wherein the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are referred to as W1, H1, and wherein the dimensions of the current picture or the dimensions of the window associated with the current picture are referred to as W2, H2, respectively, the top left position of the window associated with the current picture is referred to as (X2, Y2), and the top left position of the window associated with the co-located picture is referred to as (X1, Y1).

[1247] B5. A method according to clause B3, wherein the additional modification of the center point of the current video block is expressed as:

[1248] x0′=(x0-X2)*W1 / W2 and y0′=(y0-Y2)*H1 / H2,

[1249] wherein the center point of the current video block is (x0, y0), and the coordinates of the center point of the current video block after the additional modification are (x0′, y0′), wherein the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are referred to as W1, H1, the left top position of the window associated with the current picture is referred to as (X2, Y2), and wherein the dimensions of the current picture or the dimensions of the window associated with the current picture are referred to as W2, H2, respectively.

[1250] Some exemplary embodiments 24 of the technology described in the items of Section 4 include:

[1251] C1. A method of visual media processing (e.g., Fig.35 3500), comprising:

[1252] determining (3502) for a current video block a motion vector used in a sub-block based temporal motion vector prediction (sbTMVP) process to locate a corresponding block in a collocated picture for conversion between the current video block and a bitstream representation of the current video block, wherein the motion vector used in the sbTMVP process is calculated with respect to a point in the corresponding block in the collocated picture;

[1253] modifying (3504) the point in the corresponding block in the co-located picture by applying one or more operations; and

[1254] Converting between the current video block and a bitstream representation of the visual media data is performed (3506) based on using the point in the corresponding block in the co-located picture modified by applying the one or more operations.

[1255] C2. The method of clause C1, wherein the one or more operations include a scaling operation and / or an operation corresponding to adding an offset to the point in the corresponding block in the co-located picture.

[1256] C3. The method according to clause C1, further comprising:

[1257] Upon determining that the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are different from the dimensions of the current picture associated with the current video block or the dimensions of the window associated with the current picture, additional modification of the points in the corresponding block in the co-located picture is performed.

[1258] C4. A method according to clause C3, wherein the additional modification of the point in the corresponding block in the co-located picture is expressed as:

[1259] x′=(x-X2)*W1 / W2+X1 and y′=(y-Y2)*H1 / H2+Y1,

[1260] wherein the point in the corresponding block in the co-located picture is (x, y), and the coordinates of the point in the corresponding block in the co-located picture after the additional modification are (x′, y′), wherein the width and height of the window associated with the current picture are referred to as W2 and H2, respectively, the width and height of the window associated with the co-located picture are referred to as W1 and H1, respectively, the left top position of the window associated with the current picture is referred to as (X2, Y2), and the left top position of the window associated with the co-located picture is referred to as (X1, Y1).

[1261] C5. A method according to clause C3, wherein the additional modification of the point in the corresponding block in the co-located picture is expressed as:

[1262] x′=(x-X2)*W1 / W2 and y′=(y-Y2)*H1 / H2, wherein the point in the corresponding block in the co-located picture is (x, y), and the coordinates of the point in the corresponding block in the co-located picture after the additional modification are (x′, y′), wherein the width and height of the window associated with the current picture are referred to as W2 and H2, respectively, the width and height of the window associated with the co-located picture are referred to as W1 and H1, respectively, and the top left position of the window associated with the current picture is referred to as (X2, Y2).

[1263] Some exemplary embodiments 28 of the technology described in the items of Section 4 include:

[1264] D1. A method of visual media processing (e.g., Fig.36The method 3600 shown in FIG. 3 comprises: for converting between a video picture contained in visual media data and a bitstream representation of the visual media data using sub-pictures, determining (3602) that a rule related to one or more sub-pictures is satisfied by the conversion; and performing (3604) the conversion according to the constraint of the rule, wherein the rule specifies that the size of the sub-picture in the video picture is an integer multiple of the size of a codec tree unit associated with the video picture.

[1265] D2. The method of clause D1, wherein the rule specifies that the bitstream representation includes a variable corresponding to a width of the sub-picture.

[1266] D3. A method according to clause D1, wherein the rule specifies that the bitstream representation includes a variable corresponding to the height of the sub-picture.

[1267] D4. A method according to any one or more of clauses D2-D3, wherein the variable corresponding to the height of the sub-picture and / or the variable corresponding to the width of the sub-picture is expressed in units of luma codec tree unit size.

[1268] D5. A method according to any one or more of clauses D2 and D4, wherein the variable corresponding to the width of the sub-picture is equal to the difference between the width of the sub-picture in units of luma codec treeblock size and 1.

[1269] D6. A method according to any one or more of clauses D3 and D4, wherein the variable corresponding to the height of the sub-picture is equal to the difference between the height of the sub-picture in units of luma codec tree unit size and 1.

[1270] D7. A method as described in clause D4, wherein the luma codec tree unit size is defined as the width or height of the luma codec treeblock array.

[1271] D8. A method according to clause D4, wherein the luma codec tree unit size of a codec tree unit is defined as a variable CtbSizeY, expressed as:

[1272] CtbSizeY=(1<<(sps_log2_ctu_size_minus5+5)), where sps_log2_ctu_size_minus5 refers to the size of the syntax element.

[1273] D9. A method according to clause D8, wherein the variable corresponding to the width of the sub-picture is calculated as the width of the sub-picture multiplied by the width of CtbSizeY.

[1274] D10. A method as described in clause D8, wherein the variable corresponding to the height of the sub-picture is calculated as the sub-picture width multiplied by the height of CtbSizeY.

[1275] D11. The method of clause D1, wherein the rule specifies a value of a flag that indicates use of a sub-picture during the conversion.

[1276] D12. A method according to clause D11, wherein the flag is a Boolean value.

[1277] D13. A method according to clause D12, wherein when the flag takes the value 1, it indicates that sub-pictures are used, and when the flag takes the value 0, it indicates that sub-pictures are not used.

[1278] Some exemplary embodiments of the technology described in the items of Section 4 29 include:

[1279] E1. A method of video processing (eg, Fig.37 The method 3700 shown in FIG. 3 comprises: for a conversion between a video picture contained in visual media data and a bitstream representation of the visual media data using sub-pictures, determining (3702) that a rule related to one or more sub-pictures is satisfied by the conversion; and performing (3704) the conversion according to the rule constrained by the constraint, wherein the rule specifies that all sub-pictures in the video picture are non-overlapping and that all sub-pictures in the video picture together cover the entire video picture.

[1280] Some exemplary embodiments 30 of the technology described in the items of Section 4 include:

[1281] F1. A method for visual media processing (e.g., Fig.38 The method 3800 shown in ) comprises: for converting between a video unit of a visual media and a bitstream representation of the visual media data, determining (3802) whether a reference picture resampling (RPR) technique is used during the conversion; and performing (3804) the conversion based on the determination, wherein a flag corresponding to the determination is included in the bitstream representation at a sequence parameter set level.

[1282] F2. A method according to clause F1, wherein the flag indicates the use of the RPR technique, and wherein the flag takes a Boolean value.

[1283] F3. A method according to clause F2, wherein the flag equal to 1 specifies that the spatial resolution of the coded picture with respect to the sequence parameter set is inferred to be changeable, and the flag equal to 0 specifies that the spatial resolution of the coded picture with respect to the sequence parameter set is inferred to be unchanged.

[1284] F4. A method according to clause F2, wherein the flag equal to 1 specifies that the spatial resolution of the picture is changeable within the codec layer video sequence that references the sequence parameter set, and the flag equal to 0 specifies that the spatial resolution of the picture is not changed within the codec layer video sequence that references the sequence parameter set.

[1285] F5. A method according to any one or more of clauses F1-F4, wherein whether window information of a picture associated with the video unit is included in the bitstream representation is independent of whether the RPR technique is used.

[1286] F6. A method according to any one or more of clauses F1-F4, wherein the height and / or width of the video unit indicated in the picture parameter set (PPS) is inferred to be the same as the height and / or width of the video unit in another picture parameter set (PPS) associated with the same sequence parameter set.

[1287] F7. A method according to any one or more of clauses F1-F6, wherein the bitstream representation is configured to contain window information for a picture associated with the video unit regardless of the value of the flag.

[1288] F8. A method according to any one or more of clauses F1-F6, wherein the bitstream representation is configured to include window information for a picture associated with the video unit, and wherein the flag indicates whether the window information is used during the conversion.

[1289] F9. A method according to any one or more of clauses F1-F4, wherein the height and / or width of the video unit indicated in the picture parameter set (PPS) is inferred to be the same as the height and / or width of the video unit in another picture parameter set (PPS) associated with the same sequence parameter set.

[1290] F10. A method according to any one or more of clauses F-F4, further comprising:

[1291] Upon determining not to use the RPR technique during the conversion, the height and / or width of the video unit indicated in the picture parameter set (PPS) is inferred to be a default value for the height and / or a default value for the width of the video unit.

[1292] F11. A method according to clause F10, wherein a default value for the height and / or a default value for the width of the video unit is inferred to be a maximum height and / or a maximum width of the video unit indicated in a sequence parameter set (SPS) associated with the video unit.

[1293] Some exemplary embodiments 31 of the technology described in the items of Section 4 include:

[1294] G1. A method of visual media processing (e.g., Fig.39 ), comprising: selecting (3902) an interpolation filter during a motion compensation process to derive a prediction block for a current block of visual media data based on satisfying a condition, wherein the condition is at least partially based on determining that a resolution of a reference picture is different from a resolution of the current picture and / or a dimension of a window associated with the reference picture is different from a dimension of a window associated with the current picture; and performing (3904) a conversion between the current block of visual media data and a bitstream representation of the current block.

[1295] G2. The method according to clause G1, further comprising:

[1296] Upon determining that the resolution of the reference picture is different from the resolution of the current picture or the dimension of the window associated with the reference picture is different from the dimension of the window associated with the current picture, an interpolation filter different from another filter used when the reference picture has the same resolution as the current picture is used.

[1297] G3. A method according to clause G1, wherein the condition is based on determining that W1>a*W2 and / or H1>b*H2, wherein (W1, H1) represents the width and height of the reference image or a window associated with the reference image, and (W2, H2) represents the width and height of the current image or a window associated with the reference image, and a and b are scaling factors.

[1298] G4. A method according to clause G3, wherein a=b=1.5.

[1299] G5. A method according to clause G1, wherein the condition is also based on the height and / or width of the current block.

[1300] G6. A method according to clause G5, wherein the height and / or width of the current block individually implement one or more threshold conditions.

[1301] G7. A method according to clause G5, wherein a mathematical combination of the height and / or width of the current block implements one or more threshold conditions.

[1302] G8. A method according to clause G6, wherein achieving the one or more threshold conditions includes the height and / or width of the current block exceeding the one or more threshold conditions.

[1303] G9. A method according to clause G6, wherein achieving the one or more threshold conditions includes the height and / or width of the current block missing the one or more threshold conditions.

[1304] G10. A method according to clause G7, wherein achieving the one or more threshold conditions includes a mathematical combination of the height and / or width of the current block exceeding the one or more threshold conditions.

[1305] G11. A method according to clause G7, wherein achieving the one or more threshold conditions includes a mathematical combination of the height and / or width of the current block missing the one or more threshold conditions.

[1306] G12. A method according to any one or more of clauses G1-G11, wherein the selected interpolation filter is applied exclusively to the chroma color components of the current video block.

[1307] G13. A method according to any one or more of clauses G1-G11, wherein the selected interpolation filter is applied exclusively to the luma color component of the current video block.

[1308] G14. A method according to any one or more of clauses G1-G11, wherein the selected interpolation filtering is applied exclusively in the horizontal filtering direction.

[1309] G15. A method according to any one or more of clauses G1-G11, wherein the selected interpolation filter is applied exclusively in the vertical filtering direction.

[1310] G16. A method according to any one or more of clauses G1-G11, wherein a single-tap filter is selected as the interpolation filter so that the result of applying the interpolation filter is equivalent to replicating integer sample points.

[1311] G17. A method according to any one or more of clauses G1-G11, wherein a 4-tap interpolation filter or a 6-tap interpolation filter is selected when the condition is met.

[1312] G18. A method according to any one or more of clauses G1-G11, wherein bilinear filtering is selected as the interpolation filter when the condition is met.

[1313] Some exemplary embodiments 32 of the technology described in the items of Section 4 include:

[1314] H1. A method for visual media processing (e.g. Fig.40The method 4000 shown in ) comprises: converting between a current block of visual media data and a bitstream representation of the visual media data, determining (4002) that the current block is a combined inter-frame intra-frame prediction (CIIP) block, wherein an intra-frame prediction block of the CIIP block is generated using the size of a transform unit (TU), wherein, in the combined inter-frame intra-frame prediction (CIIP) block, a final prediction of the current block is based on a weighted sum of an inter-frame prediction of the current block and an intra-frame prediction of the current block; and performing (4004) the conversion based on the determination.

[1315] H2. The method according to clause H1, further comprising:

[1316] Upon determining that the dimension of the codec unit associated with the current block exceeds the maximum size of a transform unit, splitting the codec unit into a plurality of transform units; and

[1317] An intra prediction block and an inter prediction block are generated for each of the transformation units included in the plurality of transformation units.

[1318] H3. The method according to clause H2, further comprising:

[1319] Upon determining that the maximum size of a transform unit is less than a threshold and the dimension of the codec unit associated with the current block exceeds the maximum size of the transform unit, recursively splitting the codec unit into a plurality of transform units; and

[1320] An intra prediction block and an inter prediction block are generated for each of the transformation units included in the plurality of transformation units.

[1321] H4. A method according to clause H3, wherein the first intra-frame prediction block is based on reconstructed samples of the second intra-frame prediction block, wherein the first intra-frame prediction block and the second intra-frame prediction block are generated based on dividing the current block.

[1322] I1. A method according to any one or more of clauses A1 to H4, wherein the converting comprises generating the bitstream representation from the current video block.

[1323] I2. A method according to any one or more of clauses A1 to H4, wherein the converting comprises generating samples of the current video block from the bitstream representation.

[1324] I3. A video encoder device comprising a processor configured to implement the method according to any one or more of clauses A1 to H4.

[1325] I4. A video decoder device comprising a processor configured to implement the method according to any one or more of clauses A1 to H4.

[1326] I5. A computer-readable medium having code stored thereon which, when executed by a processor, causes the processor to implement the method according to any one or more of clauses A1 to H4.

[1327] I6. A computer-readable storage medium storing a bitstream representation generated according to any one or more of the methods described in clauses A1 to H4.

[1328] Fig.41 is a block diagram illustrating an exemplary video coding system 100, which may utilize the techniques of this disclosure.

[1329] like Fig.41 As shown, the video coding system 100 may include a source device 110 and a destination device 120. The source device 110 generates encoded video data, which may be referred to as a video encoding device. The destination device 120 may decode the encoded video data generated by the source device 110, which may be referred to as a video decoding device.

[1330] Source device 110 may include a video source 112 , a video encoder 114 , and an input / output (I / O) interface 116 .

[1331] The video source 112 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of such sources. The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bit stream. The bit stream may include a sequence of bits that form a codec representation of data of the video. The bit stream may include a codec picture and associated data. The codec picture is a codec representation of the picture. The associated data may include a sequence parameter set, a picture parameter set, and other syntax structures. The I / O interface 116 may include a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be directly transmitted to the destination device 120 via the network 130a via the I / O interface 116. The encoded video data may also be stored on a storage medium / server 130b to be accessed by the destination device 120.

[1332] Destination device 120 may include an I / O interface 126 , a video decoder 124 , and a display device 122 .

[1333] I / O interface 126 may include a receiver and / or a modem. I / O interface 126 may obtain encoded video data from source device 110 or storage medium / server 130b. Video decoder 124 may decode the encoded video data. Display device 122 may display the decoded video data to a user. Display device 122 may be integrated with destination device 120, or may be external to destination device 120, which is configured to interface with an external display device.

[1334] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other current and / or other standards.

[1335] Fig.42 is a block diagram illustrating an example of a video encoder 200, which may be Fig.41 1. The video encoder 114 in the system 100 is shown in FIG.

[1336] Video encoder 200 may be configured to perform any or all of the techniques of this disclosure. Fig.42 In the example of , the video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared among various components of the video encoder 200. In some embodiments, the processor can be configured to perform any or all of the techniques described in this disclosure.

[1337] The functional components of the video encoder 200 may include a segmentation unit 201, a prediction unit 202 which may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214.

[1338] In other examples, the video encoder 200 may include more, fewer, or different functional components. In an example, the prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[1339] In addition, some components such as the motion estimation unit 204 and the motion compensation unit 205 may be highly integrated, but in Figure 5 In the examples, they are shown separately for explanation purposes.

[1340] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[1341] The mode selection unit 203 may select one of the coding modes (intra or inter), for example, based on the error result, and provide the resulting intra or inter coded block to the residual generation unit 207 to generate residual block data and to the reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 203 may select a combined intra and inter prediction (CIIP) mode, where the prediction is based on inter prediction signaling and intra prediction signaling. The mode selection unit 203 may also select a resolution (e.g., sub-pixel or integer pixel precision) of the motion vector of the block in the case of inter prediction.

[1342] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.

[1343] Motion estimation unit 204 and motion compensation unit 205 may operate differently on the current video block, eg, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[1344] In some embodiments, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search for a reference video block of the current video block in the reference pictures of list 0 or list 1. The motion estimation unit 204 may then generate a reference index indicating a reference picture in list 0 or list 1 containing the reference video block and a motion vector indicating a spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.

[1345] In other examples, the motion estimation unit 204 may perform bidirectional prediction of the current video block, and the motion estimation unit 204 may search for a reference video block of the current video block in the reference pictures in list 0, and may also search for another reference video block of the current video block in the reference pictures in list 1. The motion estimation unit 204 may then generate a reference index indicating the reference pictures in list 0 and list 1 containing the reference video block and a motion vector indicating a spatial displacement between the reference video block and the current video block. The motion estimation unit 204 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[1346] In some embodiments, motion estimation unit 204 may output a complete set of motion information to the decoding process of the decoder.

[1347] In some embodiments, motion estimation unit 204 may not output a complete set of motion information for the current video. Instead, motion estimation unit 204 may signal the motion information of the current video block with reference to the motion information of another video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of the adjacent video block.

[1348] In one example, motion estimation unit 204 may indicate a value in a syntax structure associated with the current video block that indicates to video decoder 300 that the current video block has the same motion information as another video block.

[1349] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[1350] As discussed above, the video encoder 200 may predictively signal motion vectors.Two examples of predictive signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and Merge mode signaling.

[1351] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.

[1352] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample components of samples in the current video block.

[1353] In other examples, there may be no residual data for the current video block (eg, in skip mode), and the residual generation unit 207 may not perform a subtraction operation.

[1354] Transform processing unit 208 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[1355] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[1356] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current block for storage in the buffer 213.

[1357] After the reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[1358] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream containing the entropy-coded data.

[1359] Fig.43 is a block diagram illustrating an example of a video decoder 300, which may be Fig.41 1. The video decoder 124 in the system 100 is shown in FIG.

[1360] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Fig.43 In the example of FIG. 3 , the video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared among various components of the video decoder 300. In some embodiments, the processor can be configured to perform any or all of the techniques described in this disclosure.

[1361] exist Fig.43 In the example of FIG. 3 , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra-frame prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306, as well as a buffer 307. In some embodiments, the video decoder 300 may perform a decoding pass that is generally inverse to the process of the video encoder 200 (e.g., Fig.42 ) is passed through the encoding described by

[1362] The entropy decoding unit 301 may retrieve a coded bitstream. The coded bitstream may include entropy coded video data (e.g., coded blocks of video data). The entropy decoding unit 301 may decode the entropy coded video data, and the motion compensation unit 302 may determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information from the entropy decoded video data. The motion compensation unit 302 may determine such information, for example, by performing AMVP and Merge modes.

[1363] The motion compensation unit 302 may generate a motion compensated block, possibly interpolated based on an interpolation filter. An identifier of an interpolation filter to be used with sub-pixel precision may be included in a syntax element.

[1364] Motion compensation unit 302 may use interpolation filtering as used by video encoder 20 during encoding of a video block to calculate interpolated values ​​for sub-integer pixels of a reference block. Motion compensation unit 302 may determine the interpolation filtering used by video encoder 200 based on received syntax information and use the interpolation filtering to produce a predictive block.

[1365] The motion compensation unit 302 may use some syntax information to determine the size of blocks used to encode (multiple) frames and / or (multiple) slices of the coded video sequence, partitioning information describing how each macroblock of a picture of the coded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame coded block, and other information for decoding the coded video sequence.

[1366] The intra prediction unit 303 may use, for example, an intra prediction mode received in the bitstream to form a prediction block from spatially adjacent blocks. The inverse quantization unit 304 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.

[1367] The reconstruction unit 306 may add the residual block to the corresponding prediction block generated by the motion compensation unit 205 or the intra prediction unit 303 to form a decoded block. If desired, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides a reference block for subsequent motion compensation.

[1368] In this document, the term "video processing" or "visual media processing" may refer to video encoding, video decoding, video compression or video decompression. For example, a video compression algorithm may be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. As defined by the syntax, the bitstream representation of the current video block may correspond, for example, to bits that are co-located within the bitstream or distributed at different locations in the bitstream. For example, a macroblock may be encoded based on error residual values ​​of a transform and codec and also using bits in the header and other fields in the bitstream. In addition, as described in the above solution, during the conversion, the decoder may parse the bitstream based on a determination with the knowledge that some fields may or may not be present. Similarly, the encoder may determine that certain syntax fields will be included or not included, and generate the codec representation accordingly by including or excluding syntax fields from the codec representation.

[1369] The disclosed and other solutions, examples, embodiments, modules and functional operations described herein may be implemented in digital electronic circuits or computer software, firmware or hardware, including the structures disclosed herein and their structural equivalents, or a combination of one or more thereof. The disclosed embodiments and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium to be executed by a data processing device or to control the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that implements a machine-readable propagation signal, or a combination of one or more thereof. The term "data processing device" encompasses all devices, devices and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[1370] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language (including compiled or interpreted languages) and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[1371] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[1372] For example, processors suitable for executing computer programs include general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include or be operatively coupled to receive data from or transfer data to one or more mass storage devices or both for storing data (e.g., magnetic disks, magneto-optical disks, or optical disks). However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit.

[1373] Although the patent document includes many details, these details should not be interpreted as limitations on any subject matter or the scope of the claimed protection, but rather as descriptions of features of specific embodiments of specific technologies. Certain features described in the patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination, respectively. Moreover, although the features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features in the claimed combination may be excluded from the combination, and the claimed combination may be used in a sub-combination or a variation of the sub-combination.

[1374] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[1375] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method for visual media processing, comprising: Determining, for a current video block, a motion vector used in a sub-block based temporal motion vector prediction (sbTMVP) process to locate a corresponding block in a co-located picture for conversion between the current video block and a bitstream of the current video block, wherein the motion vector used in the sbTMVP process is calculated according to a scaling operation; Based on using the motion vector, converting between the current video block and a bitstream of visual media data; as well as Upon determining that the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are different from the dimensions of the current picture associated with the current block or the dimensions of the window associated with the current picture, a scaling operation is applied to the motion vector used in the sbTMVP process.

2. The method according to claim 1, wherein: The scaling operation includes at least one of: a right shift operation, a left shift operation and / or a right shift rounding operation applied to the motion vector used in the sbTMVP process.

3. The method according to claim 2, wherein: The right shift operation causes the motion vector to be rounded to an integer value by rounding towards zero.

4. The method of claim 1, wherein the result of the scaling operation on the motion vector used in the sbTMVP process is expressed as: MVx' = MVx*W1 / W2 and MVy' = MVy*H1 / H2, wherein MVx, MVy are motion vectors locating a block corresponding to the current video block, wherein MVx', MVy' are scaled motion vectors of the current video block, wherein the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are referred to as W1, H1, and wherein the dimensions of the current picture or the dimensions of the window associated with the current picture are referred to as W2, H2, respectively.

5. The method according to claim 1, wherein: Calculating the motion vector used in the sbTMVP process for the center point of the current video block; the method further comprises: modifying a center point of the current video block by applying one or more operations; and Converting between the current video block and a bitstream of visual media data is performed based on using the center point modified by applying the one or more operations.

6. The method according to claim 5, wherein: The one or more operations include the scaling operation and / or an operation corresponding to adding an offset to the center point.

7. The method according to claim 5, further comprising: Upon determining that the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are different from the dimensions of the current picture associated with the current block or the dimensions of the window associated with the current picture, an additional modification of the center point of the current video block is performed.

8. The method according to claim 7, wherein: The additional modification of the center point of the current video block is expressed as: x0'=(x0-X2)*W1 / W2+X1 and y0'=(y0-Y2)*H1 / H2+Y1, wherein the center point of the current video block is (x0, y0), and the coordinates of the center point of the current video block after the additional modification are (x0', y0'), wherein the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are referred to as W1, H1, and wherein the dimensions of the current picture or the dimensions of the window associated with the current picture are referred to as W2, H2, respectively, the top left position of the window associated with the current picture is referred to as (X2, Y2), and the top left position of the window associated with the co-located picture is referred to as (X1, Y1).

9. The method according to claim 7, wherein: The additional modification of the center point of the current video block is expressed as: x0' = (x0-X2)*W1 / W2 and y0' = (y0-Y2)*H1 / H2, wherein the center point of the current video block is (x0, y0), and the coordinates of the center point of the current video block after the additional modification are (x0', y0'), wherein the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are referred to as W1, H1, the left top position of the window associated with the current picture is referred to as (X2, Y2), and wherein the dimensions of the current picture or the dimensions of the window associated with the current picture are referred to as W2, H2, respectively.

10. The method according to claim 1, wherein: Calculating the motion vector used in the sbTMVP process for a point in the corresponding block in the co-located picture; the method further comprises: modifying the point in the corresponding block in the co-located picture by applying one or more operations; and Converting between the current video block and a bitstream of visual media data is performed based on using the points in the corresponding block in the co-located picture modified by applying the one or more operations.

11. The method according to claim 10, wherein: The one or more operations include the scaling operation and / or an operation corresponding to adding an offset to the point in the corresponding block in the co-located picture.

12. The method according to claim 10, further comprising: Upon determining that the dimensions of the co-located picture or the dimensions of the window associated with the co-located picture are different from the dimensions of the current picture associated with the current video block or the dimensions of the window associated with the current picture, additional modification of the points in the corresponding block in the co-located picture is performed.

13. The method according to claim 12, wherein: The additional modification of the point in the corresponding block in the co-located picture is expressed as: x'=(x-X2)*W1 / W2+X1 and y'=(y-Y2)*H1 / H2+Y1, wherein the point in the corresponding block in the co-located picture is (x, y), and the coordinates of the point in the corresponding block in the co-located picture after the additional modification are (x', y'), wherein the width and height of the window associated with the current picture are referred to as W2 and H2, respectively, the width and height of the window associated with the co-located picture are referred to as W1 and H1, respectively, the top left position of the window associated with the current picture is referred to as (X2, Y2), and the top left position of the window associated with the co-located picture is referred to as (X1, Y1).

14. The method according to claim 12, wherein: The additional modification of the point in the corresponding block in the co-located picture is expressed as: x'=(x-X2)*W1 / W2 and y'=(y-Y2)*H1 / H2, wherein the point in the corresponding block in the co-located picture is (x, y), and the coordinates of the point in the corresponding block in the co-located picture after the additional modification are (x', y'), wherein the width and height of the window associated with the current picture are referred to as W2 and H2, respectively, the width and height of the window associated with the co-located picture are referred to as W1 and H1, respectively, and the top left position of the window associated with the current picture is referred to as (X2, Y2).

15. The method according to claim 1, wherein: The converting includes generating the bitstream from the current video block.

16. The method according to claim 1, wherein: The converting includes generating samples of the current video block from the bitstream.

17. A video encoder device comprising a processor configured to implement the method according to any one of claims 1 to 16.

18. A video decoder device comprising a processor configured to implement the method according to any one of claims 1 to 16.

19. A computer readable medium having instructions stored thereon which, when executed by a processor, cause the processor to implement the method according to any one of claims 1 to 16.

Images