METHOD, APPARATUS AND NON-TRANSITORY COMPUTER-READABLE STORAGE MEDIUM FOR PROCESSING VIDEO DATA

Abstract

To provide a method related to a sub-block base prediction for a picture and a video coding.SOLUTION: A method for coding a video data, contains a step of determining a motion vector of a control point of a current video block on the basis of an affine mode. The determination contains steps of: dividing a luma component into a luma sub-block of the luma component and at least one chroma component contained in the current video block; dividing the chroma component of at least one chroma component into chroma sub-blocks; and determining a luma motion vector of each luma sub-block on the basis of the motion vector of the control point. The method is also steps of: introducing the chroma operation vector of each chroma sub-block on the basis of a luma motion vector of a plurality of corresponded luma sub-blocks and a color format of the current video block; and reconstructing the luma component on the basis of the luma motion vector of each luma sub-block.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] Pursuant to applicable patent law and / or regulations under the Paris Convention, this application timely claims priority to and the benefit of International Patent Application No. PCT / CN2018 / 092118, filed June 21, 2018, and International Patent Application No. PCT / CN2018 / 114931, filed November 10, 2018. For all purposes under U.S. law, the entire disclosures of International Patent Application No. PCT / CN2018 / 092118 and International Patent Application No. PCT / CN2018 / 114931 are incorporated by reference as part of the disclosure of this application.

[0002] This patent document is generally directed to picture and video coding techniques. [Background technology]

[0003] Motion compensation is a technique in video processing that predicts frames in a video given previous and / or future frames by accounting for the movement of the camera and / or objects in the video. Motion compensation can be used in encoding and decoding video data for video compression. Summary of the Invention

[0004] DETAILED DESCRIPTION Apparatuses, systems and methods relating to sub-block-based prediction for picture and video coding are described.

[0005] In one exemplary aspect, the disclosed techniques are used to provide a video encoding method, the method including partitioning a first component of a current video block into a first set of sub-blocks and partitioning a second component of the current video block into a second set of sub-blocks, the sub-blocks of the second component corresponding to one or more sub-blocks of the first component. The method also includes deriving motion vectors for the sub-blocks of the second component based on the motion vectors of one or more corresponding sub-blocks of the first color component based on a color format of the current video block.

[0006] In another representative aspect, the disclosed techniques are used to provide a video decoding method. The method includes receiving a video data block including a first component and at least a second component. The first component is divided into a first set of sub-blocks and the second component is divided into a second set of sub-blocks. The sub-blocks of the second component correspond to one or more sub-blocks of the first component. The method includes deriving motion vectors for sub-blocks of the second component based on a color format of the video data block based on the motion vectors of one or more corresponding sub-blocks of the first color component. The method also includes reconstructing the video data block or decoding other blocks of the video data in the same picture based on the derived motion vectors.

[0007] In yet another representative aspect, the above-described methods are embodied in the form of processor executable code and stored on a computer readable program medium.

[0008] In yet another representative aspect, a device configured or operable to perform the above-described method is disclosed, which may include a processor programmed to implement the method.

[0009] In yet another exemplary aspect, a video encoder or video decoder device may implement the methods described herein.

[0010] These and other aspects and features of the disclosed technology are described in more detail in the drawings, specification and claims. [Brief description of the drawings]

[0011] [Figure 1] 1 illustrates an example of sub-block based prediction.

[0012] [Diagram 2] 1 shows an example of a simplified affine motion model.

[0013] [Diagram 3] 1 shows an example of an affine motion vector field (MVF) for each subblock.

[0014] [Figure 4] 1 illustrates an example of motion prediction using an alternative temporal motion vector prediction (ATMVP) algorithm for a coding unit.

[0015] [Diagram 5] 1 shows an example of one CU with four sub-blocks and adjacent blocks.

[0016] [Figure 6] 1 shows an example of an optical flow trajectory used by the Bidirectional Optical Flow (BIO) algorithm.

[0017] [Figure 7] 1 shows an example of bilateral matching in a frame rate up-conversion (FRUC) algorithm.

[0018] [Figure 8] An example of template matching in the FRUC algorithm is shown below.

[0019] [Figure 9] An example of unilateral motion estimation (ME) in the FRUC method is shown.

[0020] [Figure 10] An example of deriving motion vectors (MVs) for components of sub-blocks in the 4:2:0 format according to the present technology is shown.

[0021] [Figure 11] 4 shows another example of deriving MVs of sub-block components in 4:2:0 format according to the present technique.

[0022] [Figure 12] 4 shows yet another example of deriving MVs of components of a sub-block according to the present technique.

[0023] [Figure 13] 1 is a flowchart illustrating an example of a video encoding method in accordance with one or more embodiments of the present technology.

[0024] [Figure 14] 10 is a flowchart illustrating another example of a video encoding method in accordance with one or more embodiments of the present technology.

[0025] [Figure 15] FIG. 2 is a block diagram illustrating an example of an encoding device that can be used to implement various portions of the techniques of this disclosure.

[0026] [Figure 16] FIG. 2 is a block diagram illustrating an example of an encoding device that can be used to implement various portions of the techniques of this disclosure.

[0027] [Figure 17] FIG. 2 is a block diagram illustrating an example architecture of a computer system or other control device that can be used to implement various portions of the technology of the present disclosure.

[0028] [Figure 18]FIG. 1 shows a block diagram illustrating an example embodiment of a mobile device that can be used to implement various portions of the techniques of this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Due to the increasing demand for higher resolution video, video coding methods and techniques are ubiquitous in modern technology. Video codecs typically include electronic circuits or software that compress or decompress digital video and are constantly being improved to provide higher coding efficiency. Video codecs convert uncompressed video into compressed form or vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end delay (latency). Compression formats usually comply with standard video compression specifications, such as the High Efficiency Video Coding (HEVC) standard (also known as H.265 or MPEG-H Part 2), the Versatile Video Coding standard to be finalized, or other current and / or future video coding standards.

[0030] Sub-block-based prediction was first introduced into video coding standards by the High Efficiency Video Coding (HEVC) standard. In sub-block-based prediction, a block such as a coding unit (CU) or a prediction unit (PU) is divided into several non-overlapping sub-blocks. Different sub-blocks are assigned different motion information such as reference indexes or motion vectors (MVs), and motion compensation (MC) is performed for each sub-block separately. Figure 1 shows an example of sub-block-based prediction.

[0031] Embodiments of the disclosed technology may be applied to existing video coding standards (e.g., HEVC, H.265) and future standards to improve runtime performance. Section headings are used in this document to improve the readability of the description and are not intended in any way to limit the description or embodiments (and / or implementations) to only the respective sections.

[0032] A reference software known as the Joint Exploration Model (JEM) is used to illustrate future video coding techniques. JEM employs subblock-based prediction for several coding tools, including affine prediction, alternative temporal motion vector prediction (ATMVP), spatio-temporal motion vector prediction (STMVP), bidirectional optical flow (BIO), frame rate up-conversion (FRUC), locally adaptive motion vector resolution (LAMVR), overlapping block motion compensation (OBMC), local illumination compensation (LIC), and decoder-side motion vector refinement (DMVR).

[0033] In HEVC, only the translational motion model is applied to the motion compensated prediction (MCP). However, cameras and objects have and acquire many kinds of motions, such as zoom in / zoom out, rotation, oblique motion, and / or other irregular motions. On the other hand, JEM applies a simplified affine transformation motion compensated prediction. Figure 2 shows an example of an affine motion field of a block 200 described by two control point motion vectors V0 and V1. The motion vector field (MVF) of the block 200 can be expressed by the following equation:

[0034]

number

[0035] FIG. 3 shows an example of affine MVF per subblock for block 300. To derive the motion vector for each M×N subblock, the motion vector of the center sample of each subblock can be calculated according to equation (1) and rounded to the motion vector fraction precision (e.g., 1 / 16 for JEM). A motion compensated interpolation filter can then be applied to generate a prediction for each subblock with the derived motion vector. After MCP, the high-precision motion vector for each subblock is rounded and stored with the same precision as the normal motion vector.

[0036] JEM has two affine motion modes: AF_INTER and AF_MERGE. For CUs whose width and height are both greater than 8, AF_INTER mode is applicable. A CU-level affine flag is signaled in the bitstream to indicate whether AF_INTER mode is used. In AF_INTER mode, the motion vector pair {(v0,v1)|v0={v A ,v B ,v c},v1={v D ,v E A candidate list with}} is built using adjacent blocks.

[0037] In JEM, the non-merge affine mode can only be used if both the width and height of the current block are greater than 8, and the merge affine mode can only be used if the area (i.e. width x height) of the current block is not less than 64.

[0038] 4 shows an example of an alternative temporal motion vector prediction (ATMVP) motion prediction process of a CU 400. The ATMVP method predicts the motion vectors of sub-CUs 401 in a CU 400 in two steps. The first step is to identify a corresponding block 451 in a reference picture 450 that has a temporal vector. The reference picture 450 is also called a motion source picture. The second step is to split the current CU 400 into sub-CUs 401 and obtain the motion vector and reference index of each sub-CU from the block corresponding to each sub-CU.

[0039] In the first step, the reference picture 450 and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU 400. In the second step, the corresponding block of the sub-CU 451 is identified by a time vector in the motion source picture 450 by adding the time vector to the coordinates of the current CU. For each sub-CU, the motion information of the corresponding block (e.g., the smallest motion grid that covers the center sample) is used to derive the motion information of the sub-CU. After the motion information of the corresponding N×N block is identified, it is converted into the motion vector and reference index of the current sub-CU, similar to TMVP in HEVC. Here, motion scaling and other procedures are applied.

[0040] In the spatial-temporal motion vector prediction (STMVP) method, the motion vectors of the sub-CUs are derived recursively according to the raster scan order. Figure 5 shows an example of one CU with four sub-blocks and neighboring blocks. Consider an 8x8 CU 500 that contains four sub-CUs: 4x4 sub-CUs A (501), B (502), C (503), and D (504). The neighboring 4x4 blocks in the current frame are labeled a (511), b (512), c (513), and d (514).

[0041] The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is an N×N block (block c 513) above sub-CU A 501. If this block c (513) is not available or is intra-coded, other N×N blocks above sub-CU A (501) are checked (starting from block c 513 and going from left to right). The second neighbor is a block (block b 512) to the left of sub-CU A 501. If block b (512) is not available or is intra-coded, other blocks to the left of sub-CU A 501 are checked (starting from block b 512 and going from top to bottom). The motion information obtained from the neighboring blocks for each list is scaled to the first reference frame for a given list. Then, the temporal motion vector predictor (TMVP) of sub-block A 501 is derived by following the same TMVP derivation procedure as specified in HEVC. The motion information of the collocated blocks in block D 704 is fetched and scaled accordingly. Finally, after reading and scaling the motion information, all available motion vectors are averaged for each reference list separately, and the averaged motion vector is assigned as the motion vector of the current sub-CU.

[0042] The Bidirectional Optical Flow (BIO) method is a sample-by-sample motion refinement performed on top of block-by-block motion compensation for bidirectional prediction. In some implementations, the sample-level motion refinement does not use signaling.

[0043] I (k) Let ∂I be the luminance value from reference k (k=0,1) after block motion compensation. (k) / ∂x,∂I (k) Let / ∂y be the horizontal and vertical components of the I(k) gradient, respectively. Assuming that optical flow is valid, the motion vector field (v x , v y ) is given by:

[0044]

number

[0045] Combining this optical flow equation with Hermite interpolation of the motion trajectory of each sample yields a function value I (k) and the differential value ∂I (k) / ∂x,∂I (k) We obtain a unique third-order polynomial that fits both t = 0 and t = 0. The value of this polynomial at t = 0 is the BIO prediction:

[0046]

number

[0047] FIG. 6 shows an exemplary optical flow trajectory in the bidirectional optical flow (BIO) method. Here, τ0 and τ1 indicate the distance to the reference frame. The distances τ0 and τ1 are calculated based on the POCs of Ref0 and Ref1, τ0=POC(current)-POC(Ref0), τ1=POC(Ref1)-POC(current). If both predictions come from the same time orientation (both from the past or both from the future), their signs are different (e.g., τ0·τ1<0). In this case, BIO is applied if the predictions are not from the same point of time (e.g., τ0≠τ1). Both reference regions have non-zero motion (e.g., MVx0,MVy0,MVx1,MVy1≠0), and the block motion vector is proportional to the temporal distance (e.g., MVx0 / (MVx1=MVy0 / MVy1)=-τ0 / τ1). In JEM, BIO can be applied to all bi-predictive blocks if the two predictions are from different reference pictures. BIO can be disabled for CUs with Local Illumination Compensation (LIC) enabled.

[0048] In some cases, if the merge flag is true, the FRUC flag can be signaled to the CU. If the FRUC flag is false, the merge index can be signaled and the normal merge mode is used. If the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (e.g., bilateral matching or template matching) is used to derive the motion information of the block.

[0049] On the encoder side, the decision of whether to use the FRUC merging mode for a CU is made based on the RD cost selection, similar to the case of normal merging candidates. For example, multiple matching modes (e.g., bilateral matching and template matching) are checked for a CU by using the RD cost selection. The one that leads to the smallest cost is further compared with other CU modes. If the FRUC matching mode is the most efficient mode, the FRUC flag is set to true for the CU and the associated matching mode is used.

[0050] Typically, the motion derivation process in FRUC merge mode has two steps: first, a CU-level motion search is performed, followed by sub-CU-level motion refinement. At the CU level, an initial motion vector is derived for the entire CU based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that results in the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed around the starting point. The MV that results in the minimum matching cost becomes the MV for the entire CU. Subsequently, the derived CU motion vector is used as a starting point to further refine the motion information at the sub-CU level.

[0051] FIG. 7 shows an example of bilateral matching used in the frame rate up-conversion (FRUC) method. Bilateral matching is used to derive the motion information of a current CU (700) by finding the closest match between two blocks along the motion trajectory of the current CU (700) in two different reference pictures (710, 711). Under the assumption of continuous motion trajectories, the motion vectors MV0 (701) and MV1 (702) pointing to two reference blocks are proportional to the temporal distance between the current picture and the two reference pictures, e.g., TD0 (703) and TD1 (704). In some implementations, if the current picture 700 is temporally between the two reference pictures (710, 711) and the temporal distance from the current picture to the two reference pictures is the same, the bilateral matching becomes mirror-based bidirectional MV.

[0052] FIG. 8 shows an example of template matching used in the Frame-Rate Up Conversion (FRUC) method. Template matching can be used to derive motion information of the current CU 800 by finding the closest match between a template in the current picture (e.g., a block adjacent to the top and / or left of the current CU) and a block in the reference picture 810 (e.g., the same size as the template). Except for the FRUC merge mode mentioned above, template matching can also be applied to the AMVP mode. In both JEM and HEVC, there are two candidates in AMVP. Using the template matching method, a new candidate can be derived. If the newly derived candidate by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list (e.g., by deleting the second existing AMVP candidate), and the list size is set to 2. When applied to the AMVP mode, only the CU level search is applied.

[0053] The MV candidates set at the CU level include (1) the original AMVP candidates if the current CU is in AMVP mode, (2) all merge candidates, (3) some MVs from the interpolated MV field (described below), and / or (4) the top and left neighboring motion vectors.

[0054] When using bilateral matching, each valid MV of a merging candidate can be used as an input to generate an MV pair under the assumption of bilateral matching. For example, one of the valid MVs of a merging candidate is (MVa,ref a ) The reference picture reference ref b ref a and ref b is found in another reference list B such that it is on a different side of the current picture in time. b If there is no ref b ref a The ref is determined as a different reference from the current picture, and its temporal distance from the current picture is the smallest in list B. b After is determined, the current picture and ref a , ref b MVb is derived by scaling MVa based on its temporal distance from

[0055] In some implementations, four MVs from the interpolated MV field can be added to the CU-level candidate list. More specifically, the interpolated MVs are added to the current CU at positions (0,0), (W / 2,0), (0,H / 2) and (W / 2,H / 2). When applying FRUC in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set. In some implementations, at the CU level, 15 MVs for AMVP CUs and 13 MVs for merged CUs can be added to the candidate list.

[0056] The MV candidates set at the sub-CU level include the MV determined from the CU level search, (2) the top, left, top-left, and top-right neighboring MVs, (3) a scaled version of the collocated MV from the reference picture, (4) one or more ATMVP candidates (e.g., up to four), and / or (5) one or more STMVP candidates (e.g., up to four). The scaled MV from the reference picture is derived as follows: Traverse the reference pictures in both lists. The MV of the collocated position of the sub-CU in the reference picture is scaled with reference to the starting CU level MV. The ATMVP and STMVP candidates can be the four initial candidates. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

[0057] Before coding a frame, an interpolated motion field is generated for the entire picture based on unilateral ME, and then the motion field may be used as a CU-level or sub-CU-level MV candidate later.

[0058] In some embodiments, the motion field of each reference picture in both reference lists is traversed at the 4x4 block level. Figure 9 shows an example of one-sided motion estimation (ME) 900 in the FRUC method. For each 4x4 block, if the motion associated with the block passes through the 4x4 block of the current picture and the block is not assigned to any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (similar to the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block of the current frame. If the 4x4 block does not have a scaled MV assigned, the motion of the block is marked as unavailable in the interpolated motion field.

[0059] When motion vectors point to fractional sample positions, motion compensated interpolation is necessary. To reduce complexity, bilinear interpolation can be used instead of the usual 8-tap HEVC interpolation for both bilateral matching and template matching.

[0060] The calculation of the matching cost is slightly different in different steps. When selecting a candidate from the candidate set at the CU level, the matching cost can be the sum of absolute differences (SAD) of bilateral matching or template matching. After the starting MV is determined, the matching cost C of bilateral matching in the sub-CU level search is calculated as follows:

[0061]

number

[0062] where w is a weighting factor. In some embodiments, w can be empirically set to 4. MV and MV S denote the current MV and the start MV, respectively. The SAD can still be used as the matching cost for template matching in the sub-CU level search.

[0063] In FRUC mode, MV is derived using only luma samples. The derived motion is used for both luma and chrominance for inter-MC prediction. After MV is determined, final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chrominance.

[0064] MV refinement is a pattern-based MV search based on bilateral matching cost or template matching cost. Two search patterns are supported in JEM: Unlimited Central Biased Diamond Search (UCBDS) and Adaptive Cross Search for CU-level and sub-CU-level MV refinement, respectively. For both CU-level and sub-CU-level MV refinement, MVs are directly searched with 1 / 4 luminance sample MV precision, followed by 1 / 8 luminance sample MV refinement. The search range for MV refinement in CU and sub-CU steps is set to 8 luminance samples.

[0065] In bilateral matching merge mode, bi-prediction is applied since the motion information of a CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. In template matching merge mode, the encoder can choose between uni-prediction from list0, uni-prediction from list1, or bi-prediction for the CU. The selection can be made based on the template matching cost as follows:

[0066] IF costBi <= factor * min(cost0,cost1)

[0067] Use bi-prediction;

[0068] Otherwise, if cost0 <= cost1

[0069] Use uni-prediction from list0;

[0070] Otherwise,

[0071] Use uniprediction from list1;

[0072] Here, cost0 is the SAD of list0 template matching, cost1 is the SAD of list1 template matching, and costBi is the SAD of bi-predictive template matching. For example, if the value of the coefficient is equal to 1.25, it means that the selection process is biased towards bi-prediction. Inter prediction direction selection can be applied to the CU level template matching process.

[0073] The human visual system is less sensitive to color location and motion than it is to luminance. Therefore, bandwidth can be optimized by preserving more luminance detail than color detail. In video systems, this is achieved by using color difference components. The signal is split into a luminance (Y') component and two color difference (chrominance) components. Chroma subsampling is a method of encoding images by implementing a lower resolution for the chrominance information than for the luminance information, taking advantage of the human visual system's poorer acuity for color difference than for luminance. For example, common types of subsampling include 4:2:2 (where the two chroma components are sampled at half the sample rate of the luminance), 4:1:1 (where the horizontal color resolution is quartered), and 4:2:0 (where the vertical resolution is halved compared to 4:1:1, because the Cb and Cr channels are sampled only on alternating lines). In one example, the HEVC standard specifies how to derive the MV used for MC in the chroma component (denoted as mvC) from the MV used for MC in the luma component (denoted as mv). In general, mvC is calculated as mv multiplied by a factor that depends on the color format, such as 4:2:0 or 4:2:2.

[0074] Intra-Block Copy (IBC, or Intra-Picture Block Compensation), also known as current picture referencing (CPR), has been adopted in the HEVC Screen Content Coding Extension (SCC). This tool is very useful for coding screen content video in that repeated patterns of text and graphics-rich content frequently occur within the same picture. Having a previously reconstructed block with the same or similar pattern as a predictor can effectively reduce the prediction error and thus improve the coding efficiency.

[0075] Similar to the design of the CRP in the HEVC SCC, in VVC, the use of IBC mode is signaled at both the sequence and picture levels. When IBC mode is enabled in the sequence parameter set (SPS), it can be enabled at the picture level. When IBC mode is enabled at the picture level, the current reconstructed picture is treated as a reference picture. Therefore, no block-level syntax changes are required for existing VVC inter modes to signal the use of IBC mode.

[0076] The IBC mode has the following characteristics:

[0077] ○ Treated as a normal inter mode. Therefore, merge and skip modes can be used in IBC mode as well. The merge candidate list construction is unified and contains merge candidates from neighboring positions coded in either IBC mode or HEVC inter mode. Depending on the selected merge index, the current block in merge or skip mode can be merged either with an IBC mode decoded neighboring block or with a normal inter mode coded block with a different picture as a reference picture.

[0078] The block vector prediction and coding scheme for IBC mode reuses the scheme used for motion vector prediction and coding in HEVC Inter mode (AMVP and MVD coding).

[0079] ○ IBC mode motion vectors, also called block vectors, are coded with integer pixel precision, but because the interpolation and deblocking stages require 1 / 4 pixel precision, they are stored in memory with 1 / 16 pixel precision after decoding. When used in IBC mode motion vector prediction, the stored vector predictor is right-shifted by 4.

[0080] Search range: Limited to the current CTU.

[0081] ○CPR is prohibited if affine mode / triangular mode / GBI / weighted prediction is enabled.

[0082] In some cases, pairwise average candidates are generated by averaging predefined candidate pairs in the current merge candidate list, where the predefined pairs are defined as {(0,1), (0,2), (1,2), (0,3), (1,3), (2,3)}, where the numbers represent the merge index into the merge candidate list. The averaged motion vector is calculated separately for each reference list. If both motion vectors are in one list, these two motion vectors are averaged even if they point to different reference pictures. If there is only one motion vector, use that one motion vector directly, and if there is no motion vector, leave this list disabled. The pairwise average candidate replaces the combined candidate in the HEVC standard. If the MV of two merge candidates is MV0 = (MV 0x ,MV 0y ), MV1=(MV 1x ,MV 1y ), then MV * =(MV * x ,MV * y), the MV of a pair of merge candidates, denoted as

[0083] Music Video * x =(MV 0x +MV 1x ) / 2, and

[0084] Music Video * y =(MV 0y +MV 1y ) / 2

[0085] Also, if MV0 and MV1 refer to the current picture (i.e., in CPR mode), MV * x , M.V. * y is further rounded to remove any precision greater than a full pixel, resulting in an integer MV:

[0086] Music Video * x =(MV * x / 16)<<4, and

[0087] Music Video * y =(MV * y / 16)<<4.

[0088] Note that for each pair, if one of the two is CPR coded and the other is not, then such a pair cannot be a pair average candidate.

[0089] The concept of triangular prediction mode (TPM) is to introduce a new triangular partition for motion compensated prediction. TPM divides a CU into two triangular prediction units, diagonal or anti-diagonal. Each triangular prediction unit of a CU is inter-predicted with its own uni-predictive motion vector and reference frame index derived from a single uni-predictive candidate list. After predicting the triangular prediction unit, we perform an adaptive weighting process on the diagonal edges. Then, we apply the transform and quantization process to the whole CU. Note that this mode only applies to merge mode (note: skip mode is treated as a special merge mode). TPM Uni-Prediction Candidate List

[0090] The uni-predictive candidate list, named TPM motion candidate list, consists of five uni-predictive motion vector candidates, which are derived from seven neighboring blocks, including five spatial neighboring blocks and two temporally co-located blocks. The motion vectors of the seven neighboring blocks are collected and put into the uni-predictive candidate list in the following order: uni-predictive motion vector, L0 motion vector of bi-predictive motion vector, L1 motion vector of bi-predictive motion vector, and averaged motion vector of L0 motion vector and L1 motion vector of bi-predictive motion vector. If the number of candidates is less than five, no motion vectors are added to the list. The motion candidates added to this list for TPM are called TPM candidates, and the motion information derived from spatial / temporal blocks is called regular motion candidates.

[0091] Specifically, the steps include:

[0092] (1) Obtain regular motion candidates from A1, B1, B0, A0, B2, Col, and Col2 (similar to the regular merge mode) without pruning operation.

[0093] (2) Set the variable numCurrMergeCand=0

[0094] (3) For each regular motion candidate derived from A1, B1, B0, A0, B2, Col, Col2 and numCurrMergeCand less than 5, if the regular motion candidate is uni-predictive (either List0 or List1), add it directly to the merge list as a TPM candidate and increment numCurrMergeCand by 1. We call such a TPM candidate an "originally uni-predicted candidate."

[0095] Apply complete pruning.

[0096] (4) For each motion candidate derived from A1, B1, B0, A0, B2, Col, Col2, and numCurrMergeCand less than 5, if the regular motion candidate is bi-predictive, the motion information from List0 is added to the TPM merge list (i.e., modified to uni-predictive from List0) as a new TPM candidate, and numCurrMergeCand is incremented by 1. Such TPM candidates are named "pruned List0-predictive candidates".

[0097] Complete pruning is applied.

[0098] (5) For each motion candidate derived from A1, B1, B0, A0, B2, Col, Col2 with numCurrMergeCand less than 5, if the regular motion candidate is bi-predictive, then the motion information from List1 is added to the TPM merge list (i.e., corrected to uni-predictive from List1) and numCurrMergeCand is increased by 1. Such a TPM candidate is called a "pruned List1-predictive candidate".

[0099] Complete pruning is applied.

[0100] (6) For each motion candidate derived from A1, B1, B0, A0, B2, Col, Col2 with numCurrMergeCand less than 5, if the regular motion candidate is bi-predictive, the motion information of List1 is first scaled to the reference picture in List0, and the average of the two MVs (one original from List0, the other scaled from List1) is added to the TPM merge list. We call such a candidate the average uni-predictive from the motion candidates in List0, and increase numCurrgeMerCand by 1.

[0101] Complete pruning is applied.

[0102] (7) If numCurrMergeCand is less than 5, a zero motion vector candidate is added.

[0103] If, while inserting a candidate into a list, the candidate has to be compared with all previously added candidates to see if it is identical to one of the previously added candidates, the process is called full pruning.

[0104] The scaled MV is (MV 1’x ,MV 1’y ), and the MV of list 0 is represented as (MV 0x ,MV 0y ) is assumed to be expressed as (MV * x ,MV * y The average uni-prediction from List0 motion candidates, denoted by

[0105] Music Video * x =(MV 0x +MV 1’x +1)>>1

[0106] Music Video * y =(MV 0y +MV 1’y +1)>>1.

[0107] In some existing implementations, such as affine prediction in JEM, the MVs of each subblock are calculated using an affine model, as shown in equation (1) for each component independently, which may result in misalignment of motion vectors between the luminance and chrominance components. The calculation of multiple components also results in coding inefficiencies. In some other existing implementations, the motion vector averaging operation to derive pair merge candidates / averaged uni-prediction from List0 motion candidates needs to be consistent with the rounding method used in subblock prediction. Therefore, it is desirable to obtain a unified design.

[0108] This patent document discloses a technique that can be implemented in various embodiments to calculate the MV of a sub-block of one component (e.g., a chrominance component) from the MV of one or more sub-blocks of another component (e.g., a luminance component). The disclosed technique eliminates the need to determine the MV of different components multiple times, thereby improving video coding efficiency. The disclosed technique also introduces a unified design for averaging operations and rounding methods.

[0109] The use of such techniques will become apparent in the following examples where various embodiments are described, in which, although not to be construed as limiting, Shift(x,s) is defined as Shift(x,s)=(x+off)>>s.

[0110] SignShift(x,s) is defined as follows:

[0111]

number

[0112] Here, off is an integer, e.g., 0 or 1<<(s-1).

[0113] In some embodiments, the MVs of a sub-block of one component can be derived based on the MVs of one or more sub-blocks of another component, where the MVs of the one or more sub-blocks of the other component have already been derived with an affine model, in this way eliminating the need for multiple derivations of motion information.

[0114] In some embodiments, the HEVC standard defines how to derive the MV in the chrominance component (denoted mvC) from the MV used for MC in the luminance component (denoted mv). In general, mvC is calculated by multiplying mv by a coefficient that depends on the color format, such as 4:2:0 or 4:2:2. Figure 10 shows an example of deriving the MV of the sub-block components in 4:2:0 format according to the present technology. In this example, the block size is 16x16 for Y (luminance component) and 8x8 for Cb / Cr (chrominance components). The sub-block size of the luminance component is 4x4 and the sub-block size of the chrominance component is 2x2. MV * is first derived for the 4×4 sub-block in the Y component. The MVs of the 2×2 sub-blocks in the Cb and / or Cr components are calculated as MV * In this particular example, MV * The values ​​of are first copied, and the MVs of the 2 × 2 sub-blocks in the Cb and / or Cr components are calculated according to the HEVC standard. * It is calculated based on:

[0115] FIG. 11 shows another example of deriving the MVs of the sub-block components in the 4:2:0 format according to the present technique. In this example, the block size is 16×16 for the luminance component and 8×8 for the chrominance component. The sub-block size for all components is 4×4. A 4×4 sub-block of the Cb or Cr component corresponds to a 4×4 sub-block of the Y component. The MVs of the luminance components * is calculated first. Then, the MV of the 4 × 4 sub-block in the Cb or Cr component is calculated as MV * It is derived based on

[0116] In some embodiments, MV * is calculated as the average of all corresponding sub-block MVs in the Y component: MV * = (MV0+MV1+MV2+MV3) / 4. MV * =(MV * x , M.V. * y ), MV0=(MV 0x , M.V. 0y ), MV1=(MV 1x , M.V. 1y ), MV2=(MV 2x , M.V. 2y ), MV3=(MV 3x , M.V. 3y )

[0117] In some embodiments, MV * x =Shift(MV 0x +MV 1x +MV 2x +MV 3x 2), M.V. * y=Shift(MV 0y +MV 1y +MV 2y +MV 3y , 2). In some embodiments, MV * x =SignShift(MV 0x +MV 1x +MV 2x +MV 3x 2), M.V. * y =SignShift(MV 0y +MV 1y +MV 2y +MV 3y ,2).

[0118] In some embodiments, MV * The calculation of can be done using the following operations:

[0119] 1.a MV' x =Shift(MV 0x +MV 1x,1)

[0120] 1.b MV' y =Shift(MV 0y +MV 1y ,1),

[0121] 1.c MV'' x =Shift(MV 2x +MV 3x ,1),

[0122] 1.d MV'' y =Shift(MV 2y +MV 3y ,1)

[0123] 1.e MV * x =Shift(MV' x +MV'' x ,1),

[0124] 1.fMV * y =Shift(MV' y +MV'' y ,1)

[0125] In some embodiments, MV * The calculation of can be done using the following operations:

[0126] 2.a MV' x =Shift(MV 0x +MV 2x ,1)

[0127] 2.b MV' y =Shift(MV 0y +MV 2y ,1)

[0128] 2.c MV'' x =Shift(MV 1x +MV 3x ,1)

[0129] 2.d MV'' y =Shift(MV1y +MV 3y ,1)

[0130] 2.e MV * x =Shift(MV' x +MV'' x ,1),

[0131] 2.fMV * y =Shift(MV' y +MV'' y ,1)

[0132] In some embodiments, MV * The calculation of can be done using the following operations:

[0133] 3.a MV' x =SignShift(MV 0x +MV 1x ,1)

[0134] 3.b MV' y =SignShift(MV 0y +MV 1y ,1)

[0135] 3.c MV'' x =SignShift(MV 2x +MV 3x ,1)

[0136] 3.d MV'' y =SignShift(MV 2y +MV 3y ,1),

[0137] 3.e MV * x =SignShift(MV' x +MV'' x ,1)

[0138] 3.fMV * y =SignShift(MV' y +MV''y ,1).

[0139] In some embodiments, MV * The calculation of can be done using the following operations:

[0140] 4.a MV' x =SignShift(MV 0x +MV 2x ,1)

[0141] 4.b MV' y =SignShift(MV 0y +MV 2y ,1)

[0142] 4.c MV'' x =SignShift(MV 1x +MV 3x ,1)

[0143] 4.d MV'' y =SignShift(MV 1y +MV 3y ,1)

[0144] 4.e MV * x =SignShift(MV' x +MV'' x ,1)

[0145] 4.fMV * y =SignShift(MV' y +MV'' y ,1).

[0146] In some embodiments, MV * is calculated based on the MV of the top-left sub-block in the Y component (e.g., MV0 as shown in FIG. 11). * is calculated based on the MV of the central subblock in the Y component. *is calculated based on the median MV of all corresponding sub-blocks in the Y component. In this particular example shown in FIG. * =median(MV0, MV1, MV2, MV3).

[0147] In some embodiments, the derivation of the MVs of the sub-blocks of a color component may be determined based on the color format, such as 4:2:0, 4:2:2, and / or 4:4:4. For example, if the color format is 4:4:4, no subsampling is performed. The sub-block size and the number of sub-blocks in a block of video data are the same for all components. The MV of a sub-block of one component is the same as the MV of the corresponding sub-block of another component.

[0148] As another example, if the color format is 4:2:2, the size of the sub-blocks may be the same for all components, while the number of blocks may be different for different components. * is first calculated based on the MVs of some corresponding sub-blocks. Then, the MV(s) of the corresponding sub-blocks in the Cb or Cr component is calculated as MV * Derive from.

[0149] FIG. 12 shows yet another example of deriving the MVs of the components of the sub-blocks according to the present technique. In this example, the block size is 16×16 for the luminance component and 8×16 for the chrominance component. The sub-block size for all components is 4×4. A 4×4 sub-block of the Cb or Cr component corresponds to two 4×4 sub-blocks of the Y component. The MVs of the luminance components * Then, the MV of the 4 × 4 sub-block in the Cb or Cr component is calculated as MV according to the HEVC standard. * It is derived based on.

[0150] In some embodiments, MV * is calculated as the average of all corresponding sub-block MVs in the Y component: MV * = (MV0 + MV1) / 2. MV * =(MV* x , M.V. * y ), MV0=(MV 0x , M.V. 0y ), MV1=(MV 1x , M.V. 1y )

[0151] In some embodiments, MV * x =Shift(MV 0x +MV 1x ,1), M.V. * y =Shift(MV 0y +MV 1y In some embodiments, MV * x =SignShift(MV 0x +MV 1x ,1), M.V. * y =SignShift(MV 0y +MV 1y ,1).

[0152] 13 is a flowchart of an example method 1300 for video encoding in accordance with one or more embodiments of the present technology. The method 1300 includes, at operation 1310, partitioning a first component of the current video block into a first set of sub-blocks. The method 1300 includes, at operation 1320, partitioning a second component of the current video block into a second set of sub-blocks. The sub-blocks of the second component correspond to one or more sub-blocks of the first component. The method 1300 also includes, at operation 1330, deriving motion vectors for the sub-blocks of the second component based on the motion vectors of one or more corresponding sub-blocks of the first color component based on a color format of the current video block.

[0153] 14 is a flow chart of another example method 1400 for video encoding in accordance with one or more embodiments of the present technology. The method 1400 includes, at operation 1410, receiving a block of video data including a first component and at least a second component. The first component is divided into a first set of sub-blocks and the second component is divided into a second set of sub-blocks. The sub-blocks of the second component correspond to one or more sub-blocks of the first component. The method 1400 includes, at operation 1420, deriving motion vectors for sub-blocks of the second component based on a color format of the block of video data based on the motion vectors of one or more corresponding sub-blocks of the first color component. The method 1400 includes, at operation 1430, reconstructing the block of video data or decoding other blocks of video data in the same picture based on the derived motion vectors.

[0154] In some implementations, the sub-blocks of the second component correspond to the sub-blocks of the first component. For example, these two sub-blocks may be spatially co-located in the picture. The method 1300 or 1400 further includes calculating an intermediate motion vector set by copying the motion vectors of the sub-blocks of the first component, and applying coefficients to the intermediate motion vector set to obtain derived motion vectors. The coefficients are associated with a color format.

[0155] In some embodiments, the color format is not 4:4:4 (e.g., 4:2:2 or 4:2:0) and a size of at least one subblock of the first subblock set is different from a size of at least one subblock of the second subblock set. In some embodiments, the color format is 4:4:4 and a size of at least one subblock of the first subblock set is the same as a size of at least one subblock of the second subblock set.

[0156] In some embodiments, the color format is not 4:4:4, and a size of at least one subblock of the first subblock set is the same as a size of at least one subblock of the second subblock set. The subblocks of the second component correspond to a plurality of subblocks of the first component. In some embodiments, at least one of the motion vectors of the subblocks of the second component is derived based on an average of corresponding motion vectors of the plurality of subblocks of the first component.

[0157] In some implementations, the sub-blocks of the second component correspond to four sub-blocks of the first component. The four sub-blocks are each assigned a motion vector MV0=(MV 0x , M.V. 0y ), MV1=(MV 1x , M.V. 1y ), MV2=(MV 2x , M.V. 2y ) and MV3=(MV 3x , M.V. 3y ) The method 1300 or 1400 includes an intermediate motion vector set MV * =(MV * x , M.V. * y ) and MV * x =Shift(MV 0x +MV 1x +MV 2x +MV 3x ,2) and MV * y =Shift(MV y0 +MV y1 +MV y2 +MV y3 ,2), where Shift(x,s)=(x+off)≫s, where off and s are integers, and >> represents a right shift operation. Method 1300 or 1400 also includes applying a coefficient to the intermediate motion vector set to obtain a derived motion vector, the coefficient being associated with a color format.

[0158] In some implementations, the sub-blocks of the second component correspond to four sub-blocks of the first component. The four sub-blocks are each assigned a motion vector MV0=(MV 0x , M.V. 0y ), MV1=(MV 1x , M.V. 1y ), MV2=(MV 2x , M.V. 2y ) and MV3=(MV 3x , M.V. 3y ) The method 1300 or 1400 includes an intermediate motion vector set MV * =(MV * x , M.V. * y ) to MV * x =SignShift(MV 0x +MV 1x +MV 2x +MV 3x ,2) and MV * y =SignShift(MV 0y +MV 1y +MV 2y +MV 3y ,2), where SignShift(x,s)=(x+off)≫s for x≧0 or -((-x+off)≫s for x<0, where off and s are integers, and >> represents a shift operation. Method 1300 or 1400 also includes applying a coefficient to the set of intermediate motion vectors to obtain a derived motion vector, the coefficient being associated with a color format.

[0159] In some implementations, the sub-blocks of the second component correspond to four sub-blocks of the first component. The four sub-blocks are each assigned a motion vector MV0=(MV 0x , M.V. 0y ), MV1=(MV 1x , M.V. 1y ), MV2=(MV 2x , M.V. 2y ) and MV3=(MV 3x , M.V. 3y) The method 1300 or 1400 includes an intermediate motion vector set MV * =(MV * x , M.V. * y ) and MV * x =Shift(MV' x +MV'' x ,1) and MV * y =Shift(MV' y +MV'' y ,1), where MV' x =Shift(MV 0x +MV 1x ,1),MV' y =Shift(MV 0y +MV 1y ,1), M.V. x =Shift(MV 2x +MV 3x ,1) and MV'' y =Shift(MV 2y +MV 3y ,1), Shift(x,s)=(x+off)≫s, where off and s are integers, and >> represents a right shift operation. Method 1300 or 1400 also includes applying a coefficient to the intermediate motion vector set to obtain a derived motion vector, the coefficient being associated with a color format.

[0160] In some implementations, the sub-blocks of the second component correspond to four sub-blocks of the first component. The four sub-blocks are each assigned a motion vector MV0=(MV 0x , M.V. 0y ), MV1=(MV 1x , M.V. 1y ), MV2=(MV 2x , M.V. 2y ) and MV3=(MV 3x , M.V. 3y ) The method 1300 or 1400 includes an intermediate motion vector set MV * =(MV * x , M.V. *y ) and MV * x =Shift(MV' x +MV'' x ,1), M.V. * y =Shift(MV' y +MV'' y ,1),MV' x =Shift(MV 0x +MV 2x ,1),MV' y =Shift(MV 0y +MV 2y ,1), M.V. x =Shift(MV 1x +MV 3x ,1), M.V. y =Shift(MV 1y +MV 3y ,1), where Shift(x,s)=(x+off)≫s, where off and s are integers, and >> represents a right shift operation. Method 1300 or 1400 also includes applying a coefficient to the set of intermediate motion vectors to obtain a derived motion vector, where the coefficient is associated with a color format.

[0161] In some implementations, the sub-blocks of the second component correspond to four sub-blocks of the first component. The four sub-blocks are each assigned a motion vector MV0=(MV 0x , M.V. 0y ), MV1=(MV 1x , M.V. 1y ), MV2=(MV 2x , M.V. 2y ) and MV3=(MV 3x , M.V. 3y ) The method 1300 or 1400 includes an intermediate motion vector set MV * =(MV * x , M.V. * y ) and MV * x =SignShift(MV' x +MV'' x ,1), M.V. *y =SignShift(MV' y +MV'' y ,1),MV' x =SignShift(MV 0x +MV 1x ,1),MV' y =SignShift(MV 0y +MV 1y ,1), M.V. x =SignShift(MV 2x +MV 3x ,1), M.V. y =SignShift(MV 2y +MV 3y ,1), where SignShift(x,s)=(x+off)≫s for x≧0 or -((-x+off)≫s for x<0, where off and s are integers and >> represents a shift operation. Method 1300 or 1400 also includes applying a coefficient to the set of intermediate motion vectors to obtain a derived motion vector, where the coefficient is associated with a color format.

[0162] In some implementations, the sub-blocks of the second component correspond to four sub-blocks of the first component. The four sub-blocks are each assigned a motion vector MV0=(MV 0x , M.V. 0y ), MV1=(MV 1x , M.V. 1y ), MV2=(MV 2x , M.V. 2y ) and MV3=(MV 3x , M.V. 3y ) The method 1300 or 1400 includes an intermediate motion vector set MV * =(MV * x , M.V. * y ) and MV * x =SignShift(MV' x +MV'' x ,1), M.V. * y =SignShift(MV' y +MV''y ,1),MV' x =SignShift(MV 0x +MV 2x ,1),MV' y =SignShift(MV 0y +MV 2y ,1), M.V. x =SignShift(MV 1x +MV 3x ,1), M.V. y =SignShift(MV 1y +MV 3y ,1), where SignShift(x,s)=(x+off)≫s for x≧0 or -((-x+off)≫s for x<0, where off and s are integers and >> represents a shift operation. Method 1300 or 1400 also includes applying a coefficient to the set of intermediate motion vectors to obtain a derived motion vector, where the coefficient is associated with a color format.

[0163] In some implementations, a sub-block of the second component corresponds to two sub-blocks of the first component. The two sub-blocks are respectively associated with a motion vector MV0=(MV 0x , M.V. 0y ) and MV1=(MV 1x , M.V. 1y ) The method 1300 or 1400 includes an intermediate motion vector set MV * =(MV * x , M.V. * y ) and MV * x =Shift(MV 0x +MV 1x ,1) and MV * y =Shift(MV y0 +MV y1,1), where Shift(x,s)=(x+off)≫s, where off and s are integers, and >> represents a right shift operation. Method 1300 or 1400 also includes applying a coefficient to the set of intermediate motion vectors to obtain a derived motion vector, where the coefficient is associated with a color format.

[0164] In some implementations, a sub-block of the second component corresponds to two sub-blocks of the first component. The two sub-blocks are respectively associated with a motion vector MV0=(MV 0x , M.V. 0y ) and MV1=(MV 1x , M.V. 1y ).

[0165] The method 1300 or 1400 uses an intermediate motion vector set MV * =(MV * x , M.V. * y ) to MV * x =SignShift(MV 0x +MV 1x ,1) and MV * y =SignShift(MV y0 +MV y1 ,1), where SignShift(x,s) = (x+off)≫s for x≧0 or -((-x+off)≫s for x<0).

[0166] where off and s are integers and >> represents a shift operation. The method 1300 or 1400 also includes applying a coefficient to the intermediate motion vector set to obtain a derived motion vector, the coefficient being associated with a color format.

[0167] In some embodiments, the method 1300 or 1400 further includes calculating an intermediate motion vector set based on the motion vector of a selected one of the plurality of sub-blocks of the first component, and applying a coefficient to the intermediate motion vector set to obtain a derived motion vector. The coefficient is associated with a color format. In some implementations, the selected sub-block is an upper left sub-block of the plurality of sub-blocks of the first component. In some implementations, the selected sub-block is a center sub-block of the plurality of sub-blocks of the first component.

[0168] In some embodiments, method 1300 or 1400 includes calculating an intermediate motion vector set based on a median of motion vectors of multiple sub-blocks of the first component, and applying a coefficient to the intermediate motion vector set to obtain a derived motion vector, the coefficient being associated with a color format.

[0169] In some embodiments, the application of the coefficients is specified in the High Efficiency Video Coding (HEVC) standard.

[0170] The above examples may be incorporated into the context of the methods described below, eg, methods 1300 and 1400, which may be implemented in a video decoder and / or a video encoder.

[0171] 15 is a block diagram illustrating an example encoding device 1500 that can be used to implement various portions of the techniques of this disclosure, including but not limited to method 1300 and method 1400. The encoding device 1500 includes a quantizer 1505 that compresses input data bits. The encoding device 1500 also includes an inverse quantizer 1515 that can provide the data bits to a memory 1525, and a predictor 1520 that performs motion prediction. The encoding device 1500 further includes a binary encoder 1530 that generates an encoded binary code.

[0172] 16 is a block diagram illustrating an example encoding device 1600 that can be used to implement various portions of the techniques of this disclosure, including but not limited to the method 1300 and the method 1400. The decoding device 1600 includes a bindery decoder 1605 that decodes binary codes. The decoding device 1600 also includes an inverse quantizer 1615 to provide decoded data bits to a memory 1625, and a predictor 1620 that performs motion estimation on the decoding side.

[0173] FIG. 17 is a block diagram illustrating an example of an architecture of a computer system or other control device 1700 that can be used to implement various portions of the techniques of this disclosure, including but not limited to method 1300 and method 1400. In FIG. 17, computer system 1700 includes one or more processors 1705 and memory 1710 connected via an interconnect 1725. Interconnect 1725 can represent any one or more separate physical buses, point-to-point connections, or both, connected by appropriate bridges, adapters, or controllers. Thus, interconnect 1725 may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) Standard 674 bus, sometimes referred to as "Firewire."

[0174] The processor 1705 may include, for example, a central processing unit that controls the overall operation of the host computer. In particular embodiments, the processor 1705 accomplishes this by executing software or firmware stored in memory 1710. The processor 1705 may be or include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices, or the like, or a combination of these devices.

[0175] Memory 1710 may be or include a main memory of a computer system. Memory 1710 represents any suitable form of random access memory, read-only memory, flash memory, etc., or combinations of these devices. In use, memory 1710 may include, among other things, a set of machine instructions that, when executed by processor 1705, cause processor 1705 to perform operations that implement embodiments of the techniques of this disclosure.

[0176] Also connected to the processor 1705 via interconnect 1725 is an (optional) network adapter 1715. The network adapter 1715 provides the computer system 1700 with the ability to communicate with remote devices, such as storage clients and / or other storage servers, and may be, for example, an Ethernet adapter or a Fibre Channel adapter.

[0177] FIG. 18 illustrates a block diagram of an exemplary embodiment of a mobile device 1800 that can be used to implement various portions of the techniques of this disclosure, including but not limited to method 1600. The mobile device 1800 can be a laptop, a smartphone, a tablet, a camcorder, or any other type of device capable of processing video. The mobile device 1800 includes a processor or controller 1801 for processing data and a memory 1802 in communication with the processor 1801 for storing and / or buffering data. For example, the processor 1801 can include a central processing unit (CPU) or a microcontroller device (MCU). In some implementations, the processor 1801 can include a field programmable gate array (FPGA). In some implementations, the mobile device 1800 includes or is in communication with a graphics processing unit, a video processing unit, and / or a wireless communication unit for various visual and / or communication data processing functions of a smartphone device. For example, memory 1802 can include and store processor-executable code that, when executed by processor 1801, configures mobile device 1800 to perform various operations, such as receiving information, commands, and / or data, processing information and data, and transmitting or providing the processed information / data to other devices, such as actuators or external displays.

[0178] To support various functions of the mobile device 1800, the memory 1802 may store information and data, such as instructions, software, values, images, and other data that are processed or referenced by the processor 1801. For example, various types of random access memory (RAM) devices, read only memory (ROM) devices, flash memory devices, and other suitable storage media may be used to realize the storage functions of the memory 1802. In some implementations, the mobile device 1800 includes an input / output (I / O) unit 1803 that interfaces the processor 1801 and / or the memory 1802 to other modules, units, or devices. For example, the I / O unit 1803 may interface the processor 1801 and the memory 1802 to utilize various types of wireless interfaces compatible with typical data communication standards, such as between one or more computers in the cloud and user devices. In some implementations, the mobile device 1800 may interface with other devices using wired connections via the input / output unit 1803. The mobile device 1800 may also interface with other external interfaces, such as a data storage device and / or a visual or audio display device 1804, to read and transfer data and information that may be processed by a processor, stored in a memory, or displayed on an output unit or external device of the display device 1804. For example, the display device 1804 may display a video frame that includes blocks (CU, PU, ​​or TU) to which intra-block copying is applied based on whether the block is coded using a motion compensation algorithm and according to the techniques of this disclosure.

[0179] From the foregoing, it will be understood that, for purposes of illustration, specific embodiments of the disclosed technology have been described herein, but various modifications can be made without departing from the scope of the invention, and therefore the disclosed technology is not to be limited except as by the appended claims.

[0180] The implementation and functional operations of the subject matter described in this patent document can be implemented in various systems, digital electronic circuits, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Implementations of the subject matter described herein can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for executing or controlling the operation of a data processing device. The computer readable medium can be a machine readable storage device, a machine readable storage substrate, a memory device, a composition of matter affecting a machine readable propagated signal, or one or more combinations thereof. The term "data processing unit" or "data processing device" encompasses all apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, such as code constituting a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0181] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in several coordinated files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on several computers that are located on one computer or at one site, or distributed across several sites and interconnected by a communication network.

[0182] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0183] Processors suitable for executing a computer program include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive or transfer data. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0184] The specification, together with the drawings, are to be considered merely illustrative, and illustrative is meant to be an example. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, the use of "or" is intended to include "and / or" unless the context clearly indicates otherwise.

[0185] Although this patent document contains many details, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination and initially claimed as such, one or more features from a claimed combination may, in some cases, be carved out of the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0186] Similarly, although the figures depict acts in a particular order, this should not be understood as requiring such acts to be performed in the particular order or sequential order shown, or to perform all of the acts illustrated, in order to achieve desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0187] Only a few embodiments and examples are described; other embodiments, extensions and variations can be made based on what is described and illustrated in this patent document.

Claims

1. 1. A method for processing video data, comprising the steps of: determining, during conversion between a current video unit of a video, the current video unit including a luma block and at least one chroma block, and a bitstream of the video, a motion vector of a control point of the luma block based on an affine mode; dividing the luma block into luma sub-blocks; dividing a chroma block of the at least one chroma block into chroma sub-blocks; deriving a luma motion vector for each luma sub-block based on the motion vectors of the control points; deriving a chroma motion vector for each chroma sub-block based on a luma motion vector of a plurality of corresponding luma sub-blocks and a color format of the current video unit; reconstructing the luma block based on the luma motion vector of each luma sub-block and reconstructing the chroma block based on the chroma motion vector of each chroma sub-block; Including, each luma sub-block has a first size and each chroma sub-block has a second size, the second size being equal to the first size when the color format is 4:2:0 or 4:2:2; the number of chroma sub-blocks is different from the number of luma sub-blocks when the color format is 4:2:0 or 4:2:2; method.

2. Deriving the chroma motion vector for each chroma sub-block comprises: determining that the chroma blocks are divided into at least one first chroma group, the first chroma group including two or four chroma sub-blocks according to the color format of the current video unit; deriving a group chroma motion vector for each of the at least one first chroma group; assigning the group chroma motion vector to each sub-block of the corresponding first chroma group; The method of claim 1 further comprising:

3. The method of claim 2 , wherein the group chroma motion vector is derived based on applying a scaling factor to an intermediate motion vector.

4. The first chroma group includes two chroma sub-blocks when the color format is 4:2:2, and two luma sub-blocks corresponding to the two chroma sub-blocks respectively have a motion vector MV 0 and M.V. 1 and the intermediate motion vector MV* is the MV 0 and the above-mentioned MV 1 The method of claim 3 , wherein the first and second inputs are derived based on:

5. The intermediate motion vector is the MV 0 and the above MV 1 The method of claim 4 , wherein the offset is derived based on applying an offset-based averaging operation to

6. The intermediate motion vector MV*=Shift(MV 0 +MV 1 6. The method of claim 5, wherein Shift(x,1)=(x+offset)>>1, where offset is equal to 0 or 1, and >> denotes a right shift operation.

7. The first chroma group has a color format of 4:2:0, and the upper left one of the four luma sub-blocks represents a motion vector MV 0 and the top right one of the four luma sub-blocks has a motion vector MV 1 and the bottom left one of the four luma sub-blocks has a motion vector MV 2 and the bottom right one of the four luma sub-blocks has a motion vector MV 3 , the four luma sub-blocks correspond to the four chroma sub-blocks, and the intermediate motion vector MV* is at least 0 and the above-mentioned MV 3 It is derived based on The intermediate motion vector is at least the MV 0 and the above MV 3 , derived based on applying an offset-based averaging operation to The method according to claim 3.

8. The method of claim 7, wherein the chroma blocks are reconstructed based on the group chroma motion vector. The method of claim 2 , further comprising:

9. The method described in claim 8, wherein the chroma blocks are reconstructed in units of the first chroma group.

10. the first size and the second size are equal to 4*4; The number of control points is 2 or 3. The control points include a top left control point and a top right control point, or the control points include a top left control point, a top right control point and a bottom left corner control point. The method of claim 1.

11. and determining, for the conversion between the current video unit of the video and the bitstream of the video, whether a first affine mode and a second affine mode are allowed for the current video unit based on a same dimensional constraint. The method of claim 1.

12. In the first affine mode, motion information of control points of the current video unit is derived based on at least an affine candidate of the current video unit without applying motion vector differential (MVD); the first affine mode is a merge affine mode and the second affine mode is a non-merged affine mode, or the first affine mode is a direct affine mode and the second affine mode is a non-direct affine mode; The method determining, based at least on dimensions of the current video unit, that the bitstream omits an indication of the first affine mode; determining, based at least on dimensions of the current video unit, that the bitstream omits an indication of the second affine mode; the first affine mode and the second affine mode are not allowed in response to either a height of the current video unit or a width of the current video unit being less than or equal to 8, or the first affine mode and the second affine mode are not allowed in response to both a height and a width of the current video unit being less than or equal to a threshold. The method of claim 11. constructing a merge motion candidate list for a current block of the video, the merge motion candidate list including at least two merge motion candidates; deriving a pair-wise average merging candidate from pairs of merge motion candidates included in the merge motion candidate list based on a position index into the merge motion candidate list; performing a first transformation between the current block and the bitstream of the video based on the pairwise average merging candidates; Including, The pair of merge motion candidates includes a first motion vector MV0=(MV0x, MV0y) of a reference picture list L in a first merge motion candidate and a second motion vector MV1=(MV1x, MV1y) of the reference picture list L in a second merge motion candidate; MV* = (MV*x, MV*y) is [0010] where off and s are integers and >> represents the right shift operation. Pairwise merge candidates, The method of claim 1.

14. The position indices of the pair of merge motion candidates are 0 and 1. The reference picture list L includes a reference picture list 0 (L0) or a reference picture list 1 (L1), The method further includes inserting the pairwise average merging candidate into the merge motion candidate list; If it is determined that s=0, then off=0, or If it is determined that s ≠ 0, then off = 1 << (s-1), or off=(1<<(s-1))-1, The method of claim 13, wherein

15. The method of claim 1 , wherein the converting comprises encoding the current video unit into the bitstream.

16. The method of claim 1 , wherein the converting comprises decoding the current video unit from the bitstream.

17. 1. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions that, when executed by the processor, cause the processor to: determining, during conversion between a current video unit of a video, the current video unit including a luma block and at least one chroma block, and a bitstream of the video, a motion vector of a control point of the luma block based on an affine mode; dividing the luma block into luma sub-blocks; dividing a chroma block of the at least one chroma block into chroma sub-blocks; deriving a luma motion vector for each luma sub-block based on the motion vectors of the control points; deriving a chroma motion vector for each chroma sub-block based on a luma motion vector of a plurality of corresponding luma sub-blocks and a color format of the current video unit; reconstructing the luma block based on the luma motion vector of each luma sub-block and reconstructing the chroma block based on the chroma motion vector of each chroma sub-block; Run the command, each luma sub-block has a first size and each chroma sub-block has a second size, the second size being equal to the first size when the color format is 4:2:0 or 4:2:2; the number of chroma sub-blocks is different from the number of luma sub-blocks when the color format is 4:2:0 or 4:2:2; Device.

18. A non-transitory computer-readable storage medium storing instructions, the instructions causing a processor to: determining, during conversion between a current video unit of a video, the current video unit including a luma block and at least one chroma block, and a bitstream of the video, a motion vector of a control point of the luma block based on an affine mode; dividing the luma block into luma sub-blocks; dividing a chroma block of the at least one chroma block into chroma sub-blocks; deriving a luma motion vector for each luma sub-block based on the motion vectors of the control points; deriving a chroma motion vector for each chroma sub-block based on a luma motion vector of a plurality of corresponding luma sub-blocks and a color format of the current video unit; reconstructing the luma block based on the luma motion vector of each luma sub-block and reconstructing the chroma block based on the chroma motion vector of each chroma sub-block; Run the command, each luma sub-block has a first size and each chroma sub-block has a second size, the second size being equal to the first size when the color format is 4:2:0 or 4:2:2; the number of chroma sub-blocks is different from the number of luma sub-blocks when the color format is 4:2:0 or 4:2:2; storage medium.

19. A method for storing a bitstream of video, the method comprising: determining, for a current video unit of the video including a luma block and at least one chroma block, a motion vector of a control point of the luma block based on an affine mode; dividing the luma block into luma sub-blocks; dividing a chroma block of the at least one chroma block into chroma sub-blocks; deriving a luma motion vector for each luma sub-block based on the motion vectors of the control points; deriving a chroma motion vector for each chroma sub-block based on a luma motion vector of a plurality of corresponding luma sub-blocks and a color format of the current video unit; reconstructing the luma block based on the luma motion vector of each luma sub-block and reconstructing the chroma block based on the chroma motion vector of each chroma sub-block; generating the bitstream based on the reconstruction; and storing the bitstream on a non-transitory computer-readable recording medium; Including, each luma sub-block has a first size and each chroma sub-block has a second size, the second size being equal to the first size when the color format is 4:2:0 or 4:2:2; the number of chroma sub-blocks is different from the number of luma sub-blocks when the color format is 4:2:0 or 4:2:2; method.

Images