Subblock motion vector inheritance between color components

Abstract

Devices, systems, and methods for sub-block-based prediction are described. In a typical aspect, a method for video processing includes segmenting a first component of a current video block into a first set of sub-blocks and segmenting a second component of the current video block into a second set of sub-blocks. The sub-blocks of the second component correspond to one or more sub-blocks of the first component. The method also includes deriving motion vectors for the sub-blocks of the second component based on the motion vectors of one or more corresponding sub-blocks of the first component, according to the color format of the current video block.

Description

[0001] This patent application is a divisional application of the following invention patent application:

[0002] Application Number: 201910545236.6

[0003] Application date: June 21, 2019

[0004] Invention Title: Sub-block Motion Vector Inheritance Between Color Components

[0005] Cross-reference to related applications

[0006] In accordance with the applicable Patent Law and / or the Paris Convention, this application promptly claims priority and benefit to International Patent Application No. PCT / CN2018 / 092118, filed June 21, 2018, and International Patent Application No. PCT / CN2018 / 114931, filed November 10, 2018. The entire disclosure of International Patent Application No. PCT / CN2018 / 092118 and International Patent Application No. PCT / CN2018 / 114931 is incorporated herein by reference as part of the disclosure of this application. Technical Field

[0007] This patent document generally relates to image and video coding technologies. Background Technology

[0008] Motion compensation is a video processing technique that predicts frames in a video by taking into account the motion of objects in the camera and / or video, given previous and / or future frames. Motion compensation can be used for encoding and decoding video data to achieve video compression. Summary of the Invention

[0009] Devices, systems, and methods related to sub-block-based prediction for image and video coding are described.

[0010] In one typical aspect, the disclosed technology can be used to provide a method for video encoding. This method includes segmenting a first component of a current video block into a first group of sub-blocks and segmenting a second component of the current video block into a second group of sub-blocks. The sub-blocks of the second component correspond to one or more sub-blocks of the first component. The method also includes deriving motion vectors for the sub-blocks of the second component based on the motion vectors of one or more corresponding sub-blocks of the first color component, based on the color format of the current video block.

[0011] In another typical aspect, the disclosed technology can be used to provide a method for video decoding. The method includes receiving a video data block comprising a first component and at least a second component. The first component is segmented into a first set of sub-blocks, and the second component is segmented into a second set of sub-blocks. Sub-blocks of the second component correspond to one or more sub-blocks of the first component. The method includes deriving motion vectors for the sub-blocks of the second component based on the color format of the video data block and on the motion vectors of one or more corresponding sub-blocks of the first color component. The method also includes reconstructing the video data block based on the derived motion vectors or decoding other video data blocks in the same frame.

[0012] In another typical aspect, the above methods are embodied in processor-executable code and stored in a computer-readable program medium.

[0013] In yet another typical aspect, a device configured or operable to perform the methods described above is disclosed. This device may include a processor programmed to implement the methods.

[0014] In another typical aspect, a video encoder or video decoder device may implement the methods described herein.

[0015] The above and other aspects and features of the disclosed technology are described in more detail in the accompanying drawings, description and claims. Attached Figure Description

[0016] Figure 1 An example of prediction based on sub-blocks is shown.

[0017] Figure 2 An example of a simplified affine motion model is shown.

[0018] Figure 3 An example of the affine motion vector field (MVF) for each sub-block is shown.

[0019] Figure 4 An example of motion prediction using the Optional Temporal Motion Vector Prediction (ATMVP) algorithm for coding units (CUs) is shown.

[0020] Figure 5 An example of a CU with four sub-blocks and adjacent blocks is shown.

[0021] Figure 6 An example of the optical flow trajectory used by the bidirectional optical flow (BIO) algorithm is shown.

[0022] Figure 7 An example of bidirectional matching in the Frame Rate Upconversion (FRUC) algorithm is shown.

[0023] Figure 8 An example of template matching in the FRUC algorithm is shown.

[0024] Figure 9 An example of unidirectional motion estimation (ME) in the FRUC method is shown.

[0025] Figure 10 An example is shown of deriving the motion vector (MV) of the components of a sub-block in 4:2:0 format according to this technique.

[0026] Figure 11 Another example of MV of the components of a sub-block in 4:2:0 format is shown according to this technique.

[0027] Figure 12 Another example of deriving the MV of the components of a sub-block according to this technique is shown.

[0028] Figure 13 This is a flowchart of an exemplary method for video encoding according to one or more embodiments of the present technology.

[0029] Figure 14 This is a flowchart of another exemplary method for video encoding according to one or more embodiments of the present technology.

[0030] Figure 15 This is a block diagram illustrating an exemplary coding apparatus that can be used to implement various parts of the technology disclosed herein.

[0031] Figure 16 This is a block diagram illustrating an exemplary coding apparatus that can be used to implement various parts of the technology disclosed herein.

[0032] Figure 17 This is a block diagram illustrating an example architecture of a computer system or other control device that can be used to implement various parts of the technology disclosed herein.

[0033] Figure 18 A block diagram illustrating exemplary embodiments of mobile devices that can be used to implement various parts of the present disclosure is shown. Detailed Implementation

[0034] Video coding methods and technologies are ubiquitous in modern technology due to the increasing demand for higher resolution video. Video codecs typically consist of electronic circuitry or software that compresses or decompresses digital video and are constantly being improved to provide higher coding efficiency. A video codec converts uncompressed video into a compressed format and vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end latency (delay). Compression formats typically conform to standard video compression specifications, such as the High Efficiency Video Coding (HEVC) standard (also known as H.265 or MPEG-H Part 2), a universal video coding standard yet to be finalized, or other current and / or future video coding standards.

[0035] First, the High Efficiency Video Coding (HEVC) standard introduced sub-block-based prediction into video coding standards. Using sub-block-based prediction, blocks such as coding units (CUs) or prediction units (PUs) are divided into several non-overlapping sub-blocks. Different motion information, such as reference indices or motion vectors (MVs), can be assigned to different sub-blocks, and motion compensation (MC) can be performed individually for each sub-block. Figure 1 An example of prediction based on sub-blocks is shown.

[0036] Implementations of the disclosed technologies can be applied to existing video coding standards (e.g., HEVC, H.265) and future standards to improve runtime performance. Section headings are used in this document to improve readability and do not in any way limit the discussion or embodiments (and / or implementations) to the respective sections.

[0037] Reference software called the Joint Exploration Model (JEM) is used to explore future video coding techniques. In JEM, sub-block-based predictions are employed across several coding tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), spatial-temporal motion vector prediction (STMVP), bidirectional optical flow (BIO), frame rate upconversion (FRUC), locally adaptive motion vector resolution (LAMVR), overlapping block motion compensation (OBMC), local illumination compensation (LIC), and decoder-side motion vector refinement (DMVR).

[0038] In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). However, cameras and objects may exhibit a variety of motions, such as zooming in / out, rotation, perspective motion, and / or other unconventional motions. JEM, on the other hand, applies a simplified affine transformation motion compensation prediction. Figure 2An example of the affine motion field of block 200, described by two control point motion vectors V0 and V1, is shown. The motion vector field (MVF) of block 200 can be described by the following equation:

[0039]

[0040] Figure 3 An example of an affine MVF for each sub-block of block 300 is shown. To derive the motion vector for each M×N sub-block, the motion vector of the center sample of each sub-block can be calculated according to equation (1) and rounded to the fractional precision of the motion vector (e.g., 1 / 16 in JEM). A motion-compensated interpolation filter can then be applied to generate a prediction for each sub-block using the derived motion vector. After MCP, the high-precision motion vector of each sub-block is rounded and saved to the same precision as the normal motion vector.

[0041] In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. AF_INTER mode can be applied to CUs with both width and height greater than 8. In the bitstream, affine flags at the CU level are signaled to indicate whether AF_INTER mode is used. In AF_INTER mode, motion vector pairs {(v0,v1)|v0={v...} are constructed using adjacent blocks. A ,v B ,v c},v1={v D ,v E The candidate list of}}.

[0042] In JEM, non-Merge affine mode can only be used when the width and height of the current block are both greater than 8; Merge affine mode can only be used when the area of ​​the current block (i.e., width × height) is not less than 64.

[0043] Figure 4 An example of optional temporal motion vector prediction (ATMVP) motion prediction processing for CU 400 is shown. The ATMVP method predicts the motion vectors of sub-CUs 401 within CU 400 in two steps. The first step is to identify the corresponding block 451 in reference image 450 using temporal vectors. Reference image 450 is also referred to as the motion source image. The second step is to divide the current CU 400 into sub-CUs 401 and obtain the motion vectors from the blocks corresponding to each sub-CU, as well as the reference index for each sub-CU.

[0044] In the first step, reference image 450 and its corresponding block are determined by the motion information of spatially adjacent blocks of the current CU 400. In the second step, the corresponding block of sub-CU 451 is identified by the time vector in the motion source image 450 by adding a time vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (e.g., the smallest motion grid covering the center sample) is used to derive the motion information of the sub-CU. After identifying the motion information of the corresponding N×N blocks, it is converted into the motion vector and reference index of the current sub-CU in the same manner as the TMVP of HEVC, where motion scaling and other procedures are applied.

[0045] In the spatial-temporal motion vector prediction (STMVP) method, the motion vectors of sub-CUs are recursively derived according to the raster scan order. Figure 5 An example of a CU with four sub-blocks and adjacent blocks is shown. Consider an 8×8 CU 500, which includes four 4×4 sub-CUs A(501), B(502), C(503), and D(504). The adjacent 4×4 blocks in the current frame are labeled a(511), b(512), c(513), and d(514).

[0046] Motion derivation of sub-CU A begins by identifying its two spatial neighborhoods. The first neighborhood is the N×N block (block c 513) above sub-CU A 501. If block c (513) is unavailable or internally encoded, the other N×N blocks above sub-CU A (501) are checked (from left to right, starting from block c 513). The second neighborhood is a block to the left of sub-CU A 501 (block b 512). If block b (512) is unavailable or internally encoded, the other blocks to the left of sub-CU A 501 are checked (from top to bottom, starting from block b 512). Motion information obtained from neighboring blocks for each list is scaled to the first reference frame of the given list. Next, the temporal motion vector prediction (TMVP) of sub-block A 501 is derived following the same procedure as TMVP derivation specified in HEVC. Motion information of the juxtaposed block at block D 704 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged for each reference list. Specify the average motion vector as the motion vector of the current sub-CU.

[0047] The bidirectional optical flow (BIO) method performs per-sample motion refinement on top of block-by-block motion compensation based on bidirectional prediction. In some implementations, sample-level motion refinement does not use signaling.

[0048] Let I (k) The brightness value after block motion compensation to the reference k (k = 0, 1), and I (k)The horizontal and vertical components of the gradient. Assuming optical flow is effective, the motion vector field (v) x ,v y The following formula is given:

[0049]

[0050] Combining this optical flow equation with the Hermitian interpolation of each sample's trajectory yields a unique third-order polynomial that simultaneously matches the function value I at its end. (k) and its derivative The value of this polynomial at t=0 is the BIO prediction:

[0051]

[0052] Figure 6 An exemplary optical flow trajectory in the Bidirectional Optical Flow (BIO) method is shown. Here, τ0 and τ1 represent the distances to the reference frames. The distances τ0 and τ1 are calculated based on the POCs of Ref0 and Ref1: τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current). If both predictions originate from the same time direction (both from the past or both from the future), the signs are different (e.g., τ0·τ1<0). In this case, if the predictions do not originate from the same time point (e.g., τ0≠τ1), BIO is applied. Both reference regions have non-zero motion (e.g., MVx0, MVy0, MVx1, MVy1≠0), and the block motion vector is proportional to the time distance (e.g., MVx0 / MVx1 = MVy0 / MVy1 = -τ0 / τ1). In JEM, BIO can be applied to all bidirectional prediction blocks when the two predictions originate from different reference images. When Local Illumination Compensation (LIC) is enabled for a CU, BIO can be disabled.

[0053] In some cases, for a CU, when its Merge flag is true, signaling can be sent to the FRUC flag. When the FRUC flag is false, signaling can be sent to the Merge index and the regular Merge mode can be used. When the FRUC flag is true, signaling can be sent to another FRUC mode flag to indicate which method (e.g., bidirectional matching or template matching) will be used to derive the block's motion information.

[0054] At the encoder, the decision to use the FRUCMerge mode for the CU is based on the RD cost selection made for the normal merge candidates. For example, multiple matching modes for the CU (e.g., bidirectional matching and template matching) are examined using RD cost selection. The mode that results in the lowest cost is further compared with other CU modes. If the FRUC matching mode is the most efficient mode, then for the CU, the FRUC flag is set to true, and the associated matching mode is used.

[0055] Typically, motion derivation in the FRUC Merge model involves two steps: first, a CU-level motion search is performed, followed by sub-CU-level motion refinement. At the CU level, the initial motion vector for the entire CU is derived based on bidirectional matching or template matching. First, a candidate MV list is generated, and the candidate resulting in the lowest matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bidirectional matching or template matching is performed near the starting point. The MV result with the lowest matching cost is used as the MV value for the entire CU. Subsequently, starting with the derived CU motion vector, the motion information is further refined at the sub-CU level.

[0056] Figure 7 An example of bidirectional matching used in the Frame Rate Upconversion (FRUC) method is shown. Bidirectional matching is used to obtain motion information of the current CU (700) by finding the closest match between two blocks along the motion trajectory of the current CU (700) in two different reference images (710, 711). Under the assumption of a continuous motion trajectory, the motion vectors MV0 (701) and MV1 (702) pointing to the two reference blocks are proportional to the temporal distance between the current image and the two reference images (e.g., TD0 (703) and TD1 (704)). In some embodiments, when the current image 700 is temporarily located between the two reference images (710, 711) and the temporal distance from the current image to the two reference images is the same, the bidirectional matching becomes a mirror-based bidirectional MV.

[0057] Figure 8An example of template matching used in the Frame Rate Upconversion (FRUC) method is shown. Template matching can be used to obtain motion information of the current CU 800 by finding the closest match between a template in the current image (e.g., the top and / or left adjacent block of the current CU) and a block in the reference image 1110 (e.g., of the same size as the template). In addition to the FRUC Merge mode described above, template matching can also be applied to the AMVP mode. In both JEM and HEVC, AMVP has two candidates. New candidates can be derived using the template matching method. If a newly derived candidate by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and the list size is then set to 2 (e.g., by deleting the second existing AMVP candidate). When applied to AMVP mode, only CU-level search is applied.

[0058] The MV candidates set at the CU level may include the following: (1) the original AMVP candidate if the current CU is in AMVP mode, (2) all MERGE candidates, and (3) several MVs in the interpolated MV field (described later), as well as the top and left adjacent motion vectors.

[0059] When using bidirectional matching, each valid MV of the Merge candidate can be used as input to generate MV pairs that are assumed to be bidirectionally matched. For example, a valid MV of the Merge candidate at reference list A is (MVa, ref a Then find the reference image ref of its paired bidirectional MV in another reference list B. b , making ref a and ref b Located on a different side of the current image in time. If the reference in reference list B... b If unavailable, please refer to the ref. b Determined to be consistent with reference ref a Different references, and their temporal distance to the current image is the smallest distance in list B. Determine the reference. b Then, based on the current image and reference ref a Reference b Scaling the time distance between MVAs to export MVBs.

[0060] In some implementations, four MVs from the interpolated MV field can also be added to the CU-level candidate list. More specifically, the MVs interpolated at positions (0, 0), (W / 2, 0), (0, H / 2), and (W / 2, H / 2) of the current CU are added. When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set. In some implementations, at the CU level, 15 MVs from the AMVP CU and 13 MVs from the merged CU can be added to the candidate list.

[0061] The MV candidates set at the sub-CU level include (2) the MVs determined from the CU level search, (3) the top, left, upper left, and upper right adjacent MVs, (4) scaled versions of the MVs juxtaposed in the reference image, (5) one or more ATMVP candidates (e.g., up to four) and / or (6) one or more STMVP candidates (e.g., up to four). The scaled MVs from the reference images are derived as follows. The reference images in both lists are traversed. The MVs at the juxtaposed positions of the sub-CUs in the reference images are scaled to the reference of the starting CU level MVs. The ATMVP and STMVP candidates can be the first four. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

[0062] Before encoding the frames, an interpolated motion field for the entire image is generated based on a unidirectional ME. This motion field can then be used as a MV candidate at the CU level or subCU level.

[0063] In some embodiments, the motion field of each reference image in the two reference lists is traversed at a 4×4 block level. Figure 9 An example of unidirectional motion estimation (ME) 900 in the FRUC method is shown. For each 4×4 block, if the motion associated with the block passes through 4×4 blocks in the current image, and the block has not been assigned any interpolated motion, the motion of the reference block is scaled to the current image based on temporal distances TD0 and TD1 (similar to MV scaling in TMVP in HEVC), and this scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4×4 block, the block's motion is marked as unavailable in the interpolated motion field.

[0064] Motion-compensated interpolation is required when the motion vector points to the fractional sampling position. To reduce complexity, bilinear interpolation is used instead of the conventional 8-tap HEVC interpolation for both bidirectional matching and template matching.

[0065] The calculation of matching cost differs slightly at different steps. When selecting candidates from the candidate set at the CU level, the matching cost can be the absolute sum-difference (SAD) of bidirectional matching or template matching. After determining the starting MV, the matching cost C for bidirectional matching in the sub-CU level search is calculated as follows:

[0066]

[0067] Here, w is the weighting coefficient. In some embodiments, w can be empirically set to 4. MV and MV s These indicate the current MV and the starting MV, respectively. SAD can still be used as the matching cost for pattern matching searches at the sub-CU level.

[0068] In FRUC mode, the motion signature (MV) is derived using only luma samples. The derived motion is used for both luma and chroma for inter-frame prediction in motion control (MC). After the MV is determined, the final MC is performed using an 8-taps interpolation filter for luma and a 4-taps interpolation filter for chroma.

[0069] MV refinement is a pattern-based MV search, using bidirectional matching cost or template matching cost as the criterion. JEM supports two search modes—Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search—for MV refinement at the CU level and sub-CU level, respectively. For both CU and sub-CU level MV refinement, MV is directly searched at 1 / 4 luminance sample MV precision, followed by 1 / 8 luminance sample MV refinement. The search range for MV refinement in the CU and sub-CU steps is set to 8 luminance samples.

[0070] In the bidirectional matching Merge mode, bidirectional prediction is applied because the motion information of the CU is derived based on the nearest match between two blocks along the current CU motion trajectory in two different reference images. In the template matching Merge mode, the encoder can choose for the CU from unidirectional predictions of list 0, unidirectional predictions of list 1, or bidirectional predictions. This selection can be based on the template matching cost as follows:

[0071] If costBi <= factor * min(cost0, cost1)

[0072] Then use bidirectional prediction;

[0073] Otherwise, if cost0 <= cost1

[0074] Then use the one-way prediction from list 0;

[0075] otherwise,

[0076] Use the one-way prediction from List 1;

[0077] Here, cost0 is the SAD of template matching in list 0, cost1 is the SAD of template matching in list 2, and costBi is the SAD of bidirectional template matching. For example, when the factor value is equal to 1.25, it means that the selection process is shifted towards bidirectional prediction. Inter-frame prediction direction selection can be applied to CU-level template matching processing.

[0078] The human visual system is less sensitive to the position and motion of colors than to luminance. Therefore, bandwidth can be optimized by storing more luminance detail than color detail. In video systems, this is achieved using chromatic aberration components. The signal is split into a luminance (Y') component and two chromatic aberration (chroma) components. Chromaticity subsampling is the practice of encoding an image with chroma information at a lower resolution than luminance information, taking advantage of the human visual system's lower sensitivity to chromatic aberration than to luminance. Common types of subsampling include 4:2:2 (both chroma components are sampled at half the luminance sampling rate), 4:1:1 (horizontal color resolution is one-quarter), and 4:2:0 (vertical resolution is halved compared to 4:1:1 because the Cb and Cr channels are sampled only on each spare line). In the example, the HEVC standard defines how to derive the MV (denoted as mv) for MC in the chroma component from the MV (denoted as mv) for MC in the luminance component. Generally, mvC is calculated as mv multiplied by a factor that depends on the color format, such as 4:2:0 or 4:2:2.

[0079] In HEVC Screen Content Coding Extension (SCC), intra-frame block copying (IBC or intra-frame picture block compensation), also known as current picture reference (CPR), is employed. This tool is highly effective for encoding screen content video because recurring patterns in rich text and graphic content often appear within the same picture. Using previously reconstructed blocks with the same or similar patterns as predictors can effectively reduce prediction errors and thus improve coding efficiency.

[0080] Similar to the CRP design in HEVC SCC, in VVC, signaling for the use of IBC mode is conducted at both the sequence and picture levels. When IBC mode is enabled at the Sequence Parameter Set (SPS), it can also be enabled at the picture level. When IBC mode is enabled at the picture level, the currently reconstructed picture is treated as the reference picture. Therefore, no block-level syntax changes are required on top of existing VVC inter-frame modes to signal for the use of IBC mode.

[0081] The characteristics of the IBC model include the following:

[0082] This is treated as a normal inter-frame mode. Therefore, IBC mode can also use Merge and Skip modes. The Merge candidate list is constructed uniformly, containing Merge candidates from adjacent positions, which are encoded in either IBC mode or HEVC inter-frame mode. Depending on the selected Merge index, the current block in Merge or Skip mode can be merged into a neighborhood encoded in IBC mode, or alternatively merged into a neighborhood encoded in normal inter-frame mode, using a different picture as a reference picture.

[0083] ○ The block vector prediction and coding scheme used in IBC mode reuses the scheme (AMVP and MVD coding) used in HEVC inter-frame mode.

[0084] The motion vectors used in IBC mode (also known as block vectors) are encoded with integer pixel precision, but stored in memory with 1 / 16 pixel precision after decoding, because quarter-pixel precision is required in the interpolation and deblocking stages. When used for motion vector prediction in IBC mode, the stored vector predictor is shifted 4 pixels to the right.

[0085] ○ Search scope: It is limited to the current CTU.

[0086] ○ CPR is not allowed when affine mode / triangular mode / GBI / weighted prediction is enabled.

[0087] In some cases, pairwise averaged candidates are generated by averaging predetermined candidate pairs in the current Merge candidate list, and the predefined pairs are defined as {(0,1), (0,2), (1,2), (0,3), (1,3), (2,3)}, where the numbers represent the Merge indices of the Merge candidate list. The average motion vector is calculated separately for each reference list. If two motion vectors are available in a list, they are averaged even if they point to different reference images; if only one motion vector is available, it is used directly; if no motion vector is available, the list is left invalid. Pairwise averaged candidates replace the combined candidates in the HEVC standard. Assume the MV of two Merge candidates is MV0 = (MV0 + ... 0x MV 0y ) and MV1 = (MV 1x MV 1y If ), then it is represented as MV*=(MV* x MV* y The MVs of the pairwise Merge candidates are derived as

[0088] MV* x =(MV 0x +MV 1x ) / 2, and

[0089] MV*y=(MV 0y +MV 1y ) / 2.

[0090] Additionally, when MV0 and MV1 refer to the current image (i.e., CPR mode), MV*x and MV*y are further rounded to remove portions with higher precision than full pixels to ensure that integer MV is obtained:

[0091] MV* x =(MV*) x / 16)<<4, and

[0092] [MV* y =(MV*) y / 16)<<4.

[0093] It should be noted that for each pair, if one of them is CPR-coded while the other is not, such a pair is not allowed to generate a pairwise average candidate.

[0094] The concept of Triangulation Prediction Mode (TPM) is to introduce a new triangulation for motion-compensated prediction. It divides the Cubic Queues (CU) into two triangulation prediction units along either the diagonal or anti-diagonal direction. Inter-frame prediction is performed on each triangulation prediction unit in the CU using its own unidirectional prediction motion vector derived from a single unidirectional prediction candidate list and a reference frame index. After predicting the triangulation prediction units, adaptive weighting is performed on the diagonal edges. Then, transform and quantization are applied to the entire CU. It should be noted that this mode only applies to Merge mode (note: skip mode is considered a special Merge mode). The TPM unidirectional prediction candidate list is also shown.

[0095] The unidirectional prediction candidate list, known as the TPM motion candidate list, consists of five unidirectional prediction motion vector candidates. It is derived from seven neighboring blocks, including five spatially adjacent blocks and two temporally co-located blocks. Motion vectors from the seven neighboring blocks are collected and added to the unidirectional prediction candidate list in the following order: unidirectional prediction motion vector, L0 motion vector of bidirectional prediction motion vector, L1 motion vector of bidirectional prediction motion vector, and the average motion vector of the L0 and L1 motion vectors of bidirectional prediction motion vector. If the number of candidates is less than five, a zero motion vector is added to the list. Motion candidates added to this list for TPM are called TPM candidates, and motion information derived from the spatial / temporal blocks is called regular motion candidates.

[0096] More specifically, the following steps are involved:

[0097] (1) Obtain regular motion candidates from A1, B1, B0, A0, B2, Col and Col2 (similar to those in the regular Merge mode) without performing any pruning operations.

[0098] (2) Set the variable numCurrMergeCand = 0

[0099] (3) For each regular motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 and numCurrMergeCand is less than 5, if the regular motion candidate is a one-way prediction (from list 0 or list 1), add it directly to the Merge list as a TPM candidate, and increment numCurrMergeCand by 1. Such a TPM candidate is called the "original one-way prediction candidate".

[0100] Apply full pruning

[0101] (4) For each motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 and numCurrMergeCand is less than 5, if the regular motion candidate is a bidirectional prediction, the motion information from list 0 is added as a new TPM candidate to the TPM Merge list (i.e., modified to a unidirectional prediction from list 0) and numCurrMergeCand is incremented by 1. Such a TPM candidate is called a "truncated list 0 prediction candidate".

[0102] Apply full pruning.

[0103] (5) For each motion candidate derived from A1, B1, B0, A0, B2, Col, and Col2 and numCurrMergeCand is less than 5, if the regular motion candidate is a bidirectional prediction, the motion information from List 1 is added to the TPM Merge list (i.e., modified to a unidirectional prediction from List 1) and numCurrMergeCand is incremented by 1. Such a TPM candidate is called a "truncated List 1 prediction candidate".

[0104] Apply full pruning.

[0105] (6) For each motion candidate derived from A1, B1, B0, A0, B2, Col and Col2 and numCurrMergeCand is less than 5, if the regular motion candidate is a bidirectional prediction, the motion information of list 1 is first scaled to the reference image of list 0, and the average of the two MVs (one from the original list 0 and the other is the scaled MV from list 1) is added to the TPM Merge list. Such a candidate is called the average unidirectional prediction from the motion candidate of list 0 and numCurrMergeCand is increased by 1.

[0106] Apply full pruning.

[0107] (7) If numCurrMergeCand is less than 5, add a zero motion vector candidate.

[0108] When a candidate is inserted into a list, if it must be compared with all previously added candidates to see if it is the same as one of them, such a process is called full pruning.

[0109] Assuming (MV) 1'x MV 1'y ) indicates scaling the MV and is determined by (MV 0x MV 0y ) represents the list 0MV. From (MV* x MV* y The average one-way prediction of the list of motion candidates represented by ) is defined as:

[0110] MV* x =(MV 0x +MV 1'x +1)>>1, and

[0111] MV* y =(MV 0y +MV 1'y +1)>>1.

[0112] In some existing implementations, such as affine prediction in JEM, the MV of each sub-block is calculated individually for each component using an affine model as shown in Equation (1), which can lead to misalignment of motion vectors between the luma and chroma components. Multiple calculations of components also result in coding inefficiency. In some other existing implementations, the motion vector averaging operation used to derive the Merge candidate / average unidirectional prediction from list 0 motion candidates needs to be aligned with the rounding method used in sub-block prediction. Therefore, a unified design is desired.

[0113] This patent document discloses a technique that can be implemented in various embodiments to calculate the MV of a sub-block of one component (e.g., chroma component) from the MV of one or more sub-blocks of another component (e.g., luma component). The disclosed technique eliminates the need to determine the MV of different components multiple times, thereby improving video coding efficiency. The disclosed technique also introduces a unified design regarding averaging operations and rounding methods.

[0114] The use of such techniques is illustrated in the following embodiments described for various implementations. In the following examples, which should not be construed as limiting, Shift(x,s) is defined as Shift(x,s) = (x+off) >> s, and

[0115] SignShift(x, s) is defined as:

[0116]

[0117] Here, off is an integer, for example, 0 or 1 << (s-1).

[0118] In some embodiments, the motion MV of a sub-block of one component can be derived based on the motion MV of one or more sub-blocks of another component. Here, the motion MV of one or more sub-blocks of the other component has already been derived using an affine model. This eliminates the need to derive motion information multiple times.

[0119] In some embodiments, the HEVC standard defines how to derive the MV (denoted as mv) in the chromaticity component from the MV (denoted as mv) used for the MC in the luminance component. Generally, mvC is calculated as mv multiplied by a factor that depends on the color format, such as 4:2:0 or 4:2:2. Figure 10 An example of deriving the MV of a sub-block in 4:2:0 format according to this technique is shown. In this example, the block size is 16×16 for Y (luminance component) and 8×8 for Cb / Cr (chrominance component). The sub-block size for the luminance component is 4×4, while the sub-block size for the chrominance component is 2×2. First, MV* is derived for the 4×4 sub-blocks in the Y component. The MV of the 2×2 sub-blocks in the Cb and / or Cr components can be derived based on MV*. In this specific example, the value of MV* is first copied, and the MV of the 2×2 sub-blocks in the Cb and / or Cr components is calculated based on MV* according to the HEVC standard.

[0120] Figure 11 Another example of deriving the MV of the components of a 4:2:0 sub-block according to this technique is shown. In this example, the block size of the luma component is 16×16, and the block size of the chroma component is 8×8. The sub-block size of all components is 4×4. The 4×4 sub-blocks in the Cb or Cr component correspond to four 4×4 sub-blocks in the Y component. First, the MV* of the luma component is calculated. Then, the MV of the 4×4 sub-blocks in the Cb or Cr component is derived based on the MV* according to the HEVC standard.

[0121] In some embodiments, MV* is calculated as the average of all corresponding sub-blocks MV in the Y component: MV* = (MV0 + MV1 + MV2 + MV3) / 4. Assume MV* = (MV* x MV* y ), MV0 = (MV 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ).

[0122] In some embodiments, MV* x =Shift(MV 0x +MV 1x +MV 2x +MV 3x ,2),MV* y =Shift(MV 0y +MV 1y +MV 2y +MV 3y , 2). In some embodiments, MV* x =SignShift(MV 0x +MV 1x +MV 2x +MV 3x ,2),MV* y =SignShift(MV 0y +MV 1y +MV 2y +MV 3y ,2).

[0123] In some embodiments, the following operations can be used to perform the calculation of MV*:

[0124] 1.a MV' x =Shift(MV 0x +MV 1x ,1),

[0125] 1.b MV' y =Shift(MV 0y +MV 1y ,1),

[0126] 1.c MV” x =Shift(MV 2x +MV 3x ,1),

[0127] 1.d MV” y =Shift(MV 2y +MV 3y ,1),

[0128] 1.e MV* x =Shift(MV') x +MV” x ,1), and

[0129] 1.f MV* y =Shift(MV') y +MV” y ,1).

[0130] In some embodiments, the following operations can be used to perform the calculation of MV*:

[0131] 2.a MV' x =Shift(MV 0x +MV 2x ,1),

[0132] 2.b MV' y =Shift(MV 0y +MV 2y ,1),

[0133] 2.c MV” x =Shift(MV 1x +MV 3x ,1),

[0134] 2.d MV” y =Shift(MV 1y +MV 3y ,1),

[0135] 2.e MV* x =Shift(MV') x +MV” x ,1), and

[0136] 2.f MV* y =Shift(MV') y +MV” y ,1).

[0137] In some embodiments, the following operations can be used to perform the calculation of MV*:

[0138] 3.a MV' x =SignShift(MV 0x +MV 1x ,1),

[0139] 3.b MV' y =SignShift(MV 0y +MV 1y ,1),

[0140] 3.c MV” x =SignShift(MV 2x +MV 3x ,1),

[0141] 3.d MV” y =SignShift(MV 2y +MV 3y,1),

[0142] 3.e MV* x =SignShift(MV' x +MV” x ,1), and

[0143] 3.f MV* y =SignShift(MV' y +MV” y ,1).

[0144] In some embodiments, the following operations can be used to perform the calculation of MV*:

[0145] 4.a MV' x =SignShift(MV 0x +MV 2x ,1),

[0146] 4.b MV' y =SignShift(MV 0y +MV 2y ,1),

[0147] 4.c MV” x =SignShift(MV 1x +MV 3x ,1),

[0148] 4.d MV” y =SignShift(MV 1y +MV 3y ,1),

[0149] 4.e MV* x =SignShift(MV' x +MV” x ,1), and

[0150] 4.f MV* y =SignShift(MV'y+MV) y ,1).

[0151] In some embodiments, the MV is based on the top-left sub-block in the Y component (e.g., as shown in the figure). Figure 11 MV* is calculated using MV0 (as shown). In some embodiments, MV* is calculated based on the MV of the central sub-block in the Y component. In some embodiments, MV* is calculated based on the median MV of all corresponding sub-blocks in the Y component. Figure 11 In the specific example shown, MV* = median (MV0, MV1, MV2, MV3).

[0152] In some embodiments, the MV of a sub-block of a color component can be derived based on the color format (e.g., 4:2:0, 4:2:2, and / or 4:4:4). For example, no subsampling occurs when the color format is 4:4:4. The sub-block size and the number of sub-blocks in a video data block are the same for all components. The MV of a sub-block in one component is the same as the MV of the corresponding sub-block in another component.

[0153] As another example, when the color format is 4:2:2, the sub-block size can be the same for all components, while the number of blocks for different components can be different. First, the MV* of the luminance component is calculated based on the MV of several corresponding sub-blocks. Then, the MV of the corresponding sub-blocks in the Cb or Cr component is derived from the MV*.

[0154] Figure 12 Another example of deriving the MV of the components for sub-blocks according to this technique is shown. In this example, the block size for the luma component is 16×16, and the block size for the chroma component is 8×16. The sub-block size for all components is 4×4. The 4×4 sub-blocks in the Cb or Cr component correspond to two 4×4 sub-blocks in the Y component. First, the MV* of the luma component is calculated. Then, the MV of the 4×4 sub-blocks in the Cb or Cr component is derived based on the MV* according to the HEVC standard.

[0155] In some embodiments, MV* is calculated as the average of all corresponding sub-blocks MV in the Y component: MV* = (MV0 + MV1) / 2. Assume MV* = (MV* x MV* y ), MV0 = (MV 0x MV 0y ), MV1 = (MV 1x MV 1y ),

[0156] In some embodiments, MV* x =Shift(MV 0x +MV 1x ,1),MV* y =Shift(MV 0y +MV 1y , 1). In some embodiments, MV* x =SignShift(MV 0x +MV 1x ,1),MV* y =SignShift(MV 0y +MV 1y ,1).

[0157] Figure 13This is a flowchart of an exemplary method 1300 for video encoding according to one or more embodiments of the present technology. Method 1300 includes, at operation 1310, segmenting a first component of a current video block into a first group of sub-blocks. Method 1300 includes, at operation 1320, segmenting a second component of the current video block into a second group of sub-blocks. The sub-blocks of the second component correspond to one or more sub-blocks of the first component. Method 1300 further includes, at operation 1330, deriving motion vectors of the sub-blocks of the second component based on motion vectors of one or more corresponding sub-blocks of the first color component, based on the color format of the current video block.

[0158] Figure 14 This is a flowchart of another exemplary method 1400 for video encoding according to one or more embodiments of the present technology. Method 1400 includes, at operation 1410, receiving a video data block comprising a first component and at least a second component. The first component is segmented into a first set of sub-blocks and the second component is segmented into a second set of sub-blocks. The sub-blocks of the second component correspond to one or more sub-blocks of the first component. Method 1400 includes, at operation 1420, deriving motion vectors for the sub-blocks of the second component based on the motion vectors of one or more corresponding sub-blocks of the first color component, based on the color format of the video data block. Method 1400 includes, at operation 1430, reconstructing the video data block or decoding other video data blocks in the same frame based on the derived motion vectors.

[0159] In some embodiments, a sub-block of the second component corresponds to a sub-block of the first component. For example, the two sub-blocks may spatially coexist in an image. Method 1300 or 1400 further includes calculating an intermediate motion vector set by copying the motion vector of the sub-block of the first component and applying a factor to the intermediate motion vector set to obtain a derived motion vector, wherein the factor is associated with a color format.

[0160] In some embodiments, the color format is not 4:4:4 (e.g., it is 4:2:2, or 4:2:0), and the size of at least one of the first group of sub-blocks is different from the size of at least one of the second group of sub-blocks. In some embodiments, the color format is 4:4:4, and the size of at least one of the first group of sub-blocks is the same as the size of at least one of the second group of sub-blocks.

[0161] In some embodiments, the color format is not 4:4:4, and at least one of the first group of sub-blocks has the same size as at least one of the second group of sub-blocks. The sub-blocks of the second component correspond to multiple sub-blocks of the first component. In some embodiments, at least one of the motion vectors for the sub-blocks of the second component is derived based on the average of the corresponding motion vectors of the multiple sub-blocks of the first component.

[0162] In some implementations, the sub-blocks of the second component correspond to the four sub-blocks of the first component. Each of the four sub-blocks has a motion vector MV0 = (MV... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y Method 1300 or 1400 includes calculating the intermediate motion vector set MV* = (MV*x, MV*y) as MV. * x=Shift(MV0x+MV1x+MV2x+MV3x,2) and MV * y = Shift(MVy0 + MVy1 + MVy2 + MVy3, 2), where Shift(x, s) = (x + off) >> s, where off and s are integers, and >> denotes a right shift operation. Method 1300 or 1400 also includes applying a factor to an intermediate set of motion vectors to obtain a derived motion vector, where the factor is associated with a color format.

[0163] In some implementations, the sub-blocks of the second component correspond to the four sub-blocks of the first component. Each of the four sub-blocks has a motion vector MV0 = (MV... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y Method 1300 or 1400 includes calculating the intermediate motion vector set MV* = (MV*x, MV*y) as MV. * x=SignShift(MV0x+MV1x+MV2x+MV3x,2) and MV * y = SignShift(MV0y + MV1y + MV2y + MV3y, 2), where Where off and s are integers, and >> denotes a shift operation. Method 1300 or 1400 also includes applying a factor to an intermediate set of motion vectors to obtain a derived motion vector, wherein the factor is associated with a color format.

[0164] In some implementations, the sub-blocks of the second component correspond to the four sub-blocks of the first component. Each of the four sub-blocks has a motion vector MV0 = (MV... 0x MV 0y ), MV1 = (MV 1x MV1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y Method 1300 or 1400 includes calculating the intermediate motion vector set MV* = (MV*x, MV*y) as MV. * x=Shift(MV′x+MV″x,1) and MV * y = Shift(MV′y + MV″y, 1), where MV′x = Shift(MV0x + MV1x, 1), MV′y = Shift(MV0y + MV1y, 1), MV″x = Shift(MV2x + MV3x, 1), and MV″y = Shift(MV2y + MV3y, 1), where Shift(x, s) = (x + off) >> s, where off and s are integers, and >> denotes a right shift operation. Method 1300 or 1400 also includes applying a factor to an intermediate set of motion vectors to obtain a derived motion vector, where the factor is associated with a color format.

[0165] In some implementations, the sub-blocks of the second component correspond to the four sub-blocks of the first component. Each of the four sub-blocks has a motion vector MV0 = (MV... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y Method 1300 or 1400 includes calculating the intermediate motion vector set MV* = (MV*x, MV*y) as MV. * x=Shift(MV′x+MV″x,1) and MV * y = Shift(MV′y + MV″y, 1), where MV′x = Shift(MV0x + MV2x, 1), MV′y = Shift(MV0y + MV2y, 1), MV″x = Shift(MV1x + MV3x, 1), and MV″y = Shift(MV1y + MV3y, 1), where Shift(x, s) = (x + off) >> s, where off and s are integers, and >> denotes a right shift operation. Method 1300 or 1400 also includes applying a factor to an intermediate set of motion vectors to obtain a derived motion vector, where the factor is associated with a color format.

[0166] In some implementations, the sub-blocks of the second component correspond to the four sub-blocks of the first component. Each of the four sub-blocks has a motion vector MV0 = (MV...0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y Method 1300 or 1400 includes calculating the intermediate motion vector set MV* = (MV*x, MV*y) as MV. * x=SignShift(MV′x+MV″x,1) and MV * y=SignShift(MV′y+MV″y,1), where MV′x=SignShift(MV0x+MV1x,1), MV′y=SignShift(MV0y+MV1y,1), MV″x=SignShift(MV2x+MV3x,1), and MV″y=SignShift(MV2y+MV3y,1), where Where off and s are integers, and >> denotes a shift operation. Method 1300 or 1400 also includes applying a factor to an intermediate set of motion vectors to obtain a derived motion vector, wherein the factor is associated with a color format.

[0167] In some implementations, the sub-blocks of the second component correspond to the four sub-blocks of the first component. Each of the four sub-blocks has a motion vector MV0 = (MV... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y Method 1300 or 1400 includes calculating the intermediate motion vector set MV* = (MV*x, MV*y) as MV. * x=SignShift(MV′x+MV″x,1) and MV * y=SignShift(MV′y+MV″y,1), where MV′x=SignShift(MV0x+MV2x,1), MV′y=SignShift(MV0y+MV2y,1), MV″x=SignShift(MV1x+MV3x,1), and MV″y=SignShift(MV1y+MV3y,1), where Where off and s are integers, and >> denotes a shift operation. Method 1300 or 1400 also includes applying a factor to an intermediate set of motion vectors to obtain a derived motion vector, wherein the factor is associated with a color format.

[0168] In some implementations, the sub-blocks of the second component correspond to the two sub-blocks of the first component. The two sub-blocks each have a motion vector MV0 = (MV 0x MV 0y ) and MV1 = (MV 1x MV 1y Method 1300 or 1400 includes calculating the intermediate motion vector set MV* = (MV*x, MV*y) as MV. * x = Shift(MV0x + MV1x, 1) and MV * y = Shift(MVy0 + MVy1, 1), where Shift(x, s) = (x + off) >> s, where off and s are integers, and >> denotes a right shift operation. Method 1300 or 1400 also includes applying a factor to an intermediate set of motion vectors to obtain a derived motion vector, where the factor is associated with a color format.

[0169] In some implementations, the sub-blocks of the second component correspond to the two sub-blocks of the first component. The two sub-blocks each have a motion vector MV0 = (MV 0x MV 0y ) and MV1 = (MV 1x MV 1y ).

[0170] Method 1300 or 1400 includes calculating the intermediate motion vector set MV* = (MV*x, MV*y) as MV * x = SignShift(MV0x + MV1x, 1) and MV * y = Sign(MVy0 + MVv1, 1), where

[0171]

[0172] Where off and s are integers, and >> denotes a shift operation. Method 1300 or 1400 also includes applying a factor to an intermediate set of motion vectors to obtain a derived motion vector, wherein the factor is associated with a color format.

[0173] In some embodiments, method 1300 or 1400 further includes calculating an intermediate set of motion vectors based on the motion vectors of a selected sub-block among a plurality of sub-blocks of the first component, and applying a factor to the intermediate set of motion vectors to obtain a derived motion vector. The factor is associated with a color format. In some implementations, the selected sub-block is the top-left sub-block of the plurality of sub-blocks of the first component. In some implementations, the selected sub-block is the center sub-block of the plurality of sub-blocks of the first component.

[0174] In some embodiments, method 1300 or 1400 includes calculating an intermediate set of motion vectors based on the median of motion vectors of a plurality of sub-blocks of a first component, and applying a factor to the intermediate set of motion vectors to obtain a derived motion vector, wherein the factor is associated with a color format.

[0175] In some embodiments, the application of factors is specified in the High Efficiency Video Coding (HEVC) standard.

[0176] The examples described above can be considered in the context of the methods described below (e.g., methods 1300 and 1400), which can be implemented at the video decoder and / or video encoder.

[0177] Figure 15 This is a block diagram illustrating an exemplary encoding apparatus 1500 that can be used to implement various parts of the technology disclosed herein, including (but not limited to) methods 1300 and 1400. The encoding apparatus 1500 includes a quantizer 1505 for compressing input data bits. The encoding apparatus 1500 also includes a dequantizer 1515, allowing data bits to be fed to a memory 1525 and a predictor 1520 to perform motion estimation. The encoding apparatus 1500 also includes a binary encoder 1530 to generate encoded binary code.

[0178] Figure 16 This is a block diagram illustrating an exemplary encoding apparatus 1600 that can be used to implement various parts of the technology disclosed herein, including (but not limited to) methods 1300 and 1400. The decoding apparatus 1600 includes a binary decoder 1605 for decoding binary code. The decoding apparatus 1600 also includes a dequantizer 1615, such that decoded data bits can be fed to a memory 1625 and a predictor 1620 to perform motion estimation at the decoding end.

[0179] Figure 17 This is a schematic diagram illustrating an example of the architecture of a computer system or other control device 1700 that can be used to implement various parts of the technology disclosed herein. Figure 17In this embodiment, computer system 1700 includes one or more processors 1705 and memory 1710 connected via interconnect 1725. Interconnect 1725 can represent any one or more individual physical buses, point-to-point connections, or both connected by appropriate bridges, adapters, or controllers. Therefore, interconnect 1725 can include, for example, a system bus, a peripheral component interconnect (PCI) bus, an HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an IEEE Standard 674 bus (sometimes referred to as "FireWire").

[0180] Processor 1705 may include a central processing unit (CPU) to control the overall operation of, for example, a host computer. In some embodiments, processor 1705 achieves this by executing software or firmware stored in memory 1710. Processor 1705 may be or may include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof.

[0181] Memory 1710 may be or include the main memory of a computer system. Memory 1710 represents any suitable form of random access memory (RAM), read-only memory (ROM), flash memory, etc., or a combination of these devices. In use, memory 1710 may, among other things, contain a set of machine instructions that, when executed by processor 1705, cause processor 1705 to perform operations to implement embodiments of the present disclosure.

[0182] Connected to the processor 1705 via interconnect 1725 is also (optionally) a network adapter 1715. The network adapter 1715 provides the computer system 1700 with the ability to communicate with remote devices, such as storage clients and / or other storage servers, and may be, for example, an Ethernet adapter or a Fibre Channel adapter.

[0183] Figure 18A block diagram illustrating an exemplary embodiment of a mobile device 1800 that can be used to implement various parts of the technology disclosed herein, the technology including (but not limited to) method 1600. The mobile device 1800 may be a laptop computer, smartphone, tablet computer, camera, or other device capable of processing video. The mobile device 1800 includes a processor or controller 1801 for processing data, and a memory 1802 that communicates with the processor 1801 to store and / or buffer data. For example, the processor 1801 may include a central processing unit (CPU) or a microcontroller unit (MCU). In some implementations, the processor 1801 may include a field-programmable gate array (FPGA). In some implementations, the mobile device 1800 includes or communicates with a graphics processing unit (GPU), a video processing unit (VPU), and / or a wireless communication unit to implement various visual and / or communication data processing functions of a smartphone device. For example, memory 1802 may include and store processor-executable code that, when executed by processor 1801, configures mobile device 1800 to perform various operations, such as receiving information, commands and / or data, processing information and data, and sending or providing processed information / data to another device, such as an actuator or external display.

[0184] To support the various functions of the mobile device 1800, the memory 1802 can store information and data, such as instructions, software, values, images, and other data processed or referenced by the processor 1801. For example, the storage function of the memory 1802 can be implemented using various types of random access memory (RAM) devices, read-only memory (ROM) devices, flash memory devices, and other suitable storage media. In some implementations, the mobile device 1800 includes an input / output (I / O) unit 1803 to interface the processor 1801 and / or the memory 1802 with other modules, units, or devices. For example, the I / O unit 1803 can interface with the processor 1801 and the memory 1802 to utilize various types of wireless interfaces compatible with typical data communication standards, such as between one or more computers and user equipment in the cloud. In some implementations, the mobile device 1800 can interface with other devices via a wired connection using the I / O unit 1803. The mobile device 1800 can also interface with other external interfaces (e.g., data storage) and / or visual or audio display devices 1804 to retrieve and transmit data and information that can be processed by a processor, stored in memory, or displayed on the output unit of the display device 1804 or an external device. For example, the display device 1804 can display video frames comprising blocks (CU, PU, ​​or TU) that apply intra-frame block duplication based on whether a motion compensation algorithm is used and by encoding the blocks according to disclosed techniques.

[0185] As will be understood from the foregoing, specific embodiments of the technology disclosed herein have been described for ease of explanation, but various modifications may be made without departing from the scope of the invention. Therefore, the technology disclosed herein is not limited except for the appended claims.

[0186] The subject matter and functional operations described in this patent document can be implemented in various systems, digital electronic circuits, or computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. The subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer-readable medium for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, a material composition affecting machine-readable propagation signals, or combinations thereof. The term "data processing apparatus" encompasses all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof.

[0187] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to said program, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on one or more computers located at a single site or distributed across multiple sites interconnected by a communication network.

[0188] The processing and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processing and logic flows can also be executed by special-purpose logic circuitry, and the device can be implemented as special-purpose logic circuitry, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0189] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or devices operatively coupled to one or more mass storage devices to receive data from or transfer data to, or both. However, a computer does not necessarily have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices. The processor and memory may be supplemented by or incorporated into special-purpose logic circuitry.

[0190] This specification and accompanying drawings are intended to be illustrative only, where illustrative means example. As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” should also include the plural forms. Furthermore, the use of “or” is intended to include “and / or” unless the context clearly indicates otherwise.

[0191] While this patent document contains numerous details, it should not be construed as limiting the scope of any invention or claim, but rather as a description of features of specific embodiments of a particular invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple embodiments, or in any suitable sub-combination. Moreover, although the foregoing features may be described as functioning in certain combinations, or even initially claimed to be so, in some cases one or more features from a combination of claims may be removed, and combinations of claims may involve sub-combinations or variations thereof.

[0192] Similarly, although the operations are described in a specific order in the figures, this should not be construed as requiring the operations to be performed in the specific order shown or in a sequential manner to obtain the desired result, or as requiring all of the operations shown to be performed. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0193] Only some implementation methods and examples are described. Other implementation methods, enhancements and variations can be made based on the content described and shown in this patent document.

Claims

1. A method for video decoding, comprising: Receive video data blocks comprising a luminance component and at least one chrominance component; The motion vector of the control point of the luminance component of the video data block is determined based on the affine pattern; The luminance component is divided into a first group of sub-blocks, wherein each sub-block in the first group of sub-blocks has a second size; The chromaticity component of the at least one chromaticity component is divided into a second group of sub-blocks, wherein each sub-block in the second group of sub-blocks has a first size; The motion vector of each sub-block in the first group of sub-blocks is determined based on the motion vector of the control point; Based on the motion vectors of multiple sub-blocks in the first group of sub-blocks and the color format of the video data blocks, the motion vector of each sub-block in the second group of sub-blocks is derived; and The video data block can be reconstructed based on the derived motion vectors, or other video data blocks in the same image can be decoded. Where the color format applied to the luminance component and the chrominance component is 4:2:0 or 4:2:2, the second set of sub-blocks having the first size equal to the second size is used to derive the predicted sample points of the chrominance component. The method further includes: Calculate an intermediate motion vector set based on the motion vectors of selected sub-blocks from among multiple sub-blocks of the brightness component; and A factor is applied to the intermediate motion vector set to obtain motion vectors for sub-blocks of the chromaticity components, wherein the factor is associated with the color format.

2. The method of claim 1, wherein, in response to the color format being 4:4:4, at least one of the first group of sub-blocks has the same size as at least one of the second group of sub-blocks.

3. The method according to claim 1, wherein the size of at least one of the first group of sub-blocks is 4×4 and the size of at least one of the second group of sub-blocks is 4×4.

4. The method according to claim 1, wherein the second group of sub-blocks obtained by segmenting the chromaticity component corresponds to the four sub-blocks of the luminance component, and the motion vector of one of the sub-blocks in the second group of sub-blocks is obtained based on the motion vectors of two of the four sub-blocks of the luminance component.

5. The method of claim 1, wherein at least one of the motion vectors of the sub-blocks of the chromaticity component is obtained based on the offset-based average of the corresponding motion vectors of the plurality of sub-blocks of the luminance component.

6. The method according to claim 5, wherein the sub-blocks of the chromaticity component correspond to four sub-blocks of the luminance component, and the four sub-blocks each have a motion vector MV0 = (MV0 = ... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), And among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in ,in , where off and s are integers, and >> represents a right shift operation.

7. The method according to claim 5, wherein the sub-blocks of the chromaticity component correspond to four sub-blocks of the luminance component, and the four sub-blocks each have a motion vector MV0 = (MV0 = ... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in , ,in , where off and s are integers, and >> represents a shift operation.

8. The method according to claim 5, wherein the sub-blocks of the chromaticity component correspond to four sub-blocks of the luminance component, and the four sub-blocks each have a motion vector MV0 = (MV0 = ... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in, ,in , , ,and ,in , where off and s are integers, and >> represents a right shift operation.

9. The method according to claim 5, wherein the sub-blocks of the chromaticity component correspond to four sub-blocks of the luminance component, and the four sub-blocks each have a motion vector MV0 = (MV0 = ... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ) in ,in , , ,and ,in , where off and s are integers, and >> represents a right shift operation.

10. The method according to claim 5, wherein the sub-blocks of the chromaticity component correspond to four sub-blocks of the luminance component, and the four sub-blocks each have a motion vector MV0 = (MV0 = ... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ) in ,in , , ,and ,in , where off and s are integers, and >> represents a shift operation.

11. The method according to claim 5, wherein the sub-blocks of the chromaticity component correspond to four sub-blocks of the luminance component, and the four sub-blocks each have a motion vector MV0 = (MV0 = ... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in, ,in , , ,and ,in , where off and s are integers, and >> represents a shift operation.

12. The method according to claim 5, wherein the sub-blocks of the chroma component correspond to four sub-blocks of the luminance component, and the motion vector of the sub-block of the chroma component is obtained based on the motion vectors of two sub-blocks of the corresponding four sub-blocks of the luminance component.

13. The method according to claim 12, wherein the two sub-blocks each have a motion vector MV0 = (MV 0x MV 0y ) and MV1 = (MV 1x MV 1y ), And among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in, ,in , where off and s are integers, and >> represents a right shift operation.

14. The method according to claim 12, wherein the two sub-blocks each have a motion vector MV0 = (MV 0x MV 0y ) and MV1 = (MV 1x MV 1y ), And among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ) in, ,in Where off and s are integers, and >> represents a shift operation.

15. The method of claim 1, wherein the selected sub-block includes the top-left sub-block among a plurality of sub-blocks corresponding to the luminance component.

16. The method of claim 1, wherein the selected sub-block comprises the center sub-block among a plurality of sub-blocks corresponding to the luminance component.

17. The method according to claim 1, wherein, Calculating the intermediate motion vector set includes: The intermediate motion vector set is calculated based on the median of the motion vectors of multiple sub-blocks of the brightness component.

18. The method according to any one of claims 1-17, wherein the application of the factor is specified in the High Efficiency Video Coding (HEVC) standard.

19. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 18.

20. An apparatus for video decoding, comprising: A receiving module is configured to receive a video data block including a luminance component and at least one chrominance component, determine the motion vector of the control point of the luminance component of the video data block based on an affine pattern, divide the luminance component into a first group of sub-blocks, and divide the chrominance component of the at least one chrominance component into a second group of sub-blocks, wherein each sub-block in the first group of sub-blocks has a second size, and each sub-block in the second group of sub-blocks has a first size; The processing module is configured to determine the motion vector of each sub-block in the first group of sub-blocks based on the motion vector of the control point, and to derive the motion vector of each sub-block in the second group of sub-blocks based on the motion vectors of corresponding sub-blocks in the first group of sub-blocks and the color format of the video data block; and The processing module is also used to reconstruct the video data block based on the derived motion vectors or to decode other video data blocks in the same image. Where the color format applied to the luminance component and the chrominance component is 4:2:0 or 4:2:2, the second set of sub-blocks having the first size equal to the second size is used to derive the predicted sample points of the chrominance component. The device is also configured to: Calculate an intermediate motion vector set based on the motion vectors of selected sub-blocks from among multiple sub-blocks of the brightness component; and A factor is applied to the intermediate motion vector set to obtain motion vectors for sub-blocks of the chromaticity components, wherein the factor is associated with the color format.

21. A method for video encoding, wherein, The video data block includes a luminance component and at least one chrominance component, and the method includes: The motion vector of the control point of the luminance component of the video data block is determined based on the affine pattern; The luminance component is divided into a first group of sub-blocks, wherein each sub-block in the first group of sub-blocks has a second size; The chromaticity component of the at least one chromaticity component is divided into a second group of sub-blocks, wherein each sub-block in the second group of sub-blocks has a first size; The motion vector of each sub-block in the first group of sub-blocks is determined based on the motion vector of the control point; Based on the motion vectors of corresponding sub-blocks in the first group of sub-blocks and the color format of the video data blocks, the motion vector of each sub-block in the second group of sub-blocks is derived; and The video data block can be reconstructed based on the derived motion vectors, or other video data blocks in the same image can be decoded. Where the color format applied to the luminance component and the chrominance component is 4:2:0 or 4:2:2, the second set of sub-blocks having the first size equal to the second size is used to derive the predicted sample points of the chrominance component. The method further includes: Calculate an intermediate motion vector set based on the motion vectors of selected sub-blocks from among multiple sub-blocks of the brightness component; and A factor is applied to the intermediate motion vector set to obtain motion vectors for sub-blocks of the chromaticity components, wherein the factor is associated with the color format.

22. The method of claim 21, wherein, in response to the color format being 4:4:4, at least one of the first group of sub-blocks has the same size as at least one of the second group of sub-blocks.

23. The method of claim 21, wherein the size of at least one of the first group of sub-blocks is 4×4 and the size of at least one of the second group of sub-blocks is 4×4.

24. The method of claim 21, wherein the second group of sub-blocks obtained by segmenting the chromaticity component corresponds to four sub-blocks of the luminance component, and the motion vector of one of the sub-blocks in the second group of sub-blocks is obtained based on the motion vectors of two of the four sub-blocks of the luminance component.

25. The method of claim 21, wherein at least one of the motion vectors of the sub-blocks of the chromaticity component is obtained based on an offset-based average of the corresponding motion vectors of the plurality of sub-blocks of the luminance component.

26. The method of claim 25, wherein the sub-blocks of the chroma component correspond to four sub-blocks of the luminance component, each of the four sub-blocks having a motion vector MV0 = (MV0 + 1) / 2. 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in, and ,in , where off and s are integers, and >> represents a right shift operation.

27. The method of claim 25, wherein the sub-blocks of the chroma component correspond to four sub-blocks of the luminance component, each of the four sub-blocks having a motion vector MV0 = (MV0 + 1) / 2. 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in and ,in , where off and s are integers, and >> represents a shift operation.

28. The method of claim 25, wherein the sub-blocks of the chroma component correspond to four sub-blocks of the luminance component, each of the four sub-blocks having a motion vector MV0 = (MV0 = ... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in ,in , , ,and ,in , where off and s are integers, and >> represents a right shift operation.

29. The method of claim 25, wherein the sub-blocks of the chroma component correspond to four sub-blocks of the luminance component, each of the four sub-blocks having a motion vector MV0 = (MV0 + 1) / 2. 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , As and ,in , , ,and ,in , where off and s are integers, and >> represents a right shift operation.

30. The method of claim 25, wherein the sub-blocks of the chroma component correspond to four sub-blocks of the luminance component, each of the four sub-blocks having a motion vector MV0 = (MV0 + 1) / 2. 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in, ,in , , ,and ,in , where off and s are integers, and >> represents a shift operation.

31. The method of claim 25, wherein the sub-blocks of the chroma component correspond to four sub-blocks of the luminance component, and the four sub-blocks each have a motion vector MV0 = (MV0 = ... 0x MV 0y ), MV1 = (MV 1x MV 1y ), MV2 = (MV 2x MV 2y ) and MV3 = (MV 3x MV 3y ), and among them, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in ,in , , ,and ,in , where off and s are integers, and >> represents a shift operation.

32. The method of claim 25, wherein the sub-blocks of the chroma component correspond to four sub-blocks of the luminance component, and the motion vector of the sub-block of the chroma component is obtained based on the motion vectors of two of the four sub-blocks of the corresponding luminance component.

33. The method according to claim 32, wherein the two sub-blocks respectively have motion vectors MV0 = (MV0x, MV0y) and MV1 = (MV1x, MV1y), and wherein, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ) in, ,in , where off and s are integers, and >> represents a right shift operation.

34. The method according to claim 32, wherein the two sub-blocks respectively have motion vectors MV0 = (MV0x, MV0y) and MV1 = (MV1x, MV1y), and wherein, Calculating the intermediate motion vector set includes: Calculate the intermediate motion vector set =( , ),in ,in Where off and s are integers, and >> represents a shift operation.

35. The method of claim 21, wherein the selected sub-block includes the top-left sub-block of a plurality of sub-blocks corresponding to the luminance component.

36. The method of claim 21, wherein the selected sub-block comprises the center sub-block among a plurality of sub-blocks corresponding to the luminance component.

37. The method according to claim 21, wherein, Calculating the intermediate motion vector set includes: The intermediate motion vector set is calculated based on the median of the motion vectors of multiple sub-blocks of the brightness component.

38. The method according to any one of claims 21-37, wherein the application of the factor is specified in the High Efficiency Video Coding (HEVC) standard.

39. A video encoding apparatus, wherein, The video data block includes a luminance component and at least one chrominance component, and the device includes: A determination module is used to determine the motion vector of the control point of the luminance component of the video data block based on the affine pattern; A segmentation module is used to divide the luminance component into a first group of sub-blocks, wherein each sub-block in the first group of sub-blocks has a second size; The partitioning module is further configured to divide the chromaticity component in the at least one chromaticity component into a second group of sub-blocks, wherein each sub-block in the second group of sub-blocks has a first size; The processing module is used to determine the motion vector of each sub-block in the first group of sub-blocks based on the motion vector of the control point; The processing module is further configured to derive the motion vector of each sub-block in the second group of sub-blocks based on the motion vectors of corresponding sub-blocks in the first group of sub-blocks and the color format of the video data block, and to reconstruct the video data block or decode other video data blocks in the same image based on the derived motion vectors. Where the color format applied to the luminance component and the chrominance component is 4:2:0 or 4:2:2, the second set of sub-blocks having the first size equal to the second size is used to derive the predicted sample points of the chrominance component. The video encoding device is further configured to: Calculate an intermediate motion vector set based on the motion vectors of selected sub-blocks from among multiple sub-blocks of the brightness component; and A factor is applied to the intermediate motion vector set to obtain motion vectors for sub-blocks of the chromaticity components, wherein the factor is associated with the color format.

40. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 21-38.

41. A non-transitory computer-readable storage medium for storing instructions, wherein when executed by a computer, the computer implements the method according to any one of claims 1-18 or 21-38.

42. A non-transitory computer-readable recording medium having a computer program stored thereon, said computer program, when executed by a processor, performing the following method to generate a bitstream of video. in, The video includes video data blocks, each video data block including a luminance component and at least one chrominance component, and the method includes: The motion vector of the control point of the luminance component of the video data block is determined based on the affine pattern; The luminance component is divided into a first group of sub-blocks, wherein each sub-block in the first group of sub-blocks has a second size; The chromaticity component of the at least one chromaticity component is divided into a second group of sub-blocks, wherein each sub-block in the second group of sub-blocks has a first size; The motion vector of each sub-block in the first group of sub-blocks is determined based on the motion vector of the control point; Based on the motion vectors of the corresponding multiple sub-blocks in the first group of sub-blocks and the color format of the video data block, the motion vector of each sub-block in the second group of sub-blocks is derived; The video data block is reconstructed based on the derived motion vectors, or other video data blocks in the same image are decoded; and Generate the bit stream, Where the color format applied to the luminance component and the chrominance component is 4:2:0 or 4:2:2, the second set of sub-blocks having the first size equal to the second size is used to derive the predicted sample points of the chrominance component. The method further includes: Calculate an intermediate motion vector set based on the motion vectors of selected sub-blocks from among multiple sub-blocks of the brightness component; and A factor is applied to the intermediate motion vector set to obtain motion vectors for sub-blocks of the chromaticity components, wherein the factor is associated with the color format.

43. A method for storing a video bitstream, comprising: The bit stream is generated by performing the following method; as well as The bitstream is stored in a non-transitory computer-readable recording medium. The video includes video data blocks, each video data block including a luminance component and at least one chrominance component, and the method includes: The motion vector of the control point of the luminance component of the video data block is determined based on the affine pattern; The luminance component is divided into a first group of sub-blocks, wherein each sub-block in the first group of sub-blocks has a second size; The chromaticity component of the at least one chromaticity component is divided into a second group of sub-blocks, wherein each sub-block in the second group of sub-blocks has a first size; The motion vector of each sub-block in the first group of sub-blocks is determined based on the motion vector of the control point; Based on the motion vectors of the corresponding multiple sub-blocks in the first group of sub-blocks and the color format of the video data block, the motion vector of each sub-block in the second group of sub-blocks is derived; The video data block is reconstructed or other video data blocks in the same image are decoded based on the derived motion vectors. The bitstream is generated; wherein, when the color format applied to the luminance component and the chrominance component is 4:2:0 or 4:2:2, the predicted samples of the chrominance component are derived using the second set of sub-blocks having the first size equal to the second size. The method further includes: Calculate an intermediate motion vector set based on the motion vectors of selected sub-blocks from among multiple sub-blocks of the brightness component; and A factor is applied to the intermediate motion vector set to obtain motion vectors for sub-blocks of the chromaticity components, wherein the factor is associated with the color format.

Images