The concept of interleaved prediction

Abstract

Methods, systems, and devices related to sub-block based motion prediction in video coding are described for the concept of interleaved prediction. In one typical aspect, a video processing method includes partitioning a video block into a first set of sub-blocks according to a first mode, partitioning the video block into a second set of sub-blocks according to a second mode, wherein at least one sub-block in the second set has a different size than a sub-block in the first set, and determining a prediction block that corresponds to a combination of a first intermediate prediction block generated from a prediction of the first set of sub-blocks and a second intermediate prediction block generated from a prediction of the second set of sub-blocks.

Description

[0001] Cross-reference to related applications

[0002] This application is a divisional application of the invention patent application filed on May 31, 2019, with application number 201910468364.5 and invention title "Concept of Interleaved Prediction". Technical Field

[0003] This application relates to video coding technologies, equipment, and systems. Background Technology

[0004] Motion compensation (MC) is a video processing technique that predicts frames in a video by taking into account the motion of the camera and / or objects in the video, given previous and / or future frames. Motion compensation can be used to encode video data to achieve video compression. Summary of the Invention

[0005] This document discloses methods, systems, and apparatuses related to sub-block-based motion prediction in video motion compensation.

[0006] In one typical aspect, a video processing method is disclosed. The method includes: segmenting a video block into a first group of sub-blocks according to a first mode, and segmenting the video block into a second group of sub-blocks according to a second mode. At least one sub-block in the second group has a different size than one sub-block in the first group. The method also includes determining a prediction block, which is a combination of a first intermediate prediction block generated from the first group of sub-blocks and a second intermediate prediction block generated from the second group of sub-blocks.

[0007] In another typical aspect, a video processing method is disclosed. The method includes dividing a video block into multiple sub-blocks based on the size of the video block or the type of encoding algorithm associated with the video block. One of the sub-blocks has a different size from the other sub-blocks. The method also includes generating motion vector predictions by applying the encoding algorithm to the multiple sub-blocks, and performing further processing of the video block using the motion vector predictions.

[0008] In another typical aspect, an apparatus is disclosed comprising a processor configured to implement the methods described herein.

[0009] In another typical aspect, the various techniques described herein can be implemented as a computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for implementing the methods described herein.

[0010] In another typical aspect, there is a video decoding device that can implement the methods described herein.

[0011] The details of one or more embodiments are set forth in the accompanying attachments, drawings, and description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a diagram showing an example of sub-block based prediction.

[0013] Figure 2 shows an example of an affine motion field of a block described by two control point motion vectors.

[0014] Figure 3 shows an example of an affine motion vector field for each sub-block of a block.

[0015] Figure 4 shows an example of motion vector prediction for a block 400 in AF INTER mode.

[0016] Figure 5A shows an example of the selection order of candidate blocks for a current coding unit (CU).

[0017] Figure 5B shows another example of candidate blocks for a current CU in AF MERGE mode.

[0018] Figure 6 shows an example of an alternative temporal motion vector prediction (ATMVP) motion prediction process for a CU.

[0019] Figure 7 shows an example of a CU with four sub-blocks and neighboring blocks.

[0020] Figure 8 shows an example optical flow trajectory in a bi-directional optical flow (BIO) method.

[0021] Figure 9A shows an example of an access location outside of a block.

[0022] Figure 9B shows that padding regions can be used to avoid additional memory accesses and calculations.

[0023] Figure 10 shows an example of bi-directional matching used in a frame rate up conversion (FRUC) method.

[0024] Figure 11 shows an example of template matching used in a FRUC method.

[0025] Figure 12 shows an example of a uni-directional motion estimation (ME) in a FRUC method.

[0026] Figure 13An example of interweaved prediction with two partition modes is shown in accordance with the disclosed technology.

[0027] Figure 14A An example partition mode in which a block is partitioned into 4x4 sub-blocks is shown in accordance with the disclosed technology.

[0028] Figure 14B An example partition mode in which a block is partitioned into 8x8 sub-blocks is shown in accordance with the disclosed technology.

[0029] Figure 14C An example partition mode in which a block is partitioned into 4x8 sub-blocks is shown in accordance with the disclosed technology.

[0030] Figure 14D An example partition mode in which a block is partitioned into 8x4 sub-blocks is shown in accordance with the disclosed technology.

[0031] Figure 14E An example partition mode in which a block is partitioned into inconsistent sub-blocks is shown in accordance with the disclosed technology.

[0032] Figure 14F Another example partition mode in which a block is partitioned into inconsistent sub-blocks is shown in accordance with the disclosed technology.

[0033] Figure 14G Yet another example partition mode in which a block is partitioned into inconsistent sub-blocks is shown in accordance with the disclosed technology.

[0034] Figure 15A An example flowchart of a method of video processing.

[0035] Figure 15B Another example flowchart of a method of video processing.

[0036] Figure 16 A diagram illustrating an example of an architecture of a computer system or other control device that can be used to implement various portions of the technology disclosed herein.

[0037] Figure 17 A block diagram showing an example embodiment of a mobile device that can be used to implement various portions of the technology disclosed herein.

[0038] Figure 18A 、 Figure 18B and Figure 18C An example embodiment of a partial interweaved prediction technique is shown.

[0039] Figure 19 A block diagram of an example video processing system that can implement the disclosed technology is shown. DETAILED DESCRIPTION

[0040] Global motion compensation is one of the variants of motion compensation techniques and can be used to predict the motion of a camera. However, moving objects within a frame are not adequately represented by various implementations of global motion compensation. Local motion estimation, such as block motion compensation, can be used to account for moving objects within a frame, where the frame is divided into blocks of pixels for performing motion prediction.

[0041] Subblock-based prediction, developed based on block motion compensation, was first introduced to video coding standards by High Efficiency Video Coding (HEVC) Annex I (3D-HEVC).

[0042] Figure 1 is a diagram showing an example of prediction-based subblock. Using subblock-based prediction, a block 100, such as a coding unit (CU) or a prediction unit (PU), is divided into several non-overlapping subblocks 101. Different subblocks can be assigned different motion information, such as reference indices or motion vectors (MVs). Motion compensation is then performed separately for each subblock.

[0043] To explore future video coding technologies beyond HEVC, the Video Coding Experts Group (VCEG) and the Moving Picture Experts Group (MPEG) jointly formed the Joint Video Exploration Team (JVET) in 2015. The JVET adopted many methods and added them to a reference software named Joint Exploration Model (JEM). In JEM, subblock-based prediction is employed in many coding technologies, such as affine prediction, alternative temporal motion vector prediction (ATMVP), spatial-temporal motion vector prediction (STMVP), bi-directional optical flow (BIO), and frame rate up conversion (FRUC), which are discussed in detail as follows.

[0044] Affine prediction

[0045] In HEVC, only translational motion model is applied to motion-compensated prediction (MCP). However, cameras and objects can have multiple kinds of motion, such as zooming, rotation, perspective motion, and / or other irregular motion. On the other hand, JEM applies a simplified affine transform motion-compensated prediction.

[0046] Figure 2 An example of an affine motion field of a block 200 described by two control point motion vectors V0and V1is shown. The motion vector field (MVF) of the block 200 can be described by the following equation:

[0047]

[0048] As shown in Figure 2 , (v 0x ,v 0y ) is the motion vector of the top-left control point, and (v 1x ,v 1y) is the motion vector of the top-right control point. To simplify the motion compensated prediction, a subblock-based affine transform prediction can be applied. The subblock size M x N is derived as follows:

[0049]

[0050] Here, MvPreis the motion vector fractional precision (e.g., 1 / 16 in JEM).(v 2x ,v 2y ) is the motion vector of the bottom-left control point, which is calculated according to equation (1). If needed, M and N can be down-adjusted to be the divisors of w and h, respectively.

[0051] Figure 3 An example of the affine MVF of each subblock of block 300 is shown. To derive the motion vector of each M x N subblock, the motion vector of the center sample of each subblock can be calculated according to equation (1) and rounded to the motion vector fractional precision (e.g., 1 / 16 in JEM). Then a motion compensated interpolation filter can be applied to generate the prediction of each subblock with the derived motion vector. After MCP, the high-precision motion vector of each subblock is rounded and saved as the same precision as the normal motion vector.

[0052] In JEM, there are two affine motion modes: AF INTER mode and AF MERGE mode. For a CU whose width and height are both larger than 8, AF INTER mode can be applied. In the bitstream, a CU-level affine flag is signaled to indicate whether AF INTER mode is used. In AF INTER mode, neighboring blocks are used to construct a candidate list with motion vector pairs {(v0, v1) | v0 = {v A ,v B ,v c}, v1 = {v D ,v E}}.

[0053] Figure 4 An example of the motion vector prediction (MVP) of block 400 in AF INTER mode is shown. As Figure 4As shown, v0 is selected from the motion vectors of sub-blocks A, B or C. The motion vectors of neighboring blocks can be scaled according to the reference list. The motion vectors can also be scaled according to the relationship between the picture order count (POC) of the neighboring block reference, the POC of the current CU reference and the POC of the current CU. The method of selecting v1 from neighboring sub-blocks D and E is similar. If the number of candidate lists is less than 2, the list is padded with motion vector pairs that copy each AMVP candidate. When the number of candidate lists is greater than 2, the motion vector pair candidates can be first ordered according to the neighboring motion vectors (e.g., based on the similarity of the two motion vectors in a pair of candidates). In some implementations, the first two candidates are kept. In some embodiments, a rate-distortion (RD) cost check is used to determine which motion vector pair candidate to select as the control point motion vector prediction (CPMVP) for the current CU. An index to the location of the CPMVP in the candidate list can be signaled in the bitstream. After the CPMVP of the current affine CU is determined, affine motion estimation is applied and the control point motion vectors (CPMVs) are found. The difference between the CPMVs and the CPMVP is then signaled in the bitstream.

[0054] When a CU is applied in AF_MERGE mode, it takes the first block encoded with affine mode from the available neighboring reconstructed blocks. Figure 5A An example of the selection order of the candidate blocks of the current CU 500 is shown. As shown, the selection order can be from left (501), top (502), top-right (503), bottom-left (504) to top-left (505) of the current CU 500. Figure 5A An example of the selection order of the candidate blocks of the current CU 500 is shown. As shown, the selection order can be from left (501), top (502), top-right (503), bottom-left (504) to top-left (505) of the current CU 500. Figure 5B Another example of the candidate blocks of the current CU 500 in AF_MERGE mode is shown. If the neighboring bottom-left block 501 is encoded with affine mode, as shown, the motion vectors v2, v3 and v4 containing the top-left, top-right and bottom-left corners of the CU of sub-block 501 are derived. The motion vector v0 of the top-left corner of the current CU 500 is calculated based on v2, v3 and v4. The motion vector v1 of the top-right corner of the current CU can be calculated accordingly. Figure 5B Another example of the candidate blocks of the current CU 500 in AF_MERGE mode is shown. If the neighboring bottom-left block 501 is encoded with affine mode, as shown, the motion vectors v2, v3 and v4 containing the top-left, top-right and bottom-left corners of the CU of sub-block 501 are derived. The motion vector v0 of the top-left corner of the current CU 500 is calculated based on v2, v3 and v4. The motion vector v1 of the top-right corner of the current CU can be calculated accordingly.

[0055] After the CPMVs v0 and v1 of the current CU are calculated according to the affine motion model in equation (1), the MVFs of the current CU can be generated. To identify whether the current CU is encoded with AF_MERGE mode, an affine flag can be signaled in the bitstream when at least one neighboring block is encoded with affine mode.

[0056] Alternative Temporal Motion Vector Prediction (ATMVP)

[0057] In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.

[0058] Figure 6 An example of the ATMVP motion prediction process for CU 600 is shown. The ATMVP method predicts the motion vectors of sub-CUs 601 within CU 600 in two steps. The first step is to identify a corresponding block 651 in a reference picture 650 with a temporal vector. The reference picture 650 is also referred to as a motion source picture. The second step is to divide the current CU 600 into sub-CUs 601 and obtain the motion vector and reference index for each sub-CU from its corresponding block.

[0059] In the first step, the reference picture 650 and the corresponding block are determined from the motion information of spatial neighboring blocks of the current CU 600. To avoid repeated scanning process of neighboring blocks, the first MERGE candidate in the MERGE candidate list of the current CU 600 is used. The first available motion vector and its associated reference index are set as the temporal vector and the index of the motion source picture. In this way, the corresponding block, sometimes referred to as the collocated block, can be more accurately identified compared to TMVP, which is always located at the right-bottom corner or the center position relative to the current CU.

[0060] In the second step, the corresponding block of a sub-CU 651 is identified in the motion source picture 650 by adding the temporal vector to the coordinates of the current CU. For each sub-CU, the motion information of the sub-CU is derived using the motion information of its corresponding block (e.g., the smallest motion grid covering the center sample). After the motion information of the corresponding NxN block is identified, it is converted into the motion vector and reference index of the current sub-CU in the same way as TMVP of HEVC, where motion scaling and other procedures are applied. For example, the decoder checks whether the low-delay condition is satisfied (e.g., the POCs of all reference pictures are smaller than the POC of the current picture) and possibly uses the motion vector MVx (e.g., the motion vector corresponding to the reference picture list X) to predict the motion vector MVy (e.g., X equals 0 or 1 and Y equals 1-X) of each sub-CU.

[0061] Space-time motion vector prediction (STMVP)

[0062] In the STMVP method, the motion vectors of sub-CUs are derived recursively in a raster scan order. Figure 7An example showing one CU with four sub-blocks and neighboring blocks is shown. Consider a CU 700 of 8x8, which includes four 4x4 sub-CUs A (701), B (702), C (703), and D (704). The neighboring 4x4 blocks in the current frame are labeled a (711), b (712), c (713), and d (714).

[0063] The motion derivation for sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the NxN block above sub-CU A 701 (block c 713). If this block c (713) is not available or intra coded, other NxN blocks above sub-CU A (701) are examined (from left to right, starting at block c 713). The second neighbor is the block to the left of sub-CU A 701 (block b 712). If block b (712) is not available or intra coded, other blocks to the left of sub-CU A 701 are examined (from top to bottom, starting at block b 712). The motion information obtained from each list from the neighboring blocks is scaled to the first reference frame of the given list. Next, the temporal motion vector prediction (TMVP) for sub-block A 701 is derived following the same procedure as TMVP in HEVC. The motion information of the collocated block at block D 704 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged separately for each reference list. The averaged motion vector is assigned as the motion vector of the current sub-CU.

[0064] Bi-directional optical flow (BIO)

[0065] The bi-directional optical flow (BIO) method is a sample direction motion refinement on top of bi-prediction with block-based motion compensation. In some implementations, the sample-level motion refinement does not use signaling.

[0066] Let I (k) be the luma value after block motion compensation to reference k (k = 0, 1), and be the luma value at the current sample, respectively. (k) The horizontal and vertical components of the gradient. Assuming the optical flow is valid, the motion vector field (v x ,v y ) is given by:

[0067]

[0068] Combining this optical flow equation with the Hermite interpolation of each sample motion trajectory, a unique third order polynomial is obtained that matches the function values I (k) and its derivative at the end points simultaneously. The value of this polynomial at t = 0 is the BIO prediction:

[0069]

[0070] Figure 8 An example optical flow trajectory in the bi-directional optical flow (BIO) method is shown. Here, τ0 and τ1 denote the distance to the reference frames. The distance τ0 and τ1 are computed based on the POC of Ref0 and Ref1: τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current). If both predictions come from the same temporal direction (both from the past or both from the future), the signs are different (e.g., τ0 · τ1 < 0). In this case, BIO is applied if the predictions are not from the same temporal point (e.g., τ0 ≠ τ1). Both reference regions have non-zero motion (e.g., MVx0, MVy0, MVx1, MVy1 ≠ 0), and the block motion vector is proportional to the temporal distance (e.g., MVx0 / MVx1 = MVy0 / MVy1 = -τ0 / τ1).

[0071] The motion vector field (v x ,v y ) is determined by minimizing the difference Δ of the values between A and B points. Figures 9A-9B An example is shown where the motion trajectory intersects the reference frame plane. The model uses only the first linear term of the local Taylor expansion for Δ:

[0072]

[0073] All values in the above equations depend on the sample position, denoted as (i', j'). Assuming that the motion is consistent in a local surrounding area, Δ can be minimized within a (2M+1)x(2M+1) square window Ω centered at the current prediction point (i, j), where M equals 2:

[0074]

[0075] For this optimization problem, JEM uses a simplified approach, first minimizing in the vertical direction and then in the horizontal direction. The result is as follows:

[0076]

[0077]

[0078] where,

[0079]

[0080] To avoid division by zero or very small values, a regularization parameter r and m can be introduced in equation (7) and equation (8).

[0081] r = 500 · 4 d-8 equation (10)

[0082] m = 700 · 4 d-8 Equation (11)

[0083] Here, d is the bit depth of the video samples.

[0084] To make the memory access of BIO the same as regular bi-prediction motion compensation, all the prediction and gradient values I (k) , Figure 9A An example of the access positions outside the block 900 is shown. As Figure 9A shown, in Equation (9), a (2M+1)x(2M+1) square window Ω centered on the current prediction point on the block boundary needs to access positions outside the block. In JEM, the values I (k) , outside the block are set to be equal to the nearest available values inside the block. For example, this can be implemented as a padding region 901, as Figure 9B shown.

[0085] With BIO, the motion field can be refined for each sample. To reduce the computational complexity, a block-based design of BIO is adopted in JEM. The motion refinement can be computed on a 4x4 block basis. In the block-based BIO, the s n values in Equation (9) for all samples in a 4x4 block can be aggregated, and then the aggregated values of s n are used for the derived BIO motion vector offsets for the 4x4 block. More specifically, the following equation can be used for the block-based BIO derivation:

[0086]

[0087] Here, b k denotes the group of samples belonging to the k-th 4x4 block of the prediction block. The s n in Equation (7) and Equation (8) is replaced by ((s n,bk )>>4) to derive the associated motion vector offsets.

[0088] In some cases, the MV regiment of BIO can be unreliable due to noise or irregular motion. Therefore, in BIO, the size of the MV regiment is fixed to a threshold. The threshold is determined based on whether all the reference pictures of the current picture are from one direction. For example, if all the reference pictures of the current picture are from one direction, the value of the threshold is set to 12x2 14-d , otherwise it is set to 12x2 13-d .

[0089] The gradients of BIO can be computed simultaneously by motion compensation interpolation using operations consistent with the HEVC motion compensation process (e.g., 2D separable finite impulse response (FIR)). In some embodiments, the input to the 2D separable FIR is the same reference frame as the motion compensation process, and the fractional position (fracX, fracY) according to the fractional part of the block motion vector. For the horizontal gradient The signal is first vertically interpolated using BIOfilterS corresponding to the fractional position fracY with a descaling scale shift of d-8. Then the gradient filter BIOfilterG is applied in the horizontal direction, which corresponds to the fractional position fracX with a descaling scale shift of 18-d. For the vertical gradient The gradient filter is applied vertically using BIOfilterG corresponding to the fractional position fracY with a descaling scale shift of d-8. Then the signal substitution is performed in the horizontal direction using BIOfilterS corresponding to the fractional position fracX with a descaling scale shift of 18-d. The length of the interpolation filter for the gradient computation BIOfilterG and the signal substitution BIOfilterS can be shorter (e.g., 6-tap) to keep reasonable complexity. Table 1 shows example filters for gradient computation with different fractional positions of the block motion vector in BIO. Table 2 shows example interpolation filters for prediction signal generation in BIO.

[0090] Table 1 Example filters for gradient computation in BIO

[0091] Fractional pixel position Interpolation filter for gradients (BIOfilterG) 0 {8,-39,-3,46,-17,5} 1 / 16 {8,-32,-13,50,-18,5} 1 / 8 {7,-27,-20,54,-19,5} 3 / 16 {6,-21,-29,57,-18,5} 1 / 4 {4,-17,-36,60,-15,4} 5 / 16 {3,-9,-44,61,-15,4} 3 / 8 {1,-4,-48,61,-13,3} 7 / 16 {0,1,-54,60,-9,2} 1 / 2 {-1,4,-57,57,-4,1}

[0092] Table 2 Example interpolation filters for prediction signal generation in BIO

[0093] Fractional pixel position Interpolation filter for prediction signal (BIOfilterS) 0 {0,0,64,0,0,0} 1 / 16 {1,-3,64,4,-2,0} 1 / 8 {1,-6,62,9,-3,1} 3 / 16 {2,-8,60,14,-5,1} 1 / 4 {2,-9,57,19,-7,2} 5 / 16 {3,-10,53,24,-8,2} 3 / 8 {3,-11,50,29,-9,2} 7 / 16 {3,-11,44,35,-10,3} 1 / 2 {3,-10,35,44,-11,3}

[0094] In JEM, BIO can be applied to all bi-predicted blocks when the two predictions come from different reference pictures. BIO can be disabled when local illumination compensation (LIC) is enabled for a CU.

[0095] In some embodiments, OBMC is applied to a block after the normal MC process. To reduce the computational complexity, BIO can not be applied during the OBMC process. This means that during the OBMC process, BIO is applied to the MC process of a block when its own MV is used, while BIO is not applied to the MC process of a block when the MV of a neighboring block is used.

[0096] Frame rate up conversion (FRUC)

[0097] For a CU, when its MERGE flag is true, a FRUC flag can be signaled. When the FRUC flag is false, a MERGE index can be signaled and regular MERGE mode is used. When the FRUC flag is true, another FRUC mode flag can be signaled to indicate which method (e.g., bi-directional matching or template matching) will be used to derive the motion information for the block.

[0098] At the encoder side, whether to use FRUC MERGE mode for a CU is decided based on the RD cost selection made for normal MERGE candidates. For example, multiple matching modes (e.g., bi-directional matching and template matching) for the CU are examined by using RD cost selection. The mode that results in the lowest cost is further compared with other CU modes. If the FRUC matching mode is the most efficient mode, then for the CU, the FRUC flag is set to true and the related matching mode is used.

[0099] Generally, the motion derivation process in FRUC MERGE mode has two steps: first, CU-level motion search is performed, and then sub-CU-level motion refinement is performed. At the CU level, the initial motion vector for the whole CU is derived based on bi-directional matching or template matching. First, a list of MV candidates is generated, and the candidate that results in the lowest matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bi-directional matching or template matching is performed around the starting point. The MV result of the minimum matching cost is taken as the MV value for the whole CU. Subsequently, the motion information is further refined at the sub-CU level with the derived CU motion vector as the starting point.

[0100] For example, the following derivation process is performed for WxH CU motion information derivation. In the first stage, the MV for the whole WxH CU is derived. In the second stage, the CU is further divided into MxM sub-CUs. The value of M is calculated according to (16), and D is a predefined division depth, which is set to 3 by default in JEM. Then the MV value for each sub-CU is derived.

[0101]

[0102] Figure 10An example of bi-directional matching used in the frame rate up conversion (FRUC) method is shown. Bi-directional matching is used to obtain the motion information of a current CU by finding the closest match between two blocks along the motion trajectory of the current CU (1000) in two different reference pictures (1010, 1011). Under the continuous motion trajectory assumption, the motion vectors MV0 (1001) and MV1 (1002) pointing to the two reference blocks are proportional to the temporal distance (e.g., TD0 (1003) and TD1 (1004)) between the current picture and the two reference pictures. In some embodiments, when the current picture 1000 is temporarily located between the two reference pictures (1010, 1011) and the temporal distance of the current picture to the two reference pictures is the same, the bi-directional matching becomes a mirror-based bi-directional MV.

[0103] Figure 11 An example of template matching used in the FRUC method is shown. Template matching can be used to obtain the motion information of a current CU 1100 by finding the closest match between a template (e.g., the top and / or left neighboring blocks of the current CU) in the current picture and a block (e.g., of the same size as the template) in the reference picture 1110. In addition to the FRUC MERGE mode described above, template matching can also be applied to the AMVP mode. In both JEM and HEVC, AMVP has two candidates. With the template matching method, a new candidate can be derived. If the new candidate derived by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (e.g., by deleting the second existing AMVP candidate). When applied to the AMVP mode, only CU-level search is applied.

[0104] CU-level set MV candidates can include the following: (1) the original AMVP candidate if the current CU is in the AMVP mode, (2) all MERGE candidates, (3) a number of MVs in the interpolated MV field (described later), and the top and left neighboring motion vectors.

[0105] When bi-directional matching is used, each valid MV of a MERGE candidate can be used as input to generate a pair of MVs with the hypothesis of bi-directional matching. For example, a MERGE candidate has one valid MV at the reference list A (MVa, refA). Then its counterpart bi-directional MV is found in another reference list B (MVb, refB) such that refA and refB are located on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be the same as the reference refA. a ) in the reference list A. Then its counterpart bi-directional MV is found in another reference list B (MVb, refB) such that refA and refB are located on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be the same as the reference refA. b ) in the reference list A. Then its counterpart bi-directional MV is found in another reference list B (MVb, refB) such that refA and refB are located on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be the same as the reference refA. a ) in the reference list A. Then its counterpart bi-directional MV is found in another reference list B (MVb, refB) such that refA and refB are located on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be the same as the reference refA. b ) in the reference list A. Then its counterpart bi-directional MV is found in another reference list B (MVb, refB) such that refA and refB are located on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be the same as the reference refA. b ) in the reference list A. Then its counterpart bi-directional MV is found in another reference list B (MVb, refB) such that refA and refB are located on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be the same as the reference refA. b ) in the reference list A. Then its counterpart bi-directional MV is found in another reference list B (MVb, refB) such that refA and refB are located on different sides of the current picture in time. If the reference refB in the reference list B is not available, the reference refB is determined to be the same as the reference refA. adifferent reference, and its temporal distance to the current picture is the smallest distance in the list B. The reference ref b After that, a MVb is derived by scaling the MVa between the current picture and the reference ref a , the reference ref b .

[0106] In some implementations, four MVs from the interpolated MV field can also be added to the CU-level candidate list. More specifically, the interpolated MVs at the locations (0, 0), (W / 2, 0), (0, H / 2) and (W / 2, H / 2) of the current CU are added. When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the MV candidate set at the CU level. In some implementations, at the CU level, 15 MVs for AMVP CUs and 13 MVs for MERGE CUs can be added to the candidate list.

[0107] The MV candidates set at the sub-CU level includes (1) the MVs determined from the CU-level search, (2) the top, left, top-left and top-right neighboring MVs, (3) scaled versions of collocated MVs in the reference pictures, (4) one or more ATMVP candidates (e.g., up to four) and (5) one or more STMVP candidates (e.g., up to four). The scaled MVs from the reference pictures are derived as follows. Both lists of reference pictures are traversed. The MVs at the collocated positions of the sub-CU in the reference pictures are scaled to the reference of the starting CU-level MV. The ATMVP and STMVP candidates can be the top four. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

[0108] Generation of the interpolated MV field

[0109] Before encoding the frame, an interpolated motion field for the entire picture is generated based on the uni-directional ME. This motion field can then be used as MV candidates at the CU level or sub-CU level.

[0110] In some embodiments, the motion field of each reference picture in both reference lists is traversed at the 4x4 block level. Figure 12 An example of the uni-directional motion estimation (ME) 1200 in the FRUC method is shown. For each 4x4 block, if the motion associated with the block goes through a 4x4 block in the current picture and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (in the same way as the MV scaling in TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to a 4x4 block, the motion of the block is marked as unavailable in the interpolated motion field.

[0111] Interpolation and matching cost

[0112] When the motion vector points to a fractional sample position, motion compensation interpolation is needed. To reduce complexity, bilinear interpolation is used for both bi-directional matching and template matching instead of the regular 8-tap HEVC interpolation.

[0113] The calculation of matching cost is a bit different at different steps. When selecting a candidate from the CU-level candidate set, the matching cost can be the sum of absolute difference (SAD) of bi-directional matching or template matching. After the initial MV is determined, the matching cost C of bi-directional matching at the sub-CU level search is calculated as follows:

[0114]

[0115] Here, w is a weight coefficient. In some embodiments, w can be empirically set to 4. MV and MV s indicate the current MV and the initial MV, respectively. SAD can still be used as the matching cost for pattern matching at the sub-CU level search.

[0116] In FRUC mode, the MV is derived by using only luma samples. The derived motion will be used for luma and chroma for MC inter prediction. After the MV is determined, the final MC is performed using an 8-taps interpolation filter for luma and a 4-taps interpolation filter for chroma.

[0117] MV refinement is a pattern-based MV search, based on either bi-directional matching cost or template matching cost. In JEM, two search patterns are supported—unrestricted center-biased diamond search (UCBDS) and adaptive cross search, at the CU level and sub-CU level, respectively, for MV refinement. For both CU-level and sub-CU-level MV refinement, MV is searched directly at quarter luma sample MV precision, followed by one-eighth luma sample MV refinement. The search range for MV refinement at both CU and sub-CU steps is set to 8 luma samples.

[0118] In bi-directional matching MERGE mode, bi-directional prediction is applied because the motion information of a CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. In template matching MERGE mode, the encoder can choose for the CU either from the uni-prediction of List 0, uni-prediction of List 1, or bi-prediction. The choice can be based on the template matching cost as follows:

[0119] If costBi <= factor * min(cost0, cost1)

[0120] then bi-prediction is used;

[0121] Otherwise, if cost0 <= cost1

[0122] Unidirectional prediction in List 0 is used;

[0123] Otherwise,

[0124] Unidirectional prediction in List 1 is used;

[0125] Here, cost0 is the SAD of the List 0 template matching, cost1 is the SAD of the List 2 template matching, and costBi is the SAD of the bi-directional template matching. For example, when the value of factor is equal to 1.25, it means that the selection process is biased towards bi-directional prediction. The inter prediction direction selection can be applied to the CU level template matching process.

[0126] The sub-block based prediction techniques discussed above can be used to obtain more accurate motion information for each sub-block when the sub-block size is small. However, smaller sub-blocks impose higher bandwidth requirements in motion compensation. On the other hand, for smaller sub-blocks, the derived motion information can not be accurate, especially when there is some noise in the block. Therefore, having a fixed sub-block size within a block can be sub-optimal.

[0127] Techniques are described herein that can be used in various embodiments to address the bandwidth and accuracy issues introduced by fixed sub-block size using non-uniform and / or variable sub-block size. These techniques, also referred to as interleaved prediction, use different ways of partitioning a block in order to obtain motion information more reliably without increasing the bandwidth consumption.

[0128] Using the interleaved prediction techniques, a block is partitioned into sub-blocks with one or more partition modes. A partition mode represents a way of partitioning a block into sub-blocks, including the size of the sub-blocks and the location of the sub-blocks. For each partition mode, a corresponding prediction block can be generated by deriving the motion information for each sub-block based on the partition mode. Therefore, in some embodiments, multiple prediction blocks can be generated for each prediction direction through multiple partition modes. In some embodiments, only one partition mode can be applied for each prediction direction.

[0129] Figure 13 An example of interleaved prediction with two partition modes according to the disclosed techniques is shown. A current block 1300 can be partitioned into multiple modes. For example, as shown, the current block is partitioned into mode 0 (1301) and mode 1 (1302). Two prediction blocks P0 (1303) and P1 (1304) are generated. By computing a weighted sum of P0 (1303) and P1 (1304), a final prediction block P (1305) for the current block 1300 can be generated. Figure 13

[0130] ​Generally, given X partition modes, X prediction blocks (denoted as P0, P1, …, PX) of a current block can be generated by sub-block based prediction in X partition modes. A final prediction (denoted as P) of the current block can be generated as: X-1 P = (P0 + P1 + … + PX) / X

[0131]

[0132] Here, (x, y) is the coordinate of a pixel in the block, and w i (x, y) is the weight coefficient of P i . By way of example and not limitation, the weight can be denoted as:

[0133]

[0134] N is a non-negative value. Optionally, the bit-shift operation in equation (8) can also be denoted as:

[0135]

[0136] The sum of the weights is a power of 2, and by performing a shift operation instead of a floating point division, the weighted sum P can be calculated more efficiently.

[0137] The partition modes can have different sub-block shapes, sizes, or positions. In some embodiments, the partition modes can include irregular sub-block sizes. Figures 14A-14G Examples of several partition modes for a 16x16 block are shown. In Figure 14A , the block is partitioned into 4x4 sub-blocks according to the disclosed techniques. This mode is also used in JEM. Figure 14B An example of a partition mode that partitions a block into 8x8 sub-blocks according to the disclosed techniques is shown. Figure 14C An example of a partition mode that partitions a block into 8x4 sub-blocks according to the disclosed techniques is shown. Figure 14D An example of a partition mode that partitions a block into 4x8 sub-blocks according to the disclosed techniques is shown. In Figure 14E , a portion of the block is partitioned into 4x4 sub-blocks according to the disclosed techniques. Pixels on the block boundary are partitioned into smaller sub-blocks of size 2x4, 4x2, or 2x2. Some sub-blocks can be merged to form larger sub-blocks. Figure 14F Examples of neighboring sub-blocks (such as 4x4 sub-blocks and 2x4 sub-blocks) that are merged to form larger sub-blocks of size 6x4, 4x6, or 6x6 are shown. In Figure 14G , a portion of the block is partitioned into 8x8 sub-blocks. While the block boundary pixels are partitioned into smaller sub-blocks such as 8x4, 4x8, or 4x4.

[0138] In sub-block based prediction, the shape and size of the sub-blocks can be determined based on the shape and / or size of the coding block and / or coding block information. The coding block information can include the type of coding algorithm used on the block and / or sub-blocks, such as whether the motion compensated prediction is (1) an affine prediction method, (2) an optional temporal motion vector prediction method, (3) a spatial-temporal motion vector prediction method, (4) a bi-directional optical flow method, or (5) a frame rate up conversion method. For example, in some embodiments, when the size of the current block is M x N, the size of the sub-blocks is 4 x N (or 8 x N, etc.), i.e., the sub-blocks have the same height as the current block. In some embodiments, when the size of the current block is M x N, the size of the sub-blocks is M x 4 (or M x 8, etc.), i.e., the sub-blocks have the same width as the current block. In some embodiments, when the size of the current block is M x N (where M > N), the size of the sub-blocks is A x B, where A > B (e.g., 8 x 4). Alternatively, the size of the sub-blocks is B x A (e.g., 4 x 8).

[0139] In some embodiments, the size of the current block is M x N. When M x N <= T (or min(M, N) <= T, or max(M, N) <= T, etc.), the size of the sub-blocks is A x B; when M x N > T (or min(M, N) > T, or max(M, N) > T, etc.), the size of the sub-blocks is C x D, where A <= C, B <= D. For example, if M x N <= 256, the size of the sub-blocks can be 4 x 4. In some implementations, the size of the sub-blocks is 8 x 8.

[0140] It should be noted that the interleaved prediction techniques disclosed herein can be applied to one, some, or all of the sub-block based prediction techniques. For example, the interleaved prediction techniques can be applied to affine prediction, while other sub-block based prediction techniques (e.g., ATMVP, STMVP, FRUC, or BIO) do not use interleaved prediction. As another example, all of affine, ATMVP, and STMVP apply the interleaved prediction techniques disclosed herein.

[0141] Figure 15A is an example flowchart of a method 1500 of improving motion prediction in a video system according to the disclosed technology. The method 1500 includes selecting a set of pixels from a video frame to form a block at 1502. The method 1500 includes partitioning the block into a first set of sub-blocks according to a first mode at 1504. The method 1500 includes generating a first intermediate prediction block based on the first set of sub-blocks at 1506. The method 1500 includes partitioning the block into a second set of sub-blocks according to a second mode at 1508. At least one sub-block of the second set has a different size than a size of one of the sub-blocks of the first set. The method 1500 includes generating a second intermediate prediction block based on the second set of sub-blocks at 1510. The method 1500 also includes determining a prediction block based on the first intermediate prediction block and the second intermediate prediction block at 1512.

[0142] In some embodiments, the method 1500 can be implemented without explicitly generating the intermediate prediction block. For example, the following operations can be performed on the generation of the prediction block: partitioning the video block into a first set of sub-blocks according to a first mode, partitioning the video block into a second set of sub-blocks according to a second mode, wherein at least one sub-block in the second set has a different size than a sub-block in the first set, and determining the prediction block as a combination of a first intermediate prediction block generated predictively from the first set of sub-blocks and a second intermediate prediction block generated predictively from the second set of sub-blocks.

[0143] In some embodiments, the first or second intermediate prediction block is generated using at least one of (1) an affine prediction method, (2) a selectable temporal motion vector prediction method, (3) a spatial-temporal motion vector prediction method, (4) a bi-directional optical flow method, or (5) a frame rate up conversion method. Some example embodiments of these video coding methods are disclosed herein.

[0144] In some embodiments, the sub-blocks in the first or second set have a rectangular shape. For example, a rectangular shape can refer to a non-square shape where the width and height are not equal. In some embodiments, the sub-blocks in the first set of sub-blocks have a non-uniform shape. In some embodiments, the sub-blocks in the second set of sub-blocks have a non-uniform shape. For example, a non-uniform shape can include sub-blocks of different sizes in the first (or second) set.

[0145] In some embodiments, the method includes determining the first or second mode based on a size of the video block. For example, a tall rectangular block can be split into multiple horizontal sub-blocks. In some cases, a rectangular block can be split into multiple square sub-blocks. In some embodiments, the method includes determining the first or second mode based on information from a second block that is temporally or spatially adjacent to the video block. For example, in some cases, the first or second mode for a video block can be identical to the mode of at least one temporal or spatial neighbor.

[0146] In some embodiments, the partitioning into the first set of sub-blocks is performed for motion compensation of the block in a first direction. In some embodiments, the partitioning into the second set of sub-blocks is performed for motion compensation of the block in a second direction. For example, the first direction and / or the second direction can be a backward direction of predictive coding ("B prediction") or a forward direction of predictive coding ("P prediction").

[0147] In some embodiments, for motion compensation of the block in the first direction, the method includes partitioning the block into a first set of sub-blocks and partitioning the block into a second set of sub-blocks. In some embodiments, the method further includes: motion compensating the block in a second direction by dividing the block into a third set of sub-blocks according to a third mode; generating a third intermediate prediction block based on the third set of sub-blocks; dividing the block into a fourth set of sub-blocks according to a fourth mode, wherein at least one sub-block in the fourth set is different in size from a sub-block in the third set; generating a fourth intermediate prediction block based on the fourth set of sub-blocks; determining a second prediction block based on the third intermediate prediction block and the fourth intermediate prediction block; and determining a third prediction block based on the prediction block and the second prediction block.

[0148] In some embodiments, the method includes transmitting information of a first mode and a second mode for partitioning a block to an encoding device in a block-based motion compensated video system. In some embodiments, the transmitting the information of the first mode and the second mode is performed at one of: (1) a sequence level, (2) a picture level, (3) a view level, (4) a slice level, (5) a coding tree unit, (6) a largest coding unit level, (7) a coding unit level, (8) a prediction unit level, (10) a tree unit level, or (11) a region level.

[0149] In some embodiments, determining the prediction result includes: applying a first set of weights to the first intermediate prediction block to obtain a first weighted prediction block; applying a second set of weights to the second intermediate prediction block to obtain a second weighted prediction block; and calculating a weighted sum of the first weighted prediction block and the second weighted prediction block to obtain the prediction block.

[0150] In some embodiments, the first set of weights or the second set of weights includes fixed weight values. In some embodiments, the first set of weights or the second set of weights is determined based on information from another block that is temporally or spatially adjacent to the block. In some embodiments, the first set of weights or the second set of weights is determined using an encoding algorithm used to generate the first prediction block or the second prediction block. In some implementations, at least one value in the first set of weights is different from another value in the first set of weights. In some implementations, at least one value in the second set of weights is different from another value in the second set of weights. In some implementations, the sum of the weights is equal to a power of two.

[0151] In some embodiments, the method includes transmitting weights to an encoding device in a block-based motion compensated video system. In some embodiments, the transmitting the weights is performed at one of: (1) a sequence level, (2) a picture level, (3) a view level, (4) a slice level, (5) a coding tree unit, (6) a largest coding unit level, (7) a coding unit level, (8) a prediction unit level, (10) a tree unit level, or (11) a region level.

[0152] Figure 15Bis an example flowchart of a method 1550 for improving block-based motion compensation in a video system according to the disclosed technology. The method 1550 includes selecting a set of pixels from a video frame to form a block at 1552. The method 1550 includes partitioning the block into a plurality of sub-blocks based on a size of the block or information of another block spatially or temporally adjacent to the block at 1554. At least one sub-block of the plurality of sub-blocks is of a different size than the other sub-blocks. The method 1550 also includes generating a motion vector prediction by applying an encoding algorithm to the plurality of sub-blocks at 1556. In some embodiments, the encoding algorithm includes at least one of (1) an affine prediction method, (2) a selectable temporal motion vector prediction method, (3) a spatial-temporal motion vector prediction method, (4) a bi-directional optical flow method, or (5) a frame rate up conversion method.

[0153] An alternative method includes partitioning a video block into a plurality of sub-blocks based on a size of the video block or a type of encoding algorithm associated with the video block, wherein a size of one sub-block of the plurality of sub-blocks is different than a size of other sub-blocks of the plurality of sub-blocks, generating a motion vector prediction by applying an encoding algorithm to the plurality of sub-blocks, and performing further processing of the video block using the motion vector prediction.

[0154] In the methods 1500 and 1550, partial interleaving can be implemented. Using this approach, samples in a first subset of the prediction samples are computed as a weighted combination of the first intermediate prediction block, and samples in a second subset of the prediction samples are copied from the sub-block based prediction, where the first and second subsets are based on the partitioning mode. The first and second subsets can together make up the entire prediction block, e.g., the block currently being processed. As shown in Figures 18A to 18C In various examples, the second subset excluded from interleaving can consist of (a) corner sub-blocks or (b) the topmost and bottommost rows of sub-blocks or (c) the leftmost or rightmost columns of sub-blocks, as shown. The size of the block currently being processed can be a condition for deciding whether to exclude certain sub-blocks from interleaved prediction. For example, some conditions are described later.

[0155] As further described herein, the encoding process can avoid checking affine modes for blocks split from a parent block that itself is encoded using a mode other than affine mode.

[0156] Figure 16 is a schematic diagram illustrating an example of the structure of a computer system or other control device 1600 that can be used to implement various portions of the technology disclosed herein. In Figure 16In particular embodiments, computer system 1600 includes one or more processors 1605 and memory 1610 connected by an interconnect 1625. Interconnect 1625 can represent one or more separate physical buses, point-to-point connections, or both, connected by appropriate bridges, adapters, or controllers. Accordingly, interconnect 1625 can include, for example, a system bus, a peripheral component interconnect (PCI) bus, a HyperTransport, or Industry Standard Architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 674 bus (sometimes referred to as "Firewire").

[0157] Processor 1605 can include a central processing unit (CPU) to control the overall operation of the computer system, in some embodiments, processor 1605 accomplishes this by executing software or firmware stored in memory 1610. Processor 1605 can be or include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), or the like, or a combination of these devices.

[0158] Memory 1610 can be or include the main memory of the computer system. Memory 1610 represents any appropriate form of random access memory (RAM), read only memory (ROM), flash memory, or the like, or a combination of these devices. In use, memory 1610 can contain, among other things, a set of machine instructions that, when executed by processor 1605, cause processor 1605 to perform operations to implement embodiments of the disclosed technology.

[0159] Also connected to processor 1605 by interconnect 1625 is (optional) network adapter 1615. Network adapter 1615 provides computer system 1600 with the ability to communicate with remote devices, such as storage clients and / or other storage servers, and can be, for example, an Ethernet adapter or a Fibre Channel adapter.

[0160] Figure 17A block diagram showing an example embodiment of a mobile device 1700 that can be used to implement various portions of the disclosed technology is shown. The mobile device 1700 can be a laptop computer, a smartphone, a tablet computer, a video camera, or other device capable of processing video. The mobile device 1700 includes a processor or controller 1701 to process data, and a memory 1702 in communication with the processor 1701 to store and / or buffer data. For example, the processor 1701 can include a central processing unit (CPU) or a microcontroller unit (MCU). In some implementations, the processor 1701 can include a field programmable gate array (FPGA). In some implementations, the mobile device 1700 includes or is in communication with a graphics processing unit (GPU), a video processing unit (VPU), and / or a wireless communication unit to implement various visual and / or communication data processing functions of a smartphone device. For example, the memory 1702 can include and store processor-executable code that, when executed by the processor 1701, configures the mobile device 1700 to perform various operations, such as receiving information, commands, and / or data, processing information and data, and sending or providing processed information / data to another data device, such as an actuator or an external display. To support various functions of the mobile device 1700, the memory 1702 can store information and data, such as instructions, software, values, images, and other data referenced or processed by the processor 1701. For example, the memory 1702 can be implemented using various types of random access memory (RAM) devices, read-only memory (ROM) devices, flash memory devices, and other suitable storage media. In some implementations, the mobile device 1700 includes an input / output (I / O) unit 1703 to interface the processor 1701 and / or the memory 1702 with other modules, units, or devices. For example, the I / O unit 1703 can interface the processor 1701 and the memory 1702 to utilize various wireless interfaces compatible with typical data communication standards, such as between one or more computers in the cloud and a user device. In some implementations, the mobile device 1700 can interface with other devices using wired connections through the I / O unit 1703. The I / O unit 1703 can include wireless sensors, such as infrared detectors to detect remote control signals, or other suitable wireless human interface technologies. The mobile device 1700 can also be connected with other external interfaces (e.g., data storage) and / or visual or audio display devices 1704 to retrieve and transmit data and information that can be processed by the processor, stored by the memory, or displayed by the display devices 1704 or output units of external devices. For example, the display devices 1704 can display video frames modified based on MVP (e.g., including predicted blocks 1305 as shown) in accordance with the disclosed technology. Figure 13

[0161] ​In some embodiments, a video decoder device can implement a video decoding method in which video decoding is performed using improved block-based motion prediction described herein. The method can include forming a video block using a set of pixels from a video frame. The block can be partitioned into a first set of sub-blocks according to a first mode. A first intermediate prediction block can correspond to the first set of sub-blocks. The block can include a second set of sub-blocks according to a second mode. At least one sub-block of the second set has a different size than a sub-block of the first set. The method can further determine a prediction block based on the first intermediate prediction block and a second intermediate prediction block generated from the second set of sub-blocks. Other features of the method can be similar to the method 1500 described above.

[0162] In some embodiments, a decoder-side method of video decoding can improve the quality of predicted video using block-based motion prediction by using a block of a video frame, where the block corresponds to a set of pixel blocks. The block can be partitioned into a plurality of sub-blocks based on a size of the block or information from another block that is spatially or temporally adjacent to the block, where at least one sub-block of the plurality of sub-blocks has a different size than a size of another sub-block. The decoder can use a motion vector prediction generated by applying an encoding algorithm to the plurality of sub-blocks. Other features of the method can be described with reference to Figure 15B and corresponding descriptions.

[0163] In some embodiments, a video decoding method can be implemented using a decoding device implemented on a hardware platform as described in Figure 16 and Figure 17 A video decoding method can be implemented using a decoding device implemented on a hardware platform as described in

[0164] Partial Interweaving

[0165] In some embodiments, partial interweaving prediction can be implemented as follows.

[0166] In some embodiments, interweaving prediction is applied to a portion of a current block. Prediction samples at certain positions are calculated as a weighted sum of two or more sub-block based predictions. Prediction samples at other positions are not used in the weighted sum. For example, these prediction samples are copied from a sub-block based prediction with a particular partition mode.

[0167] In some embodiments, a current block is predicted by sub-block based predictions PI and P2, which have partition modes DO and D2, respectively. The final prediction value is calculated as P = wO x PO + w1 x P1. At certain positions, wO ≠ 0 and w1 ≠ 0. But at other positions, wO = 1 and w1 = 0, that is, interweaving prediction is not applied at these positions.

[0168] In some embodiments, interweaving prediction is not applied to Figure 18A quad sub-blocks as shown.

[0169] In some embodiments, interweaving prediction is not applied toFigure 18B the leftmost column of sub-blocks and the rightmost column of sub-blocks.

[0170] In some embodiments, interleaved prediction is not applicable to Figure 18C the topmost row of sub-blocks and the bottommost row of sub-blocks.

[0171] Examples of techniques included in encoder embodiments

[0172] In some embodiments, interleaved prediction is not applied to the motion estimation (ME) process.

[0173] For example, interleaved prediction is not applied in the ME process for 6-parameter affine prediction.

[0174] For example, interleaved prediction is not applied in the ME process if the size of the current block satisfies certain conditions. Here, let the width and height of the current block be W and H, respectively, and T, T1 and T2 be integer values:

[0175] W >= T1 and H >= T2;

[0176] W <= T1 and H <= T2;

[0177] W >= T1 or H >= T2;

[0178] W <= T1 or H <= T2;

[0179] W + H >= T

[0180] W + H <= T

[0181] W x H >= T

[0182] W x H <= T

[0183] For example, if the current block is separated from the parent block and the parent block does not select affine mode at the encoder, interleaved prediction is omitted in the ME process.

[0184] Or, if the current block is separated from the parent block and the parent block does not select affine mode at the encoder, affine mode is not checked at the encoder.

[0185] Table A illustrates example performance results for using regular 2x2 affine prediction for random access (RA) configuration.

[0186] Table A Example test results for 2x2 affine prediction

[0187]

[0188]

[0189] Table B illustrates example performance results for applying interweaved prediction to uni-prediction according to embodiments of the present technology. Table C illustrates example performance results for applying interweaved prediction to bi-prediction according to embodiments of the present technology.

[0190] Table B Example test results for interweaved prediction in uni-prediction

[0191] Y U V EncT DecT Class A1 -0.05％ -0.14％ -0.02％ 101％ 100％ Class A2 -0.55％ -0.17％ -0.11％ 102％ 101％ Class B -0.33％ -0.17％ -0.20％ 101％ 101％ Class C -0.15％ -0.16％ -0.04％ 100％ 100％ Class D -0.21％ -0.09％ -0.02％ 106％ 106％ Class F -0.39％ -0.40％ -0.39％ 102％ 102％ Overall -0.27％ -0.16％ -0.11％ 101％ 101％

[0192] Table C Example test results for interweaved prediction in bi-prediction

[0193] Y U V EncT DecT Class A1 -0.09％ -0.18％ -0.12％ 103％ 102％ Class A2 -0.74％ -0.40％ -0.28％ 104％ 104％ Class B -0.37％ -0.39％ -0.35％ 103％ 102％ Class C -0.22％ -0.19％ -0.13％ 102％ 102％ Class D -0.42％ -0.28％ -0.32％ 103％ 102％ Class F -0.60％ -0.64％ -0.62％ 102％ 102％ Overall -0.38％ -0.30％ -0.23％ 103％ 102％

[0194] As shown in Tables B and C, interweaved prediction achieves major coding gains with lower complexity compared to conventional 2x2 affine prediction based coding. In particular, interweaved prediction applied to bi-prediction achieves a coding gain of 0.38% compared to the 2x2 affine method (0.47%). The encoding time and the decoding time for the 2x2 affine method are 103% and 102% respectively compared to 136% and 114% in the 2x2 affine method.

[0195] Figure 19 FIG. 19 is a block diagram illustrating an example video processing system 1900 in which various technologies disclosed herein can be implemented. Various implementations can include some or all of the components of system 1900. System 1900 can include an input 1902 for receiving video content. The video content can be received in a raw or uncompressed format (e.g., 8-bit or 10-bit multi-component pixel values), or can be received in a compressed or encoded format. Input 1902 can represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, passive optical network (PON), etc., and wireless interfaces such as Wi-Fi or cellular interfaces. Examples of peripheral bus interfaces can include a universal serial bus (USB) or a high-definition multimedia interface (HDMI) or display port, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc.

[0196] System 1900 can include an encoding component 1904, which can implement various encoding methods described herein. Encoding component 1904 can reduce the average bitrate of video from input 1902 to the output of encoding component 1904. Thus, the encoding techniques are sometimes referred to as video compression or video transcoding techniques. The output of encoding component 1904 can be stored or transmitted via a connected communication (as represented by component 1906). Component 1906 can use the stored or communicated bitstream representation of the video received at input 1902 to generate displayable video sent to a display interface 1908. The process of generating user-viewable video from a bitstream representation is sometimes referred to as video decompression.

[0197] The techniques described herein can be implemented in various electronic devices such as a mobile telephone, a laptop computer, a smart phone, or other devices that are capable of performing digital data processing and / or video display.

[0198] In one example aspect, a video processing method includes partitioning a video block into a first set of sub-blocks according to a first mode and partitioning the video block into a second set of sub-blocks according to a second mode. At least one sub-block in the second set has a different size than a sub-block in the first set. The method also includes determining a prediction block that is a combination of a first intermediate prediction block generated from the first set of sub-blocks and a second intermediate prediction block generated from the second set of sub-blocks.

[0199] In some embodiments, the first intermediate prediction block or the second intermediate prediction block is generated using at least one of (1) an affine prediction method, (2) a selectable temporal motion vector prediction method, (3) a spatial-temporal motion vector prediction method, (4) a bi-directional optical flow method, or (5) a frame rate up-conversion method.

[0200] In some embodiments, the sub-blocks in the first set or the second set have a rectangular shape. In some embodiments, the sub-blocks in the first set of sub-blocks have a non-uniform shape. In some embodiments, the sub-blocks in the second set of sub-blocks have a non-uniform shape.

[0201] In some embodiments, the method includes determining the first mode or the second mode based on a size of the video block. In some embodiments, to perform motion compensation of the video block based on the reference picture in the first reference picture list, the video block is partitioned into a first set of sub-blocks. In some embodiments, to perform motion compensation of the video block based on the reference picture in a second reference picture list different from the first reference picture list, the block is partitioned into a second set of sub-blocks. In some embodiments, to perform motion compensation of the video block from the reference picture in a second reference picture list same as the first reference picture list, the video block is partitioned into a second set of sub-blocks. In some embodiments, to perform motion compensation of the video block from the reference picture in the second reference picture list, the method partitions the video block into a third set of sub-blocks according to a third mode, generates a third intermediate prediction block based on the third set of sub-blocks, partitions the video block into a fourth set of sub-blocks according to a fourth mode, wherein at least one sub-block in the fourth set has a different size than a sub-block in the third set, generates a fourth intermediate prediction block based on the fourth set of sub-blocks, determines a second prediction block based on the third intermediate prediction block and the fourth intermediate prediction block, and determines a third prediction block from the prediction block and the second prediction block.

[0202] In some embodiments, the prediction block is determined as a weighted combination of the first intermediate prediction block weighted using a first set of weights and the second intermediate prediction block weighted using a second set of weights. In some embodiments, the first set of weights or the second set of weights includes fixed weight values. In some embodiments, at least one value in the first set of weights is different from another value in the first set of weights. In some embodiments, at least one value in the second set of weights is different from another value in the second set of weights.

[0203] In another example aspect, a video processing method includes dividing a video block into a plurality of sub-blocks according to a size of the video block or a type of encoding algorithm associated with the video block. One of the plurality of sub-blocks has a different size than other sub-blocks of the plurality of sub-blocks. The method also includes generating a motion vector prediction by applying the encoding algorithm to the plurality of sub-blocks, and further processing the video block using the motion vector prediction.

[0204] In some embodiments, the encoding algorithm includes at least one of (1) an affine prediction method, (2) a selectable temporal motion vector prediction method, (3) a spatio-temporal motion vector prediction method, (4) a bi-directional optical flow method, or (5) a frame rate up conversion method. In some embodiments, the size of the video block is M x N, where M > N, and a particular sub-block of the video block has a size of A x B, where A > B, where M, N, A, and B are integers. In some embodiments, the size of the block is M x N, where M <= N, and a particular sub-block of the video block has a size of A x B, where A <= B. In some embodiments, the method includes predicting each sub-block in a first set of sub-blocks to determine a first intermediate prediction block. In some embodiments, the method includes predicting each sub-block in a second set of sub-blocks to determine a second intermediate prediction block.

[0205] From the foregoing, it will be appreciated that specific embodiments of the disclosed technology have been described herein for purposes of illustration, but well-known modifications can be made by persons skilled in the art. Accordingly, the disclosed technology is not limited to that which has been described in the specification.

[0206] Embodiments, modules, and functional operations disclosed herein and other described embodiments, modules and functional operations can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in combinations of one or more of them. Embodiments and other embodiments disclosed can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine -readable propagated signal, or a combination of one or more of them. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also include, in addition to a processor, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus.

[0207] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0208] The processes and logic flows described herein can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, and that apparatus can also be implemented as special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0209] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0210] While this patent document contains many details, it should not be construed to limit the scope of any invention or claim in any way, but rather to describe features that are specific to certain embodiments of the present inventions. Some of the features described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0211] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring any particular order among the operations or that all illustrated operations be performed, to achieve desirable results. Additionally, the separation of various system components in the embodiments described herein should not be understood as requiring such separation in all embodiments. Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A video processing method, comprising: The video block of the current image is divided into the first group of sub-blocks according to the first pattern; The video block is divided into a second group of sub-blocks according to a second mode, wherein at least one sub-block in the second group has a different size than the sub-blocks in the first group; as well as Determine the prediction block of the video block, wherein the prediction block is a combination of a first intermediate prediction block generated from the first set of sub-blocks and a second intermediate prediction block generated from the second set of sub-blocks. In order to perform motion compensation on the video block based on reference images in the first reference image list, the video block is divided into the first group of sub-blocks; and in order to perform motion compensation on the video block based on reference images in the second reference image list, the block is divided into the second group of sub-blocks. The reference images in the first reference image list and the reference images in the second reference image list are located on different sides of the current image in time.

2. The method according to claim 1, wherein at least one of (1) an affine prediction method, (2) an optional temporal motion vector prediction method, (3) a spatiotemporal motion vector prediction method, (4) a bidirectional optical flow method, or (5) a frame rate upconversion method is used to generate the first intermediate prediction block or the second intermediate prediction block.

3. The method according to claim 1, wherein the sub-blocks in the first group or the second group have a rectangular shape.

4. The method according to any one of claims 1 to 3, wherein the sub-blocks in the first group of sub-blocks have a non-uniform shape.

5. The method according to any one of claims 1 to 3, wherein the sub-blocks in the second group of sub-blocks have a non-uniform shape.

6. The method according to any one of claims 1 to 3, comprising: The first mode or the second mode is determined based on the size of the video block.

7. The method according to any one of claims 1 to 3, wherein, in order to perform motion compensation on the video block based on reference images in the first list of reference images, the video block is divided into the first group of sub-blocks. In order to perform motion compensation on the video block from reference images in a second list of reference images that are the same as the first list of reference images, the video block is divided into the second group of sub-blocks.

8. The method of claim 7, wherein motion compensation of the video block from reference images in the second list of reference images is performed as follows: The video block is divided into a third group of sub-blocks according to the third mode; A third intermediate prediction block is generated based on the third group of sub-blocks; The video block is divided into a fourth group of sub-blocks according to a fourth mode, wherein at least one sub-block in the fourth group has a different size from the sub-blocks in the third group; A fourth intermediate prediction block is generated based on the fourth group of sub-blocks; The second prediction block is determined based on the third intermediate prediction block and the fourth intermediate prediction block; and A third prediction block is determined based on the first prediction block and the second prediction block.

9. The method according to any one of claims 1 to 3, wherein the prediction block is determined as a weighted combination of the first intermediate prediction block weighted using a first set of weights and the second intermediate prediction block weighted using a second set of weights.

10. The method of claim 9, wherein the first set of weights or the second set of weights comprises fixed weight values.

11. The method of claim 10, wherein at least one value in the first set of weights is different from another value in the first set of weights.

12. The method of claim 9, wherein at least one value in the second set of weights is different from another value in the second set of weights.

13. The method according to any one of claims 1 to 3, wherein the size of the video block is M×N, where M>N, a particular sub-block of the video block has a size of A×B, where A>B, and where M, N, A and B are integers.

14. The method according to any one of claims 1 to 3, wherein the size of the video block is M×N, where M<=N, and a particular sub-block of the video block has a size of A×B, where A<=B.

15. The method according to any one of claims 1 to 3, comprising: Perform prediction on each sub-block in the first group of sub-blocks to determine the first intermediate prediction block.

16. The method according to any one of claims 1 to 3, comprising: Perform predictions for each sub-block in the second group of sub-blocks to determine the second intermediate prediction block.

17. A video processing apparatus comprising a processor configured to implement the method of any one of claims 1 to 16.

18. A computer-readable medium storing program code that implements the method as described in any one of claims 1 to 16.

Images