Method and apparatus for sub-block based temporal motion vector prediction and reordering and refinement thereof in video coding

Abstract

A method and apparatus for video encoding and decoding using SbTMVP are provided. According to the method, a target motion offset is determined from one or more neighboring blocks, the one or more neighboring blocks comprising at least one non-neighboring spatially neighboring block of the current block. Based on the target motion offset, a co-located block in the reference picture is determined. Subblock motion information for a subblock of the current block is derived based on motion information for a corresponding subblock of the co-located block. A SbTMVP candidate of the current block is generated based on sub-block motion information of a sub-block of the current block. A current block is encoded or decoded using a motion prediction set including the SbTMVP candidates.

Description

[0001] Related references

[0002] This application claims priority to U.S. Provisional Application No. 63 / 370,508, filed on August 5, 2022. The entire contents of the above are incorporated herein by reference. Technical Field

[0003] The present invention relates to a video coding and decoding system using SbTMVP (sub-block based temporal motion vector prediction). Specifically, the present invention relates to a technology for improving the coding and decoding efficiency of SbTMVP. Background Art

[0004] Versatile video coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The standard was published as an ISO standard in February 2021: ISO / IEC23090-3:2021, Information technology - Codec representation of immersive media - Part 3: Versatile video coding. VVC is based on its previous generation High Efficiency Video Coding (HEVC) by adding more codec tools to improve codec efficiency and handle various types of video sources including three-dimensional (3D) video signals.

[0005] Figure 1AAn example adaptive inter / intra video codec system incorporating loop processing is shown. For intra prediction, prediction data is derived based on previously encoded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed at the encoder end and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by a transform (T) 118 followed by a quantization (Q) 120. The transformed and quantized residual is then encoded by an entropy encoder 122 to be included in a video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with auxiliary information (such as auxiliary information such as motion and coding modes associated with intra prediction and inter prediction) and other information (parameters associated with loop filters applied to underlying image areas, etc.). As Figure 1A As shown, auxiliary information associated with intra prediction 110, inter prediction 112 and loop filter 130 is provided to entropy encoder 122. When inter prediction mode is used, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (Inverse Quantization, referred to as IQ) 124 and inverse transformation (Inverse Transformation, referred to as IT) 126 to restore the residual. The residual is then added back to the prediction data 136 in reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in the reference picture buffer 134 and used for prediction of other frames.

[0006] like Figure 1A As shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC128 may be subject to various damages. Therefore, before the reconstructed video data is stored in the reference picture buffer 134, a loop filter 130 is generally applied to the reconstructed video data to improve the video quality. For example, a deblocking filter (DF for short), a sample adaptive offset (SAO for short), and an adaptive loop filter (ALF for short) may be used. The loop filter information may need to be incorporated into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be incorporated into the bitstream. In Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an example structure of a typical video encoder. It may correspond to a High Efficiency Video Coding (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1B The decoder shown may use similar or partially identical functional blocks as the encoder, except for transform 118 and quantization 120, since the decoder only needs inverse quantization 124 and inverse transform 126. The decoder uses an entropy decoder 140 instead of an entropy encoder 122 to decode the video bitstream into quantized transform coefficients and required codec information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 at the decoder end does not need to perform a pattern search. Instead, the decoder only needs to generate an intra-frame prediction based on the intra-frame prediction information received from the entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the intra-frame prediction information received from the entropy decoder 140 without motion estimation.

[0008] According to VVC, the input picture is divided into non-overlapping square block areas called coding tree units (CTUs), similar to HEVC. Each CTU can be divided into one or more smaller-sized coding units (CUs). The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into prediction units (PUs) as a unit to apply prediction processing such as inter-frame prediction, intra-frame prediction, etc.

[0009] The VVC standard combines various new coding and decoding tools to further improve the coding and decoding efficiency of the HEVC standard. Among the various new coding and decoding tools, some coding and decoding tools related to the present invention are reviewed below.

[0010] Sub-block based temporal motion vector prediction (SbTMVP) in VVC

[0011] VVC supports the sub-block-based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode of the CU in the current picture. SbTMVP uses the same co-located picture as TMVP. The main differences between SbTMVP and TMVP are as follows:

[0012] –TMVP predicts motion at CU level, while SbTMVP predicts motion at sub-CU level;

[0013] –TMVP obtains the temporal motion vector from the co-located block in the co-located picture (i.e., the co-located block is relative to the bottom right corner or center block of the current CU), while SbTMVP applies a motion offset before obtaining the temporal motion information from the co-located picture, where the motion offset is obtained from the motion vector of one of the spatial neighboring blocks of the current CU.

[0014] SbTMVP processing Figure 2A -B. SbTMVP predicts the motion vector of the sub-CU in the current CU in two steps. The first step is to check Figure 2A The spatial neighbor A1 in . If A1 has a motion vector that uses the co-located picture as its reference picture, then that motion vector is selected as the motion offset to be applied. If no such motion is identified, then the motion offset is set to (0,0).

[0015] In the second step, the motion offset determined in step 1 is applied (i.e., added to the coordinates of the current block) to obtain the sub-CU level motion information (motion vector and reference index) from the co-located picture, as Figure 2B shown. Figure 2B The example in assumes that the motion offset is set to the motion of block A1, where frame 220 corresponds to the current picture and frame 230 corresponds to the reference picture (i.e., the co-located picture). Then, for each sub-CU, the motion information of its corresponding block in the co-located picture (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After determining the motion information of the co-located sub-CU, it is converted into a motion vector and reference index for the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU. Figure 2B In the example, the arrows in each sub-block of the co-located picture 230 correspond to the motion vector of the co-located sub-block (the thick arrows represent L0 MVs, and the thin arrows represent L1 MVs). For the current picture 220, the arrows in each sub-block correspond to the scaled motion vector of the current sub-block (the thick arrows represent L0 MVs, and the thin arrows represent L1 MVs). If there is no motion information available for the co-located sub-CU (e.g., intra-coded sub-blocks), the default motion is used.

[0016] In VVC, the sub-block based merge mode is signaled using a combined sub-block based merge list, which contains SbTMVP candidates and affine merge candidates. The SbTMVP mode is enabled / disabled via the sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the sub-block based merge candidate list, followed by the affine merge candidate. The size of the sub-block based merge list is sent in the SPS, and the maximum allowed size of the sub-block based merge list in VVC is 5.

[0017] The sub-CU size used in SbTMVP is fixed to 8x8. Like the affine merge mode, the SbTMVP mode is only applicable to CUs whose width and height are both greater than or equal to 8.

[0018] The encoding process flow of the additional SbTMVP merge candidate is the same as that of other merge candidates, that is, for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate.

[0019] Non-adjacent spatial candidates

[0020] During the development of the VVC standard, a coding tool called non-adjacent motion vector prediction (NAMVP) was proposed in JVET-L0399 (Yu Han et al., "CE4.4.6: Improvements to merge / skip modes", ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Video Exploration Group (JVET), 12th Meeting: Macau, China, October 3-12, 2018, Document: JVET-L0399). According to the NAMVP technique, non-adjacent spatial merge candidates are inserted after the TMVP (i.e., temporal MVP) in the regular merge candidate list. The mode of the spatial merge candidate is as follows: Figure 3 The distance between non-adjacent spatial candidates and the current codec block is based on the width and height of the current codec block. Figure 3 In , each small number box corresponds to a NAMVP candidate, and the candidates are sorted by distance (as shown by the numbers in the boxes).

[0021] Multi-channel decoder-side motion vector refinement (MP-DMVR)

[0022] Multi-pass decoder-side motion vector refinement is applied. In the first pass, bilateral matching (BM) is applied to the codec block. In the second pass, BM is applied to each 16x16 sub-block within the codec block. In the third pass, the MV in each 8x8 sub-block is refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for spatial and temporal motion vector prediction.

[0023] First pass - Block-based bilateral matching MV refinement

[0024] In the first pass, the refined MV is derived by applying BM to the codec block. Similar to the decoder-side motion vector refinement (DMVR), in the bidirectional prediction operation, the refined MV is searched around the two initial MVs (i.e., MV0 and MV1) in the reference picture lists L0 and L1. Based on the minimum bilateral matching cost between the two reference blocks in L0 and L1, the refined MVs (i.e., MV0_pass1 and MV1_pass1) are derived around the initial MV.

[0025] BM performs local search and obtains integer sample precision intDeltaMV. The local search adopts a 3×3 square search pattern, with a circular search range of [–sHor, sHor] in the horizontal direction and a circular search range of [–sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension, and the maximum values ​​of sHor and sVer are both 8.

[0026] The bilateral matching cost is calculated as follows: bilCost = mvDistanceCost + sadCost. When the block size cbW*cbH is greater than 64, the MRSAD cost function is used to eliminate the DC effect of the distortion between reference blocks. When the bilCost of the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV local search terminates. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern, and the search for the minimum cost continues until the end of the search range is reached.

[0027] The existing fractional sample refinement is further applied to obtain the final deltaMV. The refined MV after the first pass is as follows:

[0028] MV0_pass1=MV0+deltaMV

[0029] MV1_pass1=MV1–deltaMV

[0030] Second pass - sub-block based bilateral matching MV refinement

[0031] In the second pass, the refined MVs are derived by applying BM to the 16×16 grid sub-blocks. For each sub-block, the refined MVs are searched in the reference picture lists L0 and L1 around the two MVs obtained during the first pass (e.g., MV0_pass1 and MV1_pass1). The refined MVs (i.e., MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1.

[0032] For each sub-block, BM performs a full search to derive integer sample precision intDeltaMV. The search range of the full search is [–sHor, sHor] in the horizontal direction and [–sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension and the maximum value of sHor and sVer is 8.

[0033] The bilateral matching cost is calculated by applying the cost factor to the SATD cost between two reference sub-blocks as follows: bilCost = satdCost * costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into up to 5 diamond search areas, such as Figure 4 As shown, 5 search areas are displayed in 5 different tones. Each search area is assigned a costFactor, which is determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond area is processed in order starting from the center of the search area. In each area, the search points are processed in raster scan order, starting from the upper left corner to the lower right corner of the area. When the minimum bilCost in the current search area is less than or equal to the threshold of sbW*sbH, the integer-pixel full search is terminated; otherwise, the integer-pixel full search continues to the next search area until all search points are checked. In addition, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than or equal to the threshold of the block area, the search process is terminated.

[0034] The existing VVC DMVR fractional sample refinement is further applied to obtain the final deltaMV (sbIdx2). Then the second pass of refined MV is obtained as follows:

[0035] ●MV0_pass2(sbIdx2)=MV0_pass1+deltaMV(sbIdx2)

[0036] ●MV1_pass2(sbIdx2)=MV1_pass1–deltaMV(sbIdx2)

[0037] The third pass - sub-block-based bidirectional optical flow MV refinement

[0038] In the third pass, the refined MV is derived by applying BDOF to the 8×8 grid subblocks. For each 8×8 subblock, BDOF refinement is used to derive scaled Vx and Vy without clipping from the refined MV of the second pass parent subblock. The derived bioMv(Vx,Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32.

[0039] The refined MVs of the third pass (e.g., MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) are derived as follows:

[0040] ●MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv

[0041] ·MV1_pass3(sbIdx3)=MV0_pass2(sbIdx2)–bioMv

[0042] Adaptive decoder-side motion vector refinement

[0043] The adaptive decoder-side motion vector refinement method is an extension of multi-pass DMVR, which consists of two new merge modes for refining MV in only one direction (L0 or L1) of bidirectional prediction of merge candidates that meet DMVR conditions. Multi-pass DMVR processing is applied to the selected merge candidates to refine the motion vector, but in the first pass (i.e. PU level) DMVR, MVD0 or MVD1 is set to zero.

[0044] The merge candidates for the new merge mode come from spatially adjacent coded blocks, TMVP, non-adjacent blocks, HMVP, and paired candidates, similar to the regular merge mode. The difference is that only candidates that meet the DMVR conditions are added to the candidate list. The two new merge modes use the same merge candidate list. If the BM candidate list contains inherited BCW weights, the DMVR processing remains unchanged unless the distortion is calculated using MRSAD or MRSATD (if the weights are unequal and the bidirectional prediction is weighted with BCW weights). The encoding and decoding of the merge index is the same as the regular merge mode.

[0045] Template matching for MV refinement

[0046] Template matching (TM) is a decoder-side MV derivation method that refines the motion information of the current CU by finding the closest match between the template in the current picture 510 (i.e., the top 514 and / or left 516 neighboring blocks of the current CU 512) and the blocks in the reference picture 520 (i.e., the same size as the template, blocks 524 and 526), ​​such as Figure 5 As shown. Figure 5In the search range 522 of [-8, +8] pixels around the position 528 in the reference picture 520 pointed to by the initial MV 530, a better MV is searched around the initial motion 530 of the current CU 512 of the current picture 510. JVET-J0021 (Yi-Wen Chen et al., "Description of SDR, HDR and 360° Video Codec Proposal by Qualcomm and Technicolor - Low Complexity and High Complexity Versions", ITU-T SG16 WP3 and ISO / IEC JTC 1 / SC 29 / WG11 Joint Collaboration Team on Video Coding (JCT-VC), 10th Meeting: San Diego, USA, April 10-20, 2018, Document: JVET-J0021) is used with the following modifications: the search step size is determined according to the AMVR mode, and in the merge mode, the TM can be cascaded with the bilateral matching process.

[0047] In AMVP mode, the MVP candidate is determined based on the template matching error to select the MVP candidate with the minimum difference between the current block template and the reference block template. Then TM is performed only for this specific MVP candidate for MV refinement. TM uses an iterative diamond search starting from full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) and refining this MVP candidate within a [-8, +8] pixel search range. The AMVP candidate can be further refined using a cross search with a search accuracy of full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode), and then using half-pixels and quarter-pixels in turn according to the AMVR mode, as shown in Table 1. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. During the search process, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than or equal to the threshold of the block area, the search process terminates.

[0048] Table 1. Search patterns of AMVR and merge patterns with AMVR.

[0049]

[0050]

[0051] In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 1, TM can be performed all the way to 1 / 8 pixel MVD accuracy or skip MVDs exceeding half-pixel MVD accuracy, depending on whether an optional interpolation filter (AMVR for half-pixel mode) is used based on the merged motion information. In addition, when TM mode is enabled, template matching can be used as an independent process between block-based and sub-block-based bilateral matching (BM) methods or as an additional MV refinement process, depending on whether BM can be enabled according to its enabling condition check.

[0052] Adaptive Re-ranking of Merge Candidates using Template Matching (ARMC-TM)

[0053] The merge candidates are adaptively re-ranked according to the cost evaluated using Template Matching (TM). The re-ranking method can be applied to the regular merge mode, Template Matching (TM) merge mode, and Affine merge mode (excluding SbTMVP candidates). For TM merge mode, the merge candidates are re-ranked before the refinement process.

[0054] After building the merge candidate list, the merge candidates are divided into multiple subgroups. For the normal merge mode and TM merge mode, the subgroup size is set to 5. For the affine merge mode, the subgroup size is set to 3. The merge candidates in each subgroup are reordered in ascending order according to the cost value based on template matching. For ARMC-TM, the candidates in a subgroup are skipped if the subgroup meets the following 2 conditions: (1) the subgroup is the last subgroup, (2) the subgroup is not the first subgroup. For simplicity, the merge candidates in the last subgroup (but not the merge candidates in the first subgroup) are not reordered.

[0055] The template matching cost of the merged candidate is measured as the sum of absolute differences (SAD) between the template samples of the current block and its corresponding reference samples. The template includes a set of reconstructed samples adjacent to the current block. The reference samples of the template are located by the motion information of the merged candidate.

[0056] When the merge candidate uses bidirectional prediction, the reference sample of the merge candidate template is also generated by bidirectional prediction, such as Figure 6 As shown. Figure 6 , block 612 corresponds to a current block in current picture 610, and blocks 622 and 632 correspond to reference blocks in reference pictures 620 and 630 in list 0 and list 1, respectively. Templates 614 and 616 are used for current block 612, templates 624 and 626 are used for reference block 622, and templates 634 and 636 are used for reference block 632. Motion vectors 640, 642, and 644 are merge candidates in list 0, and motion vectors 660, 662, and 664 are merge candidates in list 1.

[0057] For the sub-block based merge candidates with sub-block size equal to Wsub×Hsub, the template above contains multiple sub-templates of size Wsub×1, and the template on the left contains multiple sub-templates of size 1×Hsub. Figure 7 As shown in , the motion information of the sub-block in the first row and first column of the current block is used to derive the reference sample of each sub-template. Figure 7, block 712 corresponds to the current block in the current picture 710, and block 722 corresponds to the co-located block in the reference picture 720. Each small square in the current block and the co-located block corresponds to a sub-block. The dot-filled areas on the left and top of the current block correspond to the template of the current block. The boundary sub-blocks are labeled from A to G. The arrows associated with each sub-block correspond to the motion vector of the sub-block. The reference sub-blocks (labeled Aref to Gref) are positioned according to the motion vectors associated with the boundary sub-blocks.

[0058] The present invention discloses a solution for further improving the performance of SbTMVP. Summary of the invention

[0059] The present invention discloses a method and device for video encoding and decoding using SbTMVP (sub-block based temporal motion vector prediction). According to the method, input data associated with a current block is received, wherein the input data includes pixel data of the current block to be encoded by the encoder end or encoded data associated with the current block to be decoded by the decoder end. A target motion offset is determined from one or more adjacent blocks, wherein the one or more adjacent blocks include at least one non-adjacent spatial adjacent block of the current block. Based on the target motion offset, a co-located block in a reference picture is determined. Based on the motion information of the corresponding sub-block of the co-located block, sub-block motion information of the sub-block of the current block is derived. Based on the sub-block motion information of the sub-block of the current block, an SbTMVP candidate of the current block is generated. The current block is encoded or decoded using a motion prediction set including the SbTMVP candidate.

[0060] In one embodiment, the SbTMVP candidate is for skip mode, merge mode, direct mode, intra mode, inter mode, IBC mode, or a combination thereof.

[0061] In one embodiment, a target motion offset is derived from one or more motion offset candidates from the one or more neighboring blocks based on one or more schemes involving TM (template matching), ARMC-TM (adaptive reordering of merged candidates using template matching), or both. In one embodiment, the target motion offset is derived by refining a motion offset candidate with one or more templates of the current block based on TM. In another embodiment, two or more motion offset candidates from one or more neighboring blocks of the current block, one or more neighboring non-neighboring blocks of the current block, or both are determined, and wherein the two or more motion offset candidates are reordered according to ARMC-TM. In one embodiment, the first motion offset candidate of the reordered two or more motion offset candidates is selected as the target motion offset. In another embodiment, the first motion offset candidate of the reordered two or more motion offset candidates is refined by TM and used as the target motion offset.

[0062] In one embodiment, two or more motion offset candidates from one or more neighboring blocks of the current block, one or more neighboring non-neighboring blocks of the current block, or both are determined, wherein the two or more motion offset candidates are refined according to TM and then reordered according to ARMC-TM. In one embodiment, a first motion offset candidate of the reordered two or more motion offset candidates is selected as the target motion offset. In another embodiment, if one of the two or more motion offset candidates points to a co-located block without valid motion information, the TM cost or ARMC-TM cost of the one motion offset candidate is set to a large value so that the one motion offset candidate is skipped from being selected as the target motion offset.

[0063] In one embodiment, if the sub-block motion information of the target sub-block is not available, the default motion is refined by TM or BM (bilateral matching) and used as the sub-block motion information of the target sub-block. In another embodiment, if the sub-block motion information of the target sub-block is not available, the default motion is refined by TM using one or more templates of the current block. In another embodiment, if the sub-block motion information of the target sub-block is not available, the default motion is refined by BM. In another embodiment, if the sub-block motion information of the target sub-block is not available, the default motion is refined by TM and then by BM. In another embodiment, if the sub-block motion information of the target sub-block is not available, the default motion is refined by BM and then by TM. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1A An example adaptive inter / intra video codec system incorporating loop processing is shown.

[0065] Figure 1B Shown with Figure 1A The decoder corresponding to the encoder in .

[0066] Figure 2A An example of sub-block based temporal motion vector prediction (SbTMVP) in VVC is shown, where spatial neighboring blocks are checked for motion information.

[0067] Figure 2B An example of SbTMVP is shown, deriving a sub-CU motion field by applying motion offsets from spatial neighbors and scaling the motion information from the corresponding co-located sub-CU.

[0068] Figure 3 An exemplary pattern of non-adjacent spatial merging candidates is shown.

[0069] Figure 4 Shown are 5 diamond-shaped search areas for multi-pass decoder-side motion vector refinement.

[0070] Figure 5 An example of template matching for refining an initial MV by searching an area around the initial MV is shown.

[0071] Figure 6 Examples of templates for a current block and corresponding reference blocks are shown for measuring the matching cost associated with merge candidates.

[0072] Figure 7 An example of a template of a block having sub-block motion and a reference sample of the template using motion information of a sub-block of a current block is shown.

[0073] Figure 8 A flow chart illustrating an exemplary video codec system utilizing SbTMVP with motion offsets derived from one or more neighboring blocks including at least one non-immediate neighboring block according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] It is readily understood that the components of the present invention as generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the systems and methods of the present invention, as shown in the figures, is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. References in this specification to "an embodiment," "some embodiments," or similar language mean that specific features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in an embodiment" or "in some embodiments" appearing in various places throughout this specification do not necessarily all refer to the same embodiment.

[0075] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. However, those skilled in the relevant art will recognize that the present invention may be implemented without one or more specific details or using other methods, components, etc. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring various aspects of the present invention. The illustrated embodiments of the present invention will be best understood by reference to the accompanying drawings, in which the same parts are represented by the same numbers throughout. The following description is by way of example only, and simply describes some selected embodiments of the apparatus and method consistent with the present invention as claimed herein.

[0076] As mentioned above, in conventional SbTMVP, motion offset is used to locate temporally co-located blocks. The motion information of the co-located blocks is then used to derive the motion information of the sub-blocks of the current block. The motion offset is determined based on spatially adjacent blocks. In order to improve the encoding and decoding efficiency of SbTMVP for Skip, Merge, Direct, Intra mode, Inter mode and / or IBC mode, the following scheme is disclosed.

[0077] In one embodiment, the motion vectors of non-adjacent spatial neighboring blocks are used as motion offsets. In conventional methods, only the immediately adjacent neighboring blocks are used to derive motion offsets. If there is no motion information from the adjacent blocks (e.g., none of the spatially adjacent blocks are intra-coded), the motion offset is set to (0, 0), which may not provide good prediction. Therefore, the present invention proposes to allow the use of motion information from one or more non-immediately adjacent neighboring blocks to derive motion offsets. Motion offsets based on non-adjacent spatial neighboring blocks are expected to provide better predictions.

[0078] In one embodiment, motion offset derivation is related to TM and / or ARMC-TM. Some examples are listed below, however, the present invention is not limited to these examples:

[0079] - Refine the motion offset by TM using the CU template.

[0080] - Reordering two or more motion vectors of adjacent and / or non-adjacent spatially adjacent blocks using ARMC-TM and then selecting the first one or more as motion offsets. As known from ARMC-TM, the reordering is performed based on the template matching costs associated with the two or more motion vectors. The candidates are reordered first with the smallest template matching cost. In other words, the candidate with the smallest template matching cost is selected.

[0081] - Two or more motion vectors of adjacent and / or non-adjacent spatially neighboring blocks are refined using TM, then reordered by ARMC-TM, and then the first one or more are selected as motion offsets.

[0082] – Two or more motion vectors of adjacent and / or non-adjacent spatially neighboring blocks are reordered by ARMC-TM, and then the first one or more are selected as motion offsets, which are then refined by TM.

[0083] – In the TM or ARMC-TM process, if considering a motion vector as a motion offset would result in an unusable SbTMVP candidate (i.e., default motion is unusable), the TM or ARMC-TM cost associated with the motion vector is set to a large value or the motion vector is skipped.

[0084] In one embodiment, the default motion is further refined by the TM and / or BM:

[0085] – Default motion is refined by TM and CU templates.

[0086] – Default motion is refined by BM.

[0087] – Default motion is refined by TM and BM.

[0088] – Default motion is refined by BM and TM.

[0089] Any of the above-mentioned SbTMVP methods with motion offsets derived from one or more neighboring blocks (including at least one non-immediate neighboring block) can be implemented in an encoder and / or a decoder. For example, any of the proposed SbTMVP methods can be implemented in an inter-coding module and / or a merge / AMVP candidate derivation module (e.g. Figure 1A inter-frame prediction 112 in the decoder) or the motion compensation module of the decoder (e.g. Figure 1B 152) and / or the merge / AMVP candidate derivation module. Alternatively, any of the proposed methods can be implemented as a circuit coupled to the inter-frame coding module and / or the merge / AMVP candidate derivation module of the encoder and / or the motion compensation module and / or the merge / AMVP candidate derivation module of the decoder. While the inter-frame prediction 112 and MC152 are shown as separate processing units to support the SbTMVP method, they can correspond to executable software or firmware code stored on a medium (such as a hard disk or flash memory) for a CPU (central processing unit) or a programmable device (such as a DSP (digital signal processor) or an FPGA (field programmable gate array)).

[0090] Figure 8 A flowchart of an exemplary video codec system using an SbTMVP with a motion offset derived from one or more neighboring blocks including at least one non-immediate neighboring block according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented as program codes that can be executed on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, in step 810, input data associated with a current block is received, wherein the input data includes pixel data of the current block to be encoded by the encoder side or encoded data associated with the current block to be decoded by the decoder side. In step 820, a target motion offset is determined from one or more neighboring blocks including at least one non-adjacent spatial neighboring block of the current block. In step 830, a co-located block in a reference image is determined based on the target motion offset. In step 840, sub-block motion information of a sub-block of the current block is derived based on motion information of a corresponding sub-block of the co-located block. In step 850, an SbTMVP candidate is generated for the current block based on the sub-block motion information of the sub-block of the current block. In step 860, the current block is encoded or decoded using the motion prediction set including the SbTMVP candidate.

[0091] The flowchart shown is intended to illustrate an example of video encoding and decoding according to the present invention. Without departing from the spirit of the present invention, those skilled in the art may modify each step, rearrange the steps, split the steps or combine the steps to implement the present invention. In the present disclosure, specific syntax and semantics are used to illustrate examples to implement embodiments of the present invention. A technician may implement the present invention by replacing the above syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present invention.

[0092] The above description is presented to enable one of ordinary skill in the art to implement the present invention provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but to conform to the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are described in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be implemented.

[0093] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, one embodiment of the present invention can be a program code integrated into one or more circuits in a video compression chip or integrated into video compression software to perform the processing described herein. An embodiment of the present invention can also be a program code to be executed on a digital signal processor (Digital Signal Processor, referred to as DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (field programmable gate array, referred to as FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software code or firmware code that defines the specific method embodied in the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, the different code formats, styles and languages ​​of the software code and other ways of configuring the code to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0094] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects as illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A video encoding and decoding method, comprising: Receiving input data associated with a current block, wherein the input data includes pixel data of the current block to be encoded by an encoder or encoded data associated with the current block to be decoded by a decoder; determining a target motion offset from one or more neighboring blocks, the one or more neighboring blocks including at least one non-adjacent spatial neighboring block of the current block; Determining a co-located block in a reference picture based on the target motion offset; deriving sub-block motion information of multiple sub-blocks of the current block based on motion information of multiple corresponding sub-blocks of the co-located block; generating a SbTMVP (sub-block based temporal motion vector prediction) candidate of the current block based on the sub-block motion information of the plurality of sub-blocks of the current block; and The current block is encoded or decoded using a motion prediction set including the SbTMVP candidate.

2. The video encoding and decoding method according to claim 1, characterized in that: The SbTMVP candidate is for skip mode, merge mode, direct mode, inter mode, IBC mode or a combination thereof.

3. The video encoding and decoding method according to claim 1, characterized in that: The target motion offset is derived from one or more motion offset candidates of the one or more neighboring blocks based on one or more schemes involving TM (Template Matching), ARMC-TM (Adaptive Reordering of Merge Candidates with Template Matching), or both.

4. The video encoding and decoding method according to claim 3, characterized in that: The target motion offset is derived by refining a motion offset candidate based on TM using one or more templates of the current block.

5. The video encoding and decoding method according to claim 3, characterized in that: Two or more motion offset candidates are determined from one or more neighboring blocks of the current block, one or more neighboring non-neighboring blocks of the current block, or both, and wherein the two or more motion offset candidates are reordered according to the ARMC-TM.

6. The video encoding and decoding method according to claim 5, characterized in that: A first motion offset candidate among the reordered two or more motion offset candidates is selected as the target motion offset.

7. The video encoding and decoding method according to claim 5, characterized in that: The first motion offset candidate of the reordered two or more motion offset candidates is refined by the TM and used as a target motion offset.

8. The video encoding and decoding method according to claim 3, characterized in that: Two or more motion offset candidates are determined from one or more neighboring blocks of the current block, one or more neighboring non-neighboring blocks of the current block, or both, and wherein the two or more motion offset candidates are refined according to the TM and reordered according to the ARMC-TM.

9. The video encoding and decoding method according to claim 8, characterized in that: A first motion offset candidate among the reordered two or more motion offset candidates is selected as the target motion offset.

10. The video encoding and decoding method according to claim 8, characterized in that: If one of the two or more motion offset candidates points to a co-located block without valid motion information, the TM cost or ARMC-TM cost of the one motion offset candidate is set to a large value so that the one motion offset candidate is not selected as the target motion offset.

11. The video encoding and decoding method according to claim 1, characterized in that: If the sub-block motion information of the target sub-block is not available, a default motion is refined through TM or BM (Bilateral Matching) and used as the sub-block motion information of the target sub-block.

12. The video encoding and decoding method according to claim 11, characterized in that: If the sub-block motion information of the target sub-block is not available, the default motion is refined by TM using one or more templates of the current block.

13. The video encoding and decoding method according to claim 11, characterized in that: If the sub-block motion information of the target sub-block is not available, the default motion is refined by BM.

14. The video encoding and decoding method according to claim 11, characterized in that: If the sub-block motion information of the target sub-block is not available, the default motion is refined by TM and then by BM.

15. The video encoding and decoding method according to claim 11, characterized in that: If the sub-block motion information of the target sub-block is not available, the default motion is refined by BM and then by TM.

16. An apparatus for video encoding and decoding using motion vector prediction, the apparatus comprising one or more electronic devices or processors, configured to: Receive input data associated with the current block, where The input data includes pixel data of the current block to be encoded by the encoder or encoded data associated with the current block to be decoded by the decoder; determining a target motion offset from one or more neighboring blocks, the one or more neighboring blocks including at least one non-adjacent spatial neighboring block of the current block; Determining a co-located block in a reference picture based on the target motion offset; deriving sub-block motion information of multiple sub-blocks of the current block based on motion information of multiple corresponding sub-blocks of the co-located block; Based on the sub-block motion information of the multiple sub-blocks of the current block, generating a SbTMVP (sub-block based temporal motion vector prediction) candidate of the current block; as well as The current block is encoded or decoded using a motion prediction set including the SbTMVP candidate.

Images