Method and apparatus for candidate skip of predictor refinement in video coding

Abstract

Methods and apparatuses for using motion refinement with reduced bandwidth are disclosed. According to one method of the invention, a predictor refinement process is used to generate motion refinement for a current block by searching among a plurality of motion vector candidates using reference data including a target motion compensated reference block, wherein if a target motion vector candidate requires target reference data from the target motion compensated reference block that is located outside of a valid reference block, the target motion vector candidate is excluded from the search among the plurality of motion vector candidates or a replacement motion vector candidate that is closer to the center of the corresponding block of the current block is used as a replacement for the target motion vector candidate. In another method, if a target motion vector candidate belongs to one or more target fractional pixel positions, an interpolation filter with a shortened tap length is applied to the target motion vector candidate.

Description

[0001] Case Analysis

[0002] This invention is a divisional application of patent application number 201880006552.X, entitled "Method and Apparatus for Predicting Candidate Skipping in Video Encoding and Decoding".

[0003] Cross-references

[0004] This invention claims priority to U.S. Provisional Patent Application No. 62 / 445,287, filed January 12, 2017. This U.S. Provisional Patent Application is incorporated herein by reference. Technical Field

[0005] This invention relates to motion compensation using predictor refinement processes, such as model-based motion vector derivation (PMVD), bidirectional optical flow (BIO), or decoder-side motion vector refinement (DMVR), to refine the motion of prediction blocks. In particular, this invention relates to bandwidth reduction related to decoder-side motion vector refinement processes. Background Technology

[0006] Model-based derivation of motion vectors

[0007] A model-based motion vector derivation method is disclosed in VCEG-AZ07 (Jianle Chen, et al., Further improvements to HMKTA-1.0, ITU-Telecommunications Standardization Sector, Study Group 16 Question 6, VideoCoding Experts Group (VCEG), 52nd Meeting: 19–26 June 2015, Warsaw, Poland). According to VCEG-AZ07, the decoder-side motion vector derivation method uses two Frame Rate Up-Conversion (FRUC) modes. One of the FRUC modes is called bilateral matching, used for B-segments, and the other is called template matching, used for either P-segments or B-segments. Figure 1An example of a frame rate upscaling bilateral matching mode is shown, where the motion information of the current block 110 is derived based on two reference images. The motion information of the current block is derived by finding the best match between two blocks (i.e., 120 and 130) along the motion trajectory of the current block 110 in two different reference images (i.e., Ref0 and Ref1). Under the assumption of continuous motion trajectories, the motion vector MV0 associated with Ref0 and the motion vector MV1 associated with Ref1 pointing to reference blocks 120 and 130 should be proportional to the temporal distance between the current image (i.e., Cur pic) and the two reference images (Ref0 and Ref1), i.e., TD0 and TD1.

[0008] Figure 2 An example of frame rate upscaling template matching mode is shown. The adjacent regions (i.e., 220a and 220b) of the current block 210 in the current image (i.e., Cur pic) are used as templates to match the reference image (i.e., ... Figure 2 The corresponding templates (i.e., 230a and 230b) in Ref0 are matched. The best match between template 220a / 220b and template 230a / 230b will determine the decoder's derivation of motion vector 240. Although Figure 2 Only Ref0 is shown in the image; Ref1 can also be used as a reference image.

[0009] According to VCEG-AZ07, FRUC_mrg_flag is signaled when merge_flag or skip_flag is true. If FRUC_mrg_flag is 1, then FRUC_merge_mode is signaled to indicate whether bilateral match merge mode or template match merge mode is selected. If FRUC_mrg_flag is 0, it means that the regular merge mode is used in this case, and a merge index is signaled. In video encoding and decoding, to improve encoding and decoding efficiency, motion vector prediction (MVP) is used to predict the motion vectors of blocks, generating a candidate list in the process. The merge candidate list can be used for merge mode encoding and decoding blocks. When merge mode is used for encoding and decoding blocks, the block's motion information (such as motion vectors (MV)) can be represented by a candidate motion vector from the merge motion vector list. Therefore, instead of directly transmitting the block's motion information, the merge index is transmitted to the decoder. The decoder maintains the same merge list and uses the merge index to retrieve the merge candidate signaled by the merge index. Generally, the merge candidate list includes a small number of candidates, and transmitting the merge index is much more efficient than transmitting motion information. When a block is encoded and decoded in merge mode, its motion information is "merged" with the motion information of neighboring blocks by transmitting the merge index, rather than explicitly transmitting it. However, the prediction residual is still transmitted. In cases where the prediction residual is zero or very small, the prediction residual is "skipped" (i.e., skipped mode), and the block is encoded and decoded using a skipped mode with a merge index to identify the merged motion vector in the merge list.

[0010] Although the term FRUC refers to motion vector derivation with frame rate upconversion, the underlying technology is designed for the decoder to derive one or more merged motion vector candidates without explicitly transmitting motion information. Therefore, in this application, frame rate upconversion is also referred to as decoder-derived motion vectors. Since template matching is a model-based motion vector derivation technique, the template matching method with frame rate upconversion is also referred to as model-based motion vector derivation in this invention.

[0011] In the decoder-side motion vector derivation method, a new temporal motion vector prediction, called the temporally deduced motion vector prediction, is derived by scanning all motion vectors in all reference images. To derive the LIST_0 temporally deduced motion vector prediction, for each LIST_0 motion vector in the LIST_0 reference image, this motion vector is scaled to point to the current image. The 4x4 block in the current image pointed to by the scaled motion vector is the target current block. This motion vector is further scaled to point to the reference image in LIST_0 where refldx equals 0 for the target current block. This further scaled motion vector is stored in the LIST_0 motion vector field for use with the target current block. Figure 3A and Figure 3B Examples of deriving time-derived motion vector predictions for LIST_0 and LIST_1 are shown, respectively. Figure 3A and Figure 3B In this context, each small square block corresponds to a 4x4 block. The temporal extrapolation motion vector prediction process scans all motion vectors within all 4x4 blocks in all reference images to generate temporal extrapolation LIST_0 and temporal extrapolation LIST_1 motion vector predictions for the current image. For example, in... Figure 3A In the diagram, blocks 310, 312, and 314 correspond to 4x4 blocks of the current image (Cur.pic), the LIST_0 reference image with index 0 (refidx = 0), and the LIST_0 reference image with index 1 (refidx = 1), respectively. Motion vectors 320 and 330 of the two blocks in the LIST_0 reference image with index 1 are known. Then, by scaling motion vectors 320 and 330, temporally derived motion vector predictions 322 and 332 can be derived, respectively. The scaled motion vector predictions are then assigned to the corresponding blocks. Similarly, in... Figure 3B In the diagram, blocks 340, 342, and 344 correspond to the current image (Cur.pic), the LIST_1 reference image with index 0 (i.e., refidx = 0), and the 4x4 blocks of the LIST_1 reference image with index 1 (i.e., refidx = 1), respectively. Motion vectors 350 and 360 of the two blocks in the LIST_1 reference image with index 1 are known. Then, by scaling motion vectors 350 and 360, temporal derivation motion vector predictions 352 and 362 can be derived, respectively.

[0012] For both bilateral matching and template matching merging modes, two-stage matching is employed. The first stage is PU-level matching, and the second stage is sub-PU-level matching. In PU-level matching, multiple initial motion vectors from LIST_0 and LIST_1 are selected separately. These motion vectors include motion vectors from merging candidates (i.e., regular merging candidates, such as those specified in the HEVC standard) and motion vectors from time-derived motion vector predictions. Two distinct sets of initial motion vectors are generated for two lists. For each motion vector in one list, a motion vector pair is generated by including this motion vector and its mirrored counterpart derived by scaling it to the other list. For each motion vector pair, two reference blocks are compensated using this motion vector pair. The sum of absolutely differences (SAD) of the two blocks is calculated. The motion vector pair with the smallest sum of absolutely differences is selected as the optimal motion vector pair.

[0013] After the optimal motion vectors are derived for use in the prediction unit, a diamond search is performed to refine the motion vector pairs. The refinement precision is 1 / 8 pixel. The refinement search range is limited to ±1 pixel. The final motion vector pairs are those derived from the prediction unit layer. Diamond search is a well-known fast block-matching motion estimation algorithm in the field of video encoding and decoding. Therefore, the details of the diamond search algorithm will not be elaborated here.

[0014] For the second-stage sub-prediction unit layer search, the current prediction unit is divided into sub-prediction units. The depth of the sub-prediction units (e.g., 3) is signaled in the sequence parameter set (SPS). The minimum sub-prediction unit size is a 4x4 block. For each sub-prediction unit, multiple starting motion vectors are selected from LIST_0 and LIST_1, including the motion vector of the prediction unit layer derivation motion vector, the zero motion vector, the HEVC collocated TMVP of the current sub-prediction unit and the lower right block, the time-derived motion vector prediction of the current sub-prediction unit, and the motion vectors of the left / upper prediction unit / sub-prediction unit. By using a mechanism similar to the prediction unit layer search, the optimal motion vector pair of the sub-prediction unit is determined. A diamond search is performed to refine the motion vector pair. Motion compensation of the sub-prediction unit is performed to generate the predictor for this sub-prediction unit.

[0015] For template matching merging mode, the reconstructed pixels in the top 4 columns and left 4 rows are used to form the template. Template matching is performed to find the best-matching template and its corresponding motion vector. Two-stage matching is also applied to template matching. In predictor layer matching, multiple initial motion vectors from LIST_0 and LIST_1 are selected separately. These motion vectors include motion vectors from merge candidates (i.e., regular merge candidates, such as those specified in the HEVC standard) and motion vectors from time-derived motion vector predictions. Two different sets of initial motion vectors are generated for two lists. For each motion vector in one list, the sum cost of the absolute differences of the templates with this motion vector is calculated. The motion vector with the minimum cost is the best motion vector. A diamond search is then performed to refine this motion vector. The refinement precision is 1 / 8 pixel. The refinement search range is limited to ±1 pixel. The final motion vector is the motion vector derived from the predictor layer. Motion vectors in LIST_0 and LIST_1 are generated separately.

[0016] For the second-stage sub-prediction unit layer search, the current prediction unit is divided into sub-prediction units. The depth of the sub-prediction units (e.g., 3) is signaled in the sequence parameter set. The minimum sub-prediction unit size is a 4x4 block. For each sub-prediction unit located at the left or top prediction unit boundary, multiple initial motion vectors are selected from LIST_0 and LIST_1, including the motion vector of the prediction unit layer derivation motion vector, the zero motion vector, the HEVC co-occurrence TMVP of the current sub-prediction unit and the lower right block, the time-derived motion vector prediction of the current sub-prediction unit, and the motion vectors of the left / sub-prediction unit and the upper / sub-prediction unit. By using a mechanism similar to the prediction unit layer search, the optimal motion vector pair of the sub-prediction unit is determined. A diamond search is performed to refine this motion vector pair. Motion compensation for this sub-prediction unit is performed to generate the predictor for this sub-prediction unit. For prediction cells that are not located at the left prediction cell boundary or the top prediction cell boundary, the second-stage sub-prediction cell layer search is not used, and the corresponding motion vector is set to be equal to the motion vector in the first stage.

[0017] In this decoder motion vector derivation method, template matching is also used to generate motion vector predictions for inter-frame mode encoding and decoding. When a reference image is selected, template matching is performed to find the best template on the selected reference image. Its corresponding motion vector is the derivation motion vector prediction. This motion vector prediction is inserted into the first position of AMVP. AMVP stands for Advanced Motion Vector Prediction, where a candidate list is used, and the current motion vector is coded predictively. The motion vector difference between the current motion vector and the selected motion vector candidate in the candidate list is coded.

[0018] Bidirectional Optical Flow (BIO)

[0019] Bidirectional optical flow is a motion estimation / motion compensation technique disclosed in JCTVC-C204 (E. Alshina, et al., Bi-directional optical flow, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 3rd Meeting: Guangzhou, CN, 7-15 October, 2010, Document: JCTVC-C204) and VCEG-AZ05 (E. Alshina, et al., Known tools performance investigation for next generation video coding, ITU-T SG 16 Question 6, Video Coding Experts Group (VCEG), 52nd Meeting: 19–26 June 2015, Warsaw, Poland, Document: VCEG-AZ05). Bidirectional optical flow is based on... Figure 4 The assumptions about optical flow and steady-state motion shown derive sample layer motion refinement, where the current pixel 422 in slice B (i.e., bidirectional prediction slice) 420 is predicted from a pixel in reference image 0 and a pixel in reference image 1. Figure 4 As shown, the current pixel 422 is predicted from pixel B (412) in reference image 1 (410) and pixel A (432) in reference image 0 (430). Figure 4In this model, vx and vy are pixel displacement vectors in the x and y directions, respectively, derived using a bidirectional optical flow model. This model is only applicable to ground truth bidirectional prediction blocks, which are predicted from two reference images corresponding to the previous and subsequent frames. In VCEG-AZ05, bidirectional optical flow uses a 5x5 window to derive motion refinement for each sample. Therefore, for an NxN block, the motion compensation results and corresponding gradient information of the (N+4)x(N+4) block are needed to derive the sample-based motion refinement for the NxN block. According to VCEG-AZ05, a 6-tap gradient filter and a 6-tap interpolation filter are used to generate the gradient information for bidirectional optical flow. Therefore, the computational complexity of bidirectional optical flow is significantly higher than that of traditional bidirectional prediction. To further improve the performance of bidirectional optical flow, the following method is proposed.

[0020] In VCEG-AZ05, bidirectional optical flow is implemented on the HEVC reference software and is always applicable to blocks predicted with true bidirectional propagation. In HEVC, an 8-tap interpolation filter for the luma component and a 4-tap interpolation filter for the chroma component are used to perform fractional motion compensation. Considering a 5×5 window of a pixel to be processed in an 8×8 coding unit (CU) in bidirectional optical flow, the bandwidth required in the worst case increases from (8+7)x(8+7)x2 / (8x8) = 7.03 reference pixels per current pixel to (8+7+4)x(8+7+4)x2 / (8x8) = 11.288 reference pixels.

[0021] Decoder-side motion vector refinement

[0022] In JVET-D0029 (Xu Chen, et al., “Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching”, Joint Video Exploration Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 4th Meeting: Chengdu, CN, 15–21 October 2016, Document: JVET-D0029), decoder-side motion vector refinement based on bilateral template matching is disclosed. Figure 5 As shown, a template is generated using bidirectional prediction from reference blocks (i.e., 510 and 520) from MV0 and MV1. Figure 6As shown, this template is used as the new current block, and motion estimation is performed to find better matching blocks in reference images 0 and 1 (i.e., 610 and 620 respectively). The refined motion vectors are MV0' and MV1'. Then, the refined motion vectors (i.e., MV0' and MV1') are used to generate the final bidirectional prediction block for the current block.

[0023] In the decoder-side motion vector refinement, a two-stage search is used to refine the motion vectors of the current block. For example... Figure 7 As shown, for the current block, the cost of the current motion vector candidate (at the current pixel position, represented by square symbol 710) is first evaluated. In the first-stage search, an integer-pixel search is performed around the current pixel position. Eight candidates (by...) Figure 7 The eight large circles (represented by 720) are evaluated. At least one of the horizontal and vertical distances between two adjacent large circles, or between a square symbol and its adjacent large circle, is one pixel. In the first stage, the best candidate with the lowest cost is selected as the best motion vector candidate (e.g., the candidate at the location represented by circle 730). In the second stage, as... Figure 7 As shown in the eight small circles, a half-pixel square search is performed around the best motion vector candidate in the first stage. The best motion vector candidate with the lowest cost is selected as the final motion vector for final motion compensation.

[0024] To compensate for fractional motion vectors, an 8-tap interpolation filter is used in HEVC and JEM-4.0 (i.e., the reference software for JVET). In JEM-4.0, the motion vector accuracy is 1 / 16 pixel. Sixteen 8-tap filters are used. The filter coefficients are as follows.

[0025] 0 / 16-pixels:{0,0,0,64,0,0,0,0}

[0026] 1 / 16-pixel:{0,1,-3,63,4,-2,1,0}

[0027] 2 / 16-pixels:{-1,2,-5,62,8,-3,1,0}

[0028] 3 / 16-pixels:{-1,3,-8,60,13,-4,1,0}

[0029] 4 / 16-pixels: {-1,4,-10,58,17,-5,1,0}

[0030] 5 / 16-pixels:{-1,4,-11,52,26,-8,3,-1}

[0031] 6 / 16-pixels:{-1,3,-9,47,31,-10,4,-1}

[0032] 7 / 16-pixels:{-1,4,-11,45,34,-10,4,-1}

[0033] 8 / 16-pixels:{-1,4,-11,40,40,-11,4,-1}

[0034] 9 / 16-pixels:{-1,4,-10,34,45,-11,4,-1}

[0035] 10 / 16-pixels: {-1,4,-10,31,47,-9,3,-1}

[0036] 11 / 16-pixels:{-1,3,-8,26,52,-11,4,-1}

[0037] 12 / 16-pixels:{0,1,-5,17,58,-10,4,-1}

[0038] 13 / 16-pixels:{0,1,-4,13,60,-8,3,-1}

[0039] 14 / 16-pixels:{0,1,-3,8,62,-5,2,-1}

[0040] 15 / 16-pixels:{0,1,-2,4,63,-3,1,0}

[0041] There is a need to reduce bandwidth requirements for systems employing model-based motion vector derivation, bidirectional optical flow, decoder-side motion vector refinement, or other motion refinement processes. Summary of the Invention

[0042] According to a method of the present invention, a target motion compensation reference block associated with the current block is determined in a target reference image from a list of reference images, wherein the target motion compensation reference block includes additional surrounding pixels located around the corresponding block of the current block in the target reference image for interpolation filters required to perform arbitrary fractional motion vectors for the current block. One or more target fractional pixel locations are selected. Motion refinement of the current block is generated by searching through multiple motion vector candidates using reference data including the target motion compensation reference block, using a model-based motion vector derivation process, a bidirectional optical flow process, or a decoder-side motion vector refinement process, wherein if a target motion vector candidate belongs to one or more target fractional pixel locations, an interpolation filter with shortened tap length is applied to the target motion vector candidate. The one or more target fractional pixel locations correspond to multiple pixel locations from (1 / filter_precision) to ((filter_precision / 2) / filter_precision) and multiple pixel locations from ((filter_precision / 2+1) / filter_precision) to ((filter_precision-1) / filter_precision), wherein filter_precision corresponds to motion vector precision.

[0043] According to another method of the present invention, the current block is divided into multiple sub-blocks based on whether the prediction direction associated with the current block is bidirectional or unidirectional, for use in a selected motion estimation / motion compensation process that includes motion estimation / motion compensation based on the sub-blocks. Motion information associated with the multiple sub-blocks is determined. Based on the motion information associated with the multiple sub-blocks, the multiple sub-blocks are encoded or decoded using motion compensation prediction. The smallest block size of the multiple sub-blocks used for bidirectional prediction is larger than the smallest block size of the multiple sub-blocks used for unidirectional prediction. Attached Figure Description

[0044] Figure 1 An example of motion compensation using bilateral matching is shown, where the current block is predicted by two reference blocks along the motion trajectory.

[0045] Figure 2 An example of motion compensation using template matching is shown, where the template of the current block is matched with a reference template in a reference image.

[0046] Figure 3A An example of the derivation process for predicting the temporal motion vectors of the LIST_0 reference image is shown.

[0047] Figure 3B An example of the derivation process for predicting the temporal motion vectors of the LIST_1 reference image is shown.

[0048] Figure 4 An example of deriving the bidirectional optical flow for motion refinement of the offset motion vector is shown.

[0049] Figure 5 An example of decoder-side motion vector refinement is shown, where the template is first generated using bidirectional predictions from reference blocks from MV0 and MV1.

[0050] Figure 6 It shows the use of Figure 5 An example of decoder-side motion vector refinement where the generated template is used as the new current block and motion evaluation is performed to find better matching blocks from reference image 0 and reference image 1, respectively.

[0051] Figure 7 An example of a two-stage search for the motion vectors of the current block used to refine the motion vector refinement on the decoder side is shown.

[0052] Figure 8 An example of the reference data required for decoder-side motion vector refinement of an MxN block with fractional motion vectors is shown, where the (M+L-1)*(N+L-1) reference block is required for motion compensation.

[0053] Figure 9 An exemplary flowchart of a video codec system according to embodiments of the present invention is shown, which uses a predictive sub-refinement process, such as model-based motion vector derivation, bidirectional optical flow, or decoder-side motion vector refinement, to refine motion with reduced system bandwidth.

[0054] Figure 10 An exemplary flowchart of a video codec system according to an embodiment of the present invention is shown, which uses a predictive sub-refinement process, such as model-based motion vector derivation, bidirectional optical flow, or decoder-side motion vector refinement, to refine motion with reduced system bandwidth, wherein if a target motion vector candidate belongs to one or more specified target fractional pixel locations, an interpolation filter with shortened tap length is applied to the target motion vector candidate.

[0055] Figure 11 An exemplary flowchart of a video codec system according to an embodiment of the present invention is shown, which uses a selected motion estimation / motion compensation process with sub-block-based motion estimation / motion compensation to refine motion, wherein the current block is divided into multiple sub-blocks based on whether the prediction direction associated with the current block is bidirectional or unidirectional. Detailed Implementation

[0056] The following description illustrates preferred embodiments of the present invention. These embodiments are merely illustrative of the technical features of the invention and are not intended to limit the scope of the invention. The scope of protection of the present invention is defined by the claims.

[0057] As mentioned earlier, different predictor refinement techniques, such as pattern-based motion vector derivation, bidirectional optical flow, or decoder-side motion vector refinement, require access to additional reference data, which increases system bandwidth. For example, Figure 8 As shown, for an MxN block 810 with fractional motion vectors, motion compensation requires (M+L-1)*(N+L-1) reference block 825, where L is the interpolation filter tap length. In HEVC, L equals 8. For decoder-side motion vector refinement search, a ring-shaped region 820 with a one-pixel width located outside the reference block 825 is needed for the first-stage search within the (M+L-1)*(N+L-1) reference block 825 plus the ring-shaped region 820. The region corresponding to the reference block 825 plus the ring-shaped region 820 is called the reference pixel region 830. If the best candidate is located on the upper left instead of the center candidate, additional data outside the ring-shaped region 820 may be needed. For example, an additional L-shaped region 840 (i.e., an additional (M+L-1) pixel column and (N+L-1) pixel row) is needed. The additional reference pixels required to support the predictive sub-refinement tool imply additional bandwidth. This invention discloses techniques for reducing system bandwidth related to model-based motion vector derivation, bidirectional optical flow, and decoder-side motion vector refinement.

[0058] In JEM-4.0, when an 8-tap filter is used, not every filter has 8 coefficients. For example, a 3 / 16 pixel filter has only 7 coefficients, and a 1 / 16 pixel filter has only 6 coefficients. Therefore, for some motion vector candidates, the actual required reference pixels are less than... Figure 8 The reference pixels mentioned in the text. For example, if the center motion vector candidate is located at (11 / 16, 11 / 16), it requires a (M+7)*(N+7) pixel block. For the first stage search, the eight motion vector candidates are located at (11 / 16±1, 11 / 16±1) (i.e., (11 / 16, 11 / 16+1), (11 / 16, 11 / 16-1), (11 / 16+1, 11 / 16+1), (11 / 16+1, 11 / 16), (11 / 16+1, 11 / 16), (11 / 16+1, 11 / 16-1), (11 / 16-1, 11 / 16), (11 / 16-1, 11 / 16-1)), and they require a (M+7+1+1)*(N+7+1+1) pixel block (i.e., Figure 8(Referencing region 830). If the best candidate is (11 / 16+1, 11 / 16), then the eight candidates for the second-stage search are (11 / 16+1±8 / 16, 11 / 16±8 / 16) (i.e., (11 / 16+1, 11 / 16), (11 / 16+1, 11 / 16-8 / 16), (11 / 16+1+8 / 16, 11 / 16+8 / 16), (11 / 16+1+8 / 16, 11 / 16), (11 / 16+1+8 / 16, 11 / 16), (11 / 16+1+8 / 16, 11 / 16-8 / 16), (11 / 16+1-8 / 16, 11 / 16), (11 / 16+1-8 / 16, 11 / 16)). For the candidate (11 / 16+1+8 / 16, 11 / 16), a 3 / 16 pixel filter is used. This 3 / 16 pixel filter has only 7 coefficients, of which only 3 are located to the right of the current pixel. This means that there are no additional reference pixels required for motion compensation of the (11 / 16+1+8 / 16, 11 / 16) candidate. Therefore, the fractional motion vector position and filter coefficients will affect the number of pixels required for thinning. To reduce bandwidth, three methods are disclosed below.

[0059] Method 1: Candidate Skip

[0060] To reduce bandwidth requirements, candidates for skipping the search that require additional memory access were proposed. A table was created to list how many pixels are used for the filter on the right and left. For example, Table 1 shows the pixels required to the left and right of the current pixel. For predictive sub-thinning tools (e.g., model-based motion vector derivation, decoder-side motion vector thinning, and bidirectional optical flow), a valid reference block is first defined. For example, a valid reference block could be a (M+(L-1))*(N+(L-1)) block (i.e. Figure 8 The reference region 825) or (M+L+1)*(N+L+1) block (i.e., Figure 8 The reference region 830 in the reference region is used for decoder-side motion vector refinement. During the refinement process, if a candidate requires a reference pixel outside the valid block, that candidate is skipped. In the case of decoder-side motion vector refinement, the skip decision can be made based on the fractional motion vector position and pixel requirements of the filter as listed in Table 1. For example, if one-dimensional interpolation is used and the (M+(L-1)+1+1)*(N+(L-1)+1+1) pixel block is defined as the valid block, this means that the valid block includes (L / 2)+1 pixels to the left and (L / 2)+1 pixels to the right of the current pixel. In JEM-4.0, L is 8, which means there are 5 pixels to the left and 5 pixels to the right of the current pixel. For the required pixels on the left and right, we can use the following equation.

[0061] Left:

[0062] integer_part_of(refine_offset + fractional_part_of_org_MV) + Filter_required_pixel_left[(fractional_part_of(refine_offset + fractional_part_of_org_MV) % filter_precision] (1)

[0063] Right:

[0064] integer_part_of(refine_offset + fractional_part_of_org_MV) + Filter_required_pixel_right[(fractional_part_of(refine_offset + fractional_part_of_org_MV) % filter_precision) (2)

[0065] Left right Filter_required_pixel_left Filter_required_pixel_right 0 / 16-pixel: 1 0 1 / 16-pixel: 3 3 2 / 16-pixel: 4 3 3 / 16-pixel: 4 3 4 / 16-pixel: 4 3 5 / 16-pixel: 4 4 6 / 16-pixel: 4 4 7 / 16-pixel: 4 4 8 / 16-pixel: 4 4 9 / 16-pixel: 4 4 10 / 16-pixel: 4 4 11 / 16-pixel: 4 4 12 / 16-pixel: 3 4 13 / 16-pixel: 3 4 14 / 16-pixel: 3 4 15 / 16-pixel: 3 3

[0066] For example, from Table 1, if the central MV_x candidate is 3 / 16, then 4 pixels are needed on the left and 3 pixels on the right. For the first stage of search, MV_x corresponding to the (3 / 16+1) and (3 / 16–1) candidates need to be searched. For the MV_x corresponding to the (3 / 16–1) candidate, it needs more than one pixel for the left pixel, i.e., 5 pixels. For the MV_x corresponding to the (3 / 16+1) candidate, it needs more than one pixel for the right pixel, i.e., 4 pixels. Therefore, both the (3 / 16+1) and (3 / 16–1) candidates can be used for searching. If the best MV_x candidate is (3 / 16–1), then the candidates at a distance of half a pixel from the best MV_x candidate (i.e., the (3 / 16–1+8 / 16) and (3 / 16–1-8 / 16) candidates) need to be searched. For the candidate MV_x corresponding to (3 / 16–1–8 / 16), MV_x is equivalent to (–2+11 / 16). According to equations (1) and (2), integer_part_of(refine_offset+fractional_part_of_org_MV) is 2, and (fractional_part_of(refine_offset+fractional_part_of_org_MV)%filter_precision is 11, where filter_precision is 16. It requires 2+4 pixels for the left side, where 2 comes from the “-2” and 4 comes from the “11 / 16 pixel filter”. Therefore, the candidate MV_x corresponding to (3 / 16–1–8 / 16) requires more reference pixels than the effective block, and the candidate MV_x corresponding to (3 / 16–1–8 / 16) should be skipped.

[0067] Method 2: Candidate Replacement

[0068] Similar to Method-1, the valid block is first defined, and the required pixels are calculated according to equations (1) and (2). However, if a candidate is invalid, instead of skipping it, the candidate is moved closer to the center (original) motion vector. For example, if a candidate's MV_x is (X–1) and is invalid, where X is the original motion vector and "-1" is the thinning offset, the candidate position is shifted to (X–8 / 16) or (X–12 / 16) or any candidate between X and (X–1) (e.g., the valid candidate closest to (X–1)). In this way, a similar number of candidates can be checked without requiring additional bandwidth. In one embodiment, for the second-stage search, if its first-stage candidate is a replacement candidate, the reference first-stage offset should use the unreplaced offset. For example, if the original candidate of the first-stage search is (X–1) and is not a valid candidate, it is replaced by (X–12 / 16). For the second-stage candidate, it can still use (X–1 ± 8 / 16) for the second-stage search. In another embodiment, for the second-stage search, if the first-stage candidate is a replacement candidate, the reference first-stage offset should use the replaced offset. For example, if the original candidate for the first-stage search is (X–1) and is not a valid candidate, it is replaced with (X–12 / 16). For the second-stage candidate, it can use (X–12 / 16 ± 8 / 16) for the second-stage search. In another embodiment, if the first-stage candidate is a replacement candidate, the offset for the second-stage search can be reduced.

[0069] In Method-1 and Method-2, different encoding / decoding tools can have different effective reference block settings. For example, for decoder-side motion vector refinement, the effective block can be (M+L-1)*(N+L-1) blocks. For model-based motion vector derivation, the effective block can be (M+L-1+O)*(N+L-1+P) blocks, where O and P can both be 4.

[0070] In the model-based motion vector derivation, a two-stage search is performed. The first stage is a prediction unit layer search. The second stage is a sub-predictor unit layer search. In the proposed method, a valid reference block constraint is used for both the first and second stage searches. The valid reference blocks can be the same for both stages.

[0071] The proposed methods 1 and 2 can be limited to being applied to certain coding units or prediction units. For example, the proposed methods can be applied to coding units with an area greater than 64 or 256, or to bidirectional prediction blocks.

[0072] Method 3: Shorter filter tap design

[0073] In Method-3, a method is proposed to reduce the number of pixels required for filter positions from (1 / filter_precision) to ((filter_precision / 2-1) / filter_precision) and from ((filter_precision / 2+1) / filter_precision) to ((filter_precision-1) / filter_precision). For example, in JEM-4.0, a method is proposed to reduce the number of pixels required for filters corresponding to 1 / 16 pixels to 7 / 16 pixels and filters corresponding to 9 / 16 pixels to 15 / 16 pixels. If a 6-tap filter is used for filters corresponding to 1 / 16 pixels to 7 / 16 pixels and filters corresponding to 9 / 16 pixels to 15 / 16 pixels, the second-stage search for motion vector refinement on the decoder side does not require additional bandwidth.

[0074] Prediction unit segmentation based on prediction direction

[0075] In some encoding / decoding tools, the current prediction unit is divided into multiple sub-prediction units if certain constraints are met. For example, in JEM-4.0, advanced TMVP (ATMVP), model-based motion vector derivation, bidirectional optical flow, and affine prediction / compensation will divide the current prediction unit into sub-prediction units. To reduce worst-case bandwidth, it is proposed to divide the current prediction unit into different sizes based on the prediction direction. For example, a minimum size / area / width / height of M is used for bidirectional prediction blocks, and a minimum size / area / width / height of N is used for unidirectional prediction blocks. For example, the minimum area for bidirectional prediction can be 64, and the minimum area for unidirectional prediction can be 16. As another example, the minimum width / height for bidirectional prediction can be 8, and the minimum width / height for unidirectional prediction can be 4.

[0076] In another example, for the ATMVP merging mode, the minimum sub-prediction cell area is 64 if the motion vector candidate is bidirectional prediction. If the motion vector candidate is unidirectional prediction, the minimum sub-prediction cell area can be 16.

[0077] Figure 9An exemplary flowchart of a video codec system for refining motion / prediction using, for example, model-based motion vector derivation, bidirectional optical flow, or decoder-side motion vector refinement according to embodiments of the present invention, is shown. The steps shown in this flowchart, as well as other flowcharts in the present invention, can be implemented as program code executable on one or more processors (e.g., one or more CPUs) at the encoder side and / or decoder side. The steps shown in this flowchart can also be implemented in hardware, such as one or more electronic devices or processors for performing the steps in this flowchart. According to the method, in step 910, input data associated with a current block in the current image is received. In step 920, a target motion compensation reference block associated with the current block is determined in a target reference image from a list of reference images, wherein the target motion compensation reference block includes additional surrounding pixels located around the corresponding block of the current block in the target reference image for interpolation filters required to perform arbitrary fractional motion vectors of the current block. In step 930, a valid reference block associated with the target motion compensation reference block is designated. In step 940, a motion refinement for the current block is generated by searching multiple motion vector candidates using reference data including the target motion-compensated reference block, and employing a predictive sub-refinement process, such as a model-based motion vector derivation process, a bidirectional optical flow process, or a decoder-side motion vector refinement process. If a target motion vector candidate requires target reference data from a target motion-compensated reference block located outside the valid reference block, the target motion vector candidate is excluded from the search among multiple motion vector candidates, or a replacement motion vector candidate closer to the center of the corresponding block is used as a replacement for the target motion vector candidate. In step 950, the current block is encoded or decoded based on motion compensation prediction according to the motion refinement.

[0078] Figure 10An exemplary flowchart of a video codec system for refining motion with reduced system bandwidth is shown, illustrating a predictive sub-refinement process using, for example, model-based motion vector derivation, bidirectional optical flow, or decoder-side motion vector refinement according to embodiments of the present invention. In this process, if a target motion vector candidate belongs to one or more specified target fractional pixel locations, an interpolation filter with shortened tap length is applied to the target motion vector candidate. According to the method, in step 1010, input data associated with the current block in the current image is received. In step 1020, a target motion compensation reference block associated with the current block is determined in a target reference image from a list of reference images, wherein the target motion compensation reference block includes additional surrounding pixels located around the corresponding block of the current block in the target reference image for interpolation filters required to perform arbitrary fractional motion vectors of the current block. In step 1030, one or more target fractional pixel locations are selected. In step 1040, a search is performed among multiple motion vector candidates using reference data including the target motion-compensated reference block. A predictive sub-refinement process, such as a model-based motion vector derivation process, a bidirectional optical flow process, or a decoder-side motion vector refinement process, is used to generate motion refinement for the current block. If a target motion vector candidate belongs to one or more target fractional pixel locations, an interpolation filter with shortened tap length is applied to the target motion vector candidate. In step 1050, the current block is encoded or decoded based on motion-compensated prediction according to the motion refinement.

[0079] Figure 11 An exemplary flowchart of a video codec system according to an embodiment of the present invention is shown, using a selected motion estimation / motion compensation process based on sub-block motion estimation / compensation, including, for example, advanced temporal motion vector prediction, model-based motion vector derivation, bidirectional optical flow or affine prediction / compensation, and having reduced system bandwidth, to refine motion. The current block is segmented into sub-blocks based on whether the prediction direction associated with the current block is bidirectional or unidirectional. According to the method, in step 1110, input data associated with the current block in the current image is received. In step 1120, the current block is segmented into multiple current sub-blocks based on whether the prediction direction associated with the current block is bidirectional or unidirectional, for use in a selected motion estimation / motion compensation process including sub-block-based motion estimation / compensation. In step 1130, motion information associated with the multiple sub-blocks is determined. In step 1140, the multiple sub-blocks are encoded or decoded using motion compensation prediction based on the motion information associated with the multiple sub-blocks.

[0080] The flowchart shown illustrates an example of video encoding and decoding according to the present invention. Those skilled in the art can modify each step, reorganize these steps, separate a step, or combine these steps to implement the invention without departing from its spirit. Specific syntax and semantics have been used in this invention to illustrate examples of implementing embodiments of the invention. Those skilled in the art can implement the invention by replacing the syntax and semantics with equivalent syntax and semantics without departing from its spirit.

[0081] The foregoing description enables those skilled in the art to implement the invention within the content and requirements of specific applications. Various modifications of the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not limited to the specific embodiments shown and described, but is to be given the maximum scope consistent with the principles and novel features disclosed herein. Various specific details have been set forth in the foregoing detailed description in order to provide a thorough understanding of the invention. Nevertheless, it will be understood by those skilled in the art that the invention is capable of being practiced.

[0082] The embodiments of the present invention described above can be implemented in various hardware, software code, or a combination of both. For example, an embodiment of the present invention may be circuitry integrated within a video compression chip, or program code integrated into video compression software to perform the processing described herein. An embodiment of the present invention may also be program code executed on a digital signal processor (DSP) to perform the processing described herein. The present invention may also include functions executed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). According to the present invention, processors can be configured to perform specific tasks by executing machine-readable software code or firmware code that defines the specific methods implemented by the present invention. The software code or firmware code may be developed in different programming languages ​​and different formats or styles. The software code may also be compiled for different target platforms. However, different code formats, software code styles and languages, and other forms of configuration code for performing the tasks of the present invention do not depart from the spirit and scope of the present invention.

[0083] This invention may be practiced in other specific forms without departing from its spirit or essential characteristics. The examples described are illustrative in all respects only and not limiting. Therefore, the scope of the invention is indicated by the claims, not by the foregoing description. The meaning of the claims and all variations within the same scope should be included within their scope.

Claims

1. A method of video coding, the method comprising: A predictor refinement process is used to refine the motion of a block, the method comprising: receiving input data related to a current block in a current picture; determining a target motion compensated reference block related to the current block in a target reference picture from a reference picture list, wherein the target motion compensated reference block comprises extra surrounding pixels located around a corresponding block of the current block in the target reference picture, wherein the target motion compensated reference block is used to perform an interpolation filter required for any fractional motion vector of the current block; selecting one or more target fractional pixel positions; using the predictor refinement process to generate a motion refinement of the current block by searching among a plurality of motion vector candidates using reference data including the target motion compensated reference block, wherein if a target motion vector candidate belongs to the one or more target fractional pixel positions, a shortened tap length interpolation filter is applied to the target motion vector candidate; and encoding or decoding the current block based on motion compensated prediction according to the motion refinement.

2. The video coding method of claim 1, wherein, The predictor refinement process corresponds to model-based motion vector derivation, bi-directional optical flow or decoder-side motion vector refinement.

3. The video coding method of claim 1, wherein, The one or more target fractional pixel positions correspond to a plurality of pixel positions from (1 / filter_precision) to ((filter_precision / 2) / filter_precision) and a plurality of pixel positions from ((filter_precision / 2+1) / filter_precision) to ((filter_precision-1) / filter_precision), wherein the filter_precision corresponds to a motion vector precision.

4. A video coding device comprising: A video coding apparatus for refining the motion of a block using a predictor refinement process, the video coding apparatus comprising one or more electronic circuits or processors, wherein: a first electronic circuit or processor is configured to receive input data related to a current block in a current picture; a second electronic circuit or processor is configured to determine a target motion compensated reference block related to the current block in a target reference picture from a reference picture list, wherein the target motion compensated reference block comprises extra surrounding pixels located around a corresponding block of the current block in the target reference picture, wherein the target motion compensated reference block is used to perform an interpolation filter required for any fractional motion vector of the current block; selecting one or more target fractional pixel positions; using the predictor refinement process to generate a motion refinement of the current block by searching among a plurality of motion vector candidates using reference data including the target motion compensated reference block, wherein if a target motion vector candidate belongs to the one or more target fractional pixel positions, a shortened tap length interpolation filter is applied to the target motion vector candidate; and encoding or decoding the current block based on motion compensated prediction according to the motion refinement. The predictor refinement process corresponds to model-based motion vector derivation, bi-directional optical flow or decoder-side motion vector refinement.

5. The video coding apparatus of claim 4, wherein, The one or more target fractional pixel positions correspond to a plurality of pixel positions from (1 / filter_precision) to ((filter_precision / 2) / filter_precision) and a plurality of pixel positions from ((filter_precision / 2+1) / filter_precision) to ((filter_precision-1) / filter_precision), wherein the filter_precision corresponds to a motion vector precision.

6. A non-transitory computer-readable medium, comprising: A plurality of program instructions are stored, causing a processing circuit of the apparatus to perform a video coding method, and the method comprises: receiving input data related to a current block in a current picture; determining a target motion compensated reference block related to the current block in a target reference picture from a reference picture list, wherein the target motion compensated reference block comprises extra surrounding pixels located around a corresponding block of the current block in the target reference picture, wherein the target motion compensated reference block is used for performing an interpolation filter needed for any fractional motion vector of the current block; selecting one or more target fractional pixel positions; using a decoder side predictor refinement process to generate a motion refinement for the current block by searching among a plurality of motion vector candidates using reference data including the target motion compensated reference block, wherein if a target motion vector candidate belongs to the one or more target fractional pixel positions, an interpolation filter with shortened tap length is applied to the target motion vector candidate; and encoding or decoding the current block based on motion compensated prediction according to the motion refinement.

7. The non-transitory computer-readable medium of claim 6, wherein, The predictor refinement process corresponds to model based motion vector derivation, bi-directional optical flow or decoder side motion vector refinement.

Images