Candidate reordering for merge modes with motion vector differences

Abstract

A video coding system that reorders prediction candidates is provided. A video codec receives data for a block of pixels that is to be encoded or decoded as a current block of a current picture of a video. A video codec identifies possible candidate prediction locations. The video codec calculates a cost for each of the identified possible candidate predicted locations. The video codec allocates a reordering index to each of the N lowest cost candidate prediction positions from the identified possible candidate prediction positions based on the calculated cost. The video codec uses the allocated reorder index to select the candidate prediction position, where the selection is sent in or parsed from the bitstream. The video codec encodes or decodes the current block by using the selected candidate prediction position.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present disclosure is a part of a non-provisional application which claims priority to U.S. Provisional Patent Application No. 63 / 233,346 filed on August 16, 2021 and U.S. Provisional Patent Application No. 63 / 318,802 filed on March 11, 2022. The contents of the above-listed applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to video coding and decoding. In particular, the present disclosure relates to a method for sorting Merge Mode with Motion Vector Difference (MMVD) candidates. Background Art

[0004] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims listed below and are not admitted to be prior art by inclusion in this section.

[0005] High-Efficiency Video Coding (HEVC) is an international video codec standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a hybrid block-based motion compensation DCT transform coding and decoding architecture. The basic unit of compression, called a coding unit (CU), is a 2Nx2N square block. Each CU can be recursively divided into four smaller CUs until a predetermined minimum size is reached. Each CU contains one or more prediction units (PUs).

[0006] In order to improve the coding efficiency of motion vector (MV) coding in HEVC, HEVC has skip and merge modes. The skip and merge modes obtain motion information from spatially adjacent blocks (spatial candidates) or temporally co-located blocks (temporal candidates). When the PU is in skip or merge mode, the motion information is not encoded and decoded, only the index of the selected candidate is encoded and decoded. For skip mode, the residual signal is forced to zero and is not encoded and decoded. In HEVC, if a particular block is encoded as skip or merge, the candidate index is sent to indicate which candidate in the candidate set is used for merging. Each merge prediction unit (PU) reuses the MV, prediction direction and reference picture index of the selected candidate. Summary of the invention

[0007] The following summary is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious technologies described herein. Selected but not all embodiments are further described in the detailed description below. Therefore, the following summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0008] Some embodiments of the present disclosure provide a video codec system for reordering prediction candidates. The video codec receives data to be encoded or decoded as a current block of a current picture of a video. The video codec identifies possible candidate prediction positions. The video codec calculates the cost of each of the identified possible candidate prediction positions. The video codec assigns a reordering index to each of the N lowest cost candidate prediction positions from the identified possible candidate prediction positions based on the calculated cost. The video codec uses the assigned reordering index to select a candidate prediction position, where the selection is sent in a bitstream. The video codec encodes or decodes the current block by using the selected candidate prediction position. In some embodiments, N is greater than one but less than the total number of possible candidate prediction positions identified, so that only a subset of possible candidate prediction positions is identified, thereby reducing the number of bits required for encoding, decoding or sending.

[0009] In some embodiments, each candidate prediction position is a refinement position that refines the merge candidate or the starting MV by specifying a direction and distance relative to the merge candidate. In some embodiments, the encoder finds a specific refinement position with the lowest cost among the refinement positions at a specific distance or direction from the merge candidate and by identifying a set of refinement positions adjacent to the specific refinement position as possible candidate prediction positions. The specific distance or direction can be provided by the video encoder in the bitstream.

[0010] The cost of a candidate prediction position can be calculated by matching the template of the current block with the template of a reference block that is referenced by the candidate prediction position. The template of the pixel block includes the pixels on the top and left of the pixel block.

[0011] In some embodiments, the candidate prediction positions are refinement positions of the MMVD, and the selection sent in the bitstream may specify a reordering index of distance or a reordering index of direction.

[0012] In some embodiments, the transmitted selection includes (i) a group index specifying a selected candidate refinement position group of a plurality of candidate refinement position groups and (ii) one or more indices (e.g., a direction index and / or a distance index) specifying a selected candidate refinement position in the selected candidate refinement position group. In some embodiments, all of the identified possible candidate prediction positions belong to the selected candidate refinement position group. A reordering index is assigned to the N lowest cost candidate refinement positions in the selected candidate refinement position group, and the one or more indices specifying the selected candidate refinement position within the selected candidate refinement position group include the reordering index. Different candidate refinement position groups may have different numbers of member candidate refinement positions.

[0013] In some embodiments, each possible candidate prediction position identified is a representative candidate refinement position of a different candidate refinement position group. A reordering index is assigned to N groups having N lowest cost representative candidate refinement positions, respectively, N being less than or equal to the total number of groups in the plurality of candidate refinement position groups. The group index used to specify the selected candidate refinement position group is a reordering index. Each candidate refinement position group in the plurality of candidate refinement position groups has the same number of member candidate refinement positions. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated into and constitute a part of the present disclosure. The accompanying drawings illustrate embodiments of the present disclosure and together with the description are used to explain the principles of the present disclosure. It is worth noting that the drawings are not necessarily drawn to scale, as certain elements may be shown out of proportion to size in actual implementations in order to clearly illustrate the concepts of the present disclosure.

[0015] Figure 1 Motion candidates for merge mode are shown.

[0016] Figure 2 The “prediction+merging” algorithm framework for merging candidates is conceptually illustrated.

[0017] Figure 3 An example candidate re-ordering is conceptually illustrated.

[0018] Figure 4-5 An L-shaped matching method for calculating a guess cost for a selected candidate is conceptually illustrated.

[0019] Figure 6 Conceptually illustrated is the refinement of a starting MV using MVD.

[0020] Figure 7 Conceptually shows a whole set of refined positions at different directions and distances for which costs are calculated.

[0021] Figure 8 An example subset of refined positions is shown.

[0022] Fig. 9A -C conceptually illustrates a pre-selected refinement position based on a selected distance.

[0023] Fig.10 Conceptually illustrated is the reordering between refined positions having the same distance but different directions.

[0024] Fig.11 Conceptually illustrated is the reordering between refined positions having the same direction but different distances.

[0025] Fig.12 The refined positions divided into a plurality of groups are conceptually shown.

[0026] Fig.13 The reordering between different refined position groups is conceptually illustrated.

[0027] Fig.14 An example video encoder is shown that supports reordering of prediction candidates.

[0028] Fig.15 Portions of a video encoder are shown that are used to implement candidate reordering.

[0029] Fig.16 The process of encoding a pixel block using reordering of prediction candidates is conceptually illustrated.

[0030] Fig.17 An example video decoder is shown that supports reordering of prediction candidates.

[0031] Fig.18 Shown is a portion of a video decoder that is used to implement candidate reordering.

[0032] Fig.19 The process of decoding a pixel block using reordering of prediction candidates is conceptually illustrated.

[0033] Fig. 20 An electronic system that implements an embodiment of the present invention is conceptually illustrated. DETAILED DESCRIPTION

[0034] In the following detailed description, many specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. Any variations, derivations and / or extensions based on the teachings described herein are within the scope of protection of the present disclosure. In some cases, well-known methods, processes, components and / or circuits related to one or more example embodiments disclosed herein may be described at a relatively high level without details to avoid unnecessarily obscuring aspects of the teachings of the present disclosure.

[0035] 1. Candidate reordering for merge mode

[0036] Figure 1 Motion candidates for merge mode are shown. The figure shows a current block 100 of a video picture or frame, which is encoded or decoded by a video codec. As shown, up to four spatial MV candidates are derived from spatial neighbors A0, A1, B0 and B1, and one temporal MV candidate (first using TBR, then using TCTR if TBR is not available) is derived from TBR or TCTR. If any of the four spatial MV candidates is not available, position B2 is used to derive the MV candidate as a replacement. After the derivation process of the four spatial MV candidates and one temporal MV candidate, in some embodiments, redundancy removal (cropping) is applied to remove redundant MV candidates. If the number of available MV candidates is less than five after redundancy removal (cropping), three additional candidates are derived and added to the candidate set (candidate list). The video encoder selects a final candidate from the candidate set for skip or merge mode based on a rate-distortion optimization (RDO) decision, and transmits the index to the video decoder. (Skip mode and merge mode are collectively referred to as "merge mode" in this article.)

[0037] For some embodiments, merge candidates are defined as candidates of a general "prediction + merge" algorithm framework. The "prediction + merge" algorithm framework has a first part and a second part. The first part generates a candidate list (a set of) predictors, which are obtained by inheriting neighbor information or refining or processing neighbor information. The second part sends (i) a merge index to indicate which inherited neighbor in the candidate list is selected and (ii) some auxiliary information related to the merge index. In other words, the encoder sends the merge index of the selected candidate and some auxiliary information to the decoder.

[0038] Figure 2 Conceptually illustrates the "prediction + merging" algorithm framework for merging candidates. The candidate list includes many candidates that inherit neighboring information. The inherited information is then processed or refined to form new candidates. In the process, some auxiliary information for the candidates is generated and sent to the decoder.

[0039] The video codec (encoder or decoder) can handle merge candidates in different ways. First, in some embodiments, the video codec can combine two or more candidates into one candidate. Second, in some embodiments, the video codec can use the original candidate as the original MV predictor and use the current block pixels to perform a motion estimation search to find the final MVD (motion vector difference), where the auxiliary information is the MVD. Third, in some embodiments, the video codec can use the original candidate as the original MV predictor and use the current block pixels to perform a motion estimation search to find the final MVD of the L0 and L1 predictors, and the L1 predictor is the original candidate. Fourth, in some embodiments, the video codec can use the original candidate as the original MV predictor and use the current block pixels to perform a motion estimation search to find the final MVD of L1, and the L0 predictor is the original candidate. Fifth, in some embodiments, the video codec can use the original candidate as the original MV predictor and use the top or left neighboring pixels as a search template to perform an MV refinement search to find the final predictor. Sixth, the video codec can perform a MV refinement search to find the final predictor using the original candidate as the original MV predictor and the bidirectional template (the pixels on the L0 and L1 reference pictures pointed to by the candidate MV or the mirrored MV) as the search template.

[0040] For this document, the term "merge candidate" or "candidate" refers to a candidate in the general "prediction + merge" algorithm framework. The "prediction + merge" algorithm framework is not limited to the embodiments described above. Any algorithm with "prediction + merge index" behavior belongs to this framework.

[0041] In some embodiments, the video codec reorders the merge candidates, i.e., the video codec modifies the candidate order within the candidate list to obtain better coding efficiency. The reordering rule depends on some pre-calculation of the current candidates (merge candidates before reordering), such as the top neighboring condition (mode, MV, etc.) or left neighboring condition (mode, MV, etc.) of the current CU, the current CU shape, or top / left L-shaped template matching.

[0042] Figure 3 An example candidate reordering is conceptually illustrated. As shown, the example merge candidate list 300 has six candidates labeled "0" to "5". The video codec initially selects some candidates (candidates labeled "1" and "3") for reordering. The video codec then pre-calculates the costs of these candidates (the costs of the candidates labeled "1" and "3" are 100 and 50, respectively). The cost is named the guess cost of the candidate, and the lower the cost means the better the candidate. Finally, the video codec reorders the selected candidates by moving the candidates with lower costs (candidates labeled "3") to the front of the list.

[0043] Generally speaking, for a merge candidate Ci having a sequential position Oi in the merge candidate list ( N is the total number of candidates in the list, Oi=0 means Ci is at the beginning of the list, Oi=N-1 means Ci is at the end of the list), where Oi=i (C0 is 0, C1 is 1, C2 is 2, ... and so on), the video codec reorders the merge candidates in the list by changing the selected i value in Oi of Ci (changing the order of some selected candidates).

[0044] In some embodiments, merge candidate reordering can be turned off based on the size or shape of the current PU. The video codec can predetermine multiple PU sizes or shapes to turn off merge candidate reordering. In some embodiments, other conditions are also involved to turn off merge candidate reordering, such as picture size, QP value, etc., which are some predetermined values. In some embodiments, the video codec can send a flag to turn merge candidate reordering on or off. For example, a flag (such as "merge_cand_rdr_en") is sent to indicate whether "merge candidate reordering" is enabled (value 1: enabled, value 0: disabled). When the flag is not present, the value of merge_cand_rdr_en is inferred to be 1. The minimum size of the unit in the signaling merge_cand_rdr_en can also be encoded and decoded separately at the sequence level, picture level, fragment level, or PU level.

[0045] Typically, a video codec performs candidate reordering by (1) identifying one or more candidates for reordering, (2) computing a guess cost for each identified candidate, and (3) reordering the candidates based on the guess costs of the selected candidates. In some embodiments, the computed guess costs of some candidates are adjusted (cost adjustment) before the candidates are reordered.

[0046] In some embodiments, the step of selecting one or more candidates can be performed by several different methods. In some embodiments, the video codec selects all candidates with merge_index≤threshold. The threshold (threshold) is a predetermined value, and merge_index is the original order inside the merge list (merge_index is 0, 1, 2, ...). For example, if the original order of the current candidate is at the beginning of the merge list, then merge_index = 0 (corresponding to the current candidate).

[0047] In some embodiments, the video codec selects candidates for reordering based on candidate type. Candidate type is a candidate category for all candidates. The video codec first classifies all candidates into MG types, (MG=1 or 2 or 3 or other values), and then, it selects MG_S (MG_S=1,2,3..., MG_S≤MG) types from all MG types for reordering. An example of classification is to divide all candidates into 4 candidate types. Type 1 is a candidate for spatially adjacent MVs. Type 2 is a candidate for temporally adjacent MVs. Type 3 is all sub-PU candidates (e.g., Sub-PU TMVP, STMVP, affine merge candidates). Type 4 is all other candidates. In some embodiments, the video codec selects candidates based on both merge_index and candidate type.

[0048] In some embodiments, an L-shaped matching method is used to calculate the guess cost of the selected candidate. For the currently selected merge candidate, the video codec obtains an L-shaped template of the current picture and an L-shaped template of the reference picture, and compares the difference between the two templates. The L-shaped matching method has two parts or steps: (i) identifying the L-shaped template and (ii) matching the derived template.

[0049] Figure 4-5 An L-shaped matching method for calculating a guess cost for a selected candidate is conceptually illustrated. Figure 4 The L-shaped template of the current picture is shown, which includes some pixels around the top and left boundaries of the current PU. The L-shaped template of the reference picture includes some pixels around the top and left boundaries of the reference_block_for_guessing of the current merge candidate. The reference_block_for_guessing (whose width BW and height BH are the same as the current PU) is the block pointed to by the integer part of the motion vector of the current merge candidate.

[0050] Different embodiments define the L-shaped template differently. In some embodiments, all pixels of the L-shaped template are outside reference_block_for_guessing (e.g. Figure 4 In some embodiments, all pixels of the L-shaped template are within reference_block_for_guessing (e.g. Figure 4 In some embodiments, some pixels of the L-shaped template are outside the reference_block_for_guessing, and some pixels of the L-shaped template are inside the reference_block_for_guessing. Figure 5 Shown with Figure 4A similar L-shaped template of the current picture, but without the upper left pixel, and an L-shaped template of the reference picture (outer pixel embodiment) without the upper left pixel.

[0051] In some embodiments, the L-shaped matching method and the corresponding L-shaped template (named template_std) are defined as follows: Assuming that the width of the current PU is BW and the height of the current PU is BH, the L-shaped template of the current picture has a top and a left side. Define the top thickness = TTH, the left thickness = LTH, then the top includes all current picture pixels with coordinates (ltx+tj,lty-ti), where ltx is the horizontal coordinate of the upper left integer pixel of the current PU, lty is the vertical coordinate of the upper left integer pixel of the current PU, ti is the index of the pixel row (ti is 0 to (TTH-1)), and tj is the pixel index in the row (tj is 0 to BW-1). The left side includes all current picture pixels with coordinates (ltx-tjl,lty+til), where ltx is the horizontal coordinate of the upper left integer pixel of the current PU, lty is the vertical coordinate of the upper left integer pixel of the current PU, til is the pixel index in the column (til is 0 to (BH-1)), and tjl is the index of the column (tjl is 0 to (LTH-1)).

[0052] In template_std, the L-shaped template of the reference picture has a top and a left side. Define the top thickness = TTHR, the left thickness = LTHR, then the top includes all reference picture pixels with coordinates (ltxr + tjr, ltyr - tir + shifty), where ltxr is the horizontal coordinate of the upper left integer pixel of reference_block_for_guessing, ltyr is the vertical coordinate of the upper left integer pixel of reference_block_for_guessing, tir is the index of the pixel row (tir is 0 to (TTHR-1)), tjr is the pixel index in the row (tjr is 0 to BW-1), and shifty is a predetermined offset value. The left part consists of all reference picture pixels with coordinates (ltxr-tjlr+shiftx,ltyr+tilr), where ltxr is the horizontal coordinate of the upper left integer pixel of reference_block_for_guessing, ltyr is the vertical coordinate of the upper left integer pixel of reference_block_for_guessing, tilr is the pixel index in the column (tilr is 0 to (BH-1)), tjlr is the index of the column (tjlr is 0 to (LTHR-1)), and shiftx is a predetermined offset value.

[0053] If the current candidate has only L0 MV or only L1 MV, the reference picture has one L-shaped template. But if the current candidate has both L0 and L1 MV (bidirectional candidate), the reference picture has 2 L-shaped templates, one template is pointed by L0 MV and L0 reference picture, and the other template is pointed by L1MV and L1 reference picture.

[0054] In some embodiments, for the L-shaped template, the video codec has an adaptive thickness mode. The thickness is defined as the number of pixel rows at the top of the L-shaped template or the number of pixel columns in the left part of the L-shaped template. For the aforementioned L-shaped template template_std, the top thickness in the L-shaped template of the current picture is TTH, the left thickness is LTH, and the top thickness in the L-shaped template of the reference picture is TTHR, and the left thickness is LTHR. The adaptive thickness mode changes the top thickness or the left thickness according to some conditions, such as the current PU size or the current PU shape (width or height) or the QP of the current fragment. For example, if the current PU height is ≥32, the adaptive thickness mode can set the top thickness = 2, and if the current PU height is <32, the top thickness = 1.

[0055] When performing L-shaped template matching, the video codec obtains the L-shaped template of the current picture and the L-shaped template of the reference picture, and compares (matches) the difference between the two templates. The difference between the pixels in the two templates (e.g., the sum of absolute differences or SAD) is used as the cost of the MV. In some embodiments, the video codec can obtain the selected pixels from the L-shaped template of the current picture and the L-shaped template of the reference picture before calculating the difference between the selected pixels of the two L-shaped templates.

[0056] 2. MMVD Candidate Re-ranking

[0057] Merge Mode with Motion Vector Difference (MMVD) is a new codec for the Versatile Video Coding (VVC) standard. Unlike the conventional merge mode where the implicitly derived motion information is directly used for the generation of prediction samples for the current CU, in MMVD, the derived motion information is further refined by the motion vector difference (MVD). MMVD also extends the candidate list for the merge mode by adding additional MMVD candidates, which are located at refined positions based on a predetermined offset value (also called MMVD offset). Therefore, each refined position is also a candidate prediction position. The MMVD flag can be sent after sending the skip flag and the merge flag to specify whether the MMVD mode is used for the CU.

[0058] If MMVD mode is used, the selected merge candidate is refined by MVD information. MVD information includes motion magnitude information (mmvd_distance_idx, or distance index) and motion direction information (mmvd_direction_idx, or direction index). Motion magnitude information includes the offset value of the horizontal component or vertical component to be added to the starting MV (merge candidate). Motion direction information includes the direction of the MVD relative to the starting point.

[0059] Figure 6 Conceptually illustrated is the refinement of a starting MV 600 by an MVD 610. The starting MV 600 may be a selected merge candidate. The MVD 610 is defined by a refinement position having a direction and a distance (relative to the merge candidate). The starting MV 600 is refined by the MVD 610 to create a refinement MV 620 for predicting the current block. Distance information may be encoded or decoded with mmvd_distance_idx or a distance index. Direction information may be encoded or decoded with mmvd_direction_idx or a direction index.

[0060] The MVD information includes a merge candidate flag, a distance index for specifying the motion magnitude, and a direction index for indicating the motion direction. The merge candidate flag is sent to specify which of the first two merge candidates is to be used as the starting MV. The distance index is used to specify the motion magnitude information by indicating a predetermined offset from the starting MV. The offset can be added to the horizontal component or the vertical component of the starting MV. An example mapping from the distance index to the predetermined offset is specified in Table 1I-1 below.

[0061] Table 1I-1 Distance Index

[0062]

[0063] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate one of the four directions shown in Table 11-2.

[0064] Table 1I-2 MV offset symbols specified by direction index

[0065] Direction Index 00 01 10 11 x-axis + – N / A N / A y-axis N / A N / A + –

[0066] It should be noted that the meaning of the MVD symbol may vary depending on the information of the starting MV. When the starting MV is a unidirectional prediction MV or a bidirectional prediction MV, both lists point to the same side of the current picture (i.e., the picture order counts or POCs of the two reference pictures are both greater than the current picture, or both are less than the POC of the current picture), and the symbol in Table 1I-2 specifies the symbol of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV, the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and each symbol in Table 1I-2 specifies the symbol of the MV offset added to the list 0MV component of the starting MV, and the symbol of the list 1MV has the opposite value. In some embodiments, the predetermined offset (MmvdOffset) of the MMVD candidate is derived from or expressed as a distance value (MmvdDistance) and a direction symbol (MmvdSign).

[0067] In some embodiments, the video codec performs candidate reordering for the refinement positions, or more generally, candidate prediction positions, of the MMVD according to the following steps: (1) identifying possible refinement positions, (2) reordering the refinement positions in ascending order based on cost, the cost being calculated by template matching or bidirectional matching, and (3) selecting the top N refinement positions with the lowest cost as the available refinement positions, where N is less than or equal to the number of possible refinement positions determined in step (1). This reduces the number of refinement positions in the MMVD candidate list to only a subset of possible refinement positions, thereby reducing the number of bits required to encode or transmit the distance index (mmvd_distance_idx) and / or the direction index (mmvd_direction_idx).

[0068] As described above, the first step in performing candidate reordering is to identify possible or allowed refinement positions. In general, each refinement position can be represented as a pair of refinement direction and refinement distance, i.e., {(direction, distance)}. In some embodiments, the possible refinement positions {(direction, distance)} identified in the first step include the following positions, which have directions or angles in the set {n / 8*π}, where n ranges from 0 to 15, and distances in the set {1 / 4 pixel, 1 / 2 pixel, 1 pixel, 4 pixels, 8 pixels, 16 pixels, 32 pixels, 64 pixels, 128 pixels}.

[0069] Figure 7A full set of refinement positions of different directions and distances is conceptually shown, whose costs are also calculated. The video encoder and video decoder can each identify a list of N lowest cost refinement positions (i.e., refinement positions that produce MVs with lowest costs by template or bidirectional matching) and assign indices to those lowest cost refinement positions according to cost-based reordering. The video encoder can send the selection of refinement positions to the decoder by using the reordered indices. The selection of refinement positions can be represented in the bitstream as mvd_distance_idx and mmvd_direction_idx but with fewer bits.

[0070] In some embodiments, the possible refinement positions identified in the first step are a subset of the entire set of possible refinement positions. The encoder and decoder each identify a list of N lowest cost refinement positions based on the subset. The subset can be any subset of possible refinement positions negotiated by the encoder and decoder (e.g., by a video codec standard). Figure 8 An example subset of refinement positions is conceptually shown. In this example subset, the refinement positions include only distances in the set {1 / 2-pel, 1-pel} and directions in the set {n / 4*π}, with n ranging from 0 to 7. The encoder and decoder each identify a list of N lowest-cost refinement positions from the subset, and assign indices to those identified refinement positions according to a cost-based reordering. The video encoder can in turn signal the selection of refinement positions to the decoder using the reordering indices.

[0071] In some embodiments, the possible refinement positions identified in the first step include some promising refinement positions pre-selected by the video codec.The video encoder can then signal the selection of the MMVD refinement position by using the index assigned to the lowest cost refinement position among the pre-selected refinement positions. Fig. 9A -C conceptually illustrates preselected refinement positions based on a selected distance. In the illustrated example, the video codec selects a distance from a set of possible distances, and calculates the cost of refinement positions having the selected distance but in a different direction. The video codec identifies the direction (denoted as best_d) with the best cost at the selected distance. Refinement positions adjacent to or close to or adjacent to best_d at the selected distance are determined, and are assigned indices reordered according to the matching cost. The video encoder can send the selection of refinement positions to the MMVD by using the reordered indices.

[0072] Fig. 9AConceptually illustrating selecting a distance from the set of {1 / 4 pixel, 1 / 2 pixel, 1 pixel, 4 pixels, 8 pixels, 16 pixels, 32 pixels, 64 pixels, 128 pixels}. In the example, the selected distance is 1 pixel. The video codec can then calculate the cost of a refined position having a distance of 1 pixel and a direction in the set {0, 1 / 8*π, 1 / 4*π, 3 / 8*π, 1 / 2*π, 5 / 8*π, 3 / 4*π, 7 / 8*π, π, 9 / 8*π, 5 / 4*π, 11 / 8*π, 3 / 2*π, 13 / 8*π, 7 / 4*π, 15 / 8*π}, which is a guessed cost calculated by template matching or bidirectional matching. The direction with the best cost can be denoted as best_d. Fig. 9B Conceptually illustrated is identifying the best cost direction (best_d) from a refinement position at a distance (1 / 2 pixel). The video codec then calculates the cost of refinement positions near the direction best_d at the selected distance (1 / 2 pixel). Fig. 9C Conceptual illustration of calculating costs for refined positions near best_d at a selected distance. In the figure, eight refined positions adjacent to (1 pixel, best_d) are identified, reordered according to matching costs, and assigned indexes according to the reordering. The refined positions are represented as (distance, direction), which has a distance of the set {1 / 4 pixel, 1 / 2 pixel, 1 pixel, 4 pixels, 8 pixels, 16 pixels, 32 pixels, 64 pixels, 128 pixels} and the direction can be any angle in the arithmetic progression set {best_d-n / 8*π, best_d-n / 8*π+d, best_d-n / 8*π+2*d, best_d+n / 8*π}, d=1 / 8*π, 0≤n≤7.

[0073] In some embodiments, the video codec preselects refinement positions based on a selected distance. However, to find best_d for the selected distance, the video codec only checks a subset of possible directions, rather than all possible directions ({n / 8*π}, where n ranges from 0 to 15). Once the best_d direction is determined at the selected distance, the refinement positions adjacent to (1-pel, best_d) are determined, reordered according to the matching cost, and assigned indices according to the reordering. The refinement positions are represented as (distance, direction), which has a distance in the set {1 / 4 pixel, 1 / 2 pixel, 1 pixel, 4 pixels, 8 pixels, 16 pixels, 32 pixels, 64 pixels, 128 pixels}, and the direction can be any angle in the set of arithmetic progressions {best_d-n / m*π, best_d-n / m*π+d, best_d-n / m*π+2*d, best_d+n / m*π}, d=1 / m*π.

[0074] In some embodiments, the video codec preselects a refinement position based on a distance given by the video encoder. The given distance is marked in the bitstream as mmvd_distance_idx. (In other words, the distance index is not subject to cost-based reordering, and mmvd_distance_idx is not affected by the reduction in bit usage.) The video codec checks the cost of a refinement position with a given distance and direction, the direction being selected from the set {0, 1 / 8*π, 1 / 4*π, 3 / 8*π, 1 / 2*π, 5 / 8*π, 3 / 4*π, 7 / 8*π, π, 9 / 8*π, 5 / 4*π, 11 / 8*π, 3 / 2*π, 13 / 8*π, 7 / 4*π, 15 / 8*π}. In some embodiments, the video codec checks the cost of a refinement position with a given distance and direction, the direction selected from a subset of the set {0, 1 / 8*π, 1 / 4*π, 3 / 8*π, 1 / 2*π, 5 / 8*π, 3 / 4*π, 7 / 8*π, 1*π, 9 / 8*π, 5 / 4*π, 11 / 8*π, 3 / 2*π, 13 / 8*π, 7 / 4*π, 15 / 8*π}. The refined positions checked are reordered according to the matching cost and assigned indices according to the reordering.

[0075] In some embodiments, the distance index mmvd_distance_idx is given (by the encoder), and the video codec performs reordering to encode and decode the direction index mmvd_direction_idx. Conversely, in some embodiments, the direction index mmvd_direction_idx is given (by the encoder), and the video codec performs reordering to encode and decode the distance index mmvd_distance_idx. Fig.10 Conceptually illustrates the reordering between refinement positions with the same distance but different directions. In this case, mmvd_direction_idx is based on the reordered index. Fig.11 Conceptually illustrates the reordering between refinement positions with the same direction but different distances. In this case, mmvd_distance_idx is based on the reordered index.

[0076] In some embodiments, the refinement positions are divided into groups such that each refinement position can be represented as {(group, direction, distance)}. The video encoder can signal the selection of the refinement position by sending the group index, the distance index, and the direction index. In some embodiments, the video codec performs a reordering of the candidates within the selected group such that the distance index (mmvd_distance_idx) and / or the direction index (mmvd_direction_idx) are sent using the reordered indexes. For some embodiments, the member refinement positions of the selected group are the possible candidate prediction positions identified in step (1) of the candidate reordering.

[0077] Fig.12 Conceptually illustrates the refined positions divided into groups. Let A be a subset of {1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, 4 pixels, 8 pixels, 16 pixels, 32 pixels, 64 pixels, 128 pixels}, let B be a subset of {0, 1 / 8*π, 2 / 8*π, 3 / 8*π, 4 / 8*π, 5 / 8*π, 6 / 8*π, 7 / 8*π, 8 / 8*π, 9 / 8*π, 10 / 8*π, 11 / 8*π, 12 / 8*π, 13 / 8*π, 14 / 8*π, 15 / 8*π}. B is divided into several groups. Each group contains refined positions with adjacent or neihboring directions, although the sizes of different groups may not be equal.

[0078] In the example, A is the set {1 pixel, 2 pixels} and B is the set {0, 1 / 4*π, 1 / 2*π, 3 / 4*π, 1*π, 5 / 4*π, 3 / 2*π, 7 / 4*π}. Based on sets A and B, there are 16 possible positions. Among them, 6 refined positions are in group 1 (in the direction from 1 / 4*π to 3 / 4*π), 6 refined positions are in group 2 (in the direction from 3 / 2π to 2π), and 4 refined positions are in group 3 (in the direction from π to 5 / 4π). When group 2 is selected (by receiving or sending the corresponding index), the 6 refined positions of group 2 are sent using the reordered indices assigned to the members of group 2. The video codec (encoder and decoder) assigns the reordered indices to the refined positions of the members in the group based on the matching cost. The reordered indices of the selected member refined positions can be sent using mmvd_distance_idx and / or mmvd_direction_idx.

[0079] In some embodiments, the video codec (encoder or decoder) performs reordering between different groups of refinement positions so that the group index can be sent using the reordered index, while the member refinement positions within the group are sent using mmvd_distance_idx and / or mmvd_direction_idx but without reordering. In some of these embodiments, the set of possible directions (B) is divided into groups. Each group contains refinement positions with adjacent directions. All groups have the same number of member refinement positions. The video codec calculates a cost for each group (e.g., by calculating a matching cost representing the member refinement positions) and assigns indexes to different groups based on the calculated costs.

[0080] Fig.13 Conceptual illustration of the reordering between different groups of refined positions. As shown, the 16 possible refined positions are divided into four groups, W, X, Y, and Z, with each group having four directional adjacent member refined positions. The members of each group are indexed according to their relative positions to the designated representative refined positions in the group. For example, the refined position labeled (W:0) is the designated representative refined position of group W, the refined position labeled (X:0) is the designated representative refined position of group X, and so on.

[0081] The video codec calculates the cost for each group by calculating the matching cost at the representative refinement position of the group. In the example, the matching cost of refinement position "W:0" is 30, so 30 is used as the cost for group W. The matching cost of refinement position "X:0" is 50, so 50 is used as the cost for group X, and so on. In other words, the representative refinement position (W:0, X:0, Y:0, Z:0) can be considered as a possible candidate prediction position identified in step (1) of candidate reordering.

[0082] Based on the computational cost of different groups, each group is assigned a reordered index: group W is assigned a reordered group index of 1, group X is assigned a reordered group index of 2, group Y is assigned a reordered group index of 0, and group Z is assigned a reordered group index of 3. Thus, for example, if a refinement position labeled "Y:2" is selected, the encoder will signal "0" for the group index (the reordered index of group Y) and will signal "2" for the corresponding mmvd_distance_idx and / or mmvd_direction_idx.

[0083] As described above, to reduce the number of bits sent, the video codec selects the first N refinement positions with the lowest cost as the available refinement positions, N being less than or equal to the number of possible refinement positions. In this case, the video codec selects the first M groups with the lowest cost as the available groups, M being less than or equal to the number of all possible groups. This allows the video codec to send the group index using fewer bits than would be required to send all possible groups.

[0084] 3. Sample Video Encoder

[0085] Fig.14 An example video encoder 1400 supporting reordering of prediction candidates is shown. As shown, the video encoder 1400 receives an input video signal from a video source 1405 and encodes the signal into a bitstream 1495. The video encoder 1400 has several elements or modules for encoding the signal from the video source 1405, including at least some elements selected from the following: a transform module 1410, a quantization module 1411, an inverse quantization module 1414, an inverse transform module 1415, an intra-frame estimation module 1420, an intra-frame prediction module 1425, a motion compensation module 1430, a motion estimation module 1435, a loop filter 1445, a reconstructed picture buffer 1450, an MV buffer 1465, an MV prediction module 14145, and an entropy encoder 1490. The motion compensation module 1430 and the motion estimation module 1435 are part of the inter-frame prediction module 1440.

[0086] In some embodiments, modules 1410-1490 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 1410-1490 are hardware circuit modules implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 1410-1490 are shown as separate modules, some modules may be combined into a single module.

[0087] The video source 1405 provides a raw video signal that presents the pixel data of each video frame without compression. The subtractor 1408 calculates the difference between the raw video pixel data of the video source 1405 and the predicted pixel data 1413 from the motion compensation module 1430 or the intra-frame prediction module 1425. The transform module 1410 converts the difference (or residual pixel data or residual signal) into transform coefficients (e.g., by performing a discrete cosine transform or DCT). The quantization module 1411 quantizes the transform coefficients into quantization data (or quantization coefficients) 1412, which are encoded into a bitstream 1495 by the entropy encoder 1490.

[0088] The inverse quantization module 1414 dequantizes the quantized data (or quantized coefficients) 1412 to obtain transform coefficients, and the inverse transform module 1415 performs an inverse transform on the transform coefficients to generate a reconstructed residual 1419. The reconstructed residual 1419 is added to the predicted pixel data 1413 to generate reconstructed pixel data 1417. In some embodiments, the reconstructed pixel data 1417 is temporarily stored in a line buffer (line buffer not shown) for intra-frame prediction and spatial MV prediction. The reconstructed pixels are filtered by a loop filter 1445 and stored in a reconstructed picture buffer 1450. In some embodiments, the reconstructed picture buffer 1450 is a memory external to the video encoder 1400. In some embodiments, the reconstructed picture buffer 1450 is a memory internal to the video encoder 1400.

[0089] The intra estimation module 1420 performs intra prediction based on the reconstructed pixel data 1417 to generate intra prediction data. The intra prediction data is provided to the entropy encoder 1490 to be encoded into a bitstream 1495. The intra prediction data is also used by the intra prediction module 1425 to generate predicted pixel data 1413.

[0090] The motion estimation module 1435 performs inter-frame prediction by generating MVs to refer to pixel data of a previously decoded frame stored in the reconstructed picture buffer 1450. These MVs are provided to the motion compensation module 1430 to generate predicted pixel data.

[0091] Instead of encoding the complete actual MV in the bitstream, the video encoder 1400 uses MV prediction to generate a predicted MV, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 1495.

[0092] The MV prediction module 1475 generates a predicted MV based on a reference MV generated for encoding a previous video frame, i.e., a motion compensated MV for performing motion compensation. The MV prediction module 1475 obtains the reference MV from the previous video frame from the MV buffer 1465. The video encoder 1400 stores the MV generated for the current video frame in the MV buffer 1465 as a reference MV for generating a predicted MV.

[0093] The MV prediction module 1475 uses the reference MV to create a predicted MV. The predicted MV can be calculated by spatial MV prediction or temporal MV prediction. The difference (residual motion data) between the predicted MV and the motion compensation MV (MC MV) of the current frame is encoded into the bit stream 1495 by the entropy encoder 1490.

[0094] The entropy encoder 1490 encodes various parameters and data into a bitstream 1495 by using an entropy coding technique such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding. The entropy encoder 1490 encodes various header elements, flags along with quantized transform coefficients 1412 and residual motion data as syntax elements into the bitstream 1495. The bitstream 1495 is then stored in a storage device or transmitted to a decoder via a communication medium such as a network.

[0095] The loop filter 1445 performs filtering or smoothing operations on the reconstructed pixel data 1417 to reduce encoding and decoding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the filtering operation performed includes sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

[0096] Fig.15 A portion of a video encoder 1400 that implements candidate reordering is shown. Specifically, the figure shows elements of an inter prediction module 1440 of the video encoder 1400. As shown, the inter prediction module 1440 retrieves candidate motion vectors from the MV buffer 1465 and searches the contents of the reconstructed picture buffer 1450 to generate the MV of the current block.

[0097] The inter prediction module 1440 includes a motion compensation module 1430, a candidate identification module 1510, a cost calculation module 1520, a candidate reordering module 1530, and a candidate selection module 1540. In some embodiments, modules 1510-1540 are part of the motion estimation module 1435.

[0098] The candidate identification module 1510 identifies candidate prediction positions by identifying, for example, merge candidates or MMVD refinement positions based on the contents of the MV buffer 1465. The identified candidate prediction positions or refinement positions may have different directions and distances from the merge candidates. The identified candidate prediction positions may be a predetermined refinement position group ( Fig.12 ), representative refined positions of different refined position groups ( Fig.13 ), a predetermined set or subset of refined positions ( Figure 7 and Figure 8 ), specific distance ( Fig.10 ) or a specific direction ( Fig.11 ) at a refined position, a set of neighboring refined positions of a specific refined position with a preferred direction and / or distance (Figures 9A-C).

[0099] The cost calculation module 1520 calculates the cost of Figure 4 and Figure 5 The bidirectional matching or template matching is performed and the cost (guessing cost or matching cost) of each candidate prediction position is calculated by obtaining pixel data from the reconstructed picture buffer 1450. The calculated cost of each candidate position is provided to the candidate reordering module 1530, which assigns a reordered index to each candidate position, in particular, a plurality of (N) candidate positions with the lowest cost.

[0100] The candidate selection module 1540 may select one of the identified candidates to formulate the MV of the motion compensation module 1430 to generate the predicted pixel material 1413. The candidate selection module 1540 also provides the reordered index of the selected candidate to the entropy encoder 1490 so that the entropy encoder can use fewer bits to transmit the selection in the bitstream 1495. The reordered index may be used to indicate the group, distance and / or direction of the selected candidate.

[0101] Fig.16 A process 1600 for encoding a pixel block using reordering of prediction candidates is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implement encoder 1400, which performs process 1600 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing encoder 1400 performs process 1600.

[0102] The encoder receives (at block 1610) data of a block of pixels to be encoded in a bitstream as a current block of a current picture of a video;

[0103] The encoder identifies (at block 1620) possible candidate prediction positions. In some embodiments, each candidate prediction position is a refinement position that refines the merge candidate or the starting MV by specifying a direction and distance relative to the merge candidate.

[0104] In some embodiments, the encoder identifies possible candidate prediction positions by finding a specific refinement position with the lowest cost among the refinement positions at a specific distance or direction from the merge candidate (e.g., by finding the best_d direction) and by identifying a group of refinement positions adjacent to the specific refinement position as possible candidate prediction positions. The specific distance or direction may be provided by the video encoder in the bitstream.

[0105] The encoder calculates (at block 1630) a cost for each of the identified possible candidate prediction positions. The cost of the candidate prediction position can be calculated by matching a template of the current block with a template of a reference block referenced by the candidate prediction position. The template of the pixel block includes pixels at the top and left of the pixel block.

[0106] Based on the calculated costs, the encoder assigns (at block 1640) a reordered index to each of the N lowest cost candidate prediction positions among the identified possible candidate prediction positions. In some embodiments, N is a number greater than one but less than the total number of possible candidate prediction positions identified, so that only a subset of possible candidate prediction positions is identified, thereby reducing the number of bits required for encoding, decoding or transmission.

[0107] The encoder selects (at block 1650) a candidate prediction position using the assigned reordered index, where the selection is sent in the bitstream. In some embodiments where the candidate prediction position is a refinement position of the MMVD, the selection sent in the bitstream may specify a reordered index of a distance or a reordered index of a direction.

[0108] The encoder encodes (at block 1660) the current block into a bitstream using the selected candidate prediction position. Specifically, the selected candidate prediction position or refinement position is used to derive an MV for obtaining prediction material.

[0109] In some embodiments, the transmitted selection includes (i) a group index for specifying a selected candidate refinement position group from a plurality of candidate refinement position groups and (ii) one or more indices (e.g., a direction index and / or a distance index) for specifying a selected candidate refinement position within the selected candidate refinement position group. Fig.12 An example is shown in which all possible candidate prediction positions identified belong to a selected candidate refinement position group. Reordered indices are assigned to the N lowest cost candidate refinement positions of the selected candidate refinement position group, and one or more indices for specifying the selected candidate refinement position within the selected candidate refinement position group include the reordered indices. Different candidate refinement position groups may have different numbers of member candidate refinement positions. Fig.13 An example is shown in which each of the identified possible candidate prediction positions is a representative candidate refinement position of a different candidate refinement position group. A reordering index is assigned to N groups, each having N lowest-cost representative candidate refinement positions, N being less than or equal to the total number of groups in the plurality of candidate refinement position groups. The group index for specifying the selected candidate refinement position group is a reordering index. The number of member candidate refinement positions in each of the plurality of candidate refinement position groups is the same.

[0110] 4. Sample Video Decoder

[0111] In some embodiments, the encoder may send (or generate) one or more syntax elements in a bitstream so that the decoder may parse the one or more syntax elements from the bitstream.

[0112] Fig.17 An example video decoder 1700 is shown that can support reordering of prediction candidates. As shown, the video decoder 1700 is an image decoding or video decoding circuit that receives a bitstream 1795 and decodes the content of the bitstream into pixel data of a video frame for display. The video decoder 1700 has several elements or modules for decoding the bitstream 1795, including some elements selected from the following: an inverse quantization module 1711, an inverse transform module 1710, an intra-frame prediction module 1725, a motion compensation module 1730, a loop filter 1745, a decoded picture buffer 1750, an MV buffer 1765, an MV prediction module 1775 and a parser 1790. The motion compensation module 1730 is part of the inter-frame prediction module 1740.

[0113] In some embodiments, modules 1710-1790 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 1710-1790 are hardware circuit modules implemented by one or more ICs of an electronic device. Although modules 1710-1790 are shown as separate modules, some modules may be combined into a single module.

[0114] The parser 1790 (or entropy decoder) receives the bitstream 1795 and performs initial parsing according to the syntax defined by the video coding or image coding standard. The parsed syntax elements include various header elements, flags, and quantization data (or quantization coefficients) 1712. The parser 1790 parses out the various syntax elements by using entropy coding techniques (e.g., context-adaptive binary arithmetic coding (ABAC) or Huffman encoding).

[0115] The inverse quantization module 1711 dequantizes the quantized data (or quantized coefficients) 1712 to obtain transform coefficients, and the inverse transform module 1710 inversely transforms the transform coefficients 1716 to generate a reconstructed residual signal 1719. The reconstructed residual signal 1719 is added to the predicted pixel data 713 from the intra prediction module 1725 or the motion compensation module 1730 to generate decoded pixel data 1717. The decoded pixel data is filtered by the loop filter 1745 and stored in the decoded picture buffer 1750. In some embodiments, the decoded picture buffer 1750 is a memory external to the video decoder 1700. In some embodiments, the decoded picture buffer 1750 is a memory internal to the video decoder 1700.

[0116] The intra prediction module 1725 receives intra prediction data from the bitstream 1795 and, based thereon, generates predicted pixel data 1713 from decoded pixel data 1717 stored in the decoded picture buffer 1750. In some embodiments, the decoded pixel data 1717 is also stored in a line buffer (not shown) for intra prediction and spatial MV prediction.

[0117] In some embodiments, the contents of the decoded picture buffer 1750 are used for display. The display device 1755 either obtains the contents of the decoded picture buffer 1750 for direct display or obtains the contents of the decoded picture buffer to a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded picture buffer 1750 via pixel transfer.

[0118] The motion compensation module 1730 generates predicted pixel data 1713 from the decoded pixel data 1717 stored in the decoded picture buffer 1750 according to the motion compensated MV (MC MV). These motion compensated MVs are decoded by adding the residual motion data received from the bitstream 1795 to the predicted MV received from the MV prediction module 1775.

[0119] The MV prediction module 1775 generates a predicted MV based on a reference MV generated for decoding a previous video frame (e.g., a motion compensated MV for performing motion compensation). The MV prediction module 1775 obtains the reference MV of the previous video frame from the MV buffer 1765. The video decoder 1700 stores the motion compensated MV generated for decoding the current video frame in the MV buffer 1765 as a reference MV for generating a predicted MV.

[0120] The loop filter 1745 performs filtering or smoothing operations on the decoded pixel data 1717 to reduce encoding and decoding artifacts, particularly at the boundaries of pixel blocks. In some embodiments, the filtering operation performed includes sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

[0121] Fig.18 A portion of a video decoder 1700 that implements candidate reordering is shown. Specifically, the figure shows elements of an inter prediction module 1740 of the video decoder 1700. As shown, the inter prediction module 1740 retrieves candidate motion vectors from the MV buffer 1765 and searches the contents of the decoded picture buffer 1750 to generate the MV of the current block.

[0122] The inter prediction module 1740 includes a motion compensation module 1730 , a candidate identification module 1810 , a cost calculation module 1820 , a candidate reordering module 1830 , and a candidate selection module 1840 .

[0123] The candidate identification module 1810 identifies candidate prediction positions by identifying, for example, merge candidates or MMVD refinement positions based on the contents of the MV buffer 1765. The identified candidate prediction positions or refinement positions may have different directions and distances from the merge candidates. The identified candidate prediction positions may be a predetermined refinement position group ( Fig.12 ), representative refined positions of different refined position groups ( Fig.13 ), a predetermined set or subset of refined positions ( Figure 7 and Figure 8 ), specific distance ( Fig.10 ) or a specific direction ( Fig.11 ) at a refined position, a set of neighboring refined positions of a specific refined position with a preferred direction and / or distance (Figures 9A-C).

[0124] The cost calculation module 1820 calculates the cost of Figure 4 and Figure 5 The bidirectional matching or template matching is performed and the cost (guessing cost or matching cost) of each candidate prediction position is calculated by obtaining pixel data from the reconstructed picture buffer 1450. The calculated cost of each candidate position is provided to the candidate reordering module 1830, which assigns a reordered index to each candidate position, in particular, a plurality of (N) candidate positions with the lowest cost.

[0125] The candidate selection module 1840 receives a selection of a candidate from the entropy decoder 1790, which can parse the selection from the bitstream 1795. The selection of a candidate may include a reordered index that indicates the group, distance, and / or direction of the selected candidate. The candidate selection module 1840 uses the candidate selection issued by the entropy decoder 1790 to select one of the candidates identified by the candidate identification module 1810. The selected candidate is then used to formulate an MV for the motion compensation module 1730 to generate predicted pixel data 1713.

[0126] Fig.19 A process 1900 for decoding a pixel block using reordering of prediction candidates is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implement decoder 1700, which performs process 1900 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing decoder 1400 performs process 1900.

[0127] The decoder receives (at block 1910) from a bitstream data for a block of pixels to be decoded as a current block of a current picture of a video.

[0128] The decoder identifies (at block 1920) possible candidate prediction positions. In some embodiments, each candidate prediction position is a refinement position that refines the merge candidate or the starting MV by specifying a direction and distance relative to the merge candidate.

[0129] In some embodiments, the decoder identifies possible candidate prediction positions by finding a specific refinement position with the lowest cost among the refinement positions at a specific distance or direction from the merge candidate (e.g., by finding the best_d direction) and by identifying a set of refinement positions adjacent to the specific refinement position as possible candidate prediction positions. The specific distance or direction may be provided by the video decoder in the bitstream.

[0130] The decoder calculates (at block 1930) the cost of each possible candidate prediction position identified. The cost of a candidate prediction position can be calculated by matching the template of the current block with the template of the reference block that is referenced by the candidate prediction position. The template of the pixel block includes the pixels at the top and left of the pixel block.

[0131] Based on the calculated costs, the decoder assigns (at block 1940) a reordered index to each of the N lowest cost candidate prediction positions among the identified possible candidate prediction positions. In some embodiments, N is greater than one but less than the total number of possible candidate prediction positions identified, so that only a subset of possible candidate prediction positions are identified, thereby reducing the number of bits required for encoding, decoding or transmission.

[0132] The decoder uses the assigned reordered index to select (at block 1950) a candidate prediction position, where the selection is parsed from the bitstream. In some embodiments where the candidate prediction position is a refinement position of the MMVD, the selection sent in the bitstream may specify a reordered index of a distance or a reordered index of a direction.

[0133] The decoder reconstructs (at block 1960) the current block by using the selected candidate prediction position. Specifically, the selected candidate prediction position or the refined position derived MV is used to derive the MV for obtaining prediction material to reconstruct the current block.

[0134] In some embodiments, the transmitted selection includes (i) a group index specifying a selected candidate refinement position group from a plurality of candidate refinement position groups; and (ii) one or more indices specifying a selected candidate refinement position within the selected candidate refinement position group. Fig.12An example is shown in which all identified possible candidate prediction positions belong to a selected candidate refinement position group. Reordered indices are assigned to the N lowest cost candidate refinement positions of the selected candidate refinement position group, and one or more indices for specifying the selected candidate refinement position within the selected candidate refinement position group include the reordered indices. Different candidate refinement position groups may have different numbers of member candidate refinement positions. Fig.13 An example is shown in which each possible candidate prediction position identified is a representative candidate refinement position of a different candidate refinement position group. Reordered indices are assigned to N groups, each having N lowest cost representative candidate refinement positions, N being less than or equal to the total number of groups in the plurality of candidate refinement position groups. The group index that specifies the selected candidate refinement position group is the reordered index. The number of member candidate refinement positions in each candidate refinement position group in the plurality of candidate refinement position groups is the same.

[0135] 5. Example Electronic System

[0136] Many of the above features and applications are implemented as software processes, which are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, compact disc read-only memory (CD-ROM), flash drives, random access memory (RAM) chips, hard drives, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. Computer-readable media do not include carrier waves and electronic signals transmitted via wireless or wired connections.

[0137] In this specification, the term "software" is intended to include firmware residing in read-only memory or application programs stored in magnetic memory, which can be read into memory for processing by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while retaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that jointly implement the software inventions described herein are within the scope of this disclosure. In some embodiments, the software program, when installed to run on one or more electronic systems, defines one or more specific machine implementations that process and execute the operations of the software program.

[0138] Fig. 20 An electronic system 2000 for implementing some embodiments of the present disclosure is conceptually shown. The electronic system 2000 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or any other type of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 2000 includes a bus 2005, a processing unit 2010, a graphics processing unit (GPU) 2015, a system memory 2020, a network 2025, a read-only memory 2030, a permanent storage device 2035, an input device 2040, and an output device 2045.

[0139] Bus 2005 collectively represents all system, peripheral, and chipset buses that communicatively couple numerous internal devices of electronic system 2000. For example, bus 2005 communicatively couples processing unit 2010 with GPU 2015, read-only memory 2030, system memory 2020, and permanent storage device 2035.

[0140] The processing unit 2010 obtains instructions to be executed and data to be processed from these various memory units in order to perform the processing of the present disclosure. In different embodiments, the processing unit can be a single processor or a multi-core processor. Some instructions are passed to and executed by the GPU 2015. The GPU 2015 can offload various calculations or supplement the image processing provided by the processing unit 2010.

[0141] Read-only-memory (ROM) 2030 stores static data and instructions used by processing unit 2010 and other modules of the electronic system. On the other hand, permanent storage device 2035 is a read-write storage device. This device is a non-volatile storage unit that stores instructions and data even when the electronic system 2000 is turned off. Some embodiments of the present disclosure use a large-capacity memory device (such as a magnetic disk or optical disk and its corresponding magnetic disk drive) as permanent storage device 2035.

[0142] Other embodiments use removable storage devices (e.g., floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Like the permanent storage device 2035, the system memory 2020 is a read-write memory device. However, unlike the permanent storage device 2035, the system memory 2020 is a volatile read-write memory, such as a random access memory. The system memory 2020 stores some instructions and data used by the processor at runtime. In some embodiments, processing according to the present disclosure is stored in the system memory 2020, the permanent storage device 2035 and / or the read-only memory 2030. For example, according to some embodiments of the present disclosure, various memory units include instructions for processing multimedia clips. From these various memory units, the processing unit 2010 obtains instructions to be executed and data to be processed in order to perform the processing of some embodiments.

[0143] The bus 2005 is also connected to an input device 2040 and an output device 2045. The input device 2040 enables a user to convey information and select commands to the electronic system. The input device 2040 includes an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"), a camera (e.g., a webcam), a microphone or a similar device for receiving voice commands, etc. The output device 2045 displays an image generated by the electronic system or outputs data. The output device 2045 includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), and a speaker or similar audio output device. Some embodiments include devices used as input and output devices, such as a touch screen.

[0144] Finally, if Fig. 20 As shown, bus 2005 also couples electronic system 2000 to a network 2025 via a network adapter (not shown). In this manner, the computer may be part of a computer network, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of multiple networks, such as the Internet. Any or all of the components of electronic system 2000 may be used in conjunction with the present disclosure.

[0145] Some embodiments include electronic components, such as microprocessors, storage devices and memory, which store computer program instructions in machine-readable or computer-readable media (or referred to as computer-readable storage media, machine-readable media or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, double-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A compact disc, an ultra-density compact disc, any other optical or magnetic medium, and a floppy disk. A computer readable medium may store a computer program executable by at least one processing unit and including a set of instructions for performing various operations. Examples of computer programs or computer codes include machine code such as produced by a compiler, and files including high-level code executed by a computer, electronic component, or microprocessor using an interpreter.

[0146] Although the above discussion mainly involves microprocessors or multi-core processors executing software, many of the above features and applications are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0147] As used in this specification and in any claims of this application, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the terms display or display refer to displaying on an electronic device. As used in this specification and in any claims of this application, the terms "computer-readable medium", "computer-readable medium" and "machine-readable medium" are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other transient signals.

[0148] Although the present disclosure has been described with reference to many specific details, those skilled in the art will recognize that the present disclosure may be implemented in other specific forms without departing from the spirit of the present disclosure. Fig.16 and Fig.19 ) conceptually illustrates the processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the process may be implemented using several sub-processes or as part of a larger macro process. Therefore, it will be understood by those skilled in the art that the present disclosure is not limited by the foregoing illustrative details, but is defined by the appended claims.

[0149] Additional Notes

[0150] The subject matter described herein sometimes represents different elements, which are contained in or connected to other different elements.It is understandable that the described structure is only an example, and in fact, it can be implemented by many other structures to achieve the same function. Conceptually, the arrangement of any element that achieves the same function is actually "associated" to achieve the desired function.Therefore, regardless of the structure or the intermediate component, any two elements combined to achieve a specific function are considered to be "interrelated" to achieve the desired function.Similarly, any two associated elements are considered to be "operably connected" or "operably coupled" to achieve a specific function.Any two elements that can be associated with each other are also considered to be "operably coupled" to achieve a specific function.Any two elements that can be associated with each other are also considered to be "operably coupled" to achieve a specific function.Any two elements that can be associated with each other are also considered to be "operably coupled" to achieve a specific function.The specific example of operable connection includes but is not limited to physically pairable and / or physically interacting elements, and / or wirelessly interactive and / or wirelessly interactive elements, and / or logically interactive and / or logically interactive elements.

[0151] Furthermore, with respect to the use of substantially any plural and / or singular terms, those skilled in the art can translate from the plural to the singular and / or from the singular to the plural as appropriate to the context and / or application. For sake of clarity, the present invention expressly sets forth various singular / plural arrangements.

[0152] In addition, it will be understood by those skilled in the art that, in general, the terms used in the present invention, especially in the claims, and the subject matter of the claims, are generally used as "open" terms, for example, "including" should be interpreted as "including but not limited to", "having" should be interpreted as "at least having", "including" should be interpreted as "including but not limited to", etc. It will be further understood by those skilled in the art that if a specific number of claim contents are intended to be introduced, it will be clearly indicated in the claim, and will not be displayed in the absence of such contents. For example, to aid understanding, the following claims may contain the phrases "at least one" and "one or more" to introduce the contents of the claims. However, the use of these phrases should not be construed as implying the use of the indefinite article "a" or "an" to introduce the claim contents and limit any particular claim. Even when the same claim includes the introductory phrases "one or more" or "at least one", the indefinite article, such as "a" or "an", should be interpreted to mean at least one or more, and the same is true for the use of the explicit description used to introduce the claim. Furthermore, even if an introductory content explicitly refers to a specific number, a person of ordinary skill in the art will recognize that such content should be interpreted as indicating the number of references, for example, "two references" without other modifications means at least two references, or two or more references. Furthermore, where a statement similar to "at least one of A, B, and C" is used, it is generally stated so that a person of ordinary skill in the art can understand the statement, for example, "a system includes at least one of A, B, and C" will include but is not limited to a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc. A person of ordinary skill in the art will further understand that any separated words and / or phrases represented by two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of one of these terms, one of them, or both of these terms. For example, "A or B" should be understood as the possibility of "A", or "B", or "A and B".

[0153] As can be seen from the foregoing, various embodiments have been described for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the invention. Therefore, the various embodiments disclosed herein are not intended to be limiting, and the true scope and application are indicated by the claims.

Claims

1. A video decoding method, comprising: receiving data of a block of pixels from a bitstream to decode the data into a current block of a current picture of a video; identifying a plurality of possible candidate prediction locations; calculating a cost for each of the identified plurality of possible candidate prediction positions; assigning a reordered index to each of the N lowest-cost candidate prediction positions from the identified plurality of possible candidate prediction positions based on the calculated plurality of costs; selecting a candidate prediction position using the assigned reorder index, wherein the selecting is parsed from the bitstream; and The current block is reconstructed using the selected candidate prediction position.

2. The video decoding method according to claim 1, characterized in that: N is less than the total number of the plurality of possible candidate prediction positions identified.

3. The video decoding method according to claim 1, characterized in that: The cost of the candidate prediction position is calculated by combining a template of the current block and a template of a reference block, the reference block being referenced by the candidate prediction position.

4. The video decoding method according to claim 3, characterized in that: The template of a pixel block includes a plurality of pixels at the top and left side of the pixel block.

5. The video decoding method according to claim 1, characterized in that: Each candidate prediction position is a refinement position that refines the merge candidate by specifying a direction and a distance relative to the merge candidate.

6. The video decoding method according to claim 5, characterized in that: The selection parsed from the bitstream specifies a reordering index for the distance or a reordering index for the direction.

7. The video decoding method according to claim 5, characterized in that: Identifying the plurality of possible candidate prediction positions comprises: Finding a specific refined position with the lowest cost from a plurality of refined positions at a specific distance or a specific direction relative to the merge candidate; A set of refinement positions adjacent to the particular refinement position is identified as the plurality of possible candidate prediction positions.

8. The video decoding method according to claim 7, characterized in that: The specific distance or the specific direction is parsed from the bit stream.

9. The video decoding method according to claim 5, characterized in that: The selection being resolved includes (i) a set of indices for specifying a candidate refinement position group selected from a plurality of candidate refinement position groups; and (ii) one or more indices for specifying a selected candidate refinement position within the selected set of candidate refinement positions.

10. The video decoding method according to claim 9, characterized in that: All possible candidate prediction positions identified belong to the selected set of candidate refinement positions.

11. The video decoding method according to claim 10, characterized in that: A plurality of reordering indices are assigned to the N lowest-cost candidate refinement positions of the selected set of candidate refinement positions, wherein one or more indices specifying the selected candidate refinement position within the selected set of candidate refinement positions comprise a reordering indices.

12. The video decoding method according to claim 10, characterized in that: At least two different sets of candidate refinement positions have different numbers of member candidate refinement positions.

13. The video decoding method according to claim 9, characterized in that: Each possible candidate prediction position identified is a representative candidate refinement position of a different set of candidate refinement positions; The plurality of reorderings are assigned to N groups, the N groups respectively having N representative candidate refinement positions with the lowest costs, N being less than or equal to the total number of the plurality of candidate refinement position groups; as well as The group index used to specify the selected group of candidate refinement positions is a reordering index.

14. The video decoding method according to claim 13, characterized in that: Each candidate refinement position group in the plurality of candidate refinement position groups has the same number of member candidate refinement positions.

15. A video encoding method, comprising: Receiving data of a pixel block to be encoded as a current block of a current picture of a video in a bitstream; identifying a plurality of possible candidate prediction locations; calculating a cost for each of the identified plurality of possible candidate prediction positions; assigning a reordering index to each of the N lowest-cost candidate prediction positions from the identified plurality of possible candidate prediction positions based on the calculated cost; selecting a candidate prediction position using the assigned plurality of reordering indices, wherein the selection is signaled in the bitstream; as well as The current block is encoded into the bitstream using the selected candidate prediction position.

16. An electronic device comprising: The video decoder circuit is configured to perform a plurality of operations including: receiving data of a block of pixels from a bitstream to decode the data into a current block of a current picture of a video; identifying a plurality of possible candidate prediction locations; calculating a cost for each of the identified plurality of possible candidate prediction positions; assigning a reordered index to each of the N lowest-cost candidate prediction positions from the identified plurality of possible candidate prediction positions based on the calculated plurality of costs; selecting a candidate prediction position using the assigned reordered index, wherein the selection is parsed from the bitstream; and The current block is reconstructed using the selected candidate prediction position.

17. An electronic device comprising: The video encoder is configured to perform multiple operations, including: Receiving data of a block of pixels to be encoded as a current block of a current picture of a video in a bitstream; identifying a plurality of possible candidate prediction locations; calculating a cost for each of the identified plurality of possible candidate prediction positions; assigning a reordering index to each of the N lowest-cost candidate prediction positions from the identified plurality of possible candidate prediction positions based on the calculated cost; selecting a candidate prediction position using the assigned plurality of reordering indices, wherein the selection is signaled in the bitstream; and The current block is encoded into the bitstream using the selected candidate prediction position.

18. A video encoding and decoding method, comprising: Receiving data of a pixel block to be encoded or decoded as a current block of a current picture of a video in a bitstream; identifying a plurality of possible candidate prediction locations; calculating a cost for each of the identified plurality of possible candidate prediction positions; assigning a reordering index to each of the N lowest-cost candidate prediction positions from the identified plurality of possible candidate prediction positions based on the calculated cost; selecting a candidate prediction position using the assigned plurality of reordering indices, wherein the selection is signaled in the bitstream or parsed from the bitstream; and The current block is encoded or decoded using the selected candidate prediction position.

Images