Updating motion attributes of merge candidates

Abstract

A method for improving merge mode prediction by modifying motion attributes is provided. The video codec receives data of a pixel block to be encoded or decoded as a current block of a current image of a video. The video codec generates a merge candidate list for the current block. The video codec modifies the merge candidate list by changing a motion attribute of the merge candidate from a first value to a second value. The video codec identifies or receives a merge candidate selected from the modified merge candidate list. The video codec encodes or decodes the current block using the selected merge candidate. The motion attribute may be an inter-picture prediction direction, a reference index, a dual prediction index with codeunit layer weights, a local illumination compensation flag, a half-pixel filter used, or a multi-hypothesis prediction weight index.

Description

[0001] Cross-references

[0002] This application is part of a non-provisional application claiming priority to U.S. Provisional Patent Application No. 63 / 349,171 filed on June 6, 2022. The above application is hereby incorporated by reference in its entirety. Technical Field

[0003] The present invention generally relates to video encoding and decoding, and more particularly to a method for encoding and decoding pixel blocks using motion information. Background Art

[0004] Unless otherwise indicated herein, the methods described in this section are not prior art to the following claims and are not admitted to be prior art by inclusion in this section.

[0005] High-Efficiency Video Coding (HEVC) is an international video codec standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a hybrid block-based motion compensated DCT-like transform codec architecture. The basic unit of compression is called a coding unit (CU), which is a 2N×2N square pixel block. Each CU can be recursively divided into four smaller CUs until a predefined minimum size is reached. Each CU contains one or more prediction units (PUs).

[0006] Versatile video coding (VVC) is the latest international video codec standard developed by ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 Joint Video Expert Team (JVET). The input video signal is predicted from the reconstructed signal, which is derived from the coded and decoded image regions. The prediction residual signal is processed by block transform. The transform coefficients are quantized and entropy coded along with other side information in the bitstream. The reconstructed signal is generated from the prediction signal and the reconstructed residual signal after inverse transforming the dequantized transform coefficients. The reconstructed signal is also processed by loop filtering to remove coding artifacts. The decoded image is stored in a frame buffer and used to predict future images in the input video signal.

[0007] In VVC, the coded image is divided into non-overlapping square block areas represented by related coding tree units (CTUs). The leaf nodes of the coding tree correspond to coding units (CUs). The coded image can be represented by a set of slices, each slice including an integer number of CTUs. Each CTU in a slice is processed in raster scan order. Intra-picture prediction or inter-picture prediction with up to two motion vectors and reference indices is used to predict the sample values ​​of each block, so that bi-predictive (B) slices can be decoded. Intra-picture prediction or inter-picture prediction with up to one motion vector and reference index is used to predict the sample values ​​of each block, so that predicted (P) slices can be decoded. Intra-picture (I) slices are decoded using only intra-picture prediction.

[0008] Using a quadtree (QT) with a nested multi-type tree (MTT) structure, the CTU can be divided into one or more non-overlapping codec units (CUs) to accommodate various local motion and texture features. The CU is further divided into smaller CUs using one of five partition types: quadtree partition, vertical binary tree partition, horizontal binary tree partition, vertical center-side ternary tree partition, and horizontal center-side ternary tree partition.

[0009] Each CU contains one or more prediction units (PUs). The prediction unit, together with the associated CU syntax, serves as a basic unit for indicating prediction sub-information. The values ​​of the associated pixel samples within the PU are predicted using a specified prediction process. Each CU may contain one or more transform units (TUs) representing prediction residual blocks. A transform unit (TU) includes one transform block (TB) of luma samples and two corresponding transform blocks of chroma samples, each TB corresponding to one residual block sample for one color component. An integer transform is applied to the transform block. The layer values ​​of the quantized coefficients are entropy encoded and decoded in the bitstream along with other side information. The terms: coding tree block (CTB), coding block (CB), prediction block (PB), transform block (TB) are defined to specify a 2D sample array of one color component associated with a CTU, CU, PU, ​​TU, respectively. Therefore, a CTU includes one luma CTB, two chroma CTBs, and associated syntax elements. Similar relationships apply to CU, PU, ​​and TU.

[0010] For each inter-predicted CU, motion parameters including motion vector, reference picture index and reference picture list usage index and additional information are used to generate inter-predicted samples. Motion parameters can be indicated in an explicit or implicit way. When the CU is encoded and decoded using skip mode, the CU is associated with one PU and there are no significant residual coefficients, no coded motion vector differences or reference picture indices. A merge mode is specified, in which the motion parameters of the current CU are obtained from neighboring CUs, including spatial candidates and temporal candidates, as well as the additional schedule introduced in VVC. Merge mode can be applied to any inter-predicted CU. An alternative to merge mode is the explicit transmission of motion parameters, in which the motion vector, the corresponding reference picture index for each reference picture list and the reference picture list usage flag and other required information are explicitly indicated for each CU. Summary of the invention

[0011] The following summary of the invention is illustrative only and is not intended to be limiting in any way. That is, the following summary of the invention is provided to introduce the concepts, highlights, benefits, and advantages of the new and non-obvious technologies described herein. Only selected, but not all, embodiments are further described in the detailed description below. Therefore, the following summary of the invention is not intended to determine the essential characteristics of the claimed subject matter, nor is it intended to determine the scope of the claimed subject matter.

[0012] Some embodiments of the present invention provide a method for improving merge mode prediction by modifying motion attributes. A video codec receives data of a pixel block to be encoded or decoded as a current block of a current image of a video. The video codec generates a merge candidate list for the current block. The video codec modifies the merge candidate list by changing the motion attribute of the merge candidate from a first value to a second value. The video codec indicates or receives a selected plurality of merge candidates from the modified merge candidate list. The video codec encodes or decodes the current block by using the selected merge candidates.

[0013] In some embodiments, the list of merge candidates is modified when a change in a motion attribute of a first merge candidate increases an estimated cost of encoding a current block using the first merge candidate by more than a threshold. In some embodiments, the estimated cost is a template matching (TM) cost calculated by determining a difference between (i) a current template region adjacent to the current block and (ii) a reference template region adjacent to a reference block identified by the first merge candidate. In some embodiments, the encoder calculates a TM cost for each merge candidate in the list of merge candidates and reorders the list based on a plurality of calculated template matching costs for a plurality of merge candidates in the list. Selecting a merge candidate is based on the reordered list.

[0014] In some embodiments, the merge candidate list is modified by adding a second merge candidate having the modified motion attribute. In some embodiments, the merge candidate list is modified by replacing the first merge candidate with the second merge candidate having the modified motion attribute.

[0015] The motion attribute being changed may be an inter-picture prediction direction, a reference index, a bi-prediction with CU-level Weight (BCW) index, a local illumination compensation (LIC) flag, a half-pixel filter used, or a multi-hypothesis prediction (MHP) weight index. In some embodiments, the encoder changes the motion attribute of the first merge candidate by changing the reference index from identifying the first reference image to identifying the second reference image. The encoder may change the motion attribute of the first merge candidate by scaling the motion vector based on the picture order count (POC) distance of the first reference image and the picture order count (POC) distance of the second reference image. In some embodiments, the encoder changes the motion attribute of the first merge candidate by changing the bi-prediction weight index (e.g., BCW index) to select different weights for combining the first (e.g., L0) inter-picture prediction and the second (e.g., L1) inter-picture prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings are included to provide a further understanding of the present invention and are incorporated into and constitute a part of the present invention. The drawings illustrate embodiments of the present invention and together with the description are used to explain the principles of the present invention. It should be noted that the drawings are not necessarily drawn to scale, as some components may be shown in a manner that is not proportional to the size in the actual embodiment in order to clearly illustrate the concepts of the present invention.

[0017] FIG. 1 shows the reference index of the merge candidate for changing the current block in the current image.

[0018] FIG. 2 conceptually illustrates updating motion attributes of merge candidates based on Template Matching (TM) costs.

[0019] FIG. 3 conceptually illustrates adding predetermined candidates and new merge candidates with changed motion attributes to the merge candidate list.

[0020] FIG. 4A shows the current sample and the reference sample used to calculate the template matching cost of the merge candidate of the current block.

[0021] FIG. 4B conceptually illustrates a merge candidate list sorted according to calculated TM costs.

[0022] FIG. 5 illustrates an example video encoder that may implement merge mode prediction.

[0023] FIG. 6 illustrates the portion of the video encoder that generates the merge candidate list and modifies motion properties.

[0024] FIG. 7 conceptually illustrates the process of modifying the motion attributes of a merge candidate.

[0025] FIG. 8 illustrates an example video decoder that may implement merge mode prediction.

[0026] FIG. 9 illustrates the portion of a video decoder that generates a merge candidate list and modifies motion properties.

[0027] FIG. 10 conceptually illustrates the process of modifying the motion attributes of a merge candidate.

[0028] FIG. 11 conceptually illustrates an electronic system in which some embodiments of the present invention may be implemented. DETAILED DESCRIPTION

[0029] In the following detailed description, a large number of specific details are described by way of example in order to provide a thorough understanding of the relevant teachings. Any changes, derivations and / or extensions based on the teachings described herein are within the scope of protection of the present invention. In some examples, in order to avoid unnecessary confusion of aspects of the teachings of the present invention, known methods, procedures, components and / or circuits related to one or more exemplary embodiments disclosed herein are described at a relatively high level without details.

[0030] I. Merging candidate motion attributes

[0031] When initially constructing a merge candidate list for the current block (the pixel block currently being encoded or decoded), the list includes a predetermined merge candidate set. Each predetermined merge candidate has a motion attribute set, which may include (but is not limited to) the candidate's inter-picture prediction direction (single / dual prediction), one or more reference indices, a bi-prediction with CU-level Weight (BCW) index with coding unit (Coding Unit, CU) layer weight, a local illumination compensation (Local Illumination Compensation, LIC) flag, a half-pixel filter used, a multi-hypothesis prediction (Multi-Hypothesis Prediction, MHP) weight index, etc.

[0032] A. Bi-prediction with CU-level Weight (BCW)

[0033] BCW is a coding tool for enhancing bidirectional prediction. BCW allows different weights to be applied to L0 prediction and L1 prediction before combining them to generate the bi-prediction of a CU. For a CU coded and decoded by BCW, the weight parameter w is indicated for L0 prediction and L1 prediction, whereby the bidirectional prediction result P is calculated based on w according to the following formula: bi-pred .

[0034] P bi-pred =((8–w)*P0+w*P1+4)>>3

[0035] P0 represents the pixel value predicted by L0 MV (or L0 prediction). P1 represents the pixel value predicted by L1 MV (or L1 prediction). bi-pred It is the weighted average of P0 and P1 according to w. For low-latency images, that is, images using reference frames with small image order numbers (Picture Order Count, POC), possible values ​​of w include {-2, 3, 4, 5, 10}, which are also referred to as BCW candidate weights. For non-low-latency images, possible values ​​of w (BCW candidate weights) include {3, 4, 5}. In some embodiments, for merge mode, the weights are extended from {-2, 3, 4, 5, 10} to {-4, -3, -2, -1, 1, 2, 3, 5, 6, 7, 9, 10, 11, 12} or any subset thereof. When negative dual prediction weights are not supported, the weights of the merge mode are extended from {-2, 3, 4, 5, 10} to {1, 2, 3, 4, 5, 6, 7}. In addition, the negative dual prediction weights of the non-merge mode are replaced with positive weights, that is, the weights {-2, 10} are replaced with {1, 7}.

[0036] B. Local Illumination Compensation (LIC)

[0037] LIC is an inter-picture prediction technique that models the local illumination variation between a current block and its prediction block as a function of the local illumination variation between a current block template and a reference block template. The parameters of the function can be expressed as a scale α and an offset β, thereby forming a linear equation, α*p[x]+β, to compensate for illumination variation. Where p[x] is the reference sample at position x on the reference image pointed to by MV. In some embodiments, since the parameters α and β can be derived based on the current block template and the reference block template, they do not require signaling overhead. The video encoder can mark the LIC flag to enable or disable the use of LIC.

[0038] C. Multi-Hypothesis Prediction (MHP)

[0039] In the multi-hypothesis inter-frame prediction mode, in addition to the conventional bi-prediction signal, one or more additional motion compensated prediction signals are indicated. The resulting total prediction signal is obtained by weighted addition of the samples. bi and the first additional inter-frame prediction signal / hypothesis h3, the resulting prediction signal p3 is obtained according to the following formula:

[0040] p3=(1–α)p bi +αh3

[0041] The weight factor α is specified by the syntax element add_hyp_weight_idx in the bitstream of the encoded video (eg, add_hyp_weight_idx=0, α=4; add_hyp-weight_idx=1, α=−1 / 8).

[0042] In some embodiments, more than one additional prediction signal may be used, and the final generated overall prediction signal is obtained by iteratively accumulating each additional prediction signal.

[0043] p n+1 =(1-α n+1 ) n +α n+1 h n+1

[0044] The last p n (i.e., the p with the largest index n n) is the resulting overall prediction signal. In some embodiments, up to two additional prediction signals may be used (i.e., n is limited to 2). The motion parameters of each additional prediction hypothesis may be explicitly indicated by specifying a reference index, a motion vector predictor index, and a motion vector difference, or implicitly indicated by specifying a merge index. A separate multi-hypothesis merge flag may distinguish between these two indication modes.

[0045] II. Update the motion attributes of the merge candidate

[0046] In order to improve video coding efficiency, some embodiments of the present invention provide a method for changing or updating the motion attribute of a merge candidate. This is in contrast to obtaining a merge candidate in a predetermined manner, where the motion attribute remains unchanged.

[0047] In some embodiments, the inter-picture prediction direction of the merge candidate as a motion attribute can be changed. For example, a dual-prediction merge candidate with L0 prediction and L1 prediction can be changed to a candidate with only L0 prediction, and / or a candidate with only L1 prediction. A candidate with only L0 prediction or only L1 prediction can be changed to a candidate with both L0 prediction and L1 prediction.

[0048] In some embodiments, the reference index of a merge candidate as a motion attribute can be changed. According to a scaling factor, which is determined based on the Picture Order Count (POC) distance between a reference image and a current image, the motion vector of the merge candidate can be scaled. (POC is an index assigned to each image in a video sequence to indicate their temporal order or temporal position in the video). Figure 1 illustrates changing the reference index of a merge candidate for a current block 101 in a current image 100. The merge candidate initially (when predefined) has a reference index that locates a reference image 110 (curr_ref) that is a distance tb from the current image 100POC. The changed reference index locates a different reference image 120 (new_ref) that is a distance td from the current image 100POC. Based on the scaling factor td / tb, the motion vector MV that initially references a sample in the reference image 110 becomes a scaled motion vector MV' to reference a sample in the reference image 120.

[0049] In some embodiments, the reference index can be changed so that the target reference image can be changed to any reference image in the available reference list (e.g., L0 reference list, L1 reference list). For example, the L0 reference index 1 reference image can be changed to the L0 reference index 0 reference image, or the L1 reference index 1 reference image. Therefore, for a merge candidate having a motion attribute with a dual prediction motion vector with reference indexes (RefIdx_L0, RefIdx_L1), RefIdx_L0 or RefIdx_L1- can be changed to any value between 0 and N-1, where N is the length of the L0 reference list and the L1 reference list. Therefore, the reference index of the merge candidate can be changed to any one of (0, 0), (0, 1), ..., (0, N-1), (1, 0), (1, 1), ..., (1, N-1), ... (N-1, N-1). For a merge candidate with a motion attribute having a single prediction motion vector with a reference index (RefIdx) and a reference list (RefList=Li, i=0 or 1), RefList can be changed to L0 or L1. RefIdx can be changed to any value between 0 and N-1, where N is the length of the L0 reference list and the L1 reference list. Therefore, the reference index and reference list of the merge candidate can be changed to any one of (0, L0), (1, L0), ..., (N-1, L0), (0, L1), (1, L1), ..., (N-1, L1).

[0050] In some embodiments, the reference index is only allowed to be changed to a picture with a scaling factor (based on POC) not greater than 1. In some embodiments, when the L0 reference list and the L1 reference list are the same and the new inter-picture prediction direction is bi-prediction, the L0 reference index and the L1 reference index are allowed to change only when the new L0 reference picture and the new L1 reference picture (indicated by the changed L0 reference index and the changed L1 reference index) are two different pictures. For example, in a low-delay bi-prediction configuration, the POCs of the reference pictures in the reference list are all less than the POC of the current picture, and the L0 reference list is the same as the L1 reference list.

[0051] In some embodiments, when the video is encoded and decoded in a random access configuration and the new inter-picture prediction direction is bi-prediction, two reference index changes are allowed only when the new reference picture indicated by the changed index provides true bi-prediction (e.g., the new L0 reference picture and the new L1 reference picture are in opposite temporal directions relative to the current picture). When the video is encoded and decoded in a random access configuration, the POC of the reference picture in the reference list can be smaller or larger than the POC of the current picture. In some other embodiments, the reference index change is allowed only when the new reference picture indicated by the changed reference index remains in the same reference list. For example, if the reference index represented by RefIdxL0 specifies a reference picture used in the L0 reference list, the new RefIdxL0 also specifies a reference picture used in the L0 reference list.

[0052] In some embodiments, the BCW weight indicated by the BCW index of the merge candidate as a motion attribute can be changed. The BCW index value can be selected from the allowed values ​​in the current video codec settings. In some embodiments, the BCW index can be changed to indicate equal weights, or changed to any other BCW index. In some embodiments, the BCW index of the merge candidate can be changed (to a BCW index indicating equal weights or another BCW index indicating unequal weights) only when the BCW index indicates unequal weights. In some embodiments, when the BCW index of the merge candidate indicates a positive value, only the BCW index is allowed to be changed to indicate another positive value.

[0053] In some embodiments, the LIC flag of the merge candidate may be changed. The LIC flag may be changed from true (eg, indicating that LIC is enabled) to false (eg, indicating that LIC is disabled), or vice versa.

[0054] In some embodiments, the half-pixel filter used as a merge candidate for a motion attribute can be changed. For example, for a half-luminance sample position, the merge candidate can be changed from using a 6-tap interpolation filter to using a default 8-tap interpolation filter, or vice versa.

[0055] In some embodiments, the MHP weight index used as a merging candidate for a motion attribute may be changed. For example, the MHP weight index may be changed from 0 to 1, or vice versa.

[0056] III. Update the Merger Candidate List

[0057] In some embodiments, for each predetermined candidate in the merge candidate list, the motion attribute of the candidate can be changed based on a template matching (TM) cost evaluation. Specifically, in some embodiments, if the TM cost caused by changing the motion attribute of the predetermined merge candidate is less than a threshold value than the TM cost of the predetermined merge candidate with the original motion attribute, the predetermined merge candidate is replaced with an updated merge candidate with the changed motion attribute. FIG. 2 conceptually illustrates updating the motion attribute of a merge candidate based on the TM cost.

[0058] As shown, the merge candidate list 250 of the current block being encoded or decoded is initially populated with predetermined merge candidates 251-256. Each merge candidate may have a motion attribute set, which may include a candidate inter-frame prediction direction, one or more reference indices, a BCW index, a LIC flag, a half-pixel filter used, an MHP weight index, etc. In this example, predetermined merge candidate 254 (merge candidate 4) has a motion attribute set, denoted as attribute A. The video codec checks several possible changes to attribute A of merge candidate 254, including attribute A' and attribute A".

[0059] The template matching process 220 is used to calculate the TM costs of the original predetermined merge candidate 254 and the modified merge candidate 261 and the modified merge candidate 262. (The modified merge candidate 261 has the modified motion attribute A', while the modified merge candidate 262 has the modified motion attribute A"). Based on the calculated TM costs, the cost comparison process 230 is used to determine whether to replace / update / modify the merge candidate 254 to a modified merge candidate with the changed motion attribute. In this example, the merge candidate 254 is replaced with the modified merge candidate 261 (with attribute A').

[0060] In some implementations, if the TM cost of the modified merge candidate (e.g., 261 and 262) is not less than the TM cost of the original predetermined merge candidate 254 by more than a threshold, the original predetermined merge candidate 254 is not replaced or modified. Conversely, if the TM cost of the modified merge candidate is less than the TM cost of the original predetermined merge candidate 254 by more than a threshold, the modified merge candidate (261 in this example) can replace the original predetermined merge candidate 254 in the merge candidate list 250.

[0061] In some embodiments, the updated merge candidate list 260 is subjected to a candidate reordering process based on the TM costs of the candidates in the list. In some embodiments, the reordering process is performed according to the TM process described in Section IV below.

[0062] In some embodiments, to create a merge candidate list, in addition to the predetermined merge candidates, merge candidates with changed motion attributes are also added to the merge candidate list. In some embodiments, such a merge candidate list has an upper limit of a predetermined size. The TM process can then be performed on the created merge candidate list, which includes candidates with changed motion attributes.

[0063] FIG. 3 conceptually illustrates adding predetermined candidates and new merge candidates with changed motion attributes to a merge candidate list. In this example, the merge candidate list 350 for the current block originally has predetermined merge candidates 351-356, each with an original set of motion attributes. The video codec then adds new merge candidates 362, 364, and 365 to the merge candidate list 350 (to form an updated merge candidate list 360). The added new merge candidates 362, 364, and 365 have the modified motion attributes (B', D', E') of the predetermined merge candidates 352, 354, and 355, respectively.

[0064] In some embodiments, the predetermined candidates and the candidates with changed motion attributes are added to the merged candidate list in some predetermined order. For example, in some embodiments, the predetermined candidates can be added to the list before all candidates with changed motion attributes are added. For another example, a first predetermined candidate and a candidate with changed motion attributes created from the first predetermined candidate can be added to the list as a first group, and then a second predetermined candidate and a candidate with changed motion attributes created from the second predetermined candidate can be added to the list as a second group, and so on.

[0065] In some embodiments, when updating the merge candidate list, some attribute changes may be prioritized. Therefore, new merge candidates with prioritized motion attribute changes are added to the list before other new candidates with other motion attribute changes. For example, the reference index may be a motion attribute that has been prioritized to change. Therefore, a predetermined merge candidate is added to the merge candidate list, and then one or more new candidates with changed reference indexes based on the predetermined merge candidate are added to the list. Other predetermined merge candidates may then be added. Finally, new merge candidates with changed motion attributes that do not include reference index changes are added.

[0066] IV.TM Cost

[0067] In some embodiments, the template matching cost of the merge candidate is measured by the Sum of Absolute Differences (SAD) between the samples of the current template and its corresponding samples in the reference template identified by the merge candidate. Figure 4A shows the current sample and the reference sample used to calculate the template matching cost of the merge candidate for the current block 410. In some embodiments, the template matching cost of the merge candidate is measured by the Sum of Absolute Transformed Differences (SATD) between the samples of the current template and its corresponding samples in the reference template identified by the merge candidate. In some embodiments, the template matching cost of the merge candidate is measured by a combination of SAD and SATD between the samples of the current template and its corresponding samples in the reference template identified by the merge candidate.

[0068] A current block 410 is in a current image 400. A set of reconstructed samples adjacent to the current block 410 is used as a current template 415. The current block is associated with a merge candidate list 450, which includes merge candidates 451-456. Among them, merge candidate 454 is a bi-prediction candidate with motion information MV0 and MV1. MV0 locates reference block 420 in L0 reference image 401. MV1 locates reference block 430 in L1 reference image 402. Collocated reference samples of the current template 415 are located by MV0 to reference template 425 and by MV1 to reference template 435. Based on the motion properties of the merge candidate 454, a final reference sample is generated by bi-prediction from samples of reference templates 425 and 435. The template matching cost of the merge candidate 454 is the difference between the samples of the current template 415 and the final reference sample. The difference can be measured by SAD, SATD, or a combination of SAD and SATD.

[0069] Template matching cost can also be calculated for a single prediction merge candidate. Merge candidate 453 is a single prediction candidate with motion information MV0. MV0 locates reference block 440 in L0 reference image 403. The co-located reference sample of current template 415 is located by MV0 to reference template 445. Based on the samples of reference template 445 and the motion properties of merge candidate 453, a final reference sample is generated. The template matching cost of merge candidate 453 is the difference between the samples of current template 415 and the final reference sample. The difference can be measured by SAD, SATD, or a combination of SAD and SATD.

[0070] The template matching cost may be calculated for each merge candidate in the merge candidate list 450, and the merge candidate list 450 may then be sorted according to the calculated TM cost. FIG. 4B conceptually illustrates the merge candidate list 450 being sorted according to the calculated TM cost. In this example, a template matching process is performed on each merge candidate to calculate the TM cost, and the merge candidate list 450 is sorted based on the calculated TM cost to form a re-ordered candidate list 460. In some embodiments, the video encoder may check all merge candidates in the re-ordered list 460 to determine whether to modify their motion properties, while the video decoder will only check and modify the motion properties of the merge candidates selected by the indicated merge candidate index.

[0071] In some embodiments, TM cost values ​​are calculated for different bi-prediction weights, and the bi-prediction weight with the minimum TM cost value is used to predict the current block.

[0072] Adaptive Reordering of MergeCandidates with Template Matching (ARMC-TM) is a method to reorder merge candidates based on TM cost, where the signaling efficiency is improved by sorting the merge candidates in ascending order of TM cost. For TM merge mode, the merge candidates are reordered before the refinement process.

[0073] In some embodiments, after constructing the merge candidate list, the merge candidates are divided into several subgroups. For regular merge mode and TM merge mode, the subgroup size is set to 5. For affine merge mode, the subgroup size is set to 3. Based on template matching, the merge candidates in each subgroup are sorted in ascending order according to the cost value. In some embodiments, the merge candidates in the last and not the first subgroup are not reordered.

[0074] For some embodiments, the above-mentioned proposed method can be applied to conventional ARMC-TM and / or ARMC based on motion vector (MV) candidate type. For example, the proposed method can be applied to temporal motion vector prediction (TMVP) candidate ARMC, and / or non-adjacent MVP (NA-MVP) ARMC, and / or ARMC-TM.

[0075] V. Example Video Decoder

[0076] FIG. 5 illustrates an exemplary video encoder 500 that can implement merge mode prediction. As shown in the figure, the video encoder 500 receives an input video signal from a video source 505 and encodes the signal into a bitstream 595. The video encoder 500 has several components or modules for encoding the signal from the video source 505, including at least some components selected from a transform module 510, a quantization module 511, an inverse quantization module 514, an inverse transform module 515, an intra-image estimation module 520, an intra-image prediction module 525, a motion compensation module 530, a motion estimation module 535, a loop filter 545, a reconstructed image buffer 550, a motion vector (MV) buffer 565, a motion vector prediction module 575, and an entropy encoder 590. The motion compensation module 530 and the motion estimation module 535 are part of the inter-image prediction module 540.

[0077] In some embodiments, modules 510-590 are modules of software instructions being executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 510-590 are modules of hardware circuits implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 510-590 are shown as separate modules, some of these modules can be combined into a single module.

[0078] The video source 505 provides a raw video signal, which represents pixel data for each video frame without compression. The subtractor 508 calculates the difference between the raw video pixel data of the video source 505 and the predicted pixel data 513 from the motion compensation module 530 or the intra-frame prediction module 525 as a prediction residual 509. The transform module 510 transforms the difference (or residual pixel data or residual signal 508) into transform coefficients (e.g., by performing a discrete cosine transform or DCT). The quantization module 511 quantizes the transform coefficients into quantized data (or quantized coefficients) 512, which are encoded into the bitstream 595 by the entropy encoder 590.

[0079] The inverse quantization module 514 dequantizes the quantized data (or quantized coefficients) 512 to obtain transform coefficients, and the inverse transform module 515 inversely transforms the transform coefficients to generate a reconstructed residual 519. The reconstructed residual 519 is added to the predicted pixel data 513 to generate reconstructed pixel data 517. In some embodiments, the reconstructed pixel data 517 is temporarily stored in a line buffer (not shown) for intra-frame image prediction and spatial MV prediction. The reconstructed pixels are filtered by the loop filter 545 and stored in the reconstructed image buffer 550. In some embodiments, the reconstructed image buffer 550 is a storage external to the video codec 500. In some embodiments, the reconstructed image buffer 550 is a storage internal to the video encoder 500.

[0080] The image intra-picture estimation module 520 performs intra-picture prediction based on the reconstructed pixel data 517 to generate intra-picture prediction data. The intra-picture prediction data is provided to the entropy encoder 590 to be encoded into a bitstream 595. The intra-picture prediction data is also used by the intra-picture prediction module 525 to generate the predicted pixel data 513.

[0081] The motion estimation module 535 performs inter-picture prediction by generating motion vectors to reference pixel data of previously decoded frames stored in the reconstructed picture buffer 550. These motion vectors are provided to the motion compensation module 530 to generate predicted pixel data.

[0082] Instead of encoding the complete actual MV in the bitstream, the video codec 500 uses MV prediction to generate a predicted MV, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 595.

[0083] The motion vector prediction module 575 generates a predicted motion vector, i.e., a motion compensated motion vector used to perform motion compensation, based on the reference motion vector generated for encoding the previous video information frame. The motion vector prediction module 575 retrieves the reference motion vector from the previous video information frame from the motion vector buffer 565. The video encoder 500 stores these motion vectors generated for the current video information frame in the motion vector buffer 565 as reference motion vectors for generating the predicted motion vector.

[0084] The motion vector prediction module 575 uses the reference motion vector to create a predicted motion vector. The predicted motion vector can be calculated by spatial motion vector prediction or temporal motion vector prediction. The difference (residual motion data) between the predicted motion vector and the motion compensation motion vector (MC MV) of the current information frame is encoded into a bit stream 595 by the entropy encoder 590.

[0085] Using entropy coding techniques, such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding, the entropy encoder 590 encodes various parameters and data into a bitstream 595. The entropy encoder 590 encodes various header elements, flags, and quantized transform coefficients 512 and residual motion data as syntax elements into the bitstream 595. In turn, the bitstream 595 is stored in a storage device or transmitted to a decoder via a communication medium such as a network.

[0086] The loop filter 545 performs filtering or smoothing operations on the reconstructed pixel data 517 to reduce encoding and decoding artifacts, especially artifacts located at the boundaries of pixel blocks. In some embodiments, the filtering operation or smoothing operation performed by the loop filter 545 includes a deblocking filter (DBF), a sample adaptive offset (SAO) and / or an adaptive loop filter (ALF).

[0087] FIG6 illustrates the portion of the video encoder 500 that generates the merge candidate list and modifies the motion properties. Specifically, the figure illustrates the components of the motion compensation module 530 of the video encoder 500.

[0088] As shown, the motion compensation module 530 has a merge candidate list constructor 610, which generates a merge candidate list 615. The list 615 is initially generated based on the previously generated MV stored in the MV buffer 565 and includes predetermined merge candidates. The merge candidate list constructor 610 can modify the motion attributes of the predetermined merge candidates and reorder the candidates in the list 615. The merge candidate list constructor 610 can also add additional merge candidates to the list 615 based on the modified motion attributes. The modification and reordering can be based on the TM cost calculated by the TM cost calculation module 630 for each merge candidate with or without the modified motion attribute. A template matching operation is performed based on pixel samples stored in the reconstructed image buffer 550, and the pixel samples may include samples of a current template adjacent to the current block and samples of a reference template adjacent to the reference block. The reference block can be located by motion information determined based on the motion attributes of each merge candidate. Examples of the motion attributes of the merge candidates are described in Section I above.

[0089] The motion estimation module 535 provides a merge candidate selected from the list 615, which may have been reordered and / or modified by the merge candidate list constructor 610 as described above. The selection of the merge candidate is also provided to the entropy encoder 590 to be identified as a merge index. The selected merge candidate and its associated motion attribute are provided to the prediction generator 620, which retrieves the corresponding predicted pixels from the reconstructed image buffer 550. The prediction generator 620 may perform blending based on weight factors specified by the motion attributes of the selected merge candidate.

[0090] FIG. 7 conceptually illustrates a process 700 for modifying motion attributes of a merge candidate. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing encoder 500 perform process 700 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing encoder 500 performs process 700.

[0091] An encoder receives (at block 710) data to be encoded as a current block of pixels in a current picture of video.

[0092] The encoder generates (at block 720) a merge candidate list for the current block. Each merge candidate is associated with a motion attribute, which may be an inter-picture prediction direction, a reference index, a bi-prediction with CU-level Weight (BCW) index, a local illumination compensation (LIC) flag, a half-pixel filter used, or a multi-hypothesis prediction (MHP) weight index.

[0093] The encoder modifies (at block 730) the merge candidate list by changing a motion attribute of a first merge candidate from a first value to a second value. In some embodiments, the merge candidate list is modified when changing the motion attribute of the first merge candidate increases an estimated cost of encoding the current block using the first merge candidate by more than a threshold. In some embodiments, the estimated cost is a Template Matching Cost (TM) cost calculated by determining a difference between (i) a current template region adjacent to the current block and (ii) a reference template region adjacent to a reference block identified by the first merge candidate. Other cost measures may also be used as the estimated cost, such as a boundary matching (BM) cost calculated by determining a measure of discontinuity along a boundary of the current block (e.g., between reconstructed neighboring samples and predicted samples of the current block)

[0094] In some embodiments, the merge candidate list is modified by adding a second merge candidate having the modified motion attribute. In some embodiments, the merge candidate list is modified by replacing the first merge candidate with the second merge candidate having the modified motion attribute.

[0095] In some embodiments, the encoder changes the motion attribute of the first merge candidate by changing the reference index from identifying the first reference image to identifying the second reference image. The encoder may change the motion attribute of the first merge candidate by scaling the motion vector based on a picture order count (POC) distance of the first reference image and a picture order count (POC) distance of the second reference image. In some embodiments, the encoder changes the motion attribute of the first merge candidate by changing a dual prediction weight index (e.g., BCW index) to select different weights for combining the first (e.g., L0) inter-picture prediction and the second (e.g., L1) inter-picture prediction.

[0096] The encoder indicates (at block 740) to select a merge candidate from the modified merge candidate list. In some embodiments, the encoder calculates a template matching cost for each merge candidate in the merge candidate list and reorders the list based on the template matching costs of the plurality of merge candidates in the list. The merge candidate is selected based on the reordered list.

[0097] The encoder generates a prediction by encoding the current block (at block 750) using the selected merge candidate and generates a prediction residual.

[0098] VI. Sample Video Decoder

[0099] In some embodiments, the encoder may indicate (or generate) one or more syntax elements in the bitstream, so that the decoder may parse the one or more syntax elements from the bitstream.

[0100] FIG. 8 illustrates an example video decoder 800 that can implement merge mode prediction. As shown, the video decoder 800 is an image decoding or video decoding circuit that receives a bitstream 895 and decodes the content of the bitstream into pixel data of a video information frame for display. The video decoder 800 has several components or modules for decoding the bitstream 895, including some components selected from an inverse quantization module 811, an inverse transform module 810, an intra-picture prediction module 825, a motion compensation module 830, a loop filter 845, a decoded image buffer 850, a motion vector buffer 865, a motion vector prediction module 875, and a parser 890. The motion compensation module 830 is a part of the inter-picture prediction module 840.

[0101] In some embodiments, modules 810-890 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 810-890 are hardware circuit modules implemented by one or more ICs of an electronic device. Although modules 810-890 are illustrated as separate modules, some of these modules may be combined into a single module.

[0102] The parser 890 (or entropy decoder) receives the bitstream 895 and performs initial parsing according to the syntax defined by the video codec or image codec standard. The parsed syntax elements include various header elements, flags, and quantized data (or quantized coefficients) 812. The parser 890 parses out the various syntax elements by using entropy coding techniques such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding.

[0103] The inverse quantization module 811 dequantizes the quantized data (or quantized coefficients) 812 to obtain transform coefficients, and the inverse transform module 810 inversely transforms the transform coefficients 816 to generate a reconstructed residual 819. The reconstructed residual 819 is added to the predicted pixel data 813 from the intra-picture prediction module 825 or the motion compensation module 830 to generate decoded pixel data 817. The decoded pixel data is filtered by the loop filter 845 and stored in the decoded image buffer 850. In some embodiments, the decoded image buffer 850 is a storage outside the video decoder 800. In some embodiments, the decoded image buffer 850 is a storage inside the video decoder 800.

[0104] The intra-picture prediction module 825 receives the intra-picture prediction data from the bitstream 895 and generates predicted pixel data 813 based on the intra-picture prediction data from the decoded pixel data 817 stored in the decoded image buffer 850. In some embodiments, the decoded pixel data 817 is also stored in a line buffer (not shown) for intra-picture prediction and spatial MV prediction.

[0105] In some embodiments, the contents of the decoded image buffer 850 are used for display. The display device 855 retrieves the contents of the decoded image buffer 850 for direct display, or retrieves the contents of the decoded image buffer to a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded image buffer 850 via pixel transfer.

[0106] Based on the motion compensated MV (MC MV), the motion compensation module 830 generates predicted pixel data 813 from the decoded pixel data 817 stored in the decoded picture buffer 850. These motion compensated MVs are decoded by adding the residual motion data received from the bitstream 895 to the predicted MV received from the motion vector prediction module 875.

[0107] The motion vector prediction module 875 generates a predicted MV, for example, a motion compensated MV for performing motion compensation, based on a reference MV generated for decoding a previous video information frame. The motion vector prediction module 875 retrieves the reference motion vector of the previous video information frame from the motion vector buffer 865. The video decoder 800 also stores the motion compensated motion vector generated for decoding the current video information frame in the motion vector buffer 865 as a reference motion vector for generating a predicted motion vector.

[0108] The loop filter 845 performs filtering or smoothing operations on the decoded pixel data to reduce encoding and decoding artifacts, especially artifacts located at the boundaries of pixel blocks. In some embodiments, the filtering or smoothing operations performed by the loop filter 845 include a deblock filter (DBF), a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF).

[0109] FIG9 illustrates the portion of the video decoder 800 that generates the merge candidate list and modifies the motion attributes. Specifically, the figure illustrates the components of the motion compensation module 830 of the video decoder 800.

[0110] As shown, the motion compensation module 830 has a merge candidate list constructor 910, which generates a merge candidate list 915. The list 915 is initially generated based on the previously generated MV stored in the MV buffer 865 and includes predetermined merge candidates. The merge candidate list constructor 910 can modify the motion attributes of the predetermined merge candidates and reorder the candidates in the list 915. The merge candidate list constructor 910 can also add additional merge candidates to the list 915 based on the modified motion attributes. The modification and reordering can be based on the TM cost calculated by the TM cost calculation module 930 for each merge candidate with or without the modified motion attribute. A template matching operation is performed based on pixel samples stored in the decoded image buffer 850, and the pixel samples may include samples of a current template adjacent to the current block and samples of a reference template adjacent to the reference block. The reference block can be located by motion information determined based on the motion attributes of each merge candidate. Examples of the motion attributes of the merge candidates are described in Section I above.

[0111] The entropy decoder 890 may receive a merge index indicated in the bitstream 895. The received merge index is used to select a candidate from a merge candidate list 915, which may have been reordered and / or modified by the merge candidate list constructor 910 as described above. The selected merge candidate and its associated motion attribute are provided to the prediction generator 920, which obtains the corresponding predicted pixels from the decoded image buffer 850. The prediction generator 920 may perform blending based on a weight factor specified by the motion attribute of the selected merge candidate.

[0112] FIG. 10 conceptually illustrates a process 1000 for modifying motion attributes of a merge candidate. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing decoder 800 perform process 1000 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing decoder 800 performs process 1000.

[0113] The decoder receives (at block 1010) data to be decoded as a current block of pixels in a current picture of video.

[0114] The decoder generates (at block 1020) a merge candidate list for the current block. Each merge candidate is associated with a motion attribute, which may be an inter-picture prediction direction, a reference index, a bi-prediction with CU-level Weight (BCW) index, a local illumination compensation (LIC) flag, a half-pixel filter used, or a multi-hypothesis prediction (MHP) weight index.

[0115] The decoder modifies (at block 1030) the list of merge candidates by changing a motion attribute of the first merge candidate from a first value to a second value. In some embodiments, the list of merge candidates is modified when changing the motion attribute of the first merge candidate increases an estimated cost of decoding the current block using the first merge candidate by more than a threshold. In some embodiments, the estimated cost is a Template Matching Cost (TM) cost calculated by determining a difference between (i) a current template region adjacent to the current block and (ii) a reference template region adjacent to a reference block identified by the first merge candidate. Other cost measures may also be used as the estimated cost, such as a Boundary Matching (BM) cost calculated by determining a measure of discontinuity along a boundary of the current block (e.g., between reconstructed neighboring samples and predicted samples of the current block).

[0116] In some embodiments, the merge candidate list is modified by adding a second merge candidate having the modified motion attribute. In some embodiments, the merge candidate list is modified by replacing the first merge candidate with the second merge candidate having the modified motion attribute.

[0117] In some embodiments, the decoder changes the motion attribute of the first merge candidate by changing the reference index from identifying the first reference image to identifying the second reference image. The decoder may change the motion attribute of the first merge candidate by scaling the motion vector based on a picture order count (POC) distance of the first reference image and a picture order count (POC) distance of the second reference image. In some embodiments, the decoder changes the motion attribute of the first merge candidate by changing a bi-prediction weight index (e.g., BCW index) to select different weights for combining the first (e.g., L0) inter-picture prediction and the second (e.g., L1) inter-picture prediction.

[0118] The decoder receives (at block 1040) a selected merge candidate from the modified merge candidate list. In some embodiments, the decoder calculates a template matching cost for each merge candidate in the merge candidate list and reorders the list based on the template matching costs of the plurality of merge candidates in the list. The merge candidate is selected based on the reordered list.

[0119] The decoder reconstructs (at block 1050) the current block to generate a predicted block using the selected merge candidate. The decoder may then provide the reconstructed current block for display as part of the reconstructed current picture.

[0120] VII. Example Electronic System

[0121] Many of the above features and applications can be implemented as software processes, which are specified as sets of instructions recorded on a computer readable storage medium (also referred to as a computer readable medium). When these instructions are executed by one or more computing units or processing units (e.g., one or more processors, processor cores, or other processing units), the instructions cause the processing units to perform the actions represented by the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard disks, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), etc. Computer readable media do not include carriers and telecommunications signals connected wirelessly or wired.

[0122] In this specification, the term "software" is meant to include firmware in a read-only memory or an application stored in a magnetic storage device, which can be read into the memory for processing by the processor. At the same time, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program, while retaining different software inventions. In some embodiments, multiple software inventions can be implemented as independent programs. Finally, any combination of independent programs that implement the software inventions described herein together is within the scope of the present invention. In some embodiments, when installed to operate on one or more electronic systems, the software program defines one or more specific machine implementations that execute and implement the operations of the software program.

[0123] FIG. 11 conceptually illustrates an electronic system 1100 implemented in some embodiments of the present invention. The electronic system 1100 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or other types of electronic devices. The electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1100 includes a bus 1105, a processing unit 1110, a graphics-processing unit (GPU) 1115, a system memory 1120, a network 1125, a read-only memory (ROM) 1130, a permanent storage device 1135, an input device 1140, and an output device 1145.

[0124] Bus 1105 collectively represents all system buses, peripheral buses, and chipset buses that are communicatively coupled to a number of internal devices of electronic system 1100. For example, bus 1105 is communicatively coupled to processing unit 1110 via graphics processing unit 1115, read-only memory 1130, system memory 1120, and permanent storage device 1135.

[0125] From these various memory units, the processing unit 1110 retrieves instructions to execute and data to process in order to perform the processes of the present invention. In various embodiments, the processing unit may be a single processor or a multi-core processor. Certain instructions are transmitted to and executed by the image processing unit 1115. The image processing unit 1115 may offload various calculations or supplement the image processing provided by the processing unit 1110.

[0126] The read-only memory 1130 stores static data and instructions required by the processing unit 1110 or other modules of the electronic system. On the other hand, the permanent storage device 1135 is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 1100 is turned off. Some embodiments of the present invention use a large-capacity storage device (such as a magnetic disk or optical disk and its corresponding magnetic disk drive) as the permanent storage device 1135.

[0127] Other embodiments use removable storage devices (such as floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Like permanent storage device 1135, system memory 1120 is a read-write memory device. However, unlike storage device 1135, system memory 1120 is a volatile read-write memory, such as random access memory. System memory 1120 stores instructions and data that some processors need when running. In some embodiments, processing according to the present invention is stored in system memory 1120, permanent storage device 1135 and / or read-only memory 1130. For example, various memory units include instructions for processing multimedia clips according to some embodiments. For these various memory units, processing unit 1110 retrieves executed instructions and processed data in order to perform processing of certain embodiments.

[0128] The bus 1105 is also connected to input devices 1140 and output devices 1145. The input devices 1140 enable a user to communicate information and select commands to the electronic system. The input devices 1140 include an alphanumeric keyboard and pointing device (also known as a "cursor control device"), a camera (such as a webcam), a microphone or similar device for receiving voice commands, etc. The output devices 1145 display images generated by the electronic system or data output in other ways. The output devices 1145 include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD), and speakers or similar audio output devices. Some embodiments include devices such as touch screens that function as both input devices and output devices.

[0129] Finally, as shown in Figure 11, bus 1105 also couples electronic system 1100 to a network 1125 via a network adapter (not shown). In this manner, the computer may be part of a computer network (e.g., a local area network (LAN), a wide area network (WAN) or an intranet, or a network of networks (e.g., the Internet). Any or all of the components of electronic system 1100 may be used in conjunction with the present invention.

[0130] Some embodiments include electronic components, such as a microprocessor, a storage device, and a memory that stores computer program instructions to a machine-readable medium or a computer-readable medium (optionally referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable Computer readable media may store computer programs executed by at least one processing unit and include sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that produced by a compiler, and files containing high-level code executed by a computer, electronic component, or microprocessor using an interpreter.

[0131] While the above discussion refers primarily to microprocessors or multi-core processors that execute software, many of the above functions and applications are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0132] As used in the specification and any claims of the present invention, the terms "computer", "server", "processor" and "memory" refer to electronic devices or other technical devices. These terms do not include people or groups. For the purpose of this specification, the term display or display device refers to displaying on an electronic device. As used in the specification and any claims of the present invention, the terms "computer-readable medium", "computer-readable medium" and "machine-readable medium" are entirely limited to tangible, physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals and any other transient signals.

[0133] While the present invention has been described in conjunction with many specific details, it will be appreciated by those skilled in the art that the present invention may be implemented in other specific forms without departing from the spirit of the present invention. In addition, a large number of figures (including Figures 7 and 10) conceptually illustrate processes. The specific operations of these processes may not be performed in the exact order shown and described. These specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the process is implemented using several subprocesses, or as part of a larger macroprocess. Therefore, it will be appreciated by those skilled in the art that the present invention is not limited by the aforementioned illustrative details, but is defined by the scope of the patent application.

[0134] Additional Notes

[0135] The subject matter described herein sometimes represents different components that are contained in or connected to other different components. It is understood that the described structure is only an example and can actually be implemented by many other structures to implement the same function. Conceptually, any arrangement of components that implement the same function is actually "associated" in order to implement the desired function. Therefore, regardless of the structure or intermediate components, any two components combined to implement a specific function are considered to be "interrelated" to implement the desired function. Similarly, any two associated components are considered to be "operably connected" or "operably coupled" to each other to implement a specific function. Any two components that can be associated with each other are also considered to be "operably coupled" to each other to implement a specific function. Specific examples of operable connections include, but are not limited to, physically pairable and / or physically interacting components, and / or wirelessly interactable and / or wirelessly interacting components, and / or logically interacting and / or logically interacting components.

[0136] Furthermore, with respect to the use of substantially any plural and / or singular terms, those of ordinary skill in the art can translate from the plural to the singular and / or from the singular to the plural as appropriate to the context and / or application. For clarity, the various singular / plural permutations are expressly provided herein.

[0137] In addition, it will be understood by those skilled in the art that, in general, the terms used in the present invention, especially in the claims, and the subject matter of the claims, are generally used as "open" terms, for example, "including" should be interpreted as "including but not limited to", "having" should be interpreted as "at least having", "including" should be interpreted as "including but not limited to", etc. It will be further understood by those skilled in the art that if a specific number of claim contents are intended to be introduced, it will be explicitly indicated in the claim, and will not be displayed in the absence of such contents. For example, to aid understanding, the claims may include the phrases "at least one" and "one or more" to introduce the contents of the claims. However, the use of these phrases should not be interpreted as implying the use of the indefinite article "a" or "an" to introduce the contents of the claims, thereby limiting any particular patent scope. Even when the same claim includes the introductory phrases "one or more" or "at least one", the indefinite article, such as "a" or "an", should be interpreted to mean at least one or more, and the same is true for the use of the explicit description used to introduce the claims. Furthermore, even when an introductory content explicitly refers to a specific number, one of ordinary skill in the art will recognize that such content should be interpreted as indicating the number of references, for example, "two references" without other modifications means at least two references, or two or more references. Furthermore, where expressions similar to "at least one of A, B, and C" are used, they are generally expressed so that one of ordinary skill in the art can understand the expression, for example, "a system includes at least one of A, B, and C" will include but is not limited to a system with A alone, a system with B alone, a system with C alone, a system with A and B, a system with A and C, a system with B and C, and / or a system with A, B, and C, etc. One of ordinary skill in the art will further understand that any separated words and / or phrases represented by two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of one of these terms, one of them, or both of these terms. For example, "A or B" should be understood as the possibility of "A", or "B", or "A and B".

[0138] As can be seen from the foregoing, various embodiments have been described herein for illustrative purposes and that various modifications may be made without departing from the scope and spirit of the invention. Therefore, the various embodiments disclosed herein are not intended to be limiting, with the claims indicating the true scope and spirit.

Claims

1. A video encoding and decoding method, comprising: receiving data of a pixel block to be encoded or decoded as a current block of a current image of a video; Generate a merge candidate list for the current block; modifying the merge candidate list by changing a motion attribute of a first merge candidate from a first value to a second value; Indicating or receiving a selection of a merge candidate from the modified merge candidate list; as well as The current block is encoded or decoded by using the selected merge candidate.

2. The method of claim 1, further comprising: A template matching cost of each merge candidate in the merge candidate list is calculated, and the list is reordered according to the calculated template matching costs of the plurality of merge candidates in the list, wherein the merge candidate is selected based on the reordered list.

3. The method of claim 1, wherein: The merge candidate list is modified when a change in the motion attribute of the first merge candidate increases an estimated cost of encoding or decoding the current block using the first merge candidate by more than a threshold.

4. The method of claim 3, wherein: The estimated cost is calculated by determining a difference between a current template region adjacent to the current block and a reference template region adjacent to a reference block identified by the first merge candidate.

5. The method of claim 1, wherein: The merge candidate list is modified by adding a second merge candidate having the modified motion attribute.

6. The method of claim 1, wherein: The merge candidate list is modified by replacing the first merge candidate with a second merge candidate having modified motion attributes.

7. The method of claim 1, wherein: Changing the motion attribute of the first merging candidate includes: The reference index is changed from identifying the first reference image to identifying the second reference image.

8. The method of claim 7, wherein: Changing the motion attribute of the first merging candidate further includes: The motion vector is scaled based on a Picture Order Count (POC) distance of the first reference image and a Picture Order Count (POC) distance of the second reference image.

9. The method of claim 1, wherein: Changing the motion attribute of the first merging candidate includes: The bi-prediction weight index is changed to select different weights for combining the first inter-picture prediction and the second inter-picture prediction.

10. The method of claim 1, wherein: The motion attribute of the first merge candidate being changed is one of the inter-picture prediction direction, the reference index, the bi-prediction with CU-level Weight (BCW) index with the coding unit (CU) layer weight, the local illumination compensation (LIC) flag, the half-pixel filter used, and the multi-hypothesis prediction (MHP) weight index.

11. A video decoding method, comprising: receiving data of a pixel block to be decoded as a current block of a current image of a video; Generate a merge candidate list for the current block; modifying the merge candidate list by changing a motion attribute of a first merge candidate from a first value to a second value; receiving a selection of a merge candidate from the modified merge candidate list; as well as The current block is reconstructed by using the selected merge candidates.

12. A video encoding method, comprising: receiving data of a pixel block to be encoded as a current block of a current image of a video; Generate a merge candidate list for the current block; modifying the merge candidate list by changing a motion attribute of a first merge candidate from a first value to a second value; Indicates the selection of a merge candidate from the modified merge candidate list; as well as The current block is encoded using the selected merge candidates to generate a prediction block.

13. An electronic device comprising: A video codec circuit is configured to perform a plurality of operations including: receiving data of a pixel block to be encoded or decoded as a current block of a current image of a video; Generate a merge candidate list for the current block; modifying the merge candidate list by changing a motion attribute of a first merge candidate from a first value to a second value; Indicating or receiving a selection of a merge candidate from the modified merge candidate list; and The current block is encoded or decoded by using the selected merge candidate.

Images