Decoding method and decoding apparatus

Abstract

The described technology relating to methods, apparatuses, and computer readable media for determining motion vectors is relevant to both encoders and decoders. For example, a decoder receives compressed video data relating to a set of frames, the decoder computes a new motion vector for a current frame from the set of frames using a decoder-side predictor refinement technique, where the new motion vector estimates motion of the current frame based on one or more reference frames. The computing includes retrieving a first motion vector relating to the current frame, performing a first portion of a decoding process using the first motion vector, retrieving a second motion vector relating to the current frame, the second motion vector being different than the first motion vector, and performing a second portion of the decoding process using the second motion vector.

Description

[0001] Related citations

[0002] This application claims priority to U.S. Provisional Application No. 62 / 442,472, filed January 5, 2017, entitled “METHODS OF MOTION VECTOR RESTORATION FOR DECODER-SIDE PREDICTOR REFINEMENT”, filed under 35 U.S. SC §119(e), and U.S. Provisional Application No. 62 / 479,350, filed March 31, 2017, both of which are incorporated herein by reference in their entirety. Technical Field

[0003] The techniques described herein generally relate to video encoding and decoding, and in particular to decoder-side motion vector recovery. Background Technology

[0004] Video encoding and decoding involves the compression (and decompression) of digital video signals. Examples of video encoding and decoding standards include the H.264 video compression standard and its successor, High Efficiency Video Coding (HEVC). Moving video is created by taking snapshots of the signal at periodic time intervals, and playing back these snapshots or frames produces the illusion of motion. Video encoders include a prediction module that attempts to reduce redundancy using the similarity between adjacent video frames. Predicted frames are created from one or more past or future frames, often referred to as reference frames. Frames not used as reference frames are typically called non-reference frames.

[0005] Because each frame can contain tens of thousands of pixels, video encoding and decoding technologies typically don't process all the pixels of a frame at once. Therefore, the encoded frame is broken down into multiple blocks, often called macroblocks. Instead of directly encoding the raw pixel values ​​of each block, the encoder tries to find a block similar to one encoded in a reference frame. If the encoder finds a similar block, it can encode that block using motion vectors, which are two-dimensional vectors pointing to the matching block in the reference frame.

[0006] Some techniques explicitly send motion vectors to the decoder; examples of these modes include the merge mode in High Efficiency Video Coding (HEVC) and the Advanced Motion Vector Prediction (AMVP) mode. However, having to send motion vectors can consume a significant amount of data that could otherwise be used by the transimitter to encode other information. Therefore, decoder-side motion vector refinement tools can be used to refine, predict, and / or generate motion information so that it can be derived without explicit sending. Summary of the Invention

[0007] According to the subject matter disclosed in this invention, apparatus, systems, and methods for decoder-side motion vector recovery techniques are provided, which improve the execution speed and efficiency of said decoder-side motion vector recovery techniques.

[0008] Some embodiments relate to decoding methods for decoding video data, the methods including receiving compressed video data associated with a set of frames, and using decoder-side predictive sub-refinement techniques to compute a new motion vector for a current frame from the set of frames, wherein the new motion vector estimates the motion of the current frame based on one or more reference frames. The computation includes retrieving a first motion vector associated with the current frame, performing a first portion of a decoding process using the first motion vector, retrieving a second motion vector associated with the current frame, the second motion vector being different from the first motion vector, and performing a second portion of the decoding process using the second motion vector.

[0009] In some examples, the first motion vector includes an unrefined motion vector, the second motion vector includes a refined motion vector, the refined MV is refined using decoder-side predictor refinement techniques, the first part of the decoding process includes a parsing part, a motion vector derivation part, or both, and the second part of the decoding process includes a reconstruction part.

[0010] In some examples, the decoding method includes retrieving a third motion vector associated with a second frame, wherein the third motion vector is a refined motion vector; performing a first portion of the decoding process using the first motion vector and the third motion vector; and performing a second portion of the decoding process using the second motion vector and the third motion vector.

[0011] In some examples, the first part of performing the decoding process includes performing a motion vector derivation part using the first motion vector and the third motion vector, wherein the motion vector derivation part includes motion vector prediction derivation, merging candidate derivation, or both.

[0012] In some examples, the first part of performing the decoding process includes referencing the first motion vector as the motion vector for decoding the current frame.

[0013] In some examples, the decoding method includes using the second motion vector and the third motion vector to perform motion compensation, overlapping block motion compensation, deblocking, or any combination thereof.

[0014] In some examples, the decoding method includes determining that coding tree unit constraints are not applied to the compressed video data, and retrieving the first motion vector associated with the current frame, including retrieving the unrefined motion vector of the current frame and the refined motion vector associated with the second frame.

[0015] In some examples, retrieving the first motion vector in relation to the current frame includes retrieving the unrefined motion vector of the current coding tree unit row, the refined motion vector of the upper coding tree unit row, other blocks or other slices, and the refined motion vector in relation to the second frame.

[0016] Some embodiments relate to decoding methods for decoding video data. The decoding method includes receiving compressed video data associated with a set of frames, and calculating a new motion vector for a current frame from the set of frames using a decoder-side predictive sub-refinement tool, wherein the new motion vector estimates the motion of the current frame based on one or more reference frames. The calculation includes receiving a signal indicating a starting candidate index of a starting motion vector candidate list, determining a first motion vector candidate in the starting motion vector candidate list and including a second motion vector candidate whose difference from the first motion vector candidate is less than a predetermined threshold, removing the second motion vector candidate from the starting motion vector candidate list, not adding the second motion vector candidate to the starting motion vector candidate list, or both, and calculating the new motion vector based on the starting motion vector candidate list and the starting candidate index.

[0017] In some examples, the decoding method includes analyzing new motion vector candidates, which include motion vector pairs, determining, based on the analysis, whether the motion vector pairs are along the same motion trajectory, and adding the motion vector pairs to the initial motion vector candidate list.

[0018] In some examples, the decoding method includes analyzing new motion vector candidates, which include motion vector pairs; based on the analysis, determining that the motion vector pairs do not follow the same motion trajectory; splitting the motion vector pairs into two new candidate motion vector pairs; and adding the two new candidate motion vector pairs to the initial motion vector candidate list.

[0019] In some examples, the splitting includes adding a first motion vector of the motion vector pair to the first of the two new candidate motion vector pairs, filling the first of the two new candidate motion vector pairs with a motion vector that is a mirror image of the first motion vector, adding a second motion vector of the motion vector pair to the second of the two new candidate motion vector pairs, and filling the second of the two new candidate motion vector pairs with a motion vector that is a mirror image of the second motion vector.

[0020] Some embodiments relate to encoding methods for encoding video data. The method includes computing compressed video data associated with a set of frames, including computing a new motion vector for a current frame from the set of frames, wherein the new motion vector estimates the motion of the current frame based on one or more reference frames, including computing a first motion vector associated with the current frame, performing a first portion of an encoding process using the first motion vector, computing a second motion vector associated with the current frame, the second motion vector being different from the first motion vector, and performing a second portion of the encoding process using the second motion vector.

[0021] In some examples, computing the first motion vector includes computing an unrefined motion vector, an unrefined set of motion vectors, or both, and performing the first part of the encoding process includes performing a syntax encoding part, a motion vector derivation part, a motion vector prediction derivation part, or some combination thereof.

[0022] In some examples, performing the motion vector prediction derivation includes generating a merged candidate list, generating an advanced motion vector prediction candidate list, or both.

[0023] In some examples, the encoding method includes performing motion vector encoding, motion vector prediction generation, or both using the unrefined motion vectors, the set of unrefined motion vectors, or both, without using a decoder-side motion vector refining tool to refine the unrefined motion vectors, the set of unrefined motion vectors, or both.

[0024] In some examples, calculating the second motion vector includes calculating a refined motion vector, wherein the refined motion vector is calculated using encoder-side refinement techniques, the refined motion vector is stored in a set of motion vector buffers, and the second part of performing the encoding process includes performing a motion compensation part, an overlapping block motion compensation part, a deblocking part, or some combination thereof.

[0025] Some embodiments relate to apparatus for decoding video data. The apparatus includes a processor communicating with a memory, the processor executing instructions stored in the memory that cause the processor to receive compressed video data associated with a set of frames, and to compute a new motion vector for a current frame from the set of frames using decoder-side predictive sub-refinement techniques, wherein the new motion vector estimates the motion of the current frame based on one or more reference frames. The computation includes retrieving a first motion vector associated with the current frame, performing a first portion of a decoding process using the first motion vector, retrieving a second motion vector associated with the current frame, the second motion vector being different from the first motion vector, and performing a second portion of the decoding process using the second motion vector.

[0026] In some examples, the first motion vector includes an unrefined motion vector, and the second motion vector includes a refined motion vector, wherein the refined MV is refined using a decoder-side predictor refinement technique, the first part of the decoding process includes a parsing part, a motion vector derivation part, or both, and the second part of the decoding process includes a reconstruction part.

[0027] In some examples, the processor is configured to execute instructions stored in the memory that cause the processor to retrieve a third motion vector in relation to a second frame, wherein the third motion vector is a refined motion vector, to perform a first portion of the decoding process using the first motion vector and the third motion vector, and to perform a second portion of the decoding process using the second motion vector and the third motion vector.

[0028] Some embodiments relate to apparatus for decoding video data. The apparatus includes a processor communicating with a memory, the processor executing instructions stored in the memory that cause the processor to receive compressed video data associated with a set of frames, and to compute a new motion vector for a current frame from the set of frames using decoder-side predictive sub-refinement techniques, wherein the new motion vector estimates the motion of the current frame based on one or more reference frames. The computation includes receiving a signal indicating a starting candidate index for an initial motion vector candidate list, determining a first motion vector candidate in the initial motion vector candidate list and including a second motion vector candidate whose difference from the first motion vector candidate is less than a predetermined threshold, removing the second motion vector candidate from the initial motion vector candidate list, not adding the second motion vector candidate to the initial motion vector candidate list, or both, and computing the new motion vector based on the initial motion vector candidate list and the starting candidate index.

[0029] In some examples, the processor is configured to execute instructions stored in the memory that cause the processor to analyze new motion vector candidates, which include pairs of motion vectors, determine, based on the analysis, whether the pairs of motion vectors are along the same motion trajectory, and add the pairs of motion vectors to the initial list of motion vector candidates.

[0030] In some examples, the processor is configured to execute instructions stored in the memory that cause the processor to analyze new motion vector candidates, the motion vector candidates including motion vector pairs, and based on the analysis, determine that the motion vector pairs do not follow the same motion trajectory, split the motion vector pairs into two new candidate motion vector pairs, and add the two new candidate motion vector pairs to the initial motion vector candidate list.

[0031] Some embodiments relate to apparatus for encoding video data. The apparatus includes a processor communicating with a memory, the processor executing instructions stored in the memory that cause the processor to compute compressed video data in relation to a set of frames, including computing a new motion vector for a current frame from the set of frames, wherein the new motion vector estimates the motion of the current frame based on one or more reference frames, including computing a first motion vector in relation to the current frame, performing a first portion of the encoding process using the first motion vector, computing a second motion vector in relation to the current frame, the second motion vector being different from the first motion vector, and performing a second portion of the encoding process using the second motion vector.

[0032] In some examples, computing the first motion vector includes computing an unrefined motion vector, an unrefined set of motion vectors, or both, and performing the first part of the encoding process includes performing a syntax encoding part, a motion vector derivation part, a motion vector prediction derivation part, or some combination thereof.

[0033] In some examples, calculating the second motion vector includes calculating a refined motion vector, wherein the refined motion vector is refined using encoder-side refinement techniques, storing the refined motion vector in a motion vector buffer set, and performing the second part of the encoding, including performing a motion compensation part, an overlapping block motion compensation part, a deblocking part, or some combination thereof.

[0034] Therefore, the invention has been summarized relatively broadly, and the features of the disclosed subject matter are conducive to a better understanding of the detailed description below, thereby enabling a better understanding of the contribution to the prior art. Of course, there are additional features of the disclosed subject matter, which will be described below and constitute the subject matter of the appended claims. It should be understood that the wording and terminology used herein are for descriptive purposes only and should not be considered limiting. Attached Figure Description

[0035] In the diagrams, every identical or nearly identical element shown in the various diagrams is represented by the same reference symbol. For the sake of brevity, not every element is shown in every diagram, and these diagrams are not necessarily drawn to scale, but rather focus on illustrating various aspects of the technology and apparatus described herein.

[0036] Figure 1 An exemplary video codec configuration is shown.

[0037] Figure 2 An example technique for deriving motion vector prediction (MVP) for time extrapolation is shown.

[0038] Figure 3 An example technique for pattern-based motion vector derivation using bi-lateral matching merging patterns is shown.

[0039] Figure 4 An example of scaling motion trajectories is shown.

[0040] Figure 5 An example technique for pattern-based motion vector derivation using template matching to merge patterns is shown.

[0041] Figure 6 An exemplary decoder architecture is shown.

[0042] Figure 7 It shows when executing such Figure 6 An example of decoder pipeline execution when the decoder architecture is shown.

[0043] Figure 8 An example is shown when the decoder pipeline executes the decoder-side predictor refinement tool.

[0044] Figure 9 An example of a decoder-side MV refinement (DMVR) process using two reference images is shown.

[0045] Figure 10 An exemplary two-stage search process is shown, which uses bidirectional prediction blocks to search for new (e.g., better) matching blocks.

[0046] Figure 11 The overlapping block motion compensation (OBMC) performed at the sub-block level for the boundary of the motion compensation (MC) block is shown.

[0047] Figure 12A An exemplary high-level overview of the OBMC method is shown.

[0048] Figure 12B An exemplary high-level overview of the OBMC method is shown when using the initial MV.

[0049] Figure 13 A high-level description of the MV set used for the current CTU, the left column, and the top row is shown.

[0050] Figure 14 Examples of multiple candidate MV pairs, both on the same motion trajectory and not on the same motion trajectory, are shown according to some embodiments.

[0051] Figure 15 An exemplary decoding method for decoding video data using two MVs is shown according to some embodiments.

[0052] Figure 16A An exemplary method for pruning a list of motion vector candidates is shown, according to some embodiments.

[0053] Figure 16B An exemplary method for generating a candidate list of motion vectors, according to some embodiments, is shown. Detailed Implementation

[0054] The inventors have recognized and are aware that a variety of techniques can be used to improve the execution of decoder-side predictor refinement techniques, such as pattern-based motion vector derivation (PMVD), bidirectional optical flow (BIO), and decoder-side motion vector refinement (DMVR). Decoder-side predictor refinement tools can suffer from processing latency due to how motion vectors are calculated and reconstructed. These techniques can be used to allow similar runtime settings compared to the execution of conventional decoding methods that do not predict MVs (e.g., when motion vector information is transmitted from the encoder). For example, the decoding process can be adjusted so that MVs can be reconstructed earlier in the process, allowing the decoder to pre-extract the required reference pixels in a way that hides the waiting time period required to extract data. As an example of this technique, unrefined MVs can be recovered back to the MV buffer and / or remain unchanged, so that the unrefined MVs can be used by the decoder-side MV refinement tool or used to derive reference MVs or MV candidates for subsequent blocks (e.g., merging candidate lists and advanced motion vector predictor lists).

[0055] However, using such techniques (e.g., recovering unrefined MVs) can lead to blocking artifacts and / or other encoding / decoding inefficiencies. For example, in addition to using (recovered) unrefined MVs for resolution, the decoder can also use unrefined MVs for deblocking, overlapped block motion compensation (OBMC), and / or temporally in MV derivation. The techniques described herein allow the decoder to use different MVs (e.g., different from unrefined MVs) for processes performed after the resolution phase, such as deblocking, OBMV, and / or temporally in MV derivation. For example, the first MV used for resolution (e.g., MV / MVP derivation) can be an unrefined MV, and the second MV used for other processes can be a refined MV, including deblocking, OBMC, and / or temporally in MV derivation.

[0056] In some embodiments, the decoder uses two sets of motion vectors: one set of MVs is used for a first part of the decoding process (e.g., for parsing, including MV derivation and pixel pre-extraction), and a second set of MVs is used for a second part of the decoding process (e.g., for reconstruction, including motion compensation, OBMC, and / or deblocking). In some embodiments, CTU row data is introduced to allow for additional processing with refined MVs (e.g., using refined MVs from the upper CTU row). For example, the first set of MVs includes unrefined motion vectors from the current coding tree unit row, refined motion vectors from the upper coding tree unit row, and refined motion vectors related to the second frame. The second set of MVs may include refined MVs from the current image and refined MVs from other images.

[0057] These or other techniques allow pre-parsing processing to use refined MVs to avoid additional block artifacts. This technique can provide higher encoding / decoding gain compared to using unrefined MVs for processing performed after the parsing phase. These or other techniques are further described in this paper.

[0058] In the following description, numerous specific details are set forth regarding the systems and methods of the present invention, and the environments in which such systems and methods may operate, in order to provide a full understanding of the subject matter of the invention. However, the disclosed subject matter may be practiced without these specific details, and certain features well-known in the art will not be repeated to avoid complicating the disclosed subject matter, as will be apparent to those skilled in the art. Furthermore, it is understood that the examples provided below are exemplary, and other systems and methods within the scope of the disclosed subject matter are contemplated.

[0059] Figure 1 An exemplary video encoding / decoding configuration 100 according to some embodiments is illustrated. Video source 102 is a video source, such as a digital television, internet-based video, video call, and / or the like. Encoder 104 encodes the video source into encoded video. Encoder 104 may be present in the same device (e.g., a mobile phone for video calls) that generates video source 102, and / or in different devices. Receiving device 106 receives the encoded video from encoder 104. Receiving device 106 may receive the video as a video product (e.g., a digital video disc or other computer-readable medium) via a broadcast network, mobile network (e.g., a cellular network), and / or the internet. For example, receiving device 106 may be a computer, mobile phone, or television. Receiving device 106 includes a decoder 108 for decoding the encoded video and a display 110 for displaying the decoded video.

[0060] As described above, part of the decoding process relies on motion vectors. In examples where the encoder (e.g., encoder 104) does not include the final MV information directly in the encoded video, the decoder (e.g., decoder 108 in receiver 106) can employ a receiver-side prediction tool, referred to as a receiver-side predictor sub-refinement tool or a decoder-side predictor sub-refinement tool. An example of a receiver-side predictor sub-refinement tool is the mode-based motion vector derivation (PMVD) mode, which can also be called a frame rate up-conversion (FRUC) mode. For example, PMVD is described in Joint Video Exploration Team (JVET) document JVET-F1001, entitled “Algorithm Description of Joint Exploration Test Model 6 (JEM 6)”, which is incorporated herein by reference in its entirety.

[0061] Other examples of decoder-side predictor refinement tools include Bidirectional Optical Flow (BIO) and Decoder-Side Motion Vector Refinement (DMVR). For example, BIO was proposed by Samsung at the 3rd JCTVC meeting and the 52nd VCEG meeting, and is published in JCTVC-C204 and VECG-AZ05. See, for example, Elena Alshina and Alexander Alshin, “Bi-Directional Optical Flow” (JCTVC-C204) (with two additional Microsoft Excel spreadsheets), October 7-15, 2010, and E. Alshina et al., “Known Tools Performance Investigation for Next Generation Video Coding” (VCEG-AZ05) (with a Microsoft PowerPoint presentation), June 19-26, 2015, both of which are incorporated herein by reference in their entirety. BIO utilizes optical flow and the assumption of stable motion to achieve sample-level motion refinement. It is typically applied only to true bidirectional predictive blocks, which are predicted from two reference frames, one of which is the previous frame and the other is the subsequent frame. In VECG-AZ05, BIO uses a 5×5 window to derive motion refinement for a single sample. Therefore, for an N×N block, a motion compensation result of (N+4)×(N+4) blocks and the corresponding gradient information are needed to derive the sample-based motion refinement for the current block. A 6-tap gradient filter and a 6-tap interpolation filter are used to generate the gradient information in BIO. Therefore, the computational complexity of BIO is much higher than that of traditional bidirectional prediction. For additional information, see D. Marpe, H. Schwarz, and T. Wiegand, “Context-Based Adaptive Binary Arithmetic Coding in the H.264 / AVC Video Compression Standard, IEEE Transactions on Circuits and Systems for Video Technology,” Vol. 13, No. 7, pp. 620-636, July 2003, which is incorporated herein by reference in its entirety.

[0062] PMVD itself can be implemented using different modes, such as bidirectional match merge mode or template match merge mode. Typically, the mode used by the decoder is signaled in the encoded video. Therefore, the encoder signals to the decoder to use the PMVD mode, and also signals which specific PMVD mode. In some examples, FRUC_mrg_flag is signaled when merge_flag or skip_flag is true. If FRUC_mrg_flag is 1, then FRUC_merge_mode is signaled to indicate whether to select bidirectional match merge mode (e.g., combined with...). Figure 2-4 (further description) or template matching merging pattern (e.g., combining) Figure 5 (Further description).

[0063] In summary, both PMVD modes use decoded pixels to derive motion vectors for the current block. Deriving a new temporal motion vector prediction (MVP) by scanning all MVs in all reference frames is called temporally derived MVP. An image typically refers to several frames (e.g., an image consists of 16 frames), and these reference frames are placed in one or two reference image lists. For P-slices, only one reference image list is used; for B-slices, two reference lists are used. Typically, for B-slices, the two reference image lists are used to store past and future images; the one for past images is usually called LST_0, and the one for future images is called LIST_1.

[0064] To derive the MVP for the LIST_0 temporal derivation, for each LIST_0 MV in the LIST_0 reference frame, the MV is scaled to point to the current frame. The block pointed to by the scaled MV in the current frame is the target current block. The MV is further scaled to point to a reference image in List_0 where refidx equals 0, which is used for the target current block. The further scaled MV is stored in the LIST_0 MV field used for the target current block. Figure 2Example 200 of deriving the temporally deduced MVP is shown. The decoder scans all LIST_0 MVs in the LIST_0 reference image where refIdx equals 1. For each LIST_0 MV (shown by arrows 202, 204), a scaled MV pointing to the current image is derived (shown by dashed arrows 206 and 208 of reference image 201). Each scaled MV points to 4×4 blocks 210, 212 in the current image 205. Then, another scaled MV 214, 216 is assigned to 4×4 blocks 210, 212 in the current image, respectively, where the scaled MVs 214, 216 follow the associated scaled MVs 202, 204, but with a starting point in the current image 205 and an ending point in the reference image 218 where refIdx equals 0 in LIST_0. The decoder scans all MVs in all 4×4 blocks in all reference images to generate the temporally deduced LIST_0 and LIST_1 MVPs for the current frame. For each MV, the MV is scaled to obtain the crossed blocks in the current image. The decoder then computes the scaled MVP and assigns it to the crossed blocks (as indicated by the dashed arrows 206 and 208).

[0065] Figure 3 An example of the PMVD bidirectional matching merging pattern is shown. For bidirectional matching, the decoder finds the two most similar reference blocks in LIST_0 and LIST_1 that are on the same trajectory. Figure 3 As shown, for the current image (or "cur pic") 300, the decoder selects a macroblock (or "block") 302 from reference frame Ref0 304 in LIST_0 and a second block 306 from reference frame Ref1 308 in LIST_1. The decoder essentially assumes the motion is constant and uses the midpoint between the two macroblocks to generate a motion trajectory 310. The decoder uses the motion trajectory 310 to find the currently predicted macroblock (or "block") 312 in the current image 300. The decoder calculates the difference between block 302 and block 306. If the difference is only small, the decoder knows the blocks are very similar. In some examples, the decoder may calculate the sum of absolute distances (or "SAD") to calculate the difference between the two blocks. The decoder modifies the motion trajectory to minimize the difference between the blocks.

[0066] The decoder constructs initial motion vector (MV) lists from LIST_0 and LIST_1, respectively. The decoder uses 11 candidates from each list, including 7 merged MV candidates and 4 temporally derived MV predictions (or MVPs). The decoder evaluates these 11 candidates to select the optimal starting point. Specifically, the decoder searches for MV pairs that traverse two adjacent frames. When considering candidates from each list, the decoder analyzes 22 motion vectors to derive 22 motion vector pairs, generated by scanning motion trajectories. For each MV in a list, an MV pair is generated by combining this MV with its mirrored MV, which is derived by scaling the MV to another list. For each MV pair, the two reference blocks are compensated using this MV pair. Figure 4 Example 400 of scaling motion trajectories is shown. Specifically, motion trajectory 402 from the current image to ref1 in LIST_0 is scaled to motion trajectory 404 from the current image to ref0 in LIST_1. The decoder calculates the cost of each 22 motion vector pairs (e.g., using SAD) and selects the MV pair with the minimum cost as the starting point for the bidirectional matching merging pattern.

[0067] The decoder then refines the selected MV pair, searching different blocks around the starting point to determine which block is the best match. In some examples, the current PU is split into multiple sub-PUs. The depth of the sub-PU is signaled in the sequence parameter set (SPS) (e.g., 3). In some examples, the minimum sub-PU size is 4×4. For each sub-PU, multiple starting MVs from LIST_0 and LIST_1 are selected, including multiple MVs of the PU-level derivation, the 0 MV, the TMVP of the current sub-PU and the HEVC co-occurrence of the lower right block, the MVP of the current sub-PU's time derivation, and the MVs of the PUs / sub-PUs to the left and above. By using a similar mechanism in the PU-level search, the best MV pair for the sub-PU is selected. In some examples, the decoder uses the Diamond Search algorithm to search different blocks and then uses the final MV pair as the best MV pair at both the PU-level and sub-PU-level.

[0068] In summary, in some examples, the bidirectional matching merge pattern first uses a list of MVs, computes candidate MV pairs to obtain the initial MV pairs, and then refines the MV pairs to determine the final best MV pair.

[0069] For template matching merging mode, assuming the decoder decodes the current block, the decoder can use the template to find the best match using neighboring blocks. Therefore, the decoder can use neighboring blocks to find the best match and then use the best-matched motion vector. Figure 5 An example technique for template matching merging patterns is shown. (Reference) Figure 5 Template 502 includes pixels reconstructed from the top four rows and the left four columns of the current block 504 to perform matching for the current image 508 in Ref 0 506. Therefore, unlike the bidirectional matching merge mode, the template matching merge mode does not use two reference frames; it uses only one reference frame.

[0070] Similar to the bidirectional matching merging pattern, two-stage matching is also applied to template matching. In PU-level matching, 11 starting MVs are selected from LIST_0 and LIST_1 respectively. These MV packets come from 7 MVs in the merging candidate and 4 MVs from the time-derived MVP. Two distinct sets of starting MVs are generated for the two lists. For each MV in one list, the SAD cost of the template with the MV is calculated. The MV with the minimum cost is the optimal MV. Then, a diamond search is performed to refine the MVs with a refinement precision of 1 / 8 pixel, and the refinement search range is limited to ±8 pixels. The final MVs are the PU-level derived MVs, generated independently of the MVs in LIST_0 and LIST_1.

[0071] For the second stage, the sub-PU level search, the current PU is divided into multiple sub-PUs. The depth of the sub-PUs is signaled in the SPS (e.g., 3), and the minimum sub-PU size is a 4×4 block. For each sub-PU on the left or top PU boundary, multiple starting MVs from LIST_0 and LIST_1 are selected, including multiple MVs of the PU-level derived MV, 0 MV, HEVC co-occurrence TMVP of the current sub-PU and the lower right block, the time-derived MVP of the current sub-PU, and the MVs of the left and top PUs / sub-PUs. Using a similar mechanism as in the PU-level search, the best MV pair for the sub-PU is selected, and a diamond search is performed to refine the MV pair. Motion compensation is performed for this sub-PU to generate predictors for this sub-PU. For PUs not on the left or top PU boundary, the second stage, i.e., the sub-PU level search, is not applied, and the corresponding MV is set to be equal to the MV in the first stage.

[0072] When bidirectionally predicting MV pairs (e.g., in merge mode, when selecting bidirectional predictive merge candidates), a decoder-side MV refinement (DMVR) process can be performed to refine multiple LIST_0 and LIST_1 MVs for better encoding and decoding efficiency. HiSilicon presents an example of a DMVR process in JVET-D0029, titled "Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching," which is incorporated herein by reference. Figure 9 A DMVR process 900 using reference images 0 902 and 1 904 is illustrated according to some embodiments. In the DMVR process 900, bidirectional prediction blocks (bidirectional prediction templates) are generated using bidirectional predictions of reference block 906 from MV0 908 and reference block 910 from MV1 912. The bidirectional prediction blocks are used as the new current block Cur' (at the position of the original current block 914) to perform motion estimation to search for better matching blocks in reference images 0 902 and 1 904, respectively. The refined MVs (MV0' and MV1', not in...) Figure 9 (As shown in the figure) is used to generate the prediction block for the final bidirectional prediction of the current block.

[0073] In some embodiments, DMVR uses a two-stage search to refine multiple MVs of the current block to generate MV0' and MV1'. Figure 10 An exemplary two-stage search process 1000, according to some embodiments, is shown, which uses bidirectional predictive blocks to search for new (e.g., better) matching blocks. Figure 10 As shown, for the current block in reference image 0, the cost of the current MV candidate is first calculated in square block 1002 (also known as L0_pred). For example, the cost of block 1002 can be calculated as the sum of absolute differences (SAD) of (Cur'-L0_pred) to calculate the initial cost. In the first phase of the search, an integer pixel square search is performed around block 1002. As shown in this example, 8 candidates are evaluated ( Figure 10The eight large circles 1004A-1004H are collectively referred to as 1004. The distance between two adjacent circles (e.g., 1004A and 1004B), and the distance between a square block 1002 and its neighboring circle (e.g., 1004B), is 1 pixel. An 8-tap filter can be used to generate eight candidate blocks for each block 1004, and the cost of each candidate can be evaluated using SAD. The candidate with the best cost (e.g., the lowest cost if SAD is used) among the eight candidate blocks 1004 is selected as the best MV candidate in the first stage, as shown in 1004H in this example. In the second stage, a 1 / 2-pixel square search is performed around the best MV candidate from the first stage (1004H in this example), as shown in the eight small circles 1006A-1006H (collectively referred to as 1 / 2 pixel 1006). An 8-tap filter can also be used to generate candidate blocks for each 1 / 2 pixel 1006, and the cost can be determined using SAD. The MV candidate with the best cost (e.g., lowest cost) is selected as the final MV, which is used for final motion compensation. The process is repeated for reference block 1 904 to determine the final MV for reference image 1, and the final bidirectional prediction block is regenerated using the refined MV.

[0074] Figure 6An exemplary decoder architecture 600 according to some embodiments is shown. The entropy decoder includes, for example, a CABAC or CAVLC entropy decoder, which parses the syntax from the bitstream. ColMV DMA 610 stores the time MV of the same position, MV scheduling (diapatch) 612 reconstructs the MV of the block, and sends memory fetch instructions to MC cache 614 and DRAM (not shown) via memory interface arbiter 616. Inverse conversion 618 includes inverse quantization and inverse conversion (IQIT), which generates the reconstructed residual 620. Prediction 622 generates inter-frame motion compensation and intra-frame prediction predictors, deblocking 624 reduces block artifacts, and Rec DMA 626 stores the reconstructed pixels to external DRAM. Further details of exemplary components of this architecture are disclosed in T. Huang et al., “A 249MPixel / s HEVC video-decoder chip for Quad Full HD applications,” Technical Abstract of the IEEE International Solid-State Circuits Conference (ISSCC), pp. 162-163, February 2013, which is incorporated herein by reference in its entirety. Notably, for pipeline purposes, the architecture is decomposed into four stages: EC stage 602, IQIT (inverse quantization and inverse conversion) / extraction stage 604, reconstruction stage 606, and loop filtering stage 608. In HEVC and H.264, the final MV can be derived in both EC stage 602 (which includes resolution) and reconstruction stage 606. In some embodiments, the decoder derives the final MV in the resolution stage and pre-extracts the required reference pixels in the resolution stage (EC stage 602). This can, for example, reduce / hide DRAM access time.

[0075] Figure 7 It is shown that, according to some embodiments, when performing such Figure 6 The example shown illustrates a decoder pipeline execution of 700 when the decoder architecture is shown. Figure 7The process includes a resolution phase 702, during which motion vectors are reconstructed as described above. An IQ / IT phase 704-1 generates residuals for the reconstruction of the current block. A reference pixel extraction phase 704-2 extracts reference pixel data from memory; the reference frame is typically stored in external memory, such as DRAM. Therefore, if the decoder wants to perform motion compensation on the reference frame, it first needs to retrieve the reference data from external memory, which typically requires a significant wait time. An intra-frame / MC (motion compensation) reconstruction phase 706 performs prediction, and a deblocking (DB) / sample adaptive offset (SAO) phase 708 performs loop filtering to improve the quality of the decoded frame.

[0076] Typically, the decoder first decodes CU0, then CU1, and so on. To give an example of using CU0, at t0, the decoder decodes CU0 in the parsing phase 702, including reconstructing the MV. Then, at t1, CU0 is moved to the IQ / IT phase 704-1, where the decoder needs to perform pre-extraction in the previous phase (refer to pixel extraction phase 704-2) in order to perform motion compensation in the intra-frame / MC reconstruction phase 706.

[0077] like Figure 7 As seen in the diagram, to hide the wait time for fetching data from memory (e.g., so as not to affect pipeline execution), because the decoder needs to know the motion vectors before the reconstruction performed in the intra-frame / MC reconstruction stage 706, the data is pre-fetched and stored in local memory (e.g., SRAM or cache memory) in the reference pixel extraction stage 704-2. For example, in MPEG-2 / 4, H.264 / AVC, and HEVC video encoders, multiple motion vectors (MVs) can be reconstructed in the parsing stage. Based on the multiple reconstructed MVs, the required reference pixels can be extracted from DRAM and stored in local memory, such as SRAM or cache memory. In the intra-frame / MC reconstruction stage 706, reference data can be loaded from the local memory without waiting cycles.

[0078] However, the decoder-side predictive sub-refinement tool uses neighboring blocks to derive motion vectors (e.g., how PMVD, such as template matching merging mode, uses neighboring blocks to derive motion vectors). However, no template block is generated before the third stage (intra-frame / MC reconstruction stage 706). For example, when PMVD is applied, the final MV of the PMVD-encoded block depends on the PMVD search process in intra-frame / MC reconstruction stage 706, meaning that the MV cannot be reconstructed in parsing stage 702, and therefore data pre-extraction in reference pixel extraction stages 704-2 is not feasible.

[0079] Figure 8 This illustrates an example of decoder pipeline execution when the decoder-side predictive sub-refinement tool is executed. For instance, using PMVD as an example, at time t2, because the MV for CU0 depends on the PMVD search process in the intra-frame / MC reconstruction phase 706 (which also executes at t2), the MV cannot be reconstructed in the resolution phase for CU1 (at time t1), therefore the data for CU1 cannot be pre-extracted in the reference pixel extraction phase 704-2 of t2. This problem similarly affects the processing of each CU, resulting in only one CU completing processing every two time slots. For example, Figure 8 This shows that for t4 and t5, the decoder only processes CU1, compared to Figure 7 CU1 completes processing at t4 and CU2 completes processing at t5.

[0080] When decoder-side prediction refinement techniques are used for decoding, the problem of data prefetching can be solved. For example, the technique allows data prefetching to still be performed with hidden wait periods, such as... Figure 7 As shown, instead of causing as Figure 8 The delay is shown. For ease of illustration, the technique discussed below refers to PMVD as an example, although those skilled in the art will understand that the technique can be applied to other decoder-side prediction refinement techniques (e.g., BIO and DMVR).

[0081] According to some embodiments, the original candidate MV is stored in the MV buffer for the next decoding process. In some embodiments, the selected merge candidate MV (e.g., the initial or unrefined MV) is stored back in the MV buffer so that the decoder can reference adjacent blocks as well as co-located blocks / images. Thus, according to some examples, the MC of a PMVD block (e.g., performed in the intra-frame / MC reconstruction phase) uses the MV derived from PMVD, but the selected merge candidate MV is stored back in the MV buffer for future reference. For example, this can allow the MV to be reconstructed in resolution phase 702, and reference pixels can be pre-extracted in phases 704-2. If the current block is a PMVD-encoded block, a larger reference block (e.g., including a refined search range) can be pre-extracted. Thus, in some examples, the MV for the current block is not refined, but the decoder uses a refined MV for compensation.

[0082] In some examples, the decoder can be configured not to change the MV in the MV buffer. For example, the decoder can store the starting point (e.g., the starting MV) into the MV buffer and refine it to generate a refined MV. The refined MV is only used to generate motion compensation data and does not need to change the MV in the MV buffer. The MV buffer used for future reference (e.g., merging candidate lists and generating AMVP candidate lists) remains unchanged.

[0083] In some examples, the decoder can use separate buffers for refinement. For instance, the decoder can retrieve the initial MV, run PMVD, and perform refinement without storing the refined MV in the original MV buffer – the decoder stores the refined MV in a time buffer.

[0084] In some examples, the decoder can signal starting candidates for PMVD. For example, the decoder can signal a starting candidate index, which is used to select the starting MV from a list of MV candidates. This allows the decoder to know, for example, which candidate outside the initial 11 candidates will be used as the starting candidate for PMVD. The decoder can initially generate 11 starting candidates, and the encoder can signal to the decoder which candidate is best. Because the decoder knows the starting candidates, this signaling allows the decoder to skip template matching and proceed to refinement (e.g., the decoder can use template matching and a diamond search technique to refine the MV around the starting candidates). Although the MV will be refined by diamond search, in the proposed method, only the starting candidates will be stored, not the refined motion vectors.

[0085] In some examples, for PMVD (e.g., including bidirectional matching merge mode and template matching merge mode), the MVs of LIST_0 and LIST_1 in the merge candidates are used as starting MVs. In some examples, the best MV candidate can be implicitly derived by searching all these MVs, a method that would require significant memory bandwidth. In this example, a merge index is sent for either the bidirectional matching merge mode or the template matching merge mode. The sent merge index can indicate, for example, the best starting MV in LIST_0 and LIST_1 in the template matching merge mode, and the best two MV pairs (one derived from LIST_0 and the other from LIST_1) in the bidirectional matching merge mode. By sending the merge index, the template matching step can be limited to, for example, a refined search around the sent merge candidates. For bidirectional matching, the decoder can perform cost estimation to select the best MV pair from the two MV pairs and perform a refined search. For bidirectional matching, if the merge candidate is a unidirectional MV, its corresponding MV in another list can be generated using a mirrored (scaled) MV. In some embodiments, by using a predetermined MV generation method, the best starting MV and / or MV pair in LIST_0 and LIST_1 are known, and the best starting MV or best MV pair of LIST_0 and / or LIST_1 MV is explicitly signaled to reduce bandwidth requirements.

[0086] In some examples, when sending a merge index, the decoder can further utilize the selected MV to exclude or select some candidates in the first stage (PU level matching). For example, the decoder can exclude some MVs in the candidate list that are far from the selected MV. As another example, the decoder can select N MVs in the candidate list that are closest to the selected MV but are in different reference frames.

[0087] As described above, some techniques send out initial MVs by generating an initial MV candidate list and a sending candidate index (e.g., removing initial sending candidates, for example, PMVD as described above). Using PMVD as an example, because PMVD performs MV refinement, two similar initial MV candidates may have the same refined final MV. Therefore, because PMVD searches for local minima around the initial candidates, they may have the same refined final MV. Similar MVs in the candidate list can be removed from the candidate list, or the candidate list can be pruned.

[0088] The techniques described here can be used to trim and / or create a candidate list of motion vectors. Figure 16A An exemplary method 1600 for trimming a list of motion vector candidates is shown according to some embodiments. Figure 16BAn exemplary method 1650 for creating a list of motion vector candidates, according to some embodiments, is illustrated. For example, the list may initially be empty, and whenever a new candidate is added, the technique can determine whether the new candidate is redundant or associated with an existing motion vector candidate in the list. If the candidate is redundant, then no new candidate will be added.

[0089] See Figure 16A In step 1602, the decoder stores the initial motion vector candidate list. For example, a conventional merge candidate list generation process (e.g., as described above) can be used to generate the PMVD merge candidate list. See steps... Figure 16A In steps 1604-1610 of the method, for MV derivation, the newly added MV can be compared with multiple MVs already in the candidate list. If one (or more) of the multiple MVs is similar to the newly added MV, the newly added MV is removed from the list. Specifically, in step 1604, the decoder compares the newly added MV with existing candidates in the candidate MV list to determine the candidate similarity. In step 1606, the decoder compares the similarity with a predetermined threshold. If the similarity is not less than the predetermined threshold, the decoder removes the candidate in step 1608 (and proceeds to step 1610). Otherwise, if the similarity is less than the predetermined threshold, the method proceeds to step 1610. In step 1610, if there are still candidates in the candidate MV list that need to be checked, the method returns to step 1604 for each remaining candidate in the candidate list. Otherwise, if all MVs in the MV candidate list have been compared with the new candidate (and each comparison is greater than the threshold in step 1606), in step 1612, the decoder retains the new MV candidate in the MV candidate list. In step 1614, method 1600 removes the first "N" candidates from the initial motion vector candidate list. The value of N can be a predetermined value, which can be used to ensure that the final list size is less than a predetermined maximum size. In some examples, if the initial motion vector candidate list has fewer than N candidates, then step 1614 does not modify the initial motion vector candidate list, and method 1600 proceeds to step 1616 and ends.

[0090] refer to Figure 16B Method 1650 includes with Figure 16ASimilar steps to method 1600, including steps 1602, 1604, 1606, 1610, and 1606, are further discussed below. In step 1602, the decoder stores an initial motion vector candidate list. For example, the initial motion vector candidate list may be empty. In step 1652, the decoder generates new motion vector candidates. In step 1604, the decoder compares the new candidates with existing candidates in the initial motion vector candidate list to determine the similarity of the candidates. In some examples, if there are no candidates in the initial motion vector candidate list, although not shown, method 1650 may proceed directly to step 1654 and add a candidate to the initial motion vector candidate list. In step 1606, the decoder compares the similarity with a predetermined threshold. If the similarity is not less than the predetermined threshold, the decoder proceeds to step 1654 without adding a new motion vector to the list (and proceeds to step 1610). If the similarity is less than the predetermined threshold, method 1650 proceeds to step 1654 and adds a candidate to the list. From step 1654, method 1650 proceeds to step 1656 and determines whether the list size is equal to the predetermined size. If not, the method proceeds to step 1610; otherwise, the method proceeds to step 1616 and ends. In step 1610, if there are still candidates to be checked, method 1650 returns to step 1604 for each remaining candidate; otherwise, method 1650 proceeds to step 1616 and ends.

[0091] In some embodiments, the similarity of MVs can be determined based on whether (a) the reference frame index (or POC) is the same, and / or (b) whether the MV difference is less than a threshold. For example, Equation 1 can be used to calculate the sum of the absolute MV distances of MVx and MVy:

[0092] Equation 1: abs(MVx0 – MVx1) + abs(MVy0 – MVy1) <K;

[0093] Where K is the pixel distance, such as 1 / 2 pixel, one integer pixel, two integer pixels, three integer pixels, 3 and 1 / 2 integer pixels, etc.

[0094] As another example, the absolute MV distance of MVx and the absolute MV distance of MVy can be compared with K using the following Equation 2:

[0095] Equation 2: abs(MVx0 – MVx1) <K&&abs(MVy0 – MVy1)<K;

[0096] Like Equation 1, where K can be 1 / 2 pixel, one integer pixel, two integer pixels, three integer pixels, 3 and 1 / 2 integer pixels, and so on.

[0097] In some embodiments, such as for a bidirectional matching mode, candidate MV pairs can be examined to determine whether they are in the same motion trajectory. For example, the original merged candidate MVs can be examined to determine whether multiple MVs in LIST_0 and LIST_1 are in the same motion trajectory. Figure 14 Examples of candidate MV pairs in the same motion trajectory are illustrated according to some embodiments. As shown in 1402, if multiple MVs in LIST_0 and LIST_1 are in the same motion trajectory, the candidate MV is added to the list; otherwise, as shown in 1404, if multiple MVs in LIST_0 and LIST_1 are not in the same motion trajectory, the multiple MVs in LIST_0 and LIST_1 are split into two candidate MVs. For each of the two separate candidate MVs, as shown in 1406 and 1408, the missing list MV is filled with a mirror image of the MV from another list. As in another example, each bidirectional prediction MV candidate is split into two candidates, one candidate being the LIST_0 MV and the other being the LIST_1 MV. The missing list MV is then filled with a mirror image of the valid list, with each candidate (e.g., each unidirectional prediction candidate) used to generate the missing list MV.

[0098] In PMVD MV search, the MV search method can be predetermined (e.g., a three-step diamond search). For example, for a diamond search, the step size of the first step is half a pixel (1 / 2 pixel), the step size of the second step is one-quarter a pixel (1 / 4 pixel), and the step size of the third step is one-eighth a pixel (1 / 8 pixel). In some embodiments, both (a) the merge index of the starting MV and (b) the coarse-grained MVD are sent. The MVD can be the refined position index of the first step diamond search, and / or a conventional MVD, with MVD units of 1 / 16 pixel, 1 / 8 pixel, 1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, and any predetermined unit. The multiple MVs of the selected merge index, plus the sent MVD (or the refined position MV), can be used as the starting MV of PMVD, which is stored in the MV buffer for merging candidates and AMVP candidate derivation reference. In some examples, for both the encoder and / or decoder, the PMVD search can begin with the PMVD-derived initial MV, and the final PMVD-derived MV is used only for the MC. Multiple initial MVs of a PMVD-encoded block can be reconstructed during the parsing phase.

[0099] In some examples, only one MVD and / or only one MVD refinement position index is sent. If the merge candidate is a bidirectional prediction candidate, MVD is added only on LIST_0 or LIST_1. For bidirectional matching merge patterns, if MVD is added on LIST_0, then the starting MV of LIST_1 can be a mirror MV of the starting MV of LIST_0.

[0100] In some examples, coarse-grained MVD is not encoded but is derived during the decoder's search process. For instance, we can divide this search process into three stages: a first-step diamond search, a second-step cross search, and a third-step cross search. The coarse-grained MVD can be the result of the search process in the first-step diamond search or the second-step cross search.

[0101] In HEVC, the image is divided into multiple Coding Tree Units (CTUs), which are the basic processing units of HEVC. Multiple CTUs are encoded in raster scan order. In the decoder architecture of the pipeline, because the rows have already been processed, most of the information from the upper CTU rows is available during the parsing phase (e.g., including MV information). In some examples, decoder-derived MVs from multiple CTUs in the upper CTU rows can be referenced (or used), for example, for merging candidate lists and AMVP list generation. Because the information from the parsing phase is available, even if it cannot be used because the decoder-derived MVs in the current CTU row are unavailable, the decoder can use the MVs derived from these CTUs.

[0102] Therefore, in some embodiments where CTU row constraints can be used with the techniques described in this invention, the MV can be referenced (e.g., when not referencing the MV of a PMVD-encoded block) or can be derived from the PMVD in the CTU row above (e.g., when storing merge candidate MVs, storing the merge candidate MVs and the mirrored MVs, sending merge indexes to the PMVD and the bidirectional mirrored MVs (and calculating only one MV), sending merge indexes and coarse-grained MVD and / or AMVP mode and PMVD).

[0103] For example, consider the techniques discussed here regarding when to store merge candidate MVs, storing merge candidate MVs and mirrored MVs, and sending the merge index for PMVD and bidirectional mirrored MVs (and evaluating only one MV). When referring to the PMVD-encoded block in the current CTU line, the selected merge candidate MV can be used for merge candidate derivation and AMVP candidate derivation. When referring to the PMVD-encoded block in the CTU line above, multiple MVs derived from the final PMVD can be used.

[0104] As another example, consider the technique discussed here regarding the MV of blocks that do not reference PMVD encoding. When referencing a PMVD-encoded block in the current CTU line, the MV used to merge candidate derivations and AMVP candidate derivations is unavailable; when referencing a PMVD-encoded block in the CTU line above, the MV of the final PMVD derivation is used.

[0105] CTU row constraints can be changed to CTU constraints or any predetermined or derived region constraints. For example, when the MV of a PMVD-encoded block is not referenced, if a CTU constraint is applied, the MV of a PMVD-encoded block in the current CTU will be unavailable, even though the MV of a PMVD-encoded block in a different CTU is available.

[0106] Overlapping Block Motion Compensation (OBMC) is an encoding tool that can be used to reduce block artifacts in motion compensation. JVET-F1001, entitled "Algorithm Description of Joint Exploration Test Model 6 (JEM 6)," describes an example of how OBMC is performed at block boundaries, the entirety of which is incorporated herein by reference. For ease of illustration, the following description refers to JVET-F1001, but this description is not intended to limit the invention.

[0107] For OBMC, in some examples, the MV of the current block compensates for adjacent blocks. For example... Figure 11 As shown, it is in section 2.3.4 of JVET-F1001. Figure 14 The excerpt states that OBMC is performed at the sub-block level for all motion compensation (MC) block boundaries, where the "sub-block" size is set to equal to 4×4. JVET-F1001 explains that when OBMC is applied to the current sub-block, in addition to the current motion vector, if the motion vectors of four adjacent connected sub-blocks are available and if they are not the same as the current motion vector, they can also be used to derive prediction blocks for the current sub-block. These prediction blocks are based on multiple motion vectors, which are combined to generate the final prediction signal for the current sub-block.

[0108] JVET-F1001 further explains that the predicted block based on the motion vectors of neighboring sub-blocks is labeled P. N N indicates the indices of the adjacent upper, lower, left, and right sub-blocks, and P is the predicted block label based on the motion vector of the current sub-block. C When P N When P is based on the motion information of neighboring sub-blocks that include the same motion information as the current sub-block, N OBMC is not executed. Otherwise, P N Each sample is added to P CThe same sample in P N Four rows / columns were added to P C The weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P. N And weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for P C Small MC blocks are an exception (when the block height or width is equal to 4 or when the CU is encoded using a sub-CU mode), and they only P N Two rows / columns are added to P C The weighting factors {1 / 4, 1 / 8} are used for P. N And weighting factors {3 / 4, 7 / 8} are used for P C In the case of P generated based on the motion vectors of vertical (horizontal) adjacent sub-blocks N P is weighted by the same weighting factor N Multiple samples from the same row (column) are added to P C . Figure 12A An exemplary high-level overview of OBMC method 1200 is shown, where MVA 1202 represents the original MV. Using decoder-side predictor techniques, MVA 1202 is refined to MVA' 1204, which is used for OBMC at block boundaries to generate hybrid portions 1206 and 1208 based on MVA' 1204.

[0109] As described herein, a technique is provided that allows for decoder-side predictor refinement techniques with similar runtime compared to conventional decoding methods. For example, this includes using an initial MV (not the refined MV) or a partially refined MV (initial MV + transmitted MV offset) to reference both the parsing and pre-extraction phases (e.g., Figure 6 Some embodiments of adjacent blocks in stages 602 and 604). In some embodiments, this technique can lead to the use of a starting MV for other processes, such as deblocking, OBMC, and time-coordinated MV derivation. Using a starting MV for other such processes may introduce block artifacts. For example, when OBMC and / or deblocking use a recovered MV, some block artifacts may be found, so that OBMC or deblocking is not performed using a refined MV. Figure 12B An exemplary result 1250 of applying OBMC using the initial MV1202 (e.g., the recovered MV) is shown, unlike Figure 12A It has hybrid sections 1206 and 1208 based on MVA '1204'. Figure 12B The mixed portions 1252 and 1254 are based on MVA 1202. As such, in one example, this could lead to block artifacts because the MV used for adjacent blocks is MVA'1204 but the MV used for the mixed portion is MVA'1202.

[0110] To solve this post-parsing problem, multiple MVs can be used. Figure 15 An exemplary decoding method 1500 for decoding video data using two motion vectors (MVs) is illustrated according to some embodiments. In step 1502, the decoder receives compressed video data associated with a set of frames. In steps 1504-1510, the decoder calculates a new motion vector for one frame from the set of frames using decoder-side predictive sub-refinement (DSR) techniques. Specifically, in step 1504, the decoder retrieves (e.g., from a first buffer) a first motion vector (e.g., the unrefined MV) associated with the current frame. In step 1506, the decoder performs a first part of the decoding process (e.g., a parsing phase, MV / MVP derivation, and / or MV refinement techniques) using the searched first motion vector. In step 1508, the decoder retrieves a second motion vector (e.g., from a second buffer) associated with the current frame (e.g., the refined MV). In step 1510, the decoder performs a second part of the decoding process (e.g., a reconstruction phase, a motion compensation phase, a deblocking phase, and / or OBMC) using the second motion vector.

[0111] Referring to steps 1504-1510, in some embodiments two sets of MVs can be used: (1) The first set of MVs is used in the parsing phase (e.g., Figure 7 The parsing phase includes (1) the MV / MVP derivation and / or pixel pre-extraction, and (2) a second set of MVs for reconstruction (e.g., in the parsing phase). Figure 7 In the intra-frame / MC reconstruction phase, this includes processes for motion compensation, OBMC, and / or deblocking. The first set of MVs can store the original (unrefined) MV, and the second set of MVs can store the refined MV. This technique can facilitate, for example, OBMC and / or deblocking using the refined MV, avoiding additional block artifacts (e.g., artifacts caused by running OBMC and / or deblocking with the unrefined MV) and / or providing better codec gain compared to using the unrefined MV.

[0112] For example, to handle potential block artifacts, a separate set of unrefined MVs can be used in the resolution phase (e.g., for merging candidate list generation and / or AMVP candidate list generation). According to some embodiments, multiple MVs in the unrefined MV set are not refined by a decoder-side MV refinement tool and can be used for MV resolution and MV reconstruction, with the reconstructed MVs then used for reference pixel extraction. MVs refined by the decoder-side MV refinement tool can be stored in another MV buffer set, which can be used for motion compensation, OBMC, deblocking, and / or other tools that do not alter the resolution process based on the MVs.

[0113] Since the MV has already been refined in other previously refined images, using the refined MV from these other images will not introduce the above combination. Figure 8 The pre-extraction problem described. In some embodiments, a refined MV set can be used for temporal MV derivation in the parsing phase and the MV reconstruction phase. For example, for merging candidate list generation and AMVP candidate list generation, when deriving spatially adjacent MVs, an unrefined MV set is used, although when deriving temporally contiguous MVs, a refined MV set is used.

[0114] Multiple MVs in the upper CTU row may also have been refined, as described above. In some embodiments, if an MV is in the upper CTU row, the first MV set (e.g., for the parsing phase) can store multiple MVs of the second MV set (e.g., for the reconstruction phase). For example, if an MV is in the upper CTU row, then the parsing phase can access the second MV set of the upper CTU row. For example, this can reduce the size of the unrefined MV buffer. For example, the buffer size can be reduced by only needing to hold MVs for one block row and one block column of the CTU. MVs will not be referenced by adjacent space blocks in the current CTU row during the parsing phase and the MV reconstruction phase (e.g., for merging candidate list generation and AMVP candidate list generation) and can be discarded. Therefore, in some embodiments, only refined MVs need to be stored. In hardware implementations, unrefined MVs may only be stored during the parsing pipeline phase and the prefetch pipeline phase (e.g., Figure 7 (Stages 702 and 704-2 in the text). In some embodiments, the technique can use refined MVs from CUs that are processed before N previous CUs. For example, if we consider that multiple CUs before the last 5 decoded CUs can be used (e.g., without introducing a pre-extraction problem), the MVs in the multiple CUs before the last 5 decoded CUs can be used with refined MVs. In some embodiments, the same concept can be used for tile / slice boundaries. For example, if the reference MVs are in different tiles or different slices, then the resolution stage can access a second set of MVs in different tiles or different slices.

[0115] Regarding the first MV set used in the parsing phase, the first MV set (unrefined MVs) can be used for merging / AMVP candidate generation and / or initial MV generation, with the generated MVs used for reference pixel extraction. In some embodiments, if no CTU row constraints are applied, the MV set includes (a) the unrefined MVs of the current image (e.g., the left column, the top row, and the current CTU), and (b) the refined MVs of other images (e.g., temporally co-located images). For example, reference... Figure 13The MV set includes the unthinned MV of the current image in the current CTU 1302, left column 1304, and top row 1306. In some embodiments, if CTU row constraints are applied, the MV set includes (a) the unthinned MV of the current CTU row (left column and current CTU), (b) the thinned MV of the top CTU row (top row), and (c) the thinned MV of other images. For example, refer to... Figure 13 The MV set includes the unrefined MV for the current CTU row 1302 and the current CTU row 1304 in the left column, and the refined MV for the upper CTU row 1306 in the upper column.

[0116] Regarding the second MV set used in the reconstruction phase, the second MV set can be used for motion compensation, OBMC, and / or deblocking. The second MV set includes (a) the thinned MV of the current image, and (b) the thinned MV of other images. For example, refer to... Figure 13 The MV set includes a refined MV of the current image for the current CTU 1302, left column 1304, and top row 1306.

[0117] The proposed multiple MV / MV set approach can also be applied to the encoder. For example, a single, unrefined MV set can be used in the syntax encoding stage, MV derivation, and / or MVP derivation (e.g., merged candidate list generation and / or AMVP candidate list generation). According to some examples, multiple MVs in the unrefined MV set are not refined by the decoder-side MV refinement tool and can be used for MV encoding and MVP generation. Multiple MVs refined by the decoder-side refinement tool can be stored in another MV buffer set, and the refined MVs can be used for motion compensation, OBMC, deblocking, and / or other tools that do not change the parsing process based on the MVs.

[0118] Similarly, simply put, decoder-side MV thinning tools (e.g., PMVD, DMVR, and BIO) can alter the MV of a block (e.g., which can lead to the aforementioned parsing or reference pixel pre-extraction problems). In some embodiments, when the thinned MV is stored back, the difference between the thinned MV and the starting MV can be constrained to a predetermined threshold. For example, if the difference between the thinned MV and the starting MV is greater than the predetermined threshold (e.g., an integer pixel distance of 4, 8, or 16), then the thinned MV is first clipped (e.g., set to be less than or equal to the threshold) and then stored as the clipped MV. For example, the MV can be clipped by the starting MV ± 4, 8, or 16 integer pixels, and if the difference between the thinned MV and the starting MV is less than the threshold, the thinned MV is stored directly.

[0119] The impact of decoder-side MV refinement tools altering blocks of MVs can be reduced by removing the pruning process between these refined MVs and other MVs in MV / MVP derivation (e.g., during merged candidate list reconstruction or AMVP list reconstruction). For example, in some embodiments, the pruning process for removing redundancy between potential candidates is applied only to MVs that are not refined in the decoder. For these candidates that may be refined in the decoder, the refined MVs can be directly added to the candidate list without using the pruning process. In some embodiments, eliminating this pruning can be combined with other techniques described above (e.g., refined MV clipping and multiple MV / MV sets) to further reduce the impact.

[0120] In some embodiments, OBMC is applied during the refactoring phase (e.g., Figure 6 (Stage 606 in the text). Therefore, two different techniques can be used for OBMC (either alone or in combination, for example, using different techniques for sub-blocks along different edges). The first technique is to use the initial MV or a partially refined MV (e.g., an unrefined MV), which is stored in the MV buffer for OBMC. The second technique is to use the decoder-side refined MV (e.g., a refined MV) for OBMC.

[0121] The technical operations described in this invention can be implemented in any suitable manner, and the processes and decision blocks in the flowcharts above can include steps and actions in algorithms that perform these various processes. Algorithms derived from these processes can be implemented as software integrating and guiding one or more single or multi-purpose processors, as functionally equivalent circuits such as digital signal processor (DSP) circuits or application-specific integrated circuits (ASICs), or as other suitable methods. It should be understood that the flowcharts included in this invention do not describe the syntax or operation of any particular circuit or any particular coding language or type of coding language. Rather, the flowcharts illustrate functional information that those skilled in the art can use to manufacture circuits or to implement computer software algorithms to perform processes of specific devices implementing the various techniques described in this invention. It should also be understood that, unless otherwise specified herein, the sequence of steps and / or actions shown in each flowchart is merely an exemplary algorithm that can be implemented and varied in implementation and embodiments of the principles described herein.

[0122] Therefore, in some embodiments, the techniques described in this invention can be implemented as computer-readable instructions implemented as software, including application software, system software, firmware, middleware, embedded code, or any other suitable type of computer code. Such computer-readable instructions can be written using any of many suitable coding languages ​​and / or coding or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine.

[0123] When the techniques described herein are implemented as computer-executable instructions, these computer-executable instructions can be implemented in any suitable manner, including a number of functional facilities, each providing one or more operations to perform the execution of algorithmic operations according to these techniques. A "functional facility," while specifically embodied as a structural element of a computer system, causes said one or more computers to perform a specific operational role when integrated with and executed by said one or more computers; a functional facility can be part of an overall software element. For example, a functional facility can be implemented as a function of a process, or as a discrete process, or as any other suitable unit of a process. If the techniques described herein are implemented as multiple functional facilities, each functional facility can be implemented in its own way, without needing to be implemented in the same way. Furthermore, these functional facilities can be executed in parallel and / or serially as appropriate, and can pass information between each other using shared memory on the computer, using message passing protocols or any other suitable manner.

[0124] Generally, functional facilities include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types. Typically, the functionality of functional facilities can be combined or distributed as needed within the system they are permitted to operate on. In some embodiments, one or more functional facilities implementing the techniques of the present invention may simultaneously form a complete software package. In alternative embodiments, these functional facilities are applicable to interacting with each other, unrelated functional facilities, and / or processes to implement software program applications.

[0125] This invention has described some exemplary functional facilities for implementing one or more tasks. However, it should be understood that the described functional facilities and task divisions are merely illustrative of the types of functional facilities that can implement the exemplary techniques described in this invention, and these embodiments are not limited to any particular type of coding, division, or functional facility. In some embodiments, the indexing function may be implemented as a single functional facility. It should be understood that in some embodiments, some of the functional facilities described herein may be implemented together with other separate functional facilities (i.e., as a single unit or discrete units), or some of these functional facilities may not be implemented.

[0126] In some embodiments, computer-executable instructions implementing the technology described in this invention (when implemented as one or more functional facilities or in any other manner) may be encoded on one or more computer-readable media to provide functionality to the media. Computer-readable media include magnetic media such as hard disk drives, optical media such as optical discs (CDs) or digital versatile optical discs (DVDs), permanent or non-permanent solid-state memory (e.g., flash memory, magnetic RAM, etc.), or any other suitable storage media. Such computer-readable media may be implemented in any suitable manner, and as used herein, "computer-readable media" (also called "computer-readable storage media") refers to a tangible storage medium. Tangible storage media are non-transient and have at least one physical, structural element. As used herein, in a "computer-readable medium," at least one physical, structural element has at least one physical property that may be altered in some way during the creation of a process having embedded information, recording information thereon, or any other process encoding a medium having information. For example, the magnetization means that constitutes part of the magnetization result of a computer-readable medium may be altered during the recording process.

[0127] Furthermore, some of the actions described above involve storing information (e.g., data and / or instructions) in a certain way for use by these technologies. In some embodiments of these technologies, such as embodiments implemented as computer-readable instructions, information is encoded on a computer-readable storage medium. Specific architectures are referred to herein as advantageous formats for storing information, structures that can be used to give physical organization to information when encoded on a storage medium. These advantageous structures can provide functionality to the storage medium by influencing the operation of one or more processors interacting with the information, for example, by increasing the efficiency of computer operations performed by the processor.

[0128] In some, but not all, embodiments, the technology can be implemented as computer-executable instructions that can be executed on one or more suitable computing devices operating in any suitable computer system, or programmed onto one or more computing devices (or one or more processors of one or more computing devices) to execute the computer-executable instructions. The computing device or processor can be programmed to execute the instructions when the instructions are stored in a manner accessible to the computer device or processor, such as in a data storage area (e.g., an on-chip buffer or instruction register, a computer-readable storage medium accessed via a bus, a computer-readable storage medium accessed via one or more networks, and a computer-readable storage medium accessed by the device / processor). The functional facilities including these computer-executable instructions can be integrated with and direct the operation of a single multi-purpose programmable digital computing device, a coordinated system of two or more multi-purpose computing devices sharing processing power and jointly executing the technology described herein, a single computing device or a cooperative system of computing devices (co-located or geographically distributed) for executing the technology described herein, one or more field-programmable gate arrays (FPGAs) for executing the technology described herein, or any other suitable system.

[0129] A computing device may include at least one processor, a network adapter, and a computer-readable storage medium. For example, a computing device may be a desktop or laptop computer, a personal digital assistant (PDA), a smartphone, a server, or any other suitable computing device. The network adapter may be any suitable hardware and / or software enabling the computing device to communicate wired and / or wirelessly with any other suitable computing device via any suitable computing network. The computing network may include wireless access points, switches, routers, gateways, and / or other network devices, as well as any suitable wired and / or wireless communication media for exchanging data between two or more computers, including the Internet. The computer-readable medium may be suitable for storing data to be processed and / or instructions to be executed by the processor. The processor performs the processing of the data and the execution of the instructions. The data and instructions may be stored on the computer-readable storage medium.

[0130] A computing device may additionally include one or more components and peripheral devices, including input and output devices. These devices may also be used to present a user interface. Examples of output devices that can be used to provide a user interface include a printer or display for visual presentation of output, and a speaker or other sound-generating device for audible presentation of output. Examples of input devices that can be used for a user interface include a keyboard and positioning devices such as a mouse, touchpad, and digital panel. As another example, a computing device may receive input information via voice recognition or in other audible formats.

[0131] Embodiments of the technology described herein implemented in circuits and / or computer-executable instructions have been described. It should be understood that some embodiments may be in the form of methods, of which at least one example is provided. Actions performed as part of the method may be ordered in any suitable manner. Therefore, embodiments may be constructed that perform actions in an order different from that shown, which may include performing several actions simultaneously, although shown as sequential actions in the illustrative embodiments.

[0132] The various aspects of the above embodiments can be used individually, in combination, or in various arrangements not specifically discussed in the above embodiments, and are therefore not limited to the details and arrangements of the components set forth in the above description or shown in the accompanying drawings. For example, an aspect described in one embodiment can be combined in any way with aspects described in other embodiments.

[0133] The use of ordinal numbers such as "first," "second," "third," etc., in the claims does not imply any priority or order of execution of a claim element in another time sequence, wherein the actions of the method of execution are performed, but are only used as markers to distinguish one claim element with a particular name from another element with the same name (but for the use of ordinal numbers) to distinguish the claim elements.

[0134] Furthermore, the phraseology and terminology used herein are for descriptive purposes and should not be considered restrictive. The terms “including,” “comprising,” “having,” “containing,” “invoking,” and their variations, as used herein, are intended to include items listed herein, their equivalents, and any additional items.

[0135] The word "exemplary" as used herein means used as an example, instance, or illustration. Therefore, any embodiments, implementations, processes, features, etc., described herein should be understood as illustrative examples and not as preferred or advantageous examples, unless otherwise stated.

[0136] Therefore, several aspects of at least one embodiment have been described, and it should be understood that various changes, modifications, and improvements will readily occur to those skilled in the art. Such changes, modifications, and improvements are intended as part of the invention and are intended to be within the spirit and scope of the principles described herein. Therefore, the foregoing description and figures are merely exemplary.

Claims

1. A decoding method for decoding video data, characterized in that, The decoding method includes: Receive input data associated with the current encoding / decoding unit in the current image; The motion vector candidate list generation process derives a motion vector candidate list using the unrefined motion vectors in the current image and the refined motion vectors associated with co-located frames. This list includes: By generating the motion vector candidate list, the first motion vector of the previous codec unit processed before the current codec unit is used as the unrefined motion vector in the current image to derive spatial neighbor motion vector candidates. The first motion vector is an unrefined motion vector used to determine the corresponding refined motion vector of the previous codec unit, and the corresponding refined motion vector is used to perform the prediction of the previous codec unit. The corresponding refined motion vector is derived based on the first motion vector through a decoder-side predictive sub-refining technique, which is bidirectional matching or template matching. By generating the candidate list of motion vectors, a temporally co-located motion vector is derived using a second motion vector associated with the co-located frame, wherein the second motion vector is the refined motion vector. as well as Decode the current codec unit using the exported list of motion vector candidates.

2. The decoding method according to claim 1, wherein, The generation of the motion vector candidate list includes motion vector prediction derivation, candidate merging derivation, or both.

3. The decoding method according to claim 1, wherein, The refined motion vector is used to perform a partial decoding process for the current codec unit, wherein the partial decoding process includes performing motion compensation.

4. A decoding apparatus for decoding video data, the decoding apparatus comprising a processor in communication with a memory, the processor being configured to execute instructions stored in the memory, which cause the processor to: Receive input data associated with the current encoding / decoding unit in the current image; The motion vector candidate list generation process derives a motion vector candidate list using the unrefined motion vectors in the current image and the refined motion vectors associated with co-located frames. This list includes: By generating the motion vector candidate list, the first motion vector of the previous codec unit processed before the current codec unit is used as the unrefined motion vector in the current image to derive spatial neighbor motion vector candidates. The first motion vector is an unrefined motion vector used to determine the corresponding refined motion vector of the previous codec unit, and the corresponding refined motion vector is used to perform the prediction of the previous codec unit. The corresponding refined motion vector is derived based on the first motion vector through a decoder-side predictive sub-refining technique, which is bidirectional matching or template matching. By generating the candidate list of motion vectors, and using the second motion vector associated with the co-located frame, a temporally co-located motion vector is derived, wherein the second motion vector is the refined motion vector; and Decode the current codec unit using the exported list of motion vector candidates.

5. A video decoding method, characterized in that, include: Receive input data associated with the current encoding / decoding unit in the current image; For the co-occurrence encoding / decoding unit in the first co-occurrence image processed before the current image, the first unrefined motion vector is derived based on the motion vector candidate list; Using decoder-side predictive sub-refinement technology, a first refined motion vector for the co-occurrence codec unit is calculated based on the first unrefined motion vector. The decoder-side predictive sub-refinement technology is bidirectional matching or template matching. For the previous encoding / decoding unit processed before the current encoding / decoding unit in the current image, derive the second unrefined motion vector based on the motion vector candidate list; Using decoder-side predictive sub-refinement technology, the second refined motion vector of the previous codec unit is calculated based on the second unrefined motion vector. This decoder-side predictive sub-refinement technology is either bidirectional matching or template matching. The motion vector candidate list generation process generates a motion vector candidate list for the current codec unit, including: Based on the second unrefined motion vector, candidate spatial neighbor motion vectors are derived; Based on the first refined motion vector, candidate time-coordinated motion vectors are derived; and The current codec unit is decoded using the derived list of motion vector candidates.

6. The video decoding method according to claim 5, wherein, The generation of the motion vector candidate list includes motion vector prediction export, merged candidate export, or both.

7. The video decoding method according to claim 5, wherein, The current codec unit performs a portion of its decoding process using the first refined motion vector and the second refined motion vector, and this portion of the decoding process includes performing motion compensation.

8. A video decoding method, characterized in that, include: Receive compressed video data associated with a set of frames; For the current encoding / decoding unit in this frame group, derive the first unrefined motion vector based on the motion vector candidate list; Using decoder-side predictive sub-refinement technology, a refined motion vector for the current codec unit is calculated based on the first unrefined motion vector. This refined motion vector estimates the motion of the current codec unit based on one or more reference frames in the frame group, including: For the current codec unit, retrieve the first motion vector of the previous codec unit processed before the current codec unit, wherein the first motion vector is an unrefined motion vector used to determine the corresponding final refined motion vector of the previous codec unit, and the unrefined motion vector is not used to perform the prediction of the previous codec unit, and the final refined motion vector is used to perform the prediction of the previous codec unit. Retrieve a second motion vector associated with the first co-position frame, which is a refined motion vector; Using the first motion vector and the second motion vector, the first part of the decoding process of the current encoder-decoder unit is performed. This first part of the decoding process includes generating a motion vector candidate list, which includes motion vector prediction derivation, merging candidate derivation, or both. For the generation of this motion vector candidate list, spatial neighbor motion vector candidates are derived using the unrefined motion vector from the previous encoding / decoding unit. For the generation of this motion vector candidate list, use the refined motion vector to derive time-coordinated motion vector candidates; Using the refined motion vector, the second part of the decoding process of the current encoder-decoder unit is performed, and the second part of the decoding process includes motion compensation; Wherein, the first unrefined motion vector is the unrefined motion vector of the current codec unit and is used as the motion vector of the neighboring codec units processed after the current codec unit, and the refined motion vector of the current codec unit is used as the motion vector of one or more frames processed after the current frame; The decoder-side predictor refinement technique is either bidirectional matching or template matching.

Images