An Error Plane-Based Sub-Pixel Accurate Refinement Method for Decoder-Side Motion Vector Refinement

Abstract

Given that decoder-side motion vector refinement / derivation is a normative aspect of the coding system, the encoder would also have to perform the same error surface techniques to avoid drift between the encoder's reconstruction and the decoder's reconstruction. Thus, all aspects of all embodiments are applicable to both encoding and decoding systems. In template matching, refinement occurs only in references starting from the centers of sub-pixel accuracy based on explicitly signaled merge indices or implicitly derived through cost evaluation. In bilateral matching, refinement (with or without averaging templates) begins in reference lists L0 and L1, starting from their respective centers of sub-pixel accuracy based on explicitly signaled merge indices or implicitly derived through cost evaluation.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of Indian Provisional Patent Application No. 201831024666, filed on July 2, 2018, entitled "Error Plane Based Sub-Pixel Accurate Refinement Method for Decoder Side Motion Vector Refinement", which is hereby incorporated by reference in its entirety. [Background technology]

[0002] Current hybrid video codecs, such as H.264 / AVC or H.265 / HEVC, employ compression that includes predictive coding. Pictures of a video sequence are subdivided into blocks of pixels, and these blocks are subsequently coded. Instead of coding the block pixel by pixel, the entire block is predicted using already encoded pixels in its spatial or temporal proximity. The encoder further processes only the difference between the block and its prediction. This further processing typically involves transforming the block pixels into coefficients in the transform domain. These coefficients may then be further compressed by quantization and further compacted by entropy coding to form a bitstream. The bitstream further includes any signaling information that allows decoding of the encoded video. For example, the signaling information may include settings related to the encoding, such as the size of the input picture, the frame rate, a quantization step indication, predictions applied to blocks of pictures, etc. The coded signaling information and the coded signal are ordered in the bitstream in a manner known to both the encoder and the decoder. This allows the decoder to parse the coded signaling information and the coded signal.

[0003] Temporal prediction exploits the temporal correlation between pictures, also called frames, of a video. Temporal prediction is also called inter-prediction, since it is a prediction that uses the inter-dependence between different video frames. Thus, a block being encoded, also called a current block, is predicted from one or more previously encoded pictures, called reference picture(s). A reference picture is not necessarily a picture that precedes the current picture in which the current block is located in the display order of a video sequence. An encoder may code pictures in a coding order different from the display order. As a prediction of the current block, a co-located block in a reference picture may be determined. A co-located block is a block that is located in the same position in the reference picture as the current block in the current picture. Such a prediction is accurate for picture areas with no motion, i.e., no motion from one picture to another.

[0004] Motion estimation is typically used when determining the prediction of a current block to obtain a predictor that takes motion into account, i.e. a motion compensated predictor. Thus, the current block is predicted by a block in a reference picture that is located at a distance given by a motion vector from the location of the co-located block. To enable the decoder to determine the same prediction of the current block, the motion vector may be signaled in the bitstream. To further reduce the signaling overhead caused by signaling the motion vector for each of the blocks, the motion vector itself may be estimated. The motion vector estimation may be performed based on the motion vectors of neighboring blocks in the spatial and / or temporal domain.

[0005] The prediction of the current block may be calculated using one reference picture or by weighting predictions obtained from two or more reference pictures. The reference pictures may be adjacent pictures, i.e. pictures immediately preceding and / or immediately following the current picture in display order, since adjacent pictures are most likely to be similar to the current picture. In general, however, the reference pictures may be any other pictures preceding or following the current picture in display order and preceding the current picture in the bitstream (in decoding order). This may provide advantages, for example, in case of concealment and / or non-linear motion in the video content. Thus, reference picture identification information may also be signaled in the bitstream.

[0006] A special mode of inter prediction is the so-called bi-prediction, in which two reference pictures are used in generating a prediction of the current block. In particular, two predictions determined in each of the two reference pictures are combined into a prediction signal of the current block. Bi-prediction may lead to a more accurate prediction of the current block than uni-prediction, i.e., prediction using only a single reference picture. Such a more accurate prediction leads to a smaller difference (also called "residual") between the pixels of the current block and the prediction, which can be encoded more efficiently, i.e., compressed into a shorter bitstream. In general, three or more reference pictures may be used to find three or more respective reference blocks to predict the current block. That is, multi-reference inter prediction can be applied. Thus, the term multi-reference prediction includes bi-prediction and prediction using three or more reference pictures.

[0007] To provide more accurate motion estimation, the resolution of the reference picture may be increased by interpolating samples between pixels. Fractional pixel interpolation can be performed by a weighted average of the nearest pixels. In the case of half-pixel resolution, for example, bilinear interpolation is typically used. Other fractional pixels are calculated as the average of the nearest pixels weighted by the inverse of the distance between each nearest pixel and the predicted pixel.

[0008] Motion vector estimation is a computationally complex task, in which the similarity between a current block and a corresponding predicted block pointed to by a candidate motion vector in a reference picture is calculated. Typically, a search area includes M×M samples of an image, and each sample position of the M×M candidate positions is tested. This testing involves the calculation of a similarity metric between an N×N reference block C and a block R located at the tested candidate position of the search area. Due to its simplicity, the sum of absolute differences (SAD) is a metric frequently used for this purpose,

number

[0009] In the above formula, x and y define candidate positions in the search area, while indices i and j indicate samples in the reference block C and candidate block R. The candidate positions are often called block displacements or offsets, reflecting the representation of block matching as shifting the reference block in the search area and calculating the similarity between the reference block C and the overlapping part of the search area. To reduce complexity, the number of candidate motion vectors is usually reduced by restricting the candidate motion vectors to a search space. The search space may be defined, for example, by the number and / or location of pixels surrounding a location in the reference picture that corresponds to the location of the current block in the current image. After calculating the SAD for all M×M candidate positions x and y, the best matching block R is the block at the location that gives the lowest SAD, corresponding to the maximum similarity with the reference block C. On the other hand, the candidate motion vectors may be defined by a list of candidate motion vectors formed by the motion vectors of neighboring blocks.

[0010] Motion vectors are usually at least partially determined at the encoder side and signaled to the decoder in the coded bitstream. However, motion vectors may also be derived at the decoder. In such a case, the current block is not available at the decoder and cannot be used to calculate the similarity with the block to which the candidate motion vector points in the reference picture. Thus, instead of the current block, a template is used that is constructed from pixels of already decoded blocks. For example, already decoded pixels neighboring the current block may be used. Such motion estimation offers the advantage of reducing signaling, i.e., no signaling is required, since the motion vectors are derived in the same way at both the encoder and the decoder. On the other hand, the accuracy of such motion estimation may be lower.

[0011] To provide a trade-off between accuracy and signaling overhead, motion vector estimation may be split into two steps: motion vector derivation and motion vector refinement. For example, motion vector derivation may include the selection of a motion vector from a list of candidates. Such a selected motion vector may be further refined, for example, by a search in a search space. The search in the search space is based on the calculation of a cost function for each candidate motion vector, i.e., for each candidate position of the block to which the candidate motion vector points.

[0012] Document JVET-D0029: Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching, X. Chen, J. An, J. Zheng (which can be found at http: / / phenix.it-sudparis.eu / jvet / site) shows a motion vector refinement where a first motion vector is found at integer pixel resolution and further refined by a search at half pixel resolution in a search space around the first motion vector. A bilateral motion vector search based on block templates is used.

[0013] Motion vector estimation is a key feature of modern video encoders and decoders, as its efficiency in terms of quality, speed and complexity impacts the efficiency of video encoding and decoding. Summary of the Invention

[0014] The present invention relates to video encoding and decoding, and in particular to determining motion vectors.

[0015] In a first aspect of the present application, a method in a decoder-side motion vector refinement system for obtaining sub-pixel accurate delta motion vectors in one or more reference frames around respective initial sub-pixel accurate refinement center(s) may include the following steps:

[0016] performing a plurality of integer 1-pixel distance refinement operations (iterations) iteratively using a cost function to determine an integer distance refinement motion vector for each reference frame, the search center being updated after an operation (iteration) to the lowest cost location in that operation (iteration); determining that an early termination of the iterative loop occurs because the cost of the center location in a given operation (iteration) is lower than the cost of a set of 1-pixel neighboring locations around it, or determining that a predetermined number of operations (iterations) has been reached. If an early termination occurs, the method may include determining a sub-pixel distance refinement motion vector around the last search center in each reference by calculating a location having a minimum value on a parametric error surface fitted using the cost function values ​​of the last search center and the set of 1-pixel neighboring locations around it; and returning a total refinement motion vector as the sum of the determined integer distance refinement motion vector and the determined sub-pixel distance refinement motion vector for each reference frame. When a predetermined number of runs (iterations) has been reached, the method may include returning a refined motion vector corresponding to the position having the minimum cost function value across all runs (iterations).

[0017] In one implementation of the first aspect, a cost function is performed for joint refinement in both reference lists L0 and L1 such that the displacement in reference list L1 is equal and opposite to the displacement in reference list L0 in both the horizontal and vertical directions (the cost function is referred to as SBM_JOINT).

[0018] In a second aspect of the present application, a method in a decoder-side motion vector refinement system for obtaining sub-pixel accurate delta motion vectors in one or more reference frames around respective initial sub-pixel accurate refinement center(s) may include the following steps:

[0019] A step of sequentially performing a number of integer 1-pixel distance refinement operations (iterations) using a first cost function to determine an integer distance refinement motion vector for each reference frame, wherein the search center is updated after an operation (iteration) to the lowest cost position in that operation (iteration); and a step of determining that an early termination of the sequentially iterative loop occurs because the cost of the center position in a given operation (iteration) is lower than the costs of a set of 1-pixel neighboring positions around it, or determining that a predetermined number of operations (iterations) has been reached.

[0020] Using a second cost function to evaluate cost function values ​​at the last search center and a set of 1-pixel neighboring locations of the last search center: if the last search center location has the lowest second cost function value when compared with the second cost function values ​​at the set of 1-pixel neighboring locations to the last search center, determining a sub-pixel distance refinement motion vector around the best integer distance refinement location in each reference by calculating a location having a minimum value on a fitted parametric error surface using the second cost function values, and returning a total refinement motion vector as the sum of the determined integer distance refinement motion vector and the determined sub-pixel distance refinement motion vector for each reference frame. Otherwise, if the last search center location does not have the lowest second cost function value when compared with the second cost function values ​​at the set of 1-pixel neighboring locations to the last search center, the method includes returning a refinement motion vector corresponding to the location having the best second cost function value for each reference frame.

[0021] In one implementation of the second aspect, the first cost function is SBM_JOINT and the second cost function is TBM_INDEPDENDENT, which is defined as the cost function associated with performing independent refinements on both reference lists L0 and L1 against a common bilateral averaged template.

[0022] The method can be used for template matching and bilateral matching use cases.

[0023] In another aspect of the invention, the methods disclosed herein may be implemented as instructions stored on a non-transitory computer-readable medium, which may be read and executed by a processor to perform the steps of the methods described above.

[0024] In some aspects of the present invention, a method for decoder-side motion vector refinement includes determining a target integer motion vector displacement by comparing integer distance costs corresponding to candidate integer motion vector displacements with respect to an initial motion vector, determining sub-pixel motion vector displacements by performing calculations on the integer distance costs, and determining a refined motion vector based on the target integer motion vector displacements, the sub-pixel motion vector displacements, and the initial motion vector.

[0025] The present invention achieves many advantages over conventional techniques. For example, embodiments of the present invention utilize determined early termination of iteration loops based on checking the cost of a center location in a given iteration operation against the costs of a set of 1-pixel neighboring locations around the center location. By terminating iteration loops early, unnecessary computations can be reduced or eliminated.

[0026] Furthermore, an apparatus may implement the above-described methods and may be a combination of software and hardware. For example, encoding and / or decoding may be performed by an integrated circuit (semiconductor device or chip), such as a general purpose processor (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). However, embodiments of the invention are not limited to implementation on programmable hardware. Embodiments of the invention may be implemented on an application specific integrated circuit or by one or more combinations of CPU, DSP, FPGA, and ASIC components. [Brief description of the drawings]

[0027] In the following, exemplary embodiments are explained in more detail with reference to the accompanying figures and drawings.

[0028] [Figure 1] 2 is a block diagram illustrating an example structure of an encoder for encoding a video signal, according to one embodiment of the present disclosure.

[0029] [Diagram 2] 2 is a block diagram illustrating an exemplary structure of a decoder for decoding a video signal according to an embodiment of the present disclosure.

[0030] [Diagram 3] FIG. 1 is a schematic diagram illustrating an exemplary template matching suitable for bi-prediction.

[0031] [Figure 4A] FIG. 1 is a schematic diagram illustrating an exemplary template matching suitable for uni-prediction and bi-prediction.

[0032] [Figure 4B] FIG. 2 is a schematic diagram illustrating an exemplary bilateral matching suitable for uni-prediction and bi-prediction;

[0033] [Diagram 5] FIG. 1 is a flow diagram illustrating a possible implementation of a motion vector search.

[0034] [Figure 6] FIG. 2 is a schematic diagram illustrating an example of local illumination compensation applied in video coding.

[0035] [Figure 7] FIG. 2 is a schematic diagram illustrating an example of decoder-side motion vector refinement.

[0036] [Figure 8] FIG. 2 is a schematic diagram showing examples of sub-pixel positions.

[0037] [Figure 9] 1 is a block diagram of a set of 1-pixel neighborhood locations around a central pixel for obtaining sub-pixel accurate delta motion vector refinement in one or more reference frames according to an embodiment of the present disclosure.

[0038] [Figure 10] 1 is a simplified flow diagram illustrating a method for obtaining sub-pixel accurate delta motion vector refinement in one or more reference frames in a decoder-side motion vector refinement system according to some embodiments of the present disclosure.

[0039] [Figure 11] FIG. 1 is a simplified flow diagram illustrating a method for implementing certain embodiments of the present disclosure.

[0040] [Figure 12] FIG. 1 is a block diagram of an apparatus that can be used to implement various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Embodiments of the present disclosure relate to improvements in template matching applied in motion vector refinement, particularly where template matching is applied to zero-mean templates and zero-mean candidate blocks even when the mean of the best matching blocks found is not further adjusted (by local illumination control).

[0042] Template matching is used to find the best first and second motion vectors that point to the first and second reference pictures, respectively. Template matching is performed for each reference picture by template matching in a predefined search space on a location given by an initial motion vector that may be derived by or signaled to the decoder.

[0043] Template matching may be performed based on a block template derived from the block pointed to by the initial motion vector.

[0044] Such template matching to find a best matching block to obtain a predictor for a current block may be used, for example, in hybrid video encoders and / or decoders, and may be advantageously applied to encoders and / or decoders, such as HEVC, etc. In particular, further developments of HEVC or new codecs / standards may utilize embodiments of the present disclosure.

[0045] FIG. 1 shows an encoder 100 including an input 102 for receiving input image samples of a frame or picture of a video stream, and an output 172 for generating an encoded video bitstream. As used herein, the term "frame" in this disclosure is also referred to as a picture. Note that this disclosure is also applicable to interlaced fields of video. In general, a picture includes m×n pixels, which correspond to image samples, and may include one or more color components. As used herein, the following description refers to pixels as luminance samples. However, note that the motion vector search of this disclosure can be applied to any color component, including components of the search space such as chrominance or RGB. Furthermore, it may be advantageous to perform motion vector estimation for only one component and apply the estimated motion vector to more or all components.

[0046] The input blocks to be coded do not necessarily have the same size: a picture may contain blocks of different sizes and the block rasters of different pictures may also be different.

[0047] In one example embodiment, the encoder 100 is configured to perform prediction, transformation, quantization, and entropy coding on the video stream, which are performed by a transform unit 106, a quantization unit 108, and an entropy coding unit 170, respectively, to generate an encoded video bitstream.

[0048] A video stream may include multiple frames, each divided into blocks of a certain size that are either intra- or inter-coded. For example, blocks of a first frame of the video stream are intra-coded by intra prediction unit 154. Intra frames are coded using only information within the same frame so that they can be decoded independently and provide entry points in the bitstream for random access. Blocks of other frames of the video stream may be inter-coded by inter prediction unit 144, i.e., information from a previously coded frame (reference frame) is used to reduce temporal redundancy, and each block of the inter-coded frame is predicted from a block in the reference frame. Mode selection unit 160 is configured to select between blocks 155 of the frame processed by intra prediction unit 154 and blocks 145 of the frame processed by inter prediction unit 144. Mode selection unit 160 also controls parameters of intra or inter prediction. Intra-coded blocks may be provided within inter-coded frames to allow refreshing of image information. Additionally, intra frames, which contain only intra-coded blocks, may be inserted periodically into a video sequence to provide an entry point for decoding, i.e., a point at which the decoder can begin decoding without having information from previously coded frames.

[0049] The intra estimation unit 152 and the intra prediction unit 154 are units configured to perform intra prediction. In particular, the intra estimation unit 152 may derive a prediction mode also based on knowledge of the original image, while the intra prediction unit 154 provides a corresponding predictor, i.e., a sample predicted with the selected prediction mode, for differential encoding. To perform spatial or temporal prediction, the coded block may be further processed by the inverse quantization unit 110 and the inverse transform unit 112 to provide an inverse transformed block 113. The reconstruction unit 114 combines the inverse transformed block 113 with the prediction block 165 to provide a reconstructed block 115, which is provided to the loop filtering unit 120 to further improve the quality of the decoded image. The filtered block then forms a reference frame that is then stored in the decoded picture buffer 130. The inverse quantization unit 110, the inverse transform unit 112, the reconstruction unit 114 and the loop filter 120 form part of a decoder (decoding loop). Such a decoding loop (decoder) on the encoder side offers the advantage of generating the same reference frame as the reference picture to be reconstructed on the decoder side. The encoder and decoder therefore operate in a corresponding manner. The term "reconstruction" here refers to obtaining a reconstructed block 115 by adding a prediction block 165 to an inverse transformed (decoded residual) block 113.

[0050] The encoder 100 also includes an inter estimation unit 142 that receives the picture block 101 of the current frame or picture to be inter coded and one or more reference frames from the decoded picture buffer 130. Motion estimation is performed by the inter estimation unit 142, while motion compensation is performed by the inter prediction unit 144. Motion estimation is used to obtain a motion vector and a reference frame based on a certain cost function, for example also using the original image to be coded. For example, the motion estimation (inter estimation) unit 142 can provide an initial motion vector estimate. The initial motion vector can then be signaled in the bitstream directly in the form of a motion vector or as an index pointing to a motion vector candidate in a list of candidates that are built based on a predetermined rule in the encoder and the decoder alike. Motion compensation then derives a predictor of the current block as a translation of a block co-located with the current block in the reference frame to a reference block in the reference frame, i.e., by the motion vector. The inter prediction unit 144 outputs a prediction block 145 for the current block. Here, the prediction block 145 is the one that minimizes a cost function. For example, the cost function may be the difference between the current block to be coded and its prediction block, i.e., the cost function minimizes the residual block 105. The minimization of the residual block is based, for example, on the calculation of the sum of absolute differences (SAD) between all pixels (samples) of the current block and a candidate block in a candidate reference picture. In general, any other similarity metric can be used, such as the mean square error (MSE) or the structural similarity metric (SSIM).

[0051] The cost function may also be the number of bits required to code such an inter block and / or the distortion resulting from such coding. Thus, a rate-distortion optimization procedure may be used to decide on motion vector selection and / or in general on encoding parameters such as whether to use inter or intra prediction for a block and with what settings.

[0052] The intra estimation unit 152 and the inter prediction unit 154 receive as input a picture block 101 of the current frame or picture to be intra coded and one or more reference samples 117 from an already reconstructed area of ​​the current frame. Intra prediction then describes the pixels of the current block of the current frame using a function of the reference samples of the current frame. The intra prediction unit 154 outputs a prediction block for the current block, where the prediction block is advantageously one that minimizes the difference between the current block to be coded and its prediction block, i.e. minimizes the residual block. The minimization of the residual block can for example be based on a rate-distortion optimization procedure. In particular, the prediction block is obtained as a directional interpolation of the reference samples. The direction may be determined by rate-distortion optimization and / or by calculating a similarity measure as described above in connection with inter prediction.

[0053] The inter estimation unit 142 receives as input a block or a more universal-formed image sample of the current frame or picture to be inter coded and two or more already decoded pictures 231. Inter prediction then describes the current image sample of the current frame with a motion vector to a reference image sample of a reference picture. The inter prediction unit 144 outputs one or more motion vectors 145 for the current image sample, and the reference image samples pointed to by the motion vectors are advantageously those that minimize the difference between the current image sample to be coded and its reference image sample, i.e. minimize the residual image sample. A predictor for the current block is then provided by the inter prediction unit 144 for differential coding.

[0054] The difference between the current block and its prediction, i.e., residual block 105, is then transformed by transform unit 106 to generate transformed coefficients 107. The transformed coefficients 107 are quantized by quantization unit 108 and entropy coded by entropy coding unit 170. The encoded picture data 171 thus generated, i.e., encoded video bitstream, includes intra-coded and inter-coded blocks and corresponding signaling information (such as mode indication, motion vector indication, and / or intra prediction direction). Transform unit 106 may apply a linear transform, such as a discrete Fourier transform, a fast Fourier transform, or a discrete cosine transform (DFT / FFT or DCT). Transformation into the spatial frequency domain offers the advantage that the resulting coefficients 107 typically have higher values ​​at lower frequencies. Thus, after effective coefficient scanning (e.g., zigzag scanning) and quantization, the resulting sequence of values ​​typically has some larger values ​​at the beginning and a sequence of zeros at the end. This allows for more efficient encoding. Quantization unit 108 performs the actual lossy compression by reducing the resolution of the coefficient values. Entropy coding unit 170 then assigns binary codewords to the coefficient values ​​to generate a bitstream. Entropy coding unit 170 also codes signaling information (not shown in FIG. 1).

[0055] FIG. 2 shows a video decoder 200. The video decoder 200 includes a decoded picture buffer 230, an inter prediction unit 244, and an intra prediction unit 254, which is a block prediction unit. The decoded picture buffer 230 is configured to store at least one (in the case of uni-prediction) or at least two (in the case of bi-prediction) reference frames reconstructed from an encoded video bitstream, the reference frames being different from a current frame (the frame currently being decoded) of the encoded video bitstream. The intra prediction unit 254 is configured to generate a prediction block, which is an estimate of a block to be decoded. The intra prediction unit 254 is configured to generate this prediction based on reference samples obtained from the decoded picture buffer 230.

[0056] The decoder 200 is configured to decode the encoded video bitstream generated by the video encoder 100, and preferably both the decoder 200 and the encoder 100 generate identical predictions for each block to be encoded / decoded. The features of the decoded picture buffer 230 and the intra prediction unit 254 are similar to the features of the decoded picture buffer 130 and the intra prediction unit 154 of FIG.

[0057] The video decoder 200 further includes units that are also present in the video encoder 100, such as, for example, an inverse quantization unit 210, an inverse transform unit 212, and a loop filtering unit 220, which correspond to the inverse quantization unit 110, the inverse transform unit 112, and the loop filtering unit 120, respectively, of the video encoder 100.

[0058] The entropy decoding unit 204 is configured to decode the received encoded video bitstream and obtain corresponding quantized residual transform coefficients 209 and signaling information. The quantized residual transform coefficients 209 are provided to an inverse quantization unit 210 and an inverse transform unit 212 to generate a residual (inverse transformed) block. The residual block is added to a prediction block 265 in a reconstruction unit 214, and the sum is provided to a loop filtering unit 220 to obtain a decoded video. The frames of the decoded video are stored in a decoded picture buffer 230 and can serve as a decoded picture 231 for inter prediction.

[0059] In general, the intra prediction units 154 and 254 in Figures 1 and 2 can use reference samples from already encoded regions to generate a prediction signal for a block that needs to be encoded or decoded.

[0060] The entropy decoding unit 204 receives as its input the encoded bitstream 171. In general, the bitstream is first parsed, i.e., signaling parameters and residuals are extracted from the bitstream. Typically, the syntax and semantics of the bitstream are defined by a standard so that the encoder and decoder can function in an interoperable manner. As explained in the background section above, the encoded bitstream does not only contain prediction residuals. In case of motion compensated prediction, a motion vector indication is also coded in the bitstream and parsed from the bitstream in the decoder. The motion vector indication may be given by the reference picture to which the motion vector is provided and by the motion vector coordinates. So far, it has been considered to code the complete motion vector. However, in the bitstream, only the difference between the current motion vector and the previous motion vector may be encoded. This approach allows exploiting redundancy between the motion vectors of neighboring blocks.

[0061] To efficiently code reference pictures, the H.265 codec (ITU-T, H265, Series H: Audiovisual and Multimedia Systems: High Efficiency Video Coding) provides a list of reference pictures, which assigns each reference frame to a list index. The reference frames are then signaled in the bitstream by including the corresponding assigned list index in the bitstream. Such lists may be defined in the standard or signaled at the beginning of a video or a set of several frames. Note that in H.265, two lists of reference pictures are defined, called reference lists L0 and L1. A reference picture is then signaled in the bitstream by indicating the reference list (L0 or L1) and the index in that list that is associated with the desired reference picture. Providing more than one list may be advantageous for better compression. For example, reference list L0 may be used for both unidirectionally inter-predicted slices and bidirectionally inter-predicted slices, while reference list L1 may be used only for bidirectionally inter-predicted slices. In general, however, the present disclosure is not limited to any content of lists L0 and L1.

[0062] The reference lists L0 and L1 may be defined and fixed in the standard, but by signaling them at the beginning of the video sequence, greater flexibility in encoding / decoding may be achieved. Thus, the encoder may configure the lists L0 and L1 with the specific reference pictures ordered according to an index. The reference lists L0 and L1 may have the same fixed size. In general, there may be more than two lists. The motion vectors may be signaled directly by their coordinates in the reference pictures. Alternatively, as also specified in H.265, a list of candidate motion vectors may be constructed, in which an index associated with a specific motion vector can be transmitted.

[0063] The motion vector of the current block is usually correlated with the motion vectors of neighboring blocks in the current picture or in previously coded pictures. This is because the neighboring blocks are more likely to correspond to the same moving object with similar motion and the object motion is less likely to change rapidly over time. As a result, using the motion vectors in the neighboring blocks as predictors reduces the size of the signaled motion vector difference. Motion Vector Predictors (MVPs) are usually derived from already encoded / decoded motion vectors from spatially neighboring blocks or from temporally neighboring blocks in co-located pictures. In H.264 / AVC, this is done by performing a component-wise median of three spatially neighboring motion vectors. Using this approach, no predictor signaling is required. Temporal MVPs from co-located pictures are only considered in the so-called temporal direct mode of H.264 / AVC. H.264 / AVC direct mode is also used to derive motion data other than motion vectors. Thus, they are more relevant to the block merging concept in HEVC. In HEVC, the approach of implicitly deriving an MVP has been replaced by a technique known as motion vector competition, which explicitly signals which MVP from a list of MVPs is used for motion vector derivation. As a result of the variable coding quadtree block structure in HEVC, a block can have several neighboring blocks with motion vectors as potential MVP candidates. Taking the left neighborhood as an example, in the worst case, a 64x64 luma prediction block could have 16 4x4 luma prediction blocks to its left if a 64x64 luma coding tree block is not further split and its left one is split to the maximum depth.

[0064] Advanced Motion Vector Prediction (AMVP) was introduced to modify the motion vector conflicts to take such flexible block structures into account. During the development of HEVC, the initial AMVP design was significantly simplified to provide a good tradeoff between coding efficiency and implementation-friendly design. The initial design of AMVP included five MVPs from three different classes of predictors: three motion vectors from spatial neighborhoods, the median of the three spatial predictors, and a scaled motion vector from a co-located temporal neighboring block. Furthermore, the list of predictors was modified by reordering it to place the most likely motion predictor in the first position and removing redundant candidates to ensure minimal signaling overhead. The final design of the AMVP candidate list construction includes two MVP candidates: a) up to two spatial candidate MVPs derived from five spatially neighboring blocks; b) one temporal candidate MVP derived from two temporally co-located blocks if both spatial candidate MVPs are not available or they are identical; and c) a zero motion vector if spatial, temporal or both candidates are not available. More details regarding motion vector determination can be found in the book High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014, by V. Sze et al (eds.), which is incorporated herein by reference, especially Chapter 5.

[0065] To further improve the motion vector estimation without further increasing the signaling overhead, it may be beneficial to further refine the motion vectors derived at the encoder side and provided in the bitstream. The motion vector refinement may be performed in the decoder without assistance from the encoder. The encoder may use the same refinement to obtain the corresponding motion vectors in its decoder loop. The motion vector refinement is performed in a search space that includes integer and fractional pixel locations of the reference picture. For example, the fractional pixel locations may be half pixel locations, quarter pixel locations, or other fractional locations. The fractional pixel locations may be obtained from the integer (whole pixel) locations by interpolation, such as bilinear interpolation.

[0066] In bi-prediction of the current block, two prediction blocks obtained using the first motion vector of reference list L0 and the second motion vector of reference list L1, respectively, are combined into a single prediction signal, which can provide a better match to the original signal than uni-prediction, resulting in smaller residual information and potentially more efficient compression.

[0067] At the decoder, the current block is not available since it is being decoded, but a template is used for motion vector refinement: a template is an estimate of the current block and is constructed based on the image parts that have already been processed (i.e., coded at the encoder side and decoded at the decoder side).

[0068] First, an estimate of a first motion vector MV0 and an estimate of a second motion vector MV1 are received as input in the decoder 200. In the encoder 100, the motion vector estimates MV0 and MV1 can be obtained by block matching and / or by searching in a list of candidates (e.g. a merge list) formed by the motion vectors of blocks in the neighborhood of the current block (in the same picture or in an adjacent picture). MV0 and MV1 are then advantageously signaled to the decoder side in the bitstream. However, it is noted that in general, the first decision stage in the encoder could also be performed by template matching, which offered the advantage of reducing the signaling overhead.

[0069] In the decoder 200, the motion vectors MV0 and MV1 are advantageously obtained based on information in the bitstream. The motion vectors MV0 and MV1 are directly signaled or differentially signaled and / or an index in a list of motion vectors (merge list) is signaled. However, the present disclosure is not limited to signaling the motion vectors in the bitstream. Rather, according to the present disclosure, the motion vectors may be determined by template matching already in the first stage of motion vector estimation, corresponding to the operation of the encoder. The template matching of the first stage (motion vector derivation) may be performed based on a search space different from the search space of the second motion vector refinement stage. In particular, the motion vector refinement may be performed on a search space with a higher resolution (i.e., a shorter distance between search positions).

[0070] An indication of two reference pictures RefPic0 and RefPic1 to which MV0 and MV1 respectively point is also provided to the decoder. The reference pictures are stored in the decoded picture buffers in the encoder and the decoder as a result of previous processing, i.e., encoding and decoding, respectively. One of these reference pictures is selected for motion vector refinement by searching. A reference picture selection unit of the device for determining motion vectors is configured to select a first reference picture to which MV0 points and a second reference picture to which MV1 points. Following the selection, the reference picture selection unit determines whether the first reference picture or the second reference picture is used to perform the motion vector refinement. To perform the motion vector refinement, a search area in the first reference picture is defined around the candidate location to which the motion vector MV0 points. Candidate search space locations in the search area are analyzed to find the block most similar to the template block by performing template matching in the search space and determining a similarity metric, such as the sum of absolute difference (SAD). The location of the search space indicates where the top left corner of the template is to be matched. As noted above, the top left corner is merely a convention and any point in the search space, such as the center point, may generally be used to indicate the match location.

[0071] Figure 4A shows an alternative template matching that can also be applied to uniprediction. More details can be found in document JVET-A1001, in particular in the document entitled "Algorithm Description of Joint Exploration Test Model 1" by Jianle Chen et. al., accessible at http: / / phenix.it-sudparis.eu / jvet / , section 2.4.6. "Pattern matched motion vector derivation". The template in this template matching approach is determined as the samples adjacent to the current block in the current frame. As shown in Figure 1 of document JVET-A1001, already reconstructed samples adjacent to the top and left borders of the current block are taken, which can be called "L-shaped template".

[0072] According to document JVET-D0029, which is incorporated by reference, the decoder-side motion vector refinement (DMVR) has as input initial motion vectors MV0 and MV1 pointing to two respective reference pictures RefPict0 and RefPict1. These initial motion vectors are used to determine respective search spaces within RefPict0 and RefPict1. Furthermore, using the motion vectors MV0 and MV1, a template is constructed based on the respective blocks A and B (of samples) pointed to by MV0 and MV1 as follows: Template=function(Block A, Block B)

[0073] The function may be a sample clipping operation combined with a sample-wise weighted summation. The template is then used to perform template matching in a search space determined based on MV0 and MV1 in the respective reference pictures Refpic0 and Refpic1. The cost function for determining the best template match in the respective search space is SAD(Template, Block candA'), where block candA' is the candidate coding block pointed to by the candidate MV in the search space spanning the position given by MV0. Figure 3 illustrates the determination of the best matching block A' and the resulting refined motion vector MV0'. Correspondingly, the same template is used to find the best matching block B' and the corresponding motion vector MV1' pointing to block B', as shown in Figure 3. In other words, after the template is constructed based on blocks A and B pointed to by the initial motion vectors MV0 and MV1, the refined motion vectors MV0' and MV1' are found via a search on RefPic0 and RefPic1 using the template.

[0074] The motion vector derivation technique is sometimes also called frame rate up-conversion (FRUC). The initial motion vectors MV0 and MV1 may generally be indicated in the bitstream to ensure that the encoder and decoder can use the same initial point for motion vector refinement. Alternatively, the initial motion vector may be obtained by providing a list of initial candidates including one or more initial candidates. For each of them, a refined motion vector is determined, and finally, the refined motion vector with the lowest cost function is selected.

[0075] As mentioned above, the template matching motion vector derivation mode is a special merge mode based on the Frame Rate Up-Conversion (FRUC) technique. In this mode, the motion information of a block is derived on the decoder side. According to the specific implementation described in document JVET-A1001 ("Algorithm Description of Joint Exploration Test Model 1" accessible at http: / / phenix.it-sudparis.eu / jvet / ), if the merge flag is true, a FRUC flag is signaled for a CU or PU. If the FRUC flag is false, a merge index is signaled and the normal merge mode is used. If the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) should be used to derive the motion information for that block.

[0076] In summary, during the motion vector derivation process, first, an initial motion vector is derived for the entire prediction unit (PU) based on bilateral matching or template matching. First, a list of MV candidates is generated, which can be, for example, a merge list of PUs. The list is checked, and the candidate leading to the minimum matching cost is selected as the starting point (initial motion vector). Then, a local search based on bilateral matching or template matching around the starting point is performed, and the motion vector(s) (MV) that results in the minimum matching cost is taken as the MV for that PU. Then, the motion information is further refined with the derived PU motion vector as the starting point. The terms prediction unit (PU) and coding unit (coding unit) can be used interchangeably in this specification to describe a block of samples in a picture (frame).

[0077] As shown in FIG. 4B, bilateral matching (described in document JVET-A1001) is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Under the assumption of continuous motion trajectories, the motion vectors MV0 and MV1 pointing to those two reference blocks are proportional to the temporal distance between the current picture and the two reference pictures, i.e., TD0 and TD1. Thus, in an embodiment of the present disclosure, for each tested candidate vector pair, the two respective vectors lie on a straight line in the image plane. As a special case, when the current picture is temporally between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, the bilateral matching becomes a mirror-based bidirectional MV.

[0078] As shown in Figure 4A, template matching (described in document JVET-A1001) is used to derive the motion information of the current CU by finding the closest match between a template in the current picture (neighboring blocks above and / or to the left of the current CU) and a block in the reference picture (of the same size as the template). The section "Pattern matched motion vector derivation" in document JVET-A1001 describes specific implementations of the template matching and bilateral matching methods. One example discloses that the bilateral matching operation is applied only if the "merge flag" is true, indicating that the "block merge" operation mode is selected. Here, the authors of document JVET-A1001 refer to the "merge mode" of the H.265 standard. It is noted that the template matching and bilateral matching methods described in JVET-A1001 can also be applied to other video coding standards, with variations in specific implementations.

[0079] Figure 5 is a flow diagram illustrating the decoder-side motion vector refinement (DMVR) operation. According to document JVET-D0029, DMVR is applied under two conditions: 1) the prediction type is set to skip mode or merge mode, and 2) the prediction mode is bi-predictive. First, the initial motion vectors MV0 (of reference list L0) and MV1 (of reference list L1) are derived. The derivation process is performed according to the respective skip and merge operations. Here, the authors of document JVET-D0029 refer to the skip and merge modes of the H.265 standard. A description of these modes can be found in section 5.2.2.3 "Merge Motion Data Signaling and Skip Mode" of the book High Efficiency Video Coding (HEVC), Algorithms and Architectures, 2014 by v. Sze, M. Budagavi and GJ Sullivan (eds.). In H.265, when skip mode is used, it is indicated for a block that motion data is estimated instead of being explicitly signaled and that the prediction residual is zero, i.e., no transform coefficients are transmitted. When merge mode is selected, motion data is also estimated, but the prediction residual is not zero, i.e., the transform coefficients are explicitly signaled.

[0080] A parsing index is parsed (510) from the input video stream. The parsed index points to the best motion vector candidate of the MV candidate list that is constructed (520). The best motion vector candidate is then selected (530) and a template is obtained by weighted averaging (540). The DMVR (550) is applied as follows: A block template is calculated by adding together the blocks referenced by MV0 and MV1, as explained above with reference to FIG. 3. Clipping is then performed. The template is used to find a refined motion vector MV0' around the initial motion vector MV0. The search area is at integer pixel resolution (points in the search space are an integer sample distance apart from each other). A sum of absolute differences (SAD) cost metric is used to compare the template block with the new block pointed to by MV0'. The template is used to find a refined MV0" around MV0'. The search area is at half-pixel resolution (points in the search space are half the sample distance apart from each other). The terms "picture element" and "pixel" are used interchangeably herein. The same cost metric is used. The last two steps are repeated to find MV1". A new bi-predicted block is formed by adding together the blocks pointed to by MV0" and MV1". The blocks block_A' and block_B' pointed to by such refined motion vectors MV0" and MV1" are then averaged, e.g., weighted averaged (560), to obtain the final prediction.

[0081] FIG. 6 is a schematic diagram illustrating an example of local illumination compensation (LIC) that may be used in video encoding and decoding according to an embodiment of the present disclosure. Local illumination compensation (LIC) is based on a linear model for illumination changes using a scaling factor "a" and an offset "b". LIC may be adaptively enabled or disabled for each inter-mode coded coding unit (CU). When LIC is applied to a CU, a least square error method may be used to derive parameters a and b using neighboring samples of the current CU and its corresponding reference samples. More specifically, as shown in FIG. 6, subsampled (2:1 subsampled) neighboring samples of the CU and corresponding samples in the reference picture (identified by motion information of the current CU or sub-CU) are used. LIC parameters are derived and applied separately for each prediction direction. Here, subsampling 2:1 means that every other pixel on the current CU boundary and the reference block is taken. Further details on the use of scaling factors / multiplicative weighting factors and offsets can be found in document JVET-A1001, section "2.4.4. Local illumination compensation".

[0082] 7 is a schematic diagram showing a decoder-side motion vector refinement (DMVR) iteration process performed on a reference picture RefPic0. The current picture includes a current block 710, for which a motion vector MV0' should be found based on the motion vector MV0 in RefPic0. A search space containing five integer positions is determined, and the blocks pointed to by the candidate positions are called Ax. The output is the best match among the blocks Ax pointed to by the motion vector MV0'.

[0083] Whenever an explicit merge mode index is signaled, the decoder-side motion vector refinement starts from the reference index and motion vector(s) that are normatively estimated from the signaled index. When no explicit merge mode index is signaled, a set of initial motion vector candidates is evaluated at the decoder using a cost function, and the candidate with the lowest cost is selected as the starting point for refinement. Thus, regardless of whether the decoder-side motion vector derivation method is based on predicted / reconstructed neighboring block boundary samples (commonly referred to as Template Matching (TM); see FIG. 4A), (or) based on bilateral matching through minimizing the difference between corresponding patches in reference list L0 and reference list L1 (commonly referred to as Bilateral Matching (BM) cost), (or) based on the difference between averaged versions of corresponding patches in reference list L0 and reference list L1 and the displacement in L0 / L1 (referred to as DMVR cost), there is a refinement search that needs to be performed around the starting points, which may be sub-pixel accurate motion vectors.

[0084] To evaluate the cost function, an interpolation needs to be performed to derive a value at the center of sub-pixel accuracy based on the reference frame values ​​at integer grid locations. The interpolation filter can be simple, such as a bilinear interpolation filter, or it can be a longer filter, such as a 2D DCT-based separable interpolation filter. To reduce the complexity of iterating at each location considered during refinement and deriving an interpolated sample for a block, an integer-pixel distance grid of refinement points centered at a sub-pixel accurate location or locations in L0 and / or L1 was proposed in another invention. In this case, only incremental interpolation needs to be performed, since new locations close to the current best-cost location are considered. After the refinement of the integer-pixel distance grid is completed, the best integer delta MV is obtained with respect to the merge MV.

[0085] To further improve the compression gain, sub-pixel distance refinement can be performed. Half-pixel distance refinement requires an interpolated sample at half-pixel distance from the best integer distance MV position(s) in the reference frame(s). It is possible to perform sub-pixel accurate refinement jointly between L0 and L1. In this case, the displacement in L0 with respect to the best integer distance MV position in L0 is inverted to obtain the corresponding displacement in L1 with respect to the best integer distance MV position in L1. It is also possible to perform sub-pixel accurate refinement independently in L0 and L1.

[0086] 8 is a conceptual diagram illustrating an example of integer and fractional sample locations according to an embodiment of the present disclosure. With reference to FIG. 8, pixel locations designated as "A" are integer pixels, half pixel locations are designated as b, h, j, and all quarter pixel locations are designated as a, c, d, e, f, g, I, k, n, p, q, and r. To perform half pixel refinement on an 8-point square pattern, three planes are required, and similarly, for quarter pixel refinement, eight planes are required by the HEVC standard. Newer standards allow for interpolation to 1 / 16 pixel accuracy.

[0087] If "A" is an integer pixel, then pixel locations a, b and c only require horizontal interpolation, d, h and n only require vertical interpolation, and all other points require both vertical and horizontal interpolation.

[0088] During integer pixel distance refinement and sub-pixel distance refinement, it is possible to use an interpolation procedure different from that used for the final motion compensated prediction process. For example, bilinear interpolation is a simpler interpolation that can be used for refinement, while the final motion compensated prediction (MCP) may require a 2D DCT-based interpolation filter. The integer pixel grid data retrieved from the reference frame buffer needs to be kept in a first buffer in internal memory (e.g., SRAM) until all the interpolations can be completed to avoid retrieving this data multiple times from external memory (e.g., DDR). The integer pixel distance grid derived through the interpolation of integer pixel grid samples and required for the integer pixel distance refinement from the refinement center with sub-pixel accuracy needs to be kept in a second buffer in internal memory until all the integer pixel distance refinements are completed. For half-pixel distance refinements around the best integer pixel distance location (which can be a sub-pixel location relative to the integer pixel grid), considering that the distance between symmetric half-pixel refinement points on either side of the center is spaced one integer pixel distance apart, two horizontal half-pixel distance locations share an interpolated plane, two vertical half-pixel distance locations share an interpolated plane, and four diagonal half-pixel distance locations share an interpolated plane.

[0089] If the interpolation for the refinement is the same as the final MCP interpolation, it may be preferable to keep the integer-pixel distance refinement data stored in the second buffer, since this allows the final MCP interpolation calculation to be avoided when the best integer-pixel distance position turns out to be the best position after sub-pixel refinement. If the interpolation differs between the refinement and the final MCP, the second buffer can be overwritten with the interpolation for one of the three planes required for the half-pixel distance refinement.

[0090] To reduce computational load and internal memory needs, some prior art techniques evaluate only the four diagonal half-pixel distance positions, while others evaluate only the horizontal and vertical half-pixel distance positions. Thus, the set of internal memory buffers required ranges from 2 (if only the diagonal half-pixel distance positions are evaluated and the interpolation is different) to 5 (if evaluation is performed for all three half-pixel distance planes and the interpolation is not different). Some prior art techniques perform additional 1 / 4-pixel distance and 1 / 8-pixel distance refinements, but each position in these refinements requires an individual interpolation, which is computationally prohibitive and not worth the additional compression gain. If the interpolation uses a higher tap filter, the horizontal filtering is performed first, the filtered result is stored in a temporary buffer, and then the vertical filtering is performed.

[0091] Embodiments of the present disclosure provide methods and apparatus for deriving sub-pixel distance delta motion vectors around the best integer distance motion vector during decoder-side motion vector refinement / derivation without explicit refinement through an error surface technique that utilizes costs evaluated at integer-pixel distance grid locations. A parametric error surface is fitted using these costs, and the location of the minimum of the parametric error surface is obtained by solving a system of equations. Aspects of the present disclosure utilize decoder-side sub-pixel distance refinement around integer-distance pixel locations (which themselves may be sub-pixel locations with respect to the integer pixel grid) and address various variants of decoder-side motion vector refinement such as template matching, bilateral matching, and template-based bilateral matching.

[0092] Given that decoder-side motion vector refinement / derivation is a normative aspect of the encoding system, the encoder would also have to perform the same error plane techniques to avoid drift between the encoder's reconstruction and the decoder's reconstruction. Thus, all aspects of all embodiments of this disclosure are applicable to both encoding and decoding systems.

[0093] In template matching, refinement motion occurs only in references starting from the center of the sub-pixel accuracy, based on explicitly signaled merging indices or derived implicitly through cost evaluation.

[0094] In bilateral matching, refinement (with or without averaged templates) starts on reference lists L0 and L1, starting from their respective centers of sub-pixel accuracy, based on explicitly signaled merging indices or derived implicitly through cost evaluation.

[0095] Where the bilateral matching cost is evaluated based on equal and opposite horizontal and vertical displacements in reference list L1 for a given horizontal and vertical displacement in reference list L0, the positions shown in the figures are assumed to correspond to reference list L0, and the positions in reference list L1 are derived by inverting the horizontal and vertical displacements relative to the current iteration center in reference list L1.

[0096] Thus, there is a need to achieve most of the coding gain of explicit sub-pixel distance refinement without increasing memory size and computational complexity.

[0097] EMBODIMENT 1

[0098] Let N be the maximum number of integer 1-pixel distance refinement iterations that are normatively allowed. The first integer distance refinement iteration starts from the aforementioned sub-pixel accurate refinement center and proceeds until the center position has an estimated cost function value that is lower than the estimated cost function values ​​of the defined set of 1-pixel neighbors or N iterations have been performed. If the number of iterations N is reached and the center position is not the position with the minimum (lowest) cost function value when compared to its set of 1-pixel neighbors, no error surface based sub-pixel refinement is performed and the position with the minimum (lowest) cost function value across all iterations is declared as the final delta motion vector from the decoder-side motion vector refinement process. If the refinement ends with the center position with the minimum (lowest) cost function value in the set of 1-pixel neighbors of the ending iteration, the next error surface based sub-pixel refinement procedure is applied.

[0099] In the following, a set of 1-pixel neighborhood locations around a center C is defined as integer 1-pixel distances to its left (L), top (T), right (R), and bottom (B). FIG. 9 is a block diagram of a set of 1-pixel neighborhood locations around a center pixel to obtain sub-pixel accurate delta motion vector refinement in one or more reference frames, according to an embodiment of the present disclosure. With reference to FIG. 9, the origin of the 2D coordinate system is at C, and the coordinates at L, T, R, B correspond to (-1,0), (0,1), (1,0), and (0,-1). The evaluated cost function values ​​at these five locations would be E(0,0), E(-1,0), E(0,1), E(1,0), and E(0,-1). A 5-parameter error surface using a 2D parabolic function is defined as follows: E(x,y)=A*(x-x0) 2 +B*(y-y0) 2 +C (1) where E(x,y) is the evaluated cost function of the Cartesian coordinate (x,y), (x0,y0) is the Cartesian coordinate associated with the sub-pixel displacement with respect to the center (0,0) that has the smallest error, C is a parameter corresponding to the error at this position, and A and B are the error surface model parameters. A, B, C are constant values ​​for calculating E(x,y).

[0100] On the other hand, considering that A, B, C, x0, y0 are unknown parameters, by solving the five equations with the available evaluated cost function values ​​at the five positions (0,0), (-1,0), (0,-1), (1,0) and (0,1), (x0,y0) can be calculated (determined) as follows: x0=(E(-1,0)-E(1,0)) / (2*(E(-1,0)+E(1,0)-2*E(0,0))) (2) y0=(E(0,-1)-E(0,1)) / (2*(E(0,-1)+E(0,1)-2* E(0,0))) (3) x0 and y0 are coordinates associated with sub-pixel motion vector displacements. E(-1,0), E(1,0), E(0,0), E(0,-1) and E(0,1) are integer distance costs corresponding to candidate integer motion vector displacements (-1,0), (1,0), (0,0), (0,-1) with respect to the initial motion vector, respectively. The coordinates x0 and y0 can be scaled by a scale factor N depending on the sub-pixel precision of the delta motion vector, where N=2, 4, 8 and 16, for example, pixel precision ½, ¼, ⅛ and ⅛, respectively.

[0101] In another embodiment, formulas (2) and (3) can be represented as follows: x0=(E(-1,0)-E(1,0)) / (2*N*(E(-1,0)+E(1,0)-2*E(0,0))) (2') y0=(E(0,-1)-E(0,1)) / (2*N*(E(0,-1)+E(0,1)-2*E(0,0))) (3') Here, for example, N=1, 2, 4, and 8 for pixel precisions 1 / 2, 1 / 4, 1 / 8, and 1 / 16, respectively.

[0102] Note that based on equations (2) and (3), x0 can be calculated (determined) only according to the positions (-1,0), (1,0) and (0,0). Similarly, y0 can be calculated (determined) only according to the positions (0,-1), (0,1) and (0,0).

[0103] Also note that the parametric error surface cannot be fitted if the evaluated costs at all four positions (-1,0), (0,-1), (1,0), and (0,1) relative to the center (0,0), which correspond to the best integer distance displacement for the initial motion vector, are not available. In such cases, the sub-pixel accurate delta displacement is not added to the best integer distance displacement.

[0104] Depending on the sub-pixel accuracy supported by the final motion compensation, the scale factor N can be appropriately selected.

[0105] Sub-pixel motion vector displacements are limited to -0.5 and +0.5 in the x and y directions by equations (1), (2), and (3). Since we are usually interested in accuracy down to 1 / 16 pixel, the two divisions above can be efficiently performed with only shift, compare, and increment operations.

[0106] In some embodiments, a method for decoder-side motion vector refinement includes determining a target integer motion vector displacement by comparing integer distance costs corresponding to candidate integer motion vector displacements with respect to an initial motion vector, determining sub-pixel motion vector displacements by performing calculations on the integer distance costs, and determining a refined motion vector based on the target integer motion vector displacements, the sub-pixel motion vector displacements and the initial motion vector.

[0107] In one embodiment, the method may further include, prior to determining the sub-pixel motion vector displacement, determining whether the predefined motion vector displacement includes a target integer motion vector displacement; if the predefined motion vector displacement includes the target integer motion vector displacement, determining the sub-pixel motion vector displacement by performing calculations with respect to an integer distance cost. In another embodiment, the method may further include, prior to determining the sub-pixel motion vector displacement, determining whether evaluated costs are available at positions (-1,0), (0,-1), (1,0) and (0,1) for the target integer motion vector displacement; if it is determined that evaluated costs are available at positions (-1,0), (0,-1), (1,0) and (0,1) for the target integer motion vector displacement, determining the sub-pixel motion vector displacement by performing calculations with respect to an integer distance cost.

[0108] In one embodiment, the method may further include calculating a refined motion vector based on the target integer motion vector displacement and the initial motion vector if the predetermined motion vector displacement does not include the target integer motion vector displacement. In another embodiment, the method may further include calculating a refined motion vector based on the target integer motion vector displacement and the initial motion vector if it is determined that at least one or more of the evaluated costs are not available at positions (-1,0), (0,-1), (0,1), and (1,0) for the target integer motion vector displacement.

[0109] In one embodiment, determining the target integer motion vector displacement may include calculating an integer distance cost for each candidate integer motion vector displacement and selecting the candidate integer motion vector displacement corresponding to the lowest integer distance cost as the target integer motion vector displacement.

[0110] In one embodiment, the target integer motion vector displacement includes a first motion vector displacement corresponding to the reference picture list L0 and a second motion vector displacement corresponding to the reference picture list L1. The method may further include determining the first motion vector displacement by comparing integer costs corresponding to the candidate integer motion vector displacements corresponding to the reference picture list L0 and determining the second motion vector displacement by inverting the first motion vector displacement.

[0111] It should be understood that the described embodiments are exemplary and not limiting. If the integer distance refinement process results in these many evaluated cost function values ​​within a one-pixel neighborhood set, other 5-point, 6-point, and 9-point based error surface methods such as those in the referenced documents can be used.

[0112] The method of this embodiment only requires two internal memory buffers: a first buffer configured to store the integer pixel grid from the reconstructed picture buffer, and a second buffer configured to store the integer distance grid during integer distance refinement (which is utilized to obtain a parametric error surface) and can be used to overwrite with the final sub-pixel accurate motion compensated prediction samples.

[0113] According to embodiments of the present disclosure, highly accurate sub-pixel accurate delta motion vectors are obtained without explicit refinement at any of the sub-pixel accuracy levels.

[0114] If the same cost function is used throughout all iterations, then no additional interpolation or cost function evaluation calculations are required beyond those required for the integer distance refinement iterations.

[0115] 10 is a simplified flow diagram illustrating a method 1000 for obtaining sub-pixel accurate delta motion vectors in one or more reference frames in a decoder-side motion vector refinement system according to an embodiment of the present disclosure. The method 1000 may include the following steps.

[0116] Step 1001: Provide a processor, which may be one or more processing units (CPU, DSP) integrated in a video encoder and / or decoder or program code integrated in video compression software to perform the methods described herein.

[0117] Step 1003: Begin a loop by sequentially performing integer one-pixel distance refinement operations (iterative steps) by the processor using a cost function to determine an integer distance refinement motion vector for each reference frame of the one or more reference frames.

[0118] Step 1005: The processor determines a cost function value of the location of the search center of the current operation relative to the cost function values ​​of the set of 1-pixel neighboring locations. If the cost of the search center of the current iteration is determined to be the lowest, i.e., early termination from the iterative loop (1005, yes):

[0119] Step 1011: determine a sub-pixel distance refined motion vector around the last search center in each reference frame by calculating the position having a minimum value on a parametric error surface fitted using cost function values ​​of the last search center and a set of 1-pixel neighborhood positions around it;

[0120] Step 1013: For each reference frame, return a total refined motion vector as the sum of the determined integer distance refined motion vector and the determined sub-pixel distance refined motion vector.

[0121] The method further includes: if the cost function value of the position of the search center is not the lowest cost function value (1005, no), proceeding to step (1007).

[0122] Step (1007): Determine whether the current motion is the last motion (iteration). If the current motion is the last motion (1007, yes), return the refined motion vector corresponding to the position with the lowest cost value through all motions (1015). If the current motion is not the last motion (1007, no), then Movement of The center is updated to the position with the lowest cost in the calculation and the process returns to the loop and iteration step (1003).

[0123] EMBODIMENT 2

[0124] There are various variations of embodiment 1 where the cost function evaluations at integer distance positions utilized to derive the parametric error surface differ from the cost function evaluations performed during a previous iterative step of integer distance refinement.

[0125] For example, when bilateral matching is used, three types of refinement are possible. The first refinement type is to perform a joint refinement on both reference lists L0 and L1 such that the displacement in reference list L1 is equal and opposite to the displacement in reference list L0 in both horizontal and vertical directions (the first refinement type is referred to as SBM_JOINT). The second refinement type is to perform an independent refinement on both reference lists L0 and L1 for a common bilateral averaged template (the second refinement type is referred to as TBM_INDEPENDENT). The third refinement type is to perform a refinement on reference list L0 or L1 for a bilateral averaged template and invert the estimated horizontal and vertical displacements in obtaining the displacement in the other reference list (L1 if L0 was used for refinement, L0 if L1 was used for refinement) (the third refinement type is referred to as TBM_IN_1REF_NEG_IN_OTHER).

[0126] Because it is difficult to predict at which iteration an early termination of the iterative loop will occur (because the cost of the center location is lower than the cost of the surrounding set of 1-pixel neighborhoods), whenever the cost function evaluations at the integer distance locations used to derive the parametric error surface differ from the cost function evaluations performed during the previous iteration of integer distance refinement, an additional set of cost function evaluations centered on the early termination center are performed using the required cost function.

[0127] Table 1 below gives some situations during bilateral matching where additional evaluations are required after an early termination during integer distance refinement. [Table 1]

[0128] During integer distance refinement iterations, it has been observed that refinement type SBM_JOINT performs best. When obtaining independent sub-pixel delta motion vectors in L0 and L1 using refinement type TBM_INDEPENDENT, a small additional coding gain is achieved when compared to equal and opposite displacements in L0 and L1. However, independent refinement in L0 and L1 requires cost evaluations at L0 and L1 positions independently and is therefore computationally more complex than the joint equal and opposite displacement estimation option (SBM_JOINT refinement). In addition to SBM_JOINT refinement, early stopping does not require additional cost function evaluations.

[0129] 11 is a simplified flow diagram illustrating a method 1100 for obtaining sub-pixel accurate delta motion vectors in one or more reference frames around respective initial sub-pixel accurate refinement center(s) in a decoder-side motion vector refinement system according to an embodiment of the present disclosure. The method 1100 may include the following steps.

[0130] Step 1101: Provide a processor, which may be one or more processing units (CPU, DSP) integrated in a video encoder and / or decoder or program code integrated in video compression software to perform the methods described herein.

[0131] Step 1103: An integer one-pixel distance refinement operation (an iterative process) is sequentially performed by a processor using a cost function to determine an integer distance refinement motion vector for each reference frame of the one or more reference frames.

[0132] Step 1105: Determine whether the current action is the last action. If the current action is the last action (1105, yes):

[0133] Step 1111: Perform integer 1-pixel distance refinement using a second cost function to obtain an independent sub-pixel delta motion vector for each reference frame of the one or more reference frames.

[0134] If the current action is not the last action (1105, no), proceed to step 1107.

[0135] Step 1107: Determine whether the cost function value of the search center position of the current operation is the lowest cost function value in the set of 1-pixel neighboring positions. If the cost function value of the search center position is the lowest cost function value (1107, yes), proceed to step 1111 (i.e., early termination of the iterative loop). If the cost function value of the search center position is not the lowest cost function value (1107, no), proceed to step 1109.

[0136] Step 1109: Update the center to the position with the lowest cost value for that run and loop back to perform the next integer 1-pixel distance refinement run.

[0137] Step 1111: Perform integer 1-pixel distance refinement using a second cost function to obtain an independent sub-pixel delta motion vector for each reference frame of the one or more reference frames.

[0138] Step 1113: Determine whether the last search center position has the lowest second cost function value relative to the second cost function values ​​in a set of 1-pixel neighboring positions to the last search center. If the last search center position has the lowest second cost function value (1113, yes):

[0139] Step 1115: Determine a sub-pixel distance refinement motion vector around the best integer distance refinement position in each reference frame by calculating the position with a minimum on the fitted parametric error surface using the second cost function value.

[0140] Step 1117: For each reference frame, as the sum of the determined integer distance refinement MV and the determined sub-pixel distance refinement MV, Total Wash Return the Ren MV.

[0141] The embodiments of the present disclosure also provide an apparatus configured to execute the above-mentioned method. The apparatus may be a combination of software and hardware. For example, encoding and / or decoding may be performed by a chip such as a general-purpose processor (CPU), or a digital signal processor (DSP), or a field programmable gate array (FPGA). However, the embodiments of the present disclosure are not limited to programmable hardware implementation. Some embodiments of the present disclosure may be implemented using an application specific integrated circuit (ASIC) or a combination of the above-mentioned hardware components.

[0142] The encoding and / or decoding may also be implemented by program instructions or program code stored on a computer readable medium, which when executed by a processor or a computer, causes the processor or the computer to perform the steps of the method. The computer readable medium may be any medium on which program code is stored, such as a DVD, a CD, a USB (flash) drive, a hard disk, a server storage device available over a network, etc.

[0143] FIG. 12 is a block diagram of an apparatus 1200 that can be used to implement various embodiments of the present disclosure. The apparatus 1200 may be the encoding apparatus 100 shown in FIG. 1 and the decoding apparatus 200 shown in FIG. 2. Additionally, the apparatus 1200 may host one or more of the described elements. In some embodiments, the apparatus 1200 includes one or more input / output devices, such as a speaker, microphone, mouse, touch screen, keypad, keyboard, printer, display, etc. The apparatus 1200 may include one or more central processing units (CPUs) 1210, memory 1220, mass storage device 1230, video adapter 1240, and I / O interface 1260, connected to a bus. The bus may be one or more of any type of several bus architectures, including a memory bus or memory controller, a peripheral bus, a video bus, etc.

[0144] CPU 1210 may include any type of electronic data processor. Memory 1220 may include or be any type of system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or a combination thereof. In some embodiments, memory 1220 may include ROM for use during start-up and DRAM for program and data storage for use during program execution. In some embodiments, memory 1220 may be non-transitory. Mass storage 1230 may include any type of storage device that stores data, programs, and other information and makes the data, programs, and other information accessible via a bus. Mass storage 1230 may include, for example, one or more of a solid state drive, hard disk drive, magnetic disk drive, optical disk drive, and the like configured to store program code that, when executed by CPU 1210, causes the CPU to perform the methods described herein. The CPU 1210 may be configured to iteratively perform the steps described in relation to Figures 10 and 11 over several operations. The CPU 1210 may include a reference selection unit configured to select a first reference picture to which MV0 points and a second reference picture to which MV1 points. After selecting the reference pictures, the reference picture selection unit may determine whether the first reference picture or the second reference picture is used to perform motion vector refinement.

[0145] Video adapter 1240 and I / O interface 1260 provide interfaces for coupling external input / output devices to device 1200. For example, device 1200 may provide a SQL command interface to a client. As shown, examples of input and output devices include a display 1290 coupled to video adapter 1240, and any combination of mouse / keyboard / printer 1270 coupled to I / O interface 1260. Other devices may be coupled to device 1200 and additional or fewer interface cards may be utilized. For example, a serial interface card (not shown) may be used to provide a serial interface for a printer.

[0146] The apparatus 1200 may also include one or more network interfaces 1250, including a wired link, such as an Ethernet cable, and / or a wireless link to an access node or one or more networks 1280. The network interface 1250 allows the apparatus 1200 to communicate with remote units over the network 1280. For example, the network interface 1250 may provide communication to a database. In an embodiment, the apparatus 1200 is coupled to a local area network or a wide area network for data processing and communication with remote devices, such as other processing units, the Internet, remote storage facilities, etc. The apparatus 1200 may be used to encode one or more picture blocks received from an input (e.g., a network interface) and / or decode a video picture from a bitstream. The apparatus 1200 may include a bitstream parser configured to extract compressed picture blocks from the bitstream, a motion vector refinement unit configured to obtain sub-pixel accurate delta motion vector refinement, and a reconstruction unit configured to perform block reconstruction based on the obtained motion vectors.

[0147] Although a particular feature or aspect of the present disclosure may be disclosed with respect to only one of several implementations or embodiments, such feature or aspect may be combined with one or more other features or aspects of other implementations or embodiments as may be desired or advantageous for any given or particular application. Furthermore, when the terms "comprise," "have," "have" or other variations thereof are used in either the detailed description or the claims, such terms are intended to be inclusive, similar to the term "comprise." Also, the terms "exemplary," "for example," and "example" are intended merely as examples, not best or optimal. The terms "coupled" and "connected" may be used with derivatives. It should be understood that these terms may be used to indicate that two elements cooperate or interact with each other, whether in direct physical or electrical contact or not in direct contact with each other.

[0148] Although specific aspects have been illustrated and described herein, those skilled in the art will recognize that various alternative and / or equivalent implementations may be substituted for the specific aspects illustrated and described without departing from the scope of the present disclosure. This application is intended to cover any adaptations or variations of the specific aspects discussed herein.

[0149] Although elements in the following claims are described in a particular order with corresponding labeling, the elements are not necessarily intended to be limited to being implemented in that particular order, unless the claim recitation otherwise implies a particular order for implementing some or all of the elements.

[0150] Numerous alternatives, modifications, and variations will be apparent to those skilled in the art in light of the above teachings. Of course, those skilled in the art will readily recognize that there are many applications of the present invention other than those described herein. Although the present invention has been described with reference to one or more specific embodiments, those skilled in the art will recognize that many modifications can be made thereto without departing from the scope of the present invention. Thus, it should be understood that, within the scope of the appended claims and their equivalents, the present invention may be practiced otherwise than as specifically described herein.

[0151] The circuitry described above may also be a single integrated chip, however the invention is not limited thereto and the circuitry may include different pieces or pieces of hardware or a combination of hardware and software, such as a general purpose processor or DSP programmed with corresponding code.

[0152] The above flow chart is intended to illustrate an example of a decoder-side motion vector refinement technique. Those skilled in the art may modify or combine steps to implement the present disclosure without departing from the scope of the present disclosure.

[0153] In one aspect of the present disclosure, a method is provided for obtaining sub-pixel accurate delta motion vector refinement in one or more reference frames in a decoder-side motion vector refinement system. The method may include: PROVIDE PROCESSOR(1001); beginning 1003 a looping operation by iteratively performing, by a processor, integer one-pixel distance refinement operations using a cost function to determine an integer distance refinement motion vector for each reference frame of the one or more reference frames; determining (1005) by the processor whether the cost function value of the search center location is the lowest function value in a set of 1-pixel neighborhood locations; If the cost function value of the position of the search center is the lowest cost function value (1005, yes), end the loop operation; determining (1011) a sub-pixel distance refined motion vector about the search center in each reference frame by calculating a location having a minimum on a parametric error surface fitted using cost function values ​​of the search center and a set of 1-pixel neighboring locations; For each reference frame, return (1013) the sub-pixel positions obtained using the parametric error surface.

[0154] In one embodiment, the method further comprises: if the cost function value of the location of the search center is not the lowest cost function value (1005, no); Determine whether the current action is the last action (1007); If the current action is the last action (1007, yes): Return the positions in each reference frame that have the lowest cost function (1015); If the current action is not the last action (1007, no): updating (1009), by the processor, the location of the search center of the current integer 1-pixel distance refinement operation with the location having the lowest cost in the current operation; Repeat the loop operation (1003, 1005, 1007, 1009).

[0155] In one embodiment, the parametric error surface includes five pixels located at a central pixel surrounded by four surrounding pixels spaced an equal distance from the central pixel.

[0156] In one embodiment, the cost function is calculated according to the following formula: E(x,y)=A*(x-x0) 2 +B*(y-y0) 2 +C; where E(x,y) is the evaluated cost function value for coordinate (x,y), x0 and y0 are coordinates associated with the sub-pixel displacement with respect to center (0,0) that has the minimum error, C is a parameter related to the error at coordinate (x0,y0), A and B are constant values, x and y are coordinates associated with neighboring positions, and x and y are integers inclusive of -1, 0, and 1, respectively.

[0157] In one embodiment, the sub-pixel position is given by: x0=(E(-1,0)-E(1,0)) / (2*N*(E(-1,0)+E(1,0)-2*E(0,0))); y0=(E(0,-1)-E(0,1)) / (2*N*(E(0,-1)+E(0,1)-2*E(0,0))) where E(x,y)=A*(x-x0) 2 +B*(y-y0) 2 +C; where E(x,y) is the evaluated cost function value, x0 and y0 are the coordinates associated with the sub-pixel displacement relative to the center (0,0) with minimum error, C is a parameter associated with the error at coordinate (x0,y0), A and B are constant values, x and y are coordinates associated with neighboring positions, x and y are integers including -1, 0, and 1, respectively, and N is an integer 1, 2, 4, or 8 for ½, ¼, ⅛, or 1 / 16 sub-pixel accuracy.

[0158] In one embodiment, sub-pixel accurate delta motion vector refinement is obtained by template matching.

[0159] In another aspect of the present disclosure, an apparatus is provided for obtaining sub-pixel accurate delta motion vectors in one or more reference frames in a decoder-side motion vector refinement system, the apparatus including a processing unit and a non-transitory computer-readable medium having computer-readable instructions thereon to cause the processing unit to perform the following steps: starting a loop by iteratively performing N integer one-pixel distance refinement operations using a cost function to determine an integer distance refinement motion vector for each reference frame, where the position of the search center of the current iteration is updated by the position with the lowest cost in the previous iteration; determining whether the cost of the search center location is lower than the cost of any of a set of 1-pixel neighboring locations; If the cost of the search center location is the lowest cost; terminating the loop operation; determining a sub-pixel distance refined motion vector about the search center in each reference frame by calculating a location having a minimum on a parametric error surface fitted using cost function values ​​of the search center and a set of 1-pixel neighboring locations; For each reference frame, returning the sub-pixel positions obtained using the parametric error surface.

[0160] In an embodiment, the non-transitory computer readable medium having computer readable instructions thereon further causes the processing unit to perform the following steps: If the cost function value of the position of the search center is not the lowest cost function value (1005, no): determining whether the current action is the last action; If the current motion is the last motion: returning the positions in each reference frame with the lowest cost function; If the current action is not the last action: The processor The position of the search center of the current integer 1-pixel distance refinement operation is updated (1009) with the position having the lowest cost in the current refinement operation; repeating the loop operation.

[0161] In one embodiment, the cost function is calculated according to the following formula: E(x,y)=A*(x-x0) 2 +B*(y-y0)2 +C; where E(x,y) is the evaluated cost function value, x0 and y0 are the coordinates associated with the sub-pixel displacement relative to the center (0,0) with the minimum error, C is a parameter related to the error at the coordinate (x0,y0), A and B are constant values, x and y are the coordinates associated with the neighboring positions, and x and y are integers inclusive of -1, 0, and 1, respectively.

[0162] In one embodiment, the sub-pixel position is given by: x0=(E(-1,0)-E(1,0)) / (2*N*(E(-1,0)+E(1,0)-2*E(0,0))); y0=(E(0,-1)-E(0,1)) / (2*N*(E(0,-1)+E(0,1)-2*E(0,0))) where E(x,y)=A*(x-x0) 2 +B*(y-y0) 2 +C; where E(x,y) is the evaluated cost function value, x0 and y0 are the coordinates associated with the sub-pixel displacement relative to the center (0,0) with minimum error, C is a parameter associated with the error at coordinate (x0,y0), A and B are constant values, x and y are coordinates associated with neighboring positions, x and y are integers including -1, 0, and 1, respectively, and N is an integer 1, 2, 4, or 8 for ½, ¼, ⅛, or 1 / 16 sub-pixel accuracy.

[0163] Another aspect of the present disclosure provides a method in a decoder-side motion vector refinement system for obtaining sub-pixel accurate delta motion vector refinement for one or more reference frames in reference list L0 and reference list 1. The method may include: Providing a processor (1101); beginning a looping operation by iteratively performing (1103) integer one-pixel distance refinement operations by a processor using a first cost function to determine an integer distance refinement motion vector for each reference frame of the one or more reference frames; determining whether the current action is the last action (1105); If the current action is the last action (1105, yes): performing integer 1-pixel distance refinement using a second cost function to obtain independent sub-pixel delta motion vectors for each reference frame (1111); if the current action is not the last action (1105, no); determining, by the processor, whether a cost function value of the location of the search center of the current operation is a first lowest function value in a set of 1-pixel neighborhood locations (1107); If the cost function value of the search center position is the first lowest function value (1107, yes): Ending a loop operation; performing integer 1-pixel distance refinement using a second cost function to obtain independent sub-pixel delta motion vectors for each reference frame (1111); If the cost function value of the search center position of the current operation is not the first lowest function value (1107, no): updating the center to the location with the first lowest cost of the current operation; and Repeating the loop operations (1103, 1105, 1107, 1109).

[0164] In some embodiments, the method may further include: determining, by the processor, whether a second cost function value of the search center location of the obtained independent sub-pixel delta motion vector is a second lowest function value (1113); if the second cost function value is the lowest cost function value (1113, yes); determining a sub-pixel distance refined motion vector about the search center in each reference frame by calculating a location having a minimum on a fitted parametric error surface using cost function values ​​of the search center and a set of 1-pixel neighboring locations (1115); returning a total refined motion vector as the sum of the determined integer distance refined motion vector and the determined sub-pixel distance refined motion vector for each reference frame obtained using the parametric error surface for each reference frame (1117); if the second cost function value is not the second lowest cost function value (1113, no); Returning (1119) a refined motion vector corresponding to the location in each reference frame having the second lowest cost function value.

[0165] In one embodiment, the sub-pixel positions are derived according to the following formula: x0=(E(-1,0)-E(1,0)) / (2*N*(E(-1,0)+E(1,0)-2*E(0,0))) y0=(E(0,-1)-E(0,1)) / (2*N*(E(0,-1)+E(0,1)-2* E(0,0))) Where: E(x,y)=A*(x-x0) 2 +B*(y-y0) 2 +C; (x0,y0) corresponds to the sub-pixel displacement with respect to (0,0) with minimum error, C corresponds to the error at (x0,y0), A and B are constants, (x,y) correspond to nearby positions, where x includes -1,0,1 and y includes -1,0,1.

[0166] In one embodiment, each of the first and second cost functions is calculated according to the following formula: E(x,y)=A*(x-x0) 2 +B*(y-y0) 2 +C; where E(x, y) is the evaluated cost function value, x0 and y0 are the Cartesian coordinates associated with the sub-pixel displacement relative to the center (0,0) with minimum error, C is a parameter related to the error in Cartesian coordinates (x0, y0), A and B are constant values, x and y are the Cartesian coordinates associated with the neighboring positions, and x and y are integers including -1, 0, and 1, respectively.

[0167] In one embodiment, sub-pixel accurate delta motion vector refinement is obtained by bilateral matching.

[0168] In one embodiment, the bilateral matching includes performing joint refinement on both reference list L0 and reference list L1.

[0169] In one embodiment, bilateral matching involves performing independent joint refinements on both reference list L0 and reference list L1 with respect to a common bilateral averaging template.

[0170] In one embodiment, bilateral matching involves performing refinement on either reference list L0 or reference list L1 with respect to a common bilateral average template and inverting the estimated horizontal and vertical displacements when obtaining the displacements in the other reference list.

[0171] In summary, the embodiments of the present disclosure relate to a motion vector determination based on template matching for bidirectional motion vector estimation. In particular, a block template is constructed as the average of the blocks pointed to by the initial motion vector to be refined. Then, the motion vector refinement is performed by template matching in two different reference pictures. The matching is performed by finding the optimum (minimum or maximum depending on the function) of the matching function corresponding to the best matching block in each of the two reference pictures. The optimum is searched for a zero-mean template and a zero-mean candidate block (among the block positions pointed to by the motion vector candidates in the search space). In other words, the mean of the template is subtracted from the template and the mean of the candidate block is subtracted from the candidate block before performing the function optimization. Then, the predictor of the current block is calculated as the weighted average of the best matching blocks in the respective reference pictures.

Claims

1. A decoder-side motion vector refinement method performed by a decoding device, the method comprising: determining a target integer motion vector displacement by comparing integer distance costs corresponding to candidate integer motion vector displacements with respect to an initial motion vector; determining sub-pixel motion vector displacements by performing a calculation on the integer distance cost, the sub-pixel motion vector displacements being calculated using the following formula performed by the decoding device: x0=(E(−1,0)−E(1,0)) / (2×(E(−1,0)+E(1,0)−2×E(0,0))), y0=(E(0,−1)−E(0,1)) / (2×(E(0,−1)+E(0,1)−2×E(0,0))) is directly determined by where x0 and y0 are coordinates associated with the sub-pixel motion vector displacements relative to a center (0,0), and E(-1,0), E(1,0), E(0,0), E(0,-1) and E(0,1) are the integer distance costs corresponding to candidate integer motion vector displacements (-1,0), (1,0), (0,0), (0,-1) and (0,1) respectively relative to the initial motion vector; determining a refined motion vector based on the target integer motion vector displacements, the sub-pixel motion vector displacements, and the initial motion vector. method.

2. 2. The method of claim 1, wherein x0 is determined by performing at least one of a shift, a compare, and an increment operation on E(-1,0), E(1,0) and E(0,0).

3. 2. The method of claim 1, wherein y0 is determined by performing at least one of a shift, a compare, and an increment operation on E(0,-1), E(0,1) and E(0,0).

4. The method of claim 1 , wherein the sub-pixel motion vector displacements are constrained to be between −0.5 pixels and +0.5 pixels.

5. The step of determining a target integer motion vector displacement comprises: calculating an integer distance cost for each candidate integer motion vector displacement; selecting a candidate integer motion vector displacement corresponding to a lowest integer distance cost as the target integer motion vector displacement. The method of claim 1.

6. The target integer motion vector displacement includes a first motion vector displacement corresponding to a reference picture list L0 and a second motion vector displacement corresponding to a reference picture list L1, and the method includes: determining the first motion vector displacement by comparing integer costs corresponding to candidate integer motion vector displacements corresponding to the reference picture list L0; and determining the second motion vector displacement by inverting the first motion vector displacement. The method of claim 1.

7. an input for receiving encoded video data; an entropy decoding unit configured to decode the received encoded video data to obtain decoded data; and one or more processors coupled to the entropy decoding unit, the one or more processors comprising: determining a target integer motion vector displacement by comparing integer distance costs corresponding to candidate integer motion vector displacements with respect to an initial motion vector according to the decoded data; determining sub-pixel motion vector displacements by performing a calculation on the integer distance cost, the sub-pixel motion vector displacements being expressed as: x0=(E(−1,0)−E(1,0)) / (2×(E(−1,0)+E(1,0)−2×E(0,0))), y0=(E(0,−1)−E(0,1)) / (2×(E(0,−1)+E(0,1)−2×E(0,0))) is directly determined by where x0 and y0 are coordinates associated with the sub-pixel motion vector displacements relative to a center (0,0), and E(-1,0), E(1,0), E(0,0), E(0,-1) and E(0,1) are the integer distance costs corresponding to candidate integer motion vector displacements (-1,0), (1,0), (0,0), (0,-1) and (0,1) respectively relative to the initial motion vector; determining a refined motion vector based on the target integer motion vector displacements, the sub-pixel motion vector displacements, and the initial motion vector. Decoding device.

8. 8. The apparatus of claim 7, wherein x0 is determined by performing at least one of a shift, compare, and increment operation on E(-1,0), E(1,0) and E(0,0).

9. 8. The apparatus of claim 7, wherein y0 is determined by performing at least one of a shift, compare, and increment operation on E(0,-1), E(0,1) and E(0,0).

10. The apparatus of claim 7 , wherein the sub-pixel motion vector displacement is constrained to be between −0.5 pixels and +0.5 pixels.

11. The one or more processors may further comprise the steps of: determining the target integer motion vector displacement; Calculate an integer distance cost for each candidate integer motion vector displacement; by selecting the candidate integer motion vector displacement corresponding to the lowest integer distance cost as the target integer motion vector displacement; 8. The apparatus of claim 7.

12. the target integer motion vector displacement includes a first motion vector displacement corresponding to a reference picture list L0 and a second motion vector displacement corresponding to a reference picture list L1, the one or more processors: determining the first motion vector displacement by comparing integer costs corresponding to candidate integer motion vector displacements corresponding to the reference picture list L0; configured to determine the second motion vector displacement by inverting the first motion vector displacement.

8. The apparatus of claim 7.

13. A computer readable medium storing instructions which, when executed on a processor, perform the steps of the method according to any one of claims 1 to 6.

Images