Error surface-based subpixel precision refinement method for decoder-side motion vector refinement

Abstract

To prevent drift between encoder reconstruction and decoder reconstruction during decoder-side motion vector refinement / derivation, an encoding and decoding system is provided that employs the same error surface technique. [Solution] In template matching applicable to both encoding and decoding systems, refinement movement occurs only in references starting from the center of subpixel precision, based on explicitly signaled merge indices or implicitly derived through cost evaluation. In bilateral matching, refinement (with or without averaged templates) begins in reference lists L0 and L1, starting from the centers of their respective subpixel precisions, based on explicitly signaled merge indices or implicitly derived through cost evaluation.

Description

Technical Field

[0001] Cross - reference to related applications This application claims the benefit of Indian Provisional Patent Application No. 201831024666, filed on July 2, 2018, entitled "Error - surface - based refinement method for sub - pixel accuracy in decoder - side motion vector refinement", which is incorporated herein by reference in its entirety.

Background Art

[0002] Current hybrid video coders such as H.264 / AVC or H.265 / HEVC employ compression that includes predictive coding. Pictures in a video sequence are subdivided into blocks of pixels, and these blocks are then encoded. Instead of encoding each block pixel - by - pixel, the entire block is predicted using already - encoded pixels that are spatially or temporally close to that block. The encoder further processes only the difference between the block and its prediction. This further processing typically includes the conversion of block pixels to coefficients in the transform domain. These coefficients are then further compressed by quantization and made more compact by entropy coding to form a bitstream. The bitstream further includes any signaling information that enables the decoding of the encoded video. For example, the signaling information may include settings related to encoding, such as the size of the input picture, the frame rate, the quantization step indication, the prediction applied to the blocks of the picture, etc. The encoded signaling information and the encoded signals are ordered within the bitstream in a manner known to both the encoder and the decoder. Thereby, the decoder can parse the encoded signaling information and the encoded signals.

[0003] Temporal prediction utilizes the temporal correlation between pictures in a video, also known as frames. Because it uses inter-dependencies between different video frames, it is also called inter-prediction. Thus, the currently encoded block, also called the current block, is predicted from one or more previously encoded pictures, called reference pictures. A reference picture is not necessarily the picture that precedes the current block in the display order of the video sequence. The encoder may encode pictures in a different encoding order than the display order. A copositional block in the reference picture may be determined as the prediction for the current block. A copositional block is a block in the reference picture that is in the same position as the current block in the current picture. Such predictions are accurate for areas of still picture movement, i.e., areas of still picture movement from one picture to another.

[0004] Motion estimation is typically used when determining the prediction of the current block in order to obtain a motion-compensated predictor. Thus, the current block is predicted by a block in the reference picture located at a distance given by the motion vector from the position of the copositional block. The motion vector may be signaled in the bitstream to enable the decoder to determine the same prediction for the current block. To further reduce the signal transmission overhead caused by signaling the motion vector for each block, the motion vector itself may be estimated. Motion vector estimation may be performed based on the motion vectors of neighboring blocks in the spatial and / or time domain.

[0005] The prediction for the current block may be calculated using one reference picture, or by weighting predictions obtained from two or more reference pictures. The reference picture may be an adjacent picture, i.e., the picture immediately preceding and / or immediately following the current picture in display order, because adjacent pictures are most likely to be similar to the current picture. However, generally, the reference picture can be any other picture that precedes or follows the current picture in display order and precedes the current picture in the bitstream (in decoding order). This can offer advantages, for example, in the case of occlusion and / or non-linear motion in video content. Thus, reference picture identification information may also be signaled within the bitstream.

[0006] A special mode of interprediction is the so-called bi-prediction, in which two reference pictures are used when generating the prediction for the current block. Specifically, the two predictions determined in each of the two reference pictures are combined into the prediction signal for the current block. Bi-prediction can lead to a more accurate prediction of the current block than uni-prediction, i.e., prediction using only a single reference picture. Such a more accurate prediction leads to a smaller difference (also called a "residual") between the pixels of the current block and the prediction, which can be encoded more efficiently, i.e., compressed into a shorter bitstream. In general, three or more reference pictures may be used to find three or more reference blocks for each of which to predict the current block; that is, multi-reference interprediction can be applied. Thus, the term multi-reference prediction includes bi-prediction and prediction using three or more reference pictures.

[0007] To provide more accurate motion estimation, the resolution of the reference picture may be improved by interpolating samples between pixels. Fractional pixel interpolation can be performed by a weighted average of the nearest pixels. For half-pixel resolution, bilinear interpolation is typically used, for example. Other fractional pixels are calculated as an average of the nearest pixels, weighted by the reciprocal of the distance between each nearest pixel and the predicted pixel.

[0008] Motion vector estimation is a computationally complex task in which the similarity between a current block and the corresponding predicted block pointed to by a candidate motion vector in a reference picture is calculated. Typically, the search region contains M×M samples of the image, and each sample location of the M×M candidate positions is tested. This test involves calculating a similarity index between N×N reference blocks C and the blocks R located at the candidate positions being tested in the search region. Due to its simplicity, the sum of absolute differences (SAD) is a frequently used index for this purpose.

number

[0009] In the formula above, x and y define candidate positions within the search region, while indices i and j indicate samples within the reference block C and candidate block R. Candidate positions are often called block displacements or offsets, reflecting the expression of block matching as shifting the reference block within the search region and calculating the similarity between the reference block C and the overlapping portion of the search region. To reduce complexity, the number of candidate motion vectors is typically reduced by restricting the candidate motion vectors to a certain search space. The search space may be defined, for example, by the number of pixels and / or positions surrounding a position in the reference picture that corresponds to the position of the current block in the current image. After calculating the SAD for all M×M candidate positions x and y, the best-matching block R is the block of the position that gives the lowest SAD, corresponding to the greatest similarity with reference block C. On the other hand, candidate motion vectors can be defined by a list of candidate motion vectors formed by the motion vectors of neighboring blocks.

[0010] Motion vectors are typically determined, at least partially, on the encoder side and signaled to the decoder within the encoded bitstream. However, motion vectors can also be derived at the decoder. In such cases, the current block is unavailable to the decoder and cannot be used to calculate the similarity between the candidate motion vector and the block it points to in the reference picture. Therefore, a template constructed from pixels of already decoded blocks is used instead of the current block. For example, already decoded pixels adjacent to the current block may be used. Such motion estimation offers the advantage of reducing signal transmission; that is, since the motion vector is derived in the same way at both the encoder and the decoder, no signal transmission is required. On the other hand, the accuracy of such motion estimation may be lower.

[0011] To provide a trade-off between accuracy and signal transmission overhead, motion vector estimation may be divided into two steps: motion vector derivation and motion vector refinement. For example, motion vector derivation may involve selecting a motion vector from a list of candidates. Such selected motion vectors can then be further refined, for example, by searching in a search space. Searching in a search space is based on calculating a cost function for each candidate motion vector, i.e., a cost function for each candidate position in the block that the candidate motion vectors point to.

[0012] Document JVET-D0029: Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching, X. Chen, J. An, J. Zheng (this document can be found at http: / / phenix.it-sudparis.eu / jvet / site) describes motion vector refinement in which a first motion vector is found at integer pixel resolution and further refined by a search at half-pixel resolution in the search space around the first motion vector. Bilateral motion vector search based on block templates is used.

[0013] Motion vector estimation is a key feature of modern video encoders and decoders because its efficiency in terms of quality, speed, and complexity impacts the efficiency of video encoding and decoding. [Overview of the project]

[0014] This invention relates to video encoding and decoding, and more particularly to the determination of motion vectors.

[0015] In a first aspect of the present invention, a decoder-side motion vector refinement system may include the following steps for obtaining subpixel-precision delta motion vectors in one or more reference frames around each initial subpixel-precision refinement center(s):

[0016] A method for determining an integer distance-refined motion vector for each reference frame involves sequentially and iteratively performing a plurality of integer 1-pixel distance-refined operations (iterations) using a cost function, wherein the search center is updated after each operation (iteration) to the lowest cost position in that operation (iteration); and determining that the sequentially iterative loop will terminate early because the cost of the center position in a given operation (iteration) is lower than the cost of a pair of 1-pixel neighbor positions around it, or determining that a predetermined number of operations (iterations) has been reached. If early termination occurs, the method may include determining a sub-pixel distance-refined motion vector around the last search center in each reference by calculating the position having the minimum value on a parametric error surface fitted using the cost function values ​​of the last search center and the pair of 1-pixel neighbor positions around it; and returning a total refined motion vector as the sum of the determined integer distance-refined motion vectors and the determined sub-pixel distance-refined motion vectors for each reference frame. When a predetermined number of iterations of the motion (iterations) is reached, the method may include a step of returning a refined motion vector corresponding to the position having the minimum cost function value throughout all the motions (iterations).

[0017] In one implementation of the first aspect, the cost function is performed for integrated refinement in both reference lists L0 and L1 such that the displacement in reference list L1 is equal to the displacement in reference list L0 in both the horizontal and vertical directions, but in opposite directions (the cost function is referred to as SBM_JOINT).

[0018] In a second aspect of the present invention, a decoder-side motion vector refinement system may include the following steps for obtaining subpixel-precision delta motion vectors in one or more reference frames around each initial subpixel-precision refinement center(s):

[0019] A step of sequentially and iteratively performing a plurality of integer 1-pixel distance refinement operations (iterations) using a first cost function to determine an integer distance refinement motion vector for each reference frame, wherein the search center is updated after each operation (iteration) to the lowest cost position in that operation (iteration); and determining that the sequentially iterative loop terminates early because the cost of the center position in a given operation (iteration) is lower than the cost of a set of 1-pixel neighbor positions around it, or determining that a predetermined number of operations (iterations) have been reached.

[0020] The method includes the steps of evaluating the cost function values ​​at the last search center and a pair of one-pixel neighborhood locations of the last search center using a second cost function; if the last search center location has the lowest second cost function value when compared with the second cost function values ​​at the pair of one-pixel neighborhood locations of the last search center, determining the sub-pixel distance-refined motion vectors around the best integer-distance-refined location in each reference by calculating the location having the minimum value on a parametric error surface fitted using the second cost function value; and returning the total refined motion vector for each reference frame as the sum of the determined integer-distance-refined motion vectors and the determined sub-pixel distance-refined motion vectors. If the last search center location does not have the lowest second cost function value when compared with the second cost function values ​​at the pair of one-pixel neighborhood locations of the last search center, the method includes the step of returning the refined motion vector corresponding to the location having the best second cost function value for each reference frame.

[0021] In an implementation with a second aspect, the first cost function is SBM_JOINT and the second cost function is TBM_INDEPDENDENT, which is defined as a cost function related to performing independent refinements on a common bilateral averaging template in both reference lists L0 and L1.

[0022] This method can be used for template matching use cases and bilateral matching use cases.

[0023] In another aspect of the present invention, the method disclosed herein may be implemented as instructions stored on a non - transient computer - readable medium, which can be read and executed by a processor to perform the steps of the method described above.

[0024] In some aspects of the present invention, a method for decoder - side motion vector refinement includes determining a target integer motion vector displacement by comparing integer distance costs corresponding to candidate integer motion vector displacements for an initial motion vector, determining a sub - pixel motion vector displacement by performing calculations on the integer distance costs, and determining a refined motion vector based on the target integer motion vector displacement, the sub - pixel motion vector displacement, and the initial motion vector.

[0025] By the present invention, more advantages are achieved than in the prior art. For example, embodiments of the present invention utilize the determined early termination of a sequential iteration loop based on a check of the cost of a center position in a given sequential iteration operation against the costs of a set of one - pixel - neighborhood positions around the center position. By terminating the sequential iteration loop early, unnecessary calculations can be reduced or eliminated.

[0026] Furthermore, the apparatus can implement the above-described method, and may be a combination of software and hardware. For example, encoding and / or decoding may be performed by an integrated circuit (semiconductor device or chip) such as a general-purpose processor (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). However, embodiments of the present invention are not limited to implementation on programmable hardware. Embodiments of the present invention can be implemented on an application-specific integrated circuit, or by one or a combination of a CPU, a DSP, an FPGA, and an ASIC component.

Brief Description of the Drawings

[0027] Hereinafter, exemplary embodiments will be described in more detail with reference to the accompanying drawings and figures.

[0028] [Figure 1] FIG. [Number] is a block diagram showing an exemplary structure of an encoder for encoding a video signal according to an embodiment of the present disclosure.

[0029] [Figure 2] FIG. [Number] is a block diagram showing an exemplary structure of a decoder for decoding a video signal according to an embodiment of the present disclosure.

[0030] [Figure 3] FIG. [Number] is a schematic diagram showing an exemplary template matching suitable for bi-prediction.

[0031] [Figure 4A] FIG. [Number] is a schematic diagram showing an exemplary template matching suitable for uni-prediction and bi-prediction.

[0032] [Figure 4B] FIG. [Number] is a schematic diagram showing an exemplary bilateral matching suitable for uni-prediction and bi-prediction;

[0033] [Figure 5] This is a flowchart showing a possible implementation of motion vector search.

[0034] [Figure 6] This is a schematic diagram illustrating an example of local illumination compensation applied in video coding.

[0035] [Figure 7] This is a schematic diagram illustrating an example of decoder-side motion vector refinement.

[0036] [Figure 8] This is a schematic diagram showing an example of subpixel positioning.

[0037] [Figure 9] A block diagram of a set of one-pixel neighborhood positions around a central pixel for obtaining subpixel-accurate delta motion vector refinement in one or more reference frames, according to one embodiment of the present disclosure.

[0038] [Figure 10] This is a simplified flowchart illustrating a method for obtaining subpixel-accurate delta motion vector refinement in one or more reference frames in a decoder-side motion vector refinement system according to some embodiments of the present disclosure.

[0039] [Figure 11] This is a simplified flowchart illustrating a method for implementing one embodiment of the present disclosure.

[0040] [Figure 12] This is a block diagram of a device that can be used to implement various embodiments of the present disclosure. [Modes for carrying out the invention]

[0041] Embodiments of this disclosure relate to improvements in template matching applied in motion vector refinement. In particular, template matching is applied to zero-average templates and zero-average candidate blocks even when the average of the best matching blocks found is not further refined (by local illumination control).

[0042] Template matching is used to find the best first and second motion vectors pointing to the first and second reference pictures, respectively. Template matching is performed for each reference picture by template matching in a predetermined search space on a position given by an initial motion vector that can be derived by the decoder or signaled to the decoder.

[0043] Template matching may be performed based on a block template derived from the block pointed to by the initial motion vector.

[0044] Such template matching for finding the best matching block to obtain a predictor for a block may be used, for example, in hybrid video encoders and / or decoders. For example, application to encoders and / or decoders such as HEVC may be advantageous. In particular, further developments of HEVC or new codecs / standards can utilize embodiments of this disclosure.

[0045] Figure 1 shows an encoder 100 including an input 102 for receiving input image samples of a video stream frame or picture, and an output 172 for generating an encoded video bitstream. Where used herein, the term “frame” is also referred to as “picture.” Note that this disclosure is also applicable to interlaced fields of video. Generally, a picture contains m × n pixels, which correspond to image samples and may contain one or more color components. Where used herein, the following description refers to pixels as luminance samples. However, note that the motion vector search in this disclosure can be applied to any color component, including components in the search space such as chrominance or RGB. Furthermore, it may be advantageous to perform motion vector estimation on only one component and then apply the estimated motion vector to more or all components.

[0046] The input blocks to be encoded do not necessarily have to be the same size. A single picture may contain blocks of different sizes, and different pictures may have different block rasters.

[0047] In one exemplary embodiment, the encoder 100 is configured to perform prediction, transformation, quantization, and entropy coding on a video stream. Transformation, quantization, and entropy coding are performed by the transformation unit 106, the quantization unit 108, and the entropy coding unit 170, respectively, to produce an encoded video bitstream.

[0048] A video stream may contain multiple frames, each frame being divided into blocks of a certain size that are intra- or inter-encoded. For example, the first frame block of the video stream is intra-encoded by the intra-prediction unit 154. Intra-frames are encoded using only the information within the same frame so that they can be decoded independently and provide entry points in the bitstream for random access. Other frame blocks of the video stream may be inter-encoded by the inter-prediction unit 144, that is, by using information from previously encoded frames (reference frames) to reduce temporal redundancy, and each block of the inter-encoded frames is predicted from a block in the reference frame. The mode selection unit 160 is configured to select between the block 155 of frames processed by the intra-prediction unit 154 and the block 145 of frames processed by the inter-prediction unit 144. The mode selection unit 160 also controls the parameters for intra-prediction or inter-prediction. Intra-encoded blocks may be provided within inter-encoded frames to allow for image information refreshment. Furthermore, intra-frames containing only intra-encoded blocks may be periodically inserted into the video sequence to provide entry points for decoding, i.e., points from which the decoder can begin decoding without having information from previously encoded frames.

[0049] The intra-estimation unit 152 and the intra-prediction unit 154 are units configured to perform intra-prediction. In particular, the intra-estimation unit 152 may derive a prediction mode based on knowledge of the original image, while the intra-prediction unit 154 provides the corresponding predictor, i.e., the sample predicted using the selected prediction mode, for differential coding. To perform spatial or temporal prediction, the coded block may be further processed by the inverse quantization unit 110 and the inverse transform unit 112 to provide the inverse transformed block 113. The reconstruction unit 114 combines the inverse transformed block 113 with the prediction block 165 to provide the reconstructed block 115, which is provided to the loop filtering unit 120 to further improve the quality of the decoded image. The filtered block then forms a reference frame which is subsequently stored in the decode picture buffer 130. The inverse quantization unit 110, the inverse transform unit 112, the reconstruction unit 114, and the loop filter 120 form part of the decoder (decode loop). Such a decoding loop (decoder) on the encoder side has the advantage of producing the same reference frame as the reference picture that is reconstructed on the decoder side. Thus, the encoder and decoder operate in a corresponding manner. The term "reconstruction" here refers to obtaining the reconstructed block 115 by adding the predicted block 165 to the inversely transformed (decoded residual) block 113.

[0050] The encoder 100 also includes an interestimation unit 142 that receives a picture block 101 of the current frame or picture to be interencoded, and one or more reference frames from a decode picture buffer 130. Motion estimation is performed by the interestimation unit 142, while motion compensation is performed by the interpretation unit 144. Motion estimation is used, for example, to obtain motion vectors and reference frames based on some cost function, also using the original image to be encoded. For example, the motion estimation (interestimation) unit 142 can provide an initial motion vector estimation. The initial motion vector can then be signaled (transmitted as a signal) in the bitstream, either directly in the form of a motion vector, or as an index pointing to a motion vector candidate in a list of candidates constructed in the encoder and decoder according to a predetermined rule. Motion compensation then derives a predictor for the current block as a translation of the block copositioned with the current block in the reference frame to the reference block in the reference frame, i.e., by the motion vector. The interpretation unit 144 outputs a predict block 145 for the current block. Here, the predicted block 145 minimizes the cost function. For example, the cost function may be the difference between the current block to be encoded and its predicted block, i.e., the cost function minimizes the residual block 105. Minimizing the residual block is based, for example, on calculating the sum of absolute differences (SAD) between all pixels (samples) of the current block and the candidate block in the candidate reference picture. In general, any other similarity metric can be used, such as the mean square error (MSE) or the structural similarity metric (SSIM).

[0051] The cost function may also be the number of bits required to encode such interblocks and / or the distortion resulting from such encoding. Thus, a rate-distortion optimization procedure may be used to determine the motion vector selection and / or, in general, encoding parameters such as whether to use interpretation or intrapretation for blocks and with what settings.

[0052] The intra-estimation unit 152 and the inter-prediction unit 154 receive, as input, a picture block 101 of the current frame or picture to be intra-encoded, and one or more reference samples 117 from an already reconstructed region of the current frame. The intra-prediction then describes the pixels of the current block of the current frame using a function of the reference samples of the current frame. The intra-prediction unit 154 outputs a predicted block for the current block, where the predicted block advantageously minimizes the difference between the current block to be encoded and its predicted block, i.e., minimizes the residual block. Minimization of the residual block can be based, for example, on a rate-distortion optimization procedure. In particular, the predicted block is obtained as directional interpolation of the reference samples. The direction may be determined by rate-distortion optimization and / or by computing a similarity index as described above in relation to inter-prediction.

[0053] The inter-prediction unit 142 receives as input a block or more universally-formed image sample of the current frame or picture to be inter-encoded, and two or more already decoded pictures 231. The inter-prediction then describes the current image sample of the current frame using motion vectors to reference image samples of the reference picture. The inter-prediction unit 144 outputs one or more motion vectors 145 for the current image sample, the reference image sample pointed to by these motion vectors is advantageous in that it minimizes the difference between the current image sample to be encoded and its reference image sample, i.e., minimizes the residual image sample. A predictor for the current block is then provided by the inter-prediction unit 144 for differential encoding.

[0054] The difference between the current block and its prediction, i.e., the residual block 105, is then transformed by the transformation unit 106 to produce the transformed coefficients 107. The transformed coefficients 107 are quantized by the quantization unit 108 and entropy encoded by the entropy coding unit 170. The encoded picture data 171 thus produced, i.e., the encoded video bitstream, includes intra-encoded and inter-encoded blocks as well as corresponding signal transmission information (mode indication, motion vector indication, and / or intra-prediction direction, etc.). The transformation unit 106 may apply a linear transformation such as the discrete Fourier transform, the fast Fourier transform, or the discrete cosine transform (DFT / FFT or DCT). Transformation to the spatial frequency domain offers the advantage that the resulting coefficients 107 typically have higher values ​​at lower frequencies. Thus, after effective coefficient scanning (e.g., zigzag scanning) and quantization, the resulting sequence of values ​​typically has some larger values ​​at the beginning and a sequence of zeros at the end. This enables even more efficient encoding. The quantization unit 108 performs actual lossy compression by reducing the resolution of the coefficient values. The entropy coding unit 170 then assigns binary codewords to the coefficient values ​​to generate a bitstream. The entropy coding unit 170 also encodes signal transmission information (not shown in Figure 1).

[0055] Figure 2 shows the video decoder 200. The video decoder 200 includes a decode picture buffer 230, an interpretation unit 244, and an intraprediction unit 254, which is a block prediction unit. The decode picture buffer 230 is configured to store at least one (in the case of single prediction) or at least two (in the case of double prediction) reference frames reconstructed from the encoded video bitstream, the reference frames being different from the current frame (the frame currently being decoded) of the encoded video bitstream. The intraprediction unit 254 is configured to generate prediction blocks, which are estimates of the blocks to be decoded. The intraprediction unit 254 is configured to generate these predictions based on the reference samples obtained from the decode picture buffer 230.

[0056] The decoder 200 is configured to decode the encoded video bitstream generated by the video encoder 100, preferably with both the decoder 200 and the encoder 100 generating identical predictions for each block to be encoded / decoded. The features of the decode picture buffer 230 and intra prediction unit 254 are similar to those of the decode picture buffer 130 and intra prediction unit 154 in Figure 1.

[0057] The video decoder 200 further includes units that are also present in the video encoder 100, such as an inverse quantization unit 210, an inverse transform unit 212, and a loop filtering unit 220, which correspond to the inverse quantization unit 110, the inverse transform unit 112, and the loop filtering unit 120 of the video encoder 100, respectively.

[0058] The entropy decode unit 204 is configured to decode the received encoded video bitstream and obtain correspondingly quantized residual transformation coefficients 209 and signal transmission information. The quantized residual transformation coefficients 209 are provided to the inverse quantization unit 210 and the inverse transformation unit 212 to generate residual (inverse transformed) blocks. The residual blocks are added to the prediction blocks 265 in the reconstruction unit 214, and the sum is provided to the loop filtering unit 220 to obtain the decoded video. The frames of the decoded video are stored in the decode picture buffer 230 and can function as the decoded picture 231 for interprediction.

[0059] In general, the intra-prediction units 154 and 254 in Figures 1 and 2 can use reference samples from already encoded regions to generate prediction signals for blocks that need to be encoded or decoded.

[0060] The entropy decoding unit 204 receives the encoded bitstream 171 as its input. Generally, the bitstream is first parsed, i.e., the signal transfer parameters and residuals are extracted from the bitstream. Typically, the syntax and semantic content of the bitstream are defined by standards so that the encoder and decoder can function in an interoperable manner. As described in the background section above, the encoded bitstream does not contain only the prediction residuals. In the case of motion-compensated prediction, the motion vector indication is also encoded in the bitstream and parsed from the bitstream in the decoder. The motion vector indication can be given by the reference picture from which the motion vector is provided and by the motion vector coordinates. Until now, it has been considered to encode the entire motion vector. However, in the bitstream, only the difference between the current motion vector and the previous motion vector may be encoded. This approach allows for the use of redundancy between motion vectors of neighboring blocks.

[0061] To efficiently encode reference pictures, the H.265 codec (ITU-T, H265, Series H: Audiovisual and Multimedia Systems: High Efficiency Video Coding) provides a list of reference pictures, assigning each reference frame to a list index. The reference frames are then transmitted within the bitstream by including their corresponding assigned list index. Such lists may be defined by the standard or transmitted at the beginning of a video or a set of frames. Note that H.265 defines two lists of reference pictures, called reference lists L0 and L1. The reference pictures are then transmitted within the bitstream by indicating the reference list (L0 or L1) and the index in that list relating to the desired reference picture. Providing more than one list may offer advantages for better compression. For example, reference list L0 may be used for both unidirectionally interpredicted slices and bidirectionally interpredicted slices, while reference list L1 may be used only for bidirectionally interpredicted slices. However, in general, this disclosure is not limited to anything in Lists L0 and L1.

[0062] Reference lists L0 and L1 may be defined by standard and fixed, but greater flexibility in encoding / decoding can be achieved by signaling them at the beginning of the video sequence. Therefore, the encoder may construct lists L0 and L1 with specific reference pictures ordered according to an index. Reference lists L0 and L1 may have the same fixed size. In general, there may be three or more lists. Motion vectors may be signaled directly by coordinates in the reference pictures. Alternatively, a list of candidate motion vectors may be constructed, as specified in H.265, in which indices associated with specific motion vectors can be transmitted.

[0063] The motion vector of a current block is typically correlated with the motion vector of a neighboring block in the current picture or a previously encoded picture. This is because neighboring blocks are likely to correspond to the same moving object with similar motion, and the object's motion is unlikely to change rapidly over time. As a result, using the motion vector in a neighboring block as a predictor reduces the size of the signaled motion vector difference. The Motion Vector Predictor (MVP) is typically derived from already encoded / decoded motion vectors from spatially neighboring blocks or from temporally neighboring blocks in a copositional picture. In H.264 / AVC, this is done by performing a component-wise median of the motion vectors of three spatially neighboring blocks. Using this approach, signaling of the predictor is not required. Temporal MVPs from copositional pictures are considered only in the so-called temporal direct mode of H.264 / AVC. The H.264 / AVC direct mode is also used to derive motion data other than motion vectors. Therefore, they are more relevant to the block-merge concept in HEVC. In HEVC, the approach of implicitly deriving the MVP was replaced by a technique known as motion vector competition, which explicitly signals which MVP from the list of MVPs will be used for motion vector derivation. As a result of the variable coding quadtree block structure in HEVC, a single block can have several neighboring blocks with motion vectors as potential MVP candidates. Taking the left neighborhood as an example, in the worst case, if a 64x64 lumar coding tree block is not further subdivided and the one to its left is subdivided to its maximum depth, a 64x64 lumar prediction block could have 16 4x4 lumar prediction blocks to its left.

[0064] Advanced Motion Vector Prediction (AMVP) was introduced to correct motion vector races to take such flexible block structures into account. During the development of HEVC, the initial AMVP design was significantly simplified to provide a good trade-off between coding efficiency and implementation-friendly design. The initial AMVP design included five MVPs from three different classes of predictors: three motion vectors from spatial neighbors, the median of the three spatial predictors, and scaled motion vectors from copositional temporal neighbor blocks. Furthermore, the list of predictors was modified by reordering the most likely motion predictors to the first position and removing redundant candidates to ensure minimal signal transfer overhead. The final design for constructing the AMVP candidate list includes the following two MVP candidates: a) up to two spatial candidate MVPs derived from five spatial neighborhood blocks; b) one temporal candidate MVP derived from two temporal, copositional blocks if both spatial candidate MVPs are unavailable or identical; and c) zero motion vector if no spatial, temporal, or both candidates are available. Further details regarding motion vector determination can be found in the book, High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014, edited by V. Sze et al., incorporated herein by reference, particularly in Chapter 5.

[0065] To further improve motion vector estimation without further increasing signal transmission overhead, it may be beneficial to further refine the motion vector derived at the encoder side and provided in the bitstream. Motion vector refinement can be performed at the decoder without assistance from the encoder. The encoder may use the same refinement within its decoder loop to obtain the corresponding motion vector. Motion vector refinement is performed in a search space that includes integer and fractional pixel positions of the reference picture. For example, fractional pixel positions may be half-pixel positions, quarter-pixel positions, or other fractional positions. Fractional pixel positions can be obtained from integer (full pixel) positions by interpolation such as bilinear interpolation.

[0066] In current block biprediction, two prediction blocks, obtained using the first motion vector from reference list L0 and the second motion vector from reference list L1, are combined into a single prediction signal. This can provide a better fit to the original signal than single prediction, resulting in smaller residual information and potentially more efficient compression.

[0067] In the decoder, the current block is not available because it is being decoded, but a template is used for motion vector refinement. The template is an estimate of the current block and is constructed based on already processed (i.e., encoded on the encoder side and decoded on the decoder side) portions of the image.

[0068] First, estimates of the first motion vector MV0 and the second motion vector MV1 are received as input to the decoder 200. In the encoder 100, the motion vector estimates MV0 and MV1 can be obtained by block matching and / or by searching within a list of candidates (e.g., a merge list) formed by the motion vectors of neighboring blocks of the current block (within the same picture or adjacent pictures). MV0 and MV1 are then advantageously signaled to the decoder side within the bitstream. However, it should be noted that the first decision stage in the encoder could generally also be performed by template matching, which would have the advantage of reducing signal transmission overhead.

[0069] In the decoder 200, motion vectors MV0 and MV1 are advantageously obtained based on the information in the bitstream. Motion vectors MV0 and MV1 are signaled directly, or differentially, and / or their indices in a list of motion vectors (merge list) are signaled. However, this disclosure is not limited to signaling motion vectors within a bitstream. Rather, according to this disclosure, the motion vectors may already be determined by template matching in the first stage of motion vector estimation, corresponding to the operation of the encoder. The template matching in the first stage (motion vector derivation) may be performed on a different search space than the search space of the second motion vector refinement stage. In particular, motion vector refinement may be performed on a search space with higher resolution (i.e., shorter distances between search positions).

[0070] The decoder is also provided with instructions for two reference pictures, RefPic0 and RefPic1, which are pointed to by MV0 and MV1, respectively. The reference pictures are stored in the decode picture buffers in the encoder and decoder as a result of previous processing, i.e., encoding and decoding, respectively. One of these reference pictures is selected by search for motion vector refinement. The reference picture selection unit of the apparatus for determining the motion vector is configured to select a first reference picture pointed to by MV0 and a second reference picture pointed to by MV1. Following this selection, the reference picture selection unit determines whether the first or second reference picture is used to perform motion vector refinement. To perform motion vector refinement, the search region in the first reference picture is defined around the candidate position pointed to by the motion vector MV0. The candidate search space positions in the search region are analyzed to find the block that is most similar to the template block by performing template matching in the search space and determining a similarity metric such as the Sum of Absolute Differences (SAD). The search space position indicates the location where the top-left corner of the template is matched. As mentioned above, the top-left corner is merely a convention, and any point in the search space, such as the center point, can generally be used to indicate the match location.

[0071] Figure 4A shows an alternative template matching method applicable to single prediction as well. Further details can be found in document JVET-A1001, particularly in section 2.4.6, "Pattern matched motion vector derivation," of the document JVET-A1001, titled "Algorithm Description of Joint Exploration Test Model 1" by Jianle Chen et. al., accessible at http: / / phenix.it-sudparis.eu / jvet / . In this template matching approach, the template is determined as a sample adjacent to the current block in the current frame. As shown in Figure 1 of document JVET-A1001, already reconstructed samples adjacent to the upper and left boundaries of the current block are taken and may be called an "L-shaped template."

[0072] According to document JVET-D0029, which is incorporated by reference, the Decoder-Side Motion Vector Refinement (DMVR) takes as input initial motion vectors MV0 and MV1, which point to two respective reference pictures, RefPict0 and RefPict1. These initial motion vectors are used to determine the respective search spaces within RefPict0 and RefPict1. Furthermore, using motion vectors MV0 and MV1, a template is constructed based on the respective blocks A and B (of the sample) pointed to by MV0 and MV1, as follows: Template=function(Block A, Block B)

[0073] The function may be a sample clipping operation combined with a sample-by-sample weighted addition. The template is then used to perform template matching in a search space determined based on MV0 and MV1 in the respective reference pictures Refpic0 and Refpic1. The cost function for determining the best template match in each search space is SAD(Template, Block candA'), where block candA' is a candidate coded block pointed to by a candidate MV in the search space spanning the positions given by MV0. Figure 3 shows the determination of the best matching block A' and the resulting refined motion vector MV0'. Correspondingly, the same template is used to find the best matching block B' and the corresponding motion vector MV1' pointing to block B', as shown in Figure 3. In other words, after the template is constructed based on blocks A and B pointed to by the initial motion vectors MV0 and MV1, the refined motion vectors MV0' and MV1' are found through a search on RefPic0 and RefPic1 using the template.

[0074] The motion vector derivation technique is sometimes also called frame rate up-conversion (FRUC). Initial motion vectors MV0 and MV1 may generally be shown in the bitstream to ensure that the encoder and decoder can use the same initial points for motion vector refinement. Alternatively, the initial motion vectors may be obtained by providing a list of initial candidates, each containing one or more initial candidates. For each of these, a refined motion vector is determined, and finally, the refined motion vector with the lowest cost function is selected.

[0075] As described above, the motion vector derivation mode by template matching is a special merge mode based on the Frame Rate Upconversion (FRUC) technique. In this mode, the motion information of a block is derived on the decoder side. According to the specific implementation described in document JVET-A1001 ("Algorithm Description of Joint Exploration Test Model 1," accessible at http: / / phenix.it-sudparis.eu / jvet / ), if the merge flag is true, the FRUC flag is signaled for the CU or PU. If the FRUC flag is false, the merge index is signaled and the normal merge mode is used. If the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) should be used to derive the motion information for that block.

[0076] In summary, during the motion vector derivation process, initial motion vectors are first derived for the entire prediction unit (PU) based on bilateral matching or template matching. First, a list of MV candidates is generated, which may be, for example, a merge list of PUs. The list is checked, and a candidate leading to the minimum matching cost is selected as the starting point (initial motion vector). Next, a local search is performed around the starting point based on bilateral matching or template matching, and the motion vector(s) (MV) that yields the minimum matching cost is adopted as the MV for that PU. Then, the motion information is further refined using the derived PU motion vector as the starting point. The terms prediction unit (PU) and coding unit can be used interchangeably herein to describe blocks of samples within a picture (frame).

[0077] As shown in Figure 4B, bilateral matching (as described in document JVET-A1001) is used to derive motion information for the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. Assuming a continuous motion trajectory, the motion vectors MV0 and MV1 pointing to those two reference blocks are proportional to the temporal distance between the current picture and the two reference pictures, i.e., TD0 and TD1. Thus, in one embodiment of this disclosure, in each tested candidate vector pair, each of the two vectors lies on a straight line in the image plane. In a special case, where the current picture is temporally located between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, the bilateral matching becomes a mirror-based bidirectional MV.

[0078] As shown in Figure 4A, template matching (described in document JVET-A1001) is used to derive motion information for the current CU by finding the closest match between a template in the current picture (a neighboring block above and / or to the left of the current CU) and a block in the reference picture (the same size as the template). The "Pattern matched motion vector derivation" section of document JVET-A1001 describes specific implementations of template matching and bilateral matching methods. One example discloses that bilateral matching operation is applied only when the "merge flag" is true, indicating that the "block merge" operation mode is selected. Here, the authors of document JVET-A1001 refer to the "merge mode" in the H.265 standard. Note that the template matching and bilateral matching methods described in JVET-A1001 can also be applied to other video encoding standards, and variations may occur in specific implementations.

[0079] Figure 5 is a flowchart illustrating the decoder-side motion vector refinement (DMVR) operation. According to document JVET-D0029, DMVR is applied under two conditions: 1) the prediction type is set to skip mode or merge mode, and 2) the prediction mode is biprediction. First, the initial motion vectors MV0 (from reference list L0) and MV1 (from reference list L1) are derived. The derivation process is performed according to the respective skip and merge operations. Here, the authors of document JVET-D0029 refer to the skip mode and merge mode of the H.265 standard. A description of these modes can be found in section 5.2.2.3 "Merge Motion Data Signaling and Skip Mode" of the book High Efficiency Video Coding (HEVC), Algorithms and Architectures, 2014, edited by v. Sze, M. Budagavi and GJ Sullivan. In H.265, when skip mode is used, for blocks, motion data is inferred instead of being explicitly signaled, and the predicted residual is zero, meaning the conversion coefficients are not transmitted. When merge mode is selected, motion data is also inferred, but the predicted residual is not zero, meaning the conversion coefficients are explicitly signaled.

[0080] The parsed index is parsed from the input video stream (510). The parsed index points to the best motion vector candidate from the constructed (520) list of MV candidates. The best motion vector candidate is then selected (530), and a template is obtained by weighted averaging (540). DMVR (550) is applied as follows. The block template is computed by summing the blocks referenced by MV0 and MV1, as described above with reference to Figure 3. Clipping is then performed. The template is used to find the refined motion vector MV0' around the initial motion vector MV0. The search region has an integer pixel resolution (points in the search space are integer sample distances apart from each other). The difference absolute sum (SAD) cost index is used to compare the template block with the new block pointed to by MV0'. The template is used to find the refined MV0'' around MV0'. The search region has half-pixel resolution (points in the search space are separated from each other by half the sample distance). The terms “pixel” and “particle” are used interchangeably herein. The same cost metric is used. The last two steps are repeated to find MV1''. The new bipredicted block is formed by adding up the blocks pointed to by MV0'' and MV1''. The blocks block_A' and block_B' pointed to by such refined motion vectors MV0'' and MV1'' are then averaged, for example, weighted averaged (560) to obtain the final prediction.

[0081] Figure 6 is a schematic diagram showing an example of local illumination compensation (LIC) that may be used for video coding and decoding according to one embodiment of the present disclosure. Local illumination compensation (LIC) is based on a linear model of illumination variation using a scaling factor "a" and an offset "b". LIC may be adaptively enabled or disabled for each inter-mode coded unit (CU). When LIC is applied to a CU, the least squares method may be used to derive parameters a and b using neighboring samples of the current CU and its corresponding reference sample. More specifically, as shown in Figure 6, subsampled (2:1 subsampled) neighboring samples of the CU and corresponding samples in the reference picture (identified by motion information of the current CU or sub-CU) are used. The LIC parameters are derived and applied separately for each prediction direction, where 2:1 subsampling means that every other pixel on the current CU boundary and the reference block is taken. Further details on the use of scaling factors / multiplicative weighting factors and offsets can be found in section "2.4.4. Local illumination compensation" of document JVET-A1001.

[0082] Figure 7 is a schematic diagram showing the decoder-side motion vector refinement (DMVR) iteration process performed on a reference picture RefPic0. The current picture includes a current block 710, for which a motion vector MV0' should be found in RefPic0 based on the motion vector MV0. A search space containing five integer positions is determined, and the blocks pointed to by the candidate positions are denoted as Ax. The output is the best match among the blocks Ax pointed to by the motion vector MV0'.

[0083] Whenever an explicit merge mode index is signaled, decoder-side motion vector refinement begins with a reference index and motion vector(s)(single or multiple) normatively estimated from the signaled index. If no explicit merge mode index is signaled, a set of initial motion vector candidates is evaluated in the decoder using a cost function, and the lowest-cost candidate is selected as the starting point for refinement. Thus, regardless of whether the decoder-side motion vector derivation method is based on predicted / reconstructed neighbor block boundary samples (commonly referred to as template matching (TM); see Figure 4A), or on bilateral matching through minimizing the difference between corresponding patches in reference lists L0 and L1 (commonly referred to as bilateral matching (BM) cost), or on the difference between the averaged versions of corresponding patches in reference lists L0 and L1 and the displacement in L0 / L1 (referred to as DMVR cost), there is a refinement search that must be performed around starting points that may be sub-pixel-precision motion vectors.

[0084] To evaluate the cost function, interpolation must be performed to derive the value at the center of sub-pixel precision based on the reference frame value at integer grid positions. The interpolation filter may be as simple as a bilinear interpolation filter or it may be a longer filter, such as a 2D DCT-based separable interpolation filter. To reduce the complexity of iteratively deriving interpolated samples for a given block at each position considered during refinement, an integer pixel distance grid of refinement points centered on one or more sub-pixel precision positions in L0 and / or L1 was proposed in another invention. In this case, only incremental interpolation is required, as new positions close to the current best-cost position are considered. After refinement of the integer pixel distance grid is complete, the best integer delta MV with respect to merge MV is obtained.

[0085] To further improve the compression gain, sub-pixel distance refinement can be performed. Half-pixel distance refinement requires interpolated samples at a half-pixel distance from the best integer-distance MV position(s)(s) in the reference frame(s). It is possible to perform sub-pixel precision refinement jointly between L0 and L1. In this case, the displacement in L0 with respect to the best integer-distance MV position in L0 is inverted to obtain the corresponding displacement in L1 with respect to the best integer-distance MV position in L1. It is also possible to perform sub-pixel precision refinement independently in L0 and L1.

[0086] Figure 8 is a conceptual diagram showing an example of integer and fractional sample locations according to one embodiment of the present disclosure. Referring to Figure 8, the pixel location indicated as "A" is an integer pixel, the half-pixel locations are indicated as b, h, and j, and all quarter-pixel locations are indicated as a, c, d, e, f, g, I, k, n, p, q, and r. Three planes are required to perform half-pixel refinement on an 8-point square pattern, and similarly, eight planes are required for quarter-pixel refinement according to the HEVC standard. Newer standards consider interpolation with an accuracy of 1 / 16 pixels.

[0087] If "A" is an integer pixel, then pixel positions a, b, and c require only horizontal interpolation, d, h, and n require only vertical interpolation, and all other points require both vertical and horizontal interpolation.

[0088] During integer pixel distance refinement and subpixel distance refinement, it is possible to use a different interpolation procedure than the one used for the final motion-compensated prediction process. For example, bilinear interpolation is a simpler interpolation that can be used for refinement, while the final motion-compensated prediction (MCP) may require a 2D DCT-based interpolation filter. Integer pixel grid data retrieved from the reference frame buffer should be held in a first buffer in internal memory (e.g., SRAM) until all interpolation is complete, to avoid retrieving this data multiple times from external memory (e.g., DDR). The integer pixel distance grid derived through the interpolation of integer pixel grid samples and required for integer pixel distance refinement from the subpixel-precision refinement center should be maintained in a second buffer in internal memory until all integer pixel distance refinement is complete. For half-pixel distance refinements around the best integer pixel distance position (which can be a subpixel position relative to an integer pixel grid), given that the distance between symmetric half-pixel refinement points on either side of the center is 1 integer pixel distance, two horizontal half-pixel distance positions share an interpolation plane, two vertical half-pixel distance positions share an interpolation plane, and four diagonal half-pixel distance positions share an interpolation plane.

[0089] If the interpolation for refinement is the same as the final MCP interpolation, it may be preferable to retain the integer pixel distance refinement data stored in a second buffer. This is because the final MCP interpolation calculation can be avoided when it is found that the best integer pixel distance position is the best position after sub-pixel refinement. If the interpolation differs between refinement and the final MCP, the second buffer can be overwritten with the interpolation for one of the three planes required for half-pixel distance refinement.

[0090] To reduce computational load and internal memory requirements, some prior art evaluates only four diagonal half-pixel distance positions, while others evaluate only horizontal and vertical half-pixel distance positions. Therefore, the required set of internal memory buffers ranges from 2 (when only diagonal half-pixel distance positions are evaluated and interpolation differs) to 5 (when evaluation is performed for all three half-pixel distance planes and interpolation is the same). Some prior art performs additional quarter-pixel and eighth-pixel distance refinements, but each position in these refinements requires individual interpolation, which is computationally prohibitive and does not justify the additional compression gain. When using higher tap filters for interpolation, horizontal filtering is performed first, the filtered results are stored in a temporary buffer, and then vertical filtering is performed.

[0091] Embodiments of the present disclosure provide a method and apparatus for deriving a subpixel distance delta motion vector around the best integer distance motion vector during decoder-side motion vector refinement / derivation, without explicit refinement, through an error surface technique that utilizes costs evaluated at integer pixel distance grid positions. A parametric error surface is fitted using these costs, and the minimum position of the parametric error surface is obtained by solving a system of equations. Aspects of the present disclosure utilize decoder-side subpixel distance refinement around integer distance pixel positions (which themselves may be subpixel positions with respect to an integer pixel grid) and address various variations of decoder-side motion vector refinement, such as template matching, bilateral matching, and template-based bilateral matching.

[0092] Given that decoder-side motion vector refinement / derivation is a normative aspect of the encoding system, the encoder would also have to employ the same error surface technique to avoid drift between encoder and decoder reconstruction. Thus, all aspects of all embodiments of this disclosure are applicable to both encoding and decoding systems.

[0093] In template matching, refined movement occurs only in references that start from the center of subpixel precision, either based on explicitly signaled merge indices or implicitly derived through cost evaluation.

[0094] In bilateral matching, refinement (with or without an averaged template) begins in reference lists L0 and L1, starting from the center of each subpixel precision, either based on an explicitly signaled merge index or implicitly derived through cost evaluation.

[0095] If bilateral matching costs are evaluated based on equal and opposite horizontal and vertical displacements in reference list L1 for a given horizontal and vertical displacement in reference list L0, then the positions shown in the drawings are assumed to correspond to reference list L0, and the positions in reference list L1 are derived by reversing the horizontal and vertical displacements with respect to the current iteration center in reference list L1.

[0096] Therefore, it is necessary to achieve a large portion of the coding gain of explicit subpixel distance refinement without increasing memory size and computational complexity.

[0097] Embodiment 1

[0098] Let N be the maximum number of normatively allowed integer one-pixel distance refinement iterations. The first integer distance refinement iteration begins at the refinement center of the aforementioned subpixel precision and proceeds until the center position has an evaluated cost function value lower than the evaluated cost function values ​​of a defined pair of one-pixel neighbor positions, or until N iterations have been completed. If the number of iterations N is reached and the center position is not the position with the minimum cost function value when compared to a pair of its one-pixel neighbor positions, error-face-based subpixel refinement is not performed, and the position with the minimum cost function value throughout all iterations is declared as the final delta motion vector from the decoder-side motion vector refinement process. If refinement ends at a center position with the minimum cost function value in a pair of one-pixel neighbor positions in the final iteration, the next error-face-based subpixel refinement procedure is applied.

[0099] Hereinafter, a pair of one-pixel neighborhood positions around the center C are defined as integer one-pixel distances to its left (L), top (T), right (R), and bottom (B). Figure 9 is a block diagram of a pair of one-pixel neighborhood positions around a central pixel for obtaining sub-pixel-accurate delta motion vector refinement in one or more reference frames according to one embodiment of the present disclosure. Referring to Figure 9, the origin of the 2D coordinate system is at C, and the coordinates at L, T, R, and B correspond to (-1,0), (0,1), (1,0), and (0,-1). The evaluated cost function values ​​at these five positions would be E(0,0), E(-1,0), E(0,1), E(1,0), and E(0,-1). The five-parameter error surface using a 2D parabolic function is defined as follows: E(x,y)=A*(x-x0) 2 +B*(y-y0) 2 +C (1) Here, E(x,y) is the evaluated cost function of the Cartesian coordinate (x,y), (x0,y0) is the Cartesian coordinate (corresponding) related to the subpixel displacement around the center (0,0) with the minimum error, C is the parameter corresponding to the error at this position, and A and B are error surface model parameters. A, B, and C are constant values ​​for calculating E(x,y).

[0100] On the other hand, considering that A, B, C, x0, and y0 are unknown parameters, (x0,y0) can be calculated (determined) by solving the five equations using the available evaluated cost function values ​​at the five positions (0,0), (-1,0), (0,-1), (1,0), and (0,1) as follows: x0=(E(-1,0)-E(1,0)) / (2*(E(-1,0)+E(1,0)-2*E(0,0))) (2) y0=(E(0,-1)-E(0,1)) / (2*(E(0,-1)+E(0,1)-2*E(0,0))) (3) x0 and y0 are coordinates related to subpixel motion vector displacements. E(-1,0), E(1,0), E(0,0), E(0,-1), and E(0,1) are integer distance costs corresponding to candidate integer motion vector displacements (-1,0), (1,0), (0,0), and (0,-1) with respect to the initial motion vector, respectively. The coordinates x0 and y0 can be scaled by a scaling factor N depending on the subpixel precision of the delta motion vector, where N=2, 4, 8, and 16 for pixel precisions of 1 / 2, 1 / 4, 1 / 8, and 1 / 16, respectively.

[0101] In another embodiment, equations (2) and (3) can be expressed as follows: x0=(E(-1,0)-E(1,0)) / (2*N*(E(-1,0)+E(1,0)-2*E(0,0))) (2') y0=(E(0,-1)-E(0,1)) / (2*N*(E(0,-1)+E(0,1)-2*E(0,0))) (3') Here, for example, N=1, 2, 4, and 8 for pixel accuracies of 1 / 2, 1 / 4, 1 / 8, and 1 / 16, respectively.

[0102] Note that, based on equations (2) and (3), x0 can be calculated (determined) according only to the positions (-1,0), (1,0), and (0,0). Similarly, y0 can be calculated (determined) according only to the positions (0,-1), (0,1), and (0,0).

[0103] Furthermore, it should be noted that if the evaluated costs (corresponding to the best integer distance displacement relative to the initial motion vector) for all four positions (-1,0), (0,-1), (1,0), and (0,1) relative to the center (0,0) are not available, the parametric error surface cannot be fitted. In such cases, the sub-pixel-precision delta displacement is not added to the best integer distance displacement.

[0104] The scaling factor N can be appropriately selected depending on the sub-pixel accuracy supported by the final motion compensation.

[0105] The subpixel motion vector displacement is limited to -0.5 and +0.5 in the x and y directions, respectively, by equations (1), (2), and (3). Since we are usually interested in an accuracy of 1 / 16 of a pixel, the two divisions above can be efficiently performed using only shift, compare, and increment operations.

[0106] In some embodiments, a method for refining a decoder-side motion vector includes determining a target integer motion vector displacement by comparing the integer distance costs corresponding to candidate integer motion vector displacements with respect to an initial motion vector; determining a subpixel motion vector displacement by performing calculations with respect to the integer distance costs; and determining a refined motion vector based on the target integer motion vector displacement, the subpixel motion vector displacement, and the initial motion vector.

[0107] In one embodiment, the method may further include determining whether a given motion vector displacement includes a target integer motion vector displacement before determining the subpixel motion vector displacement; if the given motion vector displacement includes a target integer motion vector displacement, determining the subpixel motion vector displacement by performing a calculation with respect to the integer distance cost. In another embodiment, the method may further include determining whether the evaluated cost is available at positions (-1,0), (0,-1), (1,0), and (0,1) relative to the target integer motion vector displacement before determining the subpixel motion vector displacement; if it is determined that the evaluated cost is available at positions (-1,0), (0,-1), (1,0), and (0,1) relative to the target integer motion vector displacement, determining the subpixel motion vector displacement by performing a calculation with respect to the integer distance cost.

[0108] In one embodiment, the method may further include calculating a refined motion vector based on the target integer motion vector displacement and the initial motion vector if the predetermined motion vector displacement does not include the target integer motion vector displacement. In another embodiment, the method may further include calculating a refined motion vector based on the target integer motion vector displacement and the initial motion vector if it is determined that at least one or more of the evaluated costs are not available at the positions (-1,0), (0,-1), (0,1), and (1,0) relative to the target integer motion vector displacement.

[0109] In one embodiment, determining the target integer motion vector displacement may include calculating the integer distance cost for each candidate integer motion vector displacement and selecting the candidate integer motion vector displacement corresponding to the lowest integer distance cost as the target integer motion vector displacement.

[0110] In one embodiment, the target integer motion vector displacement includes a first motion vector displacement corresponding to a reference picture list L0 and a second motion vector displacement corresponding to a reference picture list L1. The method may further include determining the first motion vector displacement by comparing it with the integer costs corresponding to candidate integer motion vector displacements corresponding to reference picture list L0, and determining the second motion vector displacement by inverting the first motion vector displacement.

[0111] It should be understood that the embodiments described are illustrative and not limiting. If the integer distance refinement process yields many of these evaluated cost function values ​​within a one-pixel neighborhood set, other 5-point, 6-point, and 9-point based error surface methods, such as those in the reference document, can be used.

[0112] The method of this embodiment requires only two internal memory buffers. The first buffer is configured to store an integer pixel grid from the reconstructed picture buffer, and the second buffer is configured to store an integer distance grid during integer distance refinement (which is used to obtain a parametric error surface) and can be used to overwrite with the final sub-pixel-precision motion-compensated prediction samples.

[0113] According to embodiments of this disclosure, high-precision subpixel-accurate delta motion vectors can be obtained at any subpixel-accuracy level without explicit refinement.

[0114] If the same cost function is used throughout all iterations, no additional interpolation or cost function evaluation calculations are required beyond the computations necessary for the integer distance refinement iterations.

[0115] Figure 10 is a simplified flowchart illustrating a method 1000 for obtaining subpixel-accurate delta motion vectors in one or more reference frames in a decoder-side motion vector refinement system, according to one embodiment of the present disclosure. Method 1000 may include the following steps:

[0116] Step 1001: Provide a processor. The processor may be one or more processing units (CPU, DSP) integrated into a video encoder and / or decoder, or program code integrated into video compression software, in order to perform the method described herein.

[0117] Step 1003: To determine an integer distance refined motion vector for each of the one or more reference frames, the processor sequentially and iteratively performs an integer 1-pixel distance refinement operation (iterative process) using a cost function to initiate a loop.

[0118] Step 1005: The processor determines the cost function value of the search center position for the current operation relative to the cost function values ​​of a pair of one-pixel neighbor positions. If the cost of the search center for the current iteration is determined to be the lowest, i.e., early termination from the sequential iteration loop (1005, yes):

[0119] Step 1011: Determine the subpixel distance refined motion vector around the last search center in each reference frame by calculating the position having the minimum value on a parametric error surface fitted with the cost function values ​​of the last search center and a set of one-pixel neighborhood positions around it;

[0120] Step 1013: For each reference frame, return the total refined motion vector as the sum of the determined integer distance refined motion vector and the determined subpixel distance refined motion vector.

[0121] This method further includes: if the cost function value of the search center's position is not the lowest cost function value, proceed to step (1005, no), then proceed to step (1007).

[0122] Step (1007): Determine if the current action is the last action (iteration). If the current action is the last action (1007, yes), return the refined motion vector corresponding to the position with the lowest cost value throughout all actions (1015). If the current action is not the last action (1007, no), update the center to the position with the lowest cost in that rat action and return to the loop and iteration step (1003).

[0123] Embodiment 2

[0124] There are various variations of Embodiment 1 if the cost function evaluation at integer distance locations used to derive the parametric error surface differs from the cost function evaluation performed during previous iterations of integer distance refinement.

[0125] For example, when bilateral matching is used, three types of refinement are possible. The first refinement type is to perform congruent refinement on both reference lists L0 and L1 so that the displacements in reference list L1 are equal to and opposite to the displacements in reference list L0 in both the horizontal and vertical directions (the first refinement type is called SBM_JOINT). The second refinement type is to perform independent refinement on both reference lists L0 and L1 for a common bilateral averaging template (the second refinement type is called TBM_INDEPENDENT). The third refinement type is to perform refinement on either reference list L0 or L1 for the bilateral averaging template and reverse the estimated horizontal and vertical displacements when obtaining the displacements in the other reference list (L1 if L0 was used for refinement, or L0 if L1 was used for refinement) (the third refinement type is called TBM_IN_1REF_NEG_IN_OTHER).

[0126] Because it is difficult to predict which iteration will result in an early termination of the sequential iteration loop (since the cost of the central position is lower than the cost of a pair of surrounding one-pixel neighbors), whenever the cost function evaluation at integer distance positions used to derive the parametric error surface differs from the cost function evaluation performed during previous iterations of the integer distance refinement, an additional set of cost function evaluations centered on the early termination center is performed using the required cost function.

[0127] Table 1 below gives some situations during bilateral matching where additional evaluation is required after an early termination during integer distance refinement. [Table 1]

[0128] During integer-distance refined sequential iterations, the refinement type SBM_JOINT was observed to perform best. When obtaining independent subpixel delta motion vectors at L0 and L1 using the refinement type TBM_INDEPENDENT, a small additional coding gain is achieved compared to equal, opposite displacements at L0 and L1. However, independent refinement at L0 and L1 requires independent cost evaluation at L0 and L1 positions and is therefore computationally more complex than the congruent equal, opposite displacement estimation option (SBM_JOINT refinement). In addition to SBM_JOINT refinement, early termination does not require additional cost function evaluation.

[0129] Figure 11 is a simplified flowchart illustrating a method 1100 according to one embodiment of the present disclosure for obtaining subpixel-precision delta motion vectors in one or more reference frames around each initial subpixel-precision refinement center(s) in a decoder-side motion vector refinement system. The method 1100 may include the following steps:

[0130] Step 1101: Provide a processor. The processor may be one or more processing units (CPU, DSP) integrated into a video encoder and / or decoder, or program code integrated into video compression software, in order to perform the method described herein.

[0131] Step 1103: For each of the one or more reference frames, the processor sequentially and iteratively performs an integer 1-pixel distance refinement operation (iterative process) using a cost function to determine an integer distance refinement motion vector.

[0132] Step 1105: Determine if the current action is the last action. If the current action is the last action (1105, yes):

[0133] Step 1111: Perform integer 1-pixel distance refinement using a second cost function to obtain independent sub-pixel delta motion vectors for each of the one or more reference frames.

[0134] If the current operation is not the last operation (1105, no), proceed to step 1107.

[0135] Step 1107: Determine whether the cost function value of the current search center position is the lowest cost function value for a pair of 1-pixel neighbor positions. If the cost function value of the search center position is the lowest cost function value (1107, yes), proceed to step 1111 (i.e., terminate the sequential iteration loop early). If the cost function value of the search center position is not the lowest cost function value (1107, no), proceed to step 1109.

[0136] Step 1109: Update the center to the position with the lowest cost value in that operation, and return to the loop to perform the next integer 1-pixel distance refinement operation.

[0137] Step 1111: Perform integer 1-pixel distance refinement using a second cost function to obtain independent sub-pixel delta motion vectors for each of the one or more reference frames.

[0138] Step 1113: Determine whether the last search center position has the lowest second cost function value among a set of 1-pixel neighbor positions to the last search center. If the last search center position has the lowest second cost function value (1113, yes):

[0139] Step 1115: Determine the subpixel distance-refined motion vectors around the best integer distance-refined position in each reference frame by calculating the position with the minimum value on the fitted parametric error surface using the second cost function value.

[0140] Step 1117: For each reference frame, return the total refined MV as the sum of the determined integer distance refined MV and the determined subpixel distance refined MV.

[0141] Embodiments of the present disclosure also provide an apparatus configured to perform the methods described above. The apparatus may be a combination of software and hardware. For example, encoding and / or decoding may be performed by a chip such as a general-purpose processor (CPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). However, embodiments of the present disclosure are not limited to programmable hardware implementations. Some embodiments of the present disclosure may be implemented using application-specific integrated circuits (ASICs) or combinations of the hardware components described above.

[0142] Encoding and / or decoding may also be implemented by program instructions or program code stored on a computer-readable medium. When the program instructions are executed by a processor or computer, they cause the processor or computer to perform the steps of the method described above. The computer-readable medium can be any medium on which the program code is stored, such as a DVD, CD, USB (flash) drive, hard disk, or server storage device available over a network.

[0143] Figure 12 is a block diagram of a device 1200 that can be used to implement various embodiments of the present disclosure. The device 1200 may be an encoding device 100 shown in Figure 1 and a decoding device 200 shown in Figure 2. Furthermore, the device 1200 may host one or more of the described elements. In some embodiments, the device 1200 includes one or more input / output devices such as a speaker, microphone, mouse, touchscreen, keypad, keyboard, printer, and display. The device 1200 may include one or more central processing units (CPUs) 1210, memory 1220, mass storage device 1230, video adapter 1240, and I / O interface 1260 connected to a bus. The bus is one or more of several bus architectures of any type, including a memory bus or memory controller, peripheral bus, video bus, etc.

[0144] The CPU 1210 may have any type of electronic data processor. The memory 1220 may have any type of system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or a combination thereof. In some embodiments, the memory 1220 may include ROM for use at startup and DRAM for program and data storage for use while a program is running. In some embodiments, the memory 1220 may be non-temporary. The mass storage device 1230 may include any type of storage device for storing data, programs, and other information and making such data, programs, and other information accessible via a bus. The mass storage device 1230 may include, for example, one or more solid-state drives, hard disk drives, magnetic disk drives, optical disk drives, etc., which, when executed by the CPU 1210, are configured to store program code that causes the CPU to execute the method described herein. The CPU 1210 can be configured to sequentially and iteratively perform the steps described in relation to Figures 10 and 11 over several operations. The CPU 1210 may include a reference selection unit configured to select a first reference picture pointed to by MV0 and a second reference picture pointed to by MV1. After selecting the reference pictures, the reference picture selection unit may determine whether the first or second reference picture is used to perform motion vector refinement.

[0145] The video adapter 1240 and the I / O interface 1260 provide interfaces for connecting external input / output devices to device 1200. For example, device 1200 may provide a SQL command interface to a client. As shown, examples of input and output devices include a display 1290 connected to the video adapter 1240 and any combination of a mouse / keyboard / printer 1270 connected to the I / O interface 1260. Other devices may be connected to device 1200, and additional or fewer interface cards may be used. For example, a serial interface card (not shown) may be used to provide a serial interface for a printer.

[0146] The device 1200 may also include one or more network interfaces 1250, including wired links such as Ethernet cables and / or wireless links to access nodes or one or more networks 1280. The network interfaces 1250 allow the device 1200 to communicate with remote units over the network 1280. For example, the network interfaces 1250 may provide communication to a database. In one embodiment, the device 1200 is coupled to a local area network or a wide area network for communication with data processing and other processing units, the Internet, remote storage facilities, and other remote devices. The device 1200 can be used to encode one or more picture blocks received from an input (e.g., a network interface) and / or to decode video pictures from a bitstream. The device 1200 may include a bitstream parser configured to extract compressed picture blocks from a bitstream, a motion vector refinement unit configured to obtain sub-pixel-precision delta motion vector refinement, and a reconstruction unit configured to perform block reconstruction based on the obtained motion vectors.

[0147] While certain features or aspects of this disclosure may be disclosed in relation to only one of several implementations or embodiments, such features or aspects may be combined with one or more other features or aspects of other implementations or embodiments so as may be desirable and advantageous for any given or particular use. Furthermore, where the terms “include,” “have,” “possess,” or other variations thereof are used in either the detailed description or the claims, such terms are intended to be inclusive, as is the term “include.” Also, the terms “exemplary,” “for example,” and “example” are intended simply as examples, not as best or optimal. The terms “combined” and “connected” may have been used with derived terms. It should be understood that these terms may have been used to indicate that two elements cooperate or interact with each other, whether in direct physical or electrical contact or not in direct contact with each other.

[0148] While specific aspects have been illustrated and described herein, those skilled in the art will understand that a variety of alternative and / or equivalent implementations may be used in place of the specific aspects illustrated and described without departing from the scope of this disclosure. This application is intended to cover any adaptation or modification of the specific aspects discussed herein.

[0149] The elements in the following claims are described in a specific order with corresponding marks, but unless the description of the claims otherwise implies a specific order for implementing some or all of those elements, they are not necessarily intended to be limited to being implemented in that specific order.

[0150] Many alternatives, modifications, and variations will be obvious to those skilled in the art in light of the above teachings. Of course, those skilled in the art will readily recognize that there are many uses of the present invention other than those described herein. The present invention has been described with reference to one or more specific embodiments, but those skilled in the art will recognize that many modifications can be made therewith without departing from the scope of the invention. It should be understood that within the scope of the appended claims and their equivalents, the present invention can be carried out in ways other than those specifically described herein.

[0151] The circuit described above may also be a single integrated chip. However, the present invention is not limited thereto, and the circuit may include different pieces or hardware, or a combination of hardware and software such as a general-purpose processor or DSP programmed with corresponding code.

[0152] The flowchart described above is intended to illustrate an example of a decoder-side motion vector refinement technique. Those skilled in the art may modify or combine the steps to implement the disclosure without departing from the scope of the disclosure.

[0153] In one aspect of this disclosure, a method is provided for obtaining subpixel-accurate delta motion vector refinement in one or more reference frames in a decoder-side motion vector refinement system. This method may include: Processor provided (1001); To determine an integer distance-refined motion vector for each of the one or more reference frames, a loop operation is initiated by sequentially and iteratively performing an integer 1-pixel distance-refined operation by the processor using a cost function (1003); The processor determines whether the cost function value at the search center's position is the lowest function value among a pair of 1-pixel neighbor positions (1005); If the cost function value of the search center's position is the lowest cost function value (1005, yes), terminate the loop operation; The subpixel distance refined motion vector around the search center in each reference frame is determined by calculating the position having the minimum value on a parametric error surface fitted with the cost function values ​​of the search center and a pair of one-pixel neighbor positions (1011); For each reference frame, return the subpixel position obtained using the parametric error plane (1013).

[0154] In one embodiment, the method further: if the cost function value of the search center position is not the lowest cost function value (1005, no); Determine if the current operation is the last operation (1007); If the current action is the last action (1007, yes): Returns the locations within each reference frame that have the lowest cost function (1015); If the current operation is not the last operation (1007, no): The processor updates the position of the search center for the current integer 1-pixel distance refinement operation with the position that has the lowest cost in the current operation (1009); Repeat the loop operation (1003, 1005, 1007, 1009).

[0155] In one embodiment, the parametric error plane includes five pixels positioned at the central pixel, surrounded by four peripheral pixels that are equidistant from the central pixel.

[0156] In one embodiment, the cost function is calculated by the following equation: E(x,y) = A*(x-x0)² + B*(y-y0)² + C; Here, E(x,y) is the evaluated cost function value of coordinate (x,y), x0 and y0 are coordinates relating to the subpixel displacement around the center (0,0) with the minimum error, C is a parameter relating to the error at coordinate (x0,y0), A and B are constant values, x and y are coordinates relating to the neighborhood position, and x and y are integers including -1, 0 and 1, respectively.

[0157] In one embodiment, the subpixel position is obtained by the following equation: x0=(E(-1,0)-E(1,0)) / (2*N*(E(-1,0)+E(1,0)-2*E(0,0))); y0=(E(0,-1)-E(0,1)) / (2*N*(E(0,-1)+E(0,1)-2*E(0,0))) Here E(x,y) = A*(x-x0)² + B*(y-y0)² + C; E(x,y) is the evaluated cost function value, where x0 and y0 are coordinates relating to the subpixel displacement around the center (0,0) with the minimum error, C is a parameter relating to the error at coordinate (x0,y0), A and B are constant values, where x and y are coordinates relating to the neighborhood position, where x and y are integers including -1, 0, and 1, respectively, and N is an integer of 1, 2, 4, or 8 for subpixel precision of 1 / 2, 1 / 4, 1 / 8, or 1 / 16.

[0158] In one embodiment, subpixel-accurate delta motion vector refinement is achieved by template matching.

[0159] In another aspect of this disclosure, an apparatus is provided for obtaining sub-pixel-precision delta motion vectors in one or more reference frames in a decoder-side motion vector refinement system. The apparatus includes a processing unit and a non-temporary computer-readable medium having computer-readable instructions internally for causing the processing unit to perform the following steps: Steps include: initiating a loop operation by sequentially iterating through N integer 1-pixel distance refinement operations using a cost function to determine an integer distance refinement motion vector for each reference frame, wherein the position of the search center in the current iteration is updated by the position with the lowest cost in the previous operation; The steps include determining whether the cost of the search center's location is lower than the cost of any of the pair of 1-pixel neighbor locations; If the cost of the search center's location is the lowest; Steps to terminate the loop operation; The steps include determining the subpixel distance refined motion vector around the search center in each reference frame by calculating the position having the minimum value on a parametric error surface fitted using the cost function values ​​of the search center and a pair of one-pixel neighbor positions; A step that returns the subpixel position obtained using the parametric error plane for each reference frame.

[0160] In one embodiment, the non-temporary computer-readable medium, which has computer-readable instructions internally, causes the processing unit to perform the following steps: If the cost function value of the search center's position is not the lowest cost function value (1005, no): the step is to determine whether the current operation is the last operation; If the current operation is the last operation: the step of returning the positions within each reference frame with the lowest cost function; If the current operation is not the last operation: The processor will The step of updating the position of the search center for the current integer 1-pixel distance refinement operation by the position with the lowest cost in the current refinement operation (1009); and repeating the loop operation.

[0161] In one embodiment, the cost function is calculated by the following equation: E(x,y) = A*(x-x0)² + B*(y-y0)² + C; Here, E(x,y) is the evaluated cost function value, x0 and y0 are coordinates relating to the subpixel displacement around the center (0,0) with the minimum error, C is a parameter relating to the error at coordinate (x0,y0), A and B are constant values, x and y are coordinates relating to the neighborhood position, and x and y are integers including -1, 0 and 1, respectively.

[0162] In one embodiment, the subpixel position is obtained by the following equation: x0=(E(-1,0)-E(1,0)) / (2*N*(E(-1,0)+E(1,0)-2*E(0,0))); y0=(E(0,-1)-E(0,1)) / (2*N*(E(0,-1)+E(0,1)-2*E(0,0))) Here E(x,y) = A*(x-x0)² + B*(y-y0)² + C; E(x,y) is the evaluated cost function value, where x0 and y0 are coordinates relating to the subpixel displacement around the center (0,0) with the minimum error, C is a parameter relating to the error at coordinate (x0,y0), A and B are constant values, where x and y are coordinates relating to the neighborhood position, where x and y are integers including -1, 0, and 1, respectively, and N is an integer of 1, 2, 4, or 8 for subpixel precision of 1 / 2, 1 / 4, 1 / 8, or 1 / 16.

[0163] Another aspect of this disclosure provides a method for obtaining sub-pixel-accurate delta motion vector refinement in one or more reference frames in reference list L0 and reference list 1 in a decoder-side motion vector refinement system. This method may include: To provide a processor (1101); To determine an integer distance refined motion vector for each of the one or more reference frames, a loop operation is initiated by sequentially and iteratively performing an integer 1-pixel distance refined operation by the processor using a first cost function (1103); Determine whether the current operation is the last operation (1105); If the current action is the last action (1105, yes): To obtain independent subpixel delta motion vectors for each reference frame, perform integer 1-pixel distance refinement using a second cost function (1111); If the current operation is not the last operation (1105, no); The processor determines whether the cost function value of the current search center position is the first lowest function value in a pair of one-pixel neighbor positions (1107); If the cost function value of the search center's position is the first lowest function value (1107, yes): Terminate the loop operation; To obtain independent subpixel delta motion vectors for each reference frame, perform integer 1-pixel distance refinement using a second cost function (1111); If the cost function value of the current operation's search center position is not the first lowest function value (1107, no): To update the center to the position with the first lowest cost for the current operation; and Repeat the loop operation (1103, 1105, 1107, 1109).

[0164] In one embodiment, the method may further include the following: The processor determines whether the second cost function value of the search center position of the obtained independent subpixel delta motion vector is the second lowest function value (1113); If the second cost function value is the lowest cost function value (1113, yes); The subpixel distance-refined motion vector around the search center in each reference frame is determined by calculating the position having the minimum value on a parametric error surface fitted using the cost function values ​​of the search center and a set of one-pixel neighbor positions (1115); The total refined motion vector is returned as the sum of the determined integer distance refined motion vector and the determined sub-pixel distance refined motion vector for each reference frame, obtained using the parametric error surface for each reference frame (1117); If the second cost function value is not the second lowest cost function value (1113, no); To return the refined motion vector corresponding to the position within each reference frame that has the second lowest cost function value (1119).

[0165] In one embodiment, the subpixel position is derived by the following equation: x0=(E(-1,0)-E(1,0)) / (2*N*(E(-1,0)+E(1,0)-2*E(0,0))) y0=(E(0,-1)-E(0,1)) / (2*N*(E(0,-1)+E(0,1)-2*E(0,0))) Here, E(x,y) = A*(x-x0)² + B*(y-y0)² + C; (x0,y0) corresponds to the subpixel displacement with respect to (0,0) with the smallest error, C corresponds to the error at (x0,y0), A and B are constants, (x,y) corresponds to the neighborhood position, x includes -1, 0, 1, and y includes -1, 0, 1.

[0166] In one embodiment, the first cost function and the second cost function are each calculated by the following equation: E(x,y) = A*(x-x0)² + B*(y-y0)² + C; Here, E(x,y) is the evaluated cost function value, x0 and y0 are Cartesian coordinates relating to subpixel displacements around the center (0,0) with the smallest error, C is a parameter relating to the error in the Cartesian coordinates (x0,y0), A and B are constant values, x and y are Cartesian coordinates relating to the neighborhood position, and x and y are integers including -1, 0 and 1, respectively.

[0167] In one embodiment, subpixel-accurate delta motion vector refinement is achieved by bilateral matching.

[0168] In one embodiment, bilateral matching includes performing joint refinement on both reference list L0 and reference list L1.

[0169] In one embodiment, bilateral matching involves performing independent joint refinements on both reference list L0 and reference list L1 in relation to a common bilateral averaging template.

[0170] In one embodiment, bilateral matching involves reversing estimated horizontal and vertical displacements when performing refinement on either reference list L0 or reference list L1 in relation to a common bilateral averaging template and obtaining displacements in the other reference list.

[0171] In summary, embodiments of the present disclosure relate to motion vector determination based on template matching for bidirectional motion vector estimation. Specifically, a block template is constructed as the average of blocks pointed to by the initial motion vector to be refined. Motion vector refinement is then performed by template matching in two different reference pictures. Matching is performed in each of the two reference pictures by finding the optimal (function-dependent minimum or maximum) matching function corresponding to the best matching block. The optimal is searched for zero-average template and zero-average candidate blocks (among the block positions pointed to by motion vector candidates in the search space). In other words, before performing function optimization, the average of the template is subtracted from the template and the average of the candidate blocks is subtracted from the candidate blocks. The predictor of the current block is then calculated as the weighted average of the best matching blocks in each reference picture.

Claims

1. A method for refining decoder-side motion vectors performed by a decoding device, wherein the method is: The steps include determining the target integer motion vector displacement by comparing the integer distance costs corresponding to candidate integer motion vector displacements with respect to the initial motion vector; A step of determining the subpixel motion vector displacement by performing a calculation on the integer distance cost, wherein the subpixel motion vector displacement is given by the following equation: x0=(E(−1,0)−E(1,0)) / (2×(E(−1,0)+E(1,0)−2×E(0,0))), y0=(E(0,−1)−E(0,1)) / (2×(E(0,−1)+E(0,1)−2×E(0,0))) Satisfying the conditions, Here, x0 and y0 are coordinates associated with the subpixel motion vector displacement with respect to the center (0,0), and E(-1,0), E(1,0), E(0,0), E(0,-1), and E(0,1) are the integer distance costs corresponding to the candidate integer motion vector displacements (-1,0), (1,0), (0,0), (0,-1), and (0,1), respectively, with respect to the initial motion vector, and step; A step of determining a refined motion vector based on the target integer motion vector displacement, a scaled subpixel motion vector displacement, and the initial motion vector, wherein the scaled subpixel motion vector displacement is obtained by scaling the subpixel motion vector displacement according to the subpixel precision of the refined motion vector, and including, method.

2. The method according to claim 1, wherein x0 is determined by performing at least one of the following operations on E(-1,0), E(1,0), and E(0,0): shift, compare, and increment.

3. The method according to claim 1 or 2, wherein y0 is determined by performing at least one of the following operations on E(0,-1), E(0,1), and E(0,0): shift, compare, and increment.

4. The method according to any one of claims 1 to 3, wherein the subpixel motion vector displacement is limited to between -0.5 pixels and +0.5 pixels.

5. A method for refining a decoder-side motion vector performed by an encoding device, wherein the method is: The steps include determining the target integer motion vector displacement by comparing the integer distance costs corresponding to candidate integer motion vector displacements with respect to the initial motion vector; A step of determining the subpixel motion vector displacement by performing a calculation on the integer distance cost, wherein the subpixel motion vector displacement is given by the following equation: x0=(E(−1,0)−E(1,0)) / (2×(E(−1,0)+E(1,0)−2×E(0,0))), y0=(E(0,−1)−E(0,1)) / (2×(E(0,−1)+E(0,1)−2×E(0,0))) Satisfying the conditions, Here, x0 and y0 are coordinates associated with the subpixel motion vector displacement with respect to the center (0,0), and E(-1,0), E(1,0), E(0,0), E(0,-1), and E(0,1) are the integer distance costs corresponding to the candidate integer motion vector displacements (-1,0), (1,0), (0,0), (0,-1), and (0,1), respectively, with respect to the initial motion vector, and step; A step of determining a refined motion vector based on the target integer motion vector displacement, a scaled subpixel motion vector displacement, and the initial motion vector, wherein the scaled subpixel motion vector displacement is obtained by scaling the subpixel motion vector displacement according to the subpixel precision of the refined motion vector, and including, method.

6. The method according to claim 5, wherein x0 is determined by performing at least one of the following operations on E(-1,0), E(1,0), and E(0,0): shift, compare, and increment.

7. The method according to claim 5 or 6, wherein y0 is determined by performing at least one of the following operations on E(0,-1), E(0,1), and E(0,0): shift, compare, and increment.

8. The method according to any one of claims 5 to 7, wherein the subpixel motion vector displacement is limited to between -0.5 pixels and +0.5 pixels.

9. One or more processors; A computer-readable storage medium coupled to one or more processors and storing a program for execution by the one or more processors. A decoding device having, wherein the programming, when executed by the one or more processors, configures the device to perform the method described in any one of claims 1 to 4. Decoding device.

10. One or more processors; A computer-readable storage medium coupled to one or more processors and storing a program for execution by the one or more processors. An encoding device having, wherein the programming, when executed by the one or more processors, configures the device to perform the method described in any one of claims 5 to 8. Encoding device.

11. A decoding device having a processing circuit for performing the method according to any one of claims 1 to 4.

12. An encoding device having a processing circuit for performing the method described in any one of claims 5 to 8.

13. A computer program that causes a computer or one or more processors to perform the method described in any one of claims 1 to 8.

14. A method for storing a bitstream, the method being: The steps include receiving a bitstream and storing it in one or more storage media, wherein the bitstream is obtained by performing an encoding process on video data, and the encoding process is: A step of obtaining a predicted block by performing a prediction based on a refined motion vector of a picture block, wherein the refined motion vector is determined according to a target integer motion vector displacement, a scaled subpixel motion vector displacement and an initial motion vector, the scaled subpixel motion vector displacement is obtained by scaling the subpixel motion vector displacement according to the subpixel precision of the refined motion vector, the target integer motion vector displacement is determined by comparing the integer distance costs corresponding to candidate integer motion vector displacements with respect to the initial motion vector, the subpixel motion vector displacement is determined by performing a calculation on the integer distance costs, and the subpixel motion vector displacement is given by the following equation: x0=(E(−1,0)−E(1,0)) / (2×(E(−1,0)+E(1,0)−2×E(0,0))), y0=(E(0,−1)−E(0,1)) / (2×(E(0,−1)+E(0,1)−2×E(0,0))) Satisfying the conditions, Here, x0 and y0 are the coordinates of the subpixel motion vector displacement with respect to the center (0,0), and E(-1,0), E(1,0), E(0,0), E(0,-1), and E(0,1) are the integer distance costs corresponding to the candidate integer motion vector displacements (-1,0), (1,0), (0,0), (0,-1), and (0,1), respectively, with respect to the initial motion vector, and step; The steps include: obtaining a residual block based on the aforementioned picture block and the aforementioned prediction block; The steps include: performing a transformation on the residual block to obtain a transformation coefficient; The steps include: performing quantization on the aforementioned conversion coefficients to obtain quantized conversion coefficients; The steps include encoding the quantized conversion coefficients into the bitstream and including, method.

Images