Sub-pixel accurate correction method based on error surface for motion vector correction at decoding end

Abstract

Given that decoder-side motion vector correction / derivation is a standard aspect of a decoding system, the encoder also needs to perform the same error surface technique to avoid any drift between the encoder reconstruction and the decoder reconstruction. Therefore, aspects of all embodiments are applicable to both encoding and decoding systems. In template matching, the correction movement occurs only in the reference, starting from sub-pixel accurate centers, where the sub-pixel accurate centers are derived based on the explicitly indicated fusion index or implicitly derived through cost evaluation. In bilateral matching (with or without average template), the correction starts from the corresponding sub-pixel accurate centers in the reference lists L0 and L1, where the sub-pixel accurate centers are derived based on the explicitly indicated fusion index or implicitly derived through cost evaluation.

Description

[0001] This application is a divisional application. The application number of the original application is 201980040320.0, and the original application date is June 20, 2019. The entire contents of the original application are incorporated into this application by reference.

[0002] Related Applications Cross-Application

[0003] This application claims the benefit of Indian Provisional Patent Application No. 201831024666, filed on July 2, 2018, entitled “AN ERROR SURFACE BASED SUB-PIXEL ACCURATE REFINEMENT METHOD FOR DECODER SIDE MOTION VECTOR REFINEMENT”, the entire contents of which are incorporated herein by reference. Background Art

[0004] Current hybrid video codecs (such as H.264 / AVC or H.265 / HEVC) use compression including predictive decoding. The images of a video sequence are divided into pixel blocks, which are then decoded. Instead of decoding the blocks pixel by pixel, the coded pixels that are spatially or temporally adjacent to the entire block are used to predict the block. The encoder only further processes the difference between the block and its predicted block. The further processing typically includes transforming the block pixels into coefficients in the transform domain. The coefficients can then be further compressed by quantization and further compressed by entropy decoding to form a code stream. The code stream also includes any signaling information that can decode the encoded video. For example, the signaling information may include encoding-related settings, such as the size of the input image, the frame rate, the quantization step indication, the prediction applied to the image block, etc. The decoded signaling information and the decoded signal are sorted within the code stream in a manner known to the encoder and decoder. This enables the decoder to parse the decoded signaling information and the decoded signal.

[0005] Temporal prediction exploits the temporal correlation between video images (also called frames). Temporal prediction is also called inter-frame prediction because it exploits the correlation between different video frames (between frames) for prediction. Therefore, the block being encoded (also called the current block) is predicted based on one or more previously encoded images (also called one or more reference images). The reference image is not necessarily in the display order of the video sequence, but is the image before the current image in which the current block is located. The encoder can encode the images in a decoding order that is different from the display order. A co-located block in a reference image can be determined as a prediction block for the current block. A co-located block is a block located in the reference image at the same position as the current block in the current image. This prediction is accurate for static image areas, i.e. image areas that have not moved from one image to another.

[0006] In order to obtain a prediction that takes into account movement, i.e. a motion compensated prediction, motion estimation is usually employed when determining the prediction for the current block. Thus, the current block is predicted from a block in a reference image, which block is located at a distance given by the motion vector from the location of the co-located block. In order to enable the decoder to determine the same prediction for the current block, the motion vector may be indicated in the bitstream. In order to further reduce the signaling overhead caused by indicating a motion vector for each block, the motion vector itself may be estimated. Motion vector estimation may be performed based on motion vectors of neighboring blocks in the spatial and / or temporal domain.

[0007] The prediction of the current block can be calculated using one reference image or by weighting the predictions obtained from two or more reference images. The reference image can be an adjacent image, i.e. an image that immediately precedes and / or immediately follows the current image in display order, because the adjacent image is most likely to be similar to the current image. However, in general, the reference image can also be any other image in the codestream that precedes or follows the current image in display order (decoding order). This is advantageous, for example, in the case of occlusion and / or non-linear motion in the video content. Therefore, the reference image identifier can also be indicated in the codestream.

[0008] A special mode of inter prediction is bidirectional prediction, in which two reference images are used to generate a prediction of the current block. In particular, the two predictions determined for the corresponding two reference images are combined into a prediction signal for the current block. Bidirectional prediction can produce a more accurate prediction of the current block than unidirectional prediction (i.e., prediction made using only a single reference image). A more accurate prediction results in a smaller difference (also called a "residual") between the pixels of the current block and the prediction, which can be encoded more efficiently, i.e., compressed into a shorter bitstream. Typically, more than two reference images can be used to find corresponding more than two reference blocks to predict the current block, i.e., multi-reference inter prediction can be applied. Therefore, the term "multi-reference prediction" includes bidirectional prediction as well as prediction using more than two reference images.

[0009] To provide more accurate motion estimation, the resolution of the reference image can be increased by interpolating pixels between pixels. Fractional pixel interpolation can be performed by weighted averaging of the nearest pixels. For example, in the case of half-pixel resolution, bilinear interpolation is usually used. Other fractional pixels are calculated as the average of the nearest pixels weighted by the inverse of the distance between the corresponding nearest pixel and the predicted pixel.

[0010] Motion vector estimation is a computationally complex task that computes the similarity between the current block and the corresponding predicted block pointed to by the candidate motion vector in the reference image. Typically, the search region consists of M×M pixels of the image, and each pixel position of the M×M candidate positions is tested. The test consists of computing a similarity measure between an N×N reference block C and a block R at a test candidate position located in the search region. The sum of absolute difference (SAD) is a measure often used for simplicity and is given by:

[0011]

[0012] In the above formula, x and y define the candidate positions within the search area, while the indices i and j represent the pixels within the reference block C and the candidate block R. The candidate positions are often referred to as block displacements or block offsets, reflecting that block matching is represented as a displacement of the reference block within the search area, and the similarity between the reference block C and the overlapping portion of the search area is calculated. In order to reduce complexity, the number of candidate motion vectors is usually reduced by restricting them to a certain search space. For example, the search space can be defined by the number and / or position of pixels around the position in the reference image that corresponds to the position of the current block in the current image. After calculating the SAD of all M×M candidate positions x and y, the best matching block R is the block at the position with the lowest SAD, which corresponds to the maximum similarity with the reference block C. On the other hand, the candidate motion vector can be defined by a list of candidate motion vectors consisting of motion vectors of neighboring blocks.

[0013] Motion vectors are usually determined at least in part at the encoding end and indicated to the decoder within the encoded bitstream. However, motion vectors can also be derived at the decoder side. In this case, the current block is not available at the decoder side and cannot be used to calculate the similarity with the block pointed to by the candidate motion vector in the reference image. Therefore, a template constructed from the pixels of the decoded block is used instead of the current block. For example, decoded pixels adjacent to the current block can be used. The advantage of this motion estimation is reduced signaling: motion vectors are derived in the same way at the encoder and decoder side, so no signaling is required. On the other hand, the accuracy of this motion estimation may be lower.

[0014] In order to achieve a trade-off between accuracy and signaling overhead, motion vector estimation can be divided into two steps: motion vector derivation and motion vector correction. For example, motion vector derivation may include selecting a motion vector from a candidate list. Such a selected motion vector may be further corrected by searching in a search space, etc. The search in the search space is based on calculating a cost function for each candidate motion vector, i.e., a cost function for each candidate position of the block pointed to by the candidate motion vector.

[0015] X.Chen, J.An, J.Zeng published the paper JVET-D0029: Decoding-side motion vector correction based on bilateral template matching (the paper can be found at: http: / / phenix.it-sudparis.eu / jvet / site) presents motion vector correction, where a first motion vector at integer pixel resolution is found and further refined by searching at half-pixel resolution in a search space around the first motion vector. A block template-based bidirectional motion vector search is employed.

[0016] Motion vector estimation is a key feature of modern video codecs since its efficiency in terms of quality, rate and complexity directly affects the efficiency of video encoding and decoding. Summary of the invention

[0017] The present invention relates to encoding and decoding of video, and in particular to determining motion vectors.

[0018] In a first aspect of the present application, a method for obtaining sub-pixel accurate incremental motion vectors in one or more reference frames in a decoding end motion vector correction system is provided. The one or more reference frames are around corresponding one or more initial sub-pixel accurate correction centers. The method may include the following steps:

[0019] Iteratively performing a plurality of integer 1-pixel distance correction operations (iterations) using a cost function to determine an integer distance corrected motion vector for each reference frame, wherein the search center is updated to the position with the lowest cost in the operation (iteration) after the operation (iteration); determining that an early exit from the iteration loop occurs because the cost of the center position in a given operation (iteration) is lower than the cost of a set of 1-pixel neighboring positions around the center position, or determining that a predetermined number of operations (iterations) has been reached. In the event of an early exit, the method may include: determining a sub-pixel distance corrected motion vector around the last search center in each reference by calculating the position with the minimum value on the fitted parameter error surface using the cost function value of the last search center and the set of 1-pixel neighboring positions around the last search center; and returning a total corrected motion vector as the sum of the determined integer distance corrected motion vector and the determined sub-pixel distance corrected motion vector for each reference frame. In the event of a predetermined number of operations (iterations), the method may include returning a corrected motion vector corresponding to the position with the minimum cost function value in all operations (iterations).

[0020] In an implementation of the first aspect, the cost function is performed on the joint correction in the reference lists L0 and L1 so that the displacement in the reference list L1 is equal to the displacement in the reference list L0 and opposite in the horizontal and vertical directions (the cost function is called SBM_JOINT).

[0021] In a second aspect of the present application, a method for obtaining sub-pixel accurate incremental motion vectors in one or more reference frames in a decoding end motion vector correction system is provided. The one or more reference frames are obtained from a plurality of reference frames around corresponding one or more initial sub-pixel accurate correction centers, and the method may include the following steps:

[0022] Iteratively performing a plurality of integer 1-pixel distance correction operations (iterations) using a first cost function to determine an integer distance corrected motion vector for each reference frame, wherein a search center is updated after the operation (iteration) to a position with the lowest cost in the operation (iteration); determining that an early exit from an iterative loop occurs because the cost of the center position in a given operation (iteration) is lower than the cost of a set of 1-pixel neighboring positions around the center position, or determining that a predetermined number of operations (iterations) has been reached.

[0023] The cost function values ​​of the last search center and the set of 1-pixel neighboring positions of the last search center are calculated using the second cost function: if the last search center position has the lowest second cost function value compared to the second cost function value of the set of 1-pixel neighboring positions of the last search center, then the sub-pixel distance corrected motion vector around the best integer distance corrected position in each reference is determined by calculating the position with the smallest value on the fitted parameter error surface using the second cost function value; and the total corrected motion vector is returned as the sum of the determined integer distance corrected motion vector and the determined sub-pixel distance corrected motion vector for each reference frame. If the last search center position does not have the lowest second cost function value compared to the second cost function value of the set of 1-pixel neighboring positions of the last search center, the method includes: returning the corrected motion vector corresponding to the position with the best second cost function value in each reference frame.

[0024] In an implementation of the second aspect, the first cost function is SBM_JOINT, and the second cost function is TBM_INDEPDENDENT, which is defined as a cost function associated with performing independent corrections on a common bilateral average template in reference lists L0 and L1.

[0025] The method can be used both in the case of using template matching and in the case of using bilateral matching.

[0026] In another aspect of the present invention, the method disclosed herein may be implemented as instructions stored in a non-transitory computer-readable medium, which may be read and executed by a processor to perform the steps of the method.

[0027] In some aspects of the present invention, a method for decoding-side motion vector correction includes: determining a target integer motion vector displacement by comparing integer distance costs corresponding to candidate integer motion vector displacements relative to an initial motion vector; determining a sub-pixel motion vector displacement by calculating the integer distance cost; and calculating a corrected motion vector based on the target integer motion vector displacement, the sub-pixel motion vector displacement, and the initial motion vector.

[0028] Compared with conventional techniques, the present invention provides a variety of benefits. For example, embodiments of the present invention determine to exit the iteration loop early by comparing the cost of the center position in a given iteration operation with the cost of a set of 1-pixel neighboring positions around the center position. By exiting the iteration loop early, unnecessary calculations can be reduced or eliminated.

[0029] In addition, the device that can implement the above method can be a combination of software and hardware. For example, encoding and / or decoding can be performed by an integrated circuit (semiconductor device or chip) such as a general purpose processor (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). However, the embodiments of the present invention are not limited to implementation in programmable hardware. The embodiments of the present invention can be implemented in a dedicated integrated circuit, or by a combination of one or more of the CPU, DSP, FPGA and ASIC components. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Exemplary embodiments are described in detail below with reference to the accompanying drawings, in which:

[0031] Figure 1 A block diagram of an exemplary structure of an encoder for encoding a video signal provided by an embodiment of the present invention;

[0032] Figure 2 A block diagram of an exemplary structure of a decoder for decoding a video signal provided by an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of an exemplary template matching suitable for bidirectional prediction;

[0034] Figure 4A is a schematic diagram of an exemplary template matching applicable to unidirectional prediction and bidirectional prediction;

[0035] Figure 4B is a schematic diagram of an exemplary bilateral matching applicable to unidirectional prediction and bidirectional prediction;

[0036] Figure 5 A flow chart of a possible implementation of a motion vector search;

[0037] Figure 6 is a schematic diagram of an example of local illumination compensation applied to video decoding;

[0038] Figure 7 A schematic diagram of an example of motion vector correction at a decoding end;

[0039] Figure 8 is a schematic diagram of an example of a sub-pixel position;

[0040] Fig. 9 A block diagram of a set of 1-pixel neighboring positions around a center pixel for obtaining a sub-pixel accurate incremental motion vector correction in one or more reference frames according to an embodiment of the present invention;

[0041] Fig.10 A simplified flow chart of a method for obtaining sub-pixel accurate incremental motion vector correction in one or more reference frames in a decoding-side motion vector correction system provided for some embodiments of the present invention;

[0042] Fig.11 A simplified flow chart of a method for implementing an embodiment of the present invention;

[0043] Fig.12 is a block diagram of an apparatus that can be used to implement various embodiments of the present invention. DETAILED DESCRIPTION

[0044] Embodiments of this aspect are directed to improving template matching applied in motion vector correction. Specifically, template matching is applied to zero-mean templates and zero-mean candidate blocks even though the mean of the best matching block found is not further adjusted (by local illumination control).

[0045] Template matching is used to find the best first motion vector and the best second motion vector pointing to the first reference image and the second reference image, respectively. Template matching is performed for each reference image by performing template matching on a position given by an initial motion vector in a predetermined search space, wherein the initial motion vector may be derived by or indicated to the decoder.

[0046] Template matching may be performed based on a block template derived from the block pointed to by the initial motion vector.

[0047] For example, this template matching for finding the best matching block to obtain the prediction value of the current block can be adopted in a hybrid video encoder and / or decoder. For example, it can be advantageously applied to encoders and / or decoders such as HEVC. In particular, further development of HEVC or new codecs / standards can use embodiments of the present invention.

[0048] Figure 1An encoder 100 is shown, comprising an input terminal 102 for receiving input image pixels of a frame or image of a video stream and an output terminal 172 for generating an encoded video code stream. As used herein, the term "frame" in the present invention is also referred to as an image. It should be noted that the present invention is also applicable to the interlaced field of video. Typically, an image includes m×n pixels, corresponding to image pixels, and may include one or more color components. As used herein, the following description refers to pixels as luminance pixels. However, it should be noted that the motion vector search of the present invention can be applied to any color component including chrominance or components of color spaces such as RGB. In addition, it may be advantageous to perform motion vector estimation on only one component and apply the estimated motion vector to multiple or all components.

[0049] The input blocks to be decoded do not have to be of the same size. An image may include blocks of different sizes, and the block grids of different images may also be different.

[0050] In an exemplary embodiment, the encoder 100 is used to perform prediction, transformation, quantization and entropy decoding on a video stream. The transformation, quantization and entropy decoding are performed by the transformation unit 106, the quantization unit 108 and the entropy coding unit 170 respectively to generate an encoded video code stream.

[0051] The video stream may include a plurality of frames, each of which is divided into intra-frame coded blocks or inter-frame coded blocks of a certain size. For example, the blocks of the first frame of the video stream are intra-coded by the intra-frame prediction unit 154. The intra-frame is encoded using only information within the same frame, so that the intra-frame can be independently decoded, and the intra-frame can provide an entry point for random access in the code stream. Blocks of other frames of the video stream can be inter-coded by the inter-frame prediction unit 144, that is, information in a previously coded frame (reference frame) is used to reduce temporal redundancy, thereby predicting each block of the inter-frame coded frame based on a block in the reference frame. The mode selection unit 160 is used to select between a block 155 of a frame processed by the intra-frame prediction unit 154 or a block 145 of a frame processed by the inter-frame prediction unit 144. The mode selection unit 160 also controls the parameters of the intra-frame prediction or the inter-frame prediction. In order to refresh the image information, an intra-frame coded block can be provided in an inter-frame coded frame. Furthermore, intra frames containing only intra-coded blocks may be regularly inserted into the video sequence in order to provide entry points for decoding, ie points where the decoder can start decoding without information in previously encoded frames.

[0052] The intra-frame estimation unit 152 and the intra-frame prediction unit 154 are units for performing intra-frame prediction. Specifically, the intra-frame estimation unit 152 can also derive a prediction mode according to the situation of the original image, and the intra-frame prediction unit 154 provides a corresponding prediction value for differential coding, that is, a pixel predicted using the selected prediction mode. In order to perform spatial prediction or temporal prediction, the coding block can also be processed by the inverse quantization unit 110 and the inverse transform unit 112 to provide an inverse transform block 113. The reconstruction unit 114 combines the inverse transform block 113 with the prediction block 165 to provide a reconstruction block 115, which is provided to the loop filter unit 120 to further improve the quality of the decoded image. Then, the filter block forms a reference frame and is then stored in the decoded image buffer 130. The inverse quantization unit 110, the inverse transform unit 112, the reconstruction unit 114 and the loop filter 120 constitute a part of the decoder (decoding loop). The advantage of such a decoding loop (decoder) at the encoding end is that a reference frame identical to the reference image reconstructed at the decoding end is generated. Therefore, the encoder and the decoder operate in a corresponding manner. The term “reconstruction” herein refers to obtaining a reconstructed block 115 by adding the prediction block 165 to the inverse transform (decoded residual) block 113 .

[0053] The encoder 100 further comprises an inter-frame estimation unit 142 which receives a current frame or image to be inter-coded and an image block 101 of one or more reference frames from a decoded image buffer 130. Motion estimation is performed by the inter-frame estimation unit 142, while motion compensation is performed by the inter-frame prediction unit 144. Motion estimation is used to obtain a motion vector and a reference frame according to a certain cost function (e.g. also using the original image to be coded). For example, the motion estimation (inter-frame estimation) unit 142 may provide an initial motion vector estimate. The initial motion vector may then be indicated (transmitted as a signal) directly in the code stream in the form of a motion vector or as an index indicating a motion vector candidate in a candidate list, wherein the candidate list is constructed in the same way at the encoder and decoder side according to a predetermined rule. Motion compensation then derives a prediction value for the current block as a translation of a block co-located with the current block in a reference frame to a reference block in a reference frame, i.e. the prediction value for the current block is derived by the motion vector. The inter-frame prediction unit 144 outputs a prediction block 145 of the current block, wherein the prediction block 145 minimizes a cost function. For example, the cost function may be the difference between the current block to be encoded and its prediction block, i.e., the cost function minimizes the residual block 105. For example, the minimization of the residual block is based on calculating the sum of absolute differences (SAD) between all pixels (pixel points) of the current block and the candidate blocks in the candidate reference image. In general, any other similarity metric may be used, such as mean square error (MSE) or structural similarity metric (SSIM).

[0054] The cost function may also be the number of bits necessary to encode such inter-blocks and / or the distortion caused by such encoding. Thus, the rate-distortion optimization process may be used to decide on motion vector selection and / or in general to decide on encoding parameters such as whether to use inter-prediction or intra-prediction for a block, and which settings to use.

[0055] The intra-frame estimation unit 152 and the inter-frame prediction unit 154 receive as input an image block 101 of a current frame or image to be intra-coded and one or more reference pixels 117 from a reconstructed area of ​​the current frame. Then, the intra-frame prediction describes the pixels of the current block of the current frame according to a function of the reference pixels of the current frame. The intra-frame prediction unit 154 outputs a prediction block of the current block, wherein the prediction block is conducive to minimizing the difference between the current block to be coded and its prediction block, that is, minimizing the residual block. For example, the residual block can be minimized according to a rate-distortion optimization process. In particular, the prediction block is obtained as a directional interpolation of the reference pixel points. The direction can be determined by rate-distortion optimization and / or by calculating the above-mentioned similarity metric related to inter-frame prediction.

[0056] The inter-frame estimation unit 142 receives as input the current frame or image to be inter-coded and blocks or more generally image pixels of two or more decoded images 231. The inter-frame prediction then describes the current image pixels of the current frame according to the motion vectors to refer to the image pixels of the reference image. The inter-frame prediction unit 144 outputs one or more motion vectors 145 for the current image pixels, wherein the reference image pixels pointed to by the motion vectors are conducive to minimizing the difference between the current image pixels to be coded and their reference image pixels, i.e., minimizing the residual image pixels. The inter-frame prediction unit 144 then provides a prediction value for the current block for differential coding.

[0057] Then, the transform unit 106 transforms the difference between the current block and its prediction block (i.e., the residual block 105) to generate transform coefficients 107. The transform coefficients 107 are quantized by the quantization unit 108 and entropy decoded by the entropy coding unit 170. The encoded image data 171 (i.e., the encoded video code stream) generated thereby includes intra-frame coded blocks and inter-frame coded blocks and corresponding signaling information (such as mode indication, indication of motion vectors and / or intra-frame prediction direction). The transform unit 106 can apply a linear transform, such as a discrete Fourier transform (DFT), a fast Fourier transform (FFT), or a discrete cosine transform (DCT). The advantage of the spatial frequency domain transform is that the resulting coefficients 107 usually have higher values ​​at lower frequencies. Therefore, after the significant coefficient scan (such as a zigzag scan) and quantization, the resulting value sequence usually has some larger values ​​at the beginning and ends with a string of zeros. This further improves the coding efficiency. The quantization unit 108 performs the actual lossy compression by reducing the resolution of the coefficient values. The entropy coding unit 170 then assigns binary codewords to the coefficient values ​​to generate a bitstream. The entropy coding unit 170 also performs signaling information ( Figure 1 (not shown) for encoding.

[0058] Figure 2 A video decoder 200 is shown. The video decoder 200 includes a decoded image buffer 230, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, wherein the inter-frame prediction unit 244 and the intra-frame prediction unit 254 are block prediction units. The decoded image buffer 230 is used to store at least one (for unidirectional prediction) or at least two (for bidirectional prediction) reference frames reconstructed from the encoded video code stream, wherein the reference frame is different from the current frame (currently decoded frame) of the encoded video code stream. The intra-frame prediction unit 254 is used to generate a prediction block, wherein the prediction block is used to estimate the block to be decoded. The intra-frame prediction unit 254 is used to generate the prediction block according to the reference pixel points obtained from the decoded image buffer 230.

[0059] The decoder 200 is used to decode the encoded video code stream generated by the video encoder 100. Preferably, the decoder 200 and the encoder 100 generate the same prediction block for the corresponding block to be encoded / decoded. The characteristics of the decoded image buffer 230 and the intra-frame prediction unit 254 are similar to Figure 1 The features of the decoded picture buffer 130 and the intra prediction unit 154 are similar.

[0060] The video decoder 200 also includes units that are also present in the video encoder 100, for example, an inverse quantization unit 210, an inverse transform unit 212, and a loop filter unit 220, which correspond to the inverse quantization unit 110, the inverse transform unit 112, and the loop filter unit 120 of the video encoder 100, respectively.

[0061] The entropy decoding unit 204 is used to decode the received coded video code stream, and thus obtain the quantized residual transform coefficients 209 and signaling information. The quantized residual transform coefficients 209 are provided to the inverse quantization unit 210 and the inverse transform unit 212 to generate a residual (inverse transform) block. The residual block is added to the prediction block 265 in the reconstruction unit 214, and the addition is sent to the loop filter unit 220 to obtain the decoded video. The frame of the decoded video can be stored in the decoded picture buffer 230 and used as the decoded picture 231 for inter-frame prediction.

[0062] generally, Figure 1 and Figure 2 The intra prediction units 154 and 254 may use reference pixels from an already encoded region to generate a prediction signal for a block to be encoded or decoded.

[0063] The entropy decoding unit 204 receives the coded bitstream 171 as its input. Typically, the bitstream is first parsed, i.e., signaling parameters and residuals are extracted from the bitstream. Typically, the syntax and semantics of the bitstream are defined by a standard so that the encoder and decoder can communicate with each other. As described in the background section above, the coded bitstream includes not only the prediction residual. In the case of motion compensated prediction, a motion vector indication is also encoded into the bitstream and parsed from the bitstream on the decoder side. The motion vector indication can be given by providing a reference image and motion vector coordinates of the motion vector. So far, decoding of the complete motion vector has been considered. However, it is also possible to encode only the difference between the current motion vector and the previous motion vector in the bitstream. This method can exploit the redundancy between the motion vectors of neighboring blocks.

[0064] In order to efficiently decode reference images, the H.265 codec (ITU-T, H265, H Series: Audiovisual and Multimedia Systems: High Efficiency Video Coding) provides a reference image list that assigns a list index to the corresponding reference frame. The reference frame is then indicated by including the corresponding assigned list index in the codestream. Such a list can be defined in the standard, or indicated at the beginning of the video or at the beginning of a set of many frames. It should be noted that two reference image lists are defined in H.265, called reference lists L0 and L1. The reference image is then indicated in the codestream by indicating the reference list (L0 or L1) and indicating the index in the list associated with the desired reference image. Providing two or more lists may have a better compression effect. For example, reference list L0 can be used for unidirectional inter-frame prediction strips and bidirectional inter-frame prediction strips, while reference list L1 can be used only for bidirectional inter-frame prediction strips. However, in general, the present invention is not limited to any content of lists L0 and L1.

[0065] Reference lists L0 and L1 may be defined in the standard and may be fixed. However, greater flexibility in encoding / decoding may be achieved by indicating these reference lists at the beginning of a video sequence. Thus, the encoder may configure lists L0 and L1 with specific reference images sorted according to indices. Reference lists L0 and L1 may have the same fixed size. Typically, there may be more than two lists. Motion vectors may be indicated directly by coordinates in the reference image. Alternatively, H.265 also provides that a list of candidate motion vectors may be constructed and the index in the list associated with a specific motion vector may be transmitted.

[0066] The motion vector of the current block is usually associated with the motion vector of the neighboring block in the current image or the previously decoded image. This is because each neighboring block may correspond to the same moving object with similar motion, and the motion of the object is unlikely to change suddenly over time. Therefore, using the motion vector in the neighboring block as a prediction value reduces the size of the indicated motion vector difference. The motion vector predictor (MVP) is usually derived from the motion vector that has been encoded / decoded in the spatial neighboring block or the temporal neighboring block in the co-located image. In H.264 / AVC, the motion vector predictor is derived by obtaining the component-by-component median of three spatial neighboring motion vectors. Using this method, signaling of the prediction value is not required. The temporal MVP in the co-located image is only considered in the so-called temporal direct mode of H.264 / AVC. The H.264 / AVC direct mode is also used to derive other motion data besides motion vectors. Therefore, these direct modes are more related to the concept of block fusion in HEVC. In HEVC, the implicit method of deriving MVP is replaced by a technique called motion vector competition, which explicitly indicates the MVP in the MVP list for motion vector derivation. The variable decoding quadtree block structure in HEVC can cause a block to have multiple neighboring blocks that have motion vectors as potential MVP candidates. Taking the left neighbor block as an example, in the worst case, when the 64×64 luma decoding tree block is not further divided and the left neighbor block is divided to the maximum depth, the 64×64 luma prediction block can have 16 4×4 luma prediction blocks on the left.

[0067] Advanced Motion Vector Prediction (AMVP) was introduced to modify the motion vector competition to account for this flexible block structure. During the development of HEVC, the initial AMVP design was greatly simplified to achieve a good trade-off between decoding efficiency and a design that is conducive to implementation. The initial design of AMVP included five MVPs from three different predictors: three motion vectors from spatial neighboring blocks, the median of the three spatial predictors, and the scaled motion vector from the co-located temporal neighboring block. In addition, the predictor list was modified to ensure minimal signal overhead by reordering the most likely motion predictor to place the first position and removing redundant candidates. The final design of the AMVP candidate list construction includes the following two MVP candidates: (a) at most two spatial candidate MVPs, derived from five spatial neighboring blocks; (b) when the two spatial candidate MVPs are not available or are the same, one temporal candidate MVP, derived from two temporal co-located blocks; and (c) zero motion vector when the spatial candidate, the temporal candidate, or both are not available. Details on motion vector determination can be found in the book High Efficiency Video Coding (HEVC): Algorithms and Architectures, published by V. Sze et al. (eds.), 2014 and published by Springer, especially Chapter 5, which is incorporated herein by reference.

[0068] In order to further improve the motion vector estimation without further increasing the signaling overhead, further correction of the motion vectors derived at the encoding end and provided in the bitstream can be beneficial to the improvement. Motion vector correction can be performed on the decoder side without the assistance of the encoder. The encoder in the decoding loop can use the same correction to obtain the corresponding motion vector. Motion vector correction is performed in a search space including integer pixel positions and fractional pixel positions of the reference image. For example, the fractional pixel position can be a half-pixel position, a quarter-pixel position or other fractional position. The fractional pixel position can be obtained from the integer (full pixel) position by interpolation such as bilinear interpolation.

[0069] In the bidirectional prediction of the current block, the two prediction blocks obtained using the first motion vector of reference list L0 and the second motion vector of reference list L1 are combined into a single prediction signal. The adaptability of bidirectional prediction to the original signal can be better than unidirectional prediction, resulting in less residual information and potentially higher compression efficiency.

[0070] At the decoder side, the current block is not available because it is being decoded, and for motion vector correction a template is used which is an estimate of the current block and is constructed from already processed (ie encoded at the encoder and decoded at the decoder) image parts.

[0071] First, an estimate of a first motion vector MV0 and an estimate of a second motion vector MV1 are received as input on the decoder 200 side. On the encoder 100 side, the motion vector estimates MV0 and MV1 may be obtained by block matching and / or by searching in a candidate list (such as a fusion list), wherein the candidate list is formed by motion vectors of blocks adjacent to the current block (in the same image or in adjacent images). Then, MV0 and MV1 are indicated to the decoding end in the bitstream. However, it should be noted that, in general, the first determination stage on the encoder side may also be performed by template matching, which will help reduce signaling overhead.

[0072] On the decoder 200 side, motion vectors MV0 and MV1 are advantageously obtained based on the information in the code stream. Motion vectors MV0 and MV1 are indicated directly, or differentially, and / or an index in a motion vector list (fusion list) is indicated. However, the present invention is not limited to indicating motion vectors in the code stream. Instead, according to the present invention, the motion vector can be determined by template matching already existing in the first stage of motion vector estimation, corresponding to the operation of the encoder. Template matching of the first stage (motion vector derivation) can be performed based on a search space different from the search space of the second motion vector correction stage. In particular, motion vector correction can be performed on a search space with a higher resolution (i.e., a shorter distance between search positions).

[0073] An indication of two reference images RefPic0 and RefPic1 pointed to by the respective MV0 and MV1 is also provided to the decoder. The reference images are stored in decoded image buffers on the encoder and decoder sides as a result of previous processing (i.e., respective encoding and decoding). One of these reference images is selected by search for motion vector correction. A reference image selection unit of the apparatus for determining the motion vector is used to select a first reference image pointed to by MV0 and a second reference image pointed to by MV1. After the selection, the reference image selection unit determines whether the first reference image or the second reference image is used for motion vector correction. In order to perform motion vector correction, a search area in the first reference image is defined around the candidate position pointed to by the motion vector MV0. Candidate search space positions within the search area are analyzed to find the block that is most similar to the template block by performing template matching in the search space and determining a similarity measure (such as the sum of absolute difference (SAD)). The position of the search space represents the position matched by the upper left vertex of the template. As mentioned above, the upper left vertex is only a convention, and any point of the search space (such as the center point) can generally be used to represent the matching position.

[0074] Figure 4A An alternative template matching is shown that is also applicable to unidirectional prediction. For more information, see JVET-A1001, in particular Section 2.4.6: "Motion Vector Derivation for Pattern Matching" by Jianle Chen et al., "Algorithm Description of Joint Exploration Test Model 1", available at http: / / phenix.it-sudparis.eu / jvet / . The template in this template matching method is determined as the pixels in the current frame that are adjacent to the current block. As shown in JVET-A1001, Figure 1 As shown, the reconstructed pixel points adjacent to the upper boundary and the left boundary of the current block can be taken, which is called an "L-shaped template".

[0075] According to document JVET-D0029 (which is incorporated by reference), decoder-side motion vector refinement (DMVR) has as input the initial motion vectors MV0 and MV1 pointing to two corresponding reference images RefPict0 and RefPict1. These initial motion vectors are used to determine the search space in RefPict0 and RefPict1, respectively. In addition, using the motion vectors MV0 and MV1, a template is constructed according to the blocks A and B pointed to by MV0 and MV1, respectively, as follows:

[0076] Template = function (blockA, blockB).

[0077] The function may be a pixel cropping operation combined with a pixel-by-pixel weighted summation. The template is then used to perform template matching in a search space determined according to MV0 and MV1 in the corresponding reference images Refpic 0 and Refpic 1. The cost function used to determine the best template match in the corresponding search space is SAD (template, block candA'), where block candA' is the candidate decoding block pointed to by the candidate MV in the search space spanning the position given by MV0. Figure 3 The determination of the best matching block A' and the resulting modified motion vector MV0' are shown. Figure 3 As shown, the same template is used to find the best matching block B' and the corresponding motion vector MV1' pointing to block B'. ​​In other words, after constructing the template based on the blocks A and B pointed to by the initial motion vectors MV0 and MV1, the template is used to search RefPic0 and RefPic1 to find the modified motion vectors MV0' and MV1'.

[0078] Motion vector derivation technology is sometimes also called frame rate up-conversion (FRUC). The initial motion vectors MV0 and MV1 can usually be indicated in the bitstream to ensure that the encoder and decoder can use the same initial point for motion vector correction. Alternatively, the initial motion vector can be obtained by providing an initial candidate list including one or more initial candidates. For each initial motion vector, a modified motion vector is determined, and the modified motion vector with the lowest cost function is selected at the end.

[0079] As mentioned above, the motion vector derivation mode of template matching is a special fusion mode based on the frame rate up conversion (FRUC) technology. In this mode, the motion information of the block is derived at the decoding end. According to the specific implementation described in the document JVET-A1001 ("Algorithm Description of JointExploration Test Model 1", website: http: / / phenix.it-sudparis.eu / jvet / ), when the fusion flag is true, the FRUC flag is indicated for the CU or PU. When the FRUC flag is false, the fusion index is indicated and the normal fusion mode is used. When the FRUC flag is true, an additional FRUC mode flag is indicated to indicate the method used to derive the motion information of the block (bilateral matching or template matching).

[0080] To summarize, in the motion vector derivation process, the initial motion vector is first derived for the entire prediction unit (PU) based on bilateral matching or template matching. First, a list of MV candidates is generated, for example, the list can be a fusion list of PUs. Check the list and select the candidate with the smallest matching cost as the starting point (initial motion vector). Then, a local search is performed around the starting point based on bilateral matching or template matching, and one or more motion vectors (MV) with the smallest matching cost are used as the MV of the PU. Then, the motion information is further corrected using the derived PU motion vector as the starting point. The terms "prediction unit (PU)" and "coding unit (CU)" are used interchangeably in this article to describe a block of pixels within an image (frame).

[0081] like Figure 4B As shown, bilateral matching (described in document JVET-A1001) is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference images. Assuming that the motion trajectory is continuous, the motion vectors MV0 and MV1 pointing to the two reference blocks between the current image and the two reference images should be proportional to the temporal distance (i.e., TD0 and TD1). Therefore, in one embodiment of the present aspect, in each candidate vector pair tested, the two corresponding vectors are on a straight line in the image plane. As a special case, when the current image is temporally located between the two reference images and the temporal distance from the current image to the two reference images is the same, bilateral matching becomes a mirror-based bidirectional MV.

[0082] like Figure 4A As shown, template matching (described in document JVET-A1001) is used to derive the motion information of the current CU by finding the closest match between the template in the current image (the top block and / or the left neighboring block of the current CU) and the block in the reference image (the same size as the template). The "Derivation of Motion Vectors by Pattern Matching" section of document JVET-A1001 describes the specific implementation of template matching and bilateral matching methods. An example discloses that the bilateral matching operation is only applied when the "fusion flag" is true, indicating that the "block fusion" operation mode is selected. Here, the author of document JVET-A1001 refers to the "fusion mode" of the H.265 standard. It should be noted that the template matching and bilateral matching methods described in JVET-A1001 can also be applied to other video decoding standards, resulting in changes in the specific implementation methods.

[0083] Figure 5Flowchart of the decoder-side motion vector refinement (DMVR) operation. According to document JVET-D0029, DMVR is applied in the following two cases: (1) the prediction type is set to skip mode or fusion mode; (2) the prediction mode is bidirectional prediction. The first initial motion vector MV0 (from reference list L0) and MV1 (from reference list L1) are derived. The derivation process is performed according to the corresponding skip and fusion operations. Here, the authors of document JVET-D0029 refer to the skip mode and fusion mode of the H.265 standard. The description of these modes can be found in: Section 5.2.2.3 "Fused Motion Data Indication and Skip Mode" of the book "High Efficiency Video Coding (HEVC), Algorithms and Architectures" published by v. Sze, M. Budagavi and G. S. Sullivan (eds.) in 2014. In H.265, if skip mode is used, motion data is inferred for a block instead of explicitly indicated, and the prediction residual is zero, that is, the transform coefficients are not transmitted. If merge mode is selected, motion data is also inferred, but the prediction residual is not zero, that is, the transform coefficients are explicitly indicated.

[0084] Parse (510) the parsed index from the input video stream. The parsed index points to the best motion vector candidate for the constructed (520) MV candidate list. Then, the best motion vector candidate is selected (530) and the template is obtained by weighted averaging (540). DMVR is applied (550) as follows. The block template is calculated by adding the blocks referenced by MV0 and MV1, as described above in conjunction with Figure 3As described. Then cropping is performed. Use the template to find the modified motion vector MV0' around the initial motion vector MV0. The search area is an integer pixel resolution (each point in the search space is spaced apart by an integer pixel distance). Use the Sum of Absolute Difference (SAD) cost metric to compare the template block and the new block pointed to by MV0'. Use the template to find the modified MV0". around MV0'. The search area is a half-pixel resolution (each point in the search space is spaced apart by a half-pixel distance). The terms "pel" and "pixel" are used interchangeably in this article. Use the same cost metric. Repeat the last two steps to find MV1". A new bidirectional prediction block is formed by adding the blocks pointed to by MV0". Then, take an average of the blocks block_A' and block_B' pointed to by such modified motion vectors MV0" and MV1", for example, a weighted average (560) to obtain the final prediction.

[0085] Figure 6 A schematic diagram of an example of local illumination compensation (LIC) that can be used for video encoding and decoding provided by one embodiment of the present invention. Local Illumination Compensation (LIC) is based on a linear model of illumination variation, using a scaling factor "a" and an offset "b". For each coding unit (CU) of inter-frame mode decoding, LIC can be adaptively enabled or disabled. When LIC is applied to a CU, the least squares error method can be used to derive parameters a and b using the neighboring pixels of the current CU and their corresponding reference pixels. More specifically, as Figure 6 As shown, the sub-sampled (2:1 sub-sampling) neighboring pixels of the CU and the corresponding pixels in the reference image (identified by the motion information of the current CU or sub-CU) are used. LIC parameters are derived and applied separately for each prediction direction. Here, sub-sampling 2:1 means sub-sampling every two pixels on the current CU boundary and the reference block. More details on the use of scaling factors / multiplication weighting factors and offsets can be found in "Section 2.4.4 Local Illumination Compensation" of the document JVET-A1001.

[0086] Figure 7Schematic diagram of decoder-side motion vector refinement (DMVR) iterations for reference image RefPic0. The current image includes a current block 710 for which a motion vector MV0' should be found based on the motion vector MV0 in RefPic0. A search space of 5 integer positions is determined; the block pointed to by the candidate position is called Ax. The output is the best match for the block Ax pointed to by the motion vector MV0'.

[0087] Whenever an explicit merge mode index is indicated, the decoder-side motion vector correction starts with one or more motion vectors and a reference index canonically derived from the indicated index. When no explicit merge mode index is indicated, the initial motion vector candidates are evaluated at the decoder side using a cost function, and the candidate with the lowest cost is selected as the starting point for correction. Therefore, regardless of the decoder-side motion vector derivation method based on predicted / reconstructed neighboring block boundary pixels (commonly known as Template Matching (TM), the reference Figure 4A Whether it is based on minimizing the difference between the corresponding connected areas in reference list L0 and reference list L1 (usually called bilateral matching (BM) cost) (or) based on the difference between the average version of the corresponding connected areas in reference list L0 and reference list L1 and the shift of L0 / L1 (called DMVR cost), a modified search is required around the starting point which can be a sub-pixel accurate motion vector.

[0088] To evaluate the cost function, interpolation needs to be performed to derive the values ​​of the sub-pixel exact centers from the values ​​of the reference frame at integer grid positions. The interpolation filter can be as simple as a bilinear interpolation filter or it can be a longer filter such as a separable interpolation filter based on a two-dimensional DCT. In order to reduce the complexity of repeatedly deriving the interpolated pixel points of the block at each position considered during correction, another invention has proposed an integer pixel distance grid of correction points in L0 / or L1 centered at one or more sub-pixel exact positions. Taking advantage of this, only incremental interpolation needs to be performed when considering new positions close to the current best cost position. After correction of the integer pixel distance grid, an integer incremental MV that is best relative to the fused MV is obtained.

[0089] To further improve the compression gain, sub-pixel distance correction can be performed. Half-pixel distance correction requires the interpolated pixel to be at a position half a pixel away from one or more best integer distance MV positions in one or more reference frames. Sub-pixel accurate correction can be performed jointly between L0 and L1, where the displacement in L0 relative to the best integer distance MV position in L0 is inverted to obtain the corresponding displacement in L1 relative to the best integer distance MV position in L1. Sub-pixel accurate correction can also be performed independently in L0 and L1.

[0090] Figure 8 A conceptual diagram of an example of integer pixel positions and fractional pixel positions provided by an embodiment of the present invention. Figure 8 , pixel positions denoted as "A" are integer pixels; half-pixel positions are denoted as b, h, and j; all quarter-pixel positions are denoted as a, c, d, e, f, g, I, k, n, p, q, and r. To perform half-pixel correction using an 8-dot square pattern, 3 planes are required, similarly, quarter-pixel correction requires 8 planes according to the HEVC standard. New standards are considering 1 / 16 pixel accurate interpolation.

[0091] When "A" is an integer pixel, pixel positions a, b and c only need horizontal interpolation, pixel positions d, h and n only need vertical interpolation, and all other points need both vertical and horizontal interpolation.

[0092] In the integer pixel distance correction and sub-pixel distance correction processes, a different interpolation process from the interpolation used for the final motion compensated prediction process may be used. For example, bilinear interpolation is a simpler interpolation that can be used for correction, while the final motion compensated prediction (MCP) may require an interpolation filter based on a two-dimensional DCT. The integer pixel grid data obtained from the reference frame buffer needs to be kept in a first buffer in an internal memory (e.g., SRAM) until all interpolations can be completed to avoid re-acquiring the data from an external memory (e.g., DDR) multiple times. The integer pixel distance grid required for integer pixel distance correction starting from the sub-pixel accurate correction center derived by interpolation of the integer pixel grid pixels needs to be kept in a second buffer in the internal memory until all integer pixel distance corrections are completed. For the half-pixel distance correction around the best integer pixel distance position (which can be a sub-pixel position relative to the integer pixel grid), given that the distance between the symmetrical half-pixel correction points on either side of the center is separated by 1 integer pixel, 2 horizontal half-pixel distance positions share an interpolation plane, 2 vertical half-pixel distance positions share an interpolation plane, and 4 diagonal half-pixel distance positions share an interpolation plane.

[0093] When the interpolation used for correction is the same as the final MCP interpolation, it may be preferred to retain the integer pixel distance correction data stored in the second buffer, because when the best integer pixel distance position becomes the best position after sub-pixel correction, the final MCP interpolation calculation can be avoided. When the interpolation between the correction and the final MCP is different, the second buffer may be overwritten with the interpolation of one of the three planes required for the half-pixel distance correction.

[0094] To reduce computational load and internal storage requirements, some prior arts evaluate only 4 diagonal half-pixel distance positions, while other prior arts evaluate only horizontal and vertical half-pixel distance positions. Therefore, the required internal memory buffer groups range from 2 (when only diagonal half-pixel distance positions are evaluated and the interpolation is different) to 5 (when all three half-pixel distance planes are evaluated and the interpolation is the same). Although some prior arts perform additional quarter-pixel distance and eighth-pixel distance corrections, each position in these corrections requires a separate interpolation, which is computationally infeasible and not commensurate with the additional compression gain. When interpolation uses a more tapped filter, horizontal filtering is performed first, the filtering results are stored in a temporary buffer, and then vertical filtering is performed.

[0095] Embodiments of the present invention provide methods and apparatus for deriving sub-pixel distance incremental motion vectors around an optimal integer distance motion vector during a decoder-side motion vector correction / derivation process, without explicit correction, by using an error surface technique using costs evaluated at integer pixel distance grid positions. A parameter error plane is fitted using these costs, and the minimum position of the parameter error plane is obtained by solving a set of simultaneous equations. Various aspects of the present invention utilize decoder-side pixel distance correction around integer distance pixel positions (which themselves can be sub-pixel positions relative to an integer pixel grid), and handle different variants of decoder-side motion vector correction, such as template matching, bilateral matching, and template-based bilateral matching.

[0096] Since decoding-side motion vector correction / derivation is a standard aspect of a decoding system, the encoder also needs to perform the same error plane technique to avoid any drift between the encoder reconstruction and the decoder reconstruction. Therefore, aspects of all embodiments of the present invention are applicable to both encoding and decoding systems.

[0097] In template matching, correcting shifts only occur in the reference, starting from a sub-pixel accurate center derived based on an explicitly indicated fusion index or implicitly derived through cost evaluation.

[0098] In bilateral matching (with or without average template), the refinement starts from the corresponding sub-pixel accurate centers in reference lists L0 and L1, where the sub-pixel accurate centers are derived based on explicitly indicated fusion indices or implicitly derived through cost evaluation.

[0099] It should be noted that when evaluating the bilateral matching cost of given horizontal and vertical displacements in reference list L0 based on equal and opposite horizontal and vertical displacements in reference list L1, it is assumed that the position shown in the figure corresponds to reference list L0, and the position in reference list L1 is obtained by inverting the displacement in the horizontal and vertical directions in reference list L1 relative to the center of the current iteration.

[0100] Therefore, it is desirable to realize most of the decoding gain of explicit sub-pixel distance correction without increasing the memory size and computational complexity.

[0101] Example 1

[0102] Let N be the maximum number of integer 1-pixel distance correction iterations allowed by the specification. The first integer distance correction iteration starts from the corrected sub-pixel exact center described above and continues until the estimated cost function value of the center position is lower than the estimated cost function value of a predetermined set of 1-pixel neighboring positions, or until N iterations have been performed. When N iterations are reached and the center position is not the position with the minimum (lowest) cost function value compared to its set of 1-pixel neighboring positions, the error surface-based sub-pixel correction is not performed, and the position with the minimum (lowest) cost function value in all iterations is declared as the final incremental motion vector of the decoding-side motion vector correction process. When the correction is exited, the center position has the minimum (lowest) cost function value in the existing iterative set of 1-pixel neighboring positions, and the following error surface-based sub-pixel correction process is applied.

[0103] In the following, let the set of 1-pixel neighboring positions around the center C be at integer 1-pixel distances to its left (L), above (T), right (R), and below (B). Fig. 9 A block diagram of a set of 1-pixel neighboring positions around a center pixel for obtaining a sub-pixel accurate incremental motion vector correction in one or more reference frames is provided in accordance with an embodiment of the present invention. Fig. 9 , the origin of the two-dimensional coordinate system is at C, and the coordinates of L, T, R, and B correspond to (–1, 0), (0, 1), (1, 0), and (0, –1). The evaluation cost function values ​​of these five positions are E(0, 0), E(–1, 0), E(0, 1), E(1, 0), and E(0, –1). The five-parameter error surface using the two-dimensional parabolic function is defined as follows:

[0104] E(x, y) = A*(x – x0) 2+B*(y–y0) 2 +C (1)

[0105] Where E(x, y) is the estimated cost function of the Cartesian coordinate (x, y); (x0, y0) is the Cartesian coordinate with the minimum (lowest) error associated (corresponding) with the sub-pixel displacement relative to the center (0, 0); C is the parameter corresponding to the error at this position; A and B are the error surface model parameters. A, B, C are constant values ​​for calculating E(x, y).

[0106] On the other hand, considering that A, B, C, x0 and y0 are unknown parameters, using the available estimated cost function values ​​at 5 locations (0, 0), (-1, 0), (0, -1), (1, 0) and (0, 1) to solve the 5 equations, (x0, y0) can be calculated (determined) as follows:

[0107] x0=(E(–1,0)–E(1,0)) / (2*(E(–1,0)+E(1,0)–2*E(0,0))) (2)

[0108] y0=(E(0,–1)–E(0,1)) / (2*(E(0,–1)+E(0,1)–2*E(0,0))) (3)

[0109] x0 and y0 are coordinates associated with sub-pixel motion vector displacements. E(–1, 0), E(1, 0), E(0, 0), E(0, –1), and E(0, 1) are integer distance costs corresponding to candidate integer motion vector displacements (–1, 0), (1, 0), (0, 0), (0, –1), and (0, 1), respectively, relative to the initial motion vector. Coordinates x0 and y0 may be scaled by a scaling factor N according to the sub-pixel precision of the incremental motion vector, where, for example, N is 2, 4, 8, and 16 for sub-pixel precisions of 1 / 2, 1 / 4, 1 / 8, and 1 / 16, respectively.

[0110] In another embodiment, equations (2) and (3) can be expressed as:

[0111] x0=(E(–1,0)–E(1,0)) / (2*N*(E(–1,0)+E(1,0)–2*E(0,0))) (2’)

[0112] y0=(E(0,–1)–E(0,1)) / (2*N*(E(0,–1)+E(0,1)–2*E(0,0))) (3’)

[0113] Where, for example, for pixel precisions of 1 / 2, 1 / 4, 1 / 8, and 1 / 16, N is 1, 2, 4, and 8, respectively.

[0114] According to equations (2) and (3), it should be noted that x0 can be calculated (determined) only from the positions (-1, 0), (1, 0) and (0, 0). Similarly, y0 can be calculated (determined) only from the positions (0, -1), (0, 1) and (0, 0).

[0115] It should also be noted that the parametric error surface cannot be fitted when the estimated costs for all four positions (-1, 0), (0, -1), (1, 0), and (0, 1) relative to the center (0, 0) (corresponding to the best integer distance displacement relative to the initial motion vector) are not available. In such cases, the sub-pixel accurate incremental displacement is not added to the best integer distance displacement.

[0116] The scaling factor N can be appropriately selected according to the sub-pixel accuracy supported by the final motion compensation.

[0117] According to equations (1), (2) and (3), the sub-pixel motion vector displacements are bounded by –0.5 and +0.5 in the x and y directions, and since accuracy up to 1 / 16 pixel is usually of interest, this can effectively be divided into 2 with only shift, compare and increment operations required.

[0118] In some embodiments, a method for decoding-side motion vector correction includes: determining a target integer motion vector displacement by comparing integer distance costs corresponding to candidate integer motion vector displacements relative to an initial motion vector; determining a sub-pixel motion vector displacement by calculating the integer distance cost; and calculating a corrected motion vector based on the target integer motion vector displacement, the sub-pixel motion vector displacement, and the initial motion vector.

[0119] In one embodiment, the method may further include: determining whether the predetermined motion vector displacement includes the target integer motion vector displacement before determining the sub-pixel motion vector displacement; and in the case where the predetermined motion vector displacement includes the target integer motion vector displacement: determining the sub-pixel motion vector displacement by calculating the integer distance cost. In another embodiment, the method may further include: determining whether the evaluation costs at positions (-1, 0), (0, -1), (1, 0) and (0, 1) relative to the target integer motion vector displacement are available before determining the sub-pixel motion vector displacement; and in the case where the evaluation costs at positions (-1, 0), (0, -1), (1, 0) and (0, 1) are available, determining the sub-pixel motion vector displacement by calculating the integer distance cost.

[0120] In one embodiment, the method may further include: in the case where the predetermined motion vector displacement does not include the target integer motion vector displacement: calculating the revised motion vector based on the target integer motion vector displacement and the initial motion vector. In another embodiment, the method may further include: in the case where it is determined that at least one or more evaluation costs at positions (-1, 0), (0, -1), (0, 1) and (1, 0) are not available relative to the target integer motion vector displacement, calculating the revised motion vector based on the target integer motion vector displacement and the initial motion vector.

[0121] In one embodiment, determining the target integer motion vector displacement may include: calculating an integer distance cost for each candidate integer motion vector displacement; and selecting the candidate integer motion vector displacement corresponding to the minimum integer distance cost as the target integer motion vector displacement.

[0122] In one embodiment, the target integer motion vector displacement includes a first motion vector displacement corresponding to the reference picture list L0, and a second motion vector displacement corresponding to the reference picture list L1. The method may further include: determining the first motion vector displacement by comparing integer costs corresponding to candidate integer motion vector displacements corresponding to the reference picture list L0; and determining the second motion vector displacement by inverting the first motion vector displacement.

[0123] It should be understood that the described embodiments are exemplary and non-restrictive, and other 5-point, 6-point, and 9-point error surface methods in the references may be used if the integer distance correction process generates multiple of these evaluation cost function values ​​within a 1-pixel neighboring set.

[0124] The method according to this embodiment requires only two internal memory buffers. The first buffer is used to store the integer pixel grid from the reconstructed image buffer; the second buffer is used to store the integer distance grid during integer distance correction (for obtaining the parameter error surface) and can be used to predict pixel coverage with the final sub-pixel accurate motion compensation.

[0125] According to an embodiment of the present invention, a high-precision sub-pixel accurate incremental motion vector is obtained without explicit correction at any sub-pixel accuracy level.

[0126] When the same cost function is used in all iterations, no additional interpolation or cost function evaluation computation is required beyond that required for the integer distance correction iterations.

[0127] Fig.10 A simplified flowchart of a method 1000 for obtaining sub-pixel accurate incremental motion vectors in one or more reference frames in a decoding-side motion vector correction system is provided for one embodiment of the present invention. The method 1000 may include the following steps:

[0128] Step 1001: Provide a processor. The processor may be one or more processing units (CPU, DSP) integrated in a video encoder and / or decoder, or a program code integrated in a video compression software for executing the method described herein.

[0129] Step 1003: The processor iteratively performs an integer 1-pixel distance correction operation (iteration) using a cost function to determine an integer distance corrected motion vector for each of the one or more reference frames, thereby starting a loop.

[0130] Step 1005: The processor determines the cost function value of the search center position of the current operation based on the cost function value of the 1-pixel adjacent position set. If the cost of the search center of the current iteration is determined to be the lowest, that is, the iteration loop is exited early (1005, yes):

[0131] Step 1011: Determine the sub-pixel distance corrected motion vector around the last search center in each reference by calculating the position with the minimum value on the fitted parameter error surface using the cost function values ​​of the last search center and the 1-pixel neighboring positions around the last search center;

[0132] Step 1013: Return the total corrected motion vector as the sum of the determined integer distance corrected motion vector and the determined sub-pixel distance corrected motion vector for each reference frame.

[0133] The method further includes: if the cost function value of the search center position is not the lowest cost function value (1005, No), executing step (1007).

[0134] Step (1007): Determine whether the current operation is the last operation (iteration). If the current operation is the last operation (1007, yes), return the modified motion vector corresponding to the position with the lowest cost value among all operations (1015). If the current operation is not the last operation (1007, no): update the center to the position with the lowest cost in the last operation, return to the loop and repeat step (1003).

[0135] Example 2

[0136] Different variations of Embodiment 1 exist when the cost function evaluations at integer distance locations used to derive the parametric error surface are different from the cost function evaluations performed during earlier iterations of the integer distance correction.

[0137] For example, when bilateral matching is used, there can be three types of corrections. The first type of correction is to perform a joint correction in both reference lists L0 and L1, so that the displacement in reference list L1 is equal to the displacement in reference list L0 and opposite to it in the horizontal and vertical directions (the first type of correction is called SBM_JOINT). The second type of correction is to perform an independent correction for a common bilateral average template in both reference lists L0 and L1 (the second type of correction is called TBM_INDEPENDENT). The third type of correction is to perform a correction for a bilateral average template in reference lists L0 or L1, and to invert the estimated horizontal and vertical displacements when obtaining the displacement in the other reference list (if L0 is used for correction, the other reference list is L1; if L1 is used for correction, the other reference list is L0. The third type of correction is called TBM_IN_1REF_NEG_IN_OTHER).

[0138] Due to the difficulty in predicting an early exit from an iteration loop (since the cost at the center position is lower than the costs at the surrounding 1-pixel neighboring positions), when the cost function evaluation at the integer distance position used to derive the parameter error surface is different from the cost function evaluation performed during the previous iteration of the integer distance correction, an additional set of cost function evaluations centered on the early exit center are performed using the desired cost function.

[0139] Table 1 below provides some cases in the bilateral matching process where additional evaluation is required after early exit during integer distance correction.

[0140] Table 1

[0141]

[0142] It is observed that the correction type SBM_JOINT works best in integer distance correction iterations. When using correction type TBM_INDEPENDENT to obtain independent sub-pixel incremental motion vectors in L0 and L1, a small additional decoding gain can be obtained compared to equal and opposite displacements in L0 and L1. However, independent corrections in L0 and L1 require independent cost evaluations at L0 and L1 locations and are therefore more computationally complex than the joint equal and opposite displacement estimation option (SBM_JOINT correction). Early exit does not require additional cost function evaluations beyond the SBM_JOINT correction.

[0143] Fig.11A simplified flow chart of a method 1100 for obtaining sub-pixel accurate incremental motion vectors in one or more reference frames in a decoding end motion vector correction system is provided for one embodiment of the present invention. The one or more reference frames are around corresponding one or more initial sub-pixel accurate correction centers. The method 1100 may include the following steps:

[0144] Step 1101: Provide a processor. The processor may be one or more processing units (CPU, DSP) integrated in a video encoder and / or decoder, or a program code integrated in a video compression software for executing the method described herein.

[0145] Step 1103: The processor iteratively performs an integer 1-pixel distance correction operation (iteration) using a cost function to determine an integer distance corrected motion vector for each of the one or more reference frames.

[0146] Step 1105: Determine whether the current operation is the last operation. If the current operation is the last operation (1105, yes):

[0147] Step 1111: Use the second cost function to perform integer 1-pixel distance correction to obtain an independent sub-pixel incremental motion vector for each reference frame in one or more reference frames.

[0148] If the current operation is not the last operation ( 1105 , No), execute step 1107 .

[0149] Step 1107: Determine whether the cost function value of the search center position of the current operation is the lowest cost function value in the 1-pixel adjacent position set. In the case where the cost function value of the search center position is the lowest cost function value (1107, yes): Execute step 1111 (i.e., exit the iteration loop early). In the case where the cost function value of the search center position is not the lowest cost function value (1107, no): Execute step 1109.

[0150] Step 1109: Update the center to the position with the lowest cost in this operation, and return to the loop to perform the next integer 1 pixel distance correction operation.

[0151] Step 1111: Use the second cost function to perform integer 1-pixel distance correction to obtain an independent sub-pixel incremental motion vector for each reference frame in one or more reference frames.

[0152] Step 1113: For the second cost function value at the 1-pixel neighboring position set of the last search center, determine whether the last search center position has the lowest second cost function value. In the case where the last search center position has the lowest second cost function value (1113, yes):

[0153] Step 1115: Determine the sub-pixel distance corrected motion vectors around the best integer distance corrected position in each reference frame by calculating the position with the minimum value on the fitted parameter error surface using the second cost function value.

[0154] Step 1117: Return the total correction MV as the sum of the determined integer distance correction MV and the determined sub-pixel distance correction MV for each reference frame.

[0155] Embodiments of the present invention also provide a device for performing the above method. The device may be a combination of software and hardware. For example, encoding and / or decoding may be performed by a chip such as a general purpose processor (CPU), a digital signal processor (DSP), or a field programmable gate array (FPGA). However, embodiments of the present invention are not limited to programmable hardware implementations. Some embodiments of the present invention may be implemented using an application-specific integrated circuit (ASIC), or a combination of the above hardware components.

[0156] The encoding and / or decoding may also be implemented by program instructions or program codes stored in a computer-readable medium. When a processor or a computer executes the program instructions, the processor or the computer performs the steps of the above method. The computer-readable medium may be any medium storing program codes, such as a DVD, a CD, a USB (flash) drive, a hard disk, a server memory accessible via a network, etc.

[0157] Fig.12 FIG. 1 is a block diagram of an apparatus 1200 that can be used to implement various embodiments of the present invention. The apparatus 1200 can be Figure 1 The encoding device 100 and Figure 2The decoding device 200 shown. In addition, the device 1200 can accommodate one or more of the described elements. In some embodiments, the device 1200 is equipped with one or more input / output devices, such as a speaker, a microphone, a mouse, a touch screen, a keypad, a keyboard, a printer, a display, etc. The device 1200 may include one or more central processing units (CPUs) 1210, a memory 1220, a mass storage 1230, a video adapter 1240, and an I / O interface 1260 connected to a bus. The bus is one or more of several bus architectures of any type, including a memory bus or a memory controller, a peripheral bus, a video bus, etc.

[0158] CPU 1210 may have any type of electronic data processor. Memory 1220 may have or may be any type of system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), combinations thereof, and the like. In an embodiment, memory 1220 may include a ROM used at boot time and a DRAM used to store programs and data when executing programs. In some embodiments, memory 1220 may be non-transient. Mass storage 1230 may include any type of storage device that stores data, programs, and other information and enables the data, programs, and other information to be accessed via a bus. For example, mass storage 1230 includes one or more of a solid-state drive, a hard disk drive, a magnetic disk drive, an optical disk drive, and the like for storing program code, wherein when CPU 1210 executes the program code, the CPU executes the method described herein. CPU 1210 may be used to iteratively execute a combination of Fig.10 and 11 The CPU 1210 may include a reference selection unit for selecting a first reference image pointed to by MV0 and a second reference image pointed to by MV1. After selecting the reference image, the reference image selection unit may determine whether the first reference image or the second reference image is used to perform motion vector correction.

[0159] The video adapter 1240 and the I / O interface 1260 provide interfaces to couple external input and output devices to the apparatus 1200. For example, the apparatus 1200 may provide a SQL command interface to a client. As shown, examples of input and output devices include any combination of a display 1290 coupled to the video adapter 1240 and a mouse / keyboard / printer 1270 coupled to the I / O interface 1260. Other devices may be coupled to the apparatus 1200, and additional or fewer interface cards may be used. For example, a serial interface card (not shown) may be used to provide a serial interface for a printer.

[0160] The device 1200 may also include one or more network interfaces 1250, or one or more networks 1280, wherein the network interface 1250 includes a wired link such as an Ethernet cable, and / or a wireless link for accessing a node. The network interface 1250 allows the device 1200 to communicate with a remote unit through the network 1280. For example, the network interface 1250 may provide communication with a database. In one embodiment, the device 1200 is coupled to a local area network or a wide area network for data processing and communication with remote devices such as other processing units, the Internet, remote storage facilities, etc. The device 1200 may be used to encode one or more image blocks received from an input end (e.g., a network interface) and / or decode video images from a bitstream. The device 1200 may include: a bitstream parser for extracting compressed image blocks from a bitstream; a motion vector correction unit for obtaining sub-pixel accurate incremental motion vector correction; and a reconstruction unit for performing block reconstruction based on the acquired motion vectors.

[0161] Although the specific features or aspects of the present invention may have been disclosed in conjunction with only one of several implementations or embodiments, such features or aspects may be combined with one or more other features or aspects in other implementations or embodiments, as long as it is necessary or advantageous for any given or specific application. In addition, to a certain extent, the terms "including", "having", "having" or other variations of these words are used in the detailed description or claims, and such terms are similar to the term "including" and are both expressions of the meaning of inclusion. In addition, the terms "exemplary" and "for example" are only represented as examples, rather than the best or optimal. The terms "coupled" and "connected" and their derivatives can be used. It should be understood that these terms can be used to indicate that two elements cooperate or interact with each other, regardless of whether they are in direct physical contact or electrical contact, or they are not in direct contact with each other.

[0162] Although specific aspects have been illustrated and described herein, it will be appreciated by those skilled in the art that a variety of alternative and / or equivalent implementations may be substituted for the specific aspects shown and described without departing from the scope of the invention. This application is intended to cover any modifications or changes to the specific aspects discussed herein.

[0163] Although the elements in the above claims are listed in a specific order with corresponding labels, these elements are not necessarily limited to being implemented in that specific order unless the claim recitation otherwise implies a specific order for implementing some or all of these elements.

[0164] Through the above teachings, it is obvious to those skilled in the art that many substitutions, modifications and variations are possible. Of course, it is readily recognized by those skilled in the art that, in addition to the applications described herein, there are numerous other applications of the present invention. Although the present invention has been described in conjunction with one or more specific embodiments, it is recognized by those skilled in the art that, without departing from the scope of the present invention, many changes may be made to the present invention. Therefore, it should be understood that, as long as it is within the scope of the appended claims and their equivalents, the present invention may be practiced in a manner different from that specifically described herein.

[0165] The above circuit may also be a single integrated chip. However, the present invention is not limited thereto, and the circuit may include different components or hardware, or a combination of hardware and software, such as a general purpose processor or DSP programmed with corresponding codes.

[0166] The above flowchart is intended to illustrate an example of the decoding end motion vector correction technology. Those skilled in the art may modify or combine the steps to implement the present invention without departing from the spirit and scope of the present invention.

[0167] In one aspect of the present invention, a method for obtaining sub-pixel accurate incremental motion vector correction in one or more reference frames in a decoding-side motion vector correction system is provided. The method may include:

[0168] Provide a processor (1001);

[0169] The processor iteratively performs (1003) an integer 1-pixel distance correction operation using a cost function to determine an integer distance corrected motion vector for each of the one or more reference frames, thereby initiating a loop operation;

[0170] The processor determines whether the cost function value of the search center position is the lowest function value in the set of 1-pixel neighboring positions (1005);

[0171] In the case where the cost function value of the search center position is the lowest cost function value (1005, yes): the loop operation is exited.

[0172] Determine a sub-pixel distance corrected motion vector around the search center in each reference frame by calculating the position with the minimum value on the fitted parameter error surface using the cost function value of the search center and a set of 1-pixel neighboring positions (1011);

[0173] The sub-pixel positions obtained using the parametric error surface for each reference frame are returned (1013).

[0174] In one embodiment, the method may further include: in the case where the cost function value of the search center position is not the lowest cost function value (1005, No): determining whether the current operation is the last operation (1007);

[0175] If the current operation is the last operation (1007, yes):

[0176] Return the position in each reference frame with the lowest cost function (1015);

[0177] If the current operation is not the last operation (1007, No):

[0178] The processor updates (1009) the position of the search center of the current integer 1-pixel distance correction operation to the position with the lowest cost in the current operation;

[0179] The loop operation (1003, 1005, 1007, 1009) is repeated.

[0180] In one embodiment, the parameter error surface includes 5 pixels arranged such that a central pixel is surrounded by four peripheral pixels spaced equidistant from the central pixel.

[0181] In one embodiment, the cost function is calculated by the following expression:

[0182] E(x, y) = A*(x – x0) 2 +B*(y–y0) 2 +C;

[0183] Wherein, E(x, y) is the estimated cost function value of the coordinate (x, y); x0 and y0 are the coordinates with the smallest error associated with the sub-pixel displacement relative to the center (0, 0); C is a parameter associated with the error at the coordinate (x0, y0); A and B are constant values; x and y are coordinates associated with adjacent positions; x and y are each integers including -1, 0 and 1.

[0184] In one embodiment, the sub-pixel position is obtained by the following expression:

[0185] x0=(E(–1,0)–E(1,0)) / (2*N*(E(–1,0)+E(1,0)–2*E(0,0)));

[0186] y0=(E(0,–1)–E(0,1)) / (2*N*(E(0,–1)+E(0,1)–2*E(0,0)));

[0187] in,

[0188] E(x, y) = A*(x – x0) 2 +B*(y–y0) 2 +C;

[0189] E(x, y) is the estimated cost function value; x0 and y0 are the coordinates at which the error associated with the sub-pixel displacement relative to the center (0, 0) is minimized; C is a parameter associated with the error at the coordinate (x0, y0); A and B are constant values; x and y are coordinates associated with adjacent positions; x and y are each integers including -1, 0, and 1; for a sub-pixel accuracy of 1 / 2, 1 / 4, 1 / 8, or 1 / 16, N is an integer 1, 2, 4, or 8.

[0190] In one embodiment, sub-pixel accurate incremental motion vector corrections are obtained by template matching.

[0191] In another aspect of the present invention, a method for obtaining sub-pixel accurate incremental motion vectors in one or more reference frames in a decoding-side motion vector correction system is provided. The apparatus comprises a processing unit and a non-transitory computer-readable medium having computer-readable instructions therein, the non-transitory computer-readable medium being used to cause the processing unit to perform the following steps:

[0192] Iteratively performing N integer 1-pixel distance correction operations using a cost function to determine an integer distance corrected motion vector for each reference frame, thereby starting a loop operation in which the position of the search center of the current iteration is updated with the position with the lowest cost in the previous operation;

[0193] Determine whether the cost of searching the center position is lower than any cost of the set of 1-pixel neighboring positions;

[0194] In the case where the cost of searching for the center position is the lowest cost:

[0195] Exit the loop operation;

[0196] Determine a sub-pixel distance corrected motion vector around the search center in each reference frame by calculating the position with the minimum value on the fitted parameter error surface using the cost function value of the search center and a set of 1-pixel neighboring positions;

[0197] Returns the sub-pixel position obtained using the parametric error surface for each reference frame.

[0198] In one embodiment, the non-transitory computer readable medium having computer readable instructions therein further causes the processing unit to perform the following steps:

[0199] In the case where the cost function value of the search center position is not the minimum cost function value (1005, No): determining whether the current operation is the last operation;

[0200] When the current operation is the last operation: return the position with the lowest cost function in each reference frame;

[0201] When the current operation is not the last operation: the processor updates (1009) the position of the search center of the current integer 1-pixel distance correction operation to the position with the lowest cost in the current correction operation; and repeats the loop operation.

[0202] In one embodiment, the cost function is calculated by the following expression:

[0203] E(x, y) = A*(x – x0) 2 +B*(y–y0) 2 +C;

[0204] Wherein, E(x, y) is the estimated cost function value; x0 and y0 are the coordinates with the smallest error associated with the sub-pixel displacement relative to the center (0, 0); C is a parameter associated with the error at the coordinate (x0, y0); A and B are constant values; x and y are coordinates associated with adjacent positions; x and y are each integers including -1, 0 and 1.

[0205] In one embodiment, the sub-pixel position is obtained by the following expression:

[0206] x0=(E(–1,0)–E(1,0)) / (2*N*(E(–1,0)+E(1,0)–2*E(0,0)));

[0207] y0=(E(0,–1)–E(0,1)) / (2*N*(E(0,–1)+E(0,1)–2*E(0,0)));

[0208] in,

[0209] E(x, y) = A*(x – x0) 2 +B*(y–y0) 2 +C;

[0210] E(x, y) is the estimated cost function value; x0 and y0 are the coordinates at which the error associated with the sub-pixel displacement relative to the center (0, 0) is minimized; C is a parameter associated with the error at the coordinate (x0, y0); A and B are constant values; x and y are coordinates associated with adjacent positions; x and y are each integers including -1, 0, and 1; for a sub-pixel accuracy of 1 / 2, 1 / 4, 1 / 8, or 1 / 16, N is an integer 1, 2, 4, or 8.

[0211] Another aspect of the present invention provides a method for obtaining sub-pixel accurate incremental motion vector correction in one or more reference frames in reference list L0 and reference list 1 in a decoding end motion vector correction system. The method may include:

[0212] providing a processor (1101);

[0213] The processor iteratively performs (1103) an integer 1-pixel distance correction operation using a first cost function to determine an integer distance corrected motion vector for each of the one or more reference frames, thereby initiating a loop operation;

[0214] Determine whether the current operation is the last operation (1105);

[0215] When the current operation is the last operation (1105, yes):

[0216] Performing integer 1-pixel distance correction using a second cost function to obtain an independent sub-pixel incremental motion vector for each reference frame (1111);

[0217] If the current operation is not the last operation (1105, No):

[0218] The processor determines whether the cost function value of the search center position of the current operation is the first lowest function value in the 1-pixel neighboring position set (1107);

[0219] In the case where the cost function value of the search center position is the first lowest function value (1107, yes):

[0220] Exit the loop operation;

[0221] Performing integer 1-pixel distance correction using a second cost function to obtain an independent sub-pixel incremental motion vector for each reference frame (1111);

[0222] In the case where the cost function value of the search center position of the current operation is not the first lowest function value (1107, No):

[0223] Update the center to the position with the first lowest cost for the current operation;

[0224] The loop operation (1103, 1105, 1107, 1109) is repeated.

[0225] In one embodiment, the method may further include:

[0226] The processor determines whether the second cost function value of the search center position of the acquired independent sub-pixel incremental motion vector is the second lowest function value (1113);

[0227] When the second cost function value is the minimum cost function value (1113, yes):

[0228] Determine the sub-pixel distance corrected motion vector around the search center in each reference frame by calculating the position with the minimum value on the fitted parameter error surface using the cost function value of the search center and a set of 1-pixel neighboring positions (1115);

[0229] returning a total corrected motion vector as the sum of the determined integer distance corrected motion vector and the determined sub-pixel distance corrected motion vector for each reference frame obtained using the parametric error surface for each reference frame (1117);

[0230] In the case where the second cost function value is not the second lowest cost function value (1113, No):

[0231] A modified motion vector corresponding to the position having the second lowest cost function value in each reference frame is returned (1119).

[0232] In one embodiment, the sub-pixel position is obtained by:

[0233] x0=(E(–1,0)–E(1,0)) / (2*N*(E(–1,0)+E(1,0)–2*E(0,0)))

[0234] y0=(E(0,–1)–E(0,1)) / (2*N*(E(0,–1)+E(0,1)–2*E(0,0)))

[0235] in,

[0236] E(x, y) = A*(x – x0) 2 +B*(y–y0) 2 +C;

[0237] (x0, y0) corresponds to the sub-pixel displacement with the smallest error relative to (0, 0); C corresponds to the error at (x0, y0); A and B are constants; (x, y) corresponds to adjacent positions; x includes –1, 0, 1; y includes –1, 0, 1.

[0238] In one embodiment, the first cost function and the second cost function are both calculated by the following expressions:

[0239] E(x, y) = A*(x – x0) 2 +B*(y–y0) 2 +C;

[0240] Wherein, E(x, y) is the estimated cost function value; x0 and y0 are the Cartesian coordinates with the smallest error associated with the sub-pixel displacement relative to the center (0, 0); C is a parameter associated with the error at the Cartesian coordinate (x0, y0); A and B are constant values; x and y are the Cartesian coordinates associated with adjacent positions; x and y are each integers including -1, 0 and 1.

[0241] In one embodiment, sub-pixel accurate incremental motion vector corrections are obtained through bilateral matching.

[0242] In one embodiment, the bilateral matching includes performing a joint revision in both the reference list L0 and the reference list L1.

[0243] In one embodiment, bilateral matching includes performing independent joint refinements on a common bilateral average template in reference list L0 and reference list L1.

[0244] In one embodiment, bilateral matching includes: performing correction on a common bilateral average template in reference list L0 or reference list L1; and negating the estimated horizontal and vertical displacements when obtaining the displacements in the other reference list.

[0245] In summary, embodiments of the present invention relate to determining motion vectors based on template matching for bidirectional motion vector estimation. In particular, a block template is constructed as the average value of the block pointed to by the initial motion vector to be corrected. Then, motion vector correction is performed by template matching in two different reference images. Matching is performed by finding the merit (minimum or maximum, depending on the function) of the matching function corresponding to the best matching block in each of the two reference images. The merits of the zero-mean template and the zero-mean candidate block are searched (among the block positions pointed to by the motion vector candidates in the search space). In other words, before the function optimization is performed, the template is subtracted from the template's mean and the candidate block is subtracted from the candidate block's mean. Then, the prediction value of the current block is calculated as the weighted average of the best matching blocks in the corresponding reference images.

Claims

1. A decoding end motion vector correction method, characterized in that: The method comprises: Determining a target integer motion vector displacement by comparing integer distance costs corresponding to candidate integer motion vector displacements relative to the initial motion vector; Determine a sub-pixel motion vector displacement by calculating the integer distance cost; determining a modified motion vector based on the target integer motion vector displacement, the sub-pixel motion vector displacement and the initial motion vector; The sub-pixel motion vector displacement is determined by the following equation: x0 = (E(–1,0) – E(1,0)) / (2 × (E(–1,0) + E(1,0)–2×E(0,0))); and y0 = (E(0,–1)–E(0,1)) / (2×(E(0,–1)+E(0,1)–2×E(0,0))); Wherein, x0 and y0 are coordinates associated with the sub-pixel motion vector displacement relative to the center (0, 0); the center (0, 0) is the position corresponding to the target integer motion vector displacement, and the position is the position with the minimum cost value compared with the set of pixel-neighboring positions of the position; E(–1, 0), E(1, 0), E(0, 0), E(0, –1) and E(0, 1) are integer distance costs, and the integer distance costs correspond to the candidate integer motion vector displacements (–1, 0), (1, 0), (0, 0), (0, –1) and (0, 1) relative to the initial motion vector, respectively; x0 and y0 are scaled by a scaling factor N according to the sub-pixel accuracy of the incremental motion vector, and the incremental motion vector is the sum of the motion vectors corresponding to the target integer motion vector displacement and the sub-pixel motion vector displacement, respectively.

2. The method according to claim 1, characterized in that The equation for determining x0 is implemented by performing at least one of shift, comparison and addition operations on E(−1, 0), E(1, 0) and E(0, 0).

3. The method according to claim 1, characterized in that The equation for determining y0 is implemented by performing at least one of shift, comparison and addition operations on E(0,−1), E(0,1) and E(0,0).

4. The method according to claim 1, characterized in that The sub-pixel motion vector displacement is bounded between -0.5 and +0.5 pixels.

5. The method according to claim 1, characterized in that Determining the target integer motion vector displacement includes: Calculate an integer distance cost for each candidate integer motion vector displacement; The candidate integer motion vector displacement corresponding to the lowest integer distance cost is selected as the target integer motion vector displacement.

6. The method according to claim 1, characterized in that The target integer motion vector displacement comprises a first motion vector displacement corresponding to the reference picture list L0 and a second motion vector displacement corresponding to the reference picture list L1, the method further comprising: determining the first motion vector displacement by comparing integer distance costs, wherein the integer distance costs correspond to candidate integer motion vector displacements corresponding to the reference picture list L0; The second motion vector displacement is determined by inverting the first motion vector displacement.

7. The method according to claim 1, characterized in that For sub-pixel precisions of 1 / 2, 1 / 4, 1 / 8, and 1 / 16, N is 2, 4, 8, and 16, respectively.

8. A device, characterized in that: include: An input terminal for receiving encoded video data; an entropy decoding unit, configured to decode the received encoded video data to obtain decoded data; One or more processors, coupled to the entropy decoding unit, configured to: determining a target integer motion vector displacement by comparing integer distance costs corresponding to candidate integer motion vector displacements relative to an initial motion vector based on the decoded data; Determine a sub-pixel motion vector displacement by calculating the integer distance cost; determining a modified motion vector based on the target integer motion vector displacement, the sub-pixel motion vector displacement and the initial motion vector; The sub-pixel motion vector displacement is determined by the following equation: x0 = (E(–1,0) – E(1,0)) / (2 × (E(–1,0) + E(1,0) – 2×E(0,0))); and y0 = (E(0,–1) – E(0,1)) / (2 × (E(0,–1) + E(0,1) – 2×E(0,0))); Wherein, x0 and y0 are coordinates associated with the sub-pixel motion vector displacement relative to the center (0, 0); the center (0, 0) is the position corresponding to the target integer motion vector displacement, and the position is the position with the minimum cost value compared to the set of pixel neighboring positions of the position 1; E(–1, 0), E(1, 0), E(0, 0), E(0, –1) and E(0, 1) are integer distance costs, which correspond to candidate integer motion vector displacements (–1, 0), (1, 0), (0, 0), (0, –1) and (0, 1) relative to the initial motion vector, respectively; x0 and y0 are scaled by a scaling factor N according to the sub-pixel accuracy of the incremental motion vector, and the incremental motion vector is the sum of the motion vectors corresponding to the target integer motion vector displacement and the sub-pixel motion vector displacement, respectively.

9. The device according to claim 8, characterized in that The equation for determining x0 is implemented by performing at least one of shift, comparison and addition operations on E(−1, 0), E(1, 0) and E(0, 0).

10. The device according to claim 8, characterized in that The equation for determining y0 is implemented by performing at least one of shift, comparison and addition operations on E(0,−1), E(0,1) and E(0,0).

11. The device according to claim 8, characterized in that The sub-pixel motion vector displacement is bounded between -0.5 and +0.5 pixels.

12. The device according to claim 8, characterized in that The one or more processors determine the target integer motion vector displacement by: Calculate an integer distance cost for each candidate integer motion vector displacement; The candidate integer motion vector displacement corresponding to the lowest integer distance cost is selected as the target integer motion vector displacement.

13. The device according to claim 8, characterized in that The target integer motion vector displacement comprises a first motion vector displacement corresponding to reference picture list L0 and a second motion vector displacement corresponding to reference picture list L1, and the one or more processors are configured to: determining the first motion vector displacement by comparing integer distance costs, wherein the integer distance costs correspond to candidate integer motion vector displacements corresponding to the reference picture list L0; The second motion vector displacement is determined by inverting the first motion vector displacement.

14. A computer-readable medium storing instructions, characterized in that: When the instructions are executed in a processor, the steps of the method according to any one of claims 1 to 7 are performed.

15. A decoding end motion vector correction method, characterized in that: The method comprises: Receive the encoded video stream; Decoding the received encoded video code stream to obtain decoded data, where the decoded data is used to indicate an initial motion vector or the decoded data is used by a decoder to derive the initial motion vector; determining a target integer motion vector displacement by comparing integer distance costs corresponding to candidate integer motion vector displacements relative to the initial motion vector; Determine a sub-pixel motion vector displacement by calculating the integer distance cost; determining a modified motion vector based on the target integer motion vector displacement, the sub-pixel motion vector displacement and the initial motion vector; The sub-pixel motion vector displacement is determined by the following equation: x0 = (E(–1,0) – E(1,0)) / (2 × (E(–1,0) + E(1,0) – 2×E(0,0))); and y0 = (E(0,–1) – E(0,1)) / (2 × (E(0,–1) + E(0,1) – 2×E(0,0))); Wherein, x0 and y0 are coordinates associated with the sub-pixel motion vector displacement relative to the center (0, 0); the center (0, 0) is the position corresponding to the target integer motion vector displacement, and the position is the position with the minimum cost value compared with the set of pixel-neighboring positions of the position; E(–1, 0), E(1, 0), E(0, 0), E(0, –1) and E(0, 1) are integer distance costs, and the integer distance costs correspond to the candidate integer motion vector displacements (–1, 0), (1, 0), (0, 0), (0, –1) and (0, 1) relative to the initial motion vector, respectively; x0 and y0 are scaled by a scaling factor N according to the sub-pixel accuracy of the incremental motion vector, and the incremental motion vector is the sum of the motion vectors corresponding to the target integer motion vector displacement and the sub-pixel motion vector displacement, respectively.

16. The method according to claim 15, characterized in that The equation for determining x0 is implemented by performing at least one of shift, comparison and addition operations on E(−1, 0), E(1, 0) and E(0, 0).

17. The method according to claim 15, characterized in that The equation for determining y0 is implemented by performing at least one of shift, comparison and addition operations on E(0,−1), E(0,1) and E(0,0).

18. The method according to claim 15, characterized in that The sub-pixel motion vector displacement is bounded between -0.5 and +0.5 pixels.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed by one or more processors, the computer implements the method according to any one of claims 15 to 18.

Images