Motion vector correction for multi-reference prediction

Abstract

The invention relates to performing motion vector refinement in search space for multi-reference inter prediction. Two or more reference pictures are selected, one of which is used for motion vector refinement. A search space is constructed in the reference picture based on an initial estimate of a motion vector of the reference picture used for motion vector refinement. The first motion vector is refined using template matching. A second motion vector of another reference picture is calculated using an initial estimate of the second motion vector, the initial estimate of the first motion vector and the refined first motion vector.

Description

[0001] This application is a divisional application of the original application with the application number 201880044272.8 and the original filing date of 29.03.2018, the content of which is incorporated herein by reference in its entirety.

[0002] The present invention relates to the field of video coding, in particular to a motion vector estimation suitable for multi-reference inter prediction. This application claims priority from PCT / EP2017 / 066342 filed on 30.06.2017, the content of which is incorporated herein by reference. BACKGROUND

[0003] Current hybrid video codecs apply predictive coding. The images of a video sequence are subdivided into blocks of pixels, which are then coded. Instead of coding the entire block pixel by pixel, the entire block is predicted using already coded pixels that are spatially or temporally close to the block. The encoder only further processes the difference between the block and its prediction. The further processing usually comprises a transformation of the block pixels into coefficients in a transform domain. These coefficients can then be further compressed by quantization and further compressed by entropy coding to form a bitstream. The bitstream also comprises any signaling information that enables a decoder to decode the coded video. For example, the signaling can comprise settings related to the encoder settings, like the size of the input images, the frame rate, a quantization step indication, the prediction applied to the blocks of the images, etc.

[0004] Temporal prediction exploits the temporal correlation between the images, also called frames, of a video. Temporal prediction is also called inter prediction, because it is a prediction that uses dependencies between different video frames (inter). Accordingly, a block that is being coded, also called current block, is predicted from one or more previously coded images, also called reference images. The reference images are not necessarily images that precede the current image in which the current block is located, in terms of the display order of the video sequence. The encoder can code the images in a coding order that is different from the display order. As a prediction block for the current block, a collocated block in a reference image can be determined. The collocated block is the block in the reference image that is in the same position as the current block in the current image. This kind of prediction is accurate for stationary image regions, i.e. image regions that do not move from one image to another.

[0005] In order to obtain a prediction taking motion into account, i.e. a motion-compensated prediction, motion estimation is usually applied when determining the prediction block of the current block. Accordingly, the current block is predicted using a block in a reference picture, which is located at a distance given by a motion vector from the position of the collocated block. In order to enable the decoder to determine the same prediction block of the current block, the motion vector can be transmitted in the bitstream. In order to further reduce the signaling overhead caused by transmitting a motion vector for each block, the motion vector itself can be estimated. Motion vector estimation can be performed based on motion vectors of neighboring blocks in the spatial and / or temporal domain.

[0006] The prediction block of the current block can be calculated using one reference picture or by a weighted prediction block obtained from two or more reference pictures. The reference picture can be a neighboring picture, i.e. a picture immediately preceding the current picture in display order and / or a picture immediately succeeding the current picture in display order, since neighboring pictures are most likely to be similar to the current picture. However, in general, the reference picture can also be any other picture preceding the current picture in display order and / or succeeding the current picture in display order as well as preceding the current picture in the bitstream (decoding order). This can provide advantages, for example, in case of occlusions and / or non-linear movements in the video content. Accordingly, the reference picture can also be transmitted in the bitstream.

[0007] A special mode of inter prediction is so-called bi-prediction, in which two reference pictures are used to generate the prediction block of the current block. In particular, two prediction blocks determined from the two reference pictures, respectively, are combined to the prediction signal of the current block. Bi-prediction can provide a more accurate prediction for the current block compared to uni-prediction, i.e. a prediction using only one reference picture. The more accurate the prediction, the smaller the difference between the pixels of the current block and the prediction block, also referred to as “residual”, and the more efficiently the prediction block can be encoded, i.e. compressed into a shorter bitstream. In general, more than two reference pictures can be used to find respective more than two reference blocks to predict the current block, i.e. multi-reference inter prediction can be applied. Accordingly, the term “multi-reference prediction” includes bi-prediction as well as a prediction using more than two reference pictures.

[0008] In order to provide a more accurate motion estimation, the resolution of the reference picture can be enhanced by interpolating samples between pixels. Fractional pixel interpolation can be performed by a weighted average of the nearest pixels. In case of half-pixel resolution, for example, bilinear interpolation is usually used. Other fractional pixels are calculated as an average of the nearest pixels, which is weighted by the inverse of the distance between the respective nearest pixel and the pixel being predicted.

[0009] Motion vector estimation is a computationally complex task in which the similarity between the current block and the corresponding prediction block pointed to by a candidate motion vector in the reference picture is computed. To reduce the complexity, the number of candidate motion vectors is usually reduced by restricting the candidate motion vectors to a certain search space. For example, the search space can be defined by the number and / or position of pixels around the position in the reference picture that corresponds to the position of the current block in the current picture. On the other hand, the candidate motion vectors can be defined by a list of candidate motion vectors consisting of motion vectors of neighboring blocks.

[0010] Motion vectors are usually at least partly determined at the encoder side and transmitted in the coded bitstream to the decoder. However, motion vectors can also be derived at the decoder. In this case, the current block is not available at the decoder and cannot be used to compute the similarity between the current block and the block pointed to by a candidate motion vector in the reference picture. Therefore, a template consisting of pixels of already decoded blocks is used instead of the current block. For example, already decoded pixels neighboring the current block (in the current picture or in the reference picture) can be used. This kind of motion estimation provides the advantage of reduced signaling: the motion vector is derived in the same way at the encoder and at the decoder, so no signaling is necessary. On the other hand, the accuracy of this kind of motion estimation can be lower.

[0011] To balance accuracy and signaling overhead, motion vector estimation can be split into two steps: motion vector derivation and motion vector refinement. For example, the motion vector derivation can comprise selecting a motion vector from a candidate list. The selected motion vector can be further refined, for example, by a search within a search space. The search within the search space is based on computing a cost function for each candidate motion vector, i.e. for each candidate position of the block pointed to by the candidate motion vector.

[0012] The document JVET-D0029, "Decoder-side motion vector refinement based on bi-directional template matching", X. Chen, J. An and J. Zheng, (available at http: / / phenix.it-sudparis.eu / jvet / ) shows a motion vector refinement in which a first motion vector at integer pixel resolution is first found and the first motion vector is further refined by a search at half-pixel resolution within a search space around the first motion vector. SUMMARY

[0013] When multi-reference prediction is applied, motion vectors in multiple reference pictures need to be determined. Even if the motion vectors are transmitted in the first stage so that the decoder does not need to perform a further search, motion vector refinement still requires an additional search among the motion vectors of the corresponding search space. This can be a complex task requiring computational and memory resources.

[0014] The present application provides a technique for determining a first motion vector in a first reference picture and a second motion vector in a second reference picture, so that the complexity can be reduced. First, the first motion vector and the second motion vector are coarsely estimated. Then, the first motion vector is refined by searching in a search space given by the coarse estimate of the first motion vector. The second motion vector is determined based on the coarse estimate of the second motion vector and the refined first motion vector. The first motion vector and the second motion vector can be applied to inter prediction of a current block in a current picture for an encoder and / or a decoder.

[0015] According to a first aspect, the present application relates to an apparatus for determining a first motion vector in a first reference picture and a second motion vector in a second reference picture, the first motion vector and the second motion vector being applied to inter prediction of a picture block in a current picture, the apparatus comprising a motion vector refinement unit and a motion vector calculation unit. The motion vector refinement unit is configured to obtain an estimate of the first motion vector. A search space is specified based on the estimate of the first motion vector. The motion vector refinement unit is configured to search in the search space to determine the first motion vector. The motion vector calculation unit is configured to obtain an estimate of the second motion vector. The motion vector calculation unit is configured to calculate the second motion vector based on the estimate of the second motion vector and the first motion vector.

[0016] Thus, the motion vector determination can be performed with lower complexity while still maintaining the accuracy of refining the first motion vector and the refinement of the second motion vector based on the refined first motion vector.

[0017] In a possible implementation form of the apparatus according to the first aspect, the second motion vector is calculated by adding a function of a difference between the first motion vector and the estimate of the first motion vector to the estimate of the second motion vector. The function can comprise scaling and / or clipping. When determining the scaling parameter, its value can depend on a ratio between a distance of the first reference picture to the current picture and a distance of the second reference picture to the current picture.

[0018] The calculation of the second motion vector as a function of the refinement performed on the first motion vector is a low complexity estimation. Furthermore, by further modifying the second motion vector (e.g. by scaling according to the distance between the respective reference pictures), the estimation can become more accurate.

[0019] Advantageously, the apparatus further comprises a reference picture selection unit for obtaining reference pictures and selecting which one of the reference pictures is the first reference picture and the second reference picture. After this selection, the reference picture selection unit determines whether the first reference picture or the second reference picture is used for motion vector refinement. A reference picture list associates indices to be included in a bitstream with positions of reference pictures relative to the current picture. The reference picture selection unit is configured to select the first reference picture and the second reference picture based on whether the first reference picture or the second reference picture is referred to in the bitstream by an index in the predefined reference picture list in the at least two reference picture lists.

[0020] In other words, the reference picture selection unit is configured to select the first picture or the second picture based on whether the first picture or the second picture is referred to in a bitstream by an index related to a predefined reference picture list in the at least two reference picture lists, the bitstream also comprising encoded picture blocks of the video, wherein a reference picture list associates indices with positions of reference pictures relative to the current picture.

[0021] If the two reference pictures are referred to in the bitstream by an index in the same predefined reference picture list, the reference picture selection unit can select the reference picture for motion vector refinement to be the picture having the highest position in the reference picture list.

[0022] Alternatively, the reference picture for motion vector refinement can be selected to be the picture having the lowest temporal layer among the two pictures.

[0023] The reference picture for motion vector refinement can be selected to be the picture having the smallest quantization value or the picture having the shortest distance to the current picture.

[0024] As another approach, the reference picture for motion vector refinement can be selected such that an estimate of a size of a motion vector pointing to the reference picture for motion vector refinement is smaller than an estimate of a size of another motion vector.

[0025] The apparatus can further comprise a motion vector determination unit configured to determine an estimate of the first motion vector and an estimate of the second motion vector. The determination can be performed by selecting the estimate of the first motion vector and the estimate of the second motion vector from a set of motion vector candidates based on a similarity between a respective partial area of a picture involved by the motion vector candidate and the template.

[0026] A video encoder for encoding a plurality of pictures into a bitstream, comprising an inter prediction unit, a bitstream former and a reconstruction unit. The inter prediction unit comprises means for determining the first motion vector and the second motion vector and a prediction unit. The prediction unit determines the prediction block from a partial region of the first reference picture to which the first motion vector refers and a partial region of the second reference picture to which the second motion vector refers. The bitstream former includes an estimate of the first motion vector and an estimate of the second motion vector in the bitstream. The reconstruction unit reconstructs the current block from the prediction block and stores the reconstructed block in a memory.

[0027] A video decoder for decoding a plurality of pictures from a bitstream, comprising an inter prediction unit, a bitstream parser and a reconstruction unit. The inter prediction unit comprises means for determining the first motion vector and the second motion vector and a prediction unit. The prediction unit determines the prediction block from a partial region of the first reference picture to which the first motion vector refers and a partial region of the second reference picture to which the second motion vector refers. The bitstream parser obtains an estimate of the first motion vector and an estimate of the second motion vector from the bitstream. The reconstruction unit reconstructs the current block from the prediction block.

[0028] The method comprises motion vector refinement and motion vector calculation. An estimate of a first motion vector is obtained. A search space is specified based on the estimate of the first motion vector. A search is performed within the search space to determine the first motion vector. An estimate of a second motion vector is obtained. The second motion vector is calculated based on the estimate of the second motion vector and the first motion vector. The second motion vector is calculated by adding a function of a difference between the first motion vector and the estimate of the first motion vector to the estimate of the second motion vector. The function comprises scaling and / or clipping. A scaling parameter is dependent on a ratio between a distance of the first reference picture and a distance of the second reference picture from the current picture.

[0029] The method further comprises reference picture selection to obtain reference pictures and select which one of the reference pictures is the first reference picture and the second reference picture. After the selection, the reference picture selection determines whether the first reference picture or the second reference picture is used for motion vector refinement. A reference picture list associates indices to be included in the bitstream with positions of the reference pictures relative to the current picture. The reference picture selection is performed so that the first reference picture and the second reference picture are selected based on whether the first reference picture or the second reference picture is referred to in the bitstream by an index in the predefined reference picture list. If both reference pictures are referred to in the bitstream by indices in the same predefined reference picture list, the reference picture used for motion vector refinement is selected to be the picture having the highest position in the reference picture list. Alternatively, the reference picture used for motion vector refinement can be selected to be the picture having the lowest temporal layer among the two pictures. The reference picture used for motion vector refinement can be selected to be the picture having the smallest base quantization value or the picture having the shortest distance to the current picture. As another approach, the reference picture used for motion vector refinement can be selected so that the size of the estimate of the motion vector pointing to the reference picture used for motion vector refinement is smaller than the size of the estimate of the other motion vector.

[0030] The method can further determine the estimate of the first motion vector and the estimate of the second motion vector. The determination is performed by selecting the estimate of the first motion vector and the estimate of the second motion vector from a set of motion vector candidates based on a similarity between the respective partial areas of the pictures involved by the motion vector candidates.

[0031] A video encoding method for encoding a plurality of pictures into a bitstream, comprising performing inter prediction, bitstream formation and block reconstruction. The inter prediction comprises determining the first motion vector and the second motion vector and block prediction. The prediction comprises determining the predicted block from a partial area of the first reference picture involved by the first motion vector and a partial area of the second reference picture involved by the second motion vector. The bitstream formation comprises including the estimate of the first motion vector and the estimate of the second motion vector into the bitstream. The reconstruction comprises reconstructing the current block from the predicted block and storing the reconstructed block in a memory.

[0032] A video decoding method for decoding a plurality of pictures from a bitstream, comprising performing inter prediction, bitstream parsing and block reconstruction. The inter prediction comprises determining the first motion vector and the second motion vector and block prediction. The prediction determines the predicted block from a partial area of the first reference picture involved by the first motion vector and a partial area of the second reference picture involved by the second motion vector.

[0033] The code stream parsing obtains the estimated value of the first motion vector and the estimated value of the second motion vector from the code stream. The reconstructing comprises reconstructing the current block according to the prediction block.

[0034] The present application can reduce the number of search candidates in the motion vector correction process while providing similar image quality, without any impact on the coding performance. This is achieved by searching for the motion vector correction of only one reference image of the current block, while the other motion vector of the other reference image of the same current block is calculated based on the corrected motion vector.

[0035] In some embodiments of the present application,

[0036] Embodiment 1. An apparatus for determining a first motion vector (MV0") in a first reference image of a video and a second motion vector (MV1") in a second reference image of the video, the first motion vector and the second motion vector being applied to inter prediction of a block of the current image of the video, the apparatus comprising: a motion vector correction unit (530) for obtaining an estimated value (MV0) of the first motion vector and determining the first motion vector by searching within a search space (310) specified based on the estimated value of the first motion vector; a motion vector calculation unit (550) for obtaining an estimated value (MV1) of the second motion vector and calculating the second motion vector based on the estimated value of the second motion vector and the first motion vector.

[0037] Embodiment 2. The apparatus of embodiment 1, the motion vector calculation unit (550, 560) being configured to calculate the second motion vector by adding the estimated value of the second motion vector to a difference between the first motion vector and the estimated value of the first motion vector or a function of the difference between the first motion vector and the estimated value of the first motion vector.

[0038] Embodiment 3. The apparatus of embodiment 2, the function comprising scaling by a scaling factor and / or clipping.

[0039] Embodiment 4. The apparatus of embodiment 3, a value of the scaling factor depending on a ratio between a distance of the first reference image and a distance of the second reference image from the current image.

[0040] Embodiment 5. The apparatus of any one of embodiments 1 to 4, further comprising a reference image selection unit (510) configured to obtain two reference images, select the first reference image from the two reference images, and select the second reference image from the two reference images.

[0041] Embodiment 6, the apparatus according to Embodiment 5, the reference picture selection unit (510) is configured to select the first picture or the second picture based on whether the first picture or the second picture is referred to in a bitstream by an index related to a predefined reference picture list among at least two reference picture lists, the bitstream also including encoded picture blocks of the video, wherein a reference picture list associates an index with a position of a reference picture relative to the current picture.

[0042] Embodiment 7, the apparatus according to Embodiment 6, the reference picture selection unit (510) is configured to select a picture with a highest position in the reference picture list as the first reference picture if the two reference pictures are referred to in the bitstream by an index in a same predefined reference picture list.

[0043] Embodiment 8, the apparatus according to Embodiment 5, the reference picture selection unit (510) is configured to select a picture with a lowest temporal layer among the two pictures as the first reference picture.

[0044] Embodiment 9, the apparatus according to Embodiment 5, the reference picture selection unit (510) is configured to select a picture with a smallest base quantization value as the first reference picture.

[0045] Embodiment 10, the apparatus according to Embodiment 5, the reference picture selection unit (510) is configured to select a picture with a shortest distance to the current picture as the first reference picture.

[0046] Embodiment 11, the apparatus according to Embodiment 5, the reference picture selection unit (510) is configured to select the first reference picture and the second reference picture such that a magnitude of an estimate of the first motion vector is smaller than a magnitude of an estimate of the second motion vector.

[0047] Embodiment 12, the apparatus according to any one of Embodiments 1 to 11, further comprising: a motion vector estimator (820) configured to determine the estimate of the first motion vector and the estimate of the second motion vector by selecting the estimate of the first motion vector and the estimate of the second motion vector from a set of motion vector candidates based on a similarity between a template and a respective partial area of pictures involved by the motion vector candidates.

[0048] Embodiment 13. A video decoder (200) for decoding a plurality of pictures from a bitstream, comprising an inter prediction unit (210) comprising an apparatus according to any one of embodiments 1 to 12 and a prediction unit configured to determine a prediction block based on a partial region of the first reference picture to which the first motion vector refers and a partial region of the second reference picture to which the second motion vector refers; a bitstream parser (203) configured to obtain an estimate of the first motion vector and an estimate of the second motion vector from the bitstream; and a reconstruction unit (211) configured to reconstruct the current block based on the prediction block.

[0049] Embodiment 14. A video encoder (100) for encoding a plurality of pictures into a bitstream, comprising an inter prediction unit (110) comprising an apparatus according to any one of embodiments 1 to 12 and a prediction unit configured to determine a prediction block based on a partial region of the first reference picture to which the first motion vector refers and a partial region of the second reference picture to which the second motion vector refers; a bitstream former (103) configured to include an estimate of the first motion vector and an estimate of the second motion vector in the bitstream; and a reconstruction unit (111) configured to reconstruct the current block based on the prediction block and store the reconstructed block in a memory.

[0050] Embodiment 15. A method (700) for determining a first motion vector (MV0") in a first reference picture of a video and a second motion vector (MV1") in a second reference picture of the video, the first and second motion vectors being applied for inter prediction of a block of the video, the method comprising: obtaining (701) an estimate of the first motion vector (MV0); determining (703) the first motion vector by searching within a search space (310) specified based on the estimate of the first motion vector; obtaining (705) an estimate of the second motion vector (MV1); and computing (707) the second motion vector (MV1") based on the estimate of the second motion vector (MV1) and the first motion vector (MV0"). BRIEF DESCRIPTION OF DRAWINGS

[0051] The example embodiments will be described in detail below with reference to the following drawings:

[0052] Figure 1 is a block diagram of an example structure of an encoder to which motion vector derivation and refinement can be applied;

[0053] Figure 2 is a block diagram of an example structure of a decoder to which motion vector derivation and refinement can be applied;

[0054] Figure 3is a schematic diagram of motion vector refinement of a first motion vector;

[0055] Figure 4 is a schematic diagram of motion vector refinement of a second motion vector based on the refined first motion vector;

[0056] Figure 5 is a block diagram of a motion vector refinement apparatus;

[0057] Figure 6 is a schematic diagram of various configurations of search space;

[0058] Figure 7 is a schematic diagram of a method for determining a motion vector according to the present embodiment;

[0059] Figure 8 is a block diagram of an inter prediction unit for determining a motion vector and performing intra prediction according to the present embodiment. DETAILED DESCRIPTION

[0060] The present invention relates to determining motion vectors for multi-reference prediction. Motion vector determination can be used for motion estimation in video encoding and decoding. Exemplary encoders and decoders implementing motion estimation using search space construction according to the present invention are described below.

[0061] Figure 1 An encoder 100 is shown, which comprises an input for receiving an input block of a frame or image of a video stream and an output for generating an encoded video bitstream. The term "frame" is used as a synonym for image in the present invention. However, it is noted that the present invention is also applicable in the field of interlaced applications. Generally, an image comprises m by n pixels. This corresponds to image samples and can comprise one or more color components. For simplicity, the following description refers to pixels, i.e. luma samples. However, it is noted that the motion vector search of the present invention is applicable to any color component including chroma or components of a search space such as RGB. On the other hand, it can be beneficial to perform motion vector estimation on only one component and apply the determined motion vector to multiple (or all) components.

[0062] The input blocks to be encoded do not necessarily have the same size. An image can comprise blocks of different sizes and the block grid can also differ for different images.

[0063] In an illustrative implementation, the encoder 100 is configured to apply prediction, transform, quantization and entropy coding to the video stream. The transform, quantization and entropy coding are performed by a transform unit 101, a quantization unit 102 and an entropy coding unit 103, respectively, to generate an encoded video bitstream as output.

[0064] The video stream can comprise a plurality of frames, wherein each frame is divided into blocks of a certain size, which are either intra- or inter-coded. For example, the blocks of the first frame of the video stream are intra-coded by the intra-prediction unit 109. Intra-coded frames are encoded using only information from the same frame, so that the frame can be decoded independently and can provide an entry point for random access in the bitstream. The blocks of other frames of the video stream can be inter-coded by the inter-prediction unit 110: information from previously encoded frames (reference pictures) is used to reduce temporal redundancy, so that each block of an inter-coded frame is predicted from a block in a reference frame. The mode selection unit 108 is used to select whether a frame block is processed by the intra-prediction unit 109 or the inter-prediction unit 110. The block also controls the parameters of the intra- or inter-prediction. In order to achieve a refresh of the picture information, intra-coded blocks can be provided in inter-coded frames. Furthermore, intra-coded frames comprising only intra-coded blocks can be inserted uniformly in the video sequence in order to provide entry points for decoding, i.e. points at which the decoder can start decoding without using information from previously encoded frames.

[0065] The intra-prediction unit 109 is a block prediction unit. In order to perform spatial or temporal prediction, the encoded blocks can also be processed by the inverse quantization unit 104 and the inverse transform unit 105. After the blocks have been reconstructed, the loop filter unit 106 is used to further improve the quality of the decoded picture. The filtered blocks then form the reference frames, which are subsequently stored in the frame buffer 107. This decoding loop (decoder) on the encoder side provides the advantage that the same reference frames are generated as the reference pictures reconstructed on the decoder side. Accordingly, the encoder and decoder side work in a corresponding manner. Here, the term "reconstruction" means that a reconstructed block is obtained by adding a predicted block to a decoded residual block.

[0066] The inter prediction unit 110 receives as input a current frame or picture to be inter coded and blocks of one or more reference frames or pictures in the frame buffer 107. The inter prediction unit 110 applies motion estimation and motion compensation. Motion estimation serves to obtain a motion vector and a reference frame based on some cost function. Then, motion compensation describes a current block of the current frame from a reference block of the reference frame by a translation, i.e. by the motion vector, to the current frame. The inter prediction unit 110 outputs a prediction block for the current block, wherein the prediction block minimizes the cost function. For example, the cost function can be the difference between the current block to be coded and its prediction block, i.e. the cost function minimizes the residual block. For example, the residual block is minimized based on a sum of absolute difference, SAD, between all pixels (samples) of the current block and a candidate block in a candidate reference picture. However, in general, any other similarity measure can be applied, e.g. mean square error, MSE, or structural similarity, SSIM, measure.

[0067] However, the cost function can also be the number of bits necessary to code such an inter coded block and / or the distortion caused by such a coding. Thus, a rate-distortion optimization procedure can be used to decide on the motion vector selection and / or in general on coding parameters, e.g. whether to use inter prediction or intra prediction for a block and which settings to use.

[0068] The intra prediction unit 109 receives as input a block of a current frame or picture to be intra coded and one or several reference samples from a reconstructed area of the current frame. Then, the intra prediction describes the pixels of a current block of the current frame from a function of the reference samples. The intra prediction unit 109 outputs a prediction block for the current block, wherein the prediction block advantageously minimizes the difference between the current block to be coded and its prediction block, i.e. minimizes the residual block. For example, the residual block can be minimized based on a rate-distortion optimization procedure. In particular, the prediction block is obtained as a directional interpolation of the reference samples. The direction can be determined by a rate-distortion optimization and / or by computing the similarity measures mentioned above in connection with intra prediction.

[0069] Subsequently, the transform unit 101 transforms the difference between the current block and the prediction block, i.e. the residual block. The transform coefficients are quantized by the quantization unit 102 and coded by the entropy coding unit 103. The resulting coded video bitstream comprises intra coded blocks and inter coded blocks and corresponding signaling (e.g. mode indication, indication of motion vectors and / or intra prediction direction). The transform unit 101 can apply a linear transform, e.g. a Fourier or discrete cosine transform (DFT / FFT or DCT). Such a transform into the spatial frequency domain provides the advantage that the resulting coefficients usually have higher values in lower frequencies. Thus, after an efficient coefficient scan (like zig-zag) and quantization, the resulting sequence of values usually has some larger values at the beginning and ends with a string of zeros. This makes the coding more efficient. The quantization unit 102 performs the actual lossy compression by reducing the resolution of the coefficient values. Then, the entropy coding unit 103 assigns binary code words to the coefficient values to produce the bitstream. The entropy coding unit 103 also codes the signaling information (not shown in Figure 1

[0070] Figure 2 A video decoder 200 is shown. In particular, the video decoder 200 comprises a reference picture buffer 207 and an intra prediction unit 209, which is a block prediction unit. The reference picture buffer 207 is configured to store at least one reference frame reconstructed from the coded video bitstream, wherein the reference frame is different from a current frame (current decoded frame) of the coded video bitstream. The intra prediction unit 209 is configured to generate a prediction block, which is an estimate of a to-be-decoded block. The intra prediction unit 209 is configured to generate the prediction block based on reference samples obtained from the reference picture buffer 207.

[0071] The decoder 200 is configured to decode the coded video bitstream generated by the video encoder 100. Preferably, the decoder 200 and the encoder 100 both generate the same prediction block for the respective to-be-coded / decoded block. The reference picture buffer 207 and the intra prediction unit 209 have features similar to the Figure 1 The reference picture buffer 207 and the intra prediction unit 209 have features similar to the reference picture buffer 107 and the intra prediction unit 109 in the video encoder 100.

[0072] The video decoder 200 further comprises other units present in the video encoder 100, e.g. the inverse quantization unit 204, the inverse transform unit 205 and the loop filter unit 206, which correspond to the inverse quantization unit 104, the inverse transform unit 105 and the loop filter unit 106 of the video encoder 100, respectively.

[0073] ​The entropy decoding unit 203 decodes the received encoded video stream and obtains the quantized residual transform coefficients and signaling information accordingly. The quantized residual transform coefficients are transmitted to the inverse quantization unit 204 and the inverse transform unit 205 to generate residual blocks. The residual blocks are added to the prediction blocks, and the addition process is transmitted to the loop filtering unit 206 to obtain the decoded video. The frames of the decoded video can be stored in the reference image buffer 207 and used as reference frames for inter-frame prediction.

[0074] generally, Figure 1 and Figure 2 Intra-prediction units 109 and 209 can use reference samples in the encoded region to generate prediction signals for blocks that need to be encoded or decoded.

[0075] Entropy decoding unit 203 receives the encoded bitstream as its input. Generally, the bitstream is first parsed, i.e., signaling parameters and residuals are extracted from it. Typically, the syntax and semantics of the bitstream are defined by standards, allowing the encoder and decoder to operate in an interoperable manner. As described in the background section above, the encoded bitstream does not only include prediction residuals. In the case of motion-compensated prediction, motion vector indications are also encoded in the bitstream and parsed from it at the decoder. Motion vector indications can be provided by offering a reference image of the motion vectors and their coordinates. So far, the decoding of the complete motion vector has been considered. However, it is also possible to encode only the difference between the current motion vector and the previous motion vector in the bitstream. This method allows redundancy between motion vectors in adjacent blocks to be utilized.

[0076] To efficiently encode reference images, the H.265 codec (ITU-T, H.265, Series H: Audio, Video and Multimedia Systems: Efficient Video Coding) provides a list of reference images, assigning each reference frame to a list index. Reference frames are then transmitted in the bitstream by including the corresponding assigned list index. This list can be defined in the standard or transmitted at the beginning of the video or a set of frames. It should be noted that in H.265, two reference image lists are defined, referred to as L0 and L1. Reference images are then signaled and transmitted in the bitstream by indicating the list (L0 or L1) and the index in the list associated with the desired reference image. Providing two or more lists can have the advantage of better compression. For example, L0 can be used for both unidirectional and bidirectional inter-frame prediction slices, while L1 can only be used for bidirectional inter-frame prediction slices. However, in general, the invention is not limited to any of the contents of lists L0 and L1.

[0077] Lists L0 and Ll can be defined and fixed in the standard. However, it is possible to make coding / decoding more flexible by sending these two lists at the beginning of a video sequence. Accordingly, an encoder can configure lists L0 and Ll using certain reference pictures ordered according to an index. Lists L0 and Ll can have the same fixed size. There can be more than two lists in general. A motion vector can be sent directly by the coordinates in a reference picture. Or, as specified in H.265, a list of candidate motion vectors can be constructed and an index associated with a particular motion vector in the list can be sent.

[0078] The motion vector of a current block is usually related to the motion vectors of neighboring blocks in the current picture or a previously coded picture. This is because neighboring blocks are likely to correspond to the same moving object with similar motion, and the motion of the object is unlikely to change suddenly over time. Therefore, using the motion vectors in neighboring blocks as predictors reduces the size of the motion vector difference to be sent. An MVP is usually derived from the decoded motion vectors of spatial neighboring blocks or temporal neighboring blocks in a collocated picture. In H.264 / AVC, this is done by taking the median of the three spatial neighboring motion vectors component by component. Using this method, the signaling of the predictor is not needed. Temporal MVPs from a collocated picture are only considered in the so-called temporal direct mode of H.264 / AVC. The direct mode of H.264 / AVC is also used to derive other motion data besides the motion vector. Therefore, the direct mode is more related to the block merge concept in HEVC. In HEVC, the method of deriving the MVP implicitly is replaced by a technique called motion vector candidate, which explicitly indicates which MVP of a list of MVPs is used for motion vector derivation. The variable coding quad tree block structure in HEVC can make a block have several neighboring blocks whose motion vectors are potential MVP candidates. Take the left neighboring block as an example. In the worst case, when a 64x64 luma coding tree block is not split further and the left neighboring block is split to the maximum depth, the 64x64 luma prediction block can have 16 4x4 luma prediction blocks on the left side.

[0079] To account for this flexible block structure, advanced motion vector prediction (AMVP) was introduced to modify the motion vector competition. During the development of HEVC, the original AMVP design was significantly simplified to strike a good balance between coding efficiency and an easy-to-implement design. The original design of AMVP included five MVPs from three different types of predictors: three motion vectors from spatial neighboring blocks, a median of the three spatial predictors, and a scaled motion vector from a collocated, temporal neighboring block. In addition, the predictor list was modified by reordering to put the most likely motion predictor in the first position and by pruning redundant candidates to ensure minimal signaling overhead. The final design of AMVP candidate list construction included the following two MVP candidates: a) up to two spatial candidate MVPs derived from five spatial neighboring blocks; b) one temporal candidate MVP derived from two temporal collocated blocks when both spatial candidate MVPs are not available or identical; c) zero motion vector when either the spatial candidate, the temporal candidate, or both candidates are not available. Details on motion vector determination can be found in the book by V. Sze et al. (Eds.), High Efficiency Video Codding (HEVC): Algorithms and Architectures, Springer, 2014, in particular in Chapter 5, which is incorporated herein by reference.

[0080] To further improve the motion vector estimation without increasing the signaling overhead, it can be beneficial to further refine the motion vector derived at the encoder side and provided in the bitstream. The motion vector refinement can be performed at the decoder without the help of the encoder. In its decoder loop, the encoder can apply the same refinement to obtain the corresponding reference picture. The refinement is done by determining a template, determining a search space, and finding the reference picture portion in the search space that best matches the template. The position of the best matching portion determines the best motion vector, which is then used to obtain the predictor for the current block, i.e., the current block being reconstructed.

[0081] Figure 3 A search space (search area) 310 is shown, which includes integer pixel positions (full points) and fractional pixel positions (empty points) of the reference picture. In this example, the fractional pixel positions are half-pixel positions. As mentioned above, the fractional pixel positions can be obtained from the integer (full-pixel) positions by interpolation, such as bilinear interpolation.

[0082] In bi-prediction of the current block, the two prediction blocks resulting from the first motion vector in list L0 and the second motion vector in list L1 are combined into a single prediction signal that can better adapt to the original signal than a single prediction, resulting in less residual information and more efficient compression.Figure 3 A current block 320 in the current picture is also shown. At the decoder, since the current block is not available because it is being decoded due to the motion vector refinement, a template is used, which is an estimate of the current block and is constructed based on already processed (i.e. encoded at the encoder side and decoded at the decoder side) image parts.

[0083] For example, the template can be constructed based on samples belonging to already decoded (i.e. decoded before the current block) samples of the current picture. Additionally or alternatively, these samples can belong to any previously decoded reference picture. As an example, the samples used in the template construction can belong to a reference picture that is decoded before the current picture and that is located before the current picture in display order. Or, the samples can belong to a reference picture that is decoded before the current picture and that is located after the current picture in display order. Finally, the template can be constructed based on a combination of samples from two different reference pictures. It is clear to the skilled person that different approaches can be used to obtain the template, such that the constructed template can be used to estimate the current block.

[0084] First, as Figure 3 indicated, the estimate of the first motion vector MV0 and the estimate of the second motion vector MV1 are received as input at the decoder 200. At the encoder side 100, the motion vector estimates MV0 and MV1 can be obtained by block matching and / or by searching a candidate list (e.g. merge list) of motion vectors of neighboring blocks (located in the same picture or in a neighboring picture) of the current block. Then, MV0 and MV1 are advantageously signaled in the bitstream to the decoder side. However, it is to be noted that the first determination phase at the encoder can also be performed by template matching, which provides the advantage of reducing the signaling overhead.

[0085] At the decoder side 200, the motion vectors MV0 and MV1 are advantageously obtained based on information in the bitstream. MV0 and MV1 are directly signaled or signaled with distinction, and / or an index in a motion vector list (merge list) is signaled. However, the present application is not limited to signaling motion vectors in the bitstream. Instead, the motion vectors can be determined by template matching in the first phase, which corresponds to the operation at the encoder. The template matching (motion vector derivation) in the first phase can be performed based on a search space that is different from the search space of the second (refinement) phase. In particular, the refinement can be performed on a search space with higher resolution (i.e. shorter distances between search positions).

[0086] The indication of the two reference pictures pointed to by MV0 and MV1 is also provided to the decoder. As a result of the previous processing, i.e. the respective encoding and decoding, the reference pictures are stored in reference picture buffers at the encoder side and at the decoder side. A reference picture selection unit of the apparatus for determining the motion vector is configured to select a first reference picture pointed to by MV0 and a second reference picture pointed to by MV1. After this selection, the reference picture selection unit determines whether the first reference picture or the second reference picture is used for performing the motion vector refinement. In Figure 3 In the example shown in Fig. 3, the first reference picture pointed to by the motion vector MV0 is selected for the search. For the motion vector refinement, a search area 310 in the first reference picture is defined around the candidate position pointed to by the motion vector MV0. The candidate search space positions within the search area 310 are analyzed to find the block most similar to the template block by performing template matching within the search space and determining a similarity measure such as sum of absolute difference (SAD). As mentioned above, in one implementation, the template is constructed based on a combination of samples in two different reference pictures with their respective motion vectors being MV0 and MV1. The template matching is performed to find the block based on the point in the search area 310 most similar to the template. Alternatively, in another implementation, the template is constructed to find the similarity between the prediction block P0 generated based on MV0 in L0 and the prediction block P1 generated based on MV1 in L1. For the refinement of MV0, the template matching includes finding the block based on the point in the search area 310 determined by the similarity measure, e.g. SAD, between P0 and P1. The position in the search space 310 represents the top-left corner of the matching of the template 320. As mentioned above, the top-left corner is just a convention and any point in the search space, e.g. the center point 330, can be used to represent the matching position.

[0087] The candidate position with the lowest SAD value is determined as the motion vector MV0". In Figure 3 The position pointed to by MV0" is a half-pel position which is 1.5 pixels different from the initial estimate MV0 in vertical direction and the same in horizontal direction.

[0088] According to the present application, for bi-prediction or multi-reference prediction, at least one motion vector of the current block is refined by a calculation based on another refined motion vector of the current block instead of by template matching. Figure 4 This refinement is illustrated. In particular, the motion vector MV1" is calculated based on the motion vector estimate MV1 and a function of MV0" - MV0 instead of performing a second template matching. In Figure 4 In the example shown in Fig. 4, the second reference picture pointed to by the motion vector MV1 is selected for the search. For the motion vector refinement, a search area 410 in the second reference picture is defined around the candidate position pointed to by the motion vector MV1. The candidate search space positions within the search area 410 are analyzed to find the block most similar to the template block by performing template matching within the search space and determining a similarity measure such as sum of absolute difference (SAD). As mentioned above, in one implementation, the template is constructed based on a combination of samples in two different reference pictures with their respective motion vectors being MV0 and MV1. The template matching is performed to find the block based on the point in the search area 410 most similar to the template. Alternatively, in another implementation, the template is constructed to find the similarity between the prediction block P0 generated based on MV0 in L0 and the prediction block P1 generated based on MV1 in L1. For the refinement of MV1, the template matching includes finding the block based on the point in the search area 410 determined by the similarity measure, e.g. SAD, between P0 and P1. The position in the search space 410 represents the top-left corner of the matching of the template 420. As mentioned above, the top-left corner is just a convention and any point in the search space, e.g. the center point 430, can be used to represent the matching position. Figure 3The first motion vector MV0" is described to be determined and corrected. Furthermore, the second motion vector MV1" is calculated by subtracting the difference between MV0" and MV0 from the estimated value of the second motion vector MV1".

[0089] This approach makes use of the fact that most of the motion in a video sequence falls into the category of "translation motion". In translation motion, an object moves with a constant velocity (at least between frames whose time samples are close to each other). This means that the object moves with the same pixel distance in the x- and y-directions in consecutive frames (if the time sampling period does not change over time). The present invention makes use of the principle of translation motion to some extent.

[0090] In the above example, the first motion vector MV0" is corrected by template matching, while the second motion vector is corrected by calculation. However, according to the present invention, a selection process can also be performed to set which motion vectors are corrected by template matching and which motion vectors are determined by calculation. Figure 5 A block diagram showing one example of a motion vector corrector 500 is shown.

[0091] The motion vector corrector 500 can be included in a device 810 for determining motion vectors as shown in Figure 8 The device can be included in an intra prediction unit 800 which can replace the inter prediction unit 110 as shown in Figure 1 and / or can be included in an inter prediction unit 210 as shown in Figure 2

[0092] More specifically, a device 810 is provided for determining a first motion vector in a first reference image and a second motion vector in a second reference image. The first motion vector and the second motion vector are applicable for inter prediction of a block of pixels in a current image.

[0093] The device 810 comprises a motion vector corrector 500 as shown in Figure 5 detailed in the above, the motion vector corrector 500 further comprises a motion vector correction unit 530 for obtaining an estimated value MV0 of the first motion vector MV0" and determining the first motion vector MV0" by searching within a search space specified based on the estimated value MV0. The device further comprises a motion vector calculation unit 550 for obtaining an estimated value MV1 of the second motion vector MV1" and calculating the second motion vector MV1" based on the estimated value MV1 of the second motion vector MV1" and the first motion vector MV0".

[0094] ​In this example, the apparatus comprises a first branch comprising a motion vector calculation unit 530 and a motion vector calculation unit 550, a second branch comprising a motion vector calculation unit 540 and a motion vector calculation unit 560, and a switch 520 for opening one of the two branches and closing the other branch. The second branch is similar to the first branch, with the main difference being that the first motion vector MV1" and the second motion vector MV0" are outputted by processing MV1 as the estimate of the first motion vector and MV0 as the estimate of the second motion vector.

[0095] More specifically, the motion vector refinement unit 540 is configured to obtain an estimate MV1 of the first motion vector MV1" and to determine the first motion vector MV1" by searching within a search space specified based on the estimate MV1. The apparatus further comprises a motion vector calculation unit 560 configured to obtain an estimate MV0 of the second motion vector MV0" and to calculate the second motion vector MV0" based on the estimate MV0 of the second motion vector MV0" and the first motion vector MV1".

[0096] Figure 5 The application is shown to be applied in bi-prediction, in which there are two motion vectors determined in the two reference pictures, namely the motion vectors MV0" and MV1". Correspondingly, it is only necessary to take one of MV0 and MV1 for template matching and to use the other one of MV1 and MV0 for the calculation, i.e. the image to be used for the refinement is selected by template matching.

[0097] The process of decoder motion vector refinement (DMVR) is performed by the apparatus 500. The motion vector refinement unit 530 or 540 (depending on which of the motion vector estimates MV0 and MV1 the template search is performed for) receives the motion vector estimate MV0 or MV1 on the input 505 from the motion vector estimator 820 and sets a search area 310 around MV0 or MV1. Figure 3 and Figure 4 The search area in the example of Fig. 6 is 3x3 integer pixels in size and is half-pel interpolated to 7x7, so there are 49 positions in total. In general, the form and / or size of the search area can be different, the application is independent of the size and shape of the search area. The search area can have a predetermined or predefined size. In other words, the shape and size of the search space can be fixed and specified in the standard. Alternatively, the user can select one form and size out of several possible forms and sizes manually within the encoder settings and / or automatically depending on the content of the video.

[0098] Some examples of sizes and shapes of search spaces are Figure 6As shown, a full triangle marks the center pixel of the search space, and a full circle marks the remaining positions in the search space. It should be noted that the search space can also be expanded using fractional interpolation (half a pixel, a quarter a pixel, etc.). This invention is generally not limited to any specific pixel pattern.

[0099] For each location or fractional location within the search region, template matching is performed using a template that is closest to the current block, thus providing a SAD value for each search space coordinate. It's important to note that the integer pixel resolution and half-pixel resolution here describe the resolution of the search space, i.e., the displacement of the searched location relative to the uncorrected motion vector input to the process. Therefore, the search coordinates do not necessarily coincide with the actual pixel coordinates on the image plane. In other words, the integer pixel (sample) resolution of the search space does not necessarily mean that the search space location lies on an integer pixel in the reference image. Integer locations in the search space can coincide with fractional pixels in the reference image, especially if the initial motion vectors MV0, MV1 point to fractional pixels within the reference image.

[0100] The SAD values ​​of the locations within the search area are compared to determine the location with the minimum SAD value. This location with the minimum SAD value is designated as the motion vector MV0". As mentioned in the background section, SAD is merely an example; any other metric, such as MSE, SSIM, correlation coefficient, etc., can typically be used. The determined first motion vector MV0" and the estimated value of the second motion vector MV1 are passed to the motion vector calculation unit 550, where the motion vector MV1 is determined (corrected) through calculation.

[0101] As the first example method, combined with Figure 4 As described above, the motion vector calculation unit 550 is used to calculate the second motion vector by adding a function of the estimated value of the second motion vector to the difference between the estimated values ​​of the first motion vector and the first motion vector, or the difference between the estimated values ​​of the first motion vector and the first motion vector.

[0102] The following calculation can be performed by adding the difference:

[0103] MV1” = MV1 + (MV0 – MV0”) (Equation 1)

[0104] exist Figure 4 In the case shown, when one of the two reference images is located before the current image and the other is located after the current image, Equation 1 (with the negative value of the difference MV0” – MV0) is valid. Figure 4It can be seen that for extrapolating the regular motion, the difference between the current motion vector and the first motion vector in the previous picture is projected in reversed sign to the picture after the current picture.

[0105] If both reference pictures of MV0 and MV1 are located after or before the current picture, the difference is added directly without changing the sign:

[0106] MV1" = MV1 + (MV0" - MV0) (Equation 2)

[0107] The above-mentioned front / back positions are arranged in display order. The display order of pictures in a video can be different from the encoding / decoding order, i.e. from the order in which the encoded pictures are embedded in the bitstream. The display order of pictures can be given by the picture order count (POC). In particular, the POC in H.265 / HEVC is transmitted in the slice header of each slice being a picture or a partial region thereof.

[0108] The condition for determining that one of the two reference pictures is located before the current picture and the other one is located after the current picture can be implemented using the following parameters POC:

[0109] (POCi - POC0) * (POCi - POC1) < 0 (Equation 3)

[0110] where POCi is the display order (number) of the current picture, POC0 is the display order of the first reference picture, and POC1 is the display order of the second reference picture. If the condition in Equation 3 is true, the first reference picture is located before the current picture and the second reference picture is located after the current picture, and vice versa. On the other hand, if the condition of Equation 3 is not true, both reference pictures are located before or after the current picture. However, it is noted that this condition can be implemented in any way that is able to check whether the signs of the two difference expressions (POCi - POC0) and (POCi - POC1) are the same. The above Equation 3 is merely an example using the multiplication "*", because of the relatively simple implementation.

[0111] In the case of bi-prediction, where one of the reference pictures is located before the current picture and the other one is located after the current picture (the condition in Equation 3 is true), the addition with the function of the difference can be calculated as follows:

[0112] MV1" = MV1 - f(MV0" - MV0) (Equation 4)

[0113] Likewise, if both reference pictures are before or after the current picture in display order (the condition in Equation 3 is false), the calculation will add the difference without changing the sign:

[0114] MV1" = MV1 + f(MV0" - MV0) (Equation 5)

[0115] For example, the function can include scaling and / or clipping.

[0116] If the function is scaling, the calculation of the motion vector can be as follows:

[0117] MV1" = MV1 - k*(MV0" - MV0) = MV1 + k*(MV0 - MV0") (Equation 6)

[0118] where "*" denotes multiplication (scaling) and k is a scaling parameter. If k = 1, Equation 1 is obtained. For a fixed (predefined) k, Equation 6 applies to the case where one reference picture is before the current picture and the other reference picture is after the current picture. For the case where both reference pictures are before or after the current picture, the sign needs to be changed:

[0119] MV1" = MV1 + k*(MV0" - MV0) = MV1 - k*(MV0 - MV0") (Equation 7)

[0120] According to one embodiment, the value of the scaling parameter depends on the ratio between the distances of the first and second pictures from the current picture, respectively. Thus, the value of k is not predefined and not fixed, but can vary with the reference pictures from which the motion vector is defined. For example, the scaling factor k can be given as follows:

[0121] k = -(POCi - POCi) / (POCi - POC0) (Equation 8)

[0122] It is noted that Equation 8 above already takes into account the change of sign, since the scaling parameter will change sign depending on whether both reference pictures are on the same side (before or after) or on different sides of the current picture in display order.

[0123] Although it can be beneficial to take into account the distances of the first and second reference pictures from the current picture, respectively, it is still possible to set k = 1 in Equations 1 and 2, even if these distances are different from each other. This approach is simpler and the quality of the corrected motion vector can still be much higher than without the correction.

[0124] After the scaling operation, a rounding or clipping operation can be performed. Here, rounding refers to providing an integer or real number that is closest to the input value with a predefined accuracy. Clipping refers to removing numbers from the input value that have a higher accuracy than the predefined accuracy. This can be particularly beneficial in fixed-point arithmetic, which is applied in typical computing systems.

[0125] Since the motion vector correction is only performed for one of the two reference pictures, the search space for template matching is reduced by 50%.

[0126] After the above processing of the current block in the current picture, the two reference pictures and their respective associated motion vectors MV0" and MV1" are provided at the output 580. These motion vectors are used to determine a prediction value for the current block, e.g. by averaging two prediction values obtained by taking the respective portions of the reference pictures pointed to by the motion vectors MV0" and MV1" that correspond to the size and shape of the current block. This is performed by the prediction unit 890.

[0127] In general, the prediction unit 890 determines the prediction block by merging the portion of the first reference picture involved by the first motion vector MV0" and the portion of the second reference picture involved by the second motion vector MV1".

[0128] The merged prediction signal can better adapt to the original signal than a single directional prediction, resulting in less residual information and more efficient compression.

[0129] As mentioned in the previous paragraph, the output motion vectors MV0" and MV1" of the apparatus 500 can result in less residual information. Therefore, using the motion vectors MV0" and MV1" can improve the prediction performance compared to the initial motion vectors MV0 and MV1. It is noted that the apparatus 500 can be used multiple times in order to further improve the prediction performance. In this case, the output of the first application of the apparatus 500 is considered as the input of the second application, and the equations between the motion vectors (equations 1 to 8) remain equal after each application. In this case, since the motion vectors MV0" and MV1" provide a better estimate for the current block after the first application of the apparatus 500, the template used in the MV corrector unit 530 is constructed based on the samples pointed to by MV0" or the samples pointed to by MV1" in the second application.

[0130] According to an exemplary embodiment, the apparatus 500 further comprises a reference picture selection unit 510 for obtaining two reference pictures and selecting one of the two reference pictures as the first reference picture and the second reference picture.

[0131] In other words, the reference picture selection unit 510 controls which of the motion vectors of the current block is corrected by template matching and which is corrected by the calculation based on the other corrected motion vector. Several examples of how the reference picture selection unit 510 can perform the selection process are provided below.

[0132] Figure 7 A schematic diagram illustrating a method 700 of determining a first motion vector in a first reference picture and a second motion vector in a second reference picture is shown. For example, the digital video encoder 100 or the decoder 200 described above, in particular the inter prediction unit 110 or 210, can employ the process 700 to obtain the first motion vector and the second motion vector which are applied for inter prediction of a block of a picture in a current picture. The method 700 comprises the following steps: step 701 : obtaining an estimate of the first motion vector; step 703: determining the first motion vector by searching within a search space specified based on the estimate of the first motion vector; step 705: obtaining an estimate of the second motion vector; step 707: calculating the second motion vector based on the estimate of the second motion vector and the first motion vector.

[0133] Example 1

[0134] In a first example, the reference picture selection unit 510 determines which of the motion vectors based on the list to which the respective reference picture referred to by the motion vector belongs to within the search space. In particular, in order to signal the prediction value information which is used by the decoder to recover the same prediction value as used at the encoder within the bitstream, some codecs signal the reference picture by including an index associated with the particular reference picture in the reference picture list in the bitstream. For example, the reference picture list (list of reference pictures) can be a table which is available at both the encoder and the decoder and which associates the index with the position of the reference picture relative to the current picture.

[0135] There can be more than one reference picture list. For example, the reference picture list L0 and the reference picture list L1 as commonly used in H.265 / HEVC. In order to signal the reference picture in the bitstream, first the reference picture list is signaled and then the index in the signaled reference picture list is signaled.

[0136] Subsequently, the reference picture selection unit 510 is advantageously used to select the first reference picture or the second reference picture depending on whether the first reference picture or the second reference picture is referred to in the bitstream by an index in a predefined reference picture list. In this context, the term "predefined" refers, for example, to being fixedly defined in the standard or being defined for the entire video in the signaling. By way of example, the predefined list can be the list L0. Then, if the first reference picture is referred to in the reference picture list L0 and the second list is referred to in the reference picture list L1, the first motion vector pointing to the first reference picture is corrected by template matching because the first motion vector is referred to in L0 and the second motion vector is calculated because the second motion vector is not referred to in L0. However, it is noted that the predefined list is not limited to the list L0. Rather, any reference picture list used can be predefined. Typically, L0 lists more neighboring reference pictures than L1. Alternatively, L0 can comprise more reference pictures preceding the current picture in display order and L1 can comprise more pictures succeeding the current picture in display order. However, the present application is independent of which specific differences between the two or more reference picture lists exist.

[0137] It is noted that the case can occur that the first reference picture and the second reference picture to which the first motion vector and the second motion vector, respectively, point are referred to in the same reference picture list. For example, both the first reference picture and the second reference picture can be referred to in the predefined picture list L0. Alternatively, if bi-prediction is applied for the current coding block, one reference picture from list L0 and one reference picture from list L1 have to be indicated, wherein these two reference pictures can be contained together in one (or both) of the reference lists. This is because a reference picture can be present in both lists (list L0 and list L1).

[0138] If both the first reference picture and the second reference picture are contained in the predefined list (e.g. L0), the picture having the highest position in the list (L0) is selected as the reference picture for the motion vector correction by template matching of the corresponding motion vector pointing thereto.

[0139] If both reference pictures are contained in a reference picture list that is not predefined (e.g. L1 if L0 is predefined), the same selection can be made. In other words, if the reference pictures are referred to in a reference picture list other than the predefined reference picture list, the picture having the highest position in the list is selected for the template matching based motion vector correction.

[0140] In summary, the reference picture selection unit 510 is used to select the picture having the highest position in the reference picture list as the first reference picture if both reference pictures are referred to in the bitstream by an index in the same predefined reference picture list.

[0141] The present example provides a simple implementation without any additional intelligence selection process. In particular, when the parsed reference picture list is a predefined list (e.g. L0), the reference picture selection unit 510 controls the switches 520 and 570 to select the upper or lower branch in the block diagram of Figure 5

[0142] Example 2

[0143] According to a second example, the reference picture selection unit 510 is configured to select the lowest temporal layer picture of the two (or more) reference pictures as the first reference picture (corrected by template matching) for the current block. In other words, depending on the temporal layer of the reference pictures related to the motion vectors MV0 and MV1, the reference picture selection unit 510 controls the switches 520 and 580 to select the upper branch (530, 550) or the lower branch (540, 560).

[0144] It is noted that in the above Figure 5 In general, in any embodiment and example of the present application, a single motion vector correction unit 530 and a single motion vector calculation unit 550 are provided and only switched to the input of the single branch depending on the control of the reference picture selection unit 210.

[0145] In particular, the temporal layer of the two (or more) reference pictures is determined. The temporal layer of a reference picture indicates the number of pictures that have to be decoded before the reference picture can be decoded. The temporal layer is usually set at the encoder that encodes the video motion pictures of different temporal layers. The temporal layer can be contained in the code stream. Thus, the decoder can determine which reference picture belongs to which temporal layer based on the signaling information from the code stream. Accordingly, subsequently, the first reference picture or the second reference picture of the lower temporal layer is selected as the reference picture for the motion vector correction by template matching. This approach can provide the advantage of selecting the reference picture for template matching less dependent on previously decoded pictures, so that the probability of errors and distortions is lower. Accordingly, the motion vector correction procedure is more flexible.

[0146] Example 3

[0147] In a third example, the reference picture selection unit 510 is configured to select the picture with the lowest base quantization value as the first reference picture (corrected by template matching). In other words, depending on the quantization parameter of the reference pictures related to the motion vectors MV0 and MV1, the reference picture selection unit 510 controls the switches 520 and 580 to select the upper branch (530, 550) or the lower branch (540, 560).

[0148] ​The quantization value or quantization parameter in this context is the information provided in the bitstream that enables determination of the quantization step. In well-known codecs such as H.264 / AVC and H.265 / HEVC, the quantization parameter determines the value by which the coefficients to be quantized are divided. The larger the quantization value, the coarser the quantization, which typically results in a poorer quality of the reconstructed image. Hence, a lower quantization value means that a higher quality of the reconstructed image can be provided. Selecting a reference image with a lower quantization parameter means that a better quality of the reference image is used for the motion vector refinement, which in turn results in a better refinement result.

[0149] The term "base" quantization value refers to a quantization value that is common for one image slice and serves as a base for all blocks. Such a value is typically signaled in the slice header, for example. Then, the difference to the base value is typically signaled in units of blocks or processing units.

[0150] However, the present application is not limited to any particular signaling, not even the presence of such a value. The same effect can be achieved by determining a base quantization value for an image from the quantization values of the elements in the image for which the quantization values have been signaled. In other words, the term "base quantization value" refers to a general quantization value for an image.

[0151] Example 4

[0152] According to a fourth example, the reference image selection unit 510 is configured to select as the first reference image the image that is closest in distance to the current image. In other words, depending on the distance of the reference images associated with the motion vectors MV0 and MV1 to the current image, the reference image selection unit 510 controls the switches 520 and 580 to select the upper branch (530, 550) or the lower branch (540, 560).

[0153] For example, the difference between the picture order count (POC) values POC0 and POC1 of the respective reference images associated with the motion vectors MV0 and MV1 and the POC value POCi of the current image is determined. The POC value specifies the display order of the images, not the coding. Hence, an image with POC = 2 is displayed before an image with POC = 8. However, it is noted that since the present application is not limited to the application in well-known codecs such as H.264 / AVC and H.265 / HEVC, the difference between the reference images and the current image can be determined in any other way without relying on the particular POC parameter.

[0154] The first reference picture associated with the motion vector MV0 or the second reference picture associated with the motion vector MV1 having a smaller absolute POC difference (between the reference picture and the current picture) is selected as the reference picture for the motion vector refinement because it is expected that the motion vector of a closer reference picture is more accurate and / or the reference block pointed to by the motion vector is more similar to the current block. This can lead to a better refinement quality.

[0155] Example 5

[0156] According to a fifth example, the reference picture selection unit is configured to select the first reference picture and the second reference picture such that the size of the estimate of the first vector is smaller than the size of the estimate of the second motion vector. In other words, depending on the length (size) of the motion vectors MV0 and MV1 associated with the respective reference pictures, the reference picture selection unit 510 controls the switches 520 and 580 to select the upper branch (530, 550) or the lower branch (540, 560).

[0157] The absolute size of the motion vectors MV0 and MV1 pointing to the first reference picture and the second reference picture is determined using a standard procedure for determining the absolute value of a vector. For example, the sum of the squared values of each vector component of the motion vector is determined. This sum or the square root can be used as the size of the motion vector, but the calculation of the square root has a higher computational cost. Assuming that the motion between pictures is usually small, the use of a smaller motion vector for the refinement provides the advantage that the likelihood of a correct determination is higher.

[0158] Some information about the estimate of the first motion vector MV0, the estimate of the second motion vector MV1 and the indices of the reference pictures involved by MV0 and MV1 can be received as input at the decoder. The motion vector information is usually transmitted in units of blocks, which can have different sizes. The same is true for the reference picture indication. A code stream parser, which is implemented as part of the entropy decoding unit 203, extracts the motion vector information from the code stream. The motion information can directly be the coordinates of the motion vector (given in coordinates of a point (0, 0) relative to the position of a block in the reference picture which has the same position as the current block in the current picture). Alternatively, a difference between the motion vector of a block which is located before the current block in decoding order can also be transmitted. Advantageously, this can be one of the spatially or temporally neighboring blocks of the current block.

[0159] According to another example, a motion vector determination apparatus also comprising the motion vector refiner 500 further comprises a motion vector estimation unit 820 for determining the estimated value of the first motion vector and the estimated value of the second motion vector by selecting the estimated value of the first motion vector and the estimated value of the second motion vector from a set of motion vector candidates based on a similarity between the template and a respective partial area of the images to which the motion vector candidates relate. In other words, the motion vectors (MV0 and MV1) are not necessarily determined based on template matching in a search space defined in the reference images. The search space can be given by a (merge) list listing indices related to motion vectors of spatially or temporally neighboring blocks or blocks in the vicinity of the current block. That is, the present invention is not limited to the way the estimated values of the motion vectors MV0 and MV1 are derived before being refined.

[0160] In summary, in a bi-prediction operation for a coding block, two prediction blocks from a motion vector (MV) of list L0 and a MV of list L1 are combined into a single prediction signal that better adapts to the original signal than a single prediction, resulting in less residual information and more efficient compression. The bi-prediction decoding process for a current block in a current image comprises the following processing steps:

[0161] First, the estimated value of the first motion vector MV0 and the estimated value of the second motion vector MV1 are received as input at the decoder side. The two reference images to which MV0 and MV1 point are located in the image buffer of the decoder because MV0 and MV1 have been decoded before the current image is processed. One of these reference images is selected for motion vector refinement by template matching, for example the reference image to which MV0 points. For motion vector refinement, a search area in the reference image to which MV0 points is defined around the candidate point to which MV0 points. Candidate search space positions within the search area are analyzed by template matching with the current block space and determining a similarity measure. The candidate search space position with the lowest dissimilarity value is determined as the motion vector MV0". The motion vector MV1" is calculated based on a function of MV1 and MV0" - MV0 instead of performing a second template matching.

[0162] According to one embodiment of the present invention, the similarity comparison is performed by comparing the sample to which MV0" points and the sample to which MV1" points. According to Figure 4Any point in the search space 310 pointed to by MV0" has a corresponding motion vector MV1" given by one or more of Equations 1-8. To compare the similarity between the sample blocks in the template matching process, the sample block pointed to by MV0" and the sample block pointed to by MV1" can be used in a function such as the SAD function. At this point, the template consists only of samples belonging to the reference picture referred to by MV1, and a motion vector correction operation is performed on MV0. Furthermore, the template changes slightly for each point in the search space 310.

[0163] More specifically, according to the present embodiment, the following steps are applied:

[0164] Step 1: Calculate the similarity between the sample pointed to by the input motion vector MV0 and the sample pointed to by the input motion vector MV1. The input motion vector MV0 points to a point 330 in the search space.

[0165] Step 2: Select a second point in the search space 310, the second point being different from the point 330, the second point being denoted by MV0".

[0166] Step 3: Calculate the motion vector MV1" based on MV0", MV0 and MV1 using one or more of Equations 1-8.

[0167] Step 4: Calculate the similarity between the sample pointed to by the input motion vector MV0" and the sample pointed to by the input motion vector MV1". If the similarity is higher than the value calculated in Step 1, then the pair MV0" and MV1" is selected as the corrected motion vector. Otherwise, the pair MV0 and MV1 (i.e. the initial motion vector pair) is selected as the corrected motion vector.

[0168] Steps 2, 3 and 4 can be repeated to evaluate more candidate points in the search space 310. If there are no remaining search points in the search space, the corrected motion vector is output as the final corrected motion vector.

[0169] In Step 4, it is clear that if the similarity measure (such as SAD, SSIM, etc.) can make the similarity at the initial point 330 the highest, and the initial motion vector MV0 points to the initial point 330, the corresponding initial motion vector for the initial motion vector MV0 is the initial motion vector MV1 according to one or more of Equations 1-8, if this is the case, then the corrected motion vector, i.e. the output of the motion vector correction process, is MV0 and MV1.

[0170] Any motion vector pair MV0" and MV1" output by the apparatus 500 has to follow the rules described in one or more of equations 1 to 8. The specific details of the template construction and the similarity measure used in the template matching operation can be changed without affecting the present application and its benefits, which refer to the reduction of search points to be checked by pairing two motion vectors.

[0171] According to an embodiment of the present application, other processing steps for encoding and decoding follow the standard H.265 / HEVC, except for the bi-prediction process performed in the inter prediction unit as described above.

[0172] However, in general, the present application is applicable to any video decoder for decoding a plurality of images from a bitstream. Then such a decoder can comprise an inter prediction unit comprising an apparatus as claimed in any of claims 1 to 11, and a prediction unit for determining a prediction block from a partial region of a first reference image referred to by the first motion vector and a partial region of a second reference image referred to by the second motion vector. The decoder can further comprise a bitstream parser. For example, the bitstream parser can be implemented as part of the entropy decoding unit 203 for obtaining the estimated values of the first and second motion vectors from the bitstream. The video decoder can further comprise a reconstruction unit 211 for reconstructing the current block from the prediction block.

[0173] On the other hand, a video encoder for encoding a plurality of images into a bitstream can comprise an inter prediction unit comprising an apparatus as claimed in any of claims 1 to 12, and a prediction unit for determining a prediction block from a partial region of a first reference image referred to by the first motion vector and a partial region of a second reference image referred to by the second motion vector; a bitstream former implemented as part of the entropy encoding unit 103 for including the estimated values of the first and second motion vectors into the bitstream; a reconstruction unit 111 for reconstructing the current block from the prediction block and storing the reconstructed block in a memory.

[0174] The inter prediction decoding process described above is not limited to using two reference pictures. Alternatively, more than two reference pictures and associated motion vectors can also be considered. In this case, the reference picture selection unit selects more than two reference pictures, one of which is used for motion vector refinement. The reference picture used for motion vector refinement is selected using one of the methods described in examples 1 to 5 above. The remaining motion vectors are adjusted using the estimated values of the corresponding motion vectors and the motion vector refinement. In other words, the present application described above can also work if multiple reference prediction is performed. For example, if there are three reference pictures and three corresponding motion vectors, one of the three motion vectors can be determined by template matching for refinement and the other two motion vectors can be calculated. This reduces the complexity. Alternatively, two of the motion vectors can be determined by template matching for refinement and the other motion vector can be calculated based on one or both of the refined motion vectors. It is clear to those skilled in the art that the present application can be extended to any number of reference pictures and corresponding motion vectors used to construct the prediction value of the current block.

[0175] The effect of the present application is to enable bidirectional prediction to be performed at a decoder with lower processing load and memory requirements. This can be applied to any decoder, which can be included in an encoding device and / or a decoding device, i.e. on the encoder side or the decoder side.

[0176] The motion vector refinement as described above can be implemented as part of the encoding and / or decoding of a video signal (moving image). However, the motion vector refinement can also be used for other purposes of image processing, such as motion detection, motion analysis, etc.

[0177] The motion vector refinement can be implemented as a kind of apparatus. Such an apparatus can be a combination of software and hardware. For example, the motion vector refinement can be performed by a general purpose processor, a digital signal processor (DSP), a field programmable gate array (FPGA), etc. However, the present application is not limited to be implemented on programmable hardware. It can also be implemented on an application-specific integrated circuit (ASIC) or by a combination of the above hardware components.

[0178] The motion vector correction can also be implemented by program instructions stored on a computer readable medium. When the program is executed, it causes a computer to perform the following steps: obtaining an estimated value of a motion vector, determining a first reference image and a second reference image based on the estimated value, performing motion vector correction on the first motion vector, calculating a second motion vector based on the estimated value of the motion vector and the corrected first motion vector. The computer readable medium can be any medium storing the program, such as a DVD, a CD, a USB (flash) drive, a hard disk, a server storage available through a network, etc.

[0179] The encoder and / or decoder can be implemented in various devices, including a television set, a set-top box, a PC, a tablet, a smartphone, etc. It can be a software, an app, implementing the above method steps.

Claims

1. A method for video decoding using inter-frame prediction, characterized in that, The method includes: Obtain the estimated value of the first motion vector; The difference between the first motion vector and the estimated value of the first motion vector is determined by searching within the search space specified based on the estimated value of the first motion vector; An estimate of the second motion vector is obtained; the first motion vector is determined in a first reference image of the video, and the second motion vector is determined in a second reference image of the video; the first motion vector and the second motion vector are used for inter-frame prediction of image patches in the current image of the video; The second motion vector is calculated based on the estimated value of the second motion vector and the difference between the estimated values ​​of the first motion vector and the first motion vector.

2. The method according to claim 1, characterized in that, The second motion vector is calculated by adding the difference between the first motion vector and its estimated value, or a function thereof, to the estimated value of the second motion vector.

3. The method according to claim 2, characterized in that, The function includes scaling and / or clipping via a scaling factor.

4. The method according to claim 3, characterized in that, The value of the scaling factor depends on the ratio between the distances of the first reference image and the second reference image to the current image.

5. The method according to any one of claims 1 to 4, characterized in that, Also includes: Two reference images are acquired, and a first reference image is selected from the two reference images, and a second reference image is selected from the two reference images.

6. The method according to claim 5, characterized in that, The steps of acquiring two reference images, selecting the first reference image from the two reference images, and selecting the second reference image from the two reference images include: The first reference image or the second reference image is selected based on whether the first reference image or the second reference image is referenced in the bitstream by an index associated with a predefined list of reference images in at least two reference image lists. The bitstream also includes encoded image blocks of the video, wherein the reference image list associates the index with the position of the reference image relative to the current image.

7. The method according to claim 6, characterized in that, If the two reference images are referenced in the bitstream by the same index in the same predefined list of reference images, then the image with the highest position in the list of reference images is selected as the first reference image.

8. The method according to claim 1, characterized in that, Also includes: The estimated values ​​of the first motion vector and the second motion vector are determined by selecting the estimated values ​​of the first motion vector and the second motion vector from a set of motion vector candidates based on the similarity between the template and the corresponding partial regions of the image involved in the motion vector candidate.

9. A method for video coding for inter-frame prediction, characterized in that, The method includes: Obtain the estimated value of the first motion vector; The difference between the first motion vector and the estimated value of the first motion vector is determined by searching within the search space specified based on the estimated value of the first motion vector; Obtain an estimate of the second motion vector; the first motion vector is determined in a first reference image of the video, and the second motion vector is determined in a second reference image of the video; The second motion vector is calculated based on the estimated value of the second motion vector and the difference between the estimated values ​​of the first motion vector and the first motion vector. Inter-frame prediction is performed on image patches in the current image based on the second motion vector.

10. The method according to claim 9, characterized in that, The step of performing inter-frame prediction of image patches in the current image based on the second motion vector includes: A prediction block is determined based on a portion of the first reference image involved by the first motion vector and a portion of the second reference image involved by the second motion vector.

11. The method according to claim 9 or 10, characterized in that, The method further includes: The estimated values ​​of the first motion vector and the second motion vector are encoded in the bitstream.

12. A video decoder, characterized in that, The video decoder includes a processor for performing the method according to any one of claims 1 to 8.

13. A video encoder, characterized in that, The video encoder includes a processor for performing the method according to any one of claims 9 to 11.

Images