Memory access windows and padding for motion vector refinement and motion compensation

Abstract

To perform motion vector determination using template matching or bi-directional matching and motion vector based predictor generation.SOLUTION: Template matching or bilateral matching and / or predictor generation use interpolation filtering. Interpolation filtering operation accesses integer sample positions within a window, and further uses padded sample values for the integer sample positions outside the window, which are based on at least one sample within the window, and uses the accessed integer sample position values as well as the padded sample values to perform the template matching or the bilateral matching and / or the predictor generation.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] SUMMARY OF THE PRESENT EMBODIMENTS Embodiments of the present invention relate to motion vector determination and refinement, as well as predictor determination for motion compensation, which may be used during video encoding and decoding. [Background technology]

[0002] Current hybrid video codecs use predictive coding. Pictures of a video sequence are subdivided into blocks of pixels, and then these blocks are coded. Instead of coding the block pixel by pixel, the whole block is predicted using already encoded pixels that are spatially or temporally close to the block. The encoder further processes only the difference between the block and its prediction. Further processing usually includes transforming the block pixels into coefficients in the transform domain. The coefficients are then further compressed by quantization and may be further compacted by entropy coding to form a bitstream. The bitstream further includes any signaling information that allows a decoder to decode the encoded video. For example, the signaling may include settings regarding encoder settings such as the size of the input picture, the frame rate, a quantization step indication, or the prediction to be applied to the blocks of the picture.

[0003] Temporal prediction uses the temporal correlation between pictures, also called frames of the video. Temporal prediction is also called inter-prediction, since it is a prediction that uses dependencies between different video frames (inter). Thus, a block to be encoded, also called a current block, is predicted from one or more previously encoded pictures, also called reference pictures. A reference picture is not necessarily a picture preceding the current picture, in which the current block is placed in a display order of the video sequence. An encoder may encode pictures in a coding order different from the display order. As a prediction of the current block, a co-located block in a reference picture may be determined. A co-located block is a block that is located in a reference picture at the same position as the current block in the current picture. Such a prediction is accurate for stationary picture regions, i.e. picture regions that do not move from one picture to another.

[0004] Motion estimation is usually used when determining the prediction of the current block to obtain a predictor that takes into account the motion, i.e. a motion compensated predictor. The current block is thus predicted by a block in a reference picture that is located at a distance given by the motion vector from the position of the co-located block. The motion vector may be signaled in the bitstream so that the decoder can determine the same prediction of the current block. To further reduce the signaling overhead caused by signaling a motion vector for each of the blocks, the motion vector itself may be estimated. The motion vector estimation may be performed based on the motion vectors of neighboring blocks in the spatial and / or temporal domain.

[0005] The prediction of the current block may be calculated using one reference picture or by weighting predictions obtained from two or more reference pictures. Since the neighboring pictures are very likely to be similar to the current picture, the reference picture may be a neighboring picture, i.e., a picture immediately before the current picture in display order, and / or a picture immediately after the current picture. In general, however, the reference picture may be any other picture before or after the current picture in display order and before the current picture in the bitstream (decoding order). This may be advantageous, for example, in the case of occlusions and / or non-linear motion in the video content. Therefore, the identity of the reference picture may also be signaled in the bitstream.

[0006] A special mode of inter prediction is the so-called bi-prediction, in which two reference pictures are used in generating a prediction of the current block. Specifically, the two predictions determined in the respective two reference pictures are combined with the prediction signal of the current block. Bi-prediction can be a more accurate prediction of the current block than uni-prediction, i.e., prediction using only one reference picture. More accurate prediction leads to a smaller difference (also called "residual") between the pixels of the current block and the prediction, which can be encoded more efficiently, i.e., compressed into a shorter bitstream. In general, to predict the current block, three or more reference pictures may be used to find three or more reference blocks, respectively, i.e., multi-reference inter prediction can be applied. Thus, the term multi-reference prediction includes bi-prediction and prediction using three or more reference pictures.

[0007] To provide more accurate motion estimation, the resolution of the reference picture may be increased by interpolating samples between pixels. Fractional pixel interpolation may be performed by weighted averaging of the nearest pixels. In the case of half pixel resolution, for example, bilinear interpolation is usually used. Other fractional pixels are calculated as the average of the nearest pixels weighted by the inverse of the distance between each nearest pixel and the pixel to be predicted.

[0008] Motion vector estimation is a computationally complex task, in which the similarity between a current block and a corresponding predicted block (predictor) pointed to by a candidate motion vector in a reference picture is calculated. Typically, a search area contains an image of M×M samples, and each sample position of M×M candidate positions is tested. The test involves the calculation of a similarity measure between an N×N reference block C and a block R placed at the tested candidate position of the search area. For simplicity, the sum of absolute differences (SAD) is a measure often used for this purpose, given by:

number

[0009] In the above formula, x and y define candidate positions in the search area, and indices i and j indicate samples in the reference block C and candidate block R. The candidate positions are often called block displacements or offsets, reflecting the representation of block matching as a shift of the reference block in the search area and a calculation of the similarity between the reference block C and the overlapping part of the search area. To reduce the complexity, the number of candidate motion vectors is usually reduced by restricting them to a certain search space. The search space may be defined, for example, by the number and / or location of pixels surrounding a location in the reference picture that corresponds to the location of the current block in the current image. After calculating the SAD for all M×M candidate positions x and y, the best matching block R is the block at the position that results in the lowest SAD and corresponds to the maximum similarity with the reference block C. On the other hand, the candidate motion vectors may be defined by a list of candidate motion vectors formed by the motion vectors of neighboring blocks.

[0010] Motion vectors are usually at least partially determined at the encoder side and signaled to the decoder in the coded bitstream. However, motion vectors may be derived at the decoder. In such cases, the current block is not available at the decoder and cannot be used to calculate the similarity to the block to which the candidate motion vector points in the reference picture. Therefore, a template is used that is constructed outside the pixels of already decoded blocks rather than the current block ("template matching"). For example, already decoded pixels adjacent to the current block may be used. Such motion estimation has the advantage that no signaling is required since the motion vector is derived in the same way at both the encoder and the decoder, thus reducing signaling. On the other hand, the accuracy of such motion estimation may be low.

[0011] In the case of bi-prediction, instead of using a template, a comparison between candidate coding blocks in different reference pictures pointed to by candidate motion vectors may be used ("bi-directional matching"), which is described in more detail below.

[0012] To provide a trade-off between accuracy and signaling overhead, motion vector estimation may be divided into two steps: motion vector derivation and motion vector refinement. For example, motion vector derivation may include the selection of a motion vector from a candidate list. Such a selected motion vector may be further refined, for example, by searching in a search space. The search in the search space is based on the calculation of a cost function for each candidate motion vector, i.e., for each candidate position of the block to which the candidate motion vector points.

[0013] The paper "X. Chen, J. An, J. Zheng, JVET-D0029: Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching" (available at the site http: / / phenix.it-sudparis.eu / jvet / ) presents a motion vector refinement in which a first motion vector at integer pixel resolution is found and further refined by searching at 1 / 2 pixel resolution in a search space around the first motion vector.

[0014] In order to perform motion vector refinement, it is necessary to store in memory at least those samples that are needed for the current block to perform the refinement, i.e. samples that correspond to the search space, and samples that can be accessed when template matching or bidirectional matching is performed in the search space.

[0015] External memory access is a key design parameter in this hardware architecture and / or software implementation because it is slower than using internal memory, whereas internal memory on a chip is limited, e.g., due to chip size implementation. Summary of the Invention [Means for solving the problem]

[0016] The present disclosure is based on the observation that, for example, motion vector refinement when implemented in combination with fractional interpolation may require more on-chip memory size or more external memory accesses, neither of which options may be desirable. The same is true when interpolation is applied to improve the resolution of the predictor by including fractional sample positions.

[0017] In consideration of the above-mentioned problems, the present disclosure provides motion vector prediction that allows for consideration of the number of accesses to external memory and the number of samples that need to be accessible for motion vector refinement of a motion vector for a coding block, and calculates a predictor for the block.

[0018] This is achieved by padding (substituting) values ​​located outside a predefined memory access window for the purposes of motion vector refinement based on one or more samples from within the memory access window, and / or fractional interpolation with the substitution value.

[0019] This is achieved by means of the features of the independent claims.

[0020] According to an aspect of the present invention, an encoding apparatus for determining a predictor for encoding a block of a video image is provided. The encoding apparatus comprises a processing circuit configured to obtain an initial motion vector for the block, obtain a refinement motion vector for the block based on the initial motion vector, and determine a predictor for the block according to the refinement motion vector. The processing circuit is configured to determine the predictor according to the refinement motion vector using interpolation with an interpolation filter, and to use at least one sample value of integer sample positions located inside a window and padded sample values ​​for integer sample positions located outside the window for interpolation filtering, the padded sample value being based on at least one sample value of samples located inside the window.

[0021] According to another aspect of the present invention, a decoding device for determining a predictor for decoding a block of a video image is provided. The decoding device includes a processing circuit configured to obtain an initial motion vector for the block, obtain a refinement motion vector for the prediction block based on the initial motion vector, and determine a predictor for the block according to the refinement motion vector. The processing circuit is configured to determine the predictor according to the refinement motion vector using interpolation with an interpolation filter, and to use at least one sample value of integer sample positions located inside a window and padded sample values ​​for integer sample positions located outside the window for interpolation filtering, the padded sample value being based on at least one sample value of samples located inside the window.

[0022] According to yet another aspect of the present invention, there is provided an encoding method for determining a predictor for encoding a block of a video image, the encoding method including the steps of: obtaining an initial motion vector for the block, obtaining a refinement motion vector for the block based on the initial motion vector, and determining a predictor for the block according to the refinement motion vector, the determining step determining the predictor according to the refinement motion vector using interpolation with an interpolation filter, using at least one sample value of integer sample positions located inside a window and padded sample values ​​for integer sample positions located outside the window for interpolation filtering, the padded sample value being based on at least one sample value of integer samples located inside the window.

[0023] According to yet another aspect of the present invention, there is provided a decoding method for determining a predictor for decoding a block of a video image, the decoding method comprising the steps of obtaining an initial motion vector for the block, obtaining a refinement motion vector for the prediction block based on the initial motion vector, and determining a predictor according to the refinement motion vector using interpolation with an interpolation filter, using at least one sample value of integer sample positions located inside a window and padded sample values ​​for integer sample positions located outside the window for the interpolation filtering, the padded sample value being based on at least one sample value of integer samples located inside the window.

[0024] Such a method and apparatus has the advantage of limiting the number of samples made available for the purpose of predictor determination by interpolation filtering, while also avoiding further accesses to a storage device / (external) memory that stores the entire reference picture.

[0025] In an embodiment of the above aspect, the window is a window for interpolation filtering.

[0026] According to an embodiment, the interpolation may include accessing sample values ​​at integer sample positions within the window and / or padding at least one sample value for an integer sample position located outside the window with a padding value based on at least one sample within the window and / or using the accessed integer position sample values ​​and the padded sample values ​​to perform interpolation filtering.

[0027] For example, the processing circuitry may be configured to determine a refinement of the initial motion vector based on a search space of candidate motion vectors, and to determine the refinement motion vector based on the initial motion vector and the refinement.

[0028] Also, for example, the processing circuitry may be configured to determine a search space based on the initial motion vector. More specifically, the search space may be located at a position defined by the initial motion vector. For example, the search space may be centered on the initial motion vector. More specifically, according to an example, the search space may include nine integer sample motion vector candidates, i.e., the initial motion vector and eight motion vector candidates located around it.

[0029] In an embodiment, the processing circuit is further configured to reconstruct the block based on the predictor. The reconstruction may be performed in the decoding device and in the decoder loop of the encoding device. Aspects of the present disclosure related to the method may include a step of reconstructing each of the blocks based on the predictor.

[0030] For example, the processing circuitry may be configured to obtain the difference between the block and the predictor, also called the “residual.” More specifically, the processing circuitry may be configured to reconstruct the block based on the predictor and the residual, and even more specifically as a sum of the predictor and the residual.

[0031] For example, the padded sample values ​​(substitution values) are determined by mirroring the values ​​of the corresponding used (evaluated) integer sample positions relative to the axes of an interpolation filter. The interpolation filtering may be one-dimensional filtering or separable two-dimensional filtering.

[0032] In one exemplary implementation, the window is formed by all integer sample positions accessible to obtain the predictor at the position specified by the initial motion vector.

[0033] The window is defined by its vertical and / or horizontal size, for example, relative to a block having the size of a block located at the initial motion vector position, the initial motion vector position, or the vertical and horizontal lengths of an interpolation filter used to determine the predictor. The definition of a block "located" at a particular position means, in an embodiment, for example, referring to the top-left sample position of the block. Alternatively, another sample position may be referenced, or the block position may be defined by being centered at a particular position. The processing circuitry may be configured to define or determine the position and / or size of the window according to the initial motion vector position, the size of the block, and / or the vertical and horizontal lengths of an interpolation filter used to determine the predictor.

[0034] As another example, the padded sample values ​​for integer sample positions outside the window are determined to be equal to the sample values ​​of at least one used integer sample position located inside the window that is closest to the integer sample position located outside the window. In a particular example, in the case of a one-dimensional interpolation filter, the padded sample values ​​for the integer sample positions located outside the window are used to determine the predictor using interpolation with the interpolation filter, and the padded sample values ​​for all integer sample positions outside the window are determined to be equal to the sample values ​​of at least one used integer sample position located inside the window that is closest to the integer sample position located outside the window.

[0035] According to an embodiment, the number of padded sample values ​​depends on the difference between the initial motion vector and the refinement motion vector.

[0036] According to an embodiment, the processing circuit is configured to determine the refinement of the initial motion vector by template matching or template matching with bidirectional matching, which accesses sample values ​​at integer sample positions in another window, the other window including a search space and another integer sample position accessible for template matching or bidirectional matching, and further pads at least one sample value at an integer sample position located outside the other window with a padded sample (substitution) value based on at least one sample in the other window, and performs template matching or bidirectional matching using the accessed integer sample position value and the padded sample value.

[0037] In one embodiment, the search space includes at least one fractional sample position with a value obtained by interpolation filtering with a filter of a predefined tap size, the interpolation filtering using the value of the accessed integer sample position and at least one substitution (padded sample) value to obtain a value for the at least one fractional sample position.

[0038] More specifically, the padded sample values ​​used in template matching or bidirectional matching are determined by mirroring the values ​​of the corresponding estimated integer sample positions across the filter axis.

[0039] Alternatively, the padded sample value used in the template match or bidirectional match is determined to be equal to the value closest to the accessed integer sample position.

[0040] According to an embodiment, the further window is defined by its vertical and / or horizontal size relative to at least one of the search space, a block having the size of the block located at the initial motion vector position, and the initial motion vector position.

[0041] Preferably, the another window is the same as the window.

[0042] According to an embodiment, the refinement is determined by bidirectional matching, and the processing circuitry is configured to obtain at least two initial motion vectors pointing to positions in different reference pictures and forming a pair of initial motion vectors.

[0043] According to another embodiment, the refinement is determined by template matching, and the processing circuitry is further configured to obtain a template for the block and determine a refinement of the initial motion vector by template matching with the template in the search space.

[0044] In one embodiment, the search space includes at least one fractional sample position with a value obtained by interpolation filtering with a filter of a predefined tap size, the interpolation filtering using values ​​of the used integer sample positions and at least one padded sample value to obtain a value for the at least one fractional sample position.

[0045] According to an aspect of the present invention, there is provided a non-transitory computer readable storage medium storing instructions which, when executed by a processor / processing circuit, perform steps according to any aspect or embodiment described above, or a combination thereof.

[0046] According to another particular aspect of the present invention, an encoding device is provided for encoding a video image divided into predictive blocks into a bitstream or for encoding a predictive block of a video image into a bitstream, respectively. The encoding device comprises a processing circuit for determining a motion vector for the predictive block. The processing circuit is configured to obtain at least one initial motion vector and determine a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refinement motion vector. The search space is located at a position given by the initial motion vector. The encoding device further comprises an encoding circuit (which may be included in the processing circuit) for encoding a difference between the predictive block and a predictor given by the predictive block at a position based on the determined motion vector, and generating a bitstream including the encoded difference and the initial motion vector. The encoding device further comprises a motion prediction unit (which may be included in the processing circuit) for determining a predictor according to the motion vector using interpolation filtering with an interpolation filter. The interpolation filtering comprises accessing sample values ​​at integer sample positions within a predefined window, replacing at least one sample value at integer sample positions located outside the window by a replacement value based on at least one sample in the window, and performing the interpolation filtering using the accessed integer position sample value and the replacement value.

[0047] According to yet another particular aspect of the present invention, a decoding device is provided for decoding a video image divided into predictive blocks from a bitstream. The decoding device comprises an analysis unit (which may be included in a processing circuit of the decoding device) for analyzing from the bitstream an initial motion vector and an encoded difference between the predictive block and a predictor provided by the predictive block at a position specified by a refinement motion vector. The decoding device comprises a processing circuit (e.g., a processing circuit of the decoding device) for determining a motion vector for the predictive block. The processing circuit is configured to obtain at least one initial motion vector and determine a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refinement motion vector. The search space is located at the position given by the initial motion vector. Furthermore, the decoding device comprises a decoding circuit (which may be included in the processing circuit of the decoding device) for reconstructing the predictive block as a sum of the analyzed difference and the predictor provided by the predictive block at a position specified by the refinement motion vector determined by the processing circuit to determine the motion vector. The decoding circuit is further configured to determine the predictor according to the motion vector using an interpolation filtering with an interpolation filter. The interpolation filtering accesses sample values ​​at integer sample positions within a predefined window, replaces at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within the window, and performs the interpolation filtering using the accessed integer position sample values ​​and the replacement values.

[0048] According to yet another aspect of the present invention, an encoding method is provided for encoding a video image divided into predictive blocks into a bitstream. The encoding method includes determining a motion vector for the predictive block. The determining the motion vector includes obtaining at least one initial motion vector and determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refinement motion vector. The search space is located at a position given by the initial motion vector. The encoding method further includes encoding a difference between the predictive block and a predictor given by the predictive block at a position based on the determined motion vector, and generating a bitstream including the encoded difference and the initial motion vector. The method further includes determining a predictor according to the motion vector using interpolation filtering with an interpolation filter. The interpolation filtering includes accessing sample values ​​at integer sample positions within a predefined window, replacing at least one sample value at integer sample positions located outside the window by a replacement value based on at least one sample in the window, and performing the interpolation filtering using the accessed integer position sample values ​​and the replacement value.

[0049] According to yet another aspect of the present invention, a decoding method is provided for decoding a video image divided into predictive blocks from a bitstream. The decoding method includes analyzing from the bitstream an initial motion vector and an encoded difference between the predictive block and a predictor provided by the predictive block at a position specified by a refinement motion vector. The method further includes determining a motion vector for the predictive block. The determining motion vector includes obtaining at least one initial motion vector and determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refinement motion vector. The search space is located at the position given by the initial motion vector. The decoding method further includes reconstructing the predictive block as a sum of the analyzed difference and the predictor provided by the predictive block at a position specified by the refinement motion vector determined by the determining motion vector. Furthermore, the method includes determining a predictor according to the motion vector using an interpolation filtering with an interpolation filter. The interpolation filtering accesses sample values ​​at integer sample positions within a predefined window, replaces at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within the window, and performs the interpolation filtering using the accessed integer position sample values ​​and the replacement values.

[0050] Further advantages and embodiments of the invention are set forth in the dependent claims.

[0051] In the following, exemplary embodiments are explained in more detail with reference to the accompanying drawings and figures. [Brief description of the drawings]

[0052] [Figure 1] FIG. 2 is a block diagram illustrating an example structure of an encoder in which motion vector derivation and refinement may be used. [Diagram 2]2 is a block diagram illustrating an example structure of a decoder in which motion vector derivation and refinement may be used. [Diagram 3] FIG. 1 is a schematic diagram illustrating an exemplary template matching suitable for bi-prediction. [Figure 4] FIG. 1 is a schematic diagram illustrating an exemplary template matching suitable for single-prediction and bi-prediction. [Diagram 5] FIG. 2 is a block diagram illustrating a stage of motion vector derivation that operates without providing an initial motion vector that is refined in the bitstream. [Figure 6] FIG. 1 is a schematic diagram illustrating the concept of bidirectional matching. [Figure 7] FIG. 2 is a block diagram illustrating exemplary hardware for implementing embodiments of the present invention. [Figure 8] FIG. 2 is a schematic diagram illustrating an exemplary window containing a coding block and samples that should be accessible; [Figure 9] FIG. 1 is a schematic diagram illustrating an iterative search space. [Figure 10] FIG. 1 is a schematic diagram illustrating horizontal expansion of a memory access window by interpolation filtering. [Figure 11] FIG. 1 illustrates a conventional approach to perform motion vector refinement based on a search space and obtain predictions by interpolation. [Figure 12] 12 is a flowchart showing motion vector derivation and prediction acquisition according to the conventional example of FIG. 11; [Figure 13] FIG. 2 illustrates a technique for performing motion vector refinement based on a search space and obtaining a prediction by interpolation according to an exemplary embodiment of the present invention. [Figure 14] FIG. 13 is a schematic diagram showing padding of two samples to allow fractional position calculation. [Figure 15] FIG. 13 is a schematic diagram showing padding of four samples to enable template matching at the search space boundary. [Figure 16] FIG. 13 is a schematic diagram illustrating an example fractional position where no padding is required for interpolation. [Figure 17]14 is a flow diagram illustrating motion vector refinement and prediction acquisition according to the example embodiment of FIG. 13. [Figure 18] FIG. 1 is a schematic diagram illustrating a definition of a memory access window for a prediction unit. [Figure 19] FIG. 2 illustrates a method according to an embodiment. [Figure 20] FIG. 10 illustrates a technique for performing motion vector refinement based on a search space and obtaining a prediction by interpolation according to another exemplary embodiment of the present invention. [Figure 21] 21 is a flow diagram illustrating motion vector refinement and prediction acquisition according to the example embodiment of FIG. 20. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0053] This disclosure relates to adjusting the number of samples that are made accessible to perform motion vector refinement and interpolation to obtain fractional positions within a reference picture and a predictor.

[0054] As mentioned above, external memory access is one of the most important design considerations in today's hardware and software architectures. Motion vector estimation may be used in conjunction with interpolation filtering to obtain fractional positions in the search space, for example in the case of motion vector refinement, especially when it involves template matching or bidirectional matching. To use interpolation filtering, it may be necessary to increase the number of samples to be accessed from memory. However, this may lead to an increase in expensive on-chip memory or an increase in the number of external memory accesses, or on the other hand, slow down the implementation. Especially on the decoder side, such problems are undesirable as they may lead to more expensive or slower operation.

[0055] To avoid such situations, the present disclosure provides for restricting external memory access to a predefined window.

[0056] The window may be of a predefined size. For example, in motion vector refinement, the window may include at least all search space positions. The accessed samples are the samples that are actually used for a particular block and its search space to perform template matching or bidirectional matching, e.g., all samples used to calculate a cost function using a template or a bidirectional cost function. The accessible samples further include samples that are not necessarily accessed during a particular template matching or bidirectional matching for a particular block, e.g., because the search space has been iteratively determined to include different samples, and template matching or bidirectional matching for that sample does not require all of the accessible pixels. The accessible samples further include samples that may be obtained from any iteratively determined search space (i.e., assuming any block content), and possibly further include samples that are accessed to perform template matching in such a possible search space.

[0057] When using interpolation filtering to generate the final prediction, the extent to which the memory access window needs to be expanded in conventional approaches depends on the maximum difference between the determined refinement motion vector and the initial motion vector.

[0058] It should be noted that the window may be defined to be smaller than all accessible samples. Defining the window in this way allows to keep the number of samples stored in on-chip memory and fetched from external memory smaller. The present disclosure therefore limits the number of integer samples stored / cached / buffered for operations related to template matching or bidirectional matching for a particular block. This does not mean that no other samples are typically present / available in another memory or storage device. As mentioned above, typically the entire reference picture may be stored in external memory. However, for template matching or bidirectional matching, only a portion of it, i.e. the samples within the window, may be loaded and used for template matching.

[0059] Providing such a limited window allows implementation using it and only requires loading the window samples. To ensure that the encoder and decoder operate similarly, the window definition may be predefined normally or may be signaled in the bitstream. In particular, if refinement motion vectors are used to form the reference picture, both the encoder and the decoder must use the same approach.

[0060] FIG. 1 shows an encoder 100 with an input for receiving input image samples of a frame or picture of a video stream and an output for generating an encoded video bitstream. The term "frame" in this disclosure is used as a synonym for picture. However, it should be noted that the disclosure is also applicable to fields where interlacing may be applied. In general, a picture includes m×n pixels, which correspond to image samples, each of which may include one or more color components. For simplicity, the following description refers to a pixel meaning a sample of luminance. However, it should be noted that the motion vector search of the present invention may be applied to any color component, including chrominance, or to components of the search space such as RGB. On the other hand, it may be beneficial to perform motion vector estimation only for one component and apply the determined motion vector to more (or all) components.

[0061] The input blocks to be coded do not necessarily have the same size: a picture may contain blocks of different sizes, and the block rasters of different pictures may also be different.

[0062] In an illustrative implementation, the encoder 100 is configured to apply prediction, transformation, quantization, and entropy coding to the video stream, which are performed by a transform unit 106, a quantization unit 108, and an entropy encoding unit 170, respectively, to generate an encoded video bitstream as output.

[0063] A video stream may include multiple frames, and each frame is divided into blocks of a certain size that are either intra-coded or inter-coded. For example, blocks of a first frame of the video stream are intra-coded by the intra prediction unit 154. Intra frames are coded using only information within the same frame, so they can be decoded independently and can provide entry points in the bitstream for random access. Blocks of other frames of the video stream may be inter-coded by the inter prediction unit 144, where information from previously coded frames (reference frames) is used to reduce temporal redundancy, such that each block of the inter-coded frame is predicted from a block in the reference frame. The mode selection unit 160 is configured to select whether the blocks of a frame are processed by the intra prediction unit 154 or the inter prediction unit 144. This mode selection unit 160 also controls parameters of the intra prediction or inter prediction. To allow refreshing of image information, intra-coded blocks may be provided within inter-coded frames. Additionally, intra frames, which contain only intra-coded blocks, may be inserted periodically into a video sequence to provide entry points for decoding, i.e., points at which the decoder can begin decoding without having information from previously coded frames.

[0064] The intra estimation unit 152 and the intra prediction unit 154 are units that perform intra prediction. In particular, the intra estimation unit 152 may further derive a prediction mode based on knowledge of the original image, and the intra prediction unit 154 provides a corresponding predictor for differential coding, i.e., a predicted sample using the selected prediction mode. To perform spatial or temporal prediction, the coded block may be further processed by the inverse quantization unit 110 and the inverse transform unit 112. After the reconstruction of the block, the loop filtering unit 120 is applied to further improve the quality of the decoded image. Then, the filtered block forms a reference frame, which is then stored in the decoded picture buffer 130. Such a decoder loop (decoder) on the encoder side provides the advantage of generating the same reference frame as the reference picture reconstructed on the decoder side. Therefore, the encoder side and the decoder side operate in a corresponding manner. The term "reconstruction" here refers to obtaining a reconstructed block by adding a prediction block to a decoded residual block.

[0065] The inter estimation unit 142 receives as input a block of a current frame or picture to be inter coded and one or several reference frames from the decoded picture buffer 130. Motion estimation is performed by the inter estimation unit 142, and motion compensation is applied by the inter prediction unit 144. Motion estimation is used to obtain a motion vector and a reference frame based on a certain cost function, for example, also using the original image to be coded. For example, the motion estimation unit 142 may provide an initial motion vector estimate. The initial motion vector may then be signaled in the bitstream directly in the form of a vector, or as an index that refers to a motion vector candidate in a candidate list built based on a predefined rule, in the same way as in the encoder and decoder. Then, the motion compensation derives a predictor of the current block as a transformation of a block co-located with the current block in the reference frame to a reference block in the reference frame, i.e., by a motion vector. The inter prediction unit 144 outputs a prediction block for the current block, which minimizes the cost function. For example, the cost function may be the difference between the current block to be coded and its predicted block, i.e., the cost function minimizes the residual block. The minimization of the residual block is based, for example, on calculating the sum of absolute differences (SAD) of all pixels (samples) of the current block and a candidate block in the candidate reference picture. However, in general, other similarity metrics such as the mean squared error (MSE) or the structural similarity metric (SSIM) may be used.

[0066] However, the cost function may also be the number of bits required to code such an inter block and / or the distortion resulting from such coding. Thus, a rate-distortion optimization procedure may be used to determine the encoding parameters, such as the selection of motion vectors and / or in general, whether to use inter- or intra-prediction for a block and its settings.

[0067] The intra estimation unit 152 and the intra prediction unit 154 receive as input a block of the current frame or picture to be intra coded and one or several reference samples from already reconstructed areas of the current frame. Intra prediction then describes the pixels of the current block of the current frame in terms of a function of the reference samples of the current frame. The intra prediction unit 154 outputs a prediction block of the current block, which preferably minimizes the difference between the current block to be coded and its prediction block, i.e. it minimizes the residual block. The minimization of the residual block may for example be based on a rate-distortion optimization procedure. In particular, the prediction block is obtained as a directional interpolation of the reference samples. The direction may be determined by rate-distortion optimization and / or by calculating a similarity measure as mentioned above in connection with inter prediction.

[0068] The inter estimation unit 142 receives as input a block, or a more general form of image sample of the current frame or picture to be inter coded, and two or more already decoded pictures 231. Then, inter prediction describes the current image sample of the current frame in terms of a motion vector to refer to the image sample of the reference picture. The inter prediction unit 142 outputs one or more motion vectors of the current image sample, and said reference image sample pointed to by the motion vector preferably minimizes the difference between the current image sample to be coded and its reference image sample, i.e. it minimizes the residual image sample. Then, for differential coding, a predictor of the current block is provided by the inter prediction unit 144.

[0069] The difference between the current block and its prediction, i.e., the residual block 105, is then transformed by the transform unit 106. The transform coefficients 107 are quantized by the quantization unit 108 and entropy coded by the entropy coding unit 170. The encoded picture data 171 thus produced, i.e., the encoded video bitstream, includes intra-coded and inter-coded blocks and corresponding signaling (such as a mode indication, an indication of a motion vector, and / or an intra-prediction direction). The transform unit 106 may apply a linear transformation, such as a Fourier or discrete cosine transform (DFT / FFT or DCT). Such a transformation into the spatial frequency domain offers the advantage that the resulting coefficients 107 typically have higher values ​​at lower frequencies. Thus, after an effective coefficient scan (such as zigzag) and quantization, the resulting sequence of values ​​typically has some large values ​​at the beginning and ends with a run of zeros. This allows for even more efficient coding. The quantization unit 108 performs a virtual lossy compression by lowering the coefficient values ​​in resolution. An entropy encoding unit 170 then assigns binary codewords to the coefficient values ​​to generate a bitstream. The entropy encoding unit 170 also encodes signaling information (not shown in FIG. 1).

[0070] FIG. 2 shows a video decoder 200. The video decoder 200 comprises, among other things, a decoded picture buffer 230, an inter prediction unit 244, and an intra prediction unit 254, which is a block prediction unit. The decoded picture buffer 230 is configured to store at least one (for single prediction) or at least two (for bi-prediction) reference frames reconstructed from an encoded video bitstream, said reference frames being different from a current frame (currently being decoded frame) of the encoded video bitstream. The intra prediction unit 254 is configured to generate a prediction block, which is an estimate of a block to be decoded. The intra prediction unit 254 is configured to generate this prediction based on reference samples obtained from the decoded picture buffer 230.

[0071] The decoder 200 is configured to decode the encoded video bitstream generated by the video encoder 100, and preferably both the decoder 200 and the encoder 100 generate identical predictions for each block being encoded / decoded. The features of the decoded picture buffer 230, the reconstructed blocks 215, the buffer 216, and the intra prediction unit 254 are similar to the features of the decoded picture buffer 130, the reconstructed blocks 115, the buffer 116, and the intra prediction unit 154 of FIG.

[0072] The video decoder 200 includes further units also present in the video encoder 100, such as an inverse quantization unit 210, an inverse transform unit 212, and a loop filtering unit 220, which correspond to the inverse quantization unit 110, the inverse transform unit 112, and the loop filtering unit 120 of the video encoder 100, respectively.

[0073] The entropy decoding unit 204 is configured to decode the received encoded video bitstream and correspondingly obtain quantized residual transform coefficients 209 and signaling information. The quantized residual transform coefficients 209 are provided to an inverse quantization unit 210 and an inverse transform unit 212 to generate a residual block. The residual block is added to a prediction block 265, which is provided to a loop filtering unit 220 to obtain a decoded video. The frames of the decoded video may be stored in a decoded picture buffer 230 and serve as decoded pictures 231 for inter prediction.

[0074] In general, the intra prediction units 154 and 254 in Figures 1 and 2 may use reference samples from already encoded regions to generate a prediction signal for a block that needs to be encoded or decoded.

[0075] The entropy decoding unit 204 receives as its input the encoded bitstream 171. In general, the bitstream is first parsed, i.e. signaling parameters and residuals are extracted from it. Typically, the syntax and semantics of the bitstream are defined in a standard so that the encoder and the decoder can work in an interoperable manner. As explained in the background section above, the encoded bitstream does not only contain the prediction residuals. In case of motion compensated prediction, a motion vector indication is also coded in the bitstream and parsed from there in the decoder. The motion vector indication may be given by the motion vector coordinates, depending on the reference picture to which the motion vector is provided. Up to this point, the coding of the complete motion vector has been considered. However, it is also possible to code in the bitstream only the difference between the current motion vector and the previous motion vector. This approach allows to exploit the redundancy between the motion vectors of neighboring blocks.

[0076] To efficiently code reference pictures, the H.265 codec (ITU-T, H265, Series H: Audiovisual and multimedia systems: High Efficient Video Coding) provides a list of reference pictures that assigns each reference frame to a list index. The reference frames are then signaled in the bitstream by including therein the corresponding assigned list index. Such lists may be standardly defined or signaled at the beginning of a video or a set of several frames. Note that in H.265 there are two lists of reference pictures defined, called L0 and L1. The reference pictures are then signaled in the bitstream by indicating the list (L0 or L1) and the index in that list associated with the desired reference picture. Providing more than one list may be advantageous for better compression. For example, L0 may be used for both unidirectionally inter-predicted slices and bidirectionally inter-predicted slices, and L1 may be used only for bidirectionally inter-predicted slices. However, the present disclosure in general is not limited to any content of the L0 and L1 lists.

[0077] The lists L0 and L1 may be standardly defined and fixed. However, by signaling them at the beginning of the video sequence, a higher flexibility of encoding / decoding may be realized. Thus, the encoder may configure the lists L0 and L1 with specific reference pictures ordered according to their indexes. The lists of L0 and L1 may have the same fixed size. In general, there may be more than two lists. The motion vectors may be directly signaled by their coordinates in the reference pictures. Alternatively, as also specified in H.265, a list of candidate motion vectors may be constructed and the index associated with a specific motion vector in the list may be transmitted.

[0078] The motion vector of the current block is usually correlated with the motion vectors of neighboring blocks in the current picture or in previously coded pictures. This is because neighboring blocks are likely to correspond to the same moving object with similar motion, and the object motion is unlikely to change rapidly over time. As a result, by using the motion vectors in neighboring blocks as predictors, the magnitude of the signaled motion vector difference is reduced. A motion vector predictor (MVP) is usually derived from already encoded / decoded motion vectors from spatially neighboring blocks or from temporally neighboring or co-located blocks in a reference picture. In H.264 / AVC, this is done by taking the component-wise median of three spatially neighboring motion vectors. Using this approach, no predictor signaling is required. The temporal MVP from co-located blocks in a reference picture is only considered in the so-called temporal direct mode of H.264 / AVC. H.264 / AVC direct mode is also used to derive other motion data than motion vectors. Therefore, they are more related to the concept of block merging in HEVC. In HEVC, the method of implicitly deriving MVPs was replaced by a technique known as motion vector competition, which explicitly signals which MVP from a list of MVPs is used for motion vector derivation. The variable coding quadtree block structure in HEVC can result in a block having several neighboring blocks that contain motion vectors as potential MVP candidates. Taking the left neighbor as an example, in the worst case, a 64x64 luma prediction block may have 16 4x4 luma prediction blocks to its left if a 64x64 luma coding tree block is not further divided and the left block is divided to the maximum depth.

[0079] To modify the motion vector conflicts to account for such flexible block structures, evolved motion vector prediction (AMVP) was introduced. During the development of HEVC, the initial AMVP design was significantly simplified to provide a good tradeoff between coding efficiency and an easy-to-implement design. The initial design of AMVP included five MVPs from three different classes of predictors, namely three motion vectors from spatially neighboring ones, the median of the three spatial predictors, and a scaled motion vector from a co-located temporally neighboring block. Additionally, the list of predictors was modified by reordering to place the most likely motion predictor in the first position and removing redundant candidates to ensure minimal signaling overhead. The final design of the AMVP candidate list structure included the following two MVP candidates, namely a) up to two spatial candidate MVPs derived from five spatially neighboring blocks, b) one temporal candidate MVP derived from two temporal, co-located blocks in case both spatial candidate MVPs are unavailable or they are identical, and c) a zero motion vector in case the spatial candidate, the temporal candidate, or both candidates are unavailable. Further details regarding motion vector determination can be found in V. Sze et. al. (Ed.), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014, especially chapter 5, which is incorporated herein by reference.

[0080] To further improve the motion vector estimation without further increasing the signaling overhead, it would be beneficial to further refine the motion vectors derived at the encoder side and provided in the bitstream. The motion vector refinement may be performed at the decoder without assistance from the encoder. The encoder in its decoder loop may use the same refinement to obtain the corresponding motion vector. The motion vector refinement is performed in a search space, which includes integer and fractional pixel positions of the reference picture. For example, the fractional pixel positions may be 1 / 2 or 1 / 4 pixel positions, or another fractional position. The fractional pixel positions may be obtained from the integer (full pixel) positions by interpolation, such as bilinear interpolation. However, the present disclosure is equally applicable to a search space that does not include fractional pixel positions.

[0081] In bi-prediction of the current block, two prediction blocks obtained using the first motion vector in list L0 and the second motion vector in list L1, respectively, are combined into a single prediction signal, which allows for better adaptation to the original signal than uni-prediction, potentially resulting in less residual information and more efficient compression.

[0082] At the decoder, a template is used for motion vector refinement since the current block is not available during decoding; the template is an estimate of the current block and is constructed based on the image parts that have already been processed (i.e., coded at the encoder side and decoded at the decoder side).

[0083] First, an estimate of a first motion vector MV0 and an estimate of a second motion vector MV1 are received as input at the decoder 200. At the encoder side 100, the motion vector estimates MV0 and MV1 can be obtained by block matching and / or by searching in a list of candidates (such as a merge list) formed by the motion vectors of blocks adjacent to the current block (in the same picture or in an adjacent picture). MV0 and MV1 are then advantageously signaled to the decoder side in the bitstream. However, it should be noted that in general, the first decision stage at the encoder may also be performed by template matching, which has the advantage of reducing the signaling overhead.

[0084] At the decoder side 200, the motion vectors MV0 and MV1 are advantageously obtained based on information in the bitstream. MV0 and MV1 are either directly signaled or differentially signaled and / or an index in a list of motion vectors (merge list) is signaled. However, the present disclosure is not limited to signaling the motion vectors in the bitstream. Rather, the motion vectors may be determined by template matching already in the first stage, corresponding to the operation of the encoder. The template matching of the first stage (motion vector derivation) may be performed based on a search space different from the search space of the second refinement stage. In particular, the refinement may be performed in a search space with a higher resolution (i.e., a shorter distance between search positions).

[0085] An indication of the two reference pictures RefPic0 and RefPic1 pointed to by MV0 and MV1, respectively, is provided to the decoder as well. The reference pictures are stored in the decoded picture buffers at both the encoder side and the decoder side as a result of previous processing, i.e., encoding and decoding, respectively. One of these reference pictures is selected for motion vector refinement by searching. The reference picture selection units of the encoder and the decoder are respectively configured to select the first reference picture pointed to by MV0 and the second reference picture pointed to by MV1. Following the selection, the reference picture selection units determine whether the first reference picture or the second reference picture is used for performing the motion vector refinement. To perform the motion vector refinement, a search area in the first reference picture is defined around the candidate position pointed to by the motion vector MV0. The candidate search space positions in the search area are analyzed to find the block most similar to the template block by performing template matching in the search space and determining a similarity metric, such as the sum of absolute difference (SAD). The location in the search space represents the location where the top left corner of the template is matched. As already mentioned above, the top left corner is just a convention and any point in the search space, such as the center point, may generally be used to represent the matching location.

[0086] According to the above-mentioned document JVET-D0029, the decoder-side motion vector refinement (DMVR) has as input initial motion vectors MV0 and MV1, which point to two reference pictures RefPict0 and RefPict1, respectively. These initial motion vectors are used to determine the search spaces of RefPict0 and RefPict1, respectively. Furthermore, using the motion vectors MV0 and MV1, a template is constructed based on each block A and B (of samples) pointed to by MV0 and MV1, as follows: template = function((block A, block B))

[0087] The function may be a sample clipping operation combined with a weighted sum on samples. The template is then used to perform template matching in a search space determined based on MV0 and MV1 in each reference picture 0 and 1. The cost function for determining the best template match in each search space is SAD(Template, Block candA'), where block candA' is the candidate coding block pointed to by the candidate MV in the search space spanning the position given by MV0. Figure 3 shows the determination of the best matching block A' and the resulting refinement motion vector MV0'. Correspondingly, the same template is used to find the best matching block B' and the corresponding motion vector MV1' pointing to block B', as shown in Figure 3. In other words, after the template is constructed based on blocks A and B pointed to by the initial motion vectors MV0 and MV1, the refinement motion vectors MV0' and MV1' are found by searching RefPic0 and RefPic1 in the template.

[0088] The motion vector derivation technique may also be called frame rate up-conversion (FRUC). In order to ensure that the encoder and decoder can use the same initial point for motion vector refinement, the initial motion vectors MV0 and MV1 are generally indicated in the bitstream. Alternatively, the initial motion vector may be obtained by providing a list of initial candidates including one or more initial candidates. A refinement motion vector is determined for each of them, and finally, the refinement motion vector that minimizes the cost function is selected.

[0089] It is further noted that the invention is not limited to the template matching described above with reference to Fig. 3. Fig. 4 illustrates an alternative template matching that is also applicable to single prediction. More details can be found in document JVET-A1001, in particular "Section 2. 4. 6. Pattern matched motion vector derivation" of document JVET-A1001 by Jianle Chen et al., entitled "Algorithm Description of Joint Exploration Test Model 1" and accessible at the site http: / / phenix.it-sudparis.eu / jvet / . The template in this template matching approach is determined as the samples adjacent to the current block in the current frame. As shown in Fig. 1, already reconstructed samples adjacent to the top and left boundaries of the current block may be taken, called "L-shaped template".

[0090] FIG. 5 shows another type of motion vector derivation that can also be used. The input to the motion vector derivation process is a flag indicating whether motion vector derivation is applied. Implicitly, another input to the derivation process is the motion vector of a previously coded / reconstructed block that is adjacent (temporally or spatially). The motion vectors of several adjacent blocks are used as candidates for the initial search step of the motion vector derivation. The output of the process is MV0' (possibly MV1' if bi-prediction is used) and the corresponding reference picture index refPict0 and possibly refPict1, respectively. Then, the motion vector refinement stage includes template matching as described above. After finding one (uni-prediction) or two or more (bi-prediction / multi-frame prediction) motion vectors that are refined, a predictor of the current block (for bi / multi prediction, by weighted sample prediction or otherwise by referring to the samples pointed to by the refined MV) is constructed.

[0091] The present invention is not limited to the above-mentioned two matching methods (examples of template matching). For example, a third matching method called bidirectional matching (also described in document JVET-A1001) may be used for motion vector refinement, and the present invention is equally applicable. Bidirectional matching is described with reference to FIG. 6.

[0092] According to bidirectional matching, the best matching between two blocks along the motion trajectory of the current block is searched in two different reference pictures. As shown in Figure 6, bidirectional matching is used to derive the motion information of the current block by finding the closest matching of two blocks along the motion trajectory of the current block in two different reference pictures. In bidirectional matching, a cost function such as SAD(Block cand0', Block cand1') may be used where MV0 points to Block cand0' and MV1 points to Block cand1'.

[0093] In order to reduce the computational cost, a cost function is calculated and compared for the selection of suitable candidates (candidate pairs) of motion vectors, which may be meaningful to consider to satisfy the following conditions: Assuming a continuous motion trajectory, the motion vectors MV0 and MV1 pointing to two reference blocks shall be proportional to the temporal distance between the current picture and the two reference pictures, i.e., TD0 and TD1. As a special case, when the current picture is between two reference pictures in time and the temporal distance from the current picture to the two reference pictures is the same, the motion vector difference between the motion vectors of the pairs referring to one reference picture and each motion vector of the pairs referring to the other reference picture shall mirror each other.

[0094] 7 shows a processing circuit 600. The processing circuit may include any hardware, the configuration of which may be implemented by any type of programming, or hardware design combining both. For example, the processing circuit may be formed by a single processor, such as a general-purpose processor including corresponding software implementing the steps described above. On the other hand, the processing circuit may be implemented by specialized hardware, such as a DSP (digital signal processor), ASIC (application specific integrated circuit), or FPGA (field programmable gate array).

[0095] The processing circuit may include one or more of the hardware components mentioned above interconnected to perform the motion vector derivation described above. The processing circuit 600 includes computation logic to perform two functions: obtaining an initial motion vector (or multiple initial motion vectors if bi-prediction / multi-prediction is used) and a template if template matching is used 610, and motion vector refinement 620. These two functions may be implemented in the same hardware or may be performed in separate units of hardware, such as the initial motion vector and template determination unit 610 and the motion vector refinement unit 620. The processing circuit 600 may be communicatively connected to an external memory 650 in which reconstructed reference picture samples are stored. In addition, the processing circuit 600 may further include an internal memory 640 for buffering samples in a window transferred from the external memory and used for motion vector determination of the currently processed block. The processing circuit may be embodied in one chip as an integrated circuit.

[0096] It should be noted that the processing circuitry may implement other functions of the encoder and / or decoder as described with reference to Figures 1 and 2. The internal memory may be an on-chip memory such as a cache or line memory. The chip memory is preferably implemented in the encoder / decoder chip to speed up the calculations. Due to the limited size of the chip, the on-chip memory is usually small. On the other hand, the external memory can be very large, but accessing the external memory consumes more energy and is much slower. Usually, all of the required information is brought from the external memory to the on-chip memory before the calculations are performed. The worst-case external memory access (or the bandwidth that needs to be provided when designing the memory bus) represents the maximum amount of memory transfer that can occur between the external memory and the chip during the decoding of a frame or coding unit. Memory (especially the external memory) can usually only be accessed in predefined block units. In other words, single pixels usually cannot be accessed, but must be accessed in the smallest units (e.g., 8x8). The size of the on-chip memory is also an important design consideration, since larger on-chip memories are more expensive.

[0097] In other words, the above-mentioned device may be an integrated circuit further comprising an internal memory embedded within the integrated circuit and a memory access unit (interface) for fetching integer samples placed within the window from an external memory into the internal memory.

[0098] The term "prediction block" used above refers to the current block to be predicted. It is a block in an image that can be obtained by subdividing the image into equal or different sizes (e.g., by hierarchical partitioning of coding tree units into smaller unit blocks of CTUs). The block may be square, or more generally rectangular, since it is a typical shape also used in current encoders / decoders. However, the present disclosure is not limited by any size / shape of the block.

[0099] The device comprising the processing circuitry may be an encoder or decoder, or may be a device comprising such an encoder or decoder, for example a recording device and / or a playback device.

[0100] Fractional sample positions are positions between actual picture sample positions obtained by reconstructing the coded reference picture as shown in Figure 1. Therefore, fractional positions must be obtained by interpolation based on the nearest integer position. Details of the interpolation filtering can be found in Section 5. 3 Fractional Sample Interpolation in V. Sze et. al., High Efficiency Video Coding (HEVC), Springer, 2014.

[0101] Interpolation filtering is usually applied to another filter to generate another fractional pel (sample) position. As an example, the following 1D separable filter is applied to generate 1 / 4-pel and 1 / 2-pel positions in the H.265 video compression standard:

[0102] [Table 1]

[0103] As seen in the table above, the interpolation filtering requires a number of samples around the fractional pel position, corresponding to the filter taps (the number of coefficients in the table). To generate a 1 / 2 pel position, using the example filter described above, 4 integer samples from the left / top and right / bottom are required. Note that the length of the interpolation filter is different for a 1 / 4 pel sample position (7 taps) than for a 1 / 2 pel sample position (8 taps). The terms sample, pel, and pixel are used interchangeably in this application to refer to an image sample (in one or more color components) at a given time instance. A sample is essentially an intensity value.

[0104] In some embodiments of the present invention, the interpolation filter of a predefined tap size evaluates integer samples only within a window given by the integer samples accessible for template matching in the search space. The window may include many more samples than are actually used for the calculation of a certain prediction block. This is because the refinement search operation is usually performed using a fast search method (as opposed to a brute force search method), which means that some samples are not evaluated according to the stepwise progress of the search operation. As a result, the number of iterations of template matching may vary from prediction block to prediction block, as well as the samples used for the calculation for the refinement search operation.

[0105] Figure 8 shows a coding block (prediction block) and the corresponding sample of a window. It should be noted that the sample shown in Figure 8 is a reference picture sample, and the coding block here is actually a block whose size and position correspond to the current block in the current frame, for which a motion vector is derived in the reference picture. Therefore, in reality, the coding block in Figure 8 is essentially a block that is co-located with the block for which the predictor is searched. However, for the sake of brevity, this block will be referred to as the "coding block" hereinafter.

[0106] In this example, the unrefined motion vector MV0 points to an integer sample position. The search granularity of the motion vector refinement is one integer sample, which means that only integer sample points are searched since the starting point is an integer sample. In this example, the search is performed in a gradually evolving search space, which means that at each iteration the search space advances by adding a new search position according to the best direction in terms of a cost function of previously tested positions.

[0107] Such an approach is illustrated in a simplified manner in FIG. 9, where an initial motion vector points to a central point 810. A search space is gradually built around the initial motion vector position. In a first step, four adjacent positions immediately above, below, to the left, and to the right of the position 810 pointed to by the initial motion vector are tested in the same manner as the position 810 pointed to by the initial motion vector. Based on the direction in which the cost function is lowest among the five points tested, another position to be tested is added to the search space. In this example, the lowest cost function was found at the right point, so in a second step, the search space was expanded horizontally to the right by three more points. In the second step, the lowest cost function is found at the right point (relative to the lowest cost point of the first step), resulting in an expansion of the search space horizontally to the right by three more points. In a third step, the lowest cost function is again found at the right point relative to the lowest cost point of step 2, resulting in an expansion of the search space horizontally to the right by three more points. According to the example in Figure 9, three more steps are performed in the order up, up, and to the right. In the example, for each iteration, a diamond pattern (of five search points) is used, and the search space is expanded at each step to complete the missing search points.

[0108] At each iteration of the search space determination, the search space may increase by one or more integer sample positions. Returning now to FIG. 8, in this example, the maximum number of iterations of the search is four. Since a maximum of four iterations is possible, all samples shown on the left must be fetched from memory to perform the search operation, in which case the search space unfolds gradually to the left. Similarly, four samples must be extended upwards. Thus, since the refined MV can move in either direction, the search space is extended in both directions (left to right and top to bottom), and the hardware implementation requires that all samples that may be needed are fetched from external memory before the refinement search is applied. If the search space expands downwards or to the right, template matching with a template corresponding to the size of the coding block (prediction block) will require access to some of these samples, so it must be further extended by four samples. Furthermore, since the hardware implementation usually cannot accommodate irregular shapes (rectangular access is more feasible), samples at the corners (e.g., the top right) must also be fetched from memory.

[0109] It should be noted that the iterative search space expansion described above is merely an example, and the rules and number of points for expanding the search space in each iteration may be different, i.e., specified in a different way.

[0110] Figure 9 also illustrates a scenario that can arise due to the external memory access rules described above. The number of samples fetched from external memory is much larger than the samples actually used in a computation step. If we assume (for simplicity) that our template is only one sample large, then the open circles represent the samples fetched from external memory and the shaded ones represent the ones actually used. However, such redundancy is necessary if the number of accesses to external memory needs to be kept low, since the actual required samples are not yet known when the processing of the current block starts.

[0111] It should be noted that the search space may be defined in other ways, for example as a stable shape located at the position pointed to by the initial motion vector. The shape may be any shape, such as a square, rectangle, diamond, etc.

[0112] Figure 10 shows an example where the search space may further include fractional samples. In Figures 8 and 9, the motion vector search was performed on integer samples, resulting in the location shown by the solid larger dot in the access window. Here, the search is performed on samples with 1 / 2 pel resolution (the solid smaller dot), and to generate the fractional sample shown on the left, three more rows of samples must be fetched from memory, and the interpolation filter is assumed to be symmetrical and have eight taps. Furthermore, since the search operation is symmetrical (it can move iteratively left and right), the same must be applied to the left side (extending by three rows of pixels), and as a result, the fractional pixel may be located on the left side of the window.

[0113] As a result, the interpolation filtering requires that more samples be fetched from memory, as shown by the dashed lines, to include the dotted circles that represent the positions added by fractional interpolation.Similarly, allowing half a search at each vertical position requires that the window of samples accessed from memory be expanded vertically at the top and bottom as well (not shown in the example in Figure 10).

[0114] The window of memory access is defined as a rectangle that encloses all samples that need to be fetched from memory to perform a motion vector search for a prediction block (coding block). The window of memory access not only includes the actual samples that are needed, but also all remaining samples that may be accessed during the motion vector search operation. In the example of FIG. 9, the motion vector search moved to the right. However, it could have also moved to the left, which is not known in advance. Therefore, to avoid accessing the external memory multiple times, the window of memory access (or access window) includes all samples that can be accessed in each operation.

[0115] To achieve this, the present disclosure performs padding of samples located outside the memory access window when necessary, for example to interpolate fractional samples.

[0116] By providing a substitution sample (padding sample), the association with the template matching may be more favorable in a search space that further includes fractional samples. In other words, the search space of the prediction block may include at least one fractional sample position with a value obtained by interpolation filtering with a filter of a predefined tap size. The interpolation filtering uses a value of the sample position from a window and at least one substitution value to obtain a value of the at least one fractional sample position.

[0117] Figure 11 shows the basic steps of the known technique for performing motion vector derivation / refinement as outlined above. In a first step (step 1), a "search space" of at least two candidate motion vectors is obtained (in the illustrated example, there are nine candidate motion vectors). Optionally, one of the candidate motion vectors forming the search space may be considered as an initial motion vector (also called an "anchor" motion vector, or anchor candidate motion vector, according to the terminology used in this application). The initial or anchor motion vector may be, for example, a motion vector included in the bitstream and explicitly or implicitly signaled to the decoder.

[0118] In the next step (step 2), a matching process is applied to the search space to find the best matching point of the search space (in other words, the point of the search space that minimizes the cost function and corresponds to the most suitable candidate motion vector). In the illustrated example, bidirectional matching is applied, but other schemes for determining the cost function, such as template matching, may be applied as well.

[0119] In the last step (step 3) of the diagram, a final prediction is obtained by interpolation based on the most suitable candidate motion vector selected according to the process of step 2.

[0120] The concepts of Figure 11 are further explained in the flow diagram of Figure 12, where a diagram is shown specifically for the case of candidate motion vector pairs, i.e. bi-directional matching (also applicable when template matching is used for bi-directional prediction, e.g. as described above with reference to Figure 3).

[0121] In the first step (1310), a search space of candidate motion vector pairs is constructed. This step corresponds to "Step 1" in Figure 11.

[0122] In the next step (1320), a prediction is generated for each motion vector pair in the search space by applying interpolation filtering. In the next step (1330), a template is constructed (obtained) for each motion vector pair in the search space. As mentioned above, this step is optional, i.e., it is only necessary for template matching, but not for bidirectional matching. In the next step (1340), for each motion vector pair in the search space, a template matching cost (or bidirectional matching cost) is calculated by comparing the prediction blocks pointed to by the two motion vectors of the pair (or the prediction block of one candidate motion vector, and the template). In the next step (1370), the resulting motion vector pair that minimizes the template or bidirectional cost function (more generally, the best match, i.e., obtained as the most suitable by the matching process) is selected and output as the motion vector pair of the prediction block. According to the terminology mentioned above, this may be called a "refinement motion vector pair". The above steps 1320, 1330, 1340, and 1370 correspond to "step 2" in FIG. 11.

[0123] The next step (1380) is to generate a final prediction by interpolation using the motion vector pair selected in step 1370. This step corresponds to "step 3" in Figure 11.

[0124] Fig. 13 shows a process for motion vector refinement and final prediction generation according to an exemplary embodiment of the present invention. The main difference between the approach of Fig. 13 and the conventional approach shown in Fig. 11 is that the approach of Fig. 13 of the present invention uses sample padding for both motion vector refinement and final generation of predictor using interpolation filtering.

[0125] The (optional) use of padded samples for motion vector refinement is shown in additional step 2, which will be described in more detail below with reference to Figures 14 to 18. Thereby, even in the case of candidate motion vectors with fractional coordinates (i.e. fractional coordinates in the search space), the candidate blocks for which it is desired to perform bidirectional or template matching can be performed by using a predefined restricted access window, which is the same as in the case of a search space containing only integer points. The actual derivation of the cost function by applying bidirectional or template matching is performed in step 3, which is the same as step 2 in Figure 11.

[0126] Furthermore, sample padding is used in the final step 4 (again or only here) to perform the actual predictor generation. This step generally corresponds to step 3 of Fig. 11, except that it uses the padding concept. Specifically, in the given example, the sample values ​​located in the rightmost two columns and the bottom row are not accessed, but are padded with padding sample values ​​based on the other sample values ​​that are accessed.

[0127] The number of samples that are padded (replaced with replacement values ​​because they are unavailable) depends on the difference between the last selected ("refined") motion vector and the reference initial motion vector ("anchor"). In the illustrated example, this difference is 2 pixels vertically and 1 pixel horizontally. Accordingly, in the diagram of step 5, to horizontally and vertically interpolate the pixel at the intersection of the two rectangles, the sample values ​​of the pixels in the two columns to the right of the pixel and in the last row are unavailable and need to be replaced (padded) with replacement values ​​according to the predefined rules mentioned above, which will be explained in more detail below.

[0128] Preferably, if padding is also used for the search in step 2, then the same access window is used for both the step 2 and step 4 interpolation filtering operations.

[0129] Preferably, the memory access window used in steps 2 and 4 of Figure 13 is determined based on a particular candidate motion vector in the search space. The particular motion vector (called an anchor motion vector) may be, for example, a motion vector that is included in the bitstream and explicitly or implicitly signaled to the decoder. The selection decision of the anchor motion vector is performed according to predefined rules that are known to both the encoder and the decoder.

[0130] FIG. 14 shows a window of memory accesses for motion vector refinement. The window is spread around a central point 1010, which is the location pointed to by the initial motion vector. In addition, a fractional location 1020 also belongs to the search space. However, the fractional location 1020 is interpolated by a 6-tap filter using three integer sample locations to the left of the fractional location and three integer sample locations to the right of the fractional location. Since the fractional location 1020 is collinear with the integer sample locations, a one-dimensional filter may be applied for the interpolation, for example as shown in the above example described with reference to FIG. 10. As can be seen in FIG. 14, two further sample locations 1030 are needed to interpolate the fractional sample 1020, which are located outside the window. To avoid further loading of these samples from an external memory, according to the present disclosure, the values ​​of the locations 1030 outside the window are replaced with values ​​obtained based on the samples within the window.

[0131] FIG. 14 shows a fractional sample location 1020 that is halfway between two integer sample locations and thus at a 1 / 2 pixel (1 / 2 pel) location. However, the present disclosure is equally applicable to other fractional locations, such as 1 / 4 or 1 / 8 locations. Furthermore, the sample location 1020 is located on a horizontal line corresponding to a row of integer sample locations, so that horizontal 1D filtering may be used for its interpolation. However, the present disclosure is not limited to such fractional locations. The fractional locations may also be located on a vertical line corresponding to a column of integer sample locations, so that vertical 1D filtering may be used for its interpolation. Furthermore, the fractional locations do not necessarily have to be located on the same line of pixels as the integer samples. In such cases, two-dimensional filtering may be used to interpolate such samples, and such 2D filtering may be splittable into vertical 1D filtering and horizontal 1D filtering.

[0132] Examples of various fractional positions are shown in Figure 16. In particular, fractional position 1 is a position for interpolation where both horizontal and vertical filtering can be applied since it is not collinear with the rows and columns of integer sample positions. Position 2 is a position for interpolation where only horizontal filtering is used, and position 6 is a position for interpolation where only vertical filtering is used. All fractional positions shown in Figure 16 can be obtained using only integer positions within the window.

[0133] The window for memory access may be defined in various ways. Figure 18 shows an example where the memory access window is defined as an extension part EXT to the left / right or above / below the coding block (corresponding to the position of the coding block given by the initial motion vector). The amount of extension may depend on the size and shape of the coding block or the prediction block. In Figure 18, the extension parts are 4 samples long in each direction (above, below, left, right). However, it should be noted that EXT may take different values ​​for different directions depending on the block size (which may be different in size vertically and horizontally) and / or the form and size of the search space.

[0134] For example, according to the example, the window is defined as N integer sample columns and M integer sample rows for the predicted block initial motion vector, where N and M are non-zero integer values. In FIG. 18, N and M are shown to be the same size. As mentioned above, N and M may be different sizes. N and M are integers, at least one of which is non-zero. By taking the parameters N and M, as well as the template format and size, the size of the access window can be determined. In particular, if the template has T1 rows and T2 columns, the size of the memory access window can be calculated as (N+T1+N) rows and (M+T2+M) columns. This is because the search can proceed N samples to the left or right, resulting in 2N samples horizontally, and M samples up or down, resulting in 2M samples vertically. Also, the template can be aligned with T1×T2 samples at any of the 2N×2M sample positions.

[0135] On the other hand, for a particular search space construction approach such as that described with reference to Figures 8 and 9, the memory access window may be defined in terms of the maximum number of refinement iterations (search space construction iterations) and the iteration step size (in terms of the maximum sample distance achievable in each iteration), which may then be translated into the maximum amount of displacement to the left, right, up, and down. Thus, the memory access window is defined as the maximum displacement in each direction. For example, if for four iterations, each iteration may advance the search space by at most one integer sample position, then EXT=4.

[0136] In other words, according to this example, the processing circuitry is configured to determine a refinement of the initial motion vector by template matching with said template in a search space iteratively expanded in a direction given by one (or more) of the best matching positions of the search space in the last iteration, the window being defined by a predefined maximum number of iterations. The same applies mutatis mutandis to the case of bidirectional matching.

[0137] It should be noted that in general, the present disclosure is not limited to any particular shape or form or type of determination of the search space. In another example, the search space may be a rectangular sub-window of the window or equal to the window. The definition of the memory access window may be important since the corresponding signaling parameters (e.g., the parameter EXT, or the parameters N and M) may be included in the bitstream. However, the size of the memory access window may be defined normally or may be derivable based on other coding parameters (such as the template size, the prediction block size, the image resolution, etc.). Returning to FIG. 14, according to an embodiment, the window of memory access for refinement is defined around the location 1010 pointed to by the unrefined motion vector. The window identifies the maximum number of pixel samples that need to be accessed from memory to perform the motion vector refinement search. The search space here coincides with the access window. In other words, although the template size is taken to be 1x1 in this example for simplicity, it can and does usually be larger. The window of memory access is usually defined as an extension of the periphery of the coding block. In this example, the extension is 4 samples from the left / right and 4 samples from the top / bottom. If the search space position currently being tested during template matching or bidirectional matching requires samples from outside the window of memory accesses, the samples required for reconstructing the coding block are obtained by padding. This is the case for fractional position 1020 in Fig. 14. If template matching is performed at or using this position, it must be interpolated using another padding position 1030 located outside the window.

[0138] The padding samples may be generated, for example, using one of the following methods. Recent sample copy. Axis-based mirroring of interpolation filtering.

[0139] Specifically, nearest sample duplication refers to a technique in which the replacement value is determined to be equal to the value closest to the integer sample position being accessed. For example, in FIG. 14, the values ​​1030 of the two missing positions are replaced with the values ​​of the samples immediately to the right of the fractional position 1020 being interpolated. However, it should be noted that this technique is merely an example, and the present disclosure is not limited to merely duplicating the neighborhood. Alternatively, several nearest sample positions may be used to interpolate the missing sample 1030. For example, the three samples on the boundary closest to the missing sample may be used to interpolate the missing sample, for example, by weighted averaging with weights set inversely proportional to the distance of the three samples from the missing sample.

[0140] Mirroring refers to a technique in which replacement values ​​are determined by mirroring the values ​​of corresponding evaluated integer sample positions relative to the axes of the interpolation filter. For example, in Fig. 14, the two missing samples 1030 are replaced with the values ​​of the samples to the left of the fractional position 1020. Specifically, the second sample to the right of the fractional position 1020 is replaced with the value of the second position to the left of the fractional position 1020, and the third sample to the right of the fractional position 1020 is replaced with the value of the third position to the left of the fractional position 1020.

[0141] It should be noted that a combination of the two approaches described above may also be applied if the missing sample is a function of the mirrored sample and the nearest sample.

[0142] According to an embodiment, a window is formed by all integer sample positions accessible for template matching or bidirectional matching at integer sample positions in the search space. This is the case for example in Fig. 14, where additional samples outside the window are only required for template matching at non-integer fractional sample positions. In other words, in Fig. 14, in the above example, the 1 / 2 pel coordinates to be searched require samples outside the memory access window. As a result, a padding operation is applied to generate unusable samples. However, the invention is not limited thereto.

[0143] Figure 15 shows an alternative approach where the window does not include all integer sample positions accessible for template matching at integer sample positions in the search space. In the previous example, the coding block size was shown as 1x1 for simplicity. In Figure 15, the size is 4x4 integer samples. In this example, even if the search coordinates (locations where the template or bidirectional match is calculated) point to integer samples, padding can still be applied if some of the samples in the block are outside the memory access window. This approach allows the window size to be further restricted.

[0144] In more detail, Fig. 15 shows a window 1120, which includes several accessible samples. The initial motion vector points to a position that marks the top left corner of the coding block. The search space may here be considered equal to the window. However, it should be noted that it can usually be smaller than the window, which may also make sense, since if the template is larger than 1x1 integer samples, then necessarily other samples than those included in the search space are required to perform the template matching. Nevertheless, if the search space defines the size of the window, the missing samples may alternatively be padded. In Fig. 15, an exemplary search position is shown in the second row and tenth column of the window. Template matching with a template of size 4x4 integer samples would still require pixels that are outside the window. These samples are therefore padded, i.e. replaced with values ​​determined or calculated based on the samples in the window. The padding techniques mentioned above, such as nearest neighbor or mirroring, may be applied.

[0145] The window is the search space, or a block having the size of the predicted block located at the initial motion vector position, or initial motion vector position, The vertical and / or horizontal size of the

[0146] An example of window definition is shown in FIG. 18 and described above. This corresponds to determining the window size based on the size of the predicted block located at the initial motion vector position. Alternatively, the window size may be defined relative to the initial vector position by the size in one or both of the x and y directions. For example, the window may be defined to be located around the position pointed to by the initial motion vector and extending A integer samples in each direction up, down, left, and right. Alternatively, it may extend A integer samples to the left and right, respectively, and B integer samples up and down. Other definitions are possible.

[0147] It should be noted that the above examples have been provided for 1 / 2 pel interpolation. However, the present disclosure is not limited thereto. In general, any fractional position such as 1 / 4, 1 / 8, etc. may be used, i.e., interpolated using a corresponding interpolation filter.

[0148] The same definition of window size also applies to the case of padding for interpolation filtering to generate the predictor: specifically, the vertical and horizontal sizes (i.e., in the x and y directions) are determined by the vertical and horizontal lengths of the interpolation filter used to determine the predictor.

[0149] Figure 17 is a flow diagram illustrating the process shown in Figure 13 in more detail. The flow diagram of Figure 17 differs from the flow diagram of Figure 12, which illustrates a conventional process, in that steps 1320 and 1380 of Figure 12 are replaced by steps 1720 and 1780, respectively. These steps differ from the steps of Figure 12 in that padding with replacement samples is applied for both the interpolation filtering to generate candidate prediction blocks in the reference picture(s) to perform bidirectional or template matching, and for the interpolation filtering to generate the final prediction.

[0150] The processing circuitry described with reference to FIG. 7 may be used in an encoder and / or decoder such as those shown in FIGS.

[0151] In particular, an encoding device may be provided for encoding a video image divided into predictive blocks into a bitstream, the encoding device comprising an apparatus for determining a motion vector for the predictive block as described above, comprising a processing circuit, and an encoding circuit for encoding a difference between the predictive block and a predictor given by the predictive block at a position specified by the determined motion vector and generating a bitstream comprising the encoded difference and the initial motion vector.

[0152] The other units and functions of the encoder described above with reference to FIG. 1 may also be provided or implemented in the processing circuit.

[0153] Correspondingly, a decoding device is provided for decoding a video image divided into prediction blocks from a bitstream, the decoding device comprising: an analysis unit for analyzing from the bitstream an initial motion vector and an encoded difference between the prediction block and a predictor given by the prediction block at a position specified by the refinement motion vector; an apparatus for refinement motion vector determination for the prediction block as described above, comprising a processing circuit; and a decoding circuit for reconstructing the prediction block as the sum of the analyzed difference and the predictor given by the prediction block at a position based on the refinement motion vector. For example, the predictor may be given directly by the position of the refinement motion vector. However, there may be another processing step to obtain the motion vector of the current prediction block, which may further change the motion vector (filtering, clipping, further refinement, etc.).

[0154] The further units and functions of the decoder described above with reference to FIG. 2 may also be provided or implemented in a processing circuit.

[0155] Furthermore, although embodiments of the present invention have been described in terms of apparatuses comprising processing circuitry for performing motion vector refinement, the present disclosure is not limited thereto but also provides corresponding methods, which include processing steps corresponding to the steps that the processing circuitry described above is configured to perform.

[0156] The taps correspond to the filter coefficients. The tap size corresponds to the filter order. Here, the filter is assumed to be a linear filter. In some examples, the filter may be symmetric, i.e., have symmetric coefficients. However, this disclosure is not limited to symmetric or linear filters, or any type of filter. In general, the fractional position may be obtained in any manner based on neighboring samples.

[0157] A method according to an embodiment determines a motion vector for a prediction block in the following steps shown in FIG. 19. In step 1410, an initial motion vector and a template for the prediction block are obtained, and then a refinement of the initial motion vector is determined by template matching with said template in a search space. The search space is located at a position given by the initial motion vector. In 1420, a memory access window is determined. Note that this does not have to be performed explicitly. The memory access window may be predefined and known. Therefore, this step is illustrated to show that knowledge of the memory access window size is known. Then, template matching is performed in steps 1430 to 1490.

[0158] The template matching or bidirectional matching accesses integer sample positions within a predefined window, a predefined window that includes the search space, and other integer sample positions that are accessible for template matching or bidirectional matching, and replaces at least one integer sample position located outside the window with a replacement value based on at least one sample within the window. The values ​​of the accessed integer sample positions and the replacement value are used to perform the template matching. This is shown in FIG. 19, where the template matching or bidirectional matching is performed at all positions within the search space. In step 1430, it is tested whether the template matching or bidirectional matching at the current position requires one or more samples that are outside the memory access window. If so (step 1430 is yes), then in step 1440, padding is performed as described above for one or more samples that are outside the window. After this step, in step 1450, the template matching or bidirectional matching (i.e., search) is performed at the current position with the padded samples. Alternatively, if there are no needed samples outside the window (step 1430: No), then template matching or bidirectional matching (i.e., searching) is performed at the current position using the available samples in step 1450. If the current position is the last position to be tested in the search space (step 1460: Yes), then template matching ends. Otherwise (step 1460: No), the next position is obtained in step 1470 and template matching continues for the new position as described above in step 1430.

[0159] Once the template matching is completed, the best matching location is identified (eg, as the stored location with the lowest cost).

[0160] Figure 20 shows an alternative exemplary approach of motion vector refinement and predictor generation applicable within the framework of an embodiment of the present invention. The difference between this approach and the approach of Figure 13 is that in Figure 20, the interpolation filtering with padding is applied only to the final generation of the predictor using the refinement motion vector (pair). This is step 5 of Figure 20 and corresponds to step 4 of Figure 13.

[0161] On the other hand, no interpolation filtering with padding is applied to the motion vector refinement (search), i.e., step 2 of FIG. 13. Instead, in the approach of FIG. 20, if the initial motion vector (or multiple initial candidate motion vectors forming the initial search space) points to the nearest integer point, a rounding operation is performed to round the coordinates to the nearest integer point (step 2 of FIG. 20). This rounding operation is the same as a predefined vector movement, indicated by MV_round_diff, which represents the rounding direction. The rounding operation ensures that the search space is modified to obtain a modified search space that has only points with integer coordinates. As a result, the next step 3 of obtaining the best matching integer point by bidirectional (or template) matching can be performed with integer coordinates, and as a result, no interpolation filtering is required to generate a prediction for template / bidirectional matching.

[0162] FIG. 20 further includes step 4. This step is optional. In step 4, after the best matching integer point in the modified search space is obtained in step 3, the application of rounding is undone to obtain a final refinement motion vector in the (initial) search space. In other words, a vector moving in the opposite direction to step 2 is applied to the best matching (refinement) motion vector resulting from step 3. If optional step 4 is included, then a final prediction is obtained based on the refinement motion vector resulting from step 4. Alternatively, step 3 is omitted and step 5 is directly applied to the motion vector resulting from step 3. As a further alternative, the inclusion or omission of step 4 can be determined based on a predefined condition.

[0163] A flow diagram corresponding to the technique of Figure 20 is shown in Figure 21. Again, the flow diagram is created for motion vector pairs rather than single motion vectors, which are used in Figure 20 for the sake of simplicity.

[0164] The first step 1310 is the same as in Figure 20. The subsequent step 2115 corresponds to the rounding operation of step 2 of Figure 20. Insofar as the modification of the initial search space is not limited to a rounding operation, the process shown here is general, but more generally, it is assumed herein that for each motion vector pair in the initial search space, a representative motion vector pair is generated. The rounding operation of Figure 20 is a special case of this mapping of initial search space motion vectors to representative ones.

[0165] The following steps 2130 (optional for template matching) and 2140 are the same as in Figure 17, except that the operations are performed on the representative motion vectors in the modified search space. Steps 2150 and 2160 correspond to step 4 in Figure 20. Step 2150 shows the step of setting the representative motion vector pair that minimizes the template matching cost (or bidirectional matching cost) as Pair_best_rep.

[0166] Step 2160 (optional) corresponds to undoing the rounding operation shown in step 4 of Figure 20. In the next step (1370), the motion vector (pair) in the original search space is output as the motion vector for the prediction block. Steps 1370 and 1780 are the same as in Figure 17.

[0167] Further, an encoding method is provided for encoding a video image divided into predictive blocks into a bitstream, the encoding method comprising the steps of determining a motion vector for the predictive block according to any of the methods described above, encoding a difference between the predictive block and a predictor given by the predictive block at a position based on the determined motion vector, and generating a bitstream comprising the encoded difference and an initial motion vector.

[0168] The encoding method may further include the steps described with reference to the functional blocks in FIG.

[0169] Further, a decoding method is provided for decoding a video image divided into predictive blocks from a bitstream, the decoding method comprising the steps of analysing from the bitstream an initial motion vector and an encoded difference between the predictive block and a predictor provided by the predictive block at a position specified by the refinement motion vector, determining a refinement motion vector for the predictive block according to any of the methods described above, and reconstructing the predictive block as a sum of the analysed difference and the predictor provided by the predictive block at the position specified by the refinement motion vector.

[0170] The decoding method may further include the steps described with reference to the functions of the blocks in FIG.

[0171] However, it should be noted that Figures 1 and 2 are not intended to limit this disclosure, but merely provide non-limiting examples of implementation of the present invention within existing encoders and / or decoders.

[0172] The motion vector determination using sample padding as described above may be implemented as part of encoding and / or decoding of a video signal (moving image). However, the motion vector determination is not limited to being used for encoding / decoding, but may also be used for other purposes in image processing, such as motion detection, motion analysis, etc.

[0173] The motion vector determination may be implemented as an apparatus. Such an apparatus may be a combination of software and hardware. For example, the motion vector determination may be performed by a general-purpose processor, or a digital signal processor (DSP), or a chip such as a field programmable gate array (FPGA). However, the present invention is not limited to implementation on programmable hardware. It may be implemented on an application specific integrated circuit (ASIC) or by a combination of the hardware components mentioned above.

[0174] The motion vector determination may be implemented by program instructions stored in a computer-readable storage medium. When the program is executed, the computer performs the steps of the above-mentioned method. The computer-readable storage medium may be any medium on which a program is stored, such as a DVD, a CD, a USB (flash) drive, a hard disk, or a server storage available via a network.

[0175] The encoder and / or decoder may be implemented in various devices such as a TV set, a set-top box, a PC, a tablet or a smartphone, i.e. any recording device, encoding device, transcoding device, decoding device or playback device etc. It may also be a software or app implementing the method steps and may be stored / executed on a processor included in an electronic device such as those mentioned above.

[0176] In summary, the present disclosure relates to motion vector determination using template matching or bidirectional matching, and predictor generation based on the motion vector. The template matching or bidirectional matching, and / or predictor generation use interpolation filtering. The interpolation filtering operation accesses integer sample positions within a window, the window including the search space and another integer sample position accessible for template matching, pads at least one integer sample position located outside the window with a padded sample value based on at least one sample within the window, and uses the accessed integer sample position value and the padded sample value to perform template matching or bidirectional matching and predictor generation.

[0177] Further embodiments of the present invention are provided below. Please note that the numbering used in the following sections does not necessarily follow the numbering used in the previous sections.

[0178] 1. An encoding device for encoding a predictive block of a video image into a bitstream (171), the encoding device comprising: Obtaining at least one initial motion vector; determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refinement motion vector, said search space being located at a position (810) given by the initial motion vector; determining a motion vector for the predicted block (165) by encoding a difference (105) between the predicted block and a predictor provided by the predicted block at a position based on the determined motion vector to generate a bitstream (171) including the encoded difference and the initial motion vector; determining a predictor according to a motion vector using an interpolation filtering with an interpolation filter, the interpolation filtering comprising: Access sample values ​​at integer sample positions within a predefined window, replacing at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within said window; performing interpolation filtering using the accessed integer position sample values ​​and the substitution values; A processing circuit (600) configured to Encoding device.

[0179] Embodiment 2. A decoding device for decoding a video image divided into predictive blocks from a bitstream (171), the decoding device comprising: Analyzing, from the bitstream at a location specified by the refinement motion vector, an initial motion vector and an encoded difference between the prediction block and a predictor provided by the prediction block; Obtaining at least one initial motion vector; determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refinement motion vector, said search space being located at a position (810) given by the initial motion vector; determining a motion vector for the prediction block by determining a motion vector by reconstructing the predicted block as a sum of the analyzed difference and a predictor provided by the predicted block at a location specified by the refinement motion vector determined by the processing circuit (600); A processing circuit configured to The processing circuitry is further configured to determine a predictor according to the refinement motion vector using interpolation with an interpolation filter, the interpolation comprising: Access sample values ​​at integer sample positions within a predefined window, replacing at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within said window; performing an interpolation using the accessed integer sample position values ​​and the replacement values; Decoding device.

[0180] Embodiment 3. An apparatus as described in embodiment 1 or 2, wherein the replacement values ​​are determined by mirroring the values ​​of the corresponding evaluated integer sample positions relative to the axes of the interpolation filter.

[0181] Embodiment 4. An apparatus according to any one of embodiments 1 to 3, wherein the interpolation filter is a one-dimensional filter or a separable two-dimensional filter.

[0182] Embodiment 5. An apparatus according to any one of embodiments 1 to 4, wherein the window is formed by all integer sample positions accessible to obtain a predictor given by the prediction block at a position specified by the initial motion vector.

[0183] Embodiment 6. A window, a block having the size of the prediction block located at the initial motion vector position; the initial motion vector position, and The vertical and horizontal lengths of the interpolation filter used to determine the predictor 6. An apparatus according to any one of the preceding embodiments, wherein the vertical and / or horizontal size of the at least one of the first and second pixels is defined by its vertical and / or horizontal size.

[0184] Embodiment 7. The apparatus of any one of embodiments 1, 2, or 4 to 6, wherein the replacement value is determined to be equal to the value closest to the accessed integer sample position.

[0185] Embodiment 8. An apparatus according to any one of embodiments 1 to 7, wherein the number of sample values ​​to be replaced depends on the difference between the initial motion vector and the refinement motion vector.

[0186] Embodiment 9. The processing circuit (600) is configured to determine the refinement of the initial motion vector by template matching or bidirectional matching using a template; The template matching or the bidirectional matching is accessing sample values ​​at integer sample positions within another predefined window (1120), the another predefined window (1120) including the search space and another integer sample position accessible for template matching or bidirectional matching; replacing at least one sample value (1030) at an integer sample position located outside the other window with a replacement value based on at least one sample within said other window; 9. An apparatus as in any one of embodiments 1 to 8, further comprising: performing template matching or bidirectional matching using values ​​of the accessed integer sample positions and replacement values.

[0187] Embodiment 10. The search space includes at least one fractional sample position (1,2,6) with a value obtained by interpolation filtering with a filter of a predefined tap size; 10. The apparatus of embodiment 9, wherein interpolation filtering uses values ​​of the accessed integer sample positions and at least one substitution value to obtain values ​​for the at least one fractional sample position.

[0188] Embodiment 11. An apparatus as described in embodiment 10, wherein the replacement values ​​used in the template matching or bidirectional matching are determined by mirroring the values ​​of the corresponding accessed integer sample positions relative to the filter axis.

[0189] Embodiment 12. An apparatus as described in embodiment 9 or 10, wherein the replacement value used in the template matching or bidirectional matching is determined to be equal to the value closest to the accessed integer sample position.

[0190] Embodiment 13. Another predefined window (1120) search space, a block having the size of the prediction block located at the initial motion vector position; and Initial motion vector position 13. An apparatus according to any one of embodiments 9 to 12, wherein the apparatus is defined by its vertical and / or horizontal size with respect to at least one of the following:

[0191] Embodiment 14. An apparatus according to any one of embodiments 9 to 13, wherein the another predefined window (1120) is the same as the predefined window.

[0192] Embodiment 15. An apparatus as described in any one of embodiments 1 to 14, wherein the refinement is determined by bidirectional matching and the processing circuit is configured to obtain at least two initial motion vectors pointing to positions in different reference pictures (Ref0, Ref1) and forming a pair of initial motion vectors (Pair_0, Pair_1, ..., Pair_N).

[0193] Embodiment 16. The refinement is determined by template matching, and the processing circuitry further comprises: Get the template for the predicted block, determining a refinement of the initial motion vector by template matching using the template in the search space; 15. The apparatus of any one of the preceding embodiments, further configured to:

[0194] Embodiment 17. An encoding method for encoding a video image divided into predictive blocks into a bitstream, the encoding method comprising: obtaining (1310, 1410) at least one initial motion vector; determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refinement motion vector (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160), the search space being located at a position given by the initial motion vector; determining a motion vector for the prediction block, encoding a difference between the prediction block and a predictor provided by the prediction block at a position based on the determined motion vector to generate a bitstream including the encoded difference and the initial motion vector; determining (1780) a predictor according to the motion vector using interpolation filtering with an interpolation filter, the interpolation filtering comprising: Access sample values ​​at integer sample positions within a predefined window, replacing at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within said window; performing interpolation filtering using the accessed integer position sample values ​​and substitution values; The encoding method, including

[0195] Embodiment 18. A decoding method for decoding a video image divided into predictive blocks from a bitstream, the decoding method comprising: parsing, from the bitstream at a location specified by the refinement motion vector, an initial motion vector and an encoded difference between the prediction block and a predictor provided by the prediction block; obtaining (1310, 1410) at least one initial motion vector; determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refinement motion vector (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160), the search space being located at a position given by the initial motion vector; determining a motion vector for the prediction block, reconstructing the prediction block as the sum of the analyzed difference and a predictor given by the prediction block, at a position specified by the refinement motion vector determined by the step of determining the motion vector; determining (1780) a predictor according to the refinement motion vector using interpolation with an interpolation filter, the interpolation comprising: Access sample values ​​at integer sample positions within a predefined window, replacing at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within said window; performing an interpolation using the accessed integer position samples and the replacement values ​​(1780); A decoding method including:

[0196] Embodiment 19. The method according to embodiment 17 or 18, wherein the replacement values ​​are determined by mirroring the values ​​of the corresponding evaluated integer sample positions relative to the axes of the interpolation filter.

[0197] Embodiment 20. The method according to any one of embodiments 17 to 19, wherein the interpolation filter is a one-dimensional filter or a separable two-dimensional filter.

[0198] Embodiment 21. A method according to any one of embodiments 17 to 20, wherein the window is formed by all integer sample positions accessible to obtain a predictor given by the prediction block at a position specified by the initial motion vector.

[0199] Embodiment 22. A window, a block having the size of the prediction block located at the initial motion vector position; the initial motion vector position, and The vertical and horizontal lengths of the interpolation filter used to determine the predictor 22. The method according to any one of embodiments 17 to 21, wherein the vertical size and / or horizontal size of at least one of the pixels is defined by the vertical size and / or horizontal size of the pixels.

[0200] Embodiment 23. The method of any one of embodiments 17, 18, or 20 to 22, wherein the replacement value is determined to be equal to the value closest to the accessed integer sample position.

[0201] Embodiment 24. The method according to any one of embodiments 17 to 23, wherein the number of sample values ​​to be replaced depends on the difference between the initial motion vector and the refinement motion vector.

[0202] 25. The step of determining a refinement (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160) determines a refinement of the initial motion vector by template matching or bidirectional matching using a template; The template matching or the bidirectional matching is accessing sample values ​​at integer sample positions within another predefined window, the another predefined window including the search space and another integer sample position accessible for template matching or bidirectional matching; replacing (1440) at least one sample value at an integer sample position located outside the other window with a replacement value based on at least one sample within the other window; 25. The method according to any one of embodiments 17 to 24, further comprising performing template matching or bidirectional matching using the accessed integer sample position value and the replacement value (1450).

[0203] Embodiment 26. The search space includes at least one fractional sample position with a value obtained by interpolation filtering with a filter of a predefined tap size; 26. The method of embodiment 25, wherein interpolation filtering uses values ​​of the accessed integer sample positions and at least one substitution value to obtain values ​​of the at least one fractional sample position.

[0204] Embodiment 27. The method of embodiment 26, wherein the replacement values ​​used in template matching or bidirectional matching are determined by mirroring the values ​​of the corresponding evaluated integer sample positions relative to the filter axis.

[0205] Embodiment 28. The method of embodiment 26 or 27, wherein the replacement value used in the template matching or bidirectional matching is determined to be equal to the value closest to the accessed integer sample position.

[0206] Embodiment 29. Another predefined window is search space, a block having the size of the prediction block located at the initial motion vector position; and Initial motion vector position 29. The method according to any one of embodiments 25 to 28, wherein the vertical size and / or horizontal size of at least one of the pixels is defined by the vertical size and / or horizontal size of the pixels.

[0207] Embodiment 30. The method of any one of embodiments 25 to 29, wherein the another predefined window is the same as the predefined window.

[0208] Embodiment 31. A method according to any one of embodiments 17 to 30, wherein the refinement is determined by bidirectional matching and the obtaining step obtains at least two initial motion vectors pointing to positions in different reference pictures and forming an initial motion vector pair.

[0209] Embodiment 32. The refinement is determined by template matching, and the step of determining a motion vector comprises: obtaining a template for a predicted block (1330; 2130); determining (1340; 2140) a refinement of the initial motion vector by template matching using the template in the search space; 32. The method of any one of embodiments 17 to 31, further comprising:

[0210] Embodiment 33. A computer-readable storage medium storing instructions which, when executed on a processor, cause the processor to perform the method according to any one of embodiments 17 to 32. [Explanation of symbols]

[0211] 100 Encoder 101 Picture Block 102 Input 104 Residual calculation 105 Residual Blocks 106 Conversion Units 107 Conversion Factors 108 quantization units 109 Quantization Coefficients 110 Inverse Quantization Unit 111 Inverse quantization coefficients 112 Reverse conversion unit 113 Inverse Transformed Blocks 114 Reconstruction 115 Reconstructed Blocks 116 Buffers 117 Reference Samples 120 Loop Filtering Unit 121 Filtered Blocks 130 Decoded Picture Buffer 131 Decoded Picture 142 Inter-Estimation Units 144 Inter Prediction Units 152 Intra Estimation Units 154 intra prediction units 160 Mode Selection Unit 165 predicted blocks 170 Entropy Encoding Unit (Entropy Coding Unit) 171 Encoded picture data (encoded bitstream) 172 Output 200 Decoder 202 Input 204 Entropy Decoding Unit 209 Quantized Residual Transform Coefficients 210 Inverse Quantization Unit 211 Inverse quantized transform coefficients 212 Reverse conversion unit 213 Inverse Transformed Blocks 214 Reconstruction 215 reconstructed blocks 216 Buffers 217 Reference Sample 220 Loop Filtering Unit 221 Filtered Blocks 230 Decoded Picture Buffer 231 Decoded Picture 232 Output 244 Inter Prediction Units 254 intra prediction units 260 Mode Selection 265 predicted blocks 600 Processing Circuit 610 Template Determination Unit 620 Motion Vector Refinement Unit 640 Internal Memory 650 External Memory 810 center point 1010 Position pointed to by unrefined motion vector 1020 fractional position 1030 Another sample position 1120 Another Predefined Window

Claims

1. 1. An apparatus for receiving and storing a bitstream comprising coded blocks of a video image, the apparatus comprising: receiving the bitstream including the encoded blocks; storing the bitstream containing the encoded blocks; It is configured as follows: The coded blocks included in the bitstream are Obtaining an initial motion vector for a block; obtaining a refined motion vector for the block based on the initial motion vector; determining a predictor for the block according to the refined motion vector; encoding the block based on the predictor to obtain the encoded block; is generated by the refinement is determined by bilateral matching, and obtaining the initial motion vector includes obtaining at least two initial motion vectors pointing to positions in different reference pictures and forming an initial motion vector pair; determining the predictor according to the refined motion vector using interpolation with an interpolation filter, and using at least one sample value of an integer sample position located inside a window and padded sample values ​​of integer sample positions located outside the window for filtering by the interpolation filter, the padded sample value being based on at least one sample value of a sample located inside the window; Device.

2. 1. An apparatus for storing and transmitting a bitstream comprising coded blocks of a video image, said apparatus comprising: storing the bitstream containing the encoded blocks; transmitting the bitstream containing the encoded blocks. It is configured as follows: The coded blocks included in the bitstream are Obtaining an initial motion vector for a block; obtaining a refined motion vector for the block based on the initial motion vector; determining a predictor for the block according to the refined motion vector; encoding the block based on the predictor to obtain the encoded block; is generated by the refinement is determined by bilateral matching, and obtaining the initial motion vector includes obtaining at least two initial motion vectors pointing to positions in different reference pictures and forming an initial motion vector pair; determining the predictor according to the refined motion vector using interpolation with an interpolation filter, and using at least one sample value of an integer sample position located inside a window and padded sample values ​​of integer sample positions located outside the window for filtering by the interpolation filter, the padded sample value being based on at least one sample value of a sample located inside the window; Device.

3. 1. A method for receiving and storing a bitstream containing encoded blocks of a video image, comprising: receiving the bitstream including the encoded blocks; storing the bitstream containing the encoded blocks; Including, The coded blocks included in the bitstream are Obtaining the initial motion vector of the block (1310, 1410); Obtaining a refinement motion vector for the block based on the initial motion vector (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160); determining (1780) a predictor for the block according to the refined motion vector; encoding the block based on the predictor to obtain the encoded block; is generated by the refinement is determined by bilateral matching, and obtaining the initial motion vector includes obtaining at least two initial motion vectors pointing to positions in different reference pictures and forming an initial motion vector pair; the determining (1780) includes determining the predictor according to the refined motion vector using interpolation with an interpolation filter, and using at least one sample value of an integer sample position located inside a window and padded sample values ​​of integer sample positions located outside the window for filtering by the interpolation filter, the padded sample value being based on at least one sample value of a sample located inside the window. method.

4. 1. A method for storing and transmitting a bitstream containing coded blocks of a video image, comprising: storing the bitstream containing the encoded blocks; transmitting the bitstream including the encoded blocks; Including, The coded blocks included in the bitstream are Obtaining an initial motion vector for a block; obtaining a refined motion vector for the block based on the initial motion vector; determining (1780) a predictor for the block according to the refined motion vector; encoding the block based on the predictor to obtain the encoded block; is generated by the refinement is determined by bilateral matching, and obtaining the initial motion vector includes obtaining at least two initial motion vectors pointing to positions in different reference pictures and forming an initial motion vector pair; the determining (1780) includes determining the predictor according to the refined motion vector using interpolation with an interpolation filter, and using at least one sample value of an integer sample position located inside a window and padded sample values ​​of integer sample positions located outside the window for filtering by the interpolation filter, the padded sample value being based on at least one sample value of a sample located inside the window. method.

5. The generation of the coded blocks comprises: determining a refinement of the initial motion vector based on a search space of candidate motion vectors, wherein the refinement motion vector is determined based on the initial motion vector and the refinement.

5. The method of claim 3 or 4, further comprising:

6. The method of claim 5 , wherein the search space is located at a position defined by the initial motion vector.

7. The generation of the coded blocks comprises: Obtaining the difference between the block and the predictor.

7. The method of claim 3, further comprising:

8. The generation of the coded blocks comprises: reconstructing the block based on the predictor and the difference between the block and the predictor.

8. The method of claim 7, further comprising:

9. 9. The method of claim 3, wherein the padded sample values ​​are determined by mirroring the values ​​of corresponding estimated integer sample positions relative to the axes of the interpolation filter.

10. 10. The method of claim 3, wherein the interpolation filter is a one-dimensional filter or a separable two-dimensional filter.

11. 11. The method according to claim 3, wherein the window is formed by all integer sample positions accessible for obtaining the predictor at a position specified by the initial motion vector.

12. The window is a block having the size of the block located at the position of the initial motion vector; the position of the initial motion vector, and The vertical and horizontal extents of the interpolation filter used to determine the predictor.

12. The method according to claim 3, wherein the vertical and / or horizontal size of the image is defined by at least one of:

13. 13. The method of claim 3, wherein the padded sample value of the integer sample position located outside the window is determined to be equal to the sample value of at least one used integer sample position located inside the window that is closest to the integer sample position located outside the window.

14. The method according to claim 3 , wherein the number of sample values ​​to be padded depends on the difference between the initial motion vector and the refinement motion vector.

15. obtaining refinement (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160) determines refinement of the initial motion vector by the bilateral matching; The bilateral matching accessing sample values ​​at integer sample positions within another window, the other window including a search space and another integer sample position accessible for the bilateral matching; padding (1440) at least one sample value at an integer sample position located outside the other window with a padded sample value based on at least one sample within the other window; The method of any one of claims 3 to 14, further comprising performing (1450) the bilateral matching using values ​​of the accessed integer sample positions and the padded sample values.

16. the search space includes at least one fractional sample position with a value obtained by interpolation filtering with a filter of a predefined tap size; 16. The method of claim 15, wherein the interpolation filtering uses values ​​of the accessed integer sample positions and at least one of the padded sample values ​​to obtain values ​​for the at least one fractional sample position.

17. The method described in claim 16, wherein the padded sample values ​​used in the bilateral matching are determined by mirroring the values ​​of the corresponding evaluated integer sample positions relative to the axis of the filter.

18. A method as described in claim 16 or 17, wherein the padded sample value used in the bilateral matching is determined to be equal to the value closest to the accessed integer sample position.

19. The other window is the search space, a block having the size of the block located at the position of the initial motion vector; and the position of the initial motion vector 19. The method according to claim 15, wherein the vertical and / or horizontal size of at least one of the following is defined:

20. 20. The method of any one of claims 15 to 19, wherein the another window is the same as the window.

21. A computer readable storage medium storing instructions that, when executed on a processor, cause the processor to perform the method of any one of claims 3 to 20.

Images