Memory Access Windows and Padding for Motion Vector Refinement and Motion Compensation

Abstract

To perform motion vector determination using template matching or bi-directional matching and motion vector based predictor generation.SOLUTION: Template matching or bilateral matching and / or predictor generation use interpolation filtering. Interpolation filtering operation accesses integer sample positions within a window, and further uses padded sample values for the integer sample positions outside the window, which are based on at least one sample within the window, and uses the accessed integer sample position values as well as the padded sample values to perform the template matching or the bilateral matching and / or the predictor generation.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] Embodiments of the present invention relate to motion vector determination and refinement, as well as predictor determination for motion compensation, which may be used during video encoding and decoding. [Background technology]

[0002] Current hybrid video codecs use predictive coding. Pictures of a video sequence are subdivided into blocks of pixels, and these blocks are then coded. Instead of coding the block pixel by pixel, the entire block is predicted using already encoded pixels that are spatially or temporally neighboring the block. The encoder further processes only the difference between the block and its prediction. Further processing typically involves transforming the block pixels into coefficients in the transform domain. The coefficients are then further compressed by quantization and may be further compacted by entropy coding to form a bitstream. The bitstream also includes any signaling information that enables a decoder to decode the encoded video. For example, the signaling may include settings regarding encoder settings such as the input picture size, frame rate, quantization step instructions, or predictions applied to blocks of the picture.

[0003] Temporal prediction uses temporal correlation between pictures, also called frames of a video. Temporal prediction is also called inter-prediction because it uses dependencies between different video frames. Therefore, a block to be encoded, also called a current block, is predicted from one or more previously encoded pictures, also called reference pictures. A reference picture is not necessarily a picture preceding the current picture, but rather, in the current picture, the current block is arranged in the display order of the video sequence. An encoder may encode pictures in a coding order different from the display order. A co-located block in a reference picture may be determined as a prediction of the current block. A co-located block is a block in a reference picture that is located at the same position as the current block in the current picture. Such prediction is accurate for stationary picture regions, i.e., picture regions that do not move from one picture to another.

[0004] To obtain a motion-accounted predictor, i.e., a motion-compensated predictor, motion estimation is typically used to determine the prediction of a current block. Thus, the current block is predicted by a block in a reference picture that is located at a distance given by the motion vector from the position of the co-located block. The motion vector may be signaled in the bitstream so that the decoder can determine the same prediction for the current block. To further reduce the signaling overhead caused by signaling a motion vector for each block, the motion vector itself may be estimated. Motion vector estimation may be performed based on the motion vectors of neighboring blocks in the spatial and / or temporal domains.

[0005] The prediction of the current block may be calculated using one reference picture or by weighting predictions obtained from two or more reference pictures. Since neighboring pictures are highly likely to be similar to the current picture, the reference picture may be a neighboring picture, i.e., a picture immediately before and / or after the current picture in display order. However, in general, the reference picture may be any other picture before or after the current picture in display order and before the current picture in the bitstream (decoding order). This may be advantageous, for example, in cases where there is occlusion and / or nonlinear motion in the video content. Therefore, the identity of the reference picture may also be signaled in the bitstream.

[0006] A special mode of inter-prediction is so-called bi-prediction, in which two reference pictures are used to generate a prediction of a current block. Specifically, two predictions determined in each of the two reference pictures are combined with a prediction signal of the current block. Bi-prediction can result in a more accurate prediction of the current block than uni-prediction, i.e., prediction using only one reference picture. More accurate prediction leads to a smaller difference (also called "residual") between the pixels of the current block and the prediction, which can be encoded more efficiently, i.e., compressed into a shorter bitstream. Generally, to predict the current block, three or more reference pictures may be used to find three or more reference blocks, respectively, i.e., multi-reference inter-prediction can be applied. Therefore, the term multi-reference prediction includes bi-prediction and prediction using three or more reference pictures.

[0007] To provide more accurate motion estimation, the resolution of the reference picture may be increased by interpolating samples between pixels. Fractional pixel interpolation can be performed by weighted averaging of the nearest pixels. In the case of half-pixel resolution, for example, bilinear interpolation is typically used. Other fractional pixels are calculated as the average of the nearest pixels weighted by the inverse of the distance between each nearest pixel and the pixel to be predicted.

[0008] Motion vector estimation is a computationally complex task, in which the similarity between a current block and a corresponding predicted block (predictor) pointed to by a candidate motion vector in a reference picture is calculated. Typically, a search area contains an image of M × M samples, and each sample position of the M × M candidate positions is tested. The test involves calculating a similarity measure between an N × N reference block C and a block R located at the tested candidate position in the search area. For simplicity, the sum of absolute differences (SAD) is a method often used for this purpose, given by the following formula:

number

[0009] In the above formula, x and y define the candidate position in the search area, and the indices i and j indicate the samples in the reference block C and the candidate block R. The candidate position is often called the block displacement or offset, and is calculated as the shift of the reference block C in the search area and the similarity between the reference block C and the overlapping part of the search area. matching To reduce complexity, the number of candidate motion vectors is typically reduced by restricting them to a fixed search space. The search space may be defined, for example, by the number and / or location of pixels surrounding the location in the reference picture that corresponds to the location of the current block in the current image. After calculating the SAD for all M×M candidate locations x and y, the best candidate motion vector is found. matchingThe block R to be selected is the block with the lowest SAD and corresponds to the highest similarity with the reference block C. On the other hand, the candidate motion vectors may be defined by a list of candidate motion vectors formed by the motion vectors of neighboring blocks.

[0010] Motion vectors are usually determined at least in part at the encoder side and signaled to the decoder in the coded bitstream. However, motion vectors may also be derived at the decoder. In such cases, the current block is not available to the decoder and cannot be used to calculate the similarity to the block to which the candidate motion vector points in the reference picture. Therefore, a template is used ("template"), constructed externally from the pixels of an already decoded block, rather than the current block. matching "). For example, already decoded pixels neighboring the current block may be used. Such motion estimation has the advantage that no signaling is required since the motion vectors are derived in the same way in both the encoder and the decoder, reducing signaling. On the other hand, the accuracy of such motion estimation may be low.

[0011] In the case of bi-prediction, instead of using a template, a comparison between candidate coding blocks in different reference pictures, pointed to by candidate motion vectors, may be used (see " Bilateral Matching "), which is explained in more detail below.

[0012] To provide a trade-off between accuracy and signaling overhead, motion vector estimation may be divided into two steps: motion vector derivation and motion vector refinement. For example, motion vector derivation may involve selecting a motion vector from a candidate list. Such a selected motion vector may be further refined, for example, by searching within a search space. The search within the search space is based on calculating a cost function for each candidate motion vector, i.e., for each candidate position of the block to which the candidate motion vector points.

[0013] The paper "X. Chen, J. An, J. Zheng, JVET-D0029: Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching" (available at http: / / phenix.it-sudparis.eu / jvet / ) shows a motion vector refinement in which a first motion vector at integer pixel resolution is found and then further refined by searching at half pixel resolution in a search space around the first motion vector.

[0014] To perform the motion vector refinement, it is necessary to store in memory at least those samples that the current block needs to perform the refinement, i.e. the samples that correspond to the search space, and the templates in the search space. matching or Bilateral Matching are samples that can be accessed when

[0015] External memory access is a key design parameter in this hardware architecture and / or software implementation because it is slower than using internal memory, whereas internal memory on the chip is limited, for example, due to chip size implementation. Summary of the Invention [Means for solving the problem]

[0016] This disclosure is based on the observation that, for example, motion vector refinement when implemented in combination with fractional interpolation may require larger on-chip memory size or external memory accesses, both of which options may be undesirable. The same is true when interpolation is applied to improve the resolution of the predictor by including fractional sample positions.

[0017] In consideration of the above-mentioned problems, the present disclosure provides motion vector prediction that allows for consideration of the number of accesses to external memory and the number of samples that need to be made accessible for motion vector refinement of a motion vector for a coded block, and calculates a predictor for the block.

[0018] This is achieved by padding (substituting) values ​​located outside a predefined memory access window for the purposes of motion vector refinement based on one or more samples from within the memory access window, and / or fractional interpolation using the substitution value.

[0019] This is achieved by the features of the independent claims.

[0020] According to an aspect of the present invention, an encoding apparatus for determining a predictor for encoding a block of a video image is provided. The encoding apparatus includes a processing circuit configured to: obtain an initial motion vector for the block; obtain a refinement motion vector for the block based on the initial motion vector; and determine a predictor for the block according to the refinement motion vector. The processing circuit is configured to determine the predictor according to the refinement motion vector using interpolation with an interpolation filter, and to use at least one sample value of integer sample positions located inside a window and padded sample values ​​for integer sample positions located outside the window for interpolation filtering, the padded sample value being based on at least one sample value of samples located inside the window.

[0021] According to another aspect of the present invention, there is provided a decoding device for determining a predictor for decoding a block of a video image. The decoding device includes a processing circuit configured to: obtain an initial motion vector for the block; obtain a refinement motion vector for the prediction block based on the initial motion vector; and determine a predictor for the block according to the refinement motion vector. The processing circuit is configured to determine the predictor according to the refinement motion vector using interpolation with an interpolation filter, and to use at least one sample value of integer sample positions located inside a window and padded sample values ​​for integer sample positions located outside the window for interpolation filtering, the padded sample value being based on the at least one sample value of samples located inside the window.

[0022] According to yet another aspect of the present invention, there is provided an encoding method for determining a predictor for encoding a block of a video image, the encoding method including the steps of: obtaining an initial motion vector for the block; obtaining a refinement motion vector for the block based on the initial motion vector; and determining a predictor for the block according to the refinement motion vector, wherein the determining step determines the predictor according to the refinement motion vector using interpolation with an interpolation filter, using at least one sample value of integer sample positions positioned inside a window and padded sample values ​​for integer sample positions positioned outside the window for interpolation filtering, the padded sample value being based on the at least one sample value of the integer samples positioned inside the window.

[0023] According to yet another aspect of the present invention, there is provided a decoding method for determining a predictor for decoding a block of a video image, the decoding method including the steps of obtaining an initial motion vector for the block, obtaining a refinement motion vector for the prediction block based on the initial motion vector, and determining a predictor according to the refinement motion vector using interpolation with an interpolation filter, using at least one sample value of integer sample positions located inside a window and padded sample values ​​for integer sample positions located outside the window for interpolation filtering, the padded sample value being based on the at least one sample value of integer samples located inside the window.

[0024] Such a method and apparatus has the advantage of limiting the number of samples made available for the purpose of predictor determination by interpolation filtering, while also avoiding further accesses to a storage device / (external) memory that stores the entire reference picture.

[0025] In an embodiment of the above aspect, the window is a window for interpolation filtering.

[0026] According to an embodiment, the interpolation may include accessing sample values ​​at integer sample positions within the window and / or padding at least one sample value for an integer sample position located outside the window with a padding value based on at least one sample within the window and / or using the accessed integer position sample value and the padded sample value to perform interpolation filtering.

[0027] For example, the processing circuitry may be configured to determine a refinement of the initial motion vector based on a search space of candidate motion vectors, and to determine the refined motion vector based on the initial motion vector and the refinement.

[0028] Also, for example, the processing circuitry may be configured to determine a search space based on the initial motion vector. More specifically, the search space may be positioned at a location defined by the initial motion vector. For example, the search space may be centered around the initial motion vector. More specifically, according to an example, the search space may include nine integer sample motion vector candidates, i.e., the initial motion vector and eight surrounding motion vector candidates.

[0029] In an embodiment, the processing circuitry is further configured to reconstruct the blocks based on the predictor. The reconstruction may be performed in a decoding device and in a decoder loop of an encoding device. Aspects of the present disclosure relating to the method may include a step of reconstructing each of the blocks based on the predictor.

[0030] For example, the processing circuitry may be configured to obtain the difference between the block and the predictor, also called the “residual.” More specifically, the processing circuitry may be configured to reconstruct the block based on the predictor and the residual, and even more specifically as the sum of the predictor and the residual.

[0031] For example, the padded sample values ​​(substitution values) are determined by mirroring the values ​​of the corresponding used (evaluated) integer sample positions relative to the axes of the interpolation filter. The interpolation filtering may be one-dimensional filtering or separable two-dimensional filtering.

[0032] In one exemplary implementation, the window is formed by all integer sample positions accessible to obtain the predictor at the position specified by the initial motion vector.

[0033] The window is defined by its vertical and / or horizontal size, for example, relative to a block having the size of the block located at the initial motion vector position, the initial motion vector position, or the vertical and horizontal lengths of the interpolation filter used to determine the predictor. The definition of a block "located" at a particular position, in embodiments, refers to, for example, the upper-left sample position of the block. Alternatively, another sample position may be referenced, or the block position may be defined by being centered at a particular position. The processing circuitry may be configured to define or determine the position and / or size of the window, the size of the block, and / or the vertical and horizontal lengths of the interpolation filter used to determine the predictor according to the initial motion vector position.

[0034] As another example, the padded sample values ​​for integer sample positions outside the window are determined to be equal to the sample value of at least one used integer sample position located inside the window that is closest to the integer sample position located outside the window. In a particular example, in the case of a one-dimensional interpolation filter, the padded sample values ​​for multiple integer sample positions located outside the window are used to determine the predictor using interpolation with the interpolation filter, and the padded sample values ​​for all integer sample positions outside the window are determined to be equal to the sample value of at least one used integer sample position located inside the window that is closest to the integer sample position located outside the window.

[0035] According to an embodiment, the number of padded sample values ​​depends on the difference between the initial motion vector and the refinement motion vector.

[0036] According to an embodiment, the processing circuitry is matching or Bilateral Matching Template with matchingThe refinement of the initial motion vector is determined by the template matching or Bilateral Matching accesses sample values ​​at integer sample positions within another window, which is the search space and the template matching or Bilateral Matching and another integer sample position accessible for the template matching or Bilateral Matching further pads at least one sample value at an integer sample position located outside the other window with a padded sample (replacement) value based on at least one sample in the other window, and uses the value of the accessed integer sample position and the padded sample value to generate a template matching or Bilateral Matching Execute.

[0037] In one embodiment, the search space includes at least one fractional sample position with a value obtained by interpolation filtering using a filter of a predefined tap size, the interpolation filtering using the value of the accessed integer sample position and at least one replacement (padded sample) value to obtain the value of the at least one fractional sample position.

[0038] More specifically, the template matching or Bilateral Matching The padded sample values ​​used in are determined by mirroring the corresponding estimated integer sample position values ​​across the filter axis.

[0039] Or, a template matching or Bilateral Matching The padded sample value used in is determined to be equal to the value closest to the accessed integer sample position.

[0040] According to an embodiment, the other window is defined by its vertical size and / or horizontal size relative to at least one of the search space, a block having the size of the block located at the initial motion vector position, and the initial motion vector position.

[0041] Preferably, the other window is the same as the window.

[0042] According to an embodiment, the refinement is Bilateral Matching and the processing circuit is configured to obtain at least two initial motion vectors pointing to positions in different reference pictures and forming a pair of initial motion vectors.

[0043] According to another embodiment, the refinement is performed using a template matching The processing circuitry obtains a template for the block and computes a template using the template in the search space. matching The apparatus is further configured to determine a refinement of the initial motion vector by:

[0044] In one embodiment, the search space includes at least one fractional sample position with a value obtained by interpolation filtering using a filter of a predefined tap size, the interpolation filtering using values ​​of the used integer sample positions and at least one padded sample value to obtain a value for the at least one fractional sample position.

[0045] According to an aspect of the present invention, there is provided a non-transitory computer readable storage medium storing instructions that, when executed by a processor / processing circuitry, perform steps according to any of the above-described aspects or embodiments, or combinations thereof.

[0046] According to another specific aspect of the present invention, there are provided encoding devices for encoding a video image divided into predictive blocks into a bitstream or encoding a predictive block of a video image into a bitstream. The encoding device includes a processing circuit for determining a motion vector for the predictive block. The processing circuit is configured to obtain at least one initial motion vector and determine a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refined motion vector. The search space is located at a position given by the initial motion vector. The encoding device further includes an encoding circuit (which may be included in the processing circuit) for encoding a difference between the predictive block and a predictor given by the predictive block at a position based on the determined motion vector, and generating a bitstream including the encoded difference and the initial motion vector. The encoding device further includes a motion prediction unit (which may be included in the processing circuit) for determining a predictor according to the motion vector using interpolation filtering with an interpolation filter. The interpolation filtering accesses sample values ​​at integer sample positions within a predefined window, replaces at least one sample value at integer sample positions located outside the window with a replacement value based on at least one sample within the window, and performs interpolation filtering using the accessed integer sample position value and the replacement value.

[0047] According to yet another specific aspect of the present invention, a decoding device is provided for decoding a video image divided into prediction blocks from a bitstream. The decoding device includes an analysis unit (which may be included in a processing circuit of the decoding device) for analyzing, from the bitstream, an initial motion vector and an encoded difference between the prediction block and a predictor provided by the prediction block at a position specified by the refinement motion vector. The decoding device includes a processing circuit (e.g., the processing circuit of the decoding device) for determining a motion vector for the prediction block. The processing circuit is configured to obtain at least one initial motion vector and determine a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refinement motion vector. The search space is located at the position specified by the initial motion vector. Furthermore, the decoding device includes a decoding circuit (which may be included in the processing circuit of the decoding device) for reconstructing the prediction block as the sum of the analyzed difference and the predictor provided by the prediction block at the position specified by the refinement motion vector determined by the processing circuit to determine the motion vector. The decoding circuit is further configured to determine the predictor according to the motion vector using interpolation filtering using an interpolation filter. The interpolation filtering accesses sample values ​​at integer sample positions within a predefined window, replaces at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within the window, and performs the interpolation filtering using the accessed integer position sample values ​​and the replacement value.

[0048] According to yet another aspect of the present invention, there is provided an encoding method for encoding a video image divided into predictive blocks into a bitstream. The encoding method includes determining a motion vector for the predictive block. The determining the motion vector includes obtaining at least one initial motion vector and determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refined motion vector. The search space is located at a position given by the initial motion vector. The encoding method further includes encoding a difference between the predictive block and a predictor given by the predictive block at a position based on the determined motion vector, and generating a bitstream including the encoded difference and the initial motion vector. The method further includes determining a predictor according to the motion vector using interpolation filtering with an interpolation filter. The interpolation filtering accesses sample values ​​at integer sample positions within a predefined window, replaces at least one sample value at integer sample positions located outside the window with a replacement value based on at least one sample within the window, and performs interpolation filtering using the accessed integer sample position and the replacement value.

[0049] According to yet another aspect of the present invention, a decoding method for decoding a video image divided into predictive blocks from a bitstream is provided. The decoding method includes analyzing, from the bitstream, an initial motion vector and an encoded difference between the predictive block and a predictor provided by the predictive block at a position specified by the refinement motion vector. The method further includes determining a motion vector for the predictive block. The determining a motion vector includes obtaining at least one initial motion vector and determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refinement motion vector. The search space is located at the position specified by the initial motion vector. The decoding method further includes reconstructing the predictive block as the sum of the analyzed difference and the predictor provided by the predictive block at the position specified by the refinement motion vector determined by the determining a motion vector. Furthermore, the method includes determining the predictor according to the motion vector using interpolation filtering with an interpolation filter. The interpolation filtering accesses sample values ​​at integer sample positions within a predefined window, replaces at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within the window, and performs the interpolation filtering using the accessed integer position sample values ​​and the replacement value.

[0050] Further advantages and embodiments of the invention are set forth in the dependent claims.

[0051] Exemplary embodiments will now be described in more detail with reference to the accompanying drawings and figures. [Brief explanation of the drawings]

[0052] [Figure 1] FIG. 2 is a block diagram illustrating an example structure of an encoder in which motion vector derivation and refinement may be used. [Figure 2]FIG. 2 is a block diagram illustrating an example structure of a decoder in which motion vector derivation and refinement can be used. [Figure 3] FIG. 1 is a schematic diagram illustrating exemplary template matching suitable for bi-prediction. [Figure 4] FIG. 1 is a schematic diagram illustrating exemplary template matching suitable for single and bi-prediction. [Figure 5] FIG. 10 is a block diagram illustrating a stage of motion vector derivation that operates without providing an initial motion vector that is refined in the bitstream. [Figure 6] FIG. 1 is a schematic diagram illustrating the concept of bilateral matching. [Figure 7] FIG. 2 is a block diagram illustrating exemplary hardware for implementing embodiments of the present invention. [Figure 8] 1 is a schematic diagram illustrating an exemplary window containing a coding block and samples that should be accessible; [Figure 9] FIG. 1 is a schematic diagram illustrating an iterative search space. [Figure 10] FIG. 1 is a schematic diagram illustrating horizontal expansion of a memory access window through interpolation filtering. [Figure 11] FIG. 1 illustrates a conventional approach to perform motion vector refinement based on a search space and obtain predictions by interpolation. [Figure 12] 12 is a flowchart showing motion vector derivation and prediction acquisition according to the conventional example of FIG. 11. [Figure 13] FIG. 10 illustrates a technique for performing motion vector refinement based on a search space and obtaining predictions by interpolation in accordance with an exemplary embodiment of the present invention. [Figure 14] FIG. 10 is a schematic diagram illustrating padding of two samples to allow fractional position calculations. [Figure 15] FIG. 10 is a schematic diagram illustrating padding of four samples to allow template matching at the search space boundary. [Figure 16] FIG. 10 is a schematic diagram illustrating an example fractional position where interpolation does not require padding. [Figure 17] 14 is a flow diagram illustrating motion vector refinement and prediction acquisition according to the exemplary embodiment of FIG. 13. [Figure 18] FIG. 1 is a schematic diagram illustrating the definition of a memory access window for a prediction unit. [Figure 19] FIG. 1 illustrates a method according to an embodiment. [Figure 20] FIG. 10 illustrates a technique for performing motion vector refinement based on a search space and obtaining predictions by interpolation, according to another exemplary embodiment of the present invention. [Figure 21] 21 is a flow diagram illustrating motion vector refinement and prediction acquisition according to the exemplary embodiment of FIG. 20. DETAILED DESCRIPTION OF THE INVENTION

[0053] This disclosure relates to adjusting the number of samples made accessible to perform motion vector refinement and interpolation in order to obtain fractional positions within a reference picture and a predictor.

[0054] As mentioned above, external memory access is one of the most important design considerations in today's hardware and software architectures. Motion vector estimation is especially important for template matching or Bilateral Matching , may be used in conjunction with interpolation filtering to obtain fractional positions in the search space, for example in the case of motion vector refinement. Using interpolation filtering may require an increased number of samples to be accessed from memory. However, this may lead to an increase in expensive on-chip memory or an increase in the number of external memory accesses, or on the other hand, slow down the implementation. Especially on the decoder side, such issues may lead to more expensive or slower operation, which is undesirable.

[0055] To avoid such situations, the present disclosure provides for restricting external memory access to a predefined window.

[0056] The window may be of a predefined size. For example, in motion vector refinement, the window may include at least all search space positions. The samples accessed are based on the template matching or Bilateral Matching The actual samples used for a particular block and its search space to perform the cost function using a template, or Bilateral All samples used to calculate the cost function. Accessible samples are those in a particular template for a particular block. matching or Bilateral Matching It may further include samples that are not necessarily accessed in the template for that sample, for example because the search space has been iteratively determined to include different samples. matching or Bilateral Matching However, this does not require all accessible pixels. The accessible samples further include samples that may come from any iteratively determined search space (i.e., given any block content), and possibly even a template in such a possible search space. matching The sample further includes a sample that is accessed to perform the steps.

[0057] When interpolation filtering is used to generate the final prediction, the extent to which the memory access window needs to be expanded in conventional techniques depends on the maximum difference between the determined refinement motion vector and the initial motion vector.

[0058] It should be noted that the window may be defined to be smaller than all accessible samples. Defining the window in this manner allows for keeping the number of samples stored in on-chip memory and retrieved from external memory smaller. Thus, the present disclosure provides a template for a particular block. matching or Bilateral Matching This does not mean that other samples are not usually present / available in another memory or storage device. As mentioned above, the entire reference picture can usually be stored in external memory. However, the template matching or Bilateral Matching For the template, only a part of it, i.e. the sample in the window, is loaded. matching may be used for

[0059] By providing such a limited window, it is possible to implement it and actually only need to load the window samples. To ensure that the encoder and decoder behave similarly, the window definition may be predefined normally or may be signaled in the bitstream. In particular, if refinement motion vectors are used to form the reference pictures, both the encoder and the decoder must use the same technique.

[0060] FIG. 1 illustrates an encoder 100 having an input for receiving input image samples of a frame or picture of a video stream and an output for generating an encoded video bitstream. The term "frame" in this disclosure is used as a synonym for picture. However, it should be noted that this disclosure is also applicable to fields where interlacing may be applied. Generally, a picture includes m×n pixels, which correspond to image samples, each of which may include one or more color components. For simplicity, the following description refers to pixels as luminance samples. However, it should be noted that the motion vector search of the present invention can be applied to any color component, including chrominance, or to components of a search space such as RGB. On the other hand, it may be beneficial to perform motion vector estimation only for one component and apply the determined motion vector to more (or all) components.

[0061] The input blocks to be coded do not necessarily have the same size: a picture may contain blocks of different sizes, and the block rasters of different pictures may also be different.

[0062] In an illustrative implementation, encoder 100 is configured to apply prediction, transform, quantization, and entropy coding to a video stream, which are performed by transform unit 106, quantization unit 108, and entropy encoding unit 170, respectively, to produce an encoded video bitstream as output.

[0063] A video stream may include multiple frames, each divided into blocks of a specific size that are either intra-coded or inter-coded. For example, blocks of the first frame of the video stream are intra-coded by the intra prediction unit 154. Because intra frames are coded using only information within the same frame, they can be independently decoded and provide entry points within the bitstream for random access. Blocks of other frames of the video stream may be inter-coded by the inter prediction unit 144, where information from previously coded frames (reference frames) is used to reduce temporal redundancy, such that each block of the inter-coded frame is predicted from a block in the reference frame. The mode selection unit 160 is configured to select whether the blocks of a frame are processed by the intra prediction unit 154 or the inter prediction unit 144. This mode selection unit 160 also controls parameters of the intra prediction or inter prediction. To enable refreshing of image information, intra-coded blocks may be provided within inter-coded frames. Additionally, intra frames containing only intra-coded blocks may be inserted periodically into a video sequence to provide entry points for decoding, i.e., points at which a decoder can begin decoding without having information from previously coded frames.

[0064] The intra estimation unit 152 and the intra prediction unit 154 are units that perform intra prediction. In particular, the intra estimation unit 152 may further derive a prediction mode based on knowledge of the original image, and the intra prediction unit 154 provides a corresponding predictor for differential coding, i.e., a sample predicted using the selected prediction mode. To perform spatial or temporal prediction, the coded block may be further processed by the inverse quantization unit 110 and the inverse transform unit 112. After the block reconstruction, the loop filtering unit 120 is applied to further improve the quality of the decoded image. Next, the filtered block forms a reference frame, which is then stored in the decoded picture buffer 130. Such a decoder loop (decoder) on the encoder side provides the advantage of generating the same reference frame as the reference picture reconstructed on the decoder side. Therefore, the encoder side and the decoder side operate in a corresponding manner. The term "reconstruction" here refers to obtaining a reconstructed block by adding a predictive block to a decoded residual block.

[0065] The inter estimation unit 142 receives as input a block of the current frame or picture to be inter-coded and one or several reference frames from the decoded picture buffer 130. Motion estimation is performed by the inter estimation unit 142, and motion compensation is applied by the inter prediction unit 144. Motion estimation is used, for example, to obtain a motion vector and a reference frame based on a specific cost function, also using the original image to be coded. For example, the motion estimation unit 142 may provide an initial motion vector estimate. The initial motion vector may then be signaled in the bitstream directly in the form of a vector, or as an index referencing a motion vector candidate in a candidate list built based on a predefined rule, in the same way as in an encoder and decoder. Motion compensation then derives a predictor for the current block by transforming a block co-located with the current block in the reference frame to the reference block in the reference frame, i.e., by a motion vector. The inter prediction unit 144 outputs a prediction block for the current block, which minimizes the cost function. For example, the cost function may be the difference between the current block to be coded and its predicted block, i.e., the cost function minimizes the residual block. The minimization of the residual block is based, for example, on calculating the sum of absolute differences (SAD) of all pixels (samples) of the current block and a candidate block in a candidate reference picture. However, in general, other similarity metrics may be used, such as the mean squared error (MSE) or a structural similarity metric (SSIM).

[0066] However, the cost function may also be the number of bits required to encode such inter-blocks and / or the distortion resulting from such encoding. Thus, a rate-distortion optimization procedure may be used to determine the encoding parameters, such as the selection of motion vectors and / or generally whether to use inter- or intra-prediction for a block and its settings.

[0067] The intra estimation unit 152 and the intra prediction unit 154 receive as input a block of the current frame or picture to be intra-coded and one or several reference samples from already reconstructed regions of the current frame. Intra prediction then describes the pixels of the current block of the current frame in terms of a function of the reference samples of the current frame. The intra prediction unit 154 outputs a prediction block of the current block, which preferably minimizes the difference between the current block to be coded and its prediction block, i.e., it minimizes the residual block. The minimization of the residual block may be based, for example, on a rate-distortion optimization procedure. In particular, the prediction block is obtained as a directional interpolation of the reference samples. The direction may be determined by rate-distortion optimization and / or by calculating a similarity measure as mentioned above in connection with inter prediction.

[0068] The inter estimation unit 142 receives as input a block, or more generally, an image sample of the current frame or picture to be inter-coded, and two or more already decoded pictures 231. Inter prediction then describes the current image sample of the current frame in terms of a motion vector to refer to an image sample of a reference picture. The inter prediction unit 142 outputs one or more motion vectors for the current image sample, and the reference image sample pointed to by the motion vector preferably minimizes the difference between the current image sample to be coded and the reference image sample, i.e., it minimizes the residual image sample. A predictor for the current block is then provided by the inter prediction unit 144 for differential coding.

[0069] The difference between the current block and its prediction, i.e., the residual block 105, is then transformed by the transform unit 106. The transform coefficients 107 are quantized by the quantization unit 108 and entropy coded by the entropy encoding unit 170. The encoded picture data 171, i.e., the encoded video bitstream, thus generated includes intra-coded and inter-coded blocks and corresponding signaling (such as a mode indication, a motion vector indication, and / or an intra-prediction direction). The transform unit 106 may apply a linear transform such as a Fourier or discrete cosine transform (DFT / FFT or DCT). Such a transformation to the spatial frequency domain offers the advantage that the resulting coefficients 107 typically have higher values ​​at lower frequencies. Thus, after effective coefficient scanning (such as zigzag) and quantization, the resulting sequence of values ​​typically has a few large values ​​at the beginning and ends with a run of zeros. This allows for more efficient coding. The quantization unit 108 performs a virtual lossy compression by reducing the resolution of the coefficient values. An entropy coding unit 170 then assigns binary codewords to the coefficient values ​​to generate a bitstream. The entropy coding unit 170 also encodes signaling information (not shown in FIG. 1).

[0070] 2 shows a video decoder 200. The video decoder 200 includes, among other things, a decoded picture buffer 230, an inter-prediction unit 244, and an intra-prediction unit 254, which is a block prediction unit. The decoded picture buffer 230 is configured to store at least one (for uni-prediction) or at least two (for bi-prediction) reference frames reconstructed from an encoded video bitstream, the reference frames being different from a current frame (currently being decoded) of the encoded video bitstream. The intra-prediction unit 254 is configured to generate a prediction block, which is an estimate of a block to be decoded. The intra-prediction unit 254 is configured to generate this prediction based on reference samples obtained from the decoded picture buffer 230.

[0071] Decoder 200 is configured to decode the encoded video bitstream generated by video encoder 100, and preferably both decoder 200 and encoder 100 generate identical predictions for each block being encoded / decoded. The features of decoded picture buffer 230, reconstructed blocks 215, buffer 216, and intra prediction unit 254 are similar to the features of decoded picture buffer 130, reconstructed blocks 115, buffer 116, and intra prediction unit 154 of FIG. 1.

[0072] The video decoder 200 includes further units that are also present in the video encoder 100, such as an inverse quantization unit 210, an inverse transform unit 212, and a loop filtering unit 220, which correspond to the inverse quantization unit 110, the inverse transform unit 112, and the loop filtering unit 120 of the video encoder 100, respectively.

[0073] The entropy decoding unit 204 is configured to decode the received encoded video bitstream and correspondingly obtain quantized residual transform coefficients 209 and signaling information. The quantized residual transform coefficients 209 are provided to an inverse quantization unit 210 and an inverse transform unit 212 to generate a residual block. The residual block is added to a prediction block 265, which is provided to a loop filtering unit 220 to obtain decoded video. Frames of the decoded video are stored in a decoded picture buffer 230 and may serve as decoded pictures 231 for inter prediction.

[0074] In general, the intra prediction units 154 and 254 in Figures 1 and 2 may use reference samples from already encoded regions to generate a prediction signal for a block that needs to be encoded or decoded.

[0075] The entropy decoding unit 204 receives the encoded bitstream 171 as its input. Generally, the bitstream is first parsed, i.e., signaling parameters and residuals are extracted from the bitstream. Typically, the syntax and semantics of the bitstream are defined in a standard so that encoders and decoders can operate in an interoperable manner. As explained in the Background section above, the encoded bitstream does not only contain prediction residuals. In the case of motion-compensated prediction, motion vector indications are also coded in the bitstream and parsed from there by the decoder. The motion vector indications may be given by motion vector coordinates, depending on the reference picture to which the motion vectors are provided. Up to this point, we have considered coding of complete motion vectors. However, it is also possible to encode only the difference between the current motion vector and the previous motion vector in the bitstream. This approach allows for the redundancy between motion vectors of neighboring blocks to be exploited.

[0076] To efficiently encode reference pictures, the H.265 codec (ITU-T, H265, Series H: Audiovisual and multimedia systems: High Efficient Video Coding) provides a list of reference pictures that assigns each reference frame to a list index. Reference frames are then signaled in the bitstream by including their corresponding assigned list indexes. Such lists may be defined standardly or signaled at the beginning of a video or a set of several frames. Note that in H.265, there are two lists of defined reference pictures, called L0 and L1. Reference pictures are then signaled in the bitstream by indicating the list (L0 or L1) and the index within that list associated with the desired reference picture. Providing more than one list may be advantageous for better compression. For example, L0 may be used for both unidirectionally inter-predicted slices and bidirectionally inter-predicted slices, while L1 may be used only for bidirectionally inter-predicted slices. However, the present disclosure is not generally limited to any content of the L0 and L1 lists.

[0077] Lists L0 and L1 may be standardly defined and fixed. However, by signaling them at the beginning of the video sequence, greater flexibility in encoding / decoding may be achieved. Thus, the encoder may construct lists L0 and L1 with specific reference pictures ordered according to their indices. The lists L0 and L1 may have the same fixed size. In general, there may be more than two lists. Motion vectors may be signaled directly by their coordinates in the reference pictures. Alternatively, as specified in H.265, a list of candidate motion vectors may be constructed and the index associated with a specific motion vector in the list may be transmitted.

[0078] The motion vector of a current block is usually correlated with the motion vectors of neighboring blocks in the current picture or in previously coded pictures. This is because neighboring blocks are likely to correspond to the same moving object with similar motion, and the object's motion is unlikely to change rapidly over time. Consequently, using motion vectors in neighboring blocks as predictors reduces the magnitude of the signaled motion vector difference. Motion vector predictors (MVPs) are typically derived from previously encoded / decoded motion vectors from spatially neighboring blocks or from temporally neighboring or co-located blocks in a reference picture. In H.264 / AVC, this is done by taking the component-wise median of three spatially neighboring motion vectors. Using this approach, predictor signaling is not required. Temporal MVPs from co-located blocks in a reference picture are only considered in the so-called temporal direct mode of H.264 / AVC. H.264 / AVC direct mode can also be used to derive other motion data besides motion vectors. Therefore, they are more closely related to the concept of block merging in HEVC. In HEVC, the method of implicitly deriving an MVP was replaced by a technique known as motion vector competition, which explicitly signals which MVP from a list of MVPs is used for motion vector derivation. The variable coding quadtree block structure in HEVC can result in a block having several neighboring blocks that contain motion vectors as potential MVP candidates. Taking the left neighbor as an example, in the worst case, if a 64x64 luma coding tree block is not further divided and the left block is divided to the maximum depth, a 64x64 luma prediction block may have 16 4x4 luma prediction blocks to its left.

[0079] To modify motion vector conflicts to account for such flexible block structures, Advanced Motion Vector Prediction (AMVP) was introduced. During the development of HEVC, the initial AMVP design was significantly simplified to provide a good tradeoff between coding efficiency and ease of implementation. The initial design of AMVP included five MVPs from three different classes of predictors: three motion vectors from spatial neighbors, the median of the three spatial predictors, and a scaled motion vector from a co-located, temporally neighboring block. The predictor list was further reordered to place the most likely motion predictor in the first position and to remove redundant candidates to ensure minimal signaling overhead. The final design of the AMVP candidate list structure included the following two MVP candidates: a) up to two spatial candidate MVPs derived from five spatially neighboring blocks; b) one temporal candidate MVP derived from two temporal, co-located blocks if both spatial candidate MVPs are unavailable or if they are identical; and c) a zero motion vector if the spatial candidate, the temporal candidate, or both candidates are unavailable. Further details regarding motion vector determination can be found in the publication "V. Sze et. al. (Ed.), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014", particularly Chapter 5, which is incorporated herein by reference.

[0080] To further improve motion vector estimation without further increasing signaling overhead, it would be beneficial to further refine the motion vectors derived at the encoder side and provided in the bitstream. Motion vector refinement may be performed at the decoder without assistance from the encoder. The encoder in its decoder loop may use the same refinement to obtain the corresponding motion vector. Motion vector refinement is performed within a search space, which includes integer and fractional pixel positions of a reference picture. For example, the fractional pixel positions may be half-pixel or quarter-pixel positions, or another fractional position. The fractional pixel positions may be obtained from the integer (full-pixel) positions by interpolation, such as bilinear interpolation. However, the present disclosure is equally applicable to search spaces that do not include fractional pixel positions.

[0081] In bi-prediction of the current block, two prediction blocks obtained using the first motion vector in list L0 and the second motion vector in list L1, respectively, are combined into a single prediction signal, which allows for better adaptation to the original signal than uni-prediction, potentially resulting in less residual information and more efficient compression.

[0082] At the decoder, a template is used for motion vector refinement since the current block is not available during decoding; the template is an estimate of the current block and is constructed based on the image parts that have already been processed (i.e. coded at the encoder side and decoded at the decoder side).

[0083] Initially, an estimate of a first motion vector MV0 and an estimate of a second motion vector MV1 are received as input at the decoder 200. At the encoder side 100, the motion vector estimates MV0 and MV1 are matching, and / or by searching in a list of candidates (such as a merge list) formed by the motion vectors of blocks neighboring the current block (in the same picture or in neighboring pictures). MV0 and MV1 are then advantageously signaled to the decoder side in the bitstream. However, in general, the first decision step at the encoder is also based on the template matching Note that this may be done by:

[0084] At the decoder side 200, the motion vectors MV0 and MV1 are advantageously obtained based on information in the bitstream. MV0 and MV1 are either directly signaled or differentially signaled and / or their indices in a list of motion vectors (merge list) are signaled. However, the present disclosure is not limited to signaling the motion vectors in the bitstream. Rather, the motion vectors correspond to the operation of the encoder and are already obtained in the first stage from the templates. matching The first stage (motion vector derivation) template may be determined by matching may be performed based on a search space that is different from the search space of the second refinement stage. Specifically, refinement may be performed in a search space with higher resolution (i.e., shorter distances between search positions).

[0085] Indications of the two reference pictures RefPic0 and RefPic1 pointed to by MV0 and MV1, respectively, are also provided to the decoder. The reference pictures are stored in decoded picture buffers at both the encoder and decoder sides as a result of previous processing, i.e., encoding and decoding, respectively. One of these reference pictures is selected for motion vector refinement by searching. The reference picture selection units of the encoder and decoder are configured to select the first reference picture pointed to by MV0 and the second reference picture pointed to by MV1, respectively. Following the selection, the reference picture selection units determine whether the first reference picture or the second reference picture will be used to perform motion vector refinement. To perform motion vector refinement, a search region in the first reference picture is defined around the candidate position pointed to by motion vector MV0. The candidate search space positions within the search region are then searched for by a template within the search space. matching and then analyze to find the block that is most similar to the template block by determining a similarity metric such as the sum of absolute differences (SAD). The location in the search space represents the location where the top left corner of the template matches. As already mentioned above, the top left corner is just a convention, matching Any point in the search space, such as the center point, may generally be used to represent the location.

[0086] According to the above-mentioned document JVET-D0029, the decoder-side motion vector refinement (DMVR) has as input initial motion vectors MV0 and MV1, which point to two reference pictures RefPict0 and RefPict1, respectively. These initial motion vectors are used to determine the search spaces of RefPict0 and RefPict1. Furthermore, using the motion vectors MV0 and MV1, a template is constructed based on each block A and B (of samples) pointed to by MV0 and MV1, as follows: template = function((block A, block B))

[0087] The function may be a sample clipping operation combined with a weighted sum over the samples. The template is then calculated using a search space determined based on MV0 and MV1 in each reference picture 0 and 1. matching The best template in each search space is used to perform matching The cost function for determining the template is SAD(Template, Block candA'), where block candA' is the candidate coding block pointed to by the candidate MV in the search space spanned by the position given by MV0. matching 3 shows the determination of the block A' to be processed and the resulting refined motion vector MV0'. matching The same template is used to find block B' that corresponds to RefPic0 and the corresponding motion vector MV1' that points to block B'. ​​In other words, after a template is constructed based on blocks A and B pointed to by initial motion vectors MV0 and MV1, refinement motion vectors MV0' and MV1' are found by searching for RefPic0 and RefPic1 in the template.

[0088] The motion vector derivation technique may also be called frame rate up-conversion (FRUC). Initial motion vectors MV0 and MV1 are generally indicated in the bitstream to ensure that the encoder and decoder use the same initial point for motion vector refinement. Alternatively, the initial motion vector may be obtained by providing a list of initial candidates including one or more initial candidates. A refinement motion vector is determined for each of them, and finally, the refinement motion vector that minimizes a cost function is selected.

[0089] The present invention uses the template described above with reference to FIG. matching It is further noted that the present invention is not limited to the above. Figure 4 shows an alternative template that is also applicable to single prediction. matchingThis figure shows the details of the template. More details can be found in the document JVET-A1001, in particular "Section 2. 4. 6. Pattern matched motion vector derivation" of the document JVET-A1001 by Jianle Chen et al., entitled "Algorithm Description of Joint Exploration Test Model 1" and can be accessed at http: / / phenix.it-sudparis.eu / jvet / . matching The template in our method is determined as the samples adjacent to the current block in the current frame. As shown in Figure 1, we may take the already reconstructed samples adjacent to the top and left boundaries of the current block, which is called an "L-shaped template."

[0090] Figure 5 shows another type of motion vector derivation that may also be used. The input to the motion vector derivation process is a flag indicating whether motion vector derivation is applied. Implicitly, another input to the derivation process is the motion vector of a neighboring (temporally or spatially) previously coded / reconstructed block. The motion vectors of several neighboring blocks are used as candidates for the initial search step of the motion vector derivation. The outputs of the process are MV0' (possibly MV1' if bi-prediction is used) and the corresponding reference picture index refPict0 and possibly refPict1, respectively. The motion vector refinement stage then uses the template MV0' as described above. matching After finding one (uni-predictive) or two or more (bi-predictive / multi-frame predictive) refined motion vectors, a predictor for the current block (for bi- / multi-predictive) is constructed (by weighted sample prediction or otherwise by referring to the samples pointed to by the refined MVs).

[0091] The present invention is based on the above two matching How to (template matching Examples include, but are not limited to: Bilateral Matching A third called matching The method (also described in document JVET-A1001) may be used for motion vector refinement, and the invention applies as well. Bilateral Matching is explained with reference to FIG.

[0092] Bilateral Matching According to the method, the best fit between two blocks along the motion trajectory of the current block in two different reference pictures is matching As shown in Figure 6, Bilateral Matching are the closest two blocks along the motion trajectory of the current block in two different reference pictures. matching is used to derive the motion information of the current block by finding Bilateral Matching In the example, where MV0 points to Block cand0' and MV1 points to Block cand1', a cost function such as SAD(Block cand0', Block cand1') may be used.

[0093] In order to reduce the computational cost, a cost function is calculated and compared for the selection of suitable candidates (candidate pairs) of motion vectors, which may be meaningful to consider to satisfy the following condition: Assuming a continuous motion trajectory, the motion vectors MV0 and MV1 pointing to two reference blocks shall be proportional to the temporal distances between the current picture and the two reference pictures, i.e., TD0 and TD1. As a special case, when the current picture is temporally between two reference pictures and the temporal distances from the current picture to the two reference pictures are the same, the motion vector differences between the motion vectors of multiple pairs referring to one reference picture and each motion vector of the pair referring to another reference picture shall mirror each other.

[0094] 7 illustrates a processing circuit 600. The processing circuit may include any hardware, the configuration of which may be implemented by any type of programming, or a hardware design that combines both. For example, the processing circuit may be formed by a single processor, such as a general-purpose processor that includes corresponding software that performs the steps described above. Alternatively, the processing circuit may be implemented by specialized hardware, such as a DSP (digital signal processor), ASIC (application-specific integrated circuit), or FPGA (field-programmable gate array).

[0095] The processing circuitry may include one or more of the hardware components described above interconnected to perform the motion vector derivation described above. The processing circuitry 600 receives an initial motion vector (or multiple initial motion vectors if bi-prediction / multi-prediction is used), and a template matching The processing circuit 600 includes computational logic for performing two functions: obtaining a template if a template is used 610, and motion vector refinement 620. These two functions may be implemented in the same hardware or may be performed in separate units of hardware, such as the initial motion vector and template determination unit 610 and the motion vector refinement unit 620. The processing circuit 600 may be communicatively connected to an external memory 650 in which reconstructed reference picture samples are stored. Furthermore, the processing circuit 600 may further include an internal memory 640 for buffering samples within a window that are transferred from the external memory and used for motion vector determination for the currently processed block. The processing circuit may be embodied as an integrated circuit on a single chip.

[0096] Note that the processing circuit may perform other functions of the encoder and / or decoder described with reference to FIGS. 1 and 2. The internal memory may be an on-chip memory, such as a cache or line memory. The chip memory is preferably implemented on the encoder / decoder chip to speed up calculations. Due to the limited size of the chip, the on-chip memory is typically small. On the other hand, the external memory can be very large, but accessing it consumes more energy and is significantly slower. Typically, all of the necessary information is retrieved from the external memory into the on-chip memory before the calculation is performed. The worst-case external memory access (or the bandwidth that must be provided when designing the memory bus) represents the maximum amount of memory transfer that can occur between the external memory and the chip during decoding of a frame or coding unit. Memory (especially external memory) can typically only be accessed within predefined block units. In other words, single pixels are typically not accessible and must be accessed in the smallest unit (e.g., 8x8). The size of the on-chip memory is also an important design consideration because larger on-chip memories are more expensive.

[0097] In other words, the above-mentioned device may be an integrated circuit further comprising an internal memory embedded within the integrated circuit and a memory access unit (interface) for fetching integer samples arranged within the window from an external memory into the internal memory.

[0098] The term "prediction block" used above refers to the current block to be predicted. It is a block in an image that can be obtained by subdividing the image into equal or different sizes (e.g., by hierarchical partitioning of coding tree units into smaller unit blocks of CTUs). The block may be square, or more generally rectangular, since this is a typical shape also used in current encoders / decoders. However, the present disclosure is not limited by any size / shape of the block.

[0099] The device comprising the processing circuitry may be an encoder or decoder, or may be a device comprising such an encoder or decoder, for example a recording device and / or a playback device.

[0100] Fractional sample positions are positions between actual picture sample positions obtained by reconstructing the coded reference picture as shown in Figure 1. Therefore, fractional positions must be obtained by interpolation based on the nearest integer position. Details of interpolation filtering can be found in "Section 5. 3 Fractional Sample Interpolation" in the literature "V. Sze et. al., High Efficiency Video Coding (HEVC)," Springer, 2014.

[0101] Interpolation filtering is typically applied to another filter to generate another fractional pel (sample) position. As an example, the following 1D separable filter is applied to generate 1 / 4 pel and 1 / 2 pel positions in the H.265 video compression standard:

[0102] [Table 1]

[0103] As can be seen in the table above, interpolation filtering requires a number of samples around the fractional pel position, corresponding to the filter taps (the number of coefficients in the table). Using the example filter described above to generate a 1 / 2 pel position requires four integer samples from the left / top and right / bottom. Note that the length of the interpolation filter is different for a 1 / 4 pel sample position (7 taps) than for a 1 / 2 pel sample position (8 taps). The terms sample, pel, and pixel are used interchangeably herein to refer to an image sample (in one or more color components) at a given time instance. A sample is essentially an intensity value.

[0104] In some embodiments of the present invention, an interpolation filter with a predefined tap size is used to generate a template in said search space. matching The integer samples are evaluated only within a window given by the integer samples accessible for . The window may contain many more samples than are actually used in the calculation of a given prediction block. This is because the refinement search operation is usually performed using a fast search method (as opposed to a brute force search method), which means that some samples are not evaluated as the search operation progresses. As a result, the template matching The number of iterations of may vary for each prediction block, as may the samples used in the calculation for the refinement search operation.

[0105] Figure 8 shows a coding block (prediction block) and a corresponding sample of a window. Note that the sample shown in Figure 8 is a reference picture sample, and the coding block here is actually a block whose size and position correspond to the current block in the current frame, for which a motion vector is derived in the reference picture. Therefore, the coding block in Figure 8 is actually a block that is essentially co-located with the block for which the predictor is searched. However, for the sake of brevity, this block will be referred to as the "coding block" hereinafter.

[0106] In this example, the unrefined motion vector MV0 points to an integer sample position. The search granularity of the motion vector refinement is 1 integer sample, which means that only integer sample points are searched since the starting point is an integer sample. In this example, the search is performed in a gradually evolving search space, which means that at each iteration the search space advances by adding a new search position according to the best direction in terms of a cost function of previously tested positions.

[0107] Such an approach is illustrated in a simplified manner in FIG. 9 , where an initial motion vector points to a central point 810. A search space is gradually constructed around the initial motion vector location. In a first step, four adjacent locations immediately above, below, to the left, and to the right of the location 810 pointed to by the initial motion vector are tested, along with the location 810 pointed to by the initial motion vector. Based on the direction in which the cost function is lowest among the five tested points, another location to be tested is added to the search space. In this example, the lowest cost function was found at the right point, so in a second step, the search space was expanded horizontally three more points to the right. In the second step, the lowest cost function was found at a point to the right (relative to the lowest cost point from the first step), resulting in an additional horizontal expansion of the search space three points to the right. In a third step, the lowest cost function was again found at a point to the right of the lowest cost point from step 2, resulting in an additional horizontal expansion of the search space three points to the right. According to the example in Figure 9, three more steps are performed in the order up, up, and right. In the example, for each iteration, a diamond pattern (of five search points) is used, and the search space is expanded at each step to complete the missing search points.

[0108] At each iteration of the search space determination, the search space may increase by one or more integer sample positions. Returning now to FIG. 8, in this example, the maximum number of search iterations is four. Since a maximum of four iterations is possible, all samples shown on the left must be fetched from memory to perform the search operation, and in this case, the search space expansion proceeds gradually to the left. Similarly, four samples must be expanded upward. Therefore, since the refined MV can move in either direction, the search space expands in both directions (left to right and top to bottom), and a hardware implementation requires that all samples that may be needed be fetched from external memory before the refinement search is applied. If the search space expands downward or to the right, a template is used to calculate the search space using a template corresponding to the size of the coding block (prediction block). matchingHowever, some of these samples will need to be accessed, so it will need to be expanded by another four samples. Furthermore, hardware implementations usually cannot accommodate irregular shapes (rectangular accesses are more feasible), so samples in the corners (e.g., the top right) will also need to be fetched from memory.

[0109] It should be noted that the iterative search space expansion described above is merely an example, and the rules and number of points for expanding the search space in each iteration may be different, i.e., specified differently.

[0110] Figure 9 also illustrates a scenario that can arise due to the external memory access rules described above. The number of samples fetched from external memory is much larger than the samples actually used in a computation step. If we assume (for simplicity) that our template is only one sample large, the open circles represent samples fetched from external memory, while the shaded samples represent those actually used. However, this redundancy is necessary if the number of accesses to external memory needs to be kept low, since the actual required samples are not yet known when processing of the current block begins.

[0111] It should be noted that the search space may be defined in other ways, for example as a stable shape located at the position pointed to by the initial motion vector. The shape may be any shape, such as a square, rectangle, diamond, etc.

[0112] Figure 10 shows an example where the search space can further include fractional samples. In Figures 8 and 9, the motion vector search was performed on integer samples, resulting in the location indicated by the large solid dot in the access window. Here, the search is performed on samples with 1 / 2 pel resolution (the smaller solid dot), and to generate the fractional sample shown on the left, three more rows of samples must be fetched from memory, and the interpolation filter is assumed to be symmetric and have eight taps. Furthermore, because the search operation is symmetric (it can move iteratively left and right), the same must be applied to the left side (extending by three rows of pixels), resulting in the placement of fractional pixels on the left side of the window.

[0113] As a result, interpolation filtering requires an additional number of samples to be fetched from memory, as shown by the dashed lines, to include the dotted circles that represent the positions added by fractional interpolation.Similarly, allowing half a search at each vertical position requires that the window of samples accessed from memory be expanded vertically above and below (not shown in the example in Figure 10).

[0114] The memory access window is defined as a rectangle that encompasses all samples that need to be fetched from memory to perform a motion vector search for a prediction block (coding block). The memory access window not only includes the actual samples needed, but also all remaining samples that may be accessed during the motion vector search operation. In the example of Figure 9, the motion vector search moved to the right. However, it could also have moved to the left, which is not known in advance. Therefore, to avoid accessing external memory multiple times, the memory access window (or access window) includes all samples that can be accessed in each operation.

[0115] To achieve this, the present disclosure performs padding of samples located outside the memory access window when necessary, for example to interpolate fractional samples.

[0116] By providing replacement samples (padding samples), the template matching In other words, the search space of the prediction block may include at least one fractional sample position with a value obtained by interpolation filtering using a filter of a predefined tap size. The interpolation filtering uses a value of the sample position from the window and at least one substitution value to obtain a value for the at least one fractional sample position.

[0117] 11 shows the basic steps of a known technique for performing motion vector derivation / refinement as outlined above. In the first step (step 1), a "search space" of at least two candidate motion vectors is obtained (in the illustrated example, there are nine candidate motion vectors). Optionally, one of the candidate motion vectors forming the search space may be considered an initial motion vector (also referred to as an "anchor" motion vector or anchor candidate motion vector, according to the terminology used in this application). The initial or anchor motion vector may, for example, be a motion vector included in the bitstream and signaled explicitly or implicitly to the decoder.

[0118] In the next step (step 2), the best fit in the search space is matching In order to find the point matching The process is applied (in other words, the point in the search space that minimizes the cost function and corresponds to the most suitable candidate motion vector). Bilateral Matching is applied, but the template matching Other ways of determining the cost function, such as:

[0119] In the last step (step 3) of the diagram, the final prediction is obtained by interpolation based on the most suitable candidate motion vector selected according to the processing of step 2.

[0120] The concepts of Figure 11 are further explained in the flow diagram of Figure 12. In Figure 12, the case of a candidate motion vector pair, i.e. Bilateral Matching The diagram for is shown (template matching (but is also applicable when used for bi-prediction as described above with reference to FIG. 3, for example).

[0121] In the first step (1310), a search space of candidate motion vector pairs is constructed. This step corresponds to "Step 1" in Figure 11.

[0122] In the next step (1320), a prediction is generated for each motion vector pair in the search space by applying interpolation filtering. In the next step (1330), a template is constructed (obtained) for each motion vector pair in the search space. As mentioned above, this step is optional, i.e., the template matching is only necessary if Bilateral Matching In the next step (1340), for each motion vector pair in the search space, the template is matched by comparing the prediction block pointed to by the two motion vectors of the pair (or the prediction block of one candidate motion vector, and the template). matching Cost (or Bilateral Matching The next step (1370) is to calculate the template or Bilateral Minimize a cost function (more generally, the best matching Do, i.e. matching11. The process selects the most appropriate motion vector pair (obtained as the most appropriate by the process), and the resulting motion vector pair is selected and output as the motion vector pair for the prediction block. According to the terminology described above, this may be called a "refinement motion vector pair." The above steps 1320, 1330, 1340, and 1370 correspond to "step 2" in FIG. 11.

[0123] The next step (1380) is to generate the final prediction by interpolation using the motion vector pair selected in step 1370. This step corresponds to "step 3" in FIG.

[0124] Figure 13 shows a process for motion vector refinement and final prediction generation according to an exemplary embodiment of the present invention. The main difference between the approach in Figure 13 and the conventional approach shown in Figure 11 is that the approach in Figure 13 of the present invention uses sample padding for both motion vector refinement and the final generation of a predictor using interpolation filtering.

[0125] The (optional) use of padded samples in motion vector refinement is shown in additional step 2, which will be described in more detail below with reference to Figures 14 to 18. This allows for the following: even for candidate motion vectors with fractional coordinates (i.e., fractional coordinates in the search space): Bilateral Matching or template matching The candidate block desired to perform the operation can be performed by using a predefined restricted access window, which is the same as in the case of a search space containing only integer points. Bilateral Matching or template matching The actual derivation of the cost function by applying

[0126] Furthermore, sample padding is used in the final step 4 (again, or only here) to perform the actual predictor generation. This step generally corresponds to step 3 of Fig. 11, except that it uses the concept of padding. Specifically, in the given example, the sample values ​​located in the rightmost two columns and the bottom row are not accessed, but are padded with padding sample values ​​based on the other sample values ​​that are accessed.

[0127] The number of samples that are padded (replaced with replacement values ​​because they are unavailable) depends on the difference between the last selected ("refined") motion vector and the reference initial motion vector ("anchor"). In the example shown, this difference is 2 pixels vertically and 1 pixel horizontally. Accordingly, in the diagram of step 5, to interpolate the pixel at the intersection of the two rectangles horizontally and vertically, the sample values ​​of the pixels in the two columns to the right of the pixel and in the last row are unavailable and must be replaced (padded) with replacement values ​​according to the predefined rules mentioned above, which are explained in more detail below.

[0128] Preferably, if padding is also used for the search in step 2, the same access window is used for the interpolation filtering operations in both step 2 and step 4.

[0129] Preferably, the memory access window used in steps 2 and 4 of Figure 13 is determined based on a particular candidate motion vector within the search space. The particular motion vector (called an anchor motion vector) may be, for example, a motion vector included in the bitstream and explicitly or implicitly signaled to the decoder. The anchor motion vector selection decision is performed according to predefined rules known to both the encoder and the decoder.

[0130] FIG. 14 illustrates a window of memory accesses for motion vector refinement. The window extends around a central point 1010, which is the location pointed to by the initial motion vector. Additionally, a fractional location 1020 also belongs to the search space. However, the fractional location 1020 is interpolated by a six-tap filter using three integer sample locations to the left of the fractional location and three integer sample locations to the right of the fractional location. Because the fractional location 1020 is collinear with the integer sample locations, a one-dimensional filter may be applied for the interpolation, such as shown in the example described above with reference to FIG. 10. As can be seen in FIG. 14, two additional sample locations 1030 are required to interpolate the fractional sample 1020, which are located outside the window. To avoid additional loading of these samples from external memory, according to the present disclosure, the values ​​at the locations outside the window 1030 are replaced with values ​​obtained based on the samples within the window.

[0131] FIG. 14 illustrates a fractional sample location 1020 that is midway between two integer sample locations and thus at a half-pixel (½-pel) location. However, the present disclosure is equally applicable to other fractional locations, such as a quarter- or eighth-position. Furthermore, the sample location 1020 is located on a horizontal line corresponding to a row of integer sample locations, such that horizontal 1D filtering may be used for its interpolation. However, the present disclosure is not limited to such fractional locations. The fractional locations may also be located on a vertical line corresponding to a column of integer sample locations, such that vertical 1D filtering may be used for its interpolation. Furthermore, the fractional locations need not necessarily be located on the same line of pixels as the integer samples. In such cases, two-dimensional filtering may be used to interpolate such samples, and such 2D filtering may be decomposable into vertical 1D filtering and horizontal 1D filtering.

[0132] Various examples of fractional positions are shown in Figure 16. In particular, fractional position 1 is a position for interpolation where both horizontal and vertical filtering can be applied because the rows and columns of integer sample positions are not collinear. Position 2 is a position for interpolation where only horizontal filtering is used, and position 6 is a position for interpolation where only vertical filtering is used. All of the fractional positions shown in Figure 16 can be obtained using only integer positions within the window.

[0133] The window for memory access may be defined in various ways. Figure 18 shows an example in which the memory access window is defined as an extension portion EXT to the left / right or above / below the coding block (corresponding to the position of the coding block given by the initial motion vector). The amount of extension may depend on the size and shape of the coding block or the prediction block. In Figure 18, the extension portion is four samples long in each direction (above, below, left, right). However, it should be noted that EXT may take different values ​​for different directions depending on the block size (which may be different sizes in the vertical and horizontal directions) and / or the form and size of the search space.

[0134] For example, according to the example, a window may be defined as N integer sample columns and M integer sample rows relative to the predicted block initial motion vector, where N and M are non-zero integer values. Although N and M are shown in FIG. 18, they are the same size. As mentioned above, N and M may have different sizes. N and M are integers, at least one of which is non-zero. The size of the access window may be determined by taking the parameters N and M and the format and size of the template. In particular, if the template has T1 rows and T2 columns, the size of the memory access window may be calculated as (N+T1+N) rows and (M+T2+M) columns. This is because the search can proceed N samples left or right, resulting in 2N samples horizontally, and M samples up or down, resulting in 2M samples vertically. The template may also be defined as T1×T2 samples at any 2N×2M sample location. matching It can be done.

[0135] On the other hand, for a particular search space construction technique such as that described with reference to Figures 8 and 9, the memory access window may be defined in terms of the maximum number of refinement iterations (search space construction iterations) and the iteration step size (in terms of the maximum sample distance achievable in each iteration), which can then be translated into the maximum amount of displacement to the left, right, up, and down. Thus, the memory access window is defined as the maximum displacement in each direction. For example, if there are four iterations, each of which can advance the search space by at most one integer sample position, then EXT = 4.

[0136] In other words, according to this example, the processing circuitry determines the best fit of the search space in the most recent iteration. matching a search space iteratively expanded in a direction given by one (or more) of the positions of the template matching The refinement of the initial motion vector is determined by the window defined by a predefined maximum number of iterations. Bilateral Matching The same applies mutatis mutandis to the case of

[0137] It should be noted that, in general, the present disclosure is not limited to any particular shape or format or type of determination of the search space. In another example, the search space may be a rectangular sub-window of the window or may be equal to the window. The definition of the memory access window may be significant because corresponding signaling parameters (e.g., parameter EXT, or parameters N and M) may be included in the bitstream. However, the size of the memory access window may be defined in a standard manner or may be derivable based on other coding parameters (e.g., template size, prediction block size, image resolution, etc.). Returning to FIG. 14 , according to an embodiment, the memory access window for refinement is defined around the location 1010 pointed to by the unrefined motion vector. The window identifies the maximum number of pixel samples that need to be accessed from memory to perform the motion vector refinement search. The search space here coincides with the access window. In other words, although the template size is assumed to be 1x1 for simplicity in this example, it can and typically is larger. The memory access window is typically defined as an extension around the coding block. In this example, the extension is 4 samples from the left / right and 4 samples from the top / bottom. Template matching or Bilateral Matching If the search space position currently being tested during a template requires samples from outside the window of memory accesses, the samples needed to reconstruct the coding block are obtained by padding. This is the case for fractional position 1020 in Figure 14. matching is performed at or using this position, it must be interpolated using another padding position 1030 located outside the window.

[0138] The padding samples may be generated, for example, using one of the following methods: Recent sample copy. Mirroring along the axis of interpolation filtering.

[0139] Specifically, nearest sample duplication refers to a technique in which a replacement value is determined to be equal to the value closest to the integer sample position being accessed. For example, in FIG. 14 , the values ​​1030 of the two missing positions are replaced with the values ​​of the samples immediately to the right of the fractional position 1020 being interpolated. However, it should be noted that this technique is merely an example, and the present disclosure is not limited to simple duplication of neighbors. Alternatively, several nearest sample positions may be used to interpolate the missing sample 1030. For example, the three samples on the boundary closest to the missing sample may be used to interpolate the missing sample, e.g., by weighted averaging, with weights set inversely proportional to the distances of the three samples from the missing sample.

[0140] Mirroring refers to a technique in which replacement values ​​are determined by mirroring the values ​​of corresponding evaluated integer sample positions relative to the axes of the interpolation filter. For example, in Figure 14, the two missing samples 1030 are replaced with the values ​​of the samples to the left of fractional position 1020. Specifically, the second sample to the right of fractional position 1020 is replaced with the value two positions to the left of fractional position 1020, and the third sample to the right of fractional position 1020 is replaced with the value three positions to the left of fractional position 1020.

[0141] It should be noted that a combination of the two approaches described above can also be applied if the missing sample is a function of the mirrored sample and the nearest sample.

[0142] According to an embodiment, the window is a template at integer sample positions within the search space. matching or Bilateral Matching This is the case for example in Figure 14, where additional samples outside the window are formed by the template at non-integer fractional sample positions. matching14, in the example above, the half-pel coordinates being searched for require samples that are outside the memory access window. As a result, a padding operation is applied to generate unusable samples. However, the present invention is not limited to this.

[0143] FIG. 15 shows how a window can be used to find the template at integer sample positions within the search space. matching 15 shows an alternative approach that does not include all integer sample positions accessible for . In the previous examples, the coding block size was shown as 1x1 for simplicity. In Figure 15, the size is 4x4 integer samples. In this example, the search coordinates (template or Bilateral Matching Even if the location where is calculated points to an integer sample, padding can still be applied if some of the samples in the block are outside the memory access window. This technique allows us to further restrict the window size.

[0144] In more detail, Figure 15 shows a window 1120 containing several accessible samples. The initial motion vector points to a position that marks the top left corner of the coding block. The search space may here be considered equal to the window. Note, however, that it can usually be smaller than the window, which may also make sense, since if the template is larger than 1x1 integer samples, then necessarily the template matching This is because performing the above requires samples other than those included in the search space. Nevertheless, if the search space defines the size of the window, the missing samples may alternatively be padded. In Figure 15, an example search position is shown in the second row and tenth column of the window. Templates using templates with a size of 4x4 integer samples matchingrequires additional pixels that lie outside the window. These samples are therefore padded, i.e., replaced with values ​​determined or calculated based on the samples within the window. Padding techniques mentioned above, such as nearest neighbor or mirroring, may also be applied.

[0145] The window the search space, or a block having the size of the predicted block located at the initial motion vector position, or initial motion vector position, The vertical and / or horizontal size of the

[0146] An example of a window definition is shown in FIG. 18 and described above. This corresponds to determining the window size based on the size of the prediction block located at the initial motion vector position. Alternatively, the window size may be defined relative to the initial vector position by the size in one or both of the x and y directions. For example, the window may be defined to be located around the position pointed to by the initial motion vector and extend A integer samples in each of the directions up, down, left, and right. Alternatively, it may extend A integer samples to the left and right, respectively, and B integer samples up and down. Other definitions are also possible.

[0147] It should be noted that the above examples have been provided for 1 / 2 pel interpolation. However, the present disclosure is not limited thereto. In general, any fractional position, such as 1 / 4, 1 / 8, etc., may be used, i.e., interpolated using a corresponding interpolation filter.

[0148] The same definition of window size also applies to the case of padding for interpolation filtering to generate the predictor: specifically, the vertical and horizontal sizes (i.e., x and y directions) are determined by the vertical and horizontal lengths of the interpolation filter used to determine the predictor.

[0149] Figure 17 is a flow diagram showing in more detail the process shown in Figure 13. The flow diagram of Figure 17 differs from the flow diagram of Figure 12, which shows the conventional process, in that steps 1320 and 1380 of Figure 12 are replaced by steps 1720 and 1780, respectively. These steps are: Bilateral Matching or template matching The steps of FIG. 12 differ in that padding with replacement samples is applied both for the interpolation filtering to generate candidate prediction blocks in the reference picture(s) to perform (a) and for the interpolation filtering to generate the final prediction.

[0150] The processing circuitry described with reference to FIG. 7 may be used in an encoder and / or decoder such as those shown in FIGS.

[0151] In particular, an encoding device may be provided for encoding a video image divided into predictive blocks into a bitstream, the encoding device comprising: a device for determining a motion vector for the predictive block as described above, comprising a processing circuit; and an encoding circuit for encoding a difference between the predictive block and a predictor given by the predictive block at a position specified by the determined motion vector, and generating a bitstream comprising the encoded difference and the initial motion vector.

[0152] The other units and functions of the encoder described above with reference to FIG. 1 may also be provided or implemented in the processing circuitry.

[0153] Correspondingly, a decoding device is provided for decoding a video image divided into prediction blocks from a bitstream, the decoding device comprising: an analysis unit for analyzing from the bitstream an initial motion vector and an encoded difference between the prediction block and a predictor provided by the prediction block at a position specified by the refinement motion vector; an apparatus for determining a refinement motion vector for the prediction block as described above, the apparatus comprising a processing circuit; and a decoding circuit for reconstructing the prediction block as the sum of the analyzed difference and the predictor provided by the prediction block at a position based on the refinement motion vector. For example, the predictor may be directly provided by the position of the refinement motion vector. However, there may be another processing step to obtain the motion vector of the current prediction block, which may further change the motion vector (filtering, clipping, further refinement, etc.).

[0154] The other units and functions of the decoder described above with reference to FIG. 2 may also be provided or implemented in a processing circuit.

[0155] Furthermore, although embodiments of the present invention have been described in terms of apparatuses comprising processing circuitry for performing motion vector refinement, the present disclosure is not limited thereto but also provides corresponding methods, which include processing steps corresponding to the steps that the processing circuitry described above is configured to perform.

[0156] The taps correspond to the filter coefficients. The tap size corresponds to the filter order. Here, the filter is assumed to be a linear filter. In some examples, the filter may be symmetric, i.e., have symmetric coefficients. However, this disclosure is not limited to symmetric or linear filters, or any type of filter. In general, the fractional position may be obtained in any manner based on neighboring samples.

[0157] The method according to the embodiment determines the motion vector for the prediction block in the following steps, which are illustrated in Figure 19. In step 1410, an initial motion vector and template for the prediction block are obtained, and then a template is calculated using the template in a search space. matching The refinement of the initial motion vector is determined by: ∑ i = ... matching is executed.

[0158] Templates matching or Bilateral Matching is an integer sample position within a predefined window, a predefined window containing the search space, and a template matching or Bilateral Matching and another integer sample location accessible for the window, and replace at least one integer sample location located outside the window with a replacement value based on at least one sample within the window. The value of the accessed integer sample location and the replacement value are based on a template matching This is shown in Figure 19 and is used to execute the template matching or Bilateral Matching is performed at every position in the search space. In step 1430, the template at the current position matching or Bilateral Matchingrequires one or more samples outside the memory access window. If so (step 1430: yes), then in step 1440, padding is performed as described above for one or more samples outside the window. After this step, in step 1450, the template is reconstructed at the current position using the padded samples. matching or Bilateral Matching Alternatively, if there are no samples needed outside the window (step 1430: No), then in step 1450, the template is calculated at the current position using the available samples. matching or Bilateral Matching If the current position is the last position in the search space to be tested (step 1460 is yes), then the template matching Otherwise (step 1460: No), the next position is obtained in step 1470, and the template is applied to the new position in step 1430 as described above. matching continues.

[0159] Templates matching When the ,is finished, the best location (e.g., the stored location with the lowest cost) is matching The location where the

[0160] Figure 20 shows an alternative exemplary approach to motion vector refinement and predictor generation, applicable within the framework of an embodiment of the present invention. The difference between this approach and the approach of Figure 13 is that in Figure 20, interpolation filtering with padding is applied only to the final generation of the predictor, using the refined motion vector (pair). This is step 5 in Figure 20, which corresponds to step 4 in Figure 13.

[0161] On the other hand, interpolation filtering with padding is not applied to motion vector refinement (search), i.e., step 2 in Figure 13. Instead, in the approach of Figure 20, if the initial motion vector (or multiple initial candidate motion vectors forming the initial search space) points to the nearest integer point, a rounding operation is performed to round the coordinates to the nearest integer point (step 2 in Figure 20). This rounding operation is equivalent to a predefined vector shift indicated by MV_round_diff, which represents the rounding direction. The rounding operation ensures that the search space is modified to obtain a modified search space that only has points with integer coordinates. As a result, Bilateral (or template) matching By, best matching The next step 3 to get the integer point to be used can be done with integer coordinates, so that the template / Bilateral Matching No interpolation filtering is required to generate the prediction for .

[0162] Furthermore, FIG. 20 includes step 4. This step is optional. In step 4, the best fit in the modified search space is determined. matching After the corresponding integer point is obtained in step 3, the rounding is undone to obtain the final refinement motion vector in the (initial) search space. In other words, the vector moving in the opposite direction to step 2 is the best fit resulting from step 3. matching Step 5 is applied to the refined motion vectors resulting from step 3. If optional step 4 is included, then a final prediction is obtained based on the refined motion vectors resulting from step 4. Alternatively, step 3 is omitted and step 5 is applied directly to the motion vectors resulting from step 3. Further alternatively, step 4 can be included or omitted based on predetermined conditions.

[0163] A flow diagram corresponding to the approach of Figure 20 is shown in Figure 21. Again, the flow diagram is constructed for motion vector pairs rather than single motion vectors, which are used in Figure 20 for the sake of simplicity.

[0164] The first step 1310 is the same as in Figure 20. The subsequent step 2115 corresponds to the rounding operation of step 2 of Figure 20. While the process shown here is general insofar as the modification of the initial search space is not limited to rounding operations, more generally, it is assumed herein that a representative motion vector pair is generated for each motion vector pair in the initial search space. The rounding operation of Figure 20 is a special case of this mapping of initial search space motion vectors to representative ones.

[0165] Next Step 2130 (Template matching ) and 2140 are the same as in Figure 17, except that each operation is performed on a representative motion vector in the modified search space. Steps 2150 and 2160 correspond to step 4 in Figure 20. Step 2150 is the step of matching Cost (or Bilateral Matching The steps for setting a representative motion vector pair that minimizes the cost (Pair_best_rep) are shown below.

[0166] Step 2160 (optional) corresponds to undoing the rounding operation shown in step 4 of Figure 20. In the next step (1370), the motion vector (pair) in the original search space is output as the motion vector for the prediction block. Steps 1370 and 1780 are the same as in Figure 17.

[0167] Further, there is provided an encoding method for encoding a video image divided into predictive blocks into a bitstream, the encoding method comprising the steps of determining a motion vector for the predictive block according to any of the methods described above, encoding a difference between the predictive block and a predictor given by the predictive block at a position based on the determined motion vector, and generating a bitstream including the encoded difference and the initial motion vector.

[0168] The encoding method may further include steps described with reference to the functions of the blocks in FIG.

[0169] Further, a decoding method is provided for decoding a video image divided into predictive blocks from a bitstream, the decoding method comprising the steps of: analyzing from the bitstream an initial motion vector and an encoded difference between the predictive block and a predictor provided by the predictive block at a position specified by the refinement motion vector; determining a refinement motion vector for the predictive block according to any of the methods described above; and reconstructing the predictive block as the sum of the analyzed difference and the predictor provided by the predictive block at the position specified by the refinement motion vector.

[0170] The decoding method may further include the steps described with reference to the functions of the blocks in FIG.

[0171] However, it should be noted that Figures 1 and 2 are not intended to limit this disclosure, but merely provide non-limiting examples of implementation of the present invention within existing encoders and / or decoders.

[0172] The motion vector determination using sample padding as described above may be performed as part of encoding and / or decoding a video signal (moving image). However, the motion vector determination is not limited to use for encoding / decoding, but may also be used for other purposes in image processing, such as motion estimation, motion analysis, etc.

[0173] The motion vector determination may be implemented as an apparatus. Such an apparatus may be a combination of software and hardware. For example, the motion vector determination may be performed by a general-purpose processor, or a digital signal processor (DSP), or a chip such as a field programmable gate array (FPGA). However, the present invention is not limited to implementation on programmable hardware. It may be implemented on an application-specific integrated circuit (ASIC) or by a combination of the hardware components mentioned above.

[0174] The motion vector determination may be performed by program instructions stored in a computer-readable storage medium. When the program is executed, the computer performs the steps of the above-mentioned method. The computer-readable storage medium may be any medium on which a program is stored, such as a DVD, a CD, a USB (flash) drive, a hard disk, or server storage available via a network.

[0175] The encoder and / or decoder may be implemented in various devices such as a TV set, a set-top box, a PC, a tablet or a smartphone, i.e. any recording device, encoding device, transcoding device, decoding device or playback device etc. It may also be software or an app that performs the method steps and may be stored / executed on a processor included in an electronic device such as those mentioned above.

[0176] In summary, the present disclosure provides a template matching or Bilateral Matching and motion vector determination using a template. matching or Bilateral Matching , and / or predictor generation uses interpolation filtering, where the interpolation filtering operation accesses integer sample positions within a window and compares the search space with a template matchingand another integer sample position accessible for the template, padding at least one integer sample position located outside the window with a padded sample value based on at least one sample within the window, matching or Bilateral Matching and predictor generation using the values ​​of the accessed integer sample positions and the padded sample values.

[0177] Further embodiments of the present invention are provided below: Please note that the numbering used in the following sections does not necessarily follow the numbering used in the previous sections.

[0178] Embodiment 1. An encoding device for encoding a predicted block of a video image into a bitstream (171), the encoding device comprising: Obtaining at least one initial motion vector; determining a refinement of the initial motion vector based on a search space of the candidate motion vectors to obtain a refined motion vector, the search space being located at a position (810) given by the initial motion vector; determining a motion vector for the predicted block (165) by encoding a difference (105) between the predicted block and a predictor provided by the predicted block at a position based on the determined motion vector to generate a bitstream (171) including the encoded difference and the initial motion vector; determining a predictor according to a motion vector using interpolation filtering with an interpolation filter, said interpolation filtering comprising: Accesses sample values ​​at integer sample positions within a predefined window, replacing at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within said window; performing interpolation filtering using the accessed integer position sample values ​​and substitution values; a processing circuit (600) configured to: Encoding device.

[0179] Embodiment 2. A decoding device for decoding a video image divided into prediction blocks from a bitstream (171), the decoding device comprising: Analyzing, from the bitstream at a location specified by the refinement motion vector, the initial motion vector and the encoded difference between the prediction block and the predictor provided by the prediction block; Obtaining at least one initial motion vector; determining a refinement of the initial motion vector based on a search space of the candidate motion vectors to obtain a refined motion vector, the search space being located at a position (810) given by the initial motion vector; determining a motion vector for the prediction block by determining a motion vector by reconstructing the predicted block as the sum of the analyzed difference and a predictor provided by the predicted block at the location specified by the refined motion vector determined by the processing circuit (600); a processing circuit configured to: The processing circuitry is further configured to determine a predictor according to the refined motion vector using interpolation with an interpolation filter, the interpolation comprising: Accesses sample values ​​at integer sample positions within a predefined window, replacing at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within said window; Performing interpolation using the accessed integer sample position values ​​and replacement values; Decoding device.

[0180] Embodiment 3. An apparatus according to embodiment 1 or 2, wherein the replacement values ​​are determined by mirroring the values ​​of the corresponding evaluated integer sample positions relative to the axes of the interpolation filter.

[0181] Embodiment 4. An apparatus according to any one of embodiments 1 to 3, wherein the interpolation filter is a one-dimensional filter or a separable two-dimensional filter.

[0182] Embodiment 5. An apparatus according to any one of embodiments 1 to 4, wherein the window is formed by all integer sample positions accessible to obtain a predictor given by the prediction block at a position specified by the initial motion vector.

[0183] Embodiment 6. The window is a block having the size of the predicted block located at the initial motion vector position; the initial motion vector position, and The vertical and horizontal extent of the interpolation filter used to determine the predictor 6. An apparatus according to any one of embodiments 1 to 5, wherein the vertical and / or horizontal size of the at least one of the plurality of pixels is defined by its vertical and / or horizontal size relative to at least one of the plurality of pixels.

[0184] Embodiment 7. The apparatus of any one of embodiments 1, 2, or 4 to 6, wherein the replacement value is determined to be equal to the value closest to the accessed integer sample position.

[0185] Embodiment 8. An apparatus according to any one of embodiments 1 to 7, wherein the number of sample values ​​to be replaced depends on the difference between the initial motion vector and the refinement motion vector.

[0186] Embodiment 9. The processing circuit (600) performs template processing using a template. matching or Bilateral Matching and configured to determine a refinement of the initial motion vector by The template matching or the above Bilateral Matching but, The sample values ​​at integer sample positions within another predefined window (1120) are accessed, and the other predefined window (1120) is used to access the template. matching or Bilateral Matching and another integer sample position accessible for replacing at least one sample value (1030) at an integer sample position located outside the other window with a replacement value based on at least one sample within said other window; The template is created using the value of the accessed integer sample position and the replacement value. matching or Bilateral Matching 9. An apparatus according to any one of embodiments 1 to 8, which performs the steps of:

[0187] Embodiment 10. The search space includes at least one fractional sample position (1,2,6) with a value obtained by interpolation filtering with a filter of a predefined tap size; 10. The apparatus of embodiment 9, wherein interpolation filtering uses values ​​of the accessed integer sample positions and at least one substitution value to obtain values ​​for the at least one fractional sample position.

[0188] Embodiment 11. Template matching or Bilateral Matching 11. The apparatus of embodiment 10, wherein the replacement value used in is determined by mirroring the value of the corresponding accessed integer sample position relative to the axis of the filter.

[0189] Embodiment 12. Template matching or Bilateral Matching 11. An apparatus as described in embodiment 9 or 10, wherein the replacement value used in is determined to be equal to the value closest to the accessed integer sample position.

[0190] Embodiment 13. Another predefined window (1120) search space, a block having the size of the predicted block located at the initial motion vector position; and Initial Motion Vector Position 13. An apparatus according to any one of embodiments 9 to 12, wherein the apparatus is defined by its vertical and / or horizontal size relative to at least one of the following:

[0191] Embodiment 14. An apparatus according to any one of embodiments 9 to 13, wherein the another predefined window (1120) is the same as the predefined window.

[0192] Embodiment 15. The refinement Bilateral Matching and the processing circuit is configured to obtain at least two initial motion vectors that point to positions in different reference pictures (Ref0, Ref1) and form a pair of initial motion vectors (Pair_0, Pair_1, ..., Pair_N).

[0193] Embodiment 16. The refinement is performed using a template matching and the processing circuitry is determined by Get the template for the predicted block, a template using the template in the search space matching The refinement of the initial motion vector is determined by 15. The device of any one of embodiments 1 to 14, further configured to:

[0194] Embodiment 17. An encoding method for encoding a video image divided into prediction blocks into a bitstream, the encoding method comprising: obtaining at least one initial motion vector (1310, 1410); determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refined motion vector (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160), wherein the search space is located at a position given by the initial motion vector; determining a motion vector for the prediction block, encoding a difference between the predicted block and the predictor provided by the predicted block at a position based on the determined motion vector, and generating a bitstream including the encoded difference and the initial motion vector; determining (1780) a predictor according to the motion vector using interpolation filtering with an interpolation filter, the interpolation filtering comprising: Accesses sample values ​​at integer sample positions within a predefined window, replacing at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within said window; Step (1780) of performing interpolation filtering using the accessed integer position sample values ​​and substitution values; Encoding methods, including:

[0195] Embodiment 18. A decoding method for decoding a video image divided into prediction blocks from a bitstream, the decoding method comprising: analyzing, from the bitstream at the location specified by the refined motion vector, the initial motion vector and the encoded difference between the predicted block and the predictor given by the predicted block; obtaining at least one initial motion vector (1310, 1410); determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refined motion vector (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160), wherein the search space is located at a position given by the initial motion vector; determining a motion vector for the prediction block, reconstructing the predicted block as the sum of the analyzed difference and a predictor given by the predicted block at the location specified by the refined motion vector determined by the step of determining the motion vector; determining (1780) a predictor according to the refined motion vector using interpolation with an interpolation filter, the interpolation comprising: Accesses sample values ​​at integer sample positions within a predefined window, replacing at least one sample value at an integer sample position located outside the window with a replacement value based on at least one sample within said window; performing interpolation using the accessed integer position samples and replacement values; A decoding method including:

[0196] Embodiment 19. The method of embodiment 17 or 18, wherein the replacement values ​​are determined by mirroring the values ​​of the corresponding evaluated integer sample positions relative to the axes of the interpolation filter.

[0197] Embodiment 20. The method according to any one of embodiments 17 to 19, wherein the interpolation filter is a one-dimensional filter or a separable two-dimensional filter.

[0198] Embodiment 21. A method according to any one of embodiments 17 to 20, wherein the window is formed by all integer sample positions accessible to obtain the predictor given by the prediction block at the position specified by the initial motion vector.

[0199] Embodiment 22. The window is a block having the size of the predicted block located at the initial motion vector position; the initial motion vector position, and The vertical and horizontal extent of the interpolation filter used to determine the predictor 22. The method according to any one of embodiments 17 to 21, wherein the vertical size and / or horizontal size of at least one of the pixels is defined by its vertical size and / or horizontal size.

[0200] Embodiment 23. The method of any one of embodiments 17, 18, or 20 to 22, wherein the replacement value is determined to be equal to the value closest to the accessed integer sample position.

[0201] Embodiment 24. The method according to any one of embodiments 17 to 23, wherein the number of sample values ​​to be replaced depends on the difference between the initial motion vector and the refinement motion vector.

[0202] Embodiment 25. The steps (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160) of determining refinement are performed using a template matching or Bilateral Matching The refinement of the initial motion vector is determined by The template matching or the above Bilateral Matching but, Accesses sample values ​​at integer sample positions within another predefined window, where the other predefined window is a template matching or Bilateral Matching and another integer sample position accessible for replacing (1440) at least one sample value at an integer sample position located outside the other window with a replacement value based on at least one sample within the other window; The template is created using the value of the accessed integer sample position and the replacement value. matching or Bilateral Matching 25. The method of any one of embodiments 17 to 24, wherein (1450)

[0203] Embodiment 26. The search space includes at least one fractional sample position with a value obtained by interpolation filtering using a filter of a predefined tap size; 26. The method of embodiment 25, wherein interpolation filtering uses values ​​of the accessed integer sample positions and at least one substitution value to obtain values ​​for the at least one fractional sample position.

[0204] Embodiment 27. Template matching or Bilateral Matching 27. The method of embodiment 26, wherein the replacement values ​​used in are determined by mirroring the values ​​of the corresponding evaluated integer sample positions relative to the filter axis.

[0205] Embodiment 28. Template matching or Bilateral Matching 28. The method of embodiment 26 or 27, wherein the replacement value used in is determined to be equal to the value closest to the accessed integer sample position.

[0206] Embodiment 29. Another predefined window is search space, a block having the size of the predicted block located at the initial motion vector position; and Initial Motion Vector Position 29. The method according to any one of embodiments 25 to 28, wherein the vertical size and / or horizontal size of at least one of the pixels is defined.

[0207] Embodiment 30. The method of any one of embodiments 25 to 29, wherein the another predefined window is the same as the predefined window.

[0208] Embodiment 31. The refinement Bilateral Matching 31. A method according to any one of embodiments 17 to 30, wherein the step of obtaining obtains at least two initial motion vectors that point to positions in different reference pictures and form a pair of initial motion vectors.

[0209] Embodiment 32. The refinement is performed using a template matching and said step of determining a motion vector is determined by: a step (1330; 2130) of obtaining a template for the predicted block; a template using the template in the search space matching and a step (1340; 2140) of determining the refinement of the initial motion vector by 32. The method of any one of embodiments 17 to 31, further comprising:

[0210] Embodiment 33. A computer-readable storage medium storing instructions that, when executed on a processor, cause the processor to perform the method of any one of embodiments 17 to 32. [Explanation of symbols]

[0211] 100 Encoder 101 Picture Block 102 Input 104 Residual calculation 105 Residual Blocks 106 Conversion Unit 107 Conversion Factor 108 quantization units 109 quantization coefficients 110 Inverse Quantization Unit 111 Inverse quantization coefficients 112 Inverse conversion unit 113 Inverse Transformed Blocks 114 Reconstruction 115 reconstructed blocks 116 buffers 117 Reference Samples 120 Loop Filtering Unit 121 Filtered Blocks 130 Decoded Picture Buffer 131 decoded pictures 142 Inter-estimation Unit 144 Inter Prediction Units 152 Intra Estimation Units 154 intra prediction units 160 Mode Selection Unit 165 predicted blocks 170 Entropy Encoding Unit (Entropy Coding Unit) 171 Encoded Picture Data (Encoded Bitstream) 172 outputs 200 decoder 202 Input 204 Entropy Decoding Unit 209 Quantized Residual Transform Coefficients 210 Inverse Quantization Unit 211 Inverse Quantized Transform Coefficients 212 Inverse Conversion Unit 213 Inverse Transformed Blocks 214 Reconstruction 215 reconstructed blocks 216 buffers 217 Reference Sample 220 Loop Filtering Unit 221 Filtered Blocks 230 Decoded Picture Buffer 231 decoded pictures 232 output 244 Inter Prediction Units 254 intra prediction units 260 Mode Selection 265 predicted blocks 600 Processing Circuit 610 Template Determination Unit 620 Motion Vector Refinement Unit 640 internal memory 650 external memory 810 center point 1010 Position pointed to by unrefined motion vector 1020 fractional position 1030 Alternate Sample Positions 1120 Another Predefined Window

Claims

1. 1. An apparatus for receiving and storing a bitstream comprising coded blocks of a video image, the apparatus comprising: receiving the bitstream including the encoded blocks; storing the bitstream containing the encoded blocks; It is configured as follows: The coded blocks included in the bitstream are Obtaining an initial motion vector for a block; obtaining a refined motion vector for the block based on the initial motion vector; determining a predictor for the block according to the refined motion vector; encoding the block based on the predictor to obtain the encoded block; is generated by the refinement is determined by bilateral matching, and obtaining the initial motion vector includes obtaining at least two initial motion vectors pointing to positions in different reference pictures and forming an initial motion vector pair; determining the predictor according to the refined motion vector using interpolation with an interpolation filter, and using at least one sample value of an integer sample position located inside a window and padded sample values ​​of integer sample positions located outside the window for filtering by the interpolation filter, the padded sample value being based on at least one sample value of a sample located inside the window; Device.

2. 1. An apparatus for storing and transmitting a bitstream comprising coded blocks of a video image, said apparatus comprising: storing the bitstream containing the encoded blocks; transmitting the bitstream containing the encoded blocks. It is configured as follows: The coded blocks included in the bitstream are Obtaining an initial motion vector for a block; obtaining a refined motion vector for the block based on the initial motion vector; determining a predictor for the block according to the refined motion vector; encoding the block based on the predictor to obtain the encoded block; is generated by the refinement is determined by bilateral matching, and obtaining the initial motion vector includes obtaining at least two initial motion vectors pointing to positions in different reference pictures and forming an initial motion vector pair; determining the predictor according to the refined motion vector using interpolation with an interpolation filter, and using at least one sample value of an integer sample position located inside a window and padded sample values ​​of integer sample positions located outside the window for filtering by the interpolation filter, the padded sample value being based on at least one sample value of a sample located inside the window; Device.

3. 1. A method for receiving and storing a bitstream containing encoded blocks of a video image, comprising: receiving the bitstream including the encoded blocks; storing the bitstream containing the encoded blocks; Including, The coded blocks included in the bitstream are Obtaining the initial motion vector of the block (1310, 1410); Obtaining a refinement motion vector for the block based on the initial motion vector (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160); determining (1780) a predictor for the block according to the refined motion vector; encoding the block based on the predictor to obtain the encoded block; is generated by the refinement is determined by bilateral matching, and obtaining the initial motion vector includes obtaining at least two initial motion vectors pointing to positions in different reference pictures and forming an initial motion vector pair; the determining (1780) includes determining the predictor according to the refined motion vector using interpolation with an interpolation filter, and using at least one sample value of an integer sample position located inside a window and padded sample values ​​of integer sample positions located outside the window for filtering by the interpolation filter, the padded sample value being based on at least one sample value of a sample located inside the window. method.

4. 1. A method for storing and transmitting a bitstream containing coded blocks of a video image, comprising: storing the bitstream containing the encoded blocks; transmitting the bitstream including the encoded blocks; Including, The coded blocks included in the bitstream are Obtaining an initial motion vector for a block; obtaining a refined motion vector for the block based on the initial motion vector; determining (1780) a predictor for the block according to the refined motion vector; encoding the block based on the predictor to obtain the encoded block; is generated by the refinement is determined by bilateral matching, and obtaining the initial motion vector includes obtaining at least two initial motion vectors pointing to positions in different reference pictures and forming an initial motion vector pair; the determining (1780) includes determining the predictor according to the refined motion vector using interpolation with an interpolation filter, and using at least one sample value of an integer sample position located inside a window and padded sample values ​​of integer sample positions located outside the window for filtering by the interpolation filter, the padded sample value being based on at least one sample value of a sample located inside the window. method.

5. The generation of the coded blocks comprises: determining a refinement of the initial motion vector based on a search space of candidate motion vectors, wherein the refinement motion vector is determined based on the initial motion vector and the refinement.

5. The method of claim 3 or 4, further comprising:

6. The method of claim 5 , wherein the search space is located at a position defined by the initial motion vector.

7. The generation of the coded blocks comprises: Obtaining the difference between the block and the predictor.

7. The method of claim 3, further comprising:

8. The generation of the coded blocks comprises: reconstructing the block based on the predictor and the difference between the block and the predictor.

8. The method of claim 7, further comprising:

9. 9. The method of claim 3, wherein the padded sample values ​​are determined by mirroring the values ​​of corresponding estimated integer sample positions relative to the axes of the interpolation filter.

10. 10. The method of claim 3, wherein the interpolation filter is a one-dimensional filter or a separable two-dimensional filter.

11. 11. The method according to claim 3, wherein the window is formed by all integer sample positions accessible for obtaining the predictor at a position specified by the initial motion vector.

12. The window is a block having the size of the block located at the position of the initial motion vector; the position of the initial motion vector, and The vertical and horizontal extents of the interpolation filter used to determine the predictor.

12. The method according to claim 3, wherein the vertical and / or horizontal size of the image is defined by at least one of:

13. 13. The method of claim 3, wherein the padded sample value of the integer sample position located outside the window is determined to be equal to the sample value of at least one used integer sample position located inside the window that is closest to the integer sample position located outside the window.

14. The method according to claim 3 , wherein the number of sample values ​​to be padded depends on the difference between the initial motion vector and the refinement motion vector.

15. obtaining refinement (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160) determines refinement of the initial motion vector by the bilateral matching; The bilateral matching accessing sample values ​​at integer sample positions within another window, the other window including a search space and another integer sample position accessible for the bilateral matching; padding (1440) at least one sample value at an integer sample position located outside the other window with a padded sample value based on at least one sample within the other window; The method of any one of claims 3 to 14, further comprising performing (1450) the bilateral matching using values ​​of the accessed integer sample positions and the padded sample values.

16. the search space includes at least one fractional sample position with a value obtained by interpolation filtering with a filter of a predefined tap size; 16. The method of claim 15, wherein the interpolation filtering uses values ​​of the accessed integer sample positions and at least one of the padded sample values ​​to obtain values ​​for the at least one fractional sample position.

17. The method described in claim 16, wherein the padded sample values ​​used in the bilateral matching are determined by mirroring the values ​​of the corresponding evaluated integer sample positions relative to the axis of the filter.

18. A method as described in claim 16 or 17, wherein the padded sample value used in the bilateral matching is determined to be equal to the value closest to the accessed integer sample position.

19. The other window is the search space, a block having the size of the block located at the position of the initial motion vector; and the position of the initial motion vector 19. The method according to claim 15, wherein the vertical and / or horizontal size of at least one of the following is defined:

20. 20. The method of any one of claims 15 to 19, wherein the another window is the same as the window.

21. A computer readable storage medium storing instructions that, when executed on a processor, cause the processor to perform the method of any one of claims 3 to 20.

Images