Memory access window and padding for motion vector refinement and motion compensation

Abstract

The present invention provides a video image encoding and decoding apparatus and device for determining and refining motion vectors, and for determining predictors for motion compensation. [Solution] The video image encoding device includes: obtaining an initial motion vector of a block; obtaining a refined motion vector of the block based on the initial motion vector by performing a search in a search space located at a position defined by the initial motion vector; and determining a predictor of the block according to the refined motion vector, interpolation by an interpolation filter, at least one sample value of an integer sample position located inside the window, and padded sample values ​​for an integer sample position located outside the window for interpolation. The padded sample values ​​are based on at least one sample value of a sample located inside the window.

Description

Technical Field

[0001] Embodiments of the present invention relate to motion vector determination and refinement, as well as predictor determination for motion compensation, which can be used during video encoding and decoding.

Background Art

[0002] Current hybrid video coders use predictive coding. Pictures in a video sequence are subdivided into pixel blocks, and then these blocks are encoded. Instead of encoding each block pixel by pixel, the entire block is predicted using already encoded pixels that are spatially or temporally close to the block. The encoder further processes only the difference between the block and its prediction. Further processing typically includes a transformation of the block pixels to coefficients in a transform domain. Next, the coefficients can be further compressed by quantization and made more compact by entropy coding to form a bitstream. The bitstream further includes any signaling information that enables a decoder to decode the encoded video. For example, the signaling can include settings regarding encoder settings such as the size of the input picture, the frame rate, quantization step instructions, or the prediction applied to the blocks of the picture.

[0003] Temporal prediction uses the temporal correlation between pictures, also known as video frames. Temporal prediction is also called interpretation, because it uses dependencies between different video frames (inter-frames). Therefore, the block to be encoded, also called the current block, is predicted from one or more previously encoded pictures, also called reference pictures. Reference pictures are not necessarily pictures preceding the current picture; in the current picture, the current block is positioned in the order it appears in the video sequence. The encoder may encode pictures in an encoding order different from the display order. As a prediction for the current block, a block at the same position in the reference picture may be determined. A block at the same position is one that is positioned in the reference picture at the same location as the current block in the current picture. Such predictions are accurate for stationary picture regions, i.e., picture regions that do not move from one picture to another.

[0004] Motion estimation is typically used when determining the prediction for the current block in order to obtain a motion-compensated predictor. Thus, the current block is predicted by a block in the reference picture, positioned at a distance given by the motion vector from the position of a block at the same location. The motion vector may be signaled in the bitstream so that the decoder can determine the same prediction for the current block. To further reduce the signaling overhead caused by signaling the motion vector for each block, the motion vector itself may be estimated. Motion vector estimation may be performed based on the motion vectors of adjacent blocks in the spatial and / or temporal domain.

[0005] The prediction for the current block may be calculated using one reference picture, or by weighting predictions obtained from two or more reference pictures. Since adjacent pictures are very likely to be similar to the current picture, the reference pictures may be adjacent pictures, i.e., the picture immediately preceding and / or immediately following the current picture in display order. However, generally, the reference picture may be any other picture that is before or after the current picture in display order, and before the current picture in the bitstream (decode order). This can be advantageous, for example, if there is occlusion and / or non-linear motion in the video content. Therefore, the identification of the reference picture may also be signaled within the bitstream.

[0006] A special mode of interpretation is the so-called biprediction, where two reference pictures are used to generate the prediction for the current block. Specifically, the two predictions determined by each of the two reference pictures are combined with the prediction signal for the current block. Biprediction can result in a more accurate prediction of the current block than uniprediction, i.e., prediction using only one reference picture. More accurate predictions lead to smaller differences (also called "residuals") between the pixels of the current block and the predictions, which can be encoded more efficiently, i.e., compressed into a shorter bitstream. In general, to predict the current block, three or more reference pictures may be used to find three or more reference blocks, i.e., multi-reference interpretation may be applied. Thus, the term multi-reference prediction includes biprediction and predictions using three or more reference pictures.

[0007] To provide more accurate motion estimation, the resolution of the reference picture may be improved by interpolating samples between pixels. Fractional pixel interpolation can be performed by a weighted average of the nearest pixels. For 1 / 2 pixel resolution, for example, bilinear interpolation is typically used. Other fractional pixels are calculated as the average of the nearest pixels, weighted by the reciprocal of the distance between each nearest pixel and the predicted pixel.

[0008] Motion vector estimation is a computationally complex task in which the similarity between the current block and the corresponding predicted block (predictor), indicated by candidate motion vectors in a reference picture, is calculated. Typically, the search region contains M×M sample images, and each sample position of the M×M candidate locations is tested. The test involves calculating a similarity measure between N×N reference blocks C and blocks R placed at the tested candidate locations in the search region. For simplicity, the sum of absolute differences (SAD) is a method often used for this purpose and is calculated by the following formula:

number

[0009] In the above formula, x and y define candidate positions within the search region, and the indices i and j indicate samples within the reference block C and candidate block R. Candidate positions are often called block displacements or offsets and reflect the representation of block matching as a shift of the reference block within the search region and a calculation of the similarity between reference block C and the overlapping portion of the search region. To reduce complexity, the number of candidate motion vectors is usually reduced by restricting the candidate motion vectors to a certain search space. The search space may be defined, for example, by the number of pixels and / or positions enclosing the position in the reference picture corresponding to the position of the current block in the current image. After calculating the SAD for all M×M candidate positions x and y, the best-matching block R is the block at the position with the lowest SAD and corresponds to the greatest similarity with reference block C. On the other hand, candidate motion vectors may be defined by a list of candidate motion vectors formed by the motion vectors of adjacent blocks.

[0010] Motion vectors are typically determined, at least partially, on the encoder side and signaled to the decoder within the encoded bitstream. However, motion vectors may also be derived at the decoder level. In such cases, the current block is unavailable to the decoder and cannot be used to calculate the similarity of the candidate motion vector to the block it points to in the reference picture. Therefore, a template is used ("template matching"), constructed outside the pixels of an already decoded block rather than the current block. For example, an already decoded pixel adjacent to the current block may be used. This type of motion estimation has the advantage of reducing signaling, as the motion vector is derived in the same way at both the encoder and decoder, eliminating the need for signaling. On the other hand, the accuracy of such motion estimation may be lower.

[0011] In the case of bilateral prediction, instead of using templates, a comparison of candidate coded blocks in different reference pictures, pointed to by the candidate motion vectors, may be used ("bilateral matching"), which will be explained in more detail below.

[0012] To provide a trade-off between accuracy and signaling overhead, motion vector estimation may be divided into two steps: motion vector derivation and motion vector refinement. For example, motion vector derivation may include selecting a motion vector from a list of candidates. Such selected motion vectors may be further refined, for example, by searching within a search space. The search within the search space is based on calculating a cost function for each candidate motion vector, i.e., for each candidate position in the block pointed to by the candidate motion vector.

[0013] The paper "X. Chen, J. An, J. Zheng, JVET-D0029: Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching" (available at http: / / phenix.it-sudparis.eu / jvet / ) demonstrates motion vector refinement, in which a first motion vector with integer pixel resolution is found and further refined by a search at half-pixel resolution in the search space around the first motion vector.

[0014] To perform motion vector refinement, it is necessary to store at least these samples in memory, where the samples are those required for the current block to perform refinement, i.e., the samples corresponding to the search space, and the samples that may be accessed when template matching or bilateral matching is performed in the search space.

[0015] External memory access is a critical design parameter in this hardware architecture and / or software implementation. This is because accessing external memory is slower than using internal memory. On the other hand, internal memory on the chip is limited, for example, due to chip size implementation. [Overview of the Initiative] [Means for solving the problem]

[0016] This disclosure is based on the observation that, for example, motion vector refinement, when performed in combination with fractional interpolation, may require increased on-chip memory size or external memory access. Neither of these options may be desirable. The same is true when interpolation is applied to improve the predictor resolution by including fractional sample positions.

[0017] In consideration of the problems described above, this disclosure provides a motion vector prediction that takes into account the number of accesses to external memory and the number of samples that need to be made accessible for motion vector refinement of the motion vector for the encoded block, and calculates a predictor for the block.

[0018] This is achieved by padding (substituting) values ​​located outside a predefined memory access window for the purpose of motion vector refinement based on one or more samples from within the memory access window, and / or fractional interpolation using the substitution values.

[0019] This is achieved through the features of the independent claim.

[0020] According to an aspect of the present invention, an encoding device is provided for determining predictors for encoding blocks of a video image. The encoding device comprises a processing circuit configured to acquire an initial motion vector for a block, acquire a refined motion vector for the block based on the initial motion vector, and determine predictors for the block according to the refined motion vector. The processing circuit is configured to determine predictors according to the refined motion vector using interpolation with an interpolation filter, using at least one sample value of an integer sample position located inside a window and padded sample values ​​for an integer sample position located outside a window for interpolation filtering, the padded sample values ​​being based on at least one sample value of a sample located inside a window.

[0021] According to another aspect of the present invention, a decoding device is provided for determining predictors for decoding blocks of a video image. The decoding device comprises a processing circuit configured to acquire an initial motion vector for a block, acquire a refined motion vector for a predicted block based on the initial motion vector, and determine a predictor for the block according to the refined motion vector. The processing circuit is configured to determine the predictor according to the refined motion vector using interpolation with an interpolation filter, using at least one sample value of an integer sample position located inside a window and padded sample values ​​for an integer sample position located outside the window for interpolation filtering, the padded sample values ​​being based on at least one sample value of a sample located inside the window.

[0022] According to yet another aspect of the present invention, an encoding method is provided for determining predictors for encoding blocks of video images. The encoding method includes the steps of: obtaining an initial motion vector for a block; obtaining a refined motion vector for a block based on the initial motion vector; and determining a predictor for a block according to the refined motion vector, wherein the determining step determines the predictor according to the refined motion vector using interpolation with an interpolation filter, using at least one sample value for integer sample positions located inside a window and padded sample values ​​for integer sample positions located outside a window for interpolation filtering, the padded sample values ​​being based on at least one sample value for integer samples located inside a window.

[0023] According to yet another aspect of the present invention, a decoding method is provided for determining a predictor for decoding a block of video image. The decoding method includes the steps of: obtaining an initial motion vector for a block; obtaining a refined motion vector for a predicted block based on the initial motion vector; and determining a predictor according to the refined motion vector using interpolation with an interpolation filter, using at least one sample value of an integer sample position located inside a window and padded sample values ​​for an integer sample position located outside a window for interpolation filtering, the padded sample values ​​being based on at least one sample value of an integer sample located inside a window.

[0024] Such methods and apparatus have the advantage of limiting the number of samples available for predictor determination by interpolation filtering, while also avoiding further access to the storage device / (external) memory that stores the entire reference picture.

[0025] In the embodiment described above, the window is a window for interpolation filtering.

[0026] According to an embodiment, interpolation may include accessing sample values at integer sample positions within a window and / or padding at least one sample value for an integer sample position located outside the window with a padding value, which are based on at least one sample within the window and / or use the accessed integer position sample values and the padded sample values to perform interpolation filtering.

[0027] For example, the processing circuit may be configured to determine refinement of an initial motion vector based on a search space of candidate motion vectors and determine a refined motion vector based on the initial motion vector and the refinement.

[0028] Also, for example, the processing circuit may be configured to determine a search space based on an initial motion vector. More specifically, the search space may be located at a position defined by the initial motion vector. For example, the search space may be centered on the initial motion vector. More specifically, according to an example, the search space may include nine integer sample motion vector candidates, i.e., the initial motion vector and eight motion vector candidates arranged around it.

[0029] In an embodiment, the processing circuit is further configured to reconstruct a block based on a predictor. The reconstruction may be performed in a decoder device and in a decoder loop of an encoder device. Aspects of the present disclosure related to a method may include respective reconstruction steps of a block based on a predictor.

[0030] For example, the processing circuit may be configured to obtain a difference between a block and a predictor, also referred to as a "residual". More specifically, the processing circuit may be configured to reconstruct a block as a sum of a predictor and a residual, more specifically based on the predictor and the residual.

[0031] For example, padded sample values ​​(replacement values) are determined by mirroring the values ​​of the corresponding used (evaluated) integer sample positions with respect to the axis of the interpolation filter. The interpolation filtering may be one-dimensional filtering or separable two-dimensional filtering.

[0032] In one exemplary implementation, the window is formed by all integer sample positions accessible to obtain the predictor, at a position specified by the initial motion vector.

[0033] The window is defined by its vertical size and / or horizontal size relative to the size of a block placed at the initial motion vector position, the initial motion vector position, or the vertical and horizontal lengths of an interpolation filter used to determine the predictor. Defining a block "placed" at a particular position means, in embodiments, for example, referring to the top-left sample position of the block. Alternatively, another sample position may be referred to, or the block position may be defined by centering on a particular position. The processing circuit may be configured to define or determine the window position and / or size, the block size, and / or the vertical and horizontal lengths of the interpolation filter used to determine the predictor, according to the initial motion vector position.

[0034] As another example, the padded sample values ​​for integer sample positions outside the window are determined to be equal to the sample value of at least one used integer sample position located inside the window that is closest to the integer sample position located outside the window. In a particular example, for a one-dimensional interpolation filter, the padded sample values ​​for multiple integer sample positions located outside the window are used to determine the predictor using interpolation with the interpolation filter, and the padded sample values ​​for all integer sample positions outside the window are determined to be equal to the sample value of at least one used integer sample position located inside the window that is closest to the integer sample position located outside the window.

[0035] According to one embodiment, the number of padded sample values ​​depends on the difference between the initial motion vector and the refined motion vector.

[0036] According to the embodiment, the processing circuit is configured to determine the refinement of the initial motion vector by template matching with template matching or bilateral matching. The template matching or bilateral matching accesses sample values ​​at integer sample positions in another window. The other window includes a search space and another integer sample position accessible for template matching or bilateral matching, and the template matching or bilateral matching further pads at least one sample value at an integer sample position located outside the other window with a padded sample (replacement) value based on at least one sample in the other window, and performs template matching or bilateral matching using the values ​​at the accessed integer sample position and the padded sample value.

[0037] In one embodiment, the search space includes at least one fractional sample position with a value obtained by interpolation filtering using a predefined tap size filter, wherein the interpolation filtering obtains the value of the at least one fractional sample position using the value of the accessed integer sample position and at least one replaced (padding sample) value.

[0038] More specifically, the padded sample values ​​used in template matching or bilateral matching are determined by mirroring the values ​​of the corresponding evaluated integer sample positions with respect to the filter axis.

[0039] Alternatively, the padded sample values ​​used in template matching or bilateral matching are determined to be equal to the value closest to the accessed integer sample position.

[0040] According to one embodiment, another window is defined by its vertical and / or horizontal size relative to at least one of the search space, a block having the size of the block placed at the initial motion vector position, and the initial motion vector position.

[0041] Preferably, another window is the same as the window.

[0042] According to one embodiment, the refinement is determined by bilateral matching, and the processing circuit is configured to obtain at least two initial motion vectors that point to positions in different reference pictures and form a pair of initial motion vectors.

[0043] In another embodiment, refinement is determined by template matching, and the processing circuit is further configured to obtain a template of a block and determine the refinement of the initial motion vector by template matching using the template in the search space.

[0044] In one embodiment, the search space includes at least one fractional sample position with a value obtained by interpolation filtering using a predefined tap size filter, wherein the interpolation filtering obtains the value of the at least one fractional sample position using the value of the used integer sample position and at least one padded sample value.

[0045] According to an aspect of the present invention, a non-temporary computer-readable storage medium is provided that stores instructions for performing steps according to any of the above-described aspects or embodiments, or combinations thereof, when executed by a processor / processing circuit.

[0046] According to another particular aspect of the present invention, an encoding device is provided for encoding a video image divided into prediction blocks into a bitstream, or for encoding prediction blocks of a video image into a bitstream. The encoding device comprises a processing circuit for determining motion vectors for the prediction blocks. The processing circuit is configured to acquire at least one initial motion vector, determine a refinement of the initial motion vector based on a search space of candidate motion vectors, and acquire a refined motion vector. The search space is located at the positions given by the initial motion vector. The encoding device further comprises an encoding circuit (which may be included in the processing circuit) for encoding the difference between the prediction blocks and the predictors given by the prediction blocks at positions based on the determined motion vectors, and generating a bitstream containing the encoded difference and the initial motion vectors. The encoding device further comprises a motion prediction unit (which may be included in the processing circuit) for determining predictors according to motion vectors using interpolation filtering with an interpolation filter. The interpolation filtering accesses sample values ​​at integer sample positions within a predefined window, replaces at least one sample value at integer sample positions located outside the window with a replacement value based on at least one sample within the window, and performs interpolation filtering using the accessed integer position sample values ​​and replacement values.

[0047] According to yet another specific aspect of the present invention, a decoding device is provided for decoding a video image divided into prediction blocks from a bitstream. The decoding device comprises an analysis unit (which may be included in the decoding device's processing circuit) for analyzing from the bitstream an initial motion vector and an encoded difference between the prediction block and the predictor given by the prediction block at a position specified by the refined motion vector. The decoding device comprises a processing circuit (e.g., the decoding device's processing circuit) for determining the motion vector for the prediction block. The processing circuit is configured to acquire at least one initial motion vector, determine a refinement of the initial motion vector based on a search space of candidate motion vectors, and acquire a refined motion vector. The search space is located at a position given by the initial motion vector. Furthermore, the decoding device comprises a decoding circuit (which may be included in the decoding device's processing circuit) for reconstructing the prediction block as the sum of the analyzed difference and the predictor given by the prediction block at a position specified by the refined motion vector, which was determined by the processing circuit for determining the motion vector. The decoding circuit is further configured to determine the predictor according to the motion vector using interpolation filtering with an interpolation filter. Interpolation filtering accesses sample values ​​at integer sample positions within a predefined window, replaces at least one sample value at integer sample positions located outside the window with a replacement value based on at least one sample within the window, and then performs interpolation filtering using the accessed integer position sample values ​​and replacement values.

[0048] According to yet another aspect of the present invention, an encoding method is provided for encoding a video image divided into prediction blocks into a bitstream. The encoding method includes the step of determining motion vectors for the prediction blocks. The step of determining motion vectors includes the step of obtaining at least one initial motion vector, and the step of determining a refinement of the initial motion vector based on a search space of candidate motion vectors to obtain a refined motion vector. The search space is located at the positions given by the initial motion vectors. The encoding method further includes the step of encoding the difference between the prediction blocks and the predictors given by the prediction blocks at the positions based on the determined motion vectors, and generating a bitstream containing the encoded difference and the initial motion vectors. The method further includes the step of determining predictors according to the motion vectors using interpolation filtering with an interpolation filter. The interpolation filtering accesses sample values ​​at integer sample positions within a predefined window, replaces at least one sample value at integer sample positions located outside the window with a replacement value based on at least one sample within the window, and performs interpolation filtering using the accessed integer position sample values ​​and replacement values.

[0049] According to yet another aspect of the present invention, a decoding method is provided for decoding a video image divided into prediction blocks from a bitstream. The decoding method includes the step of analyzing from the bitstream an initial motion vector and an encoded difference between the prediction block and the predictor given by the prediction block at a position specified by the refined motion vector. The method further includes the step of determining the motion vector for the prediction block. The step of determining the motion vector includes the step of obtaining at least one initial motion vector and the step of obtaining a refined motion vector by determining a refinement of the initial motion vector based on a search space of candidate motion vectors. The search space is set at the position given by the initial motion vector. The decoding method further includes the step of reconstructing the prediction block as the sum of the analyzed difference and the predictor given by the prediction block at the position specified by the refined motion vector determined by the step of determining the motion vector. Furthermore, the method includes the step of determining the predictor according to the motion vector using interpolation filtering with an interpolation filter. Interpolation filtering accesses sample values ​​at integer sample positions within a predefined window, replaces at least one sample value at integer sample positions located outside the window with a replacement value based on at least one sample within the window, and then performs interpolation filtering using the accessed integer position sample values ​​and replacement values.

[0050] Further advantages and embodiments of the present invention are described in the dependent claims.

[0051] Exemplary embodiments are described in more detail below with reference to the attached drawings and figures. [Brief explanation of the drawing]

[0052] [Figure 1] This is a block diagram illustrating an exemplary encoder structure in which motion vector derivation and refinement may be used. [Figure 2]This is a block diagram illustrating an exemplary structure of a decoder in which motion vector derivation and refinement may be used. [Figure 3] This is a schematic diagram illustrating an example template matching suitable for biprediction. [Figure 4] This is a schematic diagram illustrating exemplary template matching suitable for single-prediction and bi-prediction. [Figure 5] This is a block diagram showing the steps of motion vector derivation, which operates without providing an initial motion vector to be refined within the bitstream. [Figure 6] This is a schematic diagram illustrating the concept of bilateral matching. [Figure 7] A block diagram showing exemplary hardware for carrying out embodiments of the present invention. [Figure 8] This is a schematic diagram showing an example window containing an encoded block and the sample that should be accessible. [Figure 9] This is a schematic diagram showing an iterative search space. [Figure 10] This is a schematic diagram illustrating the horizontal expansion of the memory access window through interpolation filtering. [Figure 11] This figure shows a conventional method that performs motion vector refinement based on the search space and obtains predictions through interpolation. [Figure 12] Figure 11 is a flowchart illustrating the derivation of motion vectors and the acquisition of predictions using a conventional example. [Figure 13] This figure shows a method for performing motion vector refinement based on a search space and obtaining a prediction by interpolation, according to an exemplary embodiment of the present invention. [Figure 14] This is a schematic diagram showing the padding of two samples to enable fractional position calculations. [Figure 15] This is a schematic diagram showing the padding of four samples to enable template matching at the search space boundary. [Figure 16] This is a schematic diagram illustrating exemplary fractional positions that do not require padding for interpolation. [Figure 17] Figure 13 is a flowchart illustrating motion vector refinement and prediction acquisition according to an exemplary embodiment. [Figure 18] This is a schematic diagram showing the definition of the memory access window for the prediction unit. [Figure 19] This figure shows the method according to the embodiment. [Figure 20] This figure shows a method, according to another exemplary embodiment of the present invention, for performing motion vector refinement based on a search space and obtaining predictions by interpolation. [Figure 21] Figure 20 is a flowchart illustrating motion vector refinement and prediction acquisition according to an exemplary embodiment. [Modes for carrying out the invention]

[0053] This disclosure relates to adjusting the number of samples, which are made accessible for performing motion vector refinement and interpolation, with the aim of obtaining fractional positions within a reference picture and predictor.

[0054] As mentioned above, external memory access is one of the most important design considerations in today's hardware and software architectures. Motion vector estimation may be used in conjunction with interpolation filtering to obtain fractional positions in the search space, particularly when template matching or bilateral matching is involved, for example, in the case of motion vector refinement. Using interpolation filtering may require increasing the number of samples that need to be accessed from memory. However, this can lead to an increase in expensive on-chip memory or an increase in the number of external memory accesses, and on the other hand, it can slow down implementation. Such problems are undesirable, especially on the decoder side, as they can lead to more expensive or slower operation.

[0055] To avoid such situations, this disclosure provides restrictions on external memory access to predefined windows.

[0056] The window may be of a predefined size. For example, in motion vector refinement, the window may include at least all search space positions. The accessed samples are those actually used for a particular block and its search space to perform template matching or bilateral matching, e.g., all samples used to calculate a cost function using a template or a bilateral cost function. Accessible samples further include samples that are not necessarily accessed during a particular template matching or bilateral matching for a particular block, e.g., because the search space has been iteratively determined to contain different samples, and the template matching or bilateral matching for those samples does not require all accessible pixels. Accessible samples further include samples that may be obtained from any iteratively determined search space (i.e., assume any block content), and possibly further include samples that are accessed to perform template matching in such a conceivable search space.

[0057] When generating the final prediction using interpolation filtering, the extent to which the memory access window required in conventional methods is extended depends on the maximum difference between the determined refined motion vector and the initial motion vector.

[0058] It should be noted that a window may be defined smaller than all accessible samples. Defining a window in this way makes it possible to maintain a smaller number of samples stored in on-chip memory and retrieved from external memory. Thus, this disclosure limits the number of integer samples stored / cached / buffered for operation relating to template matching or bilateral matching for a particular block. This does not mean that other samples are typically not present / available in another memory or storage device. As mentioned above, the entire reference picture may typically be stored in external memory. However, for template matching or bilateral matching, only a portion of it, i.e., the samples within the window, may be loaded and used for template matching.

[0059] By providing such a restricted window, it becomes possible to implement it using that window, and in practice, only a window sample needs to be loaded. To ensure that the encoder and decoder behave similarly, the window definition may be predefined as a standard or signaled within the bitstream. In particular, if refined motion vectors are used to form the reference picture, both the encoder and decoder must use the same technique.

[0060] Figure 1 shows an encoder 100 having an input for receiving input image samples of a video stream frame or picture, and an output for generating an encoded video bitstream. The term “frame” in this disclosure is used as a synonym for “picture.” However, it should be noted that this disclosure is also applicable to areas where interlacing may be applied. Generally, a picture contains m × n pixels. These correspond to image samples, each of which may contain one or more color components. For simplicity, the following description refers to pixels meaning luminance samples. However, it should be noted that the motion vector search of the present invention can be applied to any color component including chrominance or components of the search space such as RGB. On the other hand, it may be beneficial to perform motion vector estimation only on one component and apply the determined motion vector to more (or all) components.

[0061] The input blocks being encoded do not necessarily have the same size. A single picture may contain blocks of different sizes, and the block rasters of different pictures may also be different.

[0062] In a descriptive implementation, encoder 100 is configured to apply prediction, transformation, quantization, and entropy coding to a video stream. Transformation, quantization, and entropy coding are performed by the transformation unit 106, the quantization unit 108, and the entropy encoding unit 170, respectively, to produce an encoded video bitstream as output.

[0063] A video stream may contain multiple frames, each of which is divided into intra-encoded or inter-encoded blocks of a specific size. For example, a block of the first frame of a video stream is intra-encoded by an intra-prediction unit 154. Since intra-frames are encoded using only the information within the same frame, they can be decoded independently and provide entry points within the bitstream for random access. Blocks of other frames in the video stream may be inter-encoded by an inter-prediction unit 144, where information from a previously encoded frame (reference frame) is used to reduce temporal redundancy, so that each block of the inter-encoded frame is predicted from a block in the reference frame. A mode selection unit 160 is configured to select whether a block of a frame is processed by the intra-prediction unit 154 or the inter-prediction unit 144. This mode selection unit 160 also controls the parameters for intra-prediction or inter-prediction. To allow for image information refreshment, intra-encoded blocks may be provided within inter-encoded frames. Furthermore, intraframes containing only intra-encoded blocks may be periodically inserted into the video sequence to provide entry points for decoding, i.e., points from which the decoder can begin decoding without having information from previously encoded frames.

[0064] The intra-estimation unit 152 and intra-prediction unit 154 are units that perform intra-prediction. In particular, the intra-estimation unit 152 may further derive a prediction mode based on knowledge of the original image, and the intra-prediction unit 154 provides a corresponding predictor for differential coding, i.e., a sample predicted using the selected prediction mode. To perform spatial or temporal prediction, the coded blocks may be further processed by the inverse quantization unit 110 and the inverse transform unit 112. After the reconstruction of the blocks, the loop filtering unit 120 is applied to further improve the quality of the decoded image. The filtered blocks then form a reference frame, which is then stored in the decoded picture buffer 130. Such a decoder loop (decoder) on the encoder side has the advantage of producing the same reference frame as the reference picture reconstructed on the decoder side. Thus, the encoder side and the decoder side operate in corresponding ways. The term “reconstruction” here refers to obtaining the reconstructed blocks by adding the predicted blocks to the decoded residual blocks.

[0065] The interestimation unit 142 receives as input a block of the current frame or picture to be interencoded and one or more reference frames from the decoded picture buffer 130. The interestimation unit 142 performs motion estimation, and the interpretation unit 144 applies motion compensation. Motion estimation is used, for example, with the original image to be encoded to obtain motion vectors and reference frames based on a specific cost function. For example, the motion estimation unit 142 may provide an initial motion vector estimation. The initial motion vectors may then be signaled in the bitstream directly, in vector form, or in the same way as in encoders and decoders, as indices that refer to motion vector candidates in a candidate list built on predefined rules. Motion compensation then derives predictors for the current block by the motion vector, i.e., by transforming blocks that are identical in position to the current block in the reference frame to the reference blocks in the reference frame. The interpretation unit 144 outputs a predicted block for the current block, the predicted block minimizing the cost function. For example, the cost function may be the difference between the current block being encoded and its predicted block; that is, the cost function minimizes the residual block. Minimizing the residual block is based, for example, on calculating the sum of the absolute differences (SAD) between all pixels (samples) of the current block and the candidate block in the candidate reference picture. However, other similarity metrics such as the mean squared error (MSE) or the structural similarity metric (SSIM) may be used.

[0066] However, the cost function may also be the number of bits required to encode the distortion resulting from such interblock and / or such encoding. Thus, a rate-distortion optimization procedure may be used to determine encoding parameters such as the choice of motion vector and / or in general, whether to use inter-prediction or intra-prediction for the block and its settings.

[0067] The intra-estimation unit 152 and the intra-prediction unit 154 receive as input a block of the current frame or picture to be intra-encoded and one or more reference samples from an already reconstructed region of the current frame. The intra-prediction then describes the pixels of the current block of the current frame in terms of a function of the reference samples of the current frame. The intra-prediction unit 154 outputs a predicted block of the current block, which preferably minimizes the difference between the current block to be encoded and its predicted block, i.e., it minimizes the residual block. Minimization of the residual block may be based, for example, on a rate-distortion optimization procedure. In particular, the predicted block is obtained as a directional interpolation of the reference samples. The directional can be determined by rate-distortion optimization and / or by computing a similarity measure as mentioned above in relation to the intra-prediction.

[0068] The interpretation unit 142 receives as input an image sample in a block, or a more general form of the current frame or picture to be intercoded, and two or more already decoded pictures 231. The interpretation then describes the current image sample of the current frame in terms of motion vectors to reference the image sample of the reference picture. The interpretation unit 142 outputs one or more motion vectors of the current image sample, the reference image sample pointed to by the motion vector preferably minimizes the difference between the current image sample to be encoded and its reference image sample, i.e., it minimizes the residual image sample. Next, for differential coding, predictors of the current block are provided by the interpretation unit 144.

[0069] Next, the difference between the current block and its prediction, i.e., the residual block 105, is transformed by the transformation unit 106. The transformation coefficients 107 are quantized by the quantization unit 108 and entropy encoded by the entropy encoding unit 170. The encoded picture data 171 thus produced, i.e., the encoded video bitstream, includes intra-encoded blocks and inter-encoded blocks, as well as corresponding signaling (mode indication, motion vector indication, and / or intra-prediction direction, etc.). The transformation unit 106 may apply a linear transformation such as a Fourier or discrete cosine transform (DFT / FFT or DCT). Such a transformation to the spatial frequency domain has the advantage that the resulting coefficients 107 typically have higher values ​​at lower frequencies. Thus, after an effective coefficient scan (e.g., zigzag) and quantization, the resulting sequence of values ​​typically has some large values ​​at the beginning and ends with a run of zeros. This allows for even more efficient encoding. The quantization unit 108 performs virtually irreversible compression by reducing the coefficient values ​​of the resolution. Next, the entropy coding unit 170 assigns binary codewords to the coefficient values ​​in order to generate a bitstream. The entropy coding unit 170 also encodes signaling information (not shown in Figure 1).

[0070] Figure 2 shows the video decoder 200. The video decoder 200 comprises, in particular, a decoded picture buffer 230, an inter-prediction unit 244, and an intra-prediction unit 254, which is a block prediction unit. The decoded picture buffer 230 is configured to store at least one (for single prediction) or at least two (for dual prediction) reference frames reconstructed from the encoded video bitstream, the reference frames being different from the current frame (the frame currently being decoded) of the encoded video bitstream. The intra-prediction unit 254 is configured to generate a prediction block, which is an estimate of the block to be decoded. The intra-prediction unit 254 is configured to generate this prediction based on a reference sample taken from the decoded picture buffer 230.

[0071] The decoder 200 is configured to decode the encoded video bitstream generated by the video encoder 100, preferably with both the decoder 200 and the encoder 100 generating identical predictions for each block being encoded / decoded. The features of the decoded picture buffer 230, reconstructed block 215, buffer 216, and intra-prediction unit 254 are similar to those of the decoded picture buffer 130, reconstructed block 115, buffer 116, and intra-prediction unit 154 in Figure 1.

[0072] The video decoder 200 includes additional units also present in the video encoder 100, such as an inverse quantization unit 210, an inverse transform unit 212, and a loop filtering unit 220, which correspond to the inverse quantization unit 110, the inverse transform unit 112, and the loop filtering unit 120 of the video decoder 100, respectively.

[0073] The entropy decoding unit 204 is configured to decode the received encoded video bitstream and obtain correspondingly quantized residual transformation coefficients 209 and signaling information. The quantized residual transformation coefficients 209 are fed to the inverse quantization unit 210 and the inverse transformation unit 212 to generate residual blocks. The residual blocks are added to the prediction block 265, which is then fed to the loop filtering unit 220 to obtain the decoded video. The frames of the decoded video are stored in the decoded picture buffer 230 and can function as decoded pictures 231 for interprediction.

[0074] In general, the intra-prediction units 154 and 254 in Figures 1 and 2 may use reference samples from already encoded regions to generate prediction signals for blocks that need to be encoded or decoded.

[0075] The entropy decoding unit 204 receives the encoded bitstream 171 as its input. Generally, the bitstream is first parsed, i.e., the signaling parameters and residuals are extracted from the bitstream. Typically, the syntax and semantics of the bitstream are defined by standards so that the encoder and decoder can operate in an interoperable manner. As described in the background technology section above, the encoded bitstream does not contain only the prediction residuals. In the case of motion-compensated prediction, motion vector instructions are also encoded in the bitstream and parsed from there by the decoder. Motion vector instructions can be given by motion vector coordinates, provided by the reference picture from which the motion vectors are given. Up to this point, we have considered the encoding of the complete motion vector. However, only the difference between the current motion vector and the previous motion vector may be encoded in the bitstream. This technique can take advantage of redundancy between motion vectors of adjacent blocks.

[0076] To efficiently encode reference pictures, the H.265 codec (ITU-T, H265, Series H: Audiovisual and multimedia systems: High Efficient Video Coding) provides a list of reference pictures to which each reference frame is assigned a list index. Reference frames are then signaled in the bitstream by including their corresponding assigned list index. Such lists may be standardly defined or signaled at the beginning of a video or a set of frames. Note that H.265 has two defined lists of reference pictures, called L0 and L1. Reference pictures are then signaled in the bitstream by indicating the list (L0 or L1) and the index in that list associated with the desired reference picture. Providing more than one list can be advantageous for better compression. For example, L0 may be used for both unidirectional interpredicted slices and bidirectional interpredicted slices, while L1 may be used only for bidirectional interpredicted slices. However, this disclosure is not limited in general to anything in the L0 and L1 lists.

[0077] Lists L0 and L1 may be defined and fixed by default. However, greater flexibility in encoding / decoding can be achieved by signaling them at the beginning of the video sequence. Therefore, the encoder may construct lists L0 and L1 using specific reference pictures ordered according to their indices. The lists L0 and L1 may have the same fixed size. In general, there may be three or more lists. Motion vectors may be signaled directly by their coordinates in the reference pictures. Alternatively, a list of candidate motion vectors may be constructed, as specified in H.265, and the index associated with a particular motion vector in the list may be transmitted.

[0078] The motion vector of the current block typically correlates with the motion vectors of adjacent blocks within the current picture or in previously encoded pictures. This is because adjacent blocks are likely to correspond to the same moving object with similar motion, and the object's motion is unlikely to change abruptly over time. As a result, using the motion vectors in adjacent blocks as predictors reduces the magnitude of the signaled motion vector difference. The motion vector predictor (MVP) is typically derived from already encoded / decoded motion vectors from spatially adjacent blocks, or from temporally adjacent or identically located blocks within the reference picture. In H.264 / AVC, this is done by taking the component-wise median of three spatially adjacent motion vectors. Using this method, signaling of the predictor is not required. Temporal MVPs from identically located blocks within the reference picture are only considered in the so-called temporal direct mode of H.264 / AVC. H.264 / AVC direct mode is also used to derive other motion data besides motion vectors. Therefore, they are more closely related to the concept of block merging in HEVC. In HEVC, the method of implicitly deriving the MVP was replaced by a technique known as motion vector race, which explicitly signals which MVP to use from a list of MVPs for motion vector derivation. The variable coding quadtree block structure in HEVC can be a single block having several adjacent blocks containing motion vectors as potential MVP candidates. Taking the left neighbor as an example, in the worst case, if a 64x64 Luma coding tree block is not further subdivided and the block to the left is subdivided to its maximum depth, a 64x64 Luma prediction block may have 16 4x4 Luma prediction blocks to its left.

[0079] To correct motion vector races to account for such flexible block structures, Evolutionary Motion Vector Prediction (AMVP) was introduced. During the development of HEVC, the initial AMVP design was significantly simplified to provide a good trade-off between coding efficiency and a design that is easy to implement. The initial AMVP design included five MVPs from three different classes of predictors: three motion vectors from spatially adjacent ones, the median of three spatial predictors, and a scaled motion vector from a temporally adjacent block at the same location. Furthermore, the list of predictors was modified by rearranging the most likely motion predictor to the first position and removing redundant candidates to ensure minimal signaling overhead. The final design of the AMVP candidate list structure includes the following two MVP candidates: a) up to two spatial candidate MVPs derived from five spatially adjacent blocks, b) one temporal candidate MVP derived from two temporally adjacent blocks at the same location if both spatial candidate MVPs are unavailable or are identical, and c) zero motion vectors if no spatial, temporal, or both candidates are available. Further details regarding motion vector determination can be found in Chapter 5 of the reference "V. Sze et. al. (Ed.), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014," which is incorporated herein by reference.

[0080] To further improve motion vector estimation without further increasing signaling overhead, it would be beneficial to further refine the motion vector derived on the encoder side and provided in the bitstream. Motion vector refinement may be performed in the decoder without assistance from the encoder. The encoder within that decoder loop may use the same refinement to obtain the corresponding motion vector. Motion vector refinement is performed in the search space, which includes integer and fractional pixel positions of the reference picture. For example, a fractional pixel position may be a 1 / 2 pixel position, a 1 / 4 pixel position, or another fractional position. Fractional pixel positions may be obtained from integer (full pixel) positions by interpolation such as bilinear interpolation. However, this disclosure is equally applicable to search spaces that do not include fractional pixel positions.

[0081] In the current biprediction of blocks, two predicted blocks obtained using the first motion vector from list L0 and the second motion vector from list L1 are combined into a single predicted signal. This allows for better adaptation to the original signal than with uniprediction, potentially resulting in less residual information and more efficient compression.

[0082] In the decoder, the current block is unavailable during decoding, so a template is used for motion vector refinement. The template is an estimate of the current block and is built on already processed (i.e., encoded on the encoder side and decoded on the decoder side) image portion.

[0083] First, the estimation of the first motion vector MV0 and the estimation of the second motion vector MV1 are received as input to the decoder 200. On the encoder side 100, the motion vector estimates MV0 and MV1 may be obtained by block matching and / or by searching within a list of candidates (such as a merge list) formed by the motion vectors of blocks adjacent to the current block (within the same picture or in adjacent pictures). MV0 and MV1 are then signaled in the bitstream in the decoder's favor. However, it should be noted that the first decision stage in the encoder may also generally be performed by template matching, which has the advantage of reducing signaling overhead.

[0084] On the decoder side 200, motion vectors MV0 and MV1 are preferentially obtained based on information in the bitstream. MV0 and MV1 are either directly signaled or differentially signaled, and / or their indices in a list of motion vectors (merge list) are signaled. However, this disclosure is not limited to signaling motion vectors in the bitstream. Rather, the motion vectors may already be determined by template matching in the first stage in response to the operation of the encoder. The template matching in the first stage (motion vector derivation) may be performed on a different search space than the search space of the second refinement stage. Specifically, the refinement may be performed in a search space with higher resolution (i.e., shorter distances between search positions).

[0085] The instructions for the two reference pictures RefPic0 and RefPic1, pointed to by MV0 and MV1 respectively, are also provided to the decoder. The reference pictures are stored in decoded picture buffers on both the encoder and decoder sides as a result of previous processing, namely encoding and decoding, respectively. One of these reference pictures is selected for motion vector refinement by search. The reference picture selection units of the encoder and decoder are configured to select a first reference picture pointed to by MV0 and a second reference picture pointed to by MV1, respectively. Following the selection, the reference picture selection unit determines whether the first or second reference picture will be used for performing motion vector refinement. To perform motion vector refinement, a search region is defined in the first reference picture around the candidate position pointed to by the motion vector MV0. The candidate search space positions within the search region are analyzed to find the block most similar to the template block by performing template matching in the search space and determining a similarity metric such as the sum of absolute differences (SAD). The search space position represents the location where the top-left corner of the template matches. As already mentioned above, the top-left corner is merely a convention, and any point in the search space, such as the center point, may commonly be used to represent the matching location.

[0086] According to the aforementioned document JVET-D0029, the decoder-side motion vector refinement (DMVR) takes initial motion vectors MV0 and MV1 as input, which point to two reference pictures, RefPict0 and RefPict1. These initial motion vectors are used to determine the search space for RefPict0 and RefPict1. Furthermore, using the motion vectors MV0 and MV1, a template is constructed based on each block A and B (of the sample) pointed to by MV0 and MV1, as follows: Template = Function ((Block A, Block B))

[0087] The function may be a sample clipping operation combined with a weighted sum of samples. The template is then used to perform template matching in a search space determined based on MV0 and MV1 in each reference picture 0 and 1. The cost function for determining the best template match in each search space is SAD(Template, Block candA'), where block candA' is the candidate coded block pointed to by the candidate MV in the search space spanning the positions given by MV0. Figure 3 shows the determination of the best-matching block A' and the resulting refined motion vector MV0'. Correspondingly, as shown in Figure 3, the same template is used to find the best-matching block B' and the corresponding motion vector MV1' pointing to block B'. ​​In other words, after the template is constructed based on blocks A and B pointed to by the initial motion vectors MV0 and MV1, the refined motion vectors MV0' and MV1' are found by searching RefPic0 and RefPic1 in the template.

[0088] The motion vector derivation technique is sometimes also called frame rate up-conversion (FRUC). Initial motion vectors MV0 and MV1 are typically shown in the bitstream to ensure that encoders and decoders can reliably use the same initial points for motion vector refinement. Alternatively, the initial motion vectors may be obtained by providing a list of initial candidates, each containing one or more initial candidates. A refined motion vector is determined for each of these, and ultimately, the refined motion vector that minimizes the cost function is selected.

[0089] It should be further noted that the present invention is not limited to the template matching described above with reference to Figure 3. Figure 4 shows an alternative template matching that can also be applied to one-sided prediction. For more details, see reference JVET-A1001, in particular "Section 2.4.6. Pattern matched motion vector derivation" by Jianle Chen et al. in reference JVET-A1001, titled "Algorithm Description of Joint Exploration Test Model 1," which can be accessed at http: / / phenix.it-sudparis.eu / jvet / . In this template matching method, the template is determined as a sample adjacent to the current block in the current frame. As shown in Figure 1, an already reconstructed sample adjacent to the top and left boundaries of the current block may be taken, and this is called an "L-shaped template."

[0090] Figure 5 shows another type of motion vector derivation that may be used. The input to the motion vector derivation process is a flag indicating whether motion vector derivation is applicable. Implicitly, another input to the derivation process is the motion vector of a previously encoded / reconstructed block that is adjacent (temporally or spatially). Motion vectors of multiple adjacent blocks are used as candidates for the initial search step of motion vector derivation. The outputs of the process are, respectively, MV0' (possibly MV1' if bi-prediction is used), and the corresponding reference picture index refPict0, and possibly refPict1. The motion vector refinement step then involves template matching as described above. After finding one (single-prediction) or two or more (bi-prediction / multi-frame prediction) refined motion vectors, predictors for the current block are constructed (for bi / multi-prediction, by weighted sample prediction, or by referring to the sample pointed to by the refined MV if not).

[0091] The present invention is not limited to the two matching methods described above (examples of template matching). For example, a third matching method called bilateral matching (also described in reference JVET-A1001) may be used for motion vector refinement, and the present invention applies similarly. Bilateral matching is illustrated with reference to Figure 6.

[0092] Bilateral matching searches for the best match between two blocks along the motion trajectory of the current block in two different reference pictures. As shown in Figure 6, bilateral matching is used to derive motion information for the current block by finding the closest match between two blocks along the motion trajectory of the current block in two different reference pictures. In bilateral matching, a cost function such as SAD(Block cand0', Block cand1') may be used where MV0 points to Block cand0' and MV1 points to Block cand1'.

[0093] To reduce computational costs, a cost function is calculated and compared for selecting appropriate candidate motion vectors (candidate pairs), which is useful in considering that the following conditions are met: Assuming a continuous motion trajectory, motion vectors MV0 and MV1 pointing to two reference blocks are proportional to the temporal distance between the current picture and the two reference pictures, i.e., TD0 and TD1. In a special case, when the current picture is temporally between two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, the difference between the motion vectors of multiple pairs referencing one reference picture and the motion vectors of pairs referencing other reference pictures are mirrored to each other.

[0094] Figure 7 shows a processing circuit 600. The processing circuit may include arbitrary hardware, and its configuration may be implemented by any type of programming, or a combination of both, in a hardware design. For example, the processing circuit may be formed by a single processor, such as a general-purpose processor containing corresponding software that performs the steps described above. On the other hand, the processing circuit may be implemented by specialized hardware, such as a DSP (Digital Signal Processor) ASIC (Application-Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0095] The processing circuit may include one or more of the hardware components described above, interconnected to perform the motion vector derivation described above. The processing circuit 600 includes computational logic that performs two functions: obtaining an initial motion vector (or multiple initial motion vectors if dual / multiple predictions are used) and a template if template matching is used 610, and refining the motion vector 620. These two functions may be performed on the same hardware, or on separate hardware units, such as the initial motion vector and template determination unit 610 and the motion vector refinement unit 620. The processing circuit 600 may be communicatively connected to an external memory 650 where reconstructed reference picture samples are stored. Furthermore, the processing circuit 600 may further include an internal memory 640 that buffers samples in a window, which are transferred from the external memory and used for determining the motion vector of the block currently being processed. The processing circuit may be embodied on a single chip as an integrated circuit.

[0096] It should be noted that the processing circuit may perform other functions of the encoder and / or decoder, as described with reference to Figures 1 and 2. Internal memory may be on-chip memory such as cache or line memory. Chip memory is preferably implemented in the encoder / decoder chip to speed up computation. On-chip memory is usually small due to the size limitations of the chip. External memory, on the other hand, can be very large, but accessing it consumes more energy and access is considerably slower. Typically, all the necessary information is transferred from external memory to on-chip memory before computation is performed. The worst-case external memory access (or the bandwidth that must be provided when designing the memory bus) represents the maximum amount of memory transfer that may occur between the external memory and the chip during the decoding of a frame or coded unit. Memory (especially external memory) can usually only be accessed within predefined block units. In other words, it is usually not possible to access a single pixel, but must be accessed in the smallest units (e.g., 8x8). The size of on-chip memory is also an important design consideration, as larger on-chip memory is more expensive.

[0097] In other words, the device described above may further include an integrated circuit comprising an internal memory embedded within the integrated circuit and a memory access unit (interface) that retrieves integer samples placed within the window from external memory into the internal memory.

[0098] The term “predicted block” as used above refers to the predicted current block. Blocks in an image can be obtained by subdividing the image into equal or different sizes (for example, by hierarchical partitioning of coding tree units to make CTUs smaller unit blocks). Blocks may be square or more generally rectangular, as these are typical shapes used by current encoders / decoders. However, this disclosure is not limited to any size / shape of the block.

[0099] The device equipped with the processing circuit may be an encoder or decoder, or it may be a device equipped with such an encoder or decoder, for example, a recording device and / or a playback device.

[0100] Fractional sample positions are the positions between actual picture sample positions obtained by reconstructing the encoded reference picture, as shown in Figure 1. Therefore, fractional positions must be obtained by interpolation based on the nearest integer position. Details of interpolation filtering can be found in Section 5. 3 Fractional Sample Interpolation of the paper "V. Sze et. al., High Efficiency Video Coding (HEVC), Springer, 2014".

[0101] Interpolation filtering is typically applied to another filter to generate a different fractional Pell (sample) position. As an example, the following 1D divisible filter is applied to generate 1 / 4 Pell and 1 / 2 Pell positions in the H.265 video compression standard.

[0102] [Table 1]

[0103] As seen in the table above, interpolation filtering requires several samples around a fractional Pell position, corresponding to the filter taps (the number of coefficients in the table). To generate a 1 / 2 Pell position, using the exemplary filter described above, four integer samples are required from the left / top and right / bottom. Note that the length of the interpolation filter differs between a 1 / 4 Pell sample position (7 taps) and a 1 / 2 Pell sample position (8 taps). The terms sample, Pell, and pixel are used interchangeably in this application and represent an image sample (in one or more color components) in a given time instance. A sample is, in principle, an intensity value.

[0104] In some embodiments of the present invention, an interpolation filter of a predefined tap size evaluates integer samples only within a window given by integer samples accessible for template matching in the search space. The window may contain many more samples than are actually used to calculate a given number of prediction blocks. This is because the refined search operation is typically performed using a fast search method (as opposed to a brute-force search method), which means that some samples are not evaluated in proportion to the stepwise progress of the search operation. As a result, the number of iterations of template matching may vary from prediction block to prediction block, as well as the number of samples used for calculations for the refined search operation.

[0105] Figure 8 shows a sample of the coded block (prediction block) and the corresponding window. Note that the sample shown in Figure 8 is a reference picture sample, and the coded block here is actually a block whose size and position correspond to the current block in the current frame, from which a motion vector is derived in the reference picture. Therefore, in reality, the coded block in Figure 8 is, in short, the block at the same position as the block in which the predictor is searched. However, for the sake of brevity, this block will be referred to as the "coded block" below.

[0106] In this example, the unrefined motion vector MV0 points to an integer sample position. The search granularity for motion vector refinement is one integer sample, meaning that only integer sample points are explored, since the starting point is an integer sample. In this example, the search is performed in a gradually expanding search space. This means that the search space progresses in each iteration by adding new search positions depending on the best direction, in terms of the cost function of previously tested positions.

[0107] This method is simplified and shown in Figure 9. In Figure 9, the initial motion vector points to the center point 810. The search space is gradually constructed around the initial motion vector position. In the first step, four positions immediately above, immediately below, immediately to the left, and immediately to the right of position 810, pointed to by the initial motion vector, are tested in the same way as position 810. Another position to be tested is added to the search space based on the direction in which the cost function is lowest among the five tested points. In this example, the lowest cost function was found at the point to the right, so in the second step, the search space is expanded three more points horizontally to the right. In the second step, the lowest cost function is found at the point to the right (relative to the lowest cost point in the first step), resulting in the search space being expanded three more points horizontally to the right. In the third step, the lowest cost function is again found at the point to the right of the lowest cost point in step 2, resulting in the search space being expanded three more points horizontally to the right. In the example in Figure 9, three more steps are performed in the order of upward, upward, and rightward. In the example, a diamond pattern (consisting of five search points) is used for each iteration, and the search space is expanded in each step to complete any missing search points.

[0108] In each iteration of the search space determination, the search space can be increased by one or more integer sample positions. Returning to Figure 8, in this example, the maximum number of search iterations is 4. Since a maximum of 4 iterations are possible, all samples shown on the left must be retrieved from memory to perform the search operation, in which case the expansion of the search space gradually progresses to the left. Similarly, 4 samples need to be expanded upwards. Thus, since the refined MV can move in either direction, the search space expands in both directions (left to right and top to bottom), and hardware implementation requires that all potentially needed samples be retrieved from external memory before the refined search is applied. If the search space expands downwards or to the right, template matching with templates corresponding to the size of the coded blocks (prediction blocks) will need to access some of these samples, requiring an expansion of another 4 samples. Furthermore, since hardware implementation usually cannot retrieve irregular shapes (as rectangular access is more feasible), samples in the corners (e.g., top right) must also be retrieved from memory.

[0109] The iterative search space expansion described above is merely an example, and it should be noted that the rules and number of points for expanding the search space may differ in each iteration, i.e., they may be specified in different ways.

[0110] Figure 9 also illustrates a scenario that can arise from the external memory access rules described above. The number of samples retrieved from external memory is much greater than the number of samples actually used in the computation step. Assuming the template here is (for simplicity) only the size of one sample, the white circles represent samples retrieved from external memory, and the shaded samples represent those actually used. However, since the actual number of samples needed is not yet known when processing of the current block begins, such redundancy is necessary if it is necessary to keep the number of accesses to external memory low.

[0111] It should be noted that the search space may also be defined in another way, for example, as a stable shape placed at the position indicated by the initial motion vector. The shape may be any shape, such as a square, rectangle, or rhombus.

[0112] Figure 10 shows an example where the search space may further include fractional samples. In Figures 8 and 9, the motion vector search is performed with integer samples, resulting in the positions indicated by the larger solid lines within the access window. Here, the search is performed on samples with 1 / 2 Pell resolution (the smaller solid lines), and three more columns of samples must be retrieved from memory to generate the fractional samples shown on the left, assuming the interpolation filter is symmetric and has eight taps. Furthermore, because the search operation is symmetric (it can move iteratively left and right), the same must be applied to the left side (an extension of three columns of pixels), and as a result, fractional pixels may be located on the left side of the window.

[0113] As a result, interpolation filtering further increases the number of samples to be retrieved from memory, and includes dotted circles representing positions added by fractional interpolation, as shown by the dashed lines. Similarly, if we allow searching by half for each vertical position, the window of samples accessed from memory needs to be expanded vertically both upwards and downwards (not shown in the example in Figure 10).

[0114] The memory access window is defined as a rectangle enclosing all the samples that need to be retrieved from memory in order to perform the motion vector search on the prediction block (encoded block). The memory access window includes not only the actual samples that are needed, but also all the remaining samples that may be accessed during the motion vector search operation. In the example in Figure 9, the motion vector search was moving to the right. However, it could also have been moving to the left, which is not known in advance. Therefore, to avoid repeatedly accessing external memory, the memory access window (or access window) includes all the samples that can be accessed in each operation.

[0115] To achieve this, the disclosure, for example, performs padding of samples located outside the memory access window if necessary to interpolate fractional samples.

[0116] Providing replacement samples (padding samples) may further enhance the relevance to template matching in a search space that includes more fractional samples. In other words, the search space for prediction blocks may include at least one fractional sample position with values ​​obtained by interpolation filtering with a predefined tap size filter. Interpolation filtering uses a window and the sample position values ​​from at least one replacement value to obtain the values ​​of at least one fractional sample position.

[0117] Figure 11 shows the basic steps of a known method for performing motion vector derivation / refinement, as outlined earlier. In the first step (Step 1), a “search space” of at least two candidate motion vectors is obtained (in the illustrated example, there are nine candidate motion vectors). Optionally, one of the candidate motion vectors forming the search space may be considered the initial motion vector (also called the “anchor” motion vector or anchor candidate motion vector, in the terminology used herein). The initial or anchor motion vector may be, for example, a motion vector contained in a bitstream and explicitly or implicitly signaled to the decoder.

[0118] In the next step (Step 2), the matching process is applied to the search space to find the best-matching point in the search space (in other words, the point in the search space corresponds to the most appropriate candidate motion vector that minimizes the cost function). In the illustrated example, bilateral matching is applied, but other methods for determining the cost function, such as template matching, can be applied as well.

[0119] The final step (step 3) of the drawing is to obtain the final prediction by interpolation based on the most appropriate candidate motion vector selected according to the processing in step 2.

[0120] The concept in Figure 11 is further illustrated in the flowchart in Figure 12. Figure 12 specifically shows an example of candidate motion vector pairs, i.e., a diagram for bilateral matching (which is also applicable when template matching is used for bidirectional prediction as described above, for example, with reference to Figure 3).

[0121] In the first step (1310), a search space for candidate motion vector pairs is constructed. This step corresponds to "Step 1" in Figure 11.

[0122] In the next step (1320), predictions are generated for each motion vector pair in the search space by applying interpolation filtering. In the next step (1330), a template is constructed (obtained) for each motion vector pair in the search space. As mentioned above, this step is optional, i.e., it is only necessary for template matching and not for bilateral matching. In the next step (1340), the template matching cost (or bilateral matching cost) is calculated for each motion vector pair in the search space by comparing the prediction block pointed to by the two motion vectors of the pair (or the prediction block of one candidate motion vector, and the template). In the next step (1370), the template or bilateral cost function is minimized (more generally, the best-matching, i.e., obtained as the most appropriate by the matching process), and the resulting motion vector pair is selected and output as the motion vector pair of prediction blocks. According to the terminology above, this may also be called the "refined motion vector pair". Steps 1320, 1330, 1340, and 1370 mentioned above correspond to "Step 2" in Figure 11.

[0123] The next step (1380) generates the final prediction by interpolation using the motion vector pair selected in step 1370. This step corresponds to "Step 3" in Figure 11.

[0124] Figure 13 shows a process for motion vector refinement and final predictor generation according to an exemplary embodiment of the present invention. The main difference between the method in Figure 13 and the conventional method shown in Figure 11 is that the method in Figure 13 of the present invention uses sample padding for both motion vector refinement and final predictor generation using interpolation filtering.

[0125] An additional step 2 shows that (optionally) padded samples may be used for motion vector refinement, which will be discussed in more detail below with reference to Figures 14 to 18. This allows candidate blocks desired for bilateral matching or template matching to be performed, even in the case of candidate motion vectors with fractional coordinates (i.e., fractional coordinates in the search space), by using a predefined restricted access window, which is the same as in the case of a search space containing only integer points. The actual derivation of the cost function by applying bilateral matching or template matching is performed in step 3, which is the same as step 2 in Figure 11.

[0126] Furthermore, sample padding is used (again, or only here) in the final step 4 to perform the actual predictor generation. This step generally corresponds to step 3 in Figure 11, but differs in its use of the concept of padding. Specifically, in the given example, the sample values ​​located in the two rightmost columns and the bottom row are not accessed, but are padded with padding sample values ​​based on the other accessed sample values.

[0127] The number of samples that are padded (replaced with replacement values ​​because they are unavailable) depends on the difference between the last selected ("refined") motion vector and the reference initial motion vector ("anchor"). In the illustrated example, this difference is 2 pixels vertically and 1 pixel horizontally. Accordingly, in the figure of step 5, the sample values ​​of the two columns to the right of the pixels and the pixels in the last row cannot be used to interpolate the pixels at the intersection of the two rectangles horizontally and vertically, and must be replaced (padding) with replacement values ​​according to the predefined rules described above, which will be explained in more detail below.

[0128] Preferably, if padding is also used in the search in step 2, the same access window is used for both the interpolation filtering operations in step 2 and step 4.

[0129] Preferably, the memory access window used in steps 2 and 4 of Figure 13 is determined based on a specific candidate motion vector in the search space. This specific motion vector (called the anchor motion vector) may be, for example, a motion vector contained in the bitstream that is explicitly or implicitly signaled to the decoder. The selection of the anchor motion vector is carried out according to predefined rules known to both the encoder and the decoder.

[0130] Figure 14 shows a memory access window for motion vector refinement. The window extends around a central point 1010, which is the position indicated by the initial motion vector. Furthermore, a fractional position 1020 also belongs to the search space. However, the fractional position 1020 is interpolated by six tap filters using the three integer sample positions to the left of the fractional position and the three integer sample positions to the right of the fractional position. Since the fractional position 1020 is collinear with the integer sample positions, a one-dimensional filter may be applied to the interpolation, as shown in the example described above with reference to Figure 10, for example. As seen in Figure 14, two further sample positions 1030 are needed to interpolate the fractional sample 1020, and these are located outside the window. To avoid further loading of these samples from external memory, according to this disclosure, the values ​​of the out-of-window positions 1030 are replaced with values ​​obtained based on the samples within the window.

[0131] Figure 14 shows a fractional sample position 1020, which lies midway between two integer sample positions and is therefore at a 1 / 2 pixel (1 / 2 pelle) position. However, this disclosure is equally applicable to other fractional positions, such as 1 / 4 or 1 / 8 positions. Furthermore, sample position 1020 may be located on a horizontal line corresponding to a row of integer sample positions, and as a result, horizontal 1D filtering may be used for its interpolation. However, this disclosure is not limited to such fractional positions. Fractional positions may also be located on perpendicular lines corresponding to columns of integer sample positions, and as a result, vertical 1D filtering may be used for their interpolation. Furthermore, fractional positions do not need to be located on the same line of pixels as the integer samples. In such cases, two-dimensional filtering may be used to interpolate such samples, and such 2D filtering may be divided into vertical 1D filtering and horizontal 1D filtering.

[0132] Figure 16 shows examples of various fractional positions. In particular, fractional position 1 is an interpolation position to which both horizontal and vertical filtering can be applied because it is not located collinearly with the integer sample positions in rows and columns. Position 2 is an interpolation position to which only horizontal filtering is used, and position 6 is an interpolation position to which only vertical filtering is used. All fractional positions shown in Figure 16 can be obtained using only integer positions within the window.

[0133] The memory access window may be defined in various ways. Figure 18 shows an example where the memory access window is defined as an extension portion EXT to the left / right or above / below the coded block (corresponding to the coded block's position given by the initial motion vector). The amount of extension may depend on the size and shape of the coded or predictive block. In Figure 18, the extension portion is four samples long in each direction (up, down, left, right). However, it should be noted that EXT may take different values ​​for different directions depending on the block size (which may differ vertically and horizontally) and / or the form and size of the search space.

[0134] For example, according to the example, the window is defined as N integer sample columns and M integer sample rows with respect to the initial motion vector of the predicted block, where N and M are non-zero integer values. In Figure 18, N and M are shown, but their sizes are the same. As mentioned above, N and M may have different sizes. N and M are integers, and at least one of them is non-zero. The size of the access window can be determined by taking the parameters N and M, as well as the form and size of the template. In particular, if the template has T1 rows and T2 columns, the size of the memory access window can be calculated as (N+T1+N) rows and (M+T2+M) columns. This is because the search can move N samples to the left or right, resulting in 2N samples horizontally, and M samples up or down, resulting in 2M samples vertically. The template can also be matched with T1×T2 samples at any position in the 2N×2M sample area.

[0135] On the other hand, for certain search space construction methods as described with reference to Figures 8 and 9, the memory access window may also be defined in terms of the maximum number of refinement iterations (search space construction iterations) and the iteration step size (relative to the maximum sample distance achievable in each iteration), which can later be translated into the maximum displacement to the left, right, up, and down. Thus, the memory access window is defined as the maximum displacement in each direction. For example, if in four iterations each iteration can advance the search space by a maximum of one integer sample position, then EXT = 4.

[0136] In other words, according to this example, the processing circuit is configured to determine the refinement of the initial motion vector by template matching using the template in an iteratively expanded search space in a direction given by one (or more) of the best-matching positions in the search space in the most recent iteration, the window being defined by a predefined maximum number of iterations. The same applies to bilateral matching, with the necessary modifications.

[0137] In general, it should be noted that this disclosure is not limited to any particular shape or form, or type of determination of the search space. In another example, the search space may be a rectangular subwindow of the window, or equal to the window. The definition of the memory access window may be important because it may include corresponding signaling parameters (e.g., parameter EXT, or parameters N and M) within the bitstream. However, the size of the memory access window may be defined by standard, or it may be derivable based on other coding parameters (such as template size, prediction block size, image resolution, etc.). Returning to Figure 14, according to this embodiment, the window of memory access for refinement is defined around position 1010 pointed to by the unrefined motion vector. The window identifies the maximum number of pixel samples that need to be accessed from memory in order to perform a motion vector refinement search. The search space here coincides with the access window. In other words, for simplicity in this example, the template size is given as 1x1, but it can and does usually be larger. The window of memory access is typically defined as an extension around a coding block. In this example, the extension consists of 4 samples from the left / right and 4 samples from the top / bottom. If the search space position currently being tested during template matching or bilateral matching requires samples from outside the window of memory access, the samples needed to reconstruct the coded block are obtained through padding. This is an example of fractional position 1020 in Figure 14. If template matching is performed at or using this position, it must be interpolated using another padding position 1030 located outside the window.

[0138] Padding samples can be generated, for example, using one of the following methods: Recent sample copy. Mirroring along the axis of interpolation filtering.

[0139] Specifically, nearest sample duplication refers to a technique in which the replacement value is determined to be equal to the value closest to the integer sample position being accessed. For example, in Figure 14, the values ​​1030 at the two missing positions are replaced with the value of the sample immediately to the right of the fractional position 1020 being interpolated. However, it should be noted that this technique is merely an example, and this disclosure is not limited to simple neighborhood duplication. Alternatively, several nearest sample positions may be used to interpolate the missing sample 1030. For example, three samples on the boundary closest to the missing sample may be used to interpolate the missing sample by a weighted average using weights set inversely proportional to the distance of the three samples from the missing sample.

[0140] Mirroring refers to a technique where replacement values ​​are determined by mirroring the values ​​of the corresponding evaluated integer sample positions with respect to the axes of the interpolation filter. For example, in Figure 14, the two missing samples 1030 are replaced with the values ​​of the samples to the left of fractional position 1020. Specifically, the second sample to the right of fractional position 1020 is replaced with the value of the second position to the left of fractional position 1020, and the third sample to the right of fractional position 1020 is replaced with the value of the third position to the left of fractional position 1020.

[0141] Note that if the missing sample is a function of the mirrored sample and is the closest sample, a combination of the two methods described above may also be applicable.

[0142] According to one embodiment, the window is formed by all integer sample positions within the search space that are accessible for template matching or bilateral matching. This is the case, for example, in the example in Figure 14, where additional samples outside the window are only needed for template matching at non-integer fractional sample positions. In other words, in Figure 14, in the example described above, the 1 / 2 Pell coordinates being searched require samples outside the memory access window. As a result, a padding operation is applied to generate unusable samples. However, the present invention is not limited thereto.

[0143] Figure 15 shows an alternative technique in which the window does not include all integer sample locations accessible for template matching at integer sample locations within the search space. In the example above, the encoded block size was shown as 1x1 for simplification. In Figure 15, the size is 4x4 integer samples. In this example, even if the search coordinates (locations where template or bilateral matching is calculated) point to integer samples, padding can still be applied if some of the samples within the block are outside the memory access window. This technique makes it possible to further restrict the window size.

[0144] To elaborate, Figure 15 shows window 1120 containing several accessible samples. The initial motion vector points to a position marking the upper-left corner of the coded block. The search space may be considered equal to the window here. However, it should be noted that it can usually be smaller than the window, and this is also reasonable, because if the template is larger than 1x1 integer samples, performing template matching will inevitably require additional samples beyond those contained in the search space. Nevertheless, if the search space defines the size of the window, missing samples may be padded instead. Figure 15 shows an exemplary search position at row 2, column 10 of the window. Template matching using a template the size of 4x4 integer samples will require additional pixels outside the window. These samples are therefore padded, i.e., replaced with values ​​determined or calculated based on the samples within the window. The padding techniques described above, such as nearest neighbor or mirroring, may be applied.

[0145] The window is Search space, or A block having the size of a predicted block placed at the initial motion vector position, or Initial motion vector position, It is defined by its vertical and / or horizontal size relative to [the element].

[0146] An example of a window definition is shown in Figure 18 and described above. This corresponds to determining the window size based on the size of the prediction block placed at the initial motion vector position. Alternatively, the window size may be defined relative to the initial vector position by the size of one or both of the x and y directions. For example, the window may be defined to be placed around a position pointed to by the initial motion vector and extending by A integer samples in each of the up, down, left, and right directions. Alternatively, it may extend by A integer samples to the left and right, and by B integer samples up and down. Other definitions are also possible.

[0147] It should be noted that the above example was provided for 1 / 2 Pell interpolation. However, this disclosure is not limited thereto. In general, any fractional position such as 1 / 4, 1 / 8, etc., may be used, i.e., interpolated using the corresponding interpolation filter.

[0148] The same definition of window size also applies to the case of padding for interpolation filtering for predictor generation. Specifically, the vertical and horizontal sizes (i.e., x and y directions) are determined by the vertical and horizontal lengths of the interpolation filter used to determine the predictor.

[0149] Figure 17 is a flowchart illustrating the process shown in Figure 13 in more detail. The flowchart in Figure 17 differs from the flowchart in Figure 12, which shows the conventional process, in that steps 1320 and 1380 in Figure 12 are replaced by steps 1720 and 1780, respectively. These steps differ from the steps in Figure 12 in that padding with replacement samples is applied to both the interpolation filtering for generating candidate prediction blocks within (multiple) reference pictures to perform bilateral matching or template matching, and the interpolation filtering for generating the final prediction.

[0150] The processing circuit described with reference to Figure 7 may be used in an encoder and / or decoder as shown in Figures 1 and 2.

[0151] In particular, an encoding device may be provided for encoding a video image divided into prediction blocks into a bitstream, the encoding device comprising a device for determining motion vectors for the prediction blocks described above, which includes a processing circuit, and an encoding circuit for encoding the difference between the prediction blocks and the predictors given by the prediction blocks at the positions specified by the determined motion vectors, and generating a bitstream containing the encoded difference and the initial motion vectors.

[0152] Referring to Figure 1, other units and functions of the encoder described above may also be provided or implemented in the processing circuit.

[0153] Correspondingly, a decoding device is provided to decode a video image divided into prediction blocks from a bitstream, and the decoding device comprises an analysis unit that analyzes the initial motion vector and the encoded difference between the prediction block and the predictor given by the prediction block at a position specified by the refined motion vector from the bitstream, and a processing circuit for determining the refined motion vector for the prediction block as described above, and a decoding circuit for reconstructing the prediction block as the sum of the analyzed difference and the predictors given by the prediction block at a position based on the refined motion vector. For example, the predictor may be given directly by the position of the refined motion vector. However, there may be another processing step to obtain the motion vector of the current prediction block, which may further modify the motion vector (filtering, clipping, another refinement, etc.).

[0154] Referring to Figure 2, other units and functions of the decoder described above may also be provided or implemented in the processing circuit.

[0155] Furthermore, embodiments of the present invention have been described in terms of an apparatus equipped with a processing circuit for performing motion vector refinement. However, the present disclosure is not limited thereto and provides corresponding methods, which include processing steps corresponding to the steps to be performed by the processing circuit described above.

[0156] The taps correspond to the filter coefficients. The tap size corresponds to the filter order. Here, the filter is assumed to be a linear filter. In some examples, the filter may be symmetric, i.e., it has symmetric coefficients. However, this disclosure is not limited to symmetric filters or linear filters, or any type of filter. In general, fractional positions may be obtained in any way based on adjacent samples.

[0157] The method according to the embodiment determines the motion vector for the prediction block in the following steps shown in Figure 19. In step 1410, the initial motion vector and template of the prediction block are obtained, and then the refinement of the initial motion vector is determined by template matching using the template in the search space. The search space is placed at the position given by the initial motion vector. In 1420, the memory access window is determined. Note that this does not need to be performed explicitly. The memory access window may be predefined and known. Therefore, this step is illustrated to show that the memory access window size is known. Next, template matching is performed in steps 1430 to 1490.

[0158] Template matching or bilateral matching accesses integer sample locations within a predefined window, a predefined window containing the search space, and another integer sample location accessible for template matching or bilateral matching, and replaces at least one integer sample location located outside the window with a replacement value based on at least one sample within the window. The values ​​of the accessed integer sample locations and the replacement values ​​are used to perform template matching. This is shown in Figure 19, where template matching or bilateral matching is performed at all locations in the search space. In step 1430, it is tested whether template matching or bilateral matching at the current location requires one or more samples located outside the memory access window. If affirmative (step 1430 is yes), then in step 1440, padding as described above is performed on one or more samples located outside the window. After this step, in step 1450, template matching or bilateral matching (i.e., search) is performed at the current location using the padded samples. Alternatively, if there are no required samples outside the window (step 1430 is No), then in step 1450, template matching or bilateral matching (i.e., searching) is performed at the current position using the available samples. If the current position is the last position to be tested in the search space (step 1460 is Yes), then template matching ends. Otherwise (step 1460 is No), the next position is obtained in step 1470, and in step 1430, template matching continues for the new position as described above.

[0159] Once template matching is complete, the best-matching location is identified (for example, as the stored location with the lowest cost).

[0160] Figure 20 shows an alternative exemplary method for motion vector refinement and predictor generation applicable within the framework of the embodiments of the present invention. The difference between this method and the method in Figure 13 is that, in Figure 20, interpolation filtering with padding is applied only to the final generation of predictors using the refined motion vectors (pairs). This is step 5 in Figure 20, which corresponds to step 4 in Figure 13.

[0161] On the other hand, in motion vector refinement (search), i.e., step 2 in Figure 13, interpolation filtering with padding is not applied. Instead, in the method in Figure 20, if the initial motion vector (or multiple initial candidate motion vectors forming the initial search space) points to the nearest integer point, a rounding operation is performed to round the coordinates to the nearest integer point (step 2 in Figure 20). This rounding operation is equivalent to the predefined vector movement, indicated by MV_round_diff, which represents the rounding direction. By performing the rounding operation, the search space is modified to obtain a modified search space that contains only points with integer coordinates. As a result, the next step 3, which obtains the best-matching integer point by bilateral (or template) matching, can be performed with integer coordinates, and consequently, interpolation filtering is not required to generate predictions for template / bilateral matching.

[0162] Furthermore, Figure 20 includes step 4, which is optional. In step 4, after the best-matching integer point in the modified search space is obtained in step 3, the rounding is reversed to obtain the final refined motion vector in the (initial) search space. In other words, a vector moving in the opposite direction to that in step 2 is applied to the best-matching (refined) motion vector resulting from step 3. If step 4 is included, the final prediction is then obtained based on the refined motion vector resulting from step 4. Alternatively, step 3 is omitted, and step 5 is applied directly to the motion vector resulting from step 3. Further alternatively, whether to include or omit step 4 can be determined based on certain conditions.

[0163] A flowchart corresponding to the method in Figure 20 is shown in Figure 21. Here again, the flowchart is created for pairs of motion vectors rather than single motion vectors, which are used in Figure 20 for the purpose of simplifying the diagram.

[0164] The first step 1310 is the same as in Figure 20. The subsequent step 2115 corresponds to the rounding operation in step 2 of Figure 20. The process described herein is general, as long as the modification of the initial search space is not limited to rounding operations, but more generally, a representative pair of motion vectors is generated for each pair of motion vectors in the initial search space. The rounding operation in Figure 20 is a special case in which the initial search space motion vectors are mapped to representative ones in this way.

[0165] The next steps, 2130 (optional for template matching) and 2140, are the same as those in Figure 17, differing only in that each operation is performed with a representative motion vector in the modified search space. Steps 2150 and 2160 correspond to step 4 in Figure 20. Step 2150 shows the step of setting a representative motion vector pair that minimizes the template matching cost (or bilateral matching cost) as Pair_best_rep.

[0166] Step 2160 (optional) corresponds to undoing the rounding operation, as shown in step 4 of Figure 20. In the next step (1370), the motion vectors (pairs) in the original search space are output as motion vectors for the predicted blocks. Steps 1370 and 1780 are the same as in Figure 17.

[0167] Furthermore, an encoding method is provided for encoding a video image divided into prediction blocks into a bitstream, the encoding method comprising the steps of determining motion vectors for the prediction blocks according to any of the methods described above, and encoding the difference between the prediction blocks and the predictors given by the prediction blocks at positions based on the determined motion vectors, and generating a bitstream containing the encoded difference and the initial motion vectors.

[0168] The encoding method may further include the steps described with reference to the function of the block in Figure 1.

[0169] Furthermore, a decoding method is provided for decoding a video image divided into prediction blocks from a bitstream, the decoding method comprising the steps of: analyzing an initial motion vector and an encoded difference between the prediction block and the predictor given by the prediction block at a position specified by the refined motion vector from the bitstream; determining a refined motion vector for the prediction block according to any of the methods described above; and reconstructing the prediction block as the sum of the analyzed difference and the predictor given by the prediction block at a position specified by the refined motion vector.

[0170] The decoding method may further include the steps described with reference to the function of the block in Figure 2.

[0171] However, it should be noted that Figures 1 and 2 are not intended to limit the present disclosure. They simply provide unlimited examples of the implementation of the present invention within existing encoders and / or decoders.

[0172] Motion vector determination using sample padding as described above can be performed as part of encoding and / or decoding of video signals (moving images). However, motion vector determination is not limited to use for encoding / decoding and can also be used for other purposes in image processing, such as motion detection and motion analysis.

[0173] Motion vector determination can be implemented as a device. Such a device may be a combination of software and hardware. For example, motion vector determination may be performed by a chip such as a general-purpose processor, or a digital signal processor (DSP), or a field-programmable gate array (FPGA). However, the present invention is not limited to implementation on programmable hardware. It can be implemented on an application-specific integrated circuit (ASIC) or by a combination of the hardware components mentioned above.

[0174] Motion vector determination may be performed by program instructions stored on a computer-readable storage medium. When the program is executed, the computer performs the steps of the method described above. The computer-readable storage medium may be any medium on which the program is stored, such as a DVD, CD, USB (flash) drive, hard disk, or server storage available via a network.

[0175] The encoder and / or decoder may be implemented in various devices such as TV sets, set-top boxes, PCs, tablets, or smartphones, i.e., any recording device, encoding device, code conversion device, decoding device, or playback device. This may also be software or an app that implements the method steps, and may be stored / executed in a processor contained in the electronic devices described above.

[0176] In summary, this disclosure relates to motion vector determination using template matching or bilateral matching, and predictor generation based on motion vectors. Template matching or bilateral matching, and / or predictor generation, utilize interpolation filtering. The interpolation filtering operation accesses integer sample positions within a window, and the window, which includes the search space and another integer sample position accessible for template matching, pads at least one integer sample position located outside the window with padded sample values ​​based on at least one sample within the window, and uses the values ​​of the accessed integer sample positions and the padded sample values ​​to perform template matching or bilateral matching and predictor generation.

[0177] Another embodiment of the present invention is provided below. Note that the numbering used in the following sections does not necessarily have to follow the numbering used in the preceding sections.

[0178] Embodiment 1. An encoding device for encoding a prediction block of a video image into a bitstream (171), wherein the encoding device is Obtain at least one initial motion vector, Based on the search space of candidate motion vectors, the refinement of the initial motion vector is determined, and the refined motion vector is obtained, and the search space is placed at the position (810) given by the initial motion vector. By doing so, the motion vector for the predicted block (165) is determined. The difference (105) between the prediction block and the predictor given by the prediction block at the position based on the determined motion vector is encoded to generate a bitstream (171) containing the encoded difference and the initial motion vector. Interpolation filtering using an interpolation filter is used to determine the predictor according to the motion vector, and the interpolation filtering is performed as follows: Access the sample value at an integer sample position within a predefined window. At integer sample positions located outside the window, at least one sample value is replaced by a replacement value based on at least one sample within the window. Perform interpolation filtering using the accessed integer position sample values ​​and replacement values. The system includes a processing circuit (600) configured as follows: Encoding device.

[0179] Embodiment 2. A decoding device for decoding a video image divided into prediction blocks from a bitstream (171), wherein the decoding device is At the position specified by the refined motion vector, the initial motion vector and the encoded difference between the prediction block and the predictor given by the prediction block are analyzed from the bitstream. Obtain at least one initial motion vector, Based on the search space of candidate motion vectors, the refinement of the initial motion vector is determined, and the refined motion vector is obtained, and the search space is placed at the position (810) given by the initial motion vector. By doing so, the motion vector for the predicted block is determined, The motion vector is determined by reconstructing the prediction block as the sum of the analyzed difference and the predictor given by the prediction block at the position specified by the refined motion vector determined by the processing circuit (600). It is equipped with a processing circuit configured as follows: The processing circuit is further configured to determine a predictor according to a refined motion vector using interpolation with an interpolation filter, and the interpolation is Access the sample value at an integer sample position within a predefined window. At integer sample positions located outside the window, at least one sample value is replaced by a replacement value based on at least one sample within the window. Perform interpolation using the values ​​of the accessed integer sample positions and the replacement values. Decoding device.

[0180] Embodiment 3. The apparatus according to Embodiment 1 or 2, wherein the substitution value is determined by mirroring the value of the corresponding evaluated integer sample position with respect to the axis of the interpolation filter.

[0181] Embodiment 4. The apparatus according to any one of Embodiments 1 to 3, wherein the interpolation filter is a one-dimensional filter or a separable two-dimensional filter.

[0182] Embodiment 5. The apparatus according to any one of Embodiments 1 to 4, wherein the window is formed by all integer sample positions accessible to obtain a predictor given by the prediction block at a position specified by the initial motion vector.

[0183] Embodiment 6. The window is A block having the size of a predicted block, placed at the initial motion vector position. Initial motion vector position, and The vertical and horizontal lengths of the interpolation filter used to determine the predictor. The apparatus according to any one of embodiments 1 to 5, defined by the vertical size and / or horizontal size of at least one of the following.

[0184] Embodiment 7. The apparatus according to any one of Embodiments 1, 2, or 4 to 6, wherein the substitution value is determined to be equal to the value closest to the accessed integer sample position.

[0185] Embodiment 8. The apparatus according to any one of Embodiments 1 to 7, wherein the number of sample values ​​to be replaced depends on the difference between the initial motion vector and the refined motion vector.

[0186] Embodiment 9. The processing circuit (600) is configured to determine the refinement of the initial motion vector by template matching or bilateral matching using a template. The template matching or the bilateral matching described above, Access sample values ​​at integer sample positions within another predefined window (1120), the other predefined window (1120) includes the search space and another integer sample position, which are accessible for template matching or bilateral matching. At an integer sample position located outside of another window, at least one sample value (1030) is replaced by a replacement value based on at least one sample within the other window. An apparatus according to any one of embodiments 1 to 8, which performs template matching or bilateral matching using the values ​​and replacement values ​​of accessed integer sample positions.

[0187] Embodiment 10. The search space includes at least one fractional sample position (1, 2, 6) with values ​​obtained by interpolation filtering using a predefined tap size filter. The apparatus according to Embodiment 9, wherein interpolation filtering obtains the value of at least one fractional sample position using the value of the accessed integer sample position and at least one replacement value.

[0188] Embodiment 11. The apparatus according to Embodiment 10, wherein the substitution values ​​used in template matching or bilateral matching are determined by mirroring the values ​​of the corresponding accessed integer sample positions with respect to the filter axis.

[0189] Embodiment 12. The apparatus according to Embodiment 9 or 10, wherein the substitution value used in template matching or bilateral matching is determined to be equal to the value closest to the accessed integer sample position.

[0190] Embodiment 13. Another predefined window (1120) search space, A block having the size of a predicted block placed at the initial motion vector position, and Initial motion vector position The apparatus according to any one of embodiments 9 to 12, defined by the vertical size and / or horizontal size of at least one of the following.

[0191] Embodiment 14. The apparatus according to any one of Embodiments 9 to 13, wherein the other predefined window (1120) is the same as the predefined window.

[0192] Embodiment 15. The apparatus according to any one of Embodiments 1 to 14, wherein the refinement is determined by bilateral matching, and the processing circuit is configured to obtain at least two initial motion vectors that point to positions in different reference pictures (Ref0, Ref1) and form a pair of initial motion vectors (Pair_0, Pair_1, ..., Pair_N).

[0193] Embodiment 16. The refinement is determined by template matching, and the processing circuit is Get the template for the prediction block, The refinement of the initial motion vector is determined by template matching using the template in the search space. The apparatus according to any one of embodiments 1 to 14, further configured as follows.

[0194] Embodiment 17. An encoding method for encoding a video image divided into prediction blocks into a bitstream, wherein the encoding method is: The steps include obtaining at least one initial motion vector (1310, 1410), Steps (1320,1330,1340,1720;2115,2120,2130,2140,2150,2160) to determine the refinement of the initial motion vector based on the search space of candidate motion vectors and to obtain the refined motion vector, wherein the search space is located at the positions given by the initial motion vector, and The steps include determining the motion vector for the predicted block, The steps include: encoding the difference between the prediction block and the predictor given by the prediction block at the position based on the determined motion vector, and generating a bitstream containing the encoded difference and the initial motion vector; Step (1780) of determining a predictor according to a motion vector using interpolation filtering with an interpolation filter, wherein the interpolation filtering is Access the sample value at an integer sample position within a predefined window. At integer sample positions located outside the window, at least one sample value is replaced by a replacement value based on at least one sample within the window. Step (1780) performs interpolation filtering using the accessed integer position sample values ​​and replacement values. Encoding methods, including those mentioned.

[0195] Embodiment 18. A decoding method for decoding a video image divided into prediction blocks from a bitstream, wherein the decoding method is: The steps include: analyzing the initial motion vector and the encoded difference between the prediction block and the predictor given by the prediction block from the bitstream at a position specified by the refined motion vector; The steps include obtaining at least one initial motion vector (1310, 1410), Steps (1320,1330,1340,1720;2115,2120,2130,2140,2150,2160) to determine the refinement of the initial motion vector based on the search space of candidate motion vectors and to obtain the refined motion vector, wherein the search space is located at the positions given by the initial motion vector, and The steps include determining the motion vector for the predicted block, A step of reconstructing the prediction block as the sum of the analyzed difference and the predictor given by the prediction block, at the position specified by the refined motion vector determined by the step of determining the motion vector, Step (1780) of determining a predictor according to a refined motion vector using interpolation with an interpolation filter, wherein the interpolation is Access the sample value at an integer sample position within a predefined window. At integer sample positions located outside the window, at least one sample value is replaced by a replacement value based on at least one sample within the window. Step (1780) performs interpolation using the accessed integer position sample and replacement value. A decoding method that includes this.

[0196] Embodiment 19. The method according to Embodiment 17 or 18, wherein the substitution value is determined by mirroring the value of the corresponding evaluated integer sample position with respect to the axis of the interpolation filter.

[0197] Embodiment 20. The method according to any one of Embodiments 17 to 19, wherein the interpolation filter is a one-dimensional filter or a separable two-dimensional filter.

[0198] Embodiment 21. The method according to any one of Embodiments 17 to 20, wherein the window is formed by all integer sample positions accessible to obtain a predictor given by the prediction block at a position specified by the initial motion vector.

[0199] Embodiment 22. The window is A block having the size of a predicted block, placed at the initial motion vector position. Initial motion vector position, and The vertical and horizontal lengths of the interpolation filter used to determine the predictor. The method according to any one of embodiments 17 to 21, defined by the vertical size and / or horizontal size of at least one of the following.

[0200] Embodiment 23. The method according to any one of Embodiments 17, 18, or 20 to 22, wherein the substitution value is determined to be equal to the value closest to the accessed integer sample position.

[0201] Embodiment 24. The method according to any one of Embodiments 17 to 23, wherein the number of sample values ​​to be replaced depends on the difference between the initial motion vector and the refined motion vector.

[0202] Embodiment 25. The step of determining refinement (1320, 1330, 1340, 1720; 2115, 2120, 2130, 2140, 2150, 2160) determines the refinement of the initial motion vector by template matching or bilateral matching using a template. The template matching or the bilateral matching described above, Access sample values ​​at integer sample positions within another predefined window, the other predefined window includes the search space and other integer sample positions, which are accessible for template matching or bilateral matching. At integer sample positions located outside the other window, at least one sample value is replaced by a replacement value based on at least one sample within the other window (1440), The method according to any one of embodiments 17 to 24, wherein template matching or bilateral matching is performed using the values ​​and replacement values ​​of the accessed integer sample positions (1450).

[0203] Embodiment 26. The search space includes at least one fractional sample position with values ​​obtained by interpolation filtering using a predefined tap size filter, The method according to Embodiment 25, wherein interpolation filtering obtains the value of at least one fractional sample position using the value of the accessed integer sample position and at least one replacement value.

[0204] Embodiment 27. The method according to Embodiment 26, wherein the substitution values ​​used in template matching or bilateral matching are determined by mirroring the values ​​of the corresponding evaluated integer sample positions with respect to the filter axis.

[0205] Embodiment 28. The method according to Embodiment 26 or 27, wherein the substitution value used in template matching or bilateral matching is determined to be equal to the value closest to the accessed integer sample position.

[0206] Embodiment 29. Another predefined window, search space, A block having the size of a predicted block placed at the initial motion vector position, and Initial motion vector position The method according to any one of embodiments 25 to 28, defined by the vertical size and / or horizontal size of at least one of the following.

[0207] Embodiment 30. The method according to any one of Embodiments 25 to 29, wherein the other predefined window is the same as the predefined window.

[0208] Embodiment 31. The method according to any one of Embodiments 17 to 30, wherein the refinement is determined by bilateral matching, and the step of acquiring acquires at least two initial motion vectors that point to positions in different reference pictures and form a pair of initial motion vectors.

[0209] Embodiment 32. The refinement is determined by template matching, and the step of determining the motion vector is, Steps to obtain the prediction block template (1330;2130), The steps include determining the refinement of the initial motion vector by template matching using the template in the search space (1340;2140) and The method according to any one of embodiments 17 to 31, further including the method described above.

[0210] Embodiment 33. A computer-readable storage medium that, when executed by a processor, stores instructions causing the processor to perform the method according to any one of Embodiments 17 to 32. [Explanation of Symbols]

[0211] 100 encoders 101 Picture Blocks 102 inputs 104 Residual calculation 105 Residual Block 106 Conversion Unit 107 Conversion coefficients 108 Quantization Units 109 Quantization coefficients 110 Inverse Quantization Unit 111 Inverse Quantization Coefficient 112 Inverse Conversion Unit 113 Inversely transformed blocks 114 Reconstruction 115 Reconstructed Blocks 116 buffers 117 Reference Samples 120 Loop Filtering Unit 121 filtered blocks 130 Decoded picture buffer 131 Decoded picture 142 Interest Units 144 Interpretation Units 152 Intra Estimation Units 154 Intra Prediction Units 160 Mode Selection Unit 165 prediction blocks 170 Entropy Encoder Unit (Entropy Encoding Unit) 171 Encoded picture data (encoded bitstream) 172 Output 200 Decoders 202 inputs 204 Entropy Decode Unit 209 Quantized residual transformation coefficients 210 Inverse Quantization Unit 211 Inverse Quantization Transformation Coefficients 212 Inverse Conversion Unit 213 Inversely transformed blocks 214 Reconstruction 215 Reconstructed Blocks 216 buffers 217 Reference Samples 220 Loop Filtering Unit 221 filtered blocks 230 Decoded picture buffers 231 Decoded picture 232 Output 244 Interpretation Units 254 Intra Prediction Units 260 Mode Selection 265 Prediction Blocks 600 Processing Circuits 610 Template Determination Unit 620 Motion Vector Precision Unit 640 internal memory 650 External Memory 810 center point 1010 Position indicated by an unrefined motion vector 1020 fractional position 1030 Another sample location 1120 Another predefined window

Claims

1. A device for coding video images, Obtaining the initial motion vector of the block, By performing a search in a search space located at a position defined by the initial motion vector, the refined motion vector of the block is obtained based on the initial motion vector. Determining the predictor of the block according to the refined motion vector, interpolation by an interpolation filter, at least one sample value of an integer sample position located inside the window, and padded sample values ​​for the interpolation of an integer sample position located outside the window, wherein the padded sample values ​​are determined based on at least one sample value of a sample located inside the window. A device configured to perform the following actions.

2. A method for coding video images, Steps to obtain the initial motion vector of the block, The steps include: obtaining a refined motion vector of the block based on the initial motion vector by performing a search in a search space located at a position defined by the initial motion vector; A step of determining the predictor of the block according to the refined motion vector, interpolation by an interpolation filter, at least one sample value of an integer sample position located inside the window, and padded sample values ​​for the interpolation of an integer sample position located outside the window, wherein the padded sample values ​​are based on at least one sample value of a sample located inside the window. A method that includes this.

3. The step of obtaining the refined motion vector of the block is performed based on the candidate motion vector in the search space. The method according to claim 2, wherein the refined motion vector is determined based on the initial motion vector and the refinement process.

4. The method according to claim 2, wherein the interpolation filter is a one-dimensional filter or a separable two-dimensional filter.

5. The method according to claim 2, wherein the window is formed by all integer sample positions accessible to obtain the predictor at the position specified by the initial motion vector.

6. The aforementioned window, A block having the size of the block, positioned at the location of the initial motion vector, The position of the initial motion vector, and The vertical and horizontal lengths of the interpolation filter used to determine the predictor The method according to claim 2, defined by the vertical size and / or horizontal size for at least one of the following.

7. The method according to claim 2, wherein the padded sample value for the integer sample position located outside the window is determined to be equal to the sample value of at least one used integer sample position located inside the window that is closest to the integer sample position located outside the window.

8. The step of obtaining the refined motion vector of the block is performed by using bilateral matching. The aforementioned bilateral matching is, Accessing a sample value at an integer sample position in another window, wherein the other window includes the search space and another integer sample position accessible for the bilateral matching. At integer sample positions located outside the aforementioned separate window, at least one sample value is padded by a padded sample value based on at least one sample within the aforementioned separate window. The bilateral matching is performed using the value of the accessed integer sample position and the padded sample value. The method according to claim 2, including the method described in claim 2.

9. The search space includes at least one fractional sample position, with values ​​obtained by interpolation filtering using a predefined tap size filter. The method according to claim 3, wherein the interpolation filtering obtains the value of the at least one fractional sample position using the value of the accessed integer sample position and the at least one padded sample value.

10. The method according to claim 8, wherein the padded sample value used in the bilateral matching is determined to be equal to the value of the position closest to the accessed integer sample position.

11. The aforementioned other window, The aforementioned search space, A block having the size of the block placed at the position of the initial motion vector, and Verification of the initial motion vector The method according to claim 8, defined by the vertical size and / or horizontal size for at least one of the following.

12. The method according to claim 8, wherein the other window is the same as the window.

13. A computer-readable storage medium that, when executed by a processor, stores instructions causing the processor to perform the method described in claim 2.

14. A device for storing and decoding video or image bitstreams, It comprises a communication interface, a processor, and a storage medium. The communication interface is configured to receive and / or transmit a bitstream, and the storage medium is configured to store the bitstream. The bitstream includes information related to the initial motion vector, and the initial motion vector is used to obtain the refined motion vector of the block by performing a search in a search space located at the position defined by the initial motion vector. The refined motion vector, interpolation by the interpolation filter, at least one sample value of an integer sample position located inside the window, and padded sample values ​​for the interpolation relative to an integer sample position located outside the window are used to determine the predictor of the block, the padded sample values ​​being based on at least one sample value of a sample located inside the window, device.

Images