Video decoding method, video encoding method, and method for providing bitstream related to video data

Abstract

The present invention relates to a video decoding method, a video encoding method, and a method for providing a bitstream related to video data. The present embodiment uses an encoding tool on the encoder / decoder side based on a higher-level syntax to adjust the correction of the motion vector, where the encoding tool improves the compression performance of inter prediction. Furthermore, in order to simply perform the following steps: an integer sample search step for searching for integer offsets, in particular integer offsets related to motion vector refinement in a coding tool; and a fractional sample refinement step for searching for sub-pixel offsets.

Description

[0001] This application is a divisional application of the PCT patent application entered into China with Chinese patent application number 202080049464.5, invention name “A video decoding method, a video encoding method and a method for providing a bit stream associated with video data”, and application date July 8, 2020.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to Korean Patent Application No. 10-2019-0081926, filed on Jul. 8, 2019, and Korean Patent Application No. 10-2019-0170099, filed on Dec. 18, 2019, which are hereby incorporated by reference in their entirety. Technical Field

[0004] The present invention relates to encoding and decoding of video, and more particularly, to an encoding tool configured to improve compression performance of inter-frame prediction and a method for adjusting motion vector correction using the encoding tool at an encoder / decoder side based on a high-level syntax. Background Art

[0005] The statements in this section only provide background information related to the present invention and do not constitute prior art. Since the amount of video data is larger than the amount of voice data or still image data, storing or transmitting video data without compression processing requires a large amount of hardware resources (including memory).

[0006] Accordingly, when storing or transmitting video data, an encoder is usually used to compress the video data for storage or transmission. Then, the decoder receives the compressed video data, decompresses and reproduces the video data. Compression technologies for such videos include H.264 / AVC and High Efficiency Video Coding (HEVC), which has an encoding efficiency of about 40% higher than that of H.264 / AVC. However, image size, resolution and frame rate are gradually increasing, and accordingly, the amount of data to be encoded is also increasing. Therefore, a new compression technology with better encoding efficiency and higher image quality than existing compression technologies is needed.

[0007] In video coding, predictive coding is mainly used to improve compression performance. There is intra-frame prediction for predicting a target block to be encoded based on pre-reconstructed samples in a current image and inter-frame prediction for predicting a current block using a pre-reconstructed reference image. In particular, inter-frame prediction is widely used in video coding because it exhibits better compression performance than intra-frame prediction.

[0008] The present invention proposes a coding tool for improving the compression performance of existing inter-frame prediction. Summary of the invention

[0009] Technical issues

[0010] In the present invention, regarding a coding tool for improving compression performance of inter-frame prediction, a coding tool is utilized on the encoder / decoder side based on a high-level syntax to adjust correction of motion vectors. The main object of the present invention is to provide a method for simply performing integer sample search operations and fractional sample refinement operations associated with other coding tools for motion vector refinement.

[0011] According to one aspect of the present invention, a method for video decoding is provided, which is used to decode a coded image sequence block by block by a video decoding device. The method comprises: decoding at least one first high-level syntax element, the first high-level syntax element indicating whether each of at least one coding tool from a bitstream is allowed, the at least one coding tool including motion vector refinement; decoding a second high-level syntax element from the bitstream according to the at least one first high-level syntax element, and setting whether each of at least one coding tool is allowed at the image level, the second high-level syntax element corresponding to each of the at least one coding tool; when it is identified based on the setting at the image level that motion vector refinement is allowed for a current image including a target block, by calculating a metric between two candidate blocks within a preset offset range from an initial motion vector in a bidirectional reference image of the target block, and searching for a motion vector candidate with the smallest metric, a refined motion vector of the target block is generated; and generating an inter-frame prediction sample of the target block using the bidirectional reference image and the refined motion vector.

[0012] According to another aspect of the present invention, a video decoding device is provided for decoding a coded image sequence block by block. The device includes a decoder and a predictor, the decoder is configured to decode at least one first high-level syntax element from a bitstream, the first high-level syntax element indicating whether each of at least one coding tool is allowed, and according to the at least one first high-level syntax element, decode a second high-level syntax element from the bitstream, and set whether each of at least one coding tool is allowed at the image level, the second high-level syntax element corresponds to each of at least one coding tool, and the at least one coding tool includes motion vector refinement; the predictor is configured to generate a refined motion vector of the target block by calculating a metric between two candidate blocks within a preset offset range from an initial motion vector in a bidirectional reference image of the target block, and searching for a motion vector candidate with a minimum metric, based on the setting at the image level, in response to identifying that motion vector refinement is allowed for a current image including a target block, and searching for a motion vector candidate with a minimum metric, wherein the predictor generates an inter-frame prediction sample of the target block using the bidirectional reference image and the refined motion vector.

[0013] According to another aspect of the present invention, a video encoding method is provided for encoding a coded image sequence block by block by a video encoding device. The method comprises: encoding at least one first high-level syntax element in a sequence header, the first high-level syntax element indicating whether each of at least one coding tool is allowed, the at least one coding tool including motion vector refinement; encoding a second high-level syntax element at the picture level according to the at least one first high-level syntax element, the second high-level syntax element at the picture level being related to whether each of at least one coding tool is allowed; when it is identified based on the picture level setting that motion vector refinement is allowed for the current picture including the target block, by calculating the metric between two candidate blocks within a preset offset range from the initial motion vector in the bidirectional reference picture of the target block, and searching for the motion vector candidate with the smallest metric, a refined motion vector of the target block is generated; and generating an inter-frame prediction sample of the target block using the bidirectional reference picture and the refined motion vector.

[0014] Beneficial Effects

[0015] As is apparent from the above description, according to the present embodiment, a method is provided for simply performing integer sample search operations and fractional sample refinement operations related to motion vector refinement in a coding tool, thereby improving compression performance of inter-frame prediction. Thus, the coding efficiency of the encoder / decoder can be improved, and the complexity can be reduced accordingly. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is an exemplary block diagram of a video encoding device capable of implementing the technology of the present invention.

[0017] Figure 2 A block partition structure using a QTBTTT structure is exemplarily shown.

[0018] Figure 3a and Figure 3b A plurality of intra prediction modes including a wide-angle intra prediction mode are exemplarily shown.

[0019] Figure 4 Neighboring blocks around the current block are exemplarily shown.

[0020] Figure 5 is an exemplary block diagram of a video decoding device capable of implementing the techniques of the present invention.

[0021] Figure 6 is an exemplary diagram illustrating the concept of BDOF according to an embodiment of the present invention.

[0022] Figure 7 is an example diagram showing a method for deriving gradients for block boundary samples in bidirectional optical flow.

[0023] Figure 8a and Figure 8b is an exemplary diagram showing affine motion prediction according to an embodiment of the present invention.

[0024] Fig. 9 is an example diagram showing sub-block-by-sub-block affine motion prediction according to an embodiment of the present invention.

[0025] Fig.10 is an exemplary diagram showing a method for deriving merge candidates for affine motion prediction from translational motion vectors of neighboring blocks.

[0026] Fig.11 is an exemplary diagram illustrating the concept of a motion vector refinement encoding tool according to an embodiment of the present invention.

[0027] Fig.12 is an exemplary diagram illustrating a method for reducing the search complexity of a motion vector refinement coding tool according to an embodiment of the present invention.

[0028] Fig.13 is an exemplary diagram illustrating a method for reducing the search complexity of a motion vector refinement coding tool according to another embodiment of the present invention.

[0029] Fig.14 is an exemplary diagram illustrating a method for reducing the search complexity of a motion vector refinement coding tool according to another embodiment of the present invention.

[0030] Reference numerals

[0031] 124: Inter-frame predictor

[0032] 155: Entropy Encoder

[0033] 510: Entropy Decoder

[0034] 544: Inter-frame predictor. DETAILED DESCRIPTION

[0035] Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that when reference numerals are added to the constituent elements in the various drawings, the same reference numerals also represent the same elements, although the elements are shown in different drawings. In addition, in the following description of the embodiments of the present invention, the detailed description of the known functions and configurations incorporated herein will be omitted to avoid blurring the subject matter of the embodiments.

[0036] Figure 1 is an exemplary block diagram of a video encoding device capable of implementing the technology of the present invention. Figure 1 A video encoding apparatus and elements of the apparatus are described below.

[0037] The video encoding device includes: an image segmenter 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a rearrangement unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filter unit 180 and a memory 190. Each element of the video encoding device can be implemented in hardware or software, or a combination of hardware and software. The functions of each element can be implemented as software, and the microprocessor can be implemented to execute the software functions corresponding to each element.

[0038] A video consists of one or more sequences including multiple images. Each image is divided into multiple regions, and encoding is performed on each region. For example, an image is divided into one or more tiles (tile) or / and slices (slice). In particular, one or more tiles can be defined as tile groups. Each tile or slice is divided into one or more coding tree units (coding tree unit, CTU). Each CTU is divided into one or more coding units (coding unit, CU) through a tree structure. The information applied to each CU is encoded as the syntax of the CU, and the information commonly applied to the CU included in a CTU is encoded as the syntax of the CTU. In addition, the information commonly applied to all blocks in a slice is encoded as the syntax of the slice header, and the information applied to all blocks constituting one or more images is encoded in the picture parameter set (Picture Parameter Set, PPS) or the picture header.

[0039] In addition, information commonly referenced by one or more SPSs is encoded in a video parameter set (VideoParameter Set, VPS). In addition, information commonly referenced by multiple images is encoded in a sequence parameter set (SequenceParameter Set, SPS). In addition, information commonly applied to a tile or tile group can be encoded as the syntax of a tile header or tile group header. The syntax included in the SPS, PPS, slice header, and tile header or tile group header can be called a high-level syntax.

[0040] The image splitter 110 is configured to determine the size of a coding tree unit (CTU). Information about the size of the CTU (CTU size) is encoded as the syntax of the SPS or PPS and transmitted to the video decoding device. The image splitter 110 splits each image constituting the video into a plurality of CTUs of a predetermined size, and then recursively splits the CTU using a tree structure. In the tree structure, a leaf node is used as a coding unit (CU), which is a basic unit of coding.

[0041] The tree structure can be a quadtree (QuadTree, QT), a binary tree (BinaryTree, BT), a ternary tree (TernaryTree, TT), or a structure formed by a combination of two or more QT structures, BT structures and TT structures, wherein the quadtree (QT) is a node (or parent node) that is divided into four slave nodes (or child nodes) of the same size, the binary tree (BT) is a node that is divided into two slave nodes, and the ternary tree (TT) is a node that is divided into three slave nodes at a ratio of 1:2:1. For example, a quadtree plus binary tree (QuadTree plus BinaryTree, QTBT) structure can be used, or a quadtree plus binary tree ternary tree (QuadTree plus BinaryTree TernaryTree, QTBTTT) structure can be used. Here, BTTT can be collectively referred to as a multiple-type tree (MTT).

[0042] Figure 2 is a schematic diagram showing a method for partitioning a block using a QTBTTT structure. Figure 2 As shown, the CTU can be first split into a QT structure. The QT splitting can be repeated until the size of the split block reaches the minimum block size MinQTSize of the leaf node allowed in the QT. The first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoder 155 and notified to the video decoding device with a signal. When the leaf node of the QT is equal to or less than the maximum block size (MaxBTSize) of the root node allowed in the BT, it can be further split into one or more BT structures or TT structures. The BT structure and / or the TT structure can have multiple splitting directions. For example, there can be two directions, namely, a direction of splitting the blocks of nodes horizontally and a direction of splitting the blocks vertically. As Figure 2 As shown, when MTT splitting starts, the entropy encoder 155 encodes a second flag (mtt_split_flag) indicating whether the node is split, a flag indicating the splitting direction (vertical or horizontal) in the case of splitting, and / or a flag indicating the splitting type (binary or trifurcated), and notifies the video decoding device with a signal.

[0043] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into 4 nodes of the lower layer, the CU split flag (split_cu_flag) indicating whether the node is split may be encoded. When the value of the CU split flag (split_cu_flag) indicates that the split is not performed, the block of the node becomes a leaf node in the split tree structure and is used as a coding unit (CU), which is a basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates that the split is performed, the video encoding device starts encoding the flags from the first flag in the above-described manner.

[0044] When QTBT is used as another example of a tree structure, there may be two types of segmentation, namely, a type of horizontally segmenting a block into two blocks of the same size (i.e., symmetrical horizontal segmentation) and a type of vertically segmenting a block into two blocks of the same size (i.e., symmetrical vertical segmentation). A segmentation flag (split_flag) indicating whether each node of the BT structure is segmented into blocks of the lower layer and segmentation type information indicating the segmentation type are encoded by the entropy encoder 155 and transmitted to the video decoding device. There may be an additional type of segmenting a block of a node into two asymmetric blocks. An asymmetric segmentation type may include a type of segmenting a block into two rectangular blocks at a size ratio of 1:3, or a type of segmenting a block of a node diagonally.

[0045] In addition, the CU may have various sizes according to the QTBT or QTBTTT partitioning of the CTU. In the following, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block". When the QTBTTT partitioning is adopted, the shape of the current block may be a square or a rectangle. The predictor 120 is configured to predict the current block to generate a prediction block. The predictor 120 includes an intra predictor 122 and an inter predictor 124.

[0046] Typically, each current block in an image can be predictively encoded separately. In addition, the prediction of the current block can be performed using an intra-frame prediction technique (which uses data from an image including the current block) or an inter-frame prediction technique (which uses data of an image encoded before the image including the current block). Inter-frame prediction includes unidirectional prediction and bidirectional prediction.

[0047] The intra-frame predictor 122 is configured to predict pixels in the current block using pixels (reference pixels) located around the current block in the current image including the current block. There are multiple intra-frame prediction modes according to the prediction direction. For example, Figure 3aAs shown, the plurality of intra prediction modes may include two non-directional modes including a planar mode and a DC mode and 65 directional modes. Neighboring pixels and equations to be used are defined differently for each prediction mode.

[0048] In order to perform effective directional prediction for the current block of rectangular shape, the additional use of Figure 3b The dotted arrows in the figure indicate the directional modes (intra-prediction modes 67 to 80 and -1 to -14). These modes may be referred to as "wide angle intra-prediction modes". Figure 3b , arrows indicate corresponding reference samples used for prediction, rather than indicating a prediction direction. The prediction direction is opposite to the direction indicated by the arrow. The wide angle intra prediction mode is a mode in which prediction is performed in a direction opposite to the specific direction mode without additional bit transmission when the current block has a rectangular shape.

[0049] Specifically, in the wide-angle intra prediction mode, some wide-angle intra prediction modes that can be used for the current block can be determined based on the ratio of the width to the height of the rectangular current block. For example, when the height of the rectangular shape of the current block is less than its width, a wide-angle intra prediction mode (intra prediction modes 67 to 80) with an angle less than 45 degrees can be used. When the width of the rectangular shape of the current block is greater than its height, a wide-angle intra prediction mode (intra prediction modes -1 to -14) with an angle greater than -135 degrees can be used.

[0050] The intra-frame predictor 122 can be configured to determine the intra-frame prediction mode to be used when encoding the current block. In some examples, the intra-frame predictor 122 can be configured to encode the current block using several intra-frame prediction modes and select the appropriate intra-frame prediction mode to be used from the test mode. For example, the intra-frame predictor 122 can be configured to calculate the rate-distortion value using rate distortion analysis of several tested intra-frame prediction modes, and select the intra-frame prediction mode with the best rate-distortion characteristic among the tested modes.

[0051] The intra-frame predictor 122 is configured to select an intra-frame prediction mode from a plurality of intra-frame prediction modes and predict the current block using adjacent pixels (reference pixels) and equations determined according to the selected intra-frame prediction mode. Information about the selected intra-frame prediction mode is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0052] The inter-frame predictor 124 is configured to generate a prediction block of the current block by motion compensation. The inter-frame predictor 124 is configured to search for a block that is most similar to the current block in a reference image that has been encoded and decoded earlier than the current image, and use the searched block to generate a prediction block of the current block. Then, the inter-frame predictor is configured to generate a motion vector (MV) corresponding to the displacement between the current block in the current image and the prediction block in the reference image. Typically, motion estimation is performed on the luminance (luma) component, and the motion vector calculated based on the luminance component is used for both the luminance component and the chrominance component. The motion information including information about the reference image and information about the motion vector used to predict the current block is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0053] The inter-frame predictor 124 may be configured to perform interpolation on a reference image or a reference block to increase prediction accuracy. In other words, a subsample is interpolated between two consecutive integer samples by applying a filter coefficient to a plurality of consecutive integer samples including two integer samples. When searching for a block most similar to the current block is performed on the interpolated reference image, the motion vector may be expressed at an accuracy level of fractional sample units, rather than at an accuracy level of integer sample units.

[0054] For each target area to be encoded, for example, each unit such as a slice, tile, CTU, or CU, the precision or resolution of the motion vector may be set differently. When such an adaptive motion vector resolution (AMVR) is applied, information about the motion vector resolution to be applied to each target area should be signaled for each target area. For example, when the target area is a CU, information about the motion vector resolution applied to each CU is signaled. The information about the motion vector resolution may be information indicating the precision of the motion vector difference, which will be described later.

[0055] The inter-frame predictor 124 may be configured to perform inter-frame prediction using bidirectional prediction. In bidirectional prediction, the inter-frame predictor 124 uses two motion vectors and two reference images representing the block positions most similar to the current block in each reference image. The inter-frame predictor 124 is configured to select a first reference image and a second reference image from a reference image list 0 (RefPicList0) and a reference image list 1 (RefPicList1), respectively, search for blocks similar to the current block in each reference image, and generate a first reference block and a second reference block. Then, the inter-frame predictor 124 is configured to generate a prediction block for the current block by averaging or weighting the first reference block and the second reference block. Then, the inter-frame predictor 124 is configured to transmit motion information including information about two reference images and two motion vectors for predicting the current block to the encoder 155. Specifically, RefPicList0 may consist of images that are before the current image in the display order in the reconstructed image, and RefPicList1 may consist of images that are after the current image in the display order in the reconstructed image. However, the embodiment is not limited thereto. A pre-reconstructed image subsequent to the current image in the display order may be further included in RefPicList0, and conversely, a pre-reconstructed image prior to the current image may be further included in RefPicList1.

[0056] Various methods can be used to minimize the number of bits required to encode motion information. For example, when the reference image and motion vector of the current block are the same as the reference image and motion vector of the adjacent block, the motion information about the current block can be sent to the video decoding device through the encoding information for identifying the adjacent block. This method is called "merge mode".

[0057] In the merge mode, the inter-frame predictor 124 is configured to select a predetermined number of merge candidate blocks (hereinafter referred to as "merge candidates") from the neighboring blocks of the current block. Figure 4 As shown, all or part of the left block L, the upper block A, the upper right block AR, the lower left block BL, and the upper left block AL adjacent to the current block in the current image can be used as adjacent blocks for deriving merge candidates. In addition, in addition to the current image where the current block is located, blocks located in a reference image (which may be the same as or different from the reference image used to predict the current block) can be used as merge candidates. For example, a co-located block at the same position as the current block in the reference image or a block adjacent to the co-located block can be additionally used as a merge candidate.

[0058] The inter-frame predictor 124 uses such neighboring blocks to configure a merge list including a predetermined number of merge candidates. The inter-frame predictor 124 is configured to select a merge candidate to be used as motion information about the current block from among the merge candidates included in the merge list, and generate merge index information for identifying the selected candidate. The generated merge index information is encoded by the encoder 155 and transmitted to the video decoding device.

[0059] Another method of encoding motion information is the advanced motion vector prediction (AMVP) mode. In the AMVP mode, the inter-frame predictor 124 is configured to derive motion vector candidates for prediction of the motion vector of the current block by utilizing the neighboring blocks of the current block. Figure 2 In the current image in FIG. 1 , all or part of the left block L, the upper block A, the upper right block AR, the lower left block BL and the upper left block AL adjacent to the current block may be used as adjacent blocks for deriving motion vector candidates for prediction.

[0060] In addition, in addition to the current image including the current block, a block located in a reference image (which may be the same as or different from the reference image used to predict the current block) can be used as an adjacent block for deriving a predicted motion vector candidate. For example, a co-located block or a block adjacent to the co-located block at the same position as the current block in the reference image can be used. The inter-frame predictor 124 is configured to derive a predicted motion vector candidate using the motion vectors of the adjacent blocks, and to determine a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, a motion vector difference is calculated by subtracting the predicted motion vector from the motion vector of the current block.

[0061] The predicted motion vector can be obtained by applying a predefined function (e.g., a function for calculating a median, an average, etc.) to the predicted motion vector candidate. In particular, the video decoding device also knows the predefined function. Since the neighboring blocks used to derive the predicted motion vector candidate have been encoded and decoded, the video decoding device also already knows the motion vectors of the neighboring blocks. Accordingly, the video encoding device does not need to encode information for identifying the predicted motion vector candidate. Therefore, in this case, information about the motion vector difference and information about the reference image used to predict the current block are encoded.

[0062] The predicted motion vector may be determined by selecting any one of the predicted motion vector candidates. Specifically, information for identifying the selected predicted motion vector candidate is further encoded together with information about the motion vector difference to be used for predicting the current block and information about the reference image. The subtractor 130 is configured to subtract the prediction block generated by the intra predictor 122 or the inter predictor 124 from the current block to generate a residual block.

[0063] The transformer 140 is configured to transform the residual signal in the residual block having pixel values ​​in the spatial domain into a transform coefficient in the frequency domain. The transformer 140 may be configured to transform the residual signal in the residual block using the entire size of the residual block as a transform unit. Alternatively, the transformer 140 may be configured to divide the residual block into a plurality of sub-blocks and perform the transform using the sub-blocks as transform units. Alternatively, the residual signal may be transformed by dividing the block into two sub-blocks, namely a transform region and a non-transform region, and using only the transform region sub-block as a transform unit.

[0064] In particular, the transform region subblock may be one of two rectangular blocks having a size ratio of 1:1 based on the horizontal axis (or vertical axis). In this case, the entropy encoder 155 encodes a flag (cu_sbt_flag) indicating that only the subblock is transformed, direction (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag), and notifies the video decoding device with a signal. In addition, the size of the transform region subblock may have a size ratio of 1:3 based on the horizontal axis (or vertical axis). In particular, the entropy encoder 155 additionally encodes a flag (cu_sbt_quad_flag) that distinguishes the corresponding partitions, and notifies the video decoding device with a signal.

[0065] The transformer 140 may be configured to transform the residual block separately in the horizontal direction and the vertical direction. For the transformation, various types of transformation functions or transformation matrices may be used. For example, the paired transformation functions for horizontal transformation and vertical transformation may be defined as a multiple transform set (MTS). The transformer 140 may be configured to select a pair of transformation functions with the best transformation efficiency in the MTS, and transform the residual block in the horizontal direction and the vertical direction, respectively. The information (mts_idx) about the transformation function pair selected in the MTS is encoded by the entropy encoder 155, and the video decoding device is notified of the information by a signal.

[0066] The quantizer 145 is configured to quantize the transform coefficients output from the transformer 140 using a quantization parameter and output the quantized transform coefficients to the entropy encoder 155. For some blocks or frames, the quantizer 145 may be configured to directly quantize the relevant residual block without transformation. The quantizer 145 may be configured to apply different quantization coefficients (scaling values) according to the position of the transform coefficients in the transform block. The matrix of quantized coefficients applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to the video decoding device.

[0067] The rearrangement unit 150 may be configured to reclassify the coefficient values ​​of the residual values ​​used for quantization. The rearrangement unit 150 may be configured to change the 2D coefficient array into a 1D coefficient sequence by coefficient scanning. For example, the rearrangement unit 150 may be configured to scan the coefficients from the DC coefficient to the coefficients in the high frequency region using a zig-zag scan or a diagonal scan to output a 1D coefficient sequence. Depending on the size of the transform unit and the intra-frame prediction mode, a vertical scan or a horizontal scan may be used instead of a zig-zag scan, wherein the vertical scan scans the two-dimensional coefficient array in the column direction, and the horizontal scan scans the coefficients in the two-dimensional block shape in the row direction. In other words, the scanning mode to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan according to the size of the transform unit and the intra-frame prediction mode.

[0068] The entropy encoder 155 is configured to encode the one-dimensional quantized transform coefficient output from the rearrangement unit 150 using various encoding techniques such as context-based adaptive binary arithmetic coding (CABAC) and exponential Golomb to generate a bitstream. The entropy encoder 155 is configured to encode information related to block partitioning (e.g., CTU size, CU partition flag, QT partition flag, MTT partition type, and MTT partition direction) so that the video decoding device can partition the block in the same manner as the video encoding device.

[0069] In addition, the entropy encoder 155 is configured to encode information about a prediction type indicating whether the current block is encoded by intra-frame prediction encoding or by inter-frame prediction, and encodes intra-frame prediction information (i.e., information about the intra-frame prediction mode) or inter-frame prediction information (merge index for merge mode, information about reference image index and motion vector difference for AMVP mode) according to the prediction type. The entropy encoder 155 is also configured to encode information related to quantization (i.e., information about quantization parameters and information about quantization matrices).

[0070] The inverse quantizer 160 is configured to inversely quantize the quantized transform coefficients output from the quantizer 145 to generate transform coefficients. The inverse transformer 165 is configured to transform the transform coefficients output from the inverse quantizer 160 from the frequency domain to the spatial domain and reconstruct a residual block. The adder 170 is configured to add the reconstructed residual block and the prediction block generated by the predictor 120 to reconstruct the current block. The pixels in the reconstructed current block are used as reference pixels when performing intra-frame prediction of subsequent blocks.

[0071] The loop filter unit 180 is configured to filter the reconstructed pixels to reduce blocking artifacts, ringing artifacts, and blurring artifacts due to block-based prediction and transform / quantization. The loop filter unit 180 may include at least one of a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186. The deblocking filter 182 is configured to filter the boundaries between the reconstructed blocks to remove the block artifacts caused by block-by-block encoding / decoding, and the SAO filter 184 is configured to perform additional filtering on the deblocking filtered video.

[0072] Specifically, the SAO filter 184 is a filter for compensating for the difference between the reconstructed sample and the original sample caused by lossy coding, and performs filtering in a manner of adding the corresponding offset to each reconstructed sample. The ALF 186 is configured to perform filtering on the target sample to be filtered by applying the filter coefficient to the target sample and the adjacent samples of the target sample. The ALF 186 can be configured to divide the samples included in the image into predetermined groups, and then determine a filter to be applied to the corresponding group to perform filtering on each group differentially. Information about the filter coefficient to be used for the ALF can be encoded and notified to the video decoding device with a signal. The reconstructed blocks filtered by the loop filtering unit 180 are stored in the memory 190. Once all blocks in an image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.

[0073] Figure 5 is an exemplary functional block diagram of a video decoding device capable of implementing the technology of the present invention. Figure 5 A video decoding apparatus and elements of the apparatus are described below.

[0074] The video decoding apparatus may include an entropy decoder 510, a rearrangement unit 515, an inverse quantizer 520, an inverse transformer 530, a predictor 540, an adder 550, a loop filter unit 560, and a memory 570. Figure 1 In the video encoding device of the present invention, each element of the video decoding device can be implemented by hardware, software, or a combination of hardware and software. In addition, the function of each element can be implemented by software, and the microprocessor can be implemented to execute the software function corresponding to each element.

[0075] The entropy decoder 510 is configured to determine the current block to be decoded by decoding the bit stream generated by the video encoding device and extracting information related to block partitioning, and extract prediction information required to reconstruct the current block and information about the residual signal, etc. The entropy decoder 510 is configured to extract information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), determine the size of the CTU, and partition the image into CTUs of a determined size. Then, the decoder is configured to determine the CTU as the highest level (i.e., the root node) of the tree structure, and extract partition information about the CTU to partition the CTU using the tree structure.

[0076] For example, when the CTU is segmented using the QTBTTT structure, the first flag (QT_split_flag) related to the segmentation of the QT is extracted to segment each node into four nodes of the sublayer. For the node corresponding to the leaf node of the QT, the second flag (MTT_split_flag) related to the segmentation of the MTT and the information about the segmentation direction (vertical / horizontal) and / or the segmentation type (binary / trifurcated) are extracted, so that the corresponding leaf node is segmented with the MTT structure. Thus, each node below the leaf node of the QT is recursively segmented with the BT or TT structure.

[0077] As another example, when the QTBTTT structure is used to split the CTU, the CU split flag (split_cu_flag) indicating whether the CU is split can be extracted. When the corresponding block is split, the first flag (QT_split_flag) can be extracted. In the split operation, after zero or more recursive QT splits, zero or more recursive MTT splits may occur for each node. For example, the CTU may directly undergo MTT splitting without QT splitting, or only undergo QT splitting multiple times. As another example, when the QTBT structure is used to split the CTU, the first flag (QT_split_flag) related to QT splitting is extracted, and each node is split into four nodes of the lower layer. Then, the split flag (split_flag) indicating whether the node corresponding to the leaf node of the QT is further split with BT and the split direction information are extracted.

[0078] Once the current block to be decoded is determined by partitioning through the tree structure, the entropy decoder 510 is configured to extract information about the prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 510 is configured to extract syntax elements for intra-prediction information (intra-prediction mode) for the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 is configured to extract syntax elements for inter-prediction information, i.e., information indicating a motion vector and a reference image referenced by the motion vector.

[0079] The entropy decoder 510 is further configured to extract information about the transform coefficients of the quantized current block as information related to quantization and information about the residual signal. The rearrangement unit 515 may be configured to change the sequence of the one-dimensional quantized transform coefficients entropy decoded by the entropy decoder 510 into a 2-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the video encoding device.

[0080] The inverse quantizer 520 is configured to inversely quantize the quantized transform coefficients and inversely quantize the quantized transform coefficients using quantization parameters. The inverse quantizer 520 can be configured to apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantizer 520 can be configured to perform inverse quantization by applying a quantization coefficient (scaling value) matrix from a video encoding device to a two-dimensional quantized transform coefficient array. The inverse transformer 530 is configured to inversely transform the inverse quantized transform coefficients from the frequency domain to the spatial domain to reconstruct the residual signal, thereby generating a residual block of the current block.

[0081] In addition, when the inverse transformer 530 performs inverse transformation on only a partial region (sub-block) of the transform block, the inverse transformer 530 extracts a flag (cu_sbt_flag) indicating that only the sub-block of the transform block has been transformed, directional (vertical / horizontal) information (cu_sbt_horizontal_flag) about the sub-block, and / or position information (cu_sbt_pos_flag) about the sub-block, and inversely transforms the transform coefficient of the sub-block from the frequency domain to the spatial domain. Then, the inverse transformer 530 reconstructs the residual signal and fills the uninverse transformed region with a "0" value as a residual block, thereby generating a final residual block of the current block.

[0082] In addition, when MTS is applied, the inverse transformer 530 is configured to determine the transformation function or transformation matrix to be applied in the horizontal direction and the vertical direction respectively using the MTS information (mts_idx) signaled from the video encoding device, and inversely transform the transformation coefficients in the transformation block in the horizontal direction and the vertical direction using the determined transformation function. The predictor 540 may include an intra-frame predictor 542 and an inter-frame predictor 544. When the prediction type of the current block is intra-frame prediction, the intra-frame predictor 542 is activated, and when the prediction type of the current block is inter-frame prediction, the inter-frame predictor 544 is activated.

[0083] The intra-frame predictor 542 is configured to determine the intra-frame prediction mode of the current block among the multiple intra-frame prediction modes based on the syntax elements of the intra-frame prediction mode extracted from the entropy decoder 510, and predict the current block using reference pixels around the current block according to the intra-frame prediction mode. The inter-frame predictor 544 is configured to determine the motion vector of the current block and the reference image referenced by the motion vector using the syntax elements of the intra-frame prediction mode extracted from the entropy decoder 510, and predict the current block based on the motion vector and the reference image.

[0084] The adder 550 is configured to reconstruct the current block by adding the residual block output from the inverse transformer to the prediction block output from the inter-frame predictor or the intra-frame predictor. When intra-predicting a subsequent block to be decoded, the pixels in the reconstructed current block are used as reference pixels. The loop filtering unit 560 may include a deblocking filter 562 and an SAO filter 564. The deblocking filter 562 performs deblocking filtering on the boundaries between the reconstructed blocks to remove block artifacts caused by block-by-block decoding.

[0085] The SAO filter 564 is configured to perform additional filtering on the reconstructed blocks after deblocking filtering to compensate for the difference between the reconstructed pixels and the original pixels caused by lossy coding. The ALF 566 is configured to perform filtering on the target sample to be filtered by applying the filter coefficients to the target sample and the adjacent samples of the target sample. The ALF 566 can be configured to divide the samples in the image into predetermined groups, and then determine a filter to be applied to the corresponding group to perform filtering on each group differentially. The filter coefficients of the ALF are determined based on information about the filter coefficients decoded from the bitstream. The reconstructed blocks filtered by the loop filtering unit 560 are stored in the memory 570. When all blocks in an image are reconstructed, the reconstructed image is used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.

[0086] The present embodiment relates to encoding and decoding of video as described above. More specifically, regarding a coding tool for improving compression performance of inter-frame prediction, an encoder / decoder uses a coding tool to adjust correction of motion vectors based on a high-level syntax. In addition, the present embodiment provides a method for simply performing an integer sample search operation for searching integer offsets and a fractional sample refinement operation for searching sub-pixel offsets in a coding tool related to motion vector refinement.

[0087] The following embodiments relate to a coding tool for improving compression performance of inter-frame prediction, which can be operated by the inter-frame predictor 124 of a video encoding device and the inter-frame predictor 544 of a video decoding device. As used herein, the term "target block" may have the same meaning as the current block or coding unit (CU) described above, or may represent a local area of ​​a CU.

[0088] I. Bi-directional Optical Flow (BDOF)

[0089] Bidirectional optical flow is a technique for additionally compensating for the motion of samples predicted using bidirectional motion prediction, assuming that samples or objects constituting a video move at a constant speed and that sample values ​​hardly change. Figure 6 is an exemplary diagram illustrating the concept of a BDOF according to an embodiment of the present invention.

[0090] Assume that by (normal) bidirectional motion prediction for the target block, in the reference image Ref 0 and Ref 1 The bidirectional motion vector MV pointing to the corresponding area (ie, the reference block) that is most similar to the target block to be encoded in the current image has been determined. 0 and MV 1 The two motion vectors have values ​​representing the motion of the entire target block.

[0091] exist Figure 6 In the example, P 0 The motion vector MV 0 The reference image Ref corresponding to the sample P in the target block is indicated 0 The samples in P 1 The motion vector MV 1 The reference image Ref corresponding to the sample P in the target block is indicated 1 In addition, assuming Figure 6 The motion of the sample P in is slightly different from the overall motion of the target block. Figure 6 Ref 0 The object at sample A in the target block of the current image moves to Ref via sample P in the target block of the current image. 1 When sample B in , sample A and sample B can have values ​​very similar to each other. Accordingly, in this case, Ref 0 The point in the block that is most similar to the sample P in the target block is not the point represented by the motion vector MV 0 Indicated P 0 , but from P 0 has been shifted by a predetermined displacement vector (v x t 0 , v y t 1 ) of sample A. Ref 1 The point in the block that is most similar to the sample P in the target block is not the point represented by the motion vector MV 1 Indicated P 1 , but from P 1 has been shifted by a predetermined displacement vector (-v x t0 , -v y t 1 ) of sample B. Specifically, t 0 and t 1 Respectively represent Ref 0 and Ref 1 The temporal distance relative to the current image and is calculated based on the picture order count (POC). x , v y ) is called "optical flow".

[0092] When predicting the value of sample P of the current block in the current image, two reference samples A and B can achieve a better prediction than the value of the sample P of the current block in the current image. 0 and MV 1 Indicated reference sample P 0 and P 1 More accurate predictions. (0) (i, j) represents the sample (i, j) in the target block as represented by the motion vector MV 0 Indicates the reference image Ref 0 The value of the sample in, and I (1) (i, j) represents the sample (i, j) in the target block as represented by the motion vector MV 1 Indicates the reference image Ref 1 The value of the sample in .

[0093] As shown by the BDOF motion vector (v corresponding to the sample in the target block x , v y ) indicates the reference image Ref 0 The value of sample A in can be defined as I (0) (i+v x t 0 , j+v y t 0 ), and the reference image Ref 1 The value of sample B in can be defined as I (1) (iv x t 1 , jv y t 1 ). In particular, when linear approximation is performed using only the first-order term of the Taylor series, A and B can be expressed as Equation 1.

[0094] Equation 1

[0095]

[0096] Among them, I x (k)and I y (k) (k=0, 1) is in Ref 0 and Ref 1 The gradient value at position (i, j) in the horizontal and vertical directions. As mentioned above, t 0 and t 1 Respectively represent the Ref relative to the current image 0 and Ref 1 The time axis distance is expressed as t based on POC 0 =POC(Current)-POC(Ref 0 ), and t 1 =POC(Ref 1 )-POC (current).

[0097] The bidirectional optical flow of each sample in the block (v x , v y ) is determined as the solution that minimizes Δ, which is defined as the difference between sample A and sample B. Δ can be defined by Equation 2 using the linear approximation of A and B derived from Equation 1.

[0098] Equation 2

[0099]

[0100] For simplicity, the sample position (i, j) is omitted from each term in Equation 2.

[0101] In order to achieve a more robust optical flow estimation, it is assumed that the motion is locally consistent with neighboring samples. Considering the difference Δ in equation 2 of all samples (i', j') present in a mask Ω of a certain size centered on sample (i, j), the BDOF motion vector of the current sample (i, j) to be predicted is calculated. As shown in equation 3, the optical flow of the current sample (i, j) can be determined as the vector that minimizes the sum of the squares of the differences Δ[i', j'] obtained for each sample in the mask Ω, as shown in equation 3.

[0102] Equation 3

[0103]

[0104] The bidirectional optical flow of this embodiment can be applied to a case where one of the two reference images used for bidirectional prediction is before the current image in display order, and the other is after the current image, and the distances between the two reference images and the current image are equal to each other, that is, the differences in picture order counts (POCs) between each reference image and the current image are equal to each other. Therefore, t 0 and t 1 Can be excluded.

[0105] In addition, the bidirectional optical flow of the present embodiment can be applied only to the brightness component. For the target block to which the bidirectional prediction is applied, the bidirectional optical flow of the present embodiment is performed based on the sub-block rather than the pixel. In the following description, for simplicity, the sub-blocks of various sizes, which may be 2×2, 4×4, and 8×8, are assumed to have a size of 4×4.

[0106] Before performing the optical flow, the inter-frame predictor 124 of the video encoding apparatus is configured to generate two reference blocks for the target block using the above-mentioned bidirectional prediction. The first reference block of the two reference blocks represents the first motion vector MV 0 From the reference image Ref 0 The second reference block represents a block composed of samples of the prediction generated using the motion vector MV 1 From the reference image Ref 1 The generated predictions consist of blocks of samples.

[0107] The inter-frame predictor 124 is configured to calculate an optical flow (v) for each of the 4×4 sub-blocks constituting the target block using the horizontal gradient values ​​and the vertical gradient values ​​of the predicted samples in the first reference block and the second reference block. x , v y ). Optical flow (v x , v y ) is determined such that 0 The predicted samples are compared with the reference image Ref 1 The inter-frame predictor 124 is configured to minimize the difference between the predicted samples by using (v x , v y ) and the gradient of the predicted samples in the 4×4 sub-block to derive the sample offsets used to modify the bidirectionally predicted samples of the 4×4 sub-block.

[0108] As shown in Equation 4, the inter-frame predictor 124 is configured to calculate the horizontal gradient and the vertical gradient of the sample value at the position (i, j).

[0109] Equation 4

[0110]

[0111] Where k is 0 or 1, I (0) (i, j) and I (1) (i, j) represent the sample values ​​at the position (i, j) in the first reference block and the second reference block, respectively. In addition, shift1 is a value derived from the bit depth of the luminance component, for example, Shift1 = max(6, bit depth - 6).

[0112] In order to derive the gradient of samples located at the boundary of each reference block, samples outside the boundary of the first reference block and the second reference block are needed. Figure 6 As shown, each reference block is extended by one column to the left and right, and by one row to the top and bottom. In order to reduce the amount of calculation, each sample in the extended portion can be filled with the sample at the most adjacent position in the reference block or an integer sample. In addition, the gradient at the sample position outside the boundary of each reference block can be filled with the gradient corresponding to the sample at the most adjacent position.

[0113] The inter-frame predictor 124 is configured as follows Figure 7 As shown, using the horizontal gradient and the vertical gradient in the 6×6 window covering the 4×4 sub-block, S1, S2, S3, S5 and S6 corresponding to the autocorrelation and cross-correlation of the gradient are calculated as shown in Equation 5.

[0114] Equation 5

[0115] S 1 =∑ (i，j)∈Ω ψ x (i, j)·ψ x (i, j), S 3 =∑ (i，j)∈Ω θ(i, j)·ψ x (i, j)

[0116]

[0117] S 5 =∑ (i，j)∈Ω ψ y , (i, j)·ψ y (i, j) S 6 =∑ (i，j)∈Ω θ(i, j)·ψ y (i, j)

[0118] As mentioned above, Ω represents a mask of a certain size. In addition, as shown in the following equation 6, Ψ x (i, j) represents the sum of the horizontal gradient values ​​at position (i, j) in the first reference block and the second reference block, Ψ y (i, j) represents the sum of the vertical gradient values ​​at the position (i, j) in the first reference block and the second reference block, and θ(i, j) represents the difference between the sample value at the position (i, j) in the second reference block and the sample value at the position (i, j) in the first reference block.

[0119] Equation 6

[0120]

[0121] θ(i, j)=(I (1)(i, j)>>n b )-(I (0) (i, j)>>n b )

[0122] Among them, n a and n b is a value derived from the bit depth, and has values ​​of min(1, bit depth - 11) and min(4, bit depth - 8).

[0123] The inter-frame predictor 124 is configured to calculate the optical flow (v x , v y ), as shown in Equation 7.

[0124] Equation 7

[0125]

[0126]

[0127] in, and th′ BIO =2 max(5,BD-7) . is the floor function, and

[0128] The sample offset for modifying the predicted sample at each sample position (x, y) in the 4×4 sub-block in the target block can be calculated using the optical flow calculated for the 4×4 sub-block and the gradient value at the sample position (x, y), as shown in Equation 8.

[0129] Equation 8

[0130]

[0131] Among them, rnd() represents rounding operation.

[0132] The inter-frame predictor 124 is configured to use the sample offset b(x, y) at the position (x, y) and the prediction sample I in the first reference block. (0) (x, y) and the predicted sample I in the second reference block (1) (x, y) to generate the final predicted sample pred(x, y), as shown in Equation 9.

[0133] Equation 9

[0134] pred(x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+o offset )>>shift

[0135] Where shift is Max(3,15-bit depth), and O offset is the value used for rounding operations and is half of shift.

[0136] As described above, the bidirectional optical flow technique uses the value of the sample predicted using the motion information (two motion vectors and two reference images) for bidirectional prediction. Therefore, the inter-frame predictor 544 of the video decoding device can also perform the bidirectional optical flow in the same manner as the video encoding device using the motion information (motion vector, reference image) for bidirectional prediction received from the video encoding device. Accordingly, the video encoding device is not required to signal the video decoding device with additional information for bidirectional optical flow processing.

[0137] II. Affine Motion Prediction

[0138] The above-mentioned inter-frame prediction is a motion prediction that reflects a translational motion model. In other words, this is a technique for predicting motion in the horizontal direction (x-axis direction) and the vertical direction (y-axis direction). However, in reality, there may be various types of motion, for example, rotation, enlargement or reduction in addition to translational motion. One aspect of the present embodiment provides affine motion prediction that can reflect such various types of motion.

[0139] Figure 8a and Figure 8b is an exemplary diagram showing affine motion prediction according to an embodiment of the present invention.

[0140] There may be two types of models for affine motion prediction. One model is a model using motion vectors of two control points at the upper left and upper right corners of the target block currently to be encoded, that is, four parameters such as Figure 8a Another model is a model that uses the motion vectors of three control points at the upper left corner, upper right corner, and lower left corner of the target block, that is, six parameters, such as Figure 8b shown.

[0141] The four-parameter affine model is expressed as shown in Equation 10. The motion at the sample position (x, y) in the target block can be calculated according to Equation 10. Assume that the position of the upper left sample of the target block is (0, 0).

[0142] Equation 10

[0143]

[0144] The six-parameter affine model is expressed by Equation 11. The motion at the sample position (x, y) in the target block can be calculated according to Equation 11.

[0145] Equation 11

[0146]

[0147] Among them, (mv 0x ,mv 0y ) is the motion vector of the upper left control point, (mv 1x ,mv 1y ) is the motion vector of the upper right control point, and (mv 2x ,mv 2y ) is the motion vector of the lower left control point. In addition, W is the horizontal length of the target block, and H is the vertical length of the target block.

[0148] Affine motion prediction can be performed on each sample in the target block using the motion vector calculated according to Equation 10 or Equation 11. Alternatively, in order to reduce computational complexity, prediction can be performed on each sub-block partitioned from the target block, such as Fig. 9 As shown. For example, the size of the sub-block may be 4×4, 2×2, or 8×8. In the following exemplary embodiment, affine motion prediction for the target block is performed based on a 4×4 sub-block. This example is only for ease of explanation, and the present invention is not limited thereto.

[0149] In the sub-block based affine motion prediction, the motion vector (affine motion vector) of each sub-block is calculated by substituting the center position of each sub-block into (x, y) of Equation 10 or Equation 11. Specifically, the center position may be the actual center point of the sub-block or the lower right sample position of the center point. For example, in the case where the coordinates of the lower left sample in the 4×4 sub-block are (0, 0), the center position of the sub-block may be (1.5, 1.5) or (2, 2). Using the affine motion vector (mv) of the sub-block x ,mv y ) to generate a predicted block for each sub-block. The motion vector (mv x ,mv y ) can be set to have 1 / 16 sample accuracy. Specifically, the motion vector (mv) calculated according to Equation 1 or Equation 2 x ,mv y ) may be rounded to 1 / 16 sample unit.

[0150] Affine motion prediction can be performed not only on the luma component but also on the chroma component. In the case of a 4:2:0 video format, when affine motion prediction for the luma component is performed based on a 4×4 sub-block, affine motion prediction for the chroma component can be performed based on a 2×2 sub-block. The motion vector (mv) of each sub-block of the chroma component can be derived from the motion vector of the corresponding luma component. x ,mv y). Alternatively, the size of the sub-block used for affine motion prediction of the chrominance component may be the same as the size of the sub-block used for the luma component.

[0151] When affine motion prediction for luma components is performed based on 4×4 sub-blocks, affine motion prediction for chroma components can also be performed based on 4×4 sub-blocks. In particular, since a 4×4 sub-block for chroma components corresponds to four 4×4 sub-blocks for luma components, the motion vector (mv) of the sub-block for the chroma components can be calculated by calculating the average of the motion vectors of the four corresponding sub-blocks of the luma component. x ,mv y ).

[0152] The video encoding device is configured to perform intra prediction, inter prediction (translational motion prediction), affine motion prediction, etc., and calculate the rate-distortion (RD) cost to select the best prediction method. In order to perform affine motion prediction, the inter predictor 124 of the video encoding device determines which type of model of the two types of models to use, and determines two or three control points according to the determined type. The inter predictor 124 is configured to calculate the motion vector (mv) of each sub-block in the 4×4 sub-block in the target block using the motion vector of the control point. x ,mv y Then, the inter-frame predictor 124 is configured to use the motion vector (mv) of each sub-block. x ,mv y ) performs motion compensation in a reference image on a sub-block by sub-block basis to generate a predicted block for each sub-block in a target block.

[0153] The entropy encoder 155 of the video encoding device is configured to encode an affine-related syntax element including a flag indicating whether affine motion prediction is applied to a target block, type information indicating the type of an affine model, and motion information indicating a motion vector of each control point, and transmit it to the video decoding device. When affine motion prediction is performed, type information and motion information about a control point may be signaled, and motion vectors of as many control points as the number determined according to the type information may be signaled.

[0154] The video decoding apparatus is configured to determine the type of the affine model and the control point motion vector using the syntax elements notified by the signal, and calculate the motion vector (mv) of each 4×4 sub-block in the target block using Equation 10 or Equation 11. x ,mv y ). When the motion vector resolution information about the affine motion vector of the target block is signaled, the motion vector (mv x ,mv y) is corrected to the accuracy identified by the motion vector resolution information.

[0155] The video decoding apparatus is configured to decode the video by using a motion vector (mv x ,mv y ) Motion compensation is performed within the reference image to generate a predicted block for each subblock. In order to reduce the amount of bits required to encode the motion vector of the control point, a method such as that used in the above-mentioned normal inter-frame prediction (translational motion prediction) can be applied.

[0156] As an example, in the merge mode, the inter-frame predictor 124 of the video encoding device is configured to derive the motion vector of each control point from the neighboring blocks of the target block. Figure 4 The neighboring samples L, BL, A, AR and AL of the target block shown in the figure derive a predefined number of merge candidates to generate a merge candidate list. Each merge candidate included in the list corresponds to a pair of motion vectors of two or three control points.

[0157] First, the inter-frame predictor 124 is configured to derive merge candidates from the control point motion vectors of the neighboring blocks predicted in affine mode in the neighboring blocks. In some embodiments, the number of merge candidates derived from the neighboring blocks predicted in affine mode can be limited. For example, the inter-frame predictor 124 can be configured to derive two merge candidates from the neighboring blocks predicted in affine mode: one of L and BL and one of A, AR and AL. Priorities can be assigned in the order of L and BL and in the order of A, AR and AL. When the total number of merge candidates is greater than or equal to 3, the inter-frame predictor 124 can be configured to derive the necessary number of merge candidates from the translation motion vectors of the neighboring blocks.

[0158] Fig.10 1 is an example diagram showing a method for deriving merge candidates for affine motion prediction from translation motion vectors of adjacent blocks. The inter-frame predictor 124 is configured to derive control point motion vectors CPMV1, CPMV2, CPMV3 from adjacent block groups {B2, B3, A2}, adjacent block groups {B1, B0}, and adjacent block groups {A1, A0}, respectively. As an example, the priority in each adjacent block group can be assigned in the order of B2, B3, A2, the order of B1 and B0, and the order of A1 and A0. In addition, another control point motion vector CPMV4 is derived from the co-located block T in the reference image. The inter-frame predictor 124 is configured to generate as many merge candidates as required by combining two or three of the four control point motion vectors. The combined priority assignment is shown below. The elements in each group are listed in the order of the upper left corner, upper right corner, and lower left corner of the control point motion vector.

[0159] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0160] The inter-frame predictor 124 is configured to select a merge candidate in the merge candidate list and perform affine motion prediction on the target block. When the selected candidate consists of two control point motion vectors, affine motion prediction is performed using a four-parameter model. On the other hand, when the selected candidate consists of three control point motion vectors, affine motion prediction is performed using a six-parameter model. The entropy encoder 155 of the video encoding device encodes index information indicating the merge candidate selected from the merge candidates in the merge candidate list and notifies the video decoding device with a signal.

[0161] The entropy decoder 510 of the video decoding device decodes the index information signaled from the video encoding device. The inter-frame predictor 544 of the video decoding device constructs a merge candidate list in the same manner as the video encoding device, and is configured to perform affine motion prediction using a control point motion vector corresponding to the merge candidate indicated by the index information.

[0162] As another example, in the AMVP mode, the inter-frame predictor 124 of the video encoding device is configured to determine the type of affine model and the control point motion vector for the target block. Then, the inter-frame predictor 124 is configured to determine a motion vector difference, which is the difference between the actual control point motion vector of the target block and the predicted motion vector of each control point, and sends the motion vector difference corresponding to the control point respectively. Accordingly, the inter-frame predictor 124 of the video encoding device configures a list of a predefined number of affine AMVPs. When the target block is a 4-parameter type, the candidates included in the list are each composed of two control point motion vectors in a pair. On the other hand, when the target block is a 6-parameter type, the candidates included in the list are each composed of three control point motion vectors in a pair. The affine AMVP list can be derived using the control point motion vectors or translation motion vectors of adjacent blocks in a manner similar to the above-mentioned method of constructing a merged candidate list.

[0163] However, in order to derive candidates to be included in the affine AMVP list, there may be a restriction that only neighboring blocks are considered, the neighboring blocks referenced with Figure 4 The target blocks in the adjacent blocks are the same as the reference image.

[0164] In addition, in the AMVP mode, the affine model type of the target block should be considered. When the affine model type of the target block is a 4-parameter type, the video encoding device uses the affine model of the adjacent block to derive two control point motion vectors (the upper left and upper right control point motion vectors of the target block). When the affine model type of the target block is a 6-parameter type, the device uses the affine model of the adjacent block to derive three control point motion vectors (the upper left, upper right and lower left control point motion vectors of the target block).

[0165] When the neighboring block is of 4-parameter type, two or three control point motion vectors are predicted according to the affine model type of the target block using the two control point motion vectors of the neighboring block. For example, the affine model of the neighboring block expressed by equation 10 may be used. In equation 10, (mv 0x ,mv 0y ) and (mv 1x ,mv 1y ) are replaced by the upper left and upper right control point motion vectors of the adjacent blocks, respectively. In addition, W is replaced by the horizontal length of the adjacent blocks. The predicted motion vector for each control point of the target block is derived by placing the difference between the position of the corresponding control point of the target block and the position of the upper left corner of the adjacent block into (x, y).

[0166] When the neighboring block is of 6-parameter type, two or three control point motion vectors are predicted according to the affine model type of the target block using the three control point motion vectors of the neighboring block. For example, the affine model of the neighboring block expressed by equation 11 may be used. In equation 11, (mv 0x ,mv 0y )、(mv 1x ,mv 1y ) and (mv 2x ,mv 2y ) are replaced by the control point motion vectors of the upper left corner, upper right corner, and lower left corner of the neighboring block, respectively. The predicted motion vector for each control point of the target block is derived by placing the difference between the position of the corresponding control point of the target block and the position of the upper left corner of the neighboring block into (x, y).

[0167] The inter-frame predictor 124 of the video encoding device is configured to select a candidate in the affine AMVP list and generate a motion vector difference between the motion vector of each actual control point and the predicted motion vector of the corresponding control point of the selected candidate. The entropy encoder 155 of the video encoding device is configured to encode type information indicating the affine model type of the target block, index information indicating the selected candidate among the candidates in the affine AMVP list, and the motion vector difference corresponding to each control point, and send it to the video decoding device.

[0168] The inter-frame predictor 544 of the video decoding device is configured to determine the affine model type using the information notified by the video encoding device with a signal, and generate a motion vector difference for each control point. Then, the inter-frame predictor is configured to generate an affine AMVP list in the same manner as the video encoding device, and select a candidate indicated by the index information notified by the signal in the affine AMVP list. The inter-frame predictor 544 of the video decoding device is configured to calculate the motion vector of each control point by adding the predicted motion vector of each control point of the selected candidate to the corresponding motion vector difference.

[0169] III. Sample-by-sample adjustment of affine motion prediction samples

[0170] Subblock-by-subblock affine motion prediction for a target block has been described above. Another aspect of the invention relates to adjusting sample values ​​of predicted samples generated by subblock-by-subblock affine motion prediction on a sample-by-sample basis. Motion according to the position of each sample is additionally compensated in each subblock forming the basis for the affine motion prediction.

[0171] When the sample value in any one sub-block generated as a result of sub-block-by-sub-block affine motion prediction for the target block is I(x, y), the video encoding apparatus calculates the horizontal gradient gx(i, j) and the vertical gradient gy(i, j) at each sample position. Equation 12 may be used to calculate the gradient.

[0172] Equation 12

[0173] g x (i,j)=I(i+1,j)-I(i-1,j)

[0174] gv(i,j)=I(i,j+1)-I(i,j-1)

[0175] The sample offset ΔI(i, j) used to adjust the prediction sample is calculated by the following equation.

[0176] Equation 13

[0177] ΔI(i, j) = g x (i, j)*Δmv x (i, j)+g y (i, j)*Δmv y (i, j)

[0178] Among them, Δmv(i, j) represents the motion offset, that is, the difference between the affine motion vector in the sample (i, j) and the affine motion vector at the center position of the sub-block, and can be calculated by applying Equation 10 or Equation 11 according to the affine model type of the target block. In other words, Δmv(i, j) can be calculated according to Equation 10 or Equation 11 by subtracting the motion vector given when the sub-block center position is placed in (x, y) from the motion vector given when (i, j) is placed in (x, y). In addition, Δmv(i, j) can be calculated by substituting the horizontal offset and the vertical offset from the sub-block center position to the sample position (i, j) into (x, y) in Equation 11 and Equation 12 and removing the last term "+mv 0x ” and “+mv 0y " is used to calculate Δmv(i, j). The center position may be the actual center point of the sub-block, or may be the lower right sample position of the center point.

[0179] The motion vector of each control point of the target block used to calculate Δmv(i, j) and the difference between the sample position (i, j) and the center position of the sub-block are the same for all sub-blocks. Accordingly, the value of Δmv(i, j) can be used only to calculate one sub-block (e.g., the first sub-block) and can be reused for other sub-blocks.

[0180] The technique of the present invention is based on the assumption that the object moves at a constant speed and the variation of the sample values ​​is uniform. Therefore, by substituting the x-component (Δmv x ) and the y component (Δmv y ) are multiplied by the horizontal sample gradient value and the vertical sample gradient value to obtain the sample change in the horizontal direction and the sample change in the vertical direction. The sample offset ΔI(i, j) is calculated by adding the two sample changes. The final value of the predicted sample is calculated as follows.

[0181] Equation 14

[0182] I'(i,j)=I(i,j)+ΔI(i,j)

[0183] When the sample-by-sample adjustment for the affine motion prediction sample is applied, the inter-frame predictor 124 of the video encoding device and the inter-frame predictor 544 of the video decoding device perform the above-mentioned processing to modify the sample value of the predicted sample generated by the affine motion prediction. The gradient value is derived from the predicted sample generated by the affine motion prediction, and Δmv(i, j) is derived from the control point motion vector of the target block. Therefore, the video encoding device is not required to signal the video decoding device with additional information for the present technology.

[0184] IV. Motion Vector Refinement

[0185] As described above, in the case of inter-frame prediction using bidirectional prediction, it is determined that 0 Selected reference imageRef 0 , About the reference image Ref 0 The motion vector MV 0 , from the reference image list L 1 Selected reference imageRef 1 , and about the reference image Ref 1 The motion vector MV 1 The motion vector refinement described in this embodiment is achieved by finely adjusting the two motion vectors MV used for bidirectional prediction. 0 and MV 1 A coding tool to improve the accuracy of inter-frame prediction. Motion vector refinement can be applied to merge mode, in which the motion vector of the neighboring block is used as the motion vector of the target block. Alternatively, motion vector refinement can be applied to AMVP mode (normal inter-prediction) or affine mode.

[0186] Fig.11 1 is an exemplary diagram showing the concept of a motion vector refinement coding tool according to an embodiment of the present invention. In bidirectional prediction, the inter-frame predictor 124 of the video encoding device is configured to 0 and the reference image Ref 1 The refined motion vector is searched around the initial motion vector of . In particular, the initial motion vector represents the two motion vectors MV used for bidirectional prediction. 0 and MV 1 . Use block matching to calculate the reference image Ref 0 and Ref 1 The distortion between the two candidate blocks in . Fig.11 As shown in the reference image Ref 0 and Ref 1 In the method, the sum of absolute differences (SAD) between two candidate blocks at positions determined by respective initial motion vectors and between two candidate blocks around the positions are calculated.

[0187] In other words, the sum of the absolute values ​​of the differences between the sample values ​​corresponding to each other in the two candidate blocks is calculated. The inter-frame predictor 124 is configured to select the motion vector candidate with the smallest SAD as the refined motion vector, and use the refined motion vector to generate a bidirectional prediction signal according to the inter-frame prediction. Although SAD is used to represent the difference between the two candidate blocks, the embodiment is not limited thereto. Any metric capable of expressing the difference between the two candidate blocks, such as the sum of squared error (SSE), can be used.

[0188] Motion vector refinement can be applied when one reference image precedes the current image in display order and the other reference image follows the current image, and the two reference images are at the same distance from the current image, that is, the difference in picture order count (POC) between the reference image and the current image is the same. Fig.11 As shown, for two reference images Ref 0 and the two initial motion vectors MV in Ref1 0 and MV 1 , the refined bidirectional motion vector candidate MV in the corresponding reference picture list 0 ' and MV 1 ' can be expressed as Equation 15.

[0189] Equation 15

[0190] MV 0 ′=MV 0 +MV_offset

[0191] MV 1 ′=MV 1 -MV_offset

[0192] Wherein, MV_offset is the offset of the initial motion vector refined according to the motion vector, and represents the difference between the motion vector candidate and the corresponding initial motion vector. The offset can be the sum of an integer offset in units of integer samples and a sub-pixel offset in units of sub-pixels or sub-pixel samples. As shown in Equation 15, the candidates of the two motion vectors conform to the mirror rule of MV difference. The process of searching for the best offset of the refined motion vector includes two operations: an integer sample search for searching the integer offset, and a fractional sample refinement for searching the sub-pixel offset.

[0193] First, in an integer sample search operation, the inter-frame predictor 124 is configured to search all 25 neighboring points, including Fig.12The center of the refinement shown. In particular, the center of the refinement corresponds to the position indicated by the initial motion vector. When the SAD between the two candidate blocks in a pair at the two positions determined according to the initial motion vector is less than a specific threshold, the operation after the integer sample search can be skipped. By calculating the SAD between the two candidate blocks corresponding to the adjacent points of the position indicated by the initial motion vector, the inter-frame predictor 124 is configured to search for a point with the minimum SAD, that is, minSAD. According to the result of the integer sample search, the integer offset of the motion vector candidate with minSAD is expressed as (intOffX, intOffY).

[0194] After the integer sample search operation is completed, when intOffX of the integer offset is not -2 or 2 and intOffY is not -2 or 2 {(intOffX!=(2||-2))&&(intOffY!=(2||-2))}, the inter-frame predictor 124 sets subPelFlag, which is an indication flag for applying the fractional sample refinement operation, to a "true" value. When subPelFlag is set to "true", the inter-frame predictor 124 performs the fractional sample refinement operation. The inter-frame predictor 124 is configured to generate an optimal sub-pixel offset by solving a parameter error surface equation in a two-dimensional space, as shown in Equation 16.

[0195] Equation 16

[0196] E(x,y)=A(x min ) 2 +B(yy min ) 2 +C

[0197] Where E(x, y) is the cost at position (x, y) and can be defined as the SAD between two candidate blocks corresponding to position (x, y). In addition, (xmin, ymin) is the sample position with the minimum cost C and represents the optimal sub-pixel offset.

[0198] Assuming that according to the integer sample search, the sample position corresponding to the integer offset is (0, 0), the optimal sub-pixel offset can be calculated using the costs of blocks corresponding to all four positions (top, bottom, left and right positions) (-1, 0), (1, 0), (0, 1) and (0, 1), as shown in Equation 17.

[0199] Equation 17

[0200] x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0)))

[0201] ymin =(E(0,-1)-E(0,1)) / (2(E(0,--1)+E(0,1)-2E(0,0)))

[0202] The constraint on the cost E(x, y) can be expressed by Equation 18.

[0203] Equation 18

[0204] E(x,y)≥0&&E(0,0)minimum

[0205] Based on Equation 18, “-0.5 <x min ,y min <0.5” is satisfied.

[0206] The inter-frame predictor 124 may be configured to generate an optimal offset in sub-pixel units by adding the sub-pixel offset and the integer offset, and then add the optimal offset to the initial motion vector to obtain a refined bidirectional motion vector using Equation 15. In addition, the inter-frame predictor 124 may be configured to generate a sample of inter-frame prediction of the target block using the refined bidirectional motion vector and the bidirectional reference image.

[0207] As described above, in the motion vector refinement process, the motion vector is refined using the motion information (two motion vectors and two reference images) for bidirectional prediction. Accordingly, the inter-frame predictor 544 of the video decoding device may also be configured to perform motion vector refinement in the same manner as the video encoding device using the motion information (motion vector and reference images) for bidirectional prediction and received from the video encoding device. Therefore, there is no need to signal additional information for the motion vector refinement process from the video encoding device to the video decoding device.

[0208] When motion vector refinement is applied, various complexity issues may arise, particularly on the decoder side, since the SAD values ​​of the candidate blocks must be calculated for each target block. When fractional sample refinement operations are performed on motion vectors based on samples corresponding to integer offsets, complex equations such as those shown in Equation 16 need to be solved. This process may be difficult to optimize with hardware. When merge mode is applied to inter-frame prediction, the need for fractional sample refinement operations in motion vector refinement may be relatively low. Accordingly, the motion vector refinement process for each block encoded in merge mode increases the complexity of the video encoding / decoding device and makes it difficult to expect corresponding compression efficiency.

[0209] In view of the above, the present invention provides a method for reducing the complexity of a video encoding / decoding device by the following operation, wherein the operation determines the need for sub-pixel-based motion vector adjustment by using integer offset information obtained in an integer sample search operation, and skips the fractional sample refinement operation according to the determined result. In the above motion vector refinement process, when the SAD of the initial motion vector pair is greater than or equal to a specific threshold, the integer sample search and fractional sample refinement operations are performed. The threshold is determined according to the size of the target block. For example, the threshold can be set to a value of sbHeight×sbWidth, that is, the area of ​​the target block.

[0210] In another example according to the present embodiment, the threshold value may be determined based on the quantization parameter and the area of ​​the target block. In another example, whether to perform the integer sample search and the fractional sample refinement operation may be determined based on the index value of the neighboring block searched in the merge mode. In the integer sample search operation of the motion vector refinement process, the search operation is performed on a total of 25 integer coordinates (including the center of the refinement corresponding to the initial motion vector) from -2 to +2 horizontally and vertically, such as Fig.12 As shown in FIG. 1 , the integer offset of the position of the block with minSAD is expressed as (intOffX, intOffY).

[0211] In order to skip the complex equation solving process according to the sub-pixel-based motion vector refinement operation as shown in Equation 16 and Equation 17, the fractional sample refinement operation can be terminated in advance after performing a search for integer samples of a specific condition in the integer sample search operation. When both intOffX and intOffY are 0, the inter-frame predictor 124 of the video encoding device can be configured to skip the fractional sample refinement operation of searching for the sub-pixel offset and set the sub-pixel offset as shown in Equation 17 to zero. Alternatively, when (intOffX, intOffY) is not (0, 0), the inter-frame predictor 124 can set subPelFlag to a "true" value to apply the fractional sample refinement operation. Thus, when (intOffX, intOffY) is (0, 0), the fractional sample refinement operation can be terminated in advance.

[0212] As another aspect of an embodiment of the present invention, when the SAD obtained in the integer sample search operation is less than a certain threshold, the inter-frame predictor 124 is configured to skip the fractional sample refinement operation of the sub-pixel offset of the search vector. For example, in addition to the minSAD, the second smallest and third smallest SADs are represented as SAD_1 and SAD_2, respectively. When the condition expressed as Equation 19 is satisfied, the fractional sample refinement operation can be skipped.

[0213] Equation 19

[0214] Abs(minSAD-SAD_1) <threshold&&Abs(minSAD-SAD_2)<threshold

[0215] Here, Abs(·) is a function that changes the input to an absolute value.

[0216] As another aspect of an embodiment according to the present invention, the inter-frame predictor 124 is configured to determine whether to skip the fractional sample refinement operation of the sub-pixel offset of the search motion vector based on the size of the motion vector of the neighboring block searched in the merge mode. For example, when two or more motion vectors of two or more neighboring blocks searched in the merge mode are the same or similar (for example, the difference between the motion vectors is less than a certain threshold), the fractional sample refinement operation can be skipped.

[0217] In the merge mode, based on the size of the motion vector searched at one or more specific indexes, the inter-frame predictor 124 is configured to determine whether to skip the fractional sample refinement operation of the sub-pixel offset of the search motion vector. For example, when the values ​​of the motion vectors searched in the upper left block, the lower left block, and the upper right block are equal or differ from each other by a small amount, the fractional sample refinement operation can be skipped in the merge mode. As another example, when the size of the motion vectors searched in the upper left block, the lower left block, and the upper right block is equal to 0 or less than a specific threshold, the fractional sample refinement operation can be skipped.

[0218] In the merge mode, the inter-frame predictor 124 is configured to determine whether to skip the fractional sample refinement operation of the sub-pixel offset of the search motion vector based on the process of deriving the motion vector searched at one or more specific indexes. For example, when the values ​​of the motion vectors of the upper left block, the lower left block, and the upper right block have been derived without the motion vector refinement process, the motion vector refinement process is not additionally performed in the current block. As another example, when the fractional sample refinement operation of the motion vector refinement process is not used in the process of deriving the motion vectors of the upper left block, the lower left block, and the upper right block, the fractional sample refinement operation can be skipped in the motion vector refinement process of the current block.

[0219] When one or more neighboring blocks (e.g., the upper left block, the lower left block, and the upper right block) of the current block use the merge mode and are encoded in the skip mode (i.e., for the block, cu_skip_flag==true (TRUE)), the image encoding / decoding device is configured to determine that the accuracy requirement for the motion vector of the current block is low, and thus skip the motion vector refinement process. Specifically, when skipping the motion vector refinement process, only the fractional sample refinement operation can be skipped, or the integer sample search operation including the fractional sample refinement operation can be skipped.

[0220] As described above, in the motion vector refinement process according to the present embodiment, the search complexity of the inter predictor 124 can be reduced by terminating the fractional sample refinement operation early.

[0221] The inter-frame predictor 124 may be configured to perform early termination of the motion vector refinement process based on the motion information (two motion vectors and two reference images) used for prediction, the motion vectors of the neighboring blocks searched in the merge mode, etc. Accordingly, the inter-frame predictor 544 of the video decoding device may also be configured to perform early termination of the motion vector refinement process in the same manner as the video encoding device based on the motion information (motion vector and reference images) used for bidirectional prediction received from the video encoding device, the motion vectors of the neighboring blocks searched in the merge mode, etc. Accordingly, there is no need to signal additional information for early termination of the motion vector refinement process from the video encoding device to the video decoding device.

[0222] Hereinafter, as another aspect according to an embodiment of the present invention, a method for performing an additional fractional sample refinement operation at a boundary of a search range of a motion vector refinement process will be described.

[0223] As described above, when intOffX is -2 or 2 or intOffY is -2 or 2 {(intOffX==(2||-2))||(intOffY=(2||-2))}, the fractional sample refinement operation of the motion vector refinement process is skipped. Since the result of the motion vector refinement is limited to -1.5 to 1.5 according to the condition shown in Equation 18, the compression performance of the video encoding / decoding device may be limited. Accordingly, the motion vector refinement according to the present invention can allow fractional sample refinement operations of motion vectors in sub-pixel units from -2 to -1.5 and from 1.5 to 2 while limiting the range of integer sample search to -2 to 2, thereby improving compression performance.

[0224] Even when intOffX is -2 or 2 or intOffY is -2 or 2, the inter-frame predictor 124 of the video encoding device can be configured to set subPelFlag to a true value in order to apply the fractional sample refinement operation. When subPelFlag is set to true, the inter-frame predictor 124 can be configured to perform a fractional sample refinement operation that searches for sub-pixel offsets in units of at least 1 / 4 pixels to perform motion vector refinement for coordinates less than or equal to -1.75 or greater than or equal to 1.75. As another example, when subPelFlag is set to a true value, the inter-frame predictor 124 can be configured to perform a fractional sample refinement operation that searches for sub-pixel offsets in units of at least 1 / 8 pixels to perform motion vector refinement for coordinates of -1.875 or 1.875.

[0225] Alternatively, by deleting subPelFlag, the inter predictor 124 may be configured to perform a fractional sample refinement operation for searching for a sub-pixel offset even when intOffX is -2 or 2 or intOffY is -2 or 2. As described above, when intOffX is -2 or 2 or intOffY is -2 or 2, the inter predictor 124 may be configured to perform a fractional sample refinement operation for searching for a sub-pixel offset in units of at least 1 / 4 pixel to refine a motion vector with coordinates of -1.75 or 1.75. In addition, when intOffX is -2 or 2 or intOffY is -2 or 2, motion vector refinement with coordinates of -1.25 or 1.25 is not performed.

[0226] In order to apply equation 17, (intOffX, intOffY) requires 4 positions of the top, bottom, left and right. However, when intOffX is -2 or 2 or intOffY is -2 or 2, one or more required positions may not exist. Accordingly, the 4 positions of the top, bottom, left and right of (intOffX, intOffY) can be set by applying Max(-2, intoffx-1), intOffX and Min(intOffX+1, 2) in the horizontal direction, and Max(-2, intOffY-1), intOffY and Min(intOffY+1, 2) in the vertical direction. In other words, when (intOffX, intOffY) is located at the outermost of 25 points, the outermost point (intOffX, intOffY) can be used as the top, bottom, left or right position.

[0227] The above-mentioned motion vector refinement coding tool requires an integer sample search to be performed by SAD calculation of 25 sample positions determined by the initial motion vector, and a fractional sample refinement to be performed according to the parameter error surface equation, so it may increase the complexity. In the following, a simplified motion vector refinement process will be described. When the fractional sample refinement operation is additionally performed, even including the case where intOffX is -2 or 2 or intOffY is -2 or 2, the simplified motion vector refinement process can reduce the complexity of the video encoding / decoding device.

[0228] In an embodiment of the present invention, Fig.12As shown, the inter-frame predictor 124 is configured to calculate the SAD of positions A1, A2, A3, and A4 in an integer sample search operation to find a position with a minSAD. When the position with the minSAD is A4, the SAD of positions A41, A42, A43, and A44 around A4 is calculated. Based on the calculated SAD, the inter-frame predictor 124 is configured to perform a fractional sample refinement operation using the parameter error surface equations shown in Equations 16 and 17, thereby generating (x min ,y min ). Since the above processing does not satisfy the constraint that the cost of the center point should be the smallest in the constraint shown in Equation 18, all positions within the search range can be searched in units of 1 / 8 pixels. This search method reduces the number of searches in the integer sample search operation from 25 to 8, thereby significantly reducing the search complexity.

[0229] In another embodiment of the present invention, Fig.13 As shown, the inter-frame predictor 124 is configured to calculate the SAD of the positions A1, A2, A3, A4 and A5 in the integer sample search operation, and perform a fractional sample refinement operation based on the calculated SAD, thereby generating (x min ,y min ). Check (x in quadrants 1 to 4 min ,y min )’s approximate location. Then, when (x min ,y min ) is located in the second quadrant, for example, the inter-frame predictor 124 is configured to additionally calculate the SADs of the positions B1, B2, B3, B4, and B5. Based on the calculated SADs, the inter-frame predictor 124 is configured to additionally perform a fractional sample refinement operation, thereby generating (x min ,y min ). The above process does not satisfy the constraint that the cost of the center point should be the smallest in the constraint shown in equation 18, and accordingly all positions within the search range can be searched in units of 1 / 8 pixels. Using this search method, the search of 25 sample positions required in the integer sample search operation is replaced by the search of 10 sample positions and additional fractional sample refinement. Thus, the complexity can be reduced.

[0230] In another embodiment of the present invention, Fig.13 As shown, the inter-frame predictor 124 is configured to calculate the SAD of the positions A1, A2, A3, A4 and A5 in the integer sample search operation, and perform a fractional sample refinement operation based on the calculated SAD to generate (x min ,y min). Since the above process does not satisfy the constraint that the cost of the center point should be the smallest in the constraint shown in equation 18, all positions within the search range can be searched in units of 1 / 8 pixels. Using this search method, the number of searches in the integer sample search operation can be reduced from 25 to 5. Thus, the search complexity can be significantly reduced.

[0231] In another embodiment of the present invention, Fig.14 As shown, the inter-frame predictor 124 is configured to calculate the SAD of positions A1 to A9 in an integer sample search operation to find a position with a minSAD. For example, when A5 is a position with a minSAD, the inter-frame predictor 124 is configured to perform a fractional sample refinement operation based on the SAD pre-calculated for positions A2, A4, A6, and A8 to generate (x min ,y min ). Since the above processing does not satisfy the constraint that the cost of the center point should be the smallest among the constraints shown in Equation 18, all positions within the search range can be searched in units of 1 / 8 pixels.

[0232] When A7 is the position with the minSAD according to the result of performing the integer sample search, the SADs of the positions B1 and B2 are additionally calculated. Based on the SADs of the positions A4, A7, A8, B1, and B2, the inter-frame predictor 124 generates (x min ,y min ). Since the above processing does not satisfy the constraint that the cost of the center point should be the smallest in the constraint shown in equation 18, all positions within the search range can be searched in units of 1 / 8 pixels. Using this search method, the number of searches in the integer sample search operation can be reduced from 25 times to a minimum of 8 times and a maximum of 11 times. Thus, the search complexity can be reduced.

[0233] As described above, in the motion vector thinning process according to the present embodiment, compression performance can be improved by extending the search range of sub-pixel offsets, and search complexity can be reduced by simplifying integer sample searches.

[0234] The video encoding device can use the motion information (two motion vectors and two reference images) for prediction to expand the search range of the motion vector refinement process and simplify the search operation. Therefore, the inter-frame predictor 544 of the video decoding device can also use the motion information (motion vector and reference image) for bidirectional prediction received from the video encoding device to expand the search range of the motion vector refinement process and simplify the search operation in the same manner as the video encoding device. Therefore, it is not necessary to signal additional information from the video encoding device to the video decoding device to simplify the search operation of the motion vector refinement process.

[0235] V. Signaling Advanced Syntax

[0236] Another aspect of the present invention relates to a technique for signaling a high-level syntax for controlling the on / off of the various tools described above. The optical flow, sample-by-sample adjustment for affine motion prediction samples, and motion vector refinement described above are coding tools for improving video coding efficiency. However, for specific content such as the following, such as screen content, the various coding tools described above may not help to improve compression performance. Accordingly, applying a coding unit based on signaling whether to apply each coding tool or a coding unit based on determining whether to apply a tool may instead reduce coding efficiency or increase computational complexity. The present invention provides a signaling technique for effectively controlling the above-mentioned coding tools.

[0237] In order to encode a series of images, the video encoding device is configured to encode a first high-level syntax element at a sequence level indicating whether to disable each of the above-mentioned multiple coding tools. The first high-level syntax element can be encoded in a sequence header, that is, in an SPS. In particular, the multiple coding tools include at least one of optical flow, sample-by-sample adjustment, and motion vector refinement.

[0238] When the first high-level syntax element indicates that each of the plurality of coding tools is not enabled, the corresponding coding tool is applied to the encoding of the sequence. In other words, each of the above coding tools is not applied to any image in the sequence. When the first high-level syntax element indicates that each of the plurality of coding tools is enabled, each of the plurality of coding tools can be used to encode the corresponding sequence. For example, optical flow is applicable to all images in the sequence. In addition, motion vector refinement is applicable to all images in the sequence.

[0239] The video encoding device is configured to encode a picture level flag according to the value of a first high-level syntax element, wherein the picture level flag indicates whether a corresponding coding tool is allowed in a picture header (i.e., a PPS) of each picture belonging to a sequence. For example, when the first high-level syntax element indicates that the corresponding coding tool is allowed at the sequence level, the picture level flag indicating whether the corresponding coding tool is allowed at the picture level is encoded. When the first high-level syntax element indicates that the corresponding coding tool is not allowed at the sequence level, the picture level flag is not encoded separately. Specifically, the video decoding device is configured to automatically set the picture level flag to a value indicating that the corresponding coding tool is not allowed.

[0240] The video decoding apparatus may be configured to set whether to allow the corresponding coding tool at the picture level according to the picture level flag signaled from the video encoding apparatus or the picture level flag of the automatically set value. When the picture level flag for the coding tool according to the present embodiment is not included in the bitstream, the video decoding apparatus sets each picture level flag to a value indicating that the corresponding coding tool is not allowed at the picture level.

[0241] According to the above-mentioned syntax structure, the video encoding device can be configured to adaptively determine whether the coding tool is allowed at the sequence level and the picture level. In addition, the video decoding device can be configured to adaptively determine whether the coding tool is allowed at the sequence level and the picture level by decoding the syntax signaled from the video encoding device.

[0242] In some other embodiments of the present invention, the image level flag of the above-mentioned embodiment can be replaced by a slice level flag. In other words, when the first high-level syntax element indicates that the corresponding coding tool is allowed at the sequence level, the video encoding device encodes the slice level flag indicating whether the corresponding coding tool is allowed at the slice level in the slice header. When the first high-level syntax element indicates that the corresponding coding tool is not allowed, the slice level flag is not encoded. The video encoding device can be configured to adaptively determine whether the coding tool is allowed at the sequence level and the slice level. The video decoding device can be configured to determine whether each tool is allowed at the sequence level and the slice level based on the first high-level syntax element or the slice level flag.

[0243] In another embodiment of the present invention, in order to encode a series of images, the video encoding device is configured to encode a first high-level syntax element at the sequence level indicating whether all the above-mentioned coding tools are enabled. The first high-level syntax element can be encoded in a sequence header, that is, in an SPS. When the first high-level syntax element at the sequence level indicates that any of the multiple coding tools is not enabled, the corresponding coding tool is not applied to the encoding of the sequence. In other words, none of the above-mentioned coding tools are applied to any image in the sequence. When the first high-level syntax element at the sequence level indicates that there is no constraint on the use of all the multiple coding tools, the video encoding device is configured to encode a second high-level syntax element at the slice level indicating whether all the multiple coding tools are enabled. The second high-level syntax element can be encoded in a slice header.

[0244] When the second high-level syntax element at the slice level indicates that any one of the plurality of coding tools is not enabled, the corresponding coding tool is not applied to the encoding of the corresponding slice. In other words, none of the above coding tools are applied to the slice. When the second high-level syntax element at the slice level is 0, there is no constraint on the use of all the plurality of coding tools. In addition, when the first high-level syntax element at the sequence level indicates that all the plurality of coding tools are disabled, the second high-level syntax element at the slice level is set to not enable any one of the plurality of coding tools, and the corresponding coding tool is not applied to the encoding of the corresponding slice.

[0245] In addition, in some other embodiments according to the present invention, the second high-level syntax element at the slice level described above can be replaced by the second high-level syntax element at the picture level. The second high-level syntax element can be encoded in the picture header (i.e., PPS). When the second high-level syntax element at the picture level indicates that all multiple coding tools are disabled, the corresponding coding tools are not applied to the encoding of the picture. In other words, none of the above-mentioned coding tools are applied to the picture. When the second high-level syntax element at the picture level is 0, there is no constraint on the use of all multiple coding tools. In addition, when the first high-level syntax element at the sequence level indicates that all multiple coding tools are disabled, the second high-level syntax element at the picture level is set to not enable any of the multiple coding tools, and the corresponding coding tools are not applied to the encoding of the picture.

[0246] As described above, according to the present embodiment, by providing a method for simply performing integer sample search operations and fractional sample refinement operations related to motion vector refinement in an encoding tool for improving compression performance of inter-frame prediction, the encoding efficiency of the encoder / decoder can be improved, thereby reducing complexity.

[0247] The various functions or methods described in the present invention can be implemented with instructions stored in a non-volatile recording medium, which can be read and executed by one or more processors. Non-volatile recording media include, for example, all types of recording devices in which data is stored in a form readable by a computer system. For example, non-volatile recording media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSD).

[0248] Although exemplary embodiments have been described for illustrative purposes, those skilled in the art will appreciate that various modifications and variations are possible without departing from the spirit and scope of the embodiments. For the sake of brevity and clarity, exemplary embodiments have been described. Accordingly, those of ordinary skill will appreciate that the scope of the embodiments is not limited to the embodiments explicitly described above, but is included in the claims and their equivalents.

Claims

1. A video decoding method, which is used to decode a coded image sequence block by block by a video decoding device, the video decoding method comprising: decoding at least one first high-level syntax element from a bitstream, the first high-level syntax element indicating whether each of at least one coding tool is enabled, the at least one coding tool comprising motion vector refinement; Decoding a second high-level syntax element from a bitstream according to at least one first high-level syntax element, and setting whether each of at least one coding tool is allowed at a picture level, the second high-level syntax element corresponding to each of the at least one coding tool; When it is identified based on the setting at the picture level that motion vector refinement is allowed for the current picture including the target block, generating a refined motion vector of the target block by refining an initial motion vector obtained from a bitstream; Generate inter-frame prediction samples of target blocks using bidirectional reference images and refined motion vectors; obtaining a residual signal for a target block; Reconstruct the target block based on the inter-frame prediction samples and the residual signal. The generation of refined motion vectors includes: Integer sample search operation, which includes: calculating metrics for pairs of two candidate blocks corresponding to pairs of initial positions indicated by initial motion vectors and pairs of two neighboring positions shifted from the initial positions in integer sample units, wherein the neighboring positions in integer sample units are 24 sample positions having a square distribution around each initial position, searching among the metrics for an integer offset corresponding to the position having the smallest metric; A fractional sample refinement operation is performed only when the integer offset is not located at the outermost side of the square distribution, and the fractional sample refinement operation comprises: Using the position with the minimum metric and the top, bottom, left, and right neighboring positions of the position with the minimum metric, a sub-pixel offset is derived based on a parametric error surface equation, The offset is generated by adding the sub-pixel offset to the integer offset, A refined motion vector is generated by applying the offset to the initial motion vector.

2. The method according to claim 1, wherein: A first high-level syntax element is decoded from the sequence header.

3. The method according to claim 2, wherein: Decoding the second high-level syntax elements includes: When the first high-level syntax element indicates that motion vector refinement is allowed, a picture level flag indicating whether motion vector refinement is allowed is decoded from a picture header of each picture belonging to the sequence.

4. The method according to claim 3, wherein: When the first high-level syntax element indicates that motion vector refinement is not allowed, the picture level flag is set to a value indicating that motion vector refinement is not allowed without decoding the picture level flag from the picture header.

5. The method according to claim 1, wherein: When the first high-level syntax element indicates that motion vector refinement is not allowed, coding tools for motion vector refinement are not allowed for any picture in the sequence.

6. The method according to claim 1, wherein: When a metric between two candidate blocks corresponding to the initial motion vector is less than a preset first threshold, both the integer sample search operation and the fractional sample refinement operation are skipped.

7. The method according to claim 1, wherein: When the integer offset is zero, the fractional sample thinning operation is skipped.

8. The method according to claim 1, wherein: When each difference between the minimum metric and the metrics acquired at the integer coordinates of two or more positions in the integer sample search operation is less than a preset second threshold, the fractional sample refinement operation is skipped.

9. The method according to claim 1, wherein: When an initial motion vector of a target block is generated by a merge mode, whether to perform an integer sample search operation or a fractional sample refinement operation is determined according to the size of a motion vector of a neighboring block considered as a merge candidate or a coding mode applied to the neighboring block.

10. A video encoding method, for encoding a coded image sequence block by block by a video encoding device, the video encoding method comprising: encoding at least one first high-level syntax element in a sequence header, the first high-level syntax element indicating whether each of at least one coding tool is enabled, the at least one coding tool comprising motion vector refinement; encoding a second high-level syntax element at a picture level according to the at least one first high-level syntax element, the second high-level syntax element at the picture level being related to whether each of the at least one coding tool is enabled; When it is identified based on the setting at the picture level that motion vector refinement is allowed for the current picture including the target block, generating a refined motion vector of the target block by refining the initial motion vector obtained for the target block; Generate inter-frame prediction samples of target blocks using bidirectional reference images and refined motion vectors; generating a residual signal for a target block based on the inter-frame prediction samples, and encoding information of the residual signal, The generation of refined motion vectors includes: Integer sample search operation, which includes: calculating metrics for pairs of two candidate blocks corresponding to pairs of initial positions indicated by initial motion vectors and pairs of two neighboring positions shifted from the initial positions in integer sample units, wherein the neighboring positions in integer sample units are 24 sample positions having a square distribution around each initial position, searching among the metrics for an integer offset corresponding to the position having the smallest metric; A fractional sample refinement operation is performed only when the integer offset is not located at the outermost side of the square distribution, and the fractional sample refinement operation comprises: Using the position with the minimum metric and the top, bottom, left, and right neighboring positions of the position with the minimum metric, a sub-pixel offset is derived based on a parametric error surface equation, The offset is generated by adding the sub-pixel offset to the integer offset, A refined motion vector is generated by applying the offset to the initial motion vector.

11. A method for providing a bitstream associated with video data, the method comprising: encoding video data into a bitstream; Sending a bitstream to a video decoding device, The encoding of the video data includes: encoding at least one first high-level syntax element in a sequence header, the first high-level syntax element indicating whether each of at least one coding tool is enabled, the at least one coding tool comprising motion vector refinement; encoding a second high-level syntax element at a picture level according to the at least one first high-level syntax element, the second high-level syntax element at the picture level being related to whether each of the at least one coding tool is enabled; When it is identified based on the setting at the picture level that motion vector refinement is allowed for the current picture including the target block, generating a refined motion vector of the target block by refining the initial motion vector obtained for the target block; Generate inter-frame prediction samples of target blocks using bidirectional reference images and refined motion vectors; generating a residual signal for a target block based on the inter-frame prediction samples, and encoding information of the residual signal, The generation of refined motion vectors includes: Integer sample search operation, which includes: calculating metrics for pairs of two candidate blocks corresponding to pairs of initial positions indicated by initial motion vectors and pairs of two neighboring positions shifted from the initial positions in integer sample units, wherein the neighboring positions in integer sample units are 24 sample positions having a square distribution around each initial position, searching among the metrics for an integer offset corresponding to the position having the smallest metric; A fractional sample refinement operation is performed only when the integer offset is not located at the outermost side of the square distribution, and the fractional sample refinement operation comprises: Using the position with the minimum metric and the top, bottom, left, and right neighboring positions of the position with the minimum metric, a sub-pixel offset is derived based on a parametric error surface equation, The offset is generated by adding the sub-pixel offset to the integer offset, A refined motion vector is generated by applying the offset to the initial motion vector.

Images