Method and apparatus for encoding and decoding video by means of inter prediction

Abstract

The present invention relates to a method and an apparatus for encoding and decoding a video by means of inter prediction. Provided is an image decoding device for predicting a target block in a current image to be decoded. The image decoding apparatus includes a prediction unit that determines first and second reference images and first and second motion vectors for bidirectional prediction by decoding a bitstream, generates a first reference block from the first reference image referenced by the first motion vector, and generates a second reference block from the second reference image referenced by the second motion vector. A second reference block is generated from a second reference image referenced by the second motion vector, and the target block is predicted by means of the first reference block and the second reference block. The prediction unit includes a first encoding tool for generating a prediction block of the target block by performing bidirectional optical flow processing by means of the first reference block and the second reference block.

Description

[0001] This application is a divisional application of the PCT patent application entered into China with Chinese patent application number 202080070159.4, invention name “Method and device for encoding and decoding video by means of inter-frame prediction”, and application date September 24, 2020.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to Korean Patent Application No. 10-2019-0123491 filed on October 6, 2019 and Korean Patent Application No. 10-2019-0158564 filed on December 2, 2019, which are hereby incorporated by reference in their entirety. Technical Field

[0004] The present invention relates to video encoding and decoding, and more particularly, to an encoding tool for improving compression performance of inter-frame prediction. Background Art

[0005] Since the amount of video data is larger than that of voice data or still image data, storing or transmitting the video data without compression processing requires a large amount of hardware resources (including memory).

[0006] Accordingly, when storing or transmitting video data, an encoder is usually used to compress the video data for storage or transmission. Then, a decoder receives the compressed video data, decompresses and reproduces the video data. Compression technologies used for such videos include H.264 / AVC and High Efficiency Video Coding (HEVC), which has an approximately 40% improvement in coding efficiency over H.264 / AVC.

[0007] However, the image size, resolution and frame rate are gradually increasing, and accordingly, the amount of data to be encoded is also increasing. Therefore, a new compression technology with better encoding efficiency and higher image quality than the existing compression technology is needed.

[0008] In video coding, predictive coding is mainly used to improve compression performance. There are intra-frame predictions for predicting a target block to be encoded based on pre-reconstructed samples in a current image and inter-frame predictions for predicting a current block using a pre-reconstructed reference image. In particular, inter-frame predictions are widely used in video coding because they exhibit better compression performance than intra-frame predictions.

[0009] The present invention proposes a coding tool for improving the compression performance of existing inter-frame prediction. Summary of the invention

[0010] The present invention provides a coding tool for improving compression performance of inter-frame prediction, and in one aspect relates to a coding tool capable of compensating for various motions of an object as well as translational motion in units of blocks.

[0011] According to one aspect of the present invention, a video decoding device is provided for predicting a target block in a current image to be decoded. The device includes a predictor configured to: determine a first reference image and a second reference image and a first motion vector and a second motion vector for bidirectional prediction by decoding a bitstream, generate a first reference block from the first reference image referenced by the first motion vector, and generate a second reference block from the second reference image referenced by the second motion vector, and generate a prediction block of the target block using the first reference block and the second reference block. The predictor includes a first encoding tool, which is configured to generate a prediction block of the target block by performing bidirectional optical flow processing using the first reference block and the second reference block. In this article, when the luminance weights of each of the first reference image and the second reference image for predicting the luminance component of the target block are different from each other, the first encoding tool is not executed. In addition, when the chrominance weights of each of the first reference image and the second reference image for predicting the chrominance component of the target block are different from each other, the first encoding tool is not executed.

[0012] According to another aspect of the present invention, a video encoding device is provided for performing inter-frame prediction on a target block in a current image to be encoded. The device includes a predictor, which is configured to: determine a first motion vector and a second motion vector for bidirectional use, generate a first reference block from a first reference image referenced by the first motion vector, and generate a second reference block from a second reference image referenced by the second motion vector, and generate a prediction block of the target block using the first reference block and the second reference block. The predictor includes a first encoding tool, which is configured to generate a prediction block of the target block by performing bidirectional optical flow using the first reference block and the second reference block. In this article, when the luminance weights of each of the first reference image and the second reference image for predicting the luminance component of the target block are different from each other, the first encoding tool is not executed. Moreover, when the chrominance weights of each of the first reference image and the second reference image for predicting the chrominance component of the target block are different from each other, the first encoding tool is also not executed.

[0013] According to another aspect of the present invention, a method for predicting a target block in a current image is provided. The method includes: determining a first motion vector and a second motion vector for bidirectional use; generating a first reference block from a first reference image referenced by the first motion vector, and generating a second reference block from a second reference image referenced by the second motion vector; and predicting the target block using the first reference block and the second reference block. Predicting the target block includes: executing a first encoding tool, the first encoding tool being configured to generate a prediction block of the target block by performing bidirectional optical flow processing using the first reference block and the second reference block. In this article, when the luminance weights of each of the first reference image and the second reference image for predicting the luminance component of the target block are different from each other, the first encoding tool is not executed. In addition, when the chrominance weights of each of the first reference image and the second reference image for predicting the chrominance component of the target block are different from each other, the first encoding tool is not executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is an exemplary block diagram of a video encoding device capable of implementing the technology of the present invention.

[0015] Figure 2 A block partition structure using a QTBTTT structure is exemplarily shown.

[0016] Figure 3 A plurality of intra prediction modes are exemplarily shown.

[0017] Figure 4 Neighboring blocks around the current block are exemplarily shown.

[0018] Figure 5 is an exemplary block diagram of a video decoding device capable of implementing the techniques of the present invention.

[0019] Figure 6 is an exemplary diagram illustrating the concept of bidirectional prediction optical flow provided by the present invention.

[0020] Figure 7 is an exemplary diagram illustrating a method of deriving gradients of block boundary samples in bidirectional optical flow.

[0021] Figure 8a , Figure 8b and Fig. 9 is an exemplary diagram illustrating affine motion prediction provided by the present invention.

[0022] Fig.10 is an exemplary diagram illustrating a method of deriving a merge candidate for affine motion prediction from translational motion vectors of neighboring blocks.

[0023] Fig.11a , Fig.11b and Fig.11cis an exemplary diagram illustrating a method of deriving illumination compensation parameters according to an embodiment of illumination compensation provided by the present invention. DETAILED DESCRIPTION

[0024] Hereinafter, some embodiments of the present invention will be described in detail with reference to the attached drawings. It should be noted that when reference numerals are added to the constituent elements in the various drawings, the same reference numerals also represent the same elements even though the elements are shown in different drawings. In addition, in the following description of the present invention, the detailed description of the known functions and configurations incorporated herein will be omitted to avoid blurring the subject matter of the present invention.

[0025] Figure 1 is an exemplary block diagram of a video encoding device capable of implementing the technology of the present invention. Figure 1 A video encoding apparatus and elements of the apparatus are described below.

[0026] The video encoding apparatus includes a block partitioner 110 , a predictor 120 , a subtractor 130 , a transformer 140 , a quantizer 145 , a rearrangement unit 150 , an entropy encoder 155 , an inverse quantizer 160 , an inverse transformer 165 , an adder 170 , a loop filter unit 180 , and a memory 190 .

[0027] Each element of the video encoding device may be implemented in hardware or software, or a combination of hardware and software. The functions of each element may be implemented as software, and a microprocessor may be implemented to execute the software functions corresponding to each element.

[0028] A video consists of one or more sequences including multiple images. Each image is divided into multiple regions, and encoding is performed on each region. For example, an image is divided into one or more tiles or / and slices. Here, one or more tiles can be defined as tile groups. Each tile or slice is divided into one or more coding tree units (coding tree unit, CTU). Each CTU is divided into one or more coding units (coding unit, CU) through a tree structure. The information applied to each CU is encoded as the syntax of the CU, and the information commonly applied to the CU included in a CTU is encoded as the syntax of the CTU. In addition, the information commonly applied to all blocks in a slice is encoded as the syntax of the slice header, and the information applied to all blocks constituting one or more images is encoded in the picture parameter set (Picture Parameter Set, PPS) or the picture header. In addition, the information commonly referenced by a sequence consisting of multiple images is encoded in the sequence parameter set (Sequence Parameter Set, SPS). In addition, the information commonly applied to a tile or tile group can be encoded as the syntax of the tile header or tile group header. The syntax included in the SPS, PPS, slice header, and tile header or tile group header may be referred to as a high-level syntax.

[0029] Each of the plurality of images may be segmented into a plurality of sub-images that can be independently encoded / decoded and / or independently displayed. When sub-image segmentation is applied, information on the layout of sub-images in the image is signaled.

[0030] The block partitioner 110 determines the size of a coding tree unit (CTU). Information on the size of the CTU (CTU size) is encoded as a syntax of an SPS or a PPS and transmitted to a video decoding apparatus.

[0031] The block partitioner 110 partitions each image constituting a video into a plurality of CTUs having a predetermined size, and then recursively partitions the CTUs using a tree structure. In the tree structure, a leaf node serves as a coding unit (CU), which is a basic unit of coding.

[0032] The tree structure can be a quadtree (QuadTree, QT), a binary tree (BinaryTree, BT), a ternary tree (TernaryTree, TT), or a structure formed by a combination of two or more QT structures, BT structures and TT structures, wherein the quadtree (QT) is a node (or parent node) that is divided into four slave nodes (or child nodes) of the same size, the binary tree (BT) is a node that is divided into two slave nodes, and the ternary tree (TT) is a node that is divided into three slave nodes at a ratio of 1:2:1. For example, a quadtree plus binary tree (QuadTree plus BinaryTree, QTBT) structure can be used, or a quadtree plus binary tree ternary tree (QuadTree plus BinaryTree TernaryTree, QTBTTT) structure can be used. Here, BTTT can be collectively referred to as a multiple-type tree (MTT).

[0033] Figure 2 The QTBTTT segmentation tree structure is shown as an example. Figure 2 As shown, the CTU can be first split into a QT structure. The QT splitting can be repeated until the size of the split block reaches the minimum block size MinQTSize of the leaf node allowed in the QT. The first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoder 155 and notified to the video decoding device by a signal. When the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it can be further split into one or more BT structures or TT structures. The BT structure and / or the TT structure can have multiple splitting directions. For example, there can be two directions, namely, a direction of splitting the blocks of nodes horizontally and a direction of splitting the blocks vertically. As Figure 2As shown, when the MTT splitting starts, the second flag (mtt_split_flag) indicating whether the node is split, the flag indicating the splitting direction (vertical or horizontal) in the case of splitting, and / or the flag indicating the splitting type (binary or trifurcated) are encoded by the entropy encoder 155, and the signal is notified to the video decoding device. Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into 4 nodes of the lower layer, the CU split flag (split_cu_flag) indicating whether the node is split can be encoded. When the value of the CU split flag (split_cu_flag) indicates that the splitting is not performed, the block of the node becomes a leaf node in the split tree structure and is used as a coding unit (CU), which is a basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates that the splitting is performed, the video encoding device starts to encode the flags starting from the first flag in the above-mentioned manner.

[0034] When QTBT is used as another example of a tree structure, there may be two types of segmentation, namely, a type of horizontally segmenting a block into two blocks of the same size (i.e., symmetrical horizontal segmentation) and a type of vertically segmenting a block into two blocks of the same size (i.e., symmetrical vertical segmentation). A segmentation flag (split_flag) indicating whether each node of the BT structure is segmented into blocks of the lower layer and segmentation type information indicating the segmentation type are encoded by the entropy encoder 155 and transmitted to the video decoding device. There may be an additional type of segmenting a block of a node into two asymmetric blocks. An asymmetric segmentation type may include a type of segmenting a block into two rectangular blocks at a size ratio of 1:3, or a type of segmenting a block of a node diagonally.

[0035] A CU may have various sizes according to the QTBT or QTBTTT partitioning of a CTU. Hereinafter, a block corresponding to a CU to be encoded or decoded (i.e., a leaf node of a QTBTTT) is referred to as a "current block". When QTBTTT partitioning is adopted, the shape of the current block may be a square or a rectangle.

[0036] The predictor 120 predicts the current block to generate a predicted block. The predictor 120 includes an intra predictor 122 and an inter predictor 124.

[0037] The intra-frame predictor 122 predicts samples in the current block using samples (reference samples) located around the current block in the current image including the current block. There are multiple intra-frame prediction modes according to the prediction direction. For example, Figure 3As shown, the plurality of intra prediction modes may include 2 non-directional modes and 65 directional modes, the 2 non-directional modes including a planar mode and a DC mode. Neighboring samples and equations to be used are defined differently for each prediction mode.

[0038] The intra-frame predictor 122 may determine an intra-frame prediction mode to be used when encoding the current block. In some examples, the intra-frame predictor 122 may encode the current block using several intra-frame prediction modes and select an appropriate intra-frame prediction mode to be used from the tested modes. For example, the intra-frame predictor 122 may calculate a rate-distortion value using a rate-distortion analysis of several tested intra-frame prediction modes and may select an intra-frame prediction mode having the best rate-distortion characteristic from the tested modes.

[0039] The intra predictor 122 selects an intra prediction mode from a plurality of intra prediction modes and predicts the current block using adjacent pixels (reference pixels) and equations determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0040] The inter-frame predictor 124 generates a prediction block for the current block by motion compensation. The inter-frame predictor 124 searches for a block that is most similar to the current block in a reference image that has been encoded and decoded earlier than the current image, and uses the searched block to generate a prediction block for the current block. Then, the inter-frame predictor generates a motion vector corresponding to the displacement between the current block in the current image and the prediction block in the reference image. Typically, motion estimation is performed on the luminance (luma) component, and the motion vector calculated based on the luminance component is used for both the luminance component and the chrominance component. Motion information including information about the reference image and information about the motion vector used to predict the current block is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0041] The inter-frame predictor 124 may perform interpolation on a reference image or a reference block to increase prediction accuracy. In other words, a subsample is interpolated between two consecutive integer samples by applying a filter coefficient to a plurality of consecutive integer samples including two integer samples. When the operation of searching for the block most similar to the current block is performed on the interpolated reference image, the motion vector may be expressed at an accuracy level of fractional sample units, rather than at an accuracy level of integer sample units. For each target area to be encoded, for example, each unit such as a slice, tile, CTU or CU, the accuracy or resolution of the motion vector may be set differently. When such an adaptive motion vector resolution is applied, information about the motion vector resolution to be applied to each target area should be signaled for each target area. For example, when the target area is a CU, information about the motion vector resolution applied to each CU is signaled.

[0042] The inter-frame predictor 124 may perform inter-frame prediction using bidirectional prediction. In bidirectional prediction, the inter-frame predictor 124 uses two reference images and two motion vectors representing the block positions most similar to the current block in each reference image. The inter-frame predictor 124 selects a first reference image and a second reference image from a reference image list 0 (RefPicList0) and a reference image list 1 (RefPicList1), respectively, searches for blocks similar to the current block in each reference image, and generates a first reference block and a second reference block. Then, the inter-frame predictor 124 generates a prediction block for the current block by averaging the first reference block and the second reference block. Then, the inter-frame predictor 124 transmits motion information including information about two reference images and two motion vectors for predicting the current block to the entropy encoder 155. Here, RefPicList0 may consist of images that are before the current image in the display order in the reconstructed image, and RefPicList1 may consist of images that are after the current image in the display order in the reconstructed image. However, the embodiment is not limited thereto. A pre-reconstructed image subsequent to the current image in the display order may be further included in RefPicList0, and conversely, a pre-reconstructed image prior to the current image may be further included in RefPicList1.

[0043] The inter-frame predictor 124 can perform bidirectional prediction using weighted averaging, which is so-called weighted bidirectional prediction. The inter-frame predictor 124 determines weights to be applied to the first reference image and the second reference image, respectively. The weight assigned to the first reference image is applied to the block in the first reference image, and the weight assigned to the second reference image is applied to the block in the second reference image. The inter-frame predictor 124 applies the weight assigned to the first reference image to the first reference block and the weight assigned to the second reference image to the second reference block, thereby generating a final prediction block of the target block by weighted summation or weighted averaging of the first reference block and the second reference block. The weight information of the reference images used in the inter-frame prediction is notified to the video decoding device by a signal.

[0044] On the other hand, the weight for predicting the luma component and the weight for predicting the chroma component may be determined independently. In this case, information about the luma weight to be applied to the luma component and information about the chroma weight to be applied to the chroma component are signaled separately.

[0045] The motion information (motion vectors, reference pictures) used for inter prediction should be signaled to the video decoding device. Various methods can be used to minimize the number of bits required to encode the motion information.

[0046] For example, when the reference image and motion vector of the current block are the same as those of the adjacent block, the motion information about the current block can be transmitted to the decoding device through the encoding information for identifying the adjacent block. This method is called "merge mode".

[0047] In the merge mode, the inter predictor 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as “merge candidates”) from neighboring blocks of the current block.

[0048] like Figure 4 As shown, all or part of the left block L, the upper block A, the upper right block AR, the lower left block BL, and the upper left block AL adjacent to the current block in the current image can be used as adjacent blocks for deriving merge candidates. In addition, in addition to the current image where the current block is located, blocks located in a reference image (which may be the same as or different from the reference image used to predict the current block) can be used as merge candidates. For example, a co-located block at the same position as the current block in the reference image or a block adjacent to the co-located block can be additionally used as a merge candidate.

[0049] The inter-frame predictor 124 configures a merge list including a predetermined number of merge candidates using such adjacent blocks. The inter-frame predictor 124 selects a merge candidate to be used as motion information about the current block from among the merge candidates included in the merge list, and generates merge index information for identifying the selected candidate. The generated merge index information is encoded by the entropy encoder 155 and transmitted to the decoding device.

[0050] Another method of encoding motion information is the AMVP mode.

[0051] In the AMVP mode, the inter-frame predictor 124 derives a motion vector candidate for prediction of a motion vector of a current block by using neighboring blocks of the current block. Figure 2 In the current image in the reference image, all or part of the left block L, the upper block A, the upper right block AR, the lower left block BL, and the upper left block AL adjacent to the current block can be used as the adjacent blocks for deriving the predicted motion vector candidate. In addition, in addition to the current image including the current block, blocks located in a reference image (which may be the same as or different from the reference image used to predict the current block) can be used as the adjacent blocks for deriving the predicted motion vector candidate. For example, a co-located block at the same position as the current block in the reference image or a block adjacent to the co-located block can be used.

[0052] The inter-frame predictor 124 derives predicted motion vector candidates using motion vectors of neighboring blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, a motion vector difference is calculated by subtracting the predicted motion vector from the motion vector of the current block.

[0053] The predicted motion vector can be obtained by applying a predefined function (e.g., a function for calculating a median, an average, etc.) to the predicted motion vector candidate. In this case, the video decoding device also knows the predefined function. Since the neighboring blocks used to derive the predicted motion vector candidate have been encoded and decoded, the video decoding device also already knows the motion vectors of the neighboring blocks. Accordingly, the video encoding device does not need to encode information for identifying the predicted motion vector candidate. Therefore, in this case, information about the motion vector difference and information about the reference image used to predict the current block are encoded.

[0054] The predicted motion vector may be determined by selecting any one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is further encoded together with information about a motion vector difference to be used for predicting the current block and information about a reference image.

[0055] The subtractor 130 subtracts the prediction block generated by the intra predictor 122 or the inter predictor 124 from the current block to generate a residual block.

[0056] The transformer 140 may transform the residual signal in the residual block. The two-dimensional (2D) size of the residual block may be used as a transform unit (hereinafter "TU") of a block size for performing the transform. Alternatively, the residual block may be divided into a plurality of sub-blocks, and the residual signal in the corresponding sub-block may be transformed by using each sub-block as a TU.

[0057] The transformer 140 may divide the residual block into one or more sub-blocks and apply a transform to the one or more sub-blocks, thereby transforming the residual value of the transform block from the pixel domain to the frequency domain. In the frequency domain, the transform block is referred to as a coefficient block or a transform block containing one or more transform coefficient values. A two-dimensional transform kernel may be used for the transform, and a one-dimensional transform kernel may be used for horizontal transform and vertical transform, respectively. The transform kernel may be based on a discrete cosine transform (DCT), a discrete sine transform (DST), etc.

[0058] The transformer 140 may transform the residual block separately in the horizontal direction and the vertical direction. For the transformation, various types of transform kernels or transform matrices may be used. For example, the paired transform kernels for horizontal transformation and vertical transformation may be defined as a multiple transform set (MTS). The transformer 140 may select a pair of transform kernels with the best transformation efficiency in the MTS, and transform the residual block in the horizontal direction and the vertical direction, respectively. The information (mts_idx) about the transform kernel pair selected in the MTS is encoded by the entropy encoder 155 and notified to the video decoding device with a signal.

[0059] The quantizer 145 quantizes the transform coefficients output from the transformer 140 using the quantization parameters, and outputs the quantized transform coefficients to the entropy encoder 155. For some blocks or frames, the quantizer 145 may directly quantize the relevant residual block without transformation. The quantizer 145 may apply different quantization coefficients (scaling values) according to the position of the transform coefficients in the transform block. The quantization coefficient matrix (which is applied to the quantized transform coefficients arranged in two dimensions) may be encoded and signaled to the video decoding device.

[0060] The rearrangement unit 150 can reclassify the coefficient values ​​of the quantized residual value. The rearrangement unit 150 can change the 2D coefficient array into a 1D coefficient sequence by coefficient scanning. For example, the rearrangement unit 150 can scan the coefficients from the DC coefficient to the coefficients in the high frequency region using a zig-zag scan or a diagonal scan to output a 1D coefficient sequence. According to the size of the transform unit and the intra-frame prediction mode, the zig-zag scan can be replaced by a vertical scan or a horizontal scan, wherein the vertical scan scans the two-dimensional coefficient array in the column direction, and the horizontal scan scans the coefficients in the two-dimensional block shape in the row direction. That is, the scanning mode to be used can be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan according to the size of the transform unit and the intra-frame prediction mode.

[0061] The entropy encoder 155 encodes the one-dimensional quantized transform coefficient output from the rearrangement unit 150 using various encoding techniques such as Context-based Adaptive Binary Arithmetic Code (CABAC) and exponential Golomb to generate a bitstream.

[0062] The entropy encoder 155 encodes information related to block segmentation (e.g., CTU size, CU segmentation flag, QT segmentation flag, MTT segmentation type, and MTT segmentation direction) so that the video decoding device can segment the block in the same manner as the video encoding device. In addition, the entropy encoder 155 encodes information about the prediction type indicating whether the current block is encoded by intra-frame prediction or inter-frame prediction, and encodes intra-frame prediction information (i.e., information about the intra-frame prediction mode) or inter-frame prediction information (merge index for merge mode, information about reference image index and motion vector difference for AMVP mode) according to the prediction type. In addition, the entropy encoder 155 also encodes information related to quantization (i.e., information about quantization parameters and information about quantization matrices).

[0063] The inverse quantizer 160 inversely quantizes the quantized transform coefficient output from the quantizer 145 to generate a transform coefficient. The inverse transformer 165 transforms the transform coefficient output from the inverse quantizer 160 from the frequency domain to the spatial domain, and reconstructs a residual block.

[0064] The adder 170 adds the reconstructed residual block and the prediction block generated by the predictor 120 to reconstruct the current block. Samples in the reconstructed current block are used as reference samples when performing intra prediction of a subsequent block.

[0065] The loop filter unit 180 filters the reconstructed samples to reduce blocking artifacts, ringing artifacts, and blurring artifacts generated due to block-based prediction and transform / quantization. The loop filter unit 180 may include at least one of a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.

[0066] The deblocking filter 182 filters the boundaries between the reconstructed blocks to remove block artifacts caused by block-by-block encoding / decoding, and the SAO filter 184 performs additional filtering on the deblocked filtered video. The SAO filter 184 is a filter for compensating for the difference between the reconstructed sample and the original sample caused by lossy coding, and performs filtering in a manner of adding a corresponding offset to each reconstructed sample. The ALF 186 performs filtering on the target sample to be filtered by applying a filter coefficient to the target sample and the adjacent samples of the target sample. The ALF 186 can divide the samples included in the image into predetermined groups, and then determine a filter to be applied to the corresponding group to perform filtering differentially on each group. Information about the filter coefficient to be used for the ALF can be encoded and notified to the video decoding device with a signal.

[0067] The reconstructed blocks filtered by the loop filtering unit 180 are stored in the memory 190. Once all blocks in one image are reconstructed, the reconstructed image may be used as a reference image for inter-frame prediction of blocks in a subsequent image to be encoded.

[0068] Figure 5 is an exemplary functional block diagram of a video decoding device capable of implementing the technology of the present invention. Figure 5 A video decoding apparatus and elements of the apparatus are described.

[0069] The video decoding apparatus may include an entropy decoder 510 , a rearrangement unit 515 , an inverse quantizer 520 , an inverse transformer 530 , a predictor 540 , an adder 550 , a loop filtering unit 560 , and a memory 570 .

[0070] Similar to Figure 1 In the video encoding device of the present invention, each element of the video decoding device can be implemented by hardware, software, or a combination of hardware and software. In addition, the function of each element can be implemented by software, and the microprocessor can be implemented to execute the software function corresponding to each element.

[0071] The entropy decoder 510 determines a current block to be decoded by decoding a bit stream generated by a video encoding device and extracting information related to block partitioning, and extracts prediction information required to reconstruct the current block, information about a residual signal, and the like.

[0072] The entropy decoder 510 extracts information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), determines the size of the CTU, and divides the image into CTUs of the determined size. Then, the decoder determines the CTU as the highest level (i.e., the root node) of the tree structure, and extracts partition information about the CTU to partition the CTU using the tree structure.

[0073] For example, when the CTU is segmented using the QTBTTT structure, the first flag (QT_split_flag) related to the segmentation of the QT is extracted to segment each node into four nodes of the sublayer. For the node corresponding to the leaf node of the QT, the second flag (MTT_split_flag) related to the segmentation of the MTT and the information about the segmentation direction (vertical / horizontal) and / or the segmentation type (binary / tripartite) are extracted to segment the corresponding leaf node with the MTT structure. Thus, each node below the leaf node of the QT is recursively segmented with the BT or TT structure.

[0074] As another example, when a CTU is split using a QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU is split may be extracted. When the corresponding block is split, a first flag (QT_split_flag) may be extracted. In a split operation, after zero or more recursive QT splits, zero or more recursive MTT splits may occur for each node. For example, a CTU may directly undergo MTT splitting without undergoing QT splitting, or may only undergo QT splitting multiple times.

[0075] As another example, when the CTU is segmented using the QTBT structure, the first flag (QT_split_flag) related to the QT segmentation is extracted, and each node is segmented into four nodes of the lower layer. Then, the segmentation flag (split_flag) indicating whether the node corresponding to the leaf node of the QT is further segmented with the BT and the segmentation direction information are extracted.

[0076] Once the current block to be decoded is determined by tree structure partitioning, the entropy decoder 510 extracts information about the prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 510 extracts syntax elements of intra-prediction information (intra-prediction mode) of the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 extracts syntax elements for inter-prediction information, that is, information indicating a motion vector and a reference image referenced by the motion vector.

[0077] The entropy decoder 510 also extracts information about the quantized transform coefficient of the current block as information related to quantization and information about the residual signal.

[0078] The rearrangement unit 515 may change a sequence of one-dimensional quantized transform coefficients entropy-decoded by the entropy decoder 510 into a 2-dimensional coefficient array (ie, block) in the reverse order of coefficient scanning performed by the video encoding apparatus.

[0079] The inverse quantizer 520 inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantizer 520 may apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantizer 520 may perform inverse quantization by applying a quantization coefficient (scaling value) matrix from a video encoding device to a 2-dimensional array of quantized transform coefficients.

[0080] The inverse transformer 530 inversely transforms the inverse quantized transform coefficient from the frequency domain to the spatial domain to reconstruct the residual signal, thereby generating a reconstructed residual block of the current block. In addition, when MTS is applied, the inverse transformer 530 determines the transform function or transform matrix to be applied in the horizontal direction and the vertical direction, respectively, using the MTS information (mts_idx) signaled from the video encoding device, and inversely transforms the transform coefficient in the transform block in the horizontal direction and the vertical direction using the determined transform function.

[0081] The predictor 540 may include an intra predictor 542 and an inter predictor 544. The intra predictor 542 is activated when the prediction type of the current block is intra prediction, and the inter predictor 544 is activated when the prediction type of the current block is inter prediction.

[0082] The intra predictor 542 determines an intra prediction mode of a current block among a plurality of intra prediction modes based on a syntax element of the intra prediction mode extracted from the entropy decoder 510 , and predicts the current block using reference samples around the current block according to the intra prediction mode.

[0083] The inter predictor 544 determines a motion vector of a current block and a reference image referred to by the motion vector using a syntax element of the inter prediction extracted from the entropy decoder 510 , and predicts the current block based on the motion vector and the reference image.

[0084] Similar to the inter-frame predictor 124 of the video encoding device, the inter-frame predictor 544 can generate a prediction block of the current block using bidirectional prediction. When weighted bidirectional prediction is applied, the entropy decoder 510 extracts weight information applied to two reference images to be used for bidirectional prediction of the current block from the bitstream. The weight information may include weight information to be applied to the luminance component and weight information to be applied to the chrominance component. The inter-frame predictor 544 uses the weight information to generate a prediction block for the luminance component of the current block and a prediction block for the chrominance component.

[0085] The adder 550 reconstructs the current block by adding the residual block output from the inverse transformer to the prediction block output from the inter-frame predictor or the intra-frame predictor. When intra-predicting a subsequent block to be decoded, samples in the reconstructed current block are used as reference samples.

[0086] The loop filtering unit 560 may include at least one of a deblocking filter 562, an SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundaries between the reconstructed blocks to remove block artifacts caused by block-by-block decoding. The SAO filter 564 performs filtering in a manner of adding the reconstructed blocks after deblocking filtering the corresponding offsets so as to compensate for the difference between the reconstructed samples and the original samples caused by lossy coding. The ALF 566 performs filtering on the target sample to be filtered by applying the filter coefficients to the target sample and the adjacent samples of the target sample. The ALF 566 may divide the samples in the image into predetermined groups, and then determine a filter to be applied to the corresponding group to perform filtering differentially on each group. The filter coefficients of the ALF are determined based on information about the filter coefficients decoded from the bitstream.

[0087] The reconstructed blocks filtered by the loop filtering unit 560 are stored in the memory 570. When all blocks in one image are reconstructed, the reconstructed image is used as a reference image for inter-frame prediction of blocks in a subsequent image to be encoded.

[0088] The following disclosure relates to a coding tool for improving compression performance of inter-frame prediction, which can be operated by the inter-frame predictor 124 of a video encoding device and the inter-frame predictor 544 of a video decoding device. As used herein, the term "target block" may have the same meaning as the term "current block" or "coding unit (CU)" used above, or may represent a local area of ​​a CU.

[0089] I. Combined Inter-Intra Prediction

[0090] As described above, the target block is predicted by one of the inter prediction and the intra prediction. The combined inter-intra prediction described in the present invention is a technique for supplementing the inter prediction signal with the intra prediction signal. When the combined inter-intra prediction is applied, the inter predictor 124 of the video encoding device determines the motion vector of the target block and predicts the target block with the determined motion vector to generate the inter prediction block. The intra predictor 124 of the video encoding device predicts the target block and generates the intra prediction block using the reference samples around the target block. As the intra prediction mode for generating the intra prediction block, any one of the above-mentioned multiple intra prediction modes can be used fixedly. For example, the plane mode or the DC mode can be used as the prediction mode for generating the intra prediction block. The final prediction block is generated by the average or weighted average of the inter prediction block and the intra prediction block. The equation for calculating the final prediction block in the combined inter-intra prediction is given as follows.

[0091] [Equation 1]

[0092] P final =((4-wt)*P inter +wt*P intra +2)>>2

[0093] Here, P inter represents the inter-frame prediction block, P intra and represents an intra-frame prediction block. wt represents the weight. +2 is an offset used for rounding operation.

[0094] The weight may be determined based on whether inter-frame prediction or intra-frame prediction is used to predict the pre-encoded / decoded neighboring blocks adjacent to the target block. For example, when intra-frame prediction is performed on both the left block and the upper block of the target block, the intra-frame prediction block (P intra ) is given a larger weight. For example, wt is set to 3. When intra-frame prediction is performed on only one of the left block and the upper block, the inter-frame prediction block (P inter ) and intra prediction block (P intra ) are given the same weight. For example, wt is set to 2. When neither the left block nor the upper block is intra-predicted, the inter-prediction block (P inter ) to give a larger weight. For example, set wt to 1.

[0095] When the target block is predicted by combined inter-intra prediction, the inter predictor 544 of the video decoding apparatus extracts information about the motion vector of the target block from the bitstream to determine the motion vector of the target block. Then, the target block is predicted in the same manner as the video encoding apparatus.

[0096] Combined inter-intra prediction is a technique for supplementing an inter prediction signal with an intra prediction signal, and thus may be effective when inter prediction is more or less incorrect (e.g., when the motion vector of the target block is determined by a merge mode). Thus, combined inter-intra prediction may be applied only when the motion vector of the target block is determined by a merge mode.

[0097] II. Bidirectional Optical Flow

[0098] Bidirectional optical flow is a technique for additionally compensating for the motion of samples predicted using bidirectional motion prediction, assuming that samples or objects constituting a video move at a constant speed and that sample values ​​hardly change.

[0099] Figure 6 is an exemplary diagram showing the basic concept of BIO.

[0100] Assume that by (normal) bidirectional motion prediction for the target block, in the reference image Ref 0 and Ref 1 The bidirectional motion vector MV pointing to the corresponding area (ie, the reference block) that is most similar to the target block to be encoded in the current image has been determined. 0 and MV 1 The two motion vectors have values ​​that represent the motion of the entire target block. Figure 6 In the example, P 0 The motion vector MV 0 The reference image Ref corresponding to the sample P in the target block is indicated 0 The samples in P 1 The motion vector MV 1 The reference image Ref corresponding to the sample P in the target block is indicated 1 In addition, assuming Figure 6 The motion of the sample P in is slightly different from the overall motion of the target block. Figure 6 Ref 0 The object at sample A in the target block of the current image moves to Ref via sample P in the target block of the current image. 1 When sample B in , sample A and sample B can have values ​​very similar to each other. Moreover, in this case, Ref 0 The point in the target block that is most similar to the sample P is not represented by the bidirectional motion vector MV 0 Indicated P 0 , but from P 0 Shifted by a predetermined displacement vector (v x τ 0 , v y τ 1 ) of sample A. Ref 1The point in the target block that is most similar to the sample P is not represented by the bidirectional motion vector MV 1 Indicated P 1 , but from P 1 Shifted by a predetermined displacement vector (-v x τ 0 , -v y τ 1 ) of sample B. Here, τ 0 and τ 1 Respectively represent Ref 0 and Ref 1 The time axis distance relative to the current image and is calculated based on the picture order count (POC). In the following, (v x , v y ) is called "optical flow" or "motion offset".

[0101] When predicting the value of sample P of the current block in the current image, two reference samples A and B can achieve a better prediction than the value of the sample P of the current block in the current image. 0 and MV 1 Indicated reference sample P 0 and P 1 More accurate predictions.

[0102] I (0) (i, j) represents the motion vector MV 0 The reference image Ref corresponding to the sample (i, j) in the target block is indicated 0 The value of the sample in, and I (1) (i, j) represents the motion vector MV 1 The reference image Ref corresponding to the sample (i, j) in the target block is indicated 1 The value of the sample in .

[0103] By BIO motion vector (v x , v y ) indicates a reference image Ref corresponding to the sample in the target block 0 The value of sample A in can be defined as I (0) (i+v x τ 0 , j+v y τ 0 ), and the reference image Ref 1 The value of sample B in can be defined as I (1) (iv x τ 1 , jv y τ 1). Here, when linear approximation is performed using only the first-order term of the Taylor series, A and B can be expressed as Equation 2.

[0104] [Equation 2]

[0105]

[0106] Here, I x (k) and I y (k) (k=0, 1) is in Ref 0 and Ref 1 The gradient value in the horizontal and vertical directions at the position (i, j). In addition, τ 0 and τ 1 Respectively represent Ref 0 and Ref 1 The temporal distance relative to the current image and based on POC: τ 0 =POC(Current)-POC(Ref 0 ), and τ 1 =POC(Ref 1 )-POC(current) is used to calculate.

[0107] The bidirectional optical flow of each sample in the block (v x , v y ) is determined as the solution that minimizes Δ, which is defined as the difference between sample A and sample B. Δ can be defined by Equation 3 using the linear approximation of A and B derived from Equation 2.

[0108] [Equation 3]

[0109]

[0110] For simplicity, the sample position (i, j) is omitted from each term in Equation 3.

[0111] In order to achieve a more robust optical flow estimation, the motion is assumed to be locally consistent with neighboring samples. For the BIO motion vector of the current sample (i, j) to be predicted, the difference Δ in equation 3 of all samples (i', j') present in a mask Ω of a certain size centered on the sample (i, j) is considered. That is, the optical flow of the current sample (i, j) can be determined to make the objective function Φ(v x , v y ) minimized vector, the objective function Φ(v x , v y ) is the sum of the squares of the differences Δ[i′, j′] obtained for each sample in the mask Ω, as shown in Equation 4.

[0112] [Equation 4]

[0113]

[0114] The bidirectional optical flow of the present invention can be applied to a case where one of the two reference images used for bidirectional prediction is before the current image in display order, and the other is after the current image, and the distances between the two reference images and the current image are equal to each other, that is, the differences in the picture order count (POC) between each reference image and the current image are equal to each other. Therefore, τ 0 and τ 1 Can be ignored.

[0115] Furthermore, the bidirectional optical flow of the present invention may be applied only to the luminance component.

[0116] For a target block to which bidirectional prediction is applied, the bidirectional optical flow of the present invention is performed on a subblock basis rather than a pixel basis. In the following description, for simplicity, it is assumed that a subblock having various sizes of 2×2, 4×4, and 8×8 has a size of 4×4.

[0117] Before performing the optical flow, the inter-frame predictor 124 of the video encoding apparatus generates two reference blocks for the target block using the above-mentioned bidirectional prediction. The first reference block of the two reference blocks represents the first motion vector MV 0 According to the reference image Ref 0 The second reference block represents a block composed of samples of the prediction generated using the motion vector MV 1 According to the reference image Ref 1 The generated predictions consist of blocks of samples.

[0118] The inter-frame predictor 124 calculates a so-called optical flow (v) for each of the 4×4 sub-blocks constituting the target block using the horizontal gradient values ​​and the vertical gradient values ​​of the predicted samples in the first reference block and the second reference block. x , v y ). Optical flow (v x , v y ) is determined such that according to the reference image Ref 0 The predicted samples are compared with the reference image Ref 1 The inter-frame predictor 124 minimizes the difference between the predicted samples of x , v y ) and the gradient of the predicted samples in the 4×4 sub-block to derive the sample offsets used to modify the bidirectionally predicted samples of the 4×4 sub-block.

[0119] Specifically, the inter-frame predictor 124 calculates the horizontal gradient and the vertical gradient of the sample value at the position (i, j) using Equation 5.

[0120] [Equation 5]

[0121]

[0122] Here, k is 0 or 1, and I (0) (i, j) and I (1) (i, j) represent the sample values ​​at the position (i, j) in the first reference block and the second reference block, respectively. Shift1 is a value derived according to the bit depth of the luminance component, for example, Shift1 = max(6, bit depth - 6).

[0123] In order to derive the gradient of samples located at the boundary of each reference block, samples outside the boundary of the first reference block and the second reference block are needed. Figure 7 As shown, each reference block is extended by one column to the left and right, and by one row to the upper and lower sides. In order to reduce the amount of calculation, each sample in the extended portion can be filled with the sample at the most adjacent position in the reference block or an integer sample. Moreover, the gradient at the sample position outside the boundary of each reference block can be filled with the gradient corresponding to the sample at the most adjacent position.

[0124] The inter-frame predictor 124 uses the horizontal gradient and the vertical gradient in the 6×6 window covering the 4×4 sub-block to calculate S1, S2, S3, S5 and S6 corresponding to the autocorrelation and cross-correlation of the gradient, as shown in Figure 7 shown.

[0125] [Equation 6]

[0126]

[0127] S 5 =∑ (i，j)∈Ω ψ y (i,j) ψ (i,j) S 6 =∑ (i，j)∈Ω θ(i, j)·ψ y (i, j)

[0128] Here, Ω represents the window covering the sub-block. In addition, as shown in Equation 7 below, Ψ x (i, j) represents the sum of the horizontal gradient values ​​at the position (i, j) in the first reference block and the second reference block, Ψy(i, j) represents the sum of the vertical gradient values ​​at the position (i, j) in the first reference block and the second reference block, and θ(i, j) represents the difference between the sample value at the position (i, j) in the second reference block and the sample value at the position (i, j) in the first reference block.

[0129] [Equation 7]

[0130]

[0131]

[0132] θ(i, j)=(I (1) (i, j)>>n b )-(I (0) (i, j)>>n b )

[0133] Here, n a and n b It is a value derived from the bit depth and has values ​​of min(1, bit depth - 11) and min(4, bit depth - 8).

[0134] The inter-frame predictor 124 calculates the optical flow (v x , v y ).

[0135] [Equation 8]

[0136]

[0137] here, and th′ BIO =2 max(5，BD-7) . is a floor function, and

[0138] The sample offset of the sample used to modify the prediction at each sample position (x, y) in the 4×4 sub-block in the target block can be calculated using the optical flow calculated for the 4×4 sub-block and the gradient value at the sample position (x, y), as shown in Equation 9. In Equation 9, rnd() represents a rounding operation.

[0139] [Equation 9]

[0140]

[0141] The inter-frame predictor 124 uses the sample offset b(x, y) at the position (x, y) and the predicted sample I in the first reference block. (0) (x, y) and the predicted sample I in the second reference block (1) (x, y) to generate the final predicted sample pred(x, y), as shown in Equation 10.

[0142] [Equation 10]

[0143] pred(x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+o offset)>>shift

[0144] Here, shift is Max(3, 15-bit depth), and O offset is the value used for rounding operations and is half of shift.

[0145] As described above, the bidirectional optical flow technique uses the value of the sample predicted using the motion information (two motion vectors and two reference images) for bidirectional prediction. Therefore, the inter-frame predictor 544 of the video decoding device can also perform the bidirectional optical flow in the same manner as the video encoding device using the motion information (motion vector, reference image) for bidirectional prediction received from the video encoding device. There is no need for the video encoding device to signal the additional information for the bidirectional optical flow processing to the video decoding device.

[0146] The above-mentioned bidirectional optical flow technique can be applied to the chrominance component. In this case, in order to reduce the computational complexity, the optical flow calculated for the luminance component can be used as the optical flow for the chrominance component without recalculating the optical flow for the chrominance component (v x , v y ). Therefore, when applying bidirectional optical flow to the chrominance component, only the horizontal gradient and vertical gradient of the chrominance component of each sample need to be calculated.

[0147] The above-mentioned bidirectional optical flow itself requires a lot of calculations. In addition, when used with other coding tools, the bidirectional optical flow can further increase the computational complexity and delay the encoding / decoding process. In addition, the bidirectional optical flow combined with some coding tools may not contribute to the improvement of coding efficiency. Taking these points into account, the execution of the bidirectional optical flow may be restricted under certain conditions. The video encoding device and the video decoding device skip the bidirectional optical flow by checking whether the predefined conditions are met before executing the bidirectional optical flow. The conditions for constraining the execution of the bidirectional optical flow are described below. When one or more conditions are met, the bidirectional optical flow is not executed and skipped.

[0148] In some embodiments, bidirectional optical flow is not used with affine motion prediction, which will be described later. Since both bidirectional optical flow and affine motion prediction require a lot of calculations, the combination of the two encoding tools not only increases the complexity of the calculations, but also delays the encoding / decoding process. Therefore, when affine motion prediction is used for the target block, bidirectional optical flow for the target block is not performed.

[0149] In some other embodiments, bidirectional optical flow can be constrained to not be used with inter-frame-intra-frame merging prediction technology. When the inter-frame-intra-frame merging prediction technology is applied to the target block, the bidirectional optical flow to the target block is not applied. In intra-frame prediction, the pre-reconstructed samples around the target block are used, so the target block can be predicted after the decoding (sample reconstruction) of the adjacent blocks is completed. Therefore, when applying the combined inter-frame-intra-frame prediction and bidirectional optical flow, the bidirectional optical flow process should be stopped, and the intra-frame prediction of the target block can be performed after all adjacent blocks of the target block are decoded. This may cause a serious delay in the decoding process. Accordingly, the bidirectional optical flow may not be applied to the block to which the combined inter-frame-intra-frame prediction is applied.

[0150] In addition, the bidirectional optical flow can be restricted for use with local illumination compensation, which will be described later. For example, when local illumination compensation is applied, the bidirectional optical flow is not applied.

[0151] In some other embodiments, when the current image including the target block or the reference image referred to by the target block is an image segmented into sub-images, the bidirectional optical flow is not applied.

[0152] In some other embodiments, when weighted bidirectional prediction is performed on the target block and different weights are applied to two reference blocks (first reference block and second reference block), bidirectional optical flow is not performed. That is, when the weights to be applied to the two reference blocks (luminance components in the two reference blocks) to predict the luminance component are different from each other, the bidirectional optical flow of the target block is not performed. In addition, when the weights to be applied to the two reference blocks (chrominance components in the two reference blocks) to predict the chrominance component are different from each other, the bidirectional optical flow of the target block is not performed. As described above, the bidirectional optical flow is based on the assumption that the sample values ​​between images have almost no change. In contrast, weighted bidirectional prediction assumes that the sample values ​​between images change. Accordingly, when weighted bidirectional prediction applying different weights is performed, the bidirectional optical flow of the target block is skipped.

[0153] Alternatively, in the case of bidirectional prediction with different weights, bidirectional optical flow can be performed by applying different weights. (0) (x, y)) and the second reference block (I (1) The weights of (x, y)) are w 0 and w 1 When I in Equation 5 to Equation 10 is (0) (x, y) and I (1) (x, y) can be represented by w 0 I (0) (x, y) and w 1 I (0)That is, instead of the sample values ​​in the two reference blocks, the sample values ​​multiplied by the weights corresponding to the two reference blocks can be used to calculate the optical flow (v x , v y ), sample offset b(x, y) and the final predicted sample pred(x, y).

[0154] III. Affine Motion Prediction

[0155] The above-mentioned inter-frame prediction is a motion prediction that reflects a translational motion model. That is, it is a technique for predicting motion in the horizontal direction (x-axis direction) and the vertical direction (y-axis direction). However, in reality, there may be various types of motion, for example, rotation, enlargement or reduction in addition to translational motion. One aspect of the present invention provides affine motion prediction that can cover such various types of motion.

[0156] Figure 8a and Figure 8b is an exemplary diagram illustrating affine motion prediction.

[0157] There may be two types of models for affine motion prediction. One model is a model using motion vectors (that is, four parameters) of two control points at the upper left and upper right corners of the target block currently to be encoded, such as Figure 8a Another model is a model that uses the motion vectors (that is, six parameters) of three control points at the upper left corner, upper right corner, and lower left corner of the target block, as shown in Figure 8b shown.

[0158] The four-parameter affine model is expressed by Equation 11. The motion at the sample position (x, y) in the target block can be calculated by Equation 11. Here, it is assumed that the position of the upper left sample of the target block is (0, 0).

[0159] [Equation 11]

[0160]

[0161] The six-parameter affine model is expressed by Equation 12. The motion at the sample position (x, y) in the target block can be calculated by Equation 12.

[0162] [Equation 12]

[0163]

[0164] Here, (mv 0x , mv 0y ) is the motion vector of the upper left control point, (mv 1x , mv 1y ) is the motion vector of the upper right control point, and (mv2x , mv 2y ) is the motion vector of the lower left control point. W is a constant determined according to the horizontal length of the target block, and H is a constant determined according to the vertical length of the target block.

[0165] Affine motion prediction may be performed on each sample in the target block using the motion vector calculated by Equation 11 or Equation 12.

[0166] Alternatively, to reduce computational complexity, prediction can be performed on each sub-block segmented from the target block, such as Fig. 9 As shown. For example, the size of the sub-block may be 4×4, 2×2, or 8×8. In the following exemplary embodiment, affine motion prediction for the target block is performed based on a 4×4 sub-block. This example is only for ease of explanation, and the present invention is not limited thereto.

[0167] In the sub-block based affine motion prediction, the motion vector (affine motion vector) of each sub-block is calculated by substituting the center position of each sub-block into (x, y) of Equation 11 or Equation 12. Here, the center position may be the actual center point of the sub-block or the lower right sample position of the center point. For example, in the case where the coordinates of the lower left sample in the 4×4 sub-block are (0, 0), the center position of the sub-block may be (1.5, 1.5) or (2, 2). Using the affine motion vector (mv) of the sub-block x , mv y ) to generate a predicted block for each sub-block.

[0168] Motion vector (mv x , mv y ) can be set to have 1 / 16 sample accuracy. In this case, the motion vector (mv) calculated by equation 11 or equation 12 is x , mv y ) can be rounded in 1 / 16 sample units. Adaptive motion vector resolution can be applied to affine motion prediction, just as in normal inter prediction. In this case, information about the motion vector resolution of the target block (that is, the accuracy of the motion vector) is signaled for each target block.

[0169] Affine motion prediction can be performed not only on the luma component but also on the chroma component. In the case of a 4:2:0 video format, when affine motion prediction for the luma component is performed based on a 4×4 sub-block, affine motion prediction for the chroma component can be performed based on a 2×2 sub-block. The motion vector (mv) of each sub-block of the chroma component can be derived from the motion vector of the corresponding luma component. x , mv y). Alternatively, the size of the subblock for affine motion prediction of the chrominance component may be the same as the size of the subblock for the luma component. When affine motion prediction for the luma component is performed based on the 4×4 subblock, affine motion prediction for the chrominance component may also be performed based on the 4×4 subblock. In this case, since the 4×4 subblock for the chrominance component corresponds to the four 4×4 subblocks for the luma component, the motion vector (mv) of the subblock for the chrominance component may be calculated by calculating the average of the motion vectors of the four corresponding subblocks of the luma component. x , mv y ).

[0170] The video encoding device performs intra prediction, inter prediction (translational motion prediction), affine motion prediction, etc., and calculates the rate-distortion (RD) cost to select the best prediction method. In order to perform affine motion prediction, the inter predictor 124 of the video encoding device determines which type of model of the two types of models to use, and determines two or three control points according to the determined type. The inter predictor 124 calculates the motion vector (mv) of each sub-block of the 4×4 sub-block in the target block using the motion vector of the control point. x , mv y Then, the inter-frame predictor 124 uses the motion vector (mv x , mv y ) performs motion compensation in a reference image on a sub-block by sub-block basis to generate a predicted block for each sub-block in a target block.

[0171] The entropy encoder 155 of the video encoding device encodes an affine-related syntax element including a flag indicating whether affine motion prediction is applied to the target block, type information indicating the type of the affine model, and motion information indicating the motion vector of each control point, and transmits it to the video decoding device. When affine motion prediction is performed, type information and motion information about the control point can be signaled, and motion vectors of as many control points as the number determined according to the type information can be signaled. In addition, when adaptive motion vector resolution is applied, motion vector resolution information about the affine motion vector of the target block is signaled.

[0172] The video decoding apparatus determines the type of the affine model and the control point motion vector using the signaled syntax elements, and calculates the motion vector (mv) of each 4×4 sub-block in the target block using Equation 11 or Equation 12. x , mv y ). In the case where the motion vector resolution information about the affine motion vector of the target block is signaled, the motion vector (mv x , mv y) is corrected to the accuracy identified by the motion vector resolution information.

[0173] The video decoding apparatus uses the motion vector (mv) for each sub-block. x , mv y ) performs motion compensation within a reference picture to generate a predicted block for each sub-block.

[0174] In order to reduce the amount of bits required to encode the motion vector of the control point, a method as used in the above-mentioned normal inter-frame prediction (translational motion prediction) may be applied.

[0175] As an example, in the merge mode, the inter-frame predictor 124 of the video encoding device derives the motion vector of each control point according to the neighboring blocks of the target block. Figure 4 The neighboring samples L, BL, A, AR and AL of the target block shown in the figure derive a predefined number of merge candidates to generate a merge candidate list. Each merge candidate included in the list corresponds to a pair of motion vectors of two or three control points.

[0176] First, the inter-frame predictor 124 derives a merge candidate based on the control point motion vector of the neighboring block predicted in the affine mode in the neighboring block. In some embodiments, the number of merge candidates derived from the neighboring blocks predicted in the affine mode can be limited. For example, the inter-frame predictor 124 can derive two merge candidates based on the neighboring blocks predicted in the affine mode: one of L and BL and one of A, AR and AL. Priorities can be assigned in the order of L and BL and in the order of A, AR and AL.

[0177] When the total number of merge candidates is greater than or equal to 3, the inter predictor 124 may derive a necessary number of merge candidates according to translational motion vectors of neighboring blocks.

[0178] Fig.10 is an exemplary diagram illustrating a method of deriving merge candidates for affine motion prediction according to translational motion vectors of neighboring blocks.

[0179] The inter-frame predictor 124 derives control point motion vectors CPMV1, CPMV2, CPMV3 from the adjacent block group {B2, B3, A2}, the adjacent block group {B1, B0}, and the adjacent block group {A1, A0}, respectively. As an example, the priority in each adjacent block group can be assigned in the order of B2, B3, A2, the order of B1 and B0, and the order of A1 and A0. In addition, another control point motion vector CPMV4 is derived based on the co-located block T in the reference image. The inter-frame predictor 124 generates as many merge candidates as required by combining two or three of the four control point motion vectors. The combined priority assignment is as follows. The elements in each group are listed in the order of the upper left corner, upper right corner, and lower left corner of the control point motion vector.

[0180] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0181] The inter-frame predictor 124 selects a merge candidate in the merge candidate list and performs affine motion prediction on the target block. When the selected candidate includes two control point motion vectors, affine motion prediction is performed using a four-parameter model. On the other hand, when the selected candidate includes three control point motion vectors, affine motion prediction is performed using a six-parameter model. The entropy encoder 155 of the video encoding device encodes index information indicating the merge candidate selected from the merge candidates in the merge candidate list and notifies the video decoding device with a signal.

[0182] The entropy decoder 510 of the video decoding device decodes the index information signaled from the video encoding device. The inter-frame predictor 544 of the video decoding device constructs a merge candidate list in the same manner as the video encoding device, and performs affine motion prediction using a control point motion vector corresponding to the merge candidate indicated by the index information.

[0183] As another example, in the AMVP mode, the inter-frame predictor 124 of the video encoding device determines the type of affine model and the control point motion vector for the target block. Then, the inter-frame predictor 124 calculates the motion vector difference, which is the difference between the actual control point motion vector of the target block and the predicted motion vector of each control point, and sends the motion vector difference corresponding to the control point respectively. To this end, the inter-frame predictor 124 of the video encoding device configures a predefined number of lists of affine AMVP. When the target block is a 4-parameter type, the candidates included in the list are each composed of two control point motion vectors in a pair. On the other hand, when the target block is a 6-parameter type, the candidates included in the list are each composed of three control point motion vectors in a pair. The affine AMVP list can be derived using the control point motion vectors or translation motion vectors of adjacent blocks in a manner similar to the above-mentioned method of constructing a merged candidate list.

[0184] However, in order to derive candidates to be included in the affine AMVP list, there may be a restriction that only neighboring blocks are considered, the neighboring blocks referenced with Figure 4 The target blocks in the adjacent blocks are the same as the reference image.

[0185] In addition, in the AMVP mode, the affine model type of the target block should be considered. When the affine model type of the target block is a 4-parameter type, the video encoding device uses the affine model of the adjacent block to derive two control point motion vectors (the upper left and upper right control point motion vectors of the target block). When the affine model type of the target block is a 6-parameter type, the device uses the affine model of the adjacent block to derive three control point motion vectors (the upper left, upper right and lower left control point motion vectors of the target block).

[0186] When the neighboring block is of 4-parameter type, two or three control point motion vectors are predicted according to the affine model type of the target block using the two control point motion vectors of the neighboring block. For example, the affine model of the neighboring block expressed by equation 11 may be used. In equation 11, (mv 0x , mv 0y ) and (mv 1x , mv 1y ) are replaced by the upper left and upper right control point motion vectors of the adjacent blocks, respectively. W is replaced by the horizontal length of the adjacent blocks. The predicted motion vector for each control point of the target block is derived by placing the difference between the position of the corresponding control point of the target block and the position of the upper left corner of the adjacent block into (x, y).

[0187] When the neighboring block is of 6-parameter type, two or three control point motion vectors are predicted according to the affine model type of the target block using the three control point motion vectors of the neighboring block. For example, the affine model of the neighboring block expressed by equation 12 may be used. In equation 12, (mv0x , mv 0y )、(mv 1x , mv 1y ) and (mv 2x , mv 2y ) are replaced by the control point motion vectors of the upper left corner, upper right corner and lower left corner of the adjacent block, respectively. W and H are replaced by the horizontal length and vertical length of the adjacent block, respectively. The predicted motion vector for each control point of the target block is derived by placing the difference between the position of the corresponding control point of the target block and the position of the upper left corner of the adjacent block into (x, y).

[0188] The inter-frame predictor 124 of the video encoding device selects a candidate in the affine AMVP list and generates a motion vector difference between the motion vector of each actual control point and the predicted motion vector of the corresponding control point of the selected candidate. The entropy encoder 155 of the video encoding device encodes type information indicating the affine model type of the target block, index information indicating the selected candidate among the candidates in the affine AMVP list, and the motion vector difference corresponding to each control point, and transmits it to the video decoding device.

[0189] The inter-frame predictor 544 of the video decoding device determines the affine model type using the information signaled from the video encoding device, and generates a motion vector difference for each control point. Then, the inter-frame predictor generates an affine AMVP list in the same manner as the video encoding device, and selects a candidate indicated by the index information signaled in the affine AMVP list. The inter-frame predictor 544 of the video decoding device calculates the motion vector of each control point by adding the predicted motion vector of each control point of the selected candidate to the corresponding motion vector difference.

[0190] IV. Sample-by-sample adjustment for affine motion prediction samples

[0191] Subblock-by-subblock affine motion prediction for a target block has been described above. Another aspect of the invention relates to adjusting the sample values ​​of predicted samples generated by subblock-by-subblock affine motion prediction on a sample-by-sample basis. Motion according to the position of each sample is additionally compensated in each subblock forming the basis for the affine motion prediction.

[0192] When the sample value in any one sub-block generated as a result of sub-block-by-sub-block affine motion prediction for the target block is I(x, y), the video encoding apparatus calculates the horizontal gradient g at each sample position. x (i, j) and vertical gradient g y (i, j). Equation 13 can be used to calculate the gradient.

[0193] [Equation 13]

[0194] gx (i,j)=I(i+1,j)-I(i-1,j)

[0195] g y (i,j)=I(i,j+1)-I(i,j-1)

[0196] The sample offset ΔI(i, j) used to adjust the prediction sample is calculated by the following equation.

[0197] [Equation 14]

[0198] ΔI(i,j)=g x (i, j))*Δmv x (i, j)+g y (i, j)*Δmv y (i, j)

[0199] Here, Δmv(i, j) represents a motion offset, that is, the difference between the affine motion vector in the sample (i, j) and the affine motion vector at the center position of the sub-block, and can be calculated by applying Equation 11 or Equation 12 according to the affine model type of the target block. That is, Δmv(i, j) can be calculated according to Equation 11 or Equation 12 by subtracting the motion vector given when the sub-block center position is placed in (x, y) from the motion vector given when (i, j) is placed in (x, y). In other words, Δmv(i, j) can be calculated by substituting the horizontal offset and the vertical offset from the sub-block center position to the sample position (i, j) into (x, y) in Equations 11 and 12 and eliminating the last term "+mv 0x ” and “+mv 0y ” is used to calculate Δmv(i, j) using the equation obtained by . The center position may be the actual center point of the sub-block, or may be the lower right sample position of the center point.

[0200] The motion vector of each control point of the target block used to calculate Δmv(i, j) and the difference between the sample position (i, j) and the center position of the sub-block are the same for all sub-blocks. Accordingly, the value of Δmv(i, j) can be used only to calculate one sub-block (e.g., the first sub-block) and can be reused for other sub-blocks.

[0201] The technique of the present invention is based on the assumption that the object moves at a constant speed and the variation of the sample values ​​is uniform. Therefore, by substituting the x-component (Δmv x ) and the y component (Δmv y ) are multiplied by the horizontal sample gradient value and the vertical sample gradient value respectively to obtain the sample variation in the horizontal direction and the sample variation in the vertical direction. The sample offset ΔI(i, j) is calculated by adding the two sample variations.

[0202] The final value of the predicted sample is calculated as follows.

[0203] [Equation 15]

[0204] I′(i, j)=I(i, j)+ΔI(i, j)

[0205] When the sample-by-sample adjustment for the affine motion prediction sample is applied, the inter-frame predictor 124 of the video encoding device and the inter-frame predictor 544 of the video decoding device perform the above-mentioned process to modify the sample value of the predicted sample generated by the affine motion prediction. The gradient value is derived from the predicted sample generated by the affine motion prediction, and Δmv(i, j) is derived from the control point motion vector of the target block. Therefore, the video encoding device is not required to signal additional information for the process of the present technology to the video decoding device.

[0206] The sample-by-sample adjustment technique described above for the affine motion prediction samples is applied to the luma component. Furthermore, this technique can even be applied to the chroma components. In this case, the Δmvx and Δmv calculated for the luma component are y Can be used as Δmv for chrominance components x and Δmv y That is, the video encoding device and the video decoding device calculate the gradient value for the predicted sample of the chrominance component generated by affine motion prediction. Then, the gradient value of the chrominance component and the Δmv calculated for the luminance component can be calculated by x and Δmv y Substitute into Equation 14 and Equation 15 to adjust the predicted samples of the chroma component generated by affine motion prediction.

[0207] In the case of bidirectional prediction, Equations 14 and 15 are applied to each of the two reference images. The video encoding device and the video decoding device generate two reference blocks by performing sample-by-sample adjustment on the affine prediction samples of each of the reference image of the reference image list 0 and the reference image of the reference image list 1. The two reference blocks are generated by Equations 14 and 15. The final prediction block of the target block can be generated by averaging between the two reference blocks. When the bit depth is 10, the process of generating the final prediction block is expressed as follows.

[0208] [Equation 16]

[0209]

[0210] In Equation 16, “I 0 (i, j) + clip3()" is a prediction sample in a reference block generated by applying the present technique to a reference image in reference image list 0, and "I 1(i, j)+clip3()" is the predicted sample in the reference block generated by applying the present technology to the reference image of reference image list 1.

[0211] In order to prevent a delay caused by performing a sample-by-sample adjustment technique on affine motion prediction samples, it may be determined whether application of the technique is appropriate before performing the technique, so that the technique may be skipped and not performed.

[0212] As an example, a video encoding device may determine whether to apply the present technology based on a predefined image area, and signal a flag indicating whether to apply the technology to a video decoding device. Here, the predefined image area may be an image sequence, an image, or a slice. When the application of the present technology is determined on a sequence-by-sequence, image-by-image, or slice-by-slice basis, the flag may be included in a sequence header (SPS), an image header (PPS), or a slice header. The video decoding device may extract the flag contained in the bitstream and determine whether to apply the present technology to a block in an image area corresponding to the extracted flag.

[0213] As another example, whether to apply the present technique to a target block may be predetermined based on the control point motion vector of the target block. When the values ​​of the control point motion vectors of the target block are all the same, the technique is not applied. In the case where the affine type of the target block is a 4-parameter model, when the control point motion vectors of the upper left corner and the upper right corner are the same, the technique is not performed. In the case of a 6-parameter model, when the control point motion vectors of the upper left corner, the upper right corner, and the lower left corner are the same, the technique is not performed.

[0214] As another example, whether to apply the present technology may be determined based on the angle between the control point motion vectors. For example, when the angle between the control point motion vectors is an obtuse angle (i.e., the dot product between the vectors is negative), the present technology may not be applied. Alternatively, when the angle between the control point motion vectors is an acute angle (i.e., the dot product between the vectors is positive), the application of the present technology may be limited.

[0215] As another example, when the control point motion vector of the target block refers to reference images in different reference image lists, the application of the present technology may be excluded.

[0216] As another example, to minimize latency, the technique may be limited for use with combined inter-intra prediction techniques, such as in the case of bidirectional optical flow. In addition, the application of the present technique may be excluded in the case of applying local illumination compensation described below or in the case of bidirectional prediction.

[0217] As another example, in unidirectional or bidirectional prediction, if the reference picture referred to by the target block is not a short-term reference picture, the technique is not performed.

[0218] As another example, when a current image including the target block or a reference image referred to by the target block is an image divided into sub-images, the technology is not applied.

[0219] As another example, when bidirectional prediction is performed and different weights are applied to two prediction blocks (first reference block and second reference block) generated by affine motion prediction, the present technique is not performed. That is, when the luminance weights to be applied to the two reference blocks (luminance components in the two reference blocks) to predict the luminance component are different from each other, the technique is not applied to the target block. Also, when the chrominance weights to be applied to the two reference blocks (chrominance components in the two reference blocks) to predict the chrominance component are different from each other, the technique is not applied to the target block.

[0220] Alternatively, in the case of bidirectional prediction applying different weights, the present technology may be performed by applying different weights. For example, a video encoding device or a video decoding device generates two reference blocks by performing sample-by-sample adjustment on affine prediction samples of each of the reference image of reference image list 0 and the reference image of reference image list 1. Hereinafter, a final prediction block is generated by applying corresponding weights to the two reference blocks.

[0221] Even after performing the sample-by-sample adjustment technique on the affine motion prediction samples, if Δmv x and Δmv y If the value of is less than a predetermined threshold, the execution of this technology can be stopped.

[0222] V. Local illumination compensation

[0223] The local illumination compensation technology is a coding technology that uses a linear model to compensate for the illumination change between the target block and the predicted block. The inter-frame predictor 124 of the video encoding device uses the motion vector (translation motion vector) of the target block to determine the reference block in the reference image, and uses the pre-reconstructed samples around the reference block (on the upper and left sides of the reference block) and the pre-reconstructed samples around the target block (on the upper and left sides of the reference block) to obtain the parameters of the linear model for illumination compensation.

[0224] When the pre-reconstructed samples around the reference block are called x and the corresponding pre-reconstructed samples around the target block are called y, the parameters "A" and "b" are derived as in Equation 17 such that the sum of the squares of the differences between y and (Ax+b) is minimized.

[0225] [Equation 17]

[0226] argmin{∑(y-Ax-b) 2}

[0227] The final prediction sample is generated by applying the weight A and the offset b to the sample in the prediction block (reference block) generated by the motion vector of the target block, as shown in Equation 18. In Equation 18, pred[x][y] is the prediction sample at the (x, y) position generated by the motion vector of the target block, and pred LIC [x][y] are the final predicted samples after illumination compensation.

[0228] [Equation 18]

[0229] pred LIC [x][y]=A*pred[x][y]+b

[0230] Another aspect of the invention relates to techniques for combining illumination compensation techniques with affine motion prediction.

[0231] As described above, when sub-block-by-sub-block affine motion prediction is applied to a target block, a motion vector is generated for each sub-block. The corresponding motion vector for each sub-block can be used to derive illumination compensation parameters, and then the corresponding motion vector can be used to perform illumination compensation in units of sub-blocks. However, this not only increases the complexity of the calculation, but also causes serious delay problems. Since the reconstructed samples in each sub-block require illumination compensation for subsequent sub-blocks, the illumination compensation process for the sub-block should be suspended until the adjacent sub-blocks are reconstructed (that is, until both the prediction block and the residual block for the sub-block are reconstructed). The present invention is intended to solve these problems.

[0232] Fig.11a , Fig.11b and Fig.11c Various examples of determining the position of a reference block to derive illumination compensation parameters according to an embodiment of the present invention are shown. In this embodiment, one illumination compensation parameter set (A, b) for a target block is derived, and the same parameters are applied to all sub-blocks in the target block. That is, the entire target block is modified with one illumination compensation parameter set.

[0233] like Fig.11a As shown, the inter-frame predictor 124 of the video encoding device can determine the position of the reference block in the reference image using the affine motion vector of the sub-block located at the upper left of the target block or the motion vector of the upper left corner control point of the target block. The pre-reconstructed samples around the determined reference block are used for parameter derivation. Alternatively, as Fig.11b As shown, the position of the reference block can be determined using the affine motion vector of the central sub-block in the target block. Once the position of the reference block is determined, the illumination compensation parameters are derived using the pre-reconstructed samples adjacent to the upper and left sides of the reference block and the corresponding pre-reconstructed samples adjacent to the upper and left sides of the target block.

[0234] As another example, multiple sub-blocks in the target block may be used. Fig.11c As shown, the inter-frame predictor 124 determines the reference sub-block corresponding to each boundary sub-block using the affine motion vector of the sub-block (boundary sub-block) located at the boundary in the target block. Samples for deriving illumination compensation parameters are extracted from pre-reconstructed samples adjacent to the boundary sub-block and the corresponding reference sub-block in the target block, respectively. For the sub-block located at the upper boundary in the target block and the corresponding reference sub-block, samples are extracted from the pre-reconstructed samples adjacent to the upper side. For the sub-block located at the left boundary in the target block and the corresponding reference sub-block, samples are extracted from the pre-reconstructed samples adjacent to the left side.

[0235] The one or more coding tools described above may be used to improve the prediction performance of inter-frame prediction. In order to address issues such as complexity or delay, applying certain coding tools may require excluding other coding tools from being applied.

[0236] On the other hand, both the sample-by-sample adjustment of the affine prediction sample and the bidirectional optical flow are techniques for modifying the predicted sample after prediction, and the modification is performed using the gradient of the sample. Accordingly, in order to reduce computational complexity and hardware complexity, the equation of the bidirectional optical flow may be modified in the form of an equation for sample-by-sample adjustment of the affine prediction sample for bidirectional prediction. Alternatively, the equation for sample-by-sample adjustment of the affine prediction sample for bidirectional prediction may be modified in the form of an equation for the bidirectional optical flow.

[0237] By substituting Equation 9 into Equation 10, the equation for obtaining the final predicted sample in the bidirectional optical flow can be expressed as follows.

[0238] [Equation 19]

[0239]

[0240] Equation 19 is expressed in the form of Equation 16 as follows.

[0241] [Equation 20]

[0242]

[0243] That is, the final predicted sample to which the bidirectional optical flow is applied can be calculated by Equation 20 instead of Equation 19. Since Equation 20 is expressed in the form of a sample-by-sample adjustment technique similar to that of the affine predicted sample, there is no need to separately design hardware to implement the equation according to the bidirectional optical flow technique. In addition, since Equation 20 is expressed in the form of an average of a predicted block of a reference image in reference image list 0 and a predicted block of a reference image in reference image list 1, hardware design is simplified.

[0244] In addition, the motion offset of the bidirectional optical flow technique (optical flow) (v x , v y ) and the motion offset (Δmv) of the sample-by-sample adjustment technique for affine prediction. x , Δmv y ) can match each other in terms of motion vector accuracy. For example, the motion offsets of the two techniques can both be expressed with an accuracy of 1 / 32 sub-sample units.

[0245] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions described in one or more examples can be implemented as hardware, software, firmware or any combination thereof. It should be understood that the functional components described herein have been labeled as "... units" to further emphasize their implementation independence.

[0246] The various functions or methods described in the present invention can be implemented as instructions stored in a non-volatile recording medium, which can be read and executed by one or more processors. Non-volatile recording media include, for example, all types of recording devices in which data is stored in a form that can be read by a computer system. For example, non-volatile recording media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSD).

[0247] Although exemplary embodiments have been described for illustrative purposes, it will be appreciated by those skilled in the art that various modifications and variations are possible without departing from the spirit and scope of the embodiments. For the sake of brevity and clarity, exemplary embodiments have been described. Accordingly, it will be appreciated by those of ordinary skill that the scope of the embodiments is not limited to the embodiments explicitly described above, but is included in the claims and their equivalents.

Claims

1. A video decoding method for reconstructing a target block in a current image to be decoded, the video decoding method comprising: Determining a first reference image and a second reference image and a first motion vector and a second motion vector for bidirectional prediction by decoding a bit stream; generating a first reference block from a first reference image referenced by a first motion vector, and generating a second reference block from a second reference image referenced by a second motion vector; generating a prediction block of the target block using the first reference block and the second reference block; generating a residual block of the target block based on a residual signal decoded from the bitstream; Reconstruct the target block based on the prediction block of the target block and the residual block of the target block, The step of generating a prediction block of a target block by using the first reference block and the second reference block includes: executing a first encoding tool configured to generate a prediction block of a target block by performing a bidirectional optical flow process using a first reference block and a second reference block, wherein the first coding tool is applied only to the luminance component of the target block, wherein the first encoding tool is not performed when the luminance weights assigned to each of the first reference image and the second reference image for predicting the luminance component of the target block are different from each other, and When chroma weights assigned to each of the first reference image and the second reference image for predicting a chroma component of the target block are different from each other, the first encoding tool is not performed.

2. The video decoding method according to claim 1, further comprising: When the luminance weights assigned to each of the first reference image and the second reference image are different from each other, predicting a luminance component of the target block by applying the luminance weights respectively corresponding to the first reference block and the second reference block to the first reference block and the second reference block; as well as When the chroma weights assigned to each of the first reference image and the second reference image are different from each other, the chroma components of the target block are predicted by applying the chroma weights respectively corresponding to the first reference block and the second reference block to the first reference block and the second reference block.

3. The video decoding method according to claim 1, wherein: When executing the first encoding tool, generating a prediction block for a sub-block partitioned from a target block includes: generating a first horizontal gradient and a first vertical gradient for each luminance sample of a subblock in a first reference block corresponding to a subblock of a target block, generating a second horizontal gradient and a second vertical gradient for each luminance sample of a sub-block in a second reference block corresponding to a sub-block of the target block, calculating a motion offset corresponding to a sub-block of the target block using the first horizontal gradient and the second horizontal gradient and the first vertical gradient and the second vertical gradient for the luma sample, and Luminance samples in a sub-block of a target block are predicted using luma sample values ​​of a sub-block of a first reference block, luma sample values ​​of a sub-block of a second reference block, and a motion offset.

4. The video decoding method according to claim 3, wherein: The luminance samples in the sub-block of the prediction target block include: calculating a sample offset of a luma sample position in the sub-block of the target block using a difference between a first horizontal gradient and a second horizontal gradient corresponding to the luma sample position, a difference between a first vertical gradient and a second vertical gradient corresponding to the luma sample position, and a motion offset corresponding to the sub-block of the target block, and A luma sample at a luma sample position is predicted using luma sample values ​​in a first reference block and a second reference block corresponding to the luma sample position and a sample offset for the luma sample position.

5. The video decoding method according to claim 1, wherein: Generating a prediction block of a target block using the first reference block and the second reference block further includes: executing a second encoding tool configured to generate an inter-prediction block using the first reference block and the second reference block, generate an intra-prediction block by performing intra-prediction on the target block, and generate a prediction block of the target block by weighted averaging the inter-prediction block and the intra-prediction block, Wherein, the execution of the second encoding tool restricts the execution of the first encoding tool.

6. The video decoding method according to claim 5, wherein: When the second encoding tool is executed, the intra prediction block is generated using the planar mode among the plurality of intra prediction modes.

7. The video decoding method according to claim 5, wherein: The weight value used for weighted averaging is determined by the number of intra prediction blocks in neighboring blocks including the left block and the upper block of the target block.

8. A video encoding method for encoding a target block in a current image to be encoded, the video encoding method comprising: determining a first motion vector and a second motion vector for bidirectional use; generating a first reference block from a first reference image referenced by a first motion vector, and generating a second reference block from a second reference image referenced by a second motion vector; generating a prediction block of the target block using the first reference block and the second reference block; Generate a residual block based on the prediction block of the target block and encode the residual block. The step of generating a prediction block of a target block by using the first reference block and the second reference block includes: executing a first encoding tool configured to generate a prediction block of a target block by performing a bidirectional optical flow using a first reference block and a second reference block, wherein the first coding tool is applied only to the luminance component of the target block, wherein the first encoding tool is not performed when the luminance weights assigned to each of the first reference image and the second reference image for predicting the luminance component of the target block are different from each other, and When chroma weights assigned to each of the first reference image and the second reference image for predicting a chroma component of the target block are different from each other, the first encoding tool is not performed.

9. The video encoding method according to claim 8, wherein: When executing the first encoding tool, generating a prediction block for a sub-block partitioned from a target block includes: generating a first horizontal gradient and a first vertical gradient for each luminance sample of a subblock in a first reference block corresponding to a subblock of a target block, generating a second horizontal gradient and a second vertical gradient for each luminance sample of a sub-block in a second reference block corresponding to a sub-block of the target block, calculating a motion offset corresponding to a sub-block of the target block using the first horizontal gradient and the second horizontal gradient and the first vertical gradient and the second vertical gradient for the luma sample, and Luminance samples in a sub-block of a target block are predicted using luma sample values ​​of a sub-block of a first reference block, luma sample values ​​of a sub-block of a second reference block, and a motion offset.

10. The video encoding method according to claim 9, wherein: The luminance samples in the sub-block of the prediction target block include: calculating a sample offset of a luma sample position in the sub-block of the target block using a difference between a first horizontal gradient and a second horizontal gradient corresponding to the luma sample position, a difference between a first vertical gradient and a second vertical gradient corresponding to the luma sample position, and a motion offset corresponding to the sub-block of the target block, and A luma sample at a luma sample position is predicted using luma sample values ​​in a first reference block and a second reference block corresponding to the luma sample position and a sample offset for the luma sample position.

11. The video encoding method according to claim 8, wherein: Generating a prediction block of a target block using the first reference block and the second reference block further includes: executing a second encoding tool configured to generate an inter-prediction block using the first reference block and the second reference block, generate an intra-prediction block by performing intra-prediction on the target block, and generate a prediction block of the target block by weighted averaging the inter-prediction block and the intra-prediction block, Wherein, the execution of the second encoding tool restricts the execution of the first encoding tool.

12. The video encoding method according to claim 8, wherein: The weight used for weighted averaging is determined by the number of intra-prediction blocks in neighboring blocks including the left block and the upper block of the target block.

13. An apparatus for sending a bitstream comprising video data to a decoding apparatus, the apparatus comprising at least one processor, the at least one processor being configured to: generating a bitstream by encoding the video data based on encoding the target block in a current picture to be encoded; Sending a bit stream to a decoding device, in, The at least one processor is configured to encode the target block by the following steps: determining a first motion vector and a second motion vector for bidirectional use; generating a first reference block from a first reference image referenced by a first motion vector, and generating a second reference block from a second reference image referenced by a second motion vector; generating a prediction block of the target block using the first reference block and the second reference block; Generate a residual block based on the prediction block of the target block and encode the residual block. The step of using the first reference block and the second reference block to predict the target block includes: executing a first encoding tool configured to generate a prediction block of a target block by performing a bidirectional optical flow process using a first reference block and a second reference block, wherein the first coding tool is applied only to the luminance component of the target block, wherein the first encoding tool is not performed when the luminance weights assigned to each of the first reference image and the second reference image for predicting the luminance component of the target block are different from each other, and When chroma weights assigned to each of the first reference image and the second reference image for predicting a chroma component of the target block are different from each other, the first encoding tool is not performed.

Images