Video decoding method for predicting the target block, video encoding method for predicting the inter-image of the target block, and a bitstream transmission device containing encoded video data.

Abstract

The invention proposes a video decoding method for predicting the target block in the current image to be decoded. The method comprises the following steps: identifying the first and second reference images and the first and second motion vectors for double prediction by decoding the bit stream, generating the first reference block from the first reference image referenced by the first motion vector and generating the second reference block from the second reference image referenced by the second motion vector, and generating the predicted block of the target block using the first and second reference blocks. The invention also proposes a bit stream transmission device containing encoded video data. (Fig. 5)

Description

Method and device for encoding and decoding video using inter prediction

[0001] The present invention relates to encoding and decoding of images (videos), and more particularly, to a coding tool for improving compression performance of inter prediction.

[0002] Since video data has a large amount of data compared to voice data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.

[0003] Therefore, when storing or transmitting video data, an encoder is used to compress the video data and store or transmit it, and a decoder receives the compressed video data, decompresses it, and plays it back. Examples of such video compression technologies include H.264 / AVC, as well as HEVC (High Efficiency Video Coding), which improves encoding efficiency by about 40% compared to H.264 / AVC.

[0004] However, as the size, resolution, and frame rate of images are gradually increasing, and the amount of data that needs to be encoded is also increasing, a new compression technology that has better encoding efficiency and better image quality improvement than existing compression technologies is required.

[0005] In image (video) encoding, predictive coding is primarily used to improve compression performance. There are two types of prediction: intraprediction, which uses previously restored samples within the current picture to predict the target block to be encoded, and interprediction, which uses previously restored reference pictures to predict the current block. Interprediction, in particular, offers superior compression performance compared to intraprediction, and is widely used in video encoding.

[0006] In this disclosure, we propose a coding tool that improves the compression performance of existing inter prediction.

[0007] The present disclosure relates to a coding tool for improving the compression performance of inter prediction, and in one aspect, to a coding tool capable of compensating for various motions of an object as well as translation motion at the block level.

[0008] One aspect of the present disclosure provides a video decoding device for predicting a target block in a current picture to be decoded. The video decoding device includes a prediction unit that determines first and second reference pictures and first and second motion vectors for bi-prediction by decoding a bitstream, generates a first reference block from the first reference picture referenced by the first motion vector, generates a second reference block from the second reference picture referenced by the second motion vector, and predicts the target block using the first and second reference blocks. The prediction unit includes a first coding tool that generates a prediction block of the target block by performing a bi-directional optical flow process using the first and second reference blocks. When the luma weights assigned to the first and second reference pictures respectively for predicting the luma component of the target block are different from each other, the first coding tool is not executed, and when the chroma weights assigned to the first and second reference pictures respectively for predicting the chroma component of the target block are different from each other, the first coding tool is not executed.

[0009] Another aspect of the present disclosure provides a video encoding device for inter-predicting a target block within a current picture to be encoded. The video encoding device includes a prediction unit that determines a first motion vector and a second motion vector for bi-prediction, generates a first reference block from a first reference picture referenced by the first motion vector, generates a second reference block from a second reference picture referenced by the second motion vector, and predicts the target block using the first and second reference blocks. The prediction unit includes a first coding tool that generates a prediction block of the target block by performing a bi-directional optical flow process using the first and second reference blocks. When the luma weights assigned to the first and second reference pictures respectively for predicting the luma component of the target block are different from each other, the first coding tool is not executed, and when the chroma weights assigned to the first and second reference pictures respectively for predicting the chroma component of the target block are different from each other, the first coding tool is not executed.

[0010] Another aspect of the present disclosure provides a method for predicting a target block within a current picture. The method comprises the steps of: determining a first motion vector and a second motion vector for bi-prediction; generating a first reference block from a first reference picture referenced by the first motion vector and generating a second reference block from a second reference picture referenced by the second motion vector; and predicting the target block using the first and second reference blocks. The step of predicting the target block includes the step of executing a first coding tool that generates a prediction block of the target block by performing a bi-directional optical flow process using the first and second reference blocks, wherein the first coding tool is not executed when luma weights assigned to the first and second reference pictures respectively for predicting a luma component of the target block are different from each other, and the first coding tool is not executed when chroma weights assigned to the first and second reference pictures respectively for predicting a chroma component of the target block are different from each other.

[0011] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure.

[0012] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT structure.

[0013] Figure 3 is a diagram showing multiple intra prediction modes.

[0014] Figure 4 is an example diagram of the surrounding blocks of the current block.

[0015] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure.

[0016] FIG. 6 is an exemplary diagram illustrating the concept of bi-prediction optical flow provided by the present disclosure.

[0017] Figure 7 is an example diagram illustrating a method for deriving a gradient for block boundary samples in bidirectional optical flow.

[0018] FIG. 8 and FIG. 9 are exemplary diagrams for explaining affine motion prediction provided by the present disclosure.

[0019] Figure 10 is an example diagram illustrating a method for deriving merge candidates for affine motion prediction from translation motion vectors of surrounding blocks.

[0020] FIG. 11 is an exemplary diagram illustrating a method for deriving lighting compensation parameters according to one embodiment of lighting compensation provided by the present disclosure.

[0021] Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. When assigning identification numbers to components in each drawing, it should be noted that, where possible, identical components are assigned the same numbers, even if they appear in different drawings. Furthermore, when describing the present invention, detailed descriptions of known related components or functions will be omitted if they are deemed to obscure the gist of the present invention.

[0022] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image encoding device and its subcomponents will be described with reference to FIG. 1.

[0023] The video encoding device may be configured to include a block division unit (110), a prediction unit (120), a subtractor (130), a transformation unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse transformation unit (165), an adder (170), a loop filter unit (180), and a memory (190).

[0024] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0025] A single image (video) is composed of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or / slice is divided into one or more Coding Tree Units (CTUs). Each CTU is then divided into one or more Coding Units (CUs) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks within a single slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced in a sequence composed of multiple pictures is encoded in a Sequence Parameter Set (SPS). In addition, information commonly applied to a single tile or a group of tiles may be encoded as syntax of a tile or tile group header. Syntaxes included in an SPS, PPS, slice header, tile, or tile group header may be referred to as high-level syntax.

[0026] Each of the multiple pictures may be divided into multiple subpictures that can be independently encoded / decoded and / or displayed. When subpicture division is applied, information regarding the layout of the subpictures within the picture is signaled.

[0027] The block division unit (110) determines the size of the CTU (Coding Tree Unit). Information about the size of the CTU (CTU size) is encoded as a syntax of SPS or PPS and transmitted to the image decoding device.

[0028] The block division unit (110) divides each picture constituting an image into a plurality of Coding Tree Units (CTUs) having a predetermined size, and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a coding unit (CU), which is the basic unit of encoding.

[0029] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, or a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that mixes two or more of the QT structures, BT structures, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be combined and referred to as a MTT (Multiple-Type Tree).

[0030] Figure 2 shows the QTBTTT split tree structure. As shown in Figure 2, a CTU can first be split into a QT structure. The quadtree splitting can be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoding unit (155) and signaled to the image decoding device. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in BT, it can be further split into one or more of the BT structure or the TT structure. There can be multiple splitting directions in the BT structure and / or the TT structure. For example, there can be two directions in which the block of the corresponding node is split horizontally and two directions in which the block is split vertically. As shown in Fig. 2, when MTT splitting starts, a second flag (mtt_split_flag) indicating whether nodes have been split, and if splitting has been performed, a flag additionally indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (Binary or Ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. Alternatively, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of the lower layer. If the CU split flag (split_cu_flag) value indicates that it is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is the basic unit of encoding.If the CU split flag (split_cu_flag) value indicates that the CU is split, the video encoding device starts encoding from the first flag in the manner described above.

[0031] As another example of a tree structure, when QTBT is used, there may be two types: a type that horizontally splits the block of the corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that vertically splits it (i.e., symmetric vertical splitting). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating the type of split are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may additionally be a type that splits the block of the corresponding node into two blocks of an asymmetrical shape. The asymmetric shape may include a shape that splits the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that splits the block of the corresponding node in a diagonal direction.

[0032] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." Depending on the QTBTTT partitioning employed, the current block may be rectangular as well as square.

[0033] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

[0034] The intra prediction unit (122) predicts samples within the current block using samples (reference samples) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in Fig. 3, the multiple intra prediction modes may include two non-directional modes including a planar mode and a DC mode, and 65 directional modes. The surrounding samples to be used and the calculation formula are defined differently depending on each prediction mode.

[0035] The intra prediction unit (122) can determine an intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit (122) can encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra prediction unit (122) can calculate rate-distortion values ​​using rate-distortion analysis for multiple tested intra prediction modes and select an intra prediction mode with the best rate-distortion characteristics among the tested modes.

[0036] The intra prediction unit (122) selects one intra prediction mode from among multiple intra prediction modes and predicts the current block using surrounding samples (reference samples) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0037] The inter prediction unit (124) generates a prediction block for the current block through a motion compensation process. The inter prediction unit (124) searches for a block most similar to the current block within reference pictures that were encoded and decoded before the current picture, and uses the searched block to generate a prediction block for the current block. Then, a motion vector corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture is generated. Generally, motion estimation is performed on the luma component, and a motion vector calculated based on the luma component is used for both the luma component and the chroma component. Motion information including information on the reference picture used to predict the current block and information on the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0038] The inter prediction unit (124) may perform interpolation for a reference picture or a reference block to improve the accuracy of prediction. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed for the interpolated reference picture, the motion vector can be expressed up to a precision in decimal units rather than a precision in integer sample units. The precision of the motion vector, i.e., the motion vector resolution, can be set differently for each target region to be encoded, such as a slice, tile, CTU, CU, etc. When such an adaptive motion vector resolution is applied, information on the motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on the motion vector resolution applied to each CU is signaled.

[0039] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors indicating the block position most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture to generate a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged to generate a prediction block for the current block. Then, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is transmitted to the encoding unit (150). Here, reference picture list 0 may be composed of pictures that are before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures that are after the current picture in display order among the restored pictures. However, this is not necessarily limited to this, and restored pictures that are after the current picture in display order may be additionally included in reference picture list 0, and conversely, restored pictures that are before the current picture may be additionally included in reference picture list 1.

[0040] The inter prediction unit (124) may also perform bidirectional prediction using a weighted average, so-called weighted bidirectional prediction. The inter prediction unit (124) determines weights to be applied to the first reference picture and the second reference picture, respectively. The weights assigned to the first reference picture are applied to blocks in the first reference picture, and the weights assigned to the second reference picture are applied to blocks in the second reference picture. The inter prediction unit (124) applies the weights assigned to the first reference picture to the first reference block, and the weights assigned to the second reference picture to the second reference block, thereby generating a final prediction block of the target block through a weighted sum or weighted average operation of the first reference block and the second reference block. Weight information for the reference pictures used for inter prediction is signaled to the video decoding device.

[0041] Meanwhile, the weights used to predict the luma component and the weights used to predict the chroma component can be determined independently. In this case, information about the luma weight to be applied to the luma component and the chroma weight to be applied to the chroma component are separately signaled.

[0042] Motion information (motion vectors, reference pictures) used for inter prediction must be signaled to the video decoding device. Various methods can be used to minimize the number of bits required to encode the motion information.

[0043] For example, if the reference picture and motion vector of the current block are identical to those of a neighboring block, the motion information of the current block can be transmitted to the decoding device by encoding information that can identify the neighboring block. This method is called 'merge mode'.

[0044] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.

[0045] As the surrounding blocks for deriving merge candidates, all or part of the left block (L), upper block (A), upper right block (AR), lower left block (BL), and upper left block (AL) adjacent to the current block within the current picture may be used, as illustrated in FIG. 4. In addition, a block located within a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block within the reference picture or blocks adjacent to the co-located block may be additionally used as a merge candidate.

[0046] The inter prediction unit (124) uses these surrounding blocks to construct a merge list containing a predetermined number of merge candidates. Among the merge candidates included in the merge list, a merge candidate to be used as motion information of the current block is selected and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the encoding unit (155) and transmitted to the decoding device.

[0047] Another method for encoding motion information is AMVP mode.

[0048] In AMVP mode, the inter prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive predicted motion vector candidates, all or some of the left block (L), upper block (A), upper right block (AR), lower left block (BL), and upper left block (AL) adjacent to the current block in the current picture shown in FIG. 2 may be used. In addition, a block located in a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as the neighboring block used to derive predicted motion vector candidates. For example, a block located in the same location as the current block (co-located block) in the reference picture or blocks adjacent to the block in the same location may be used.

[0049] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of these surrounding blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to produce a differential motion vector.

[0050] The predicted motion vector can be obtained by applying a predefined function (e.g., median, mean, etc.) to the predicted motion vector candidates. In this case, the image decoding device also knows the predefined function. In addition, since the surrounding blocks used to derive the predicted motion vector candidates are blocks that have already been encoded and decoded, the image decoding device also already knows the motion vectors of the surrounding blocks. Therefore, the image encoding device does not need to encode information for identifying the predicted motion vector candidates. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.

[0051] Alternatively, the predicted motion vector can be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the differential motion vector and the reference picture used to predict the current block.

[0052] The subtractor (130) subtracts the prediction block generated by the intra prediction unit (122) or inter prediction unit (124) from the current block to generate a residual block.

[0053] The transformation unit (140) can transform residual signals within a residual block. The two-dimensional size of the residual block can be used as a transform unit (hereinafter referred to as "TU"), which is a block size for performing transformation. Alternatively, the residual block can be divided into multiple sub-blocks and each sub-block can be used as a TU to transform the residual signals within the corresponding sub-blocks.

[0054] The transform unit (140) divides the residual block into one or more sub-blocks, applies a transform to the one or more sub-blocks, and transforms the residual values ​​of the transform blocks from the pixel domain to the frequency domain. In the frequency domain, the transformed blocks are referred to as coefficient blocks or transform blocks that include one or more transform coefficient values. A two-dimensional transform kernel may be used for the transform, and a one-dimensional transform kernel may be used for each of the horizontal and vertical directions. The transform kernel may be based on a discrete cosine transform (DCT), a discrete sine transform (DST), etc.

[0055] The transformation unit (140) can individually perform transformations in the horizontal and vertical directions on the residual block or transformation unit. For the transformation, various types of transformation kernels or transformation matrices can be used. For example, a pair of transformation kernels for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one transformation kernel pair with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) on the transformation kernel pair selected among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0056] The quantization unit (145) quantizes the transform coefficients output from the transform unit (140) using quantization parameters and outputs the quantized transform coefficients to the entropy encoding unit (155). The quantization unit (145) may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit (145) may also apply different quantization coefficients (scaling values) according to the positions of the transform coefficients within the transform block. A matrix of quantization coefficients applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to an image decoding device.

[0057] The reordering unit (150) can reorder coefficient values ​​for quantized residual values. The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence through coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from the DC coefficient to the coefficients of the high-frequency region using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans a two-dimensional coefficient array in the column direction or a horizontal scan that scans two-dimensional block-shaped coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, a scanning method to be used among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan may be determined.

[0058] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transform coefficients output from the rearrangement unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

[0059] In addition, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoding device can divide the block in the same manner as the image encoding device. In addition, the entropy encoding unit (155) encodes information about a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., information about an intra prediction mode) or inter prediction information (in the case of a merge mode, a merge index, and in the case of an AMVP mode, information about a reference picture index and a differential motion vector) according to the prediction type. In addition, the entropy encoding unit (155) encodes information related to quantization, that is, information about a quantization parameter and information about a quantization matrix.

[0060] The inverse quantization unit (160) inversely quantizes the quantized transform coefficients output from the quantization unit (145) to generate transform coefficients. The inverse transform unit (165) transforms the transform coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

[0061] The addition unit (170) adds the restored residual block and the prediction block generated by the prediction unit (120) to restore the current block. Samples within the restored current block are used as reference samples when intra-predicting the next block.

[0062] The loop filter unit (180) performs filtering on restored samples to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation / quantization. The loop filter unit (180) may include one or more of a deblocking filter (182), a sample adaptive offset (SAO) filter (184), and an adaptive loop filter (ALF, 186).

[0063] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-based encoding / decoding, and the SAO filter (184) performs additional filtering on the deblocking-filtered image. The SAO filter (184) is a filter used to compensate for differences between restored samples and original samples caused by lossy coding, and is performed by adding an offset corresponding to each restored sample. The ALF (186) performs filtering on the target sample by applying filter coefficients to the target sample to be filtered and the surrounding samples of the target sample. The ALF (186) can divide the samples included in the image into a predetermined group and then determine one filter to be applied to the group to differentially perform filtering on each group. Information on the filter coefficients to be used in the ALF can be encoded and signaled to the image decoding device.

[0064] The restored blocks filtered through the loop filter unit (180) are stored in the memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0065] FIG. 5 is an exemplary functional block diagram of an image decoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image decoding device and its subcomponents will be described with reference to FIG. 5.

[0066] The video decoding device may be configured to include an entropy decoding unit (510), a rearrangement unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).

[0067] Similar to the video encoding device of FIG. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0068] The entropy decoding unit (510) decodes the bitstream generated by the image encoding device to extract information related to block division, thereby determining the current block to be decoded, and extracts prediction information and information on residual signals required to restore the current block.

[0069] The entropy decoding unit (510) extracts information about the CTU size from the Sequence Parameter Set (SPS) or the Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and the CTU is divided using the tree structure by extracting division information about the CTU.

[0070] For example, when splitting a CTU using the QTBTTT structure, first, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, the second flag (MTT_split_flag) related to the splitting of MTT and the split direction (vertical / horizontal) and / or split type (binary / ternary) information are extracted, and the corresponding leaf nodes are split into the MTT structure. Through this, each node below the leaf nodes of QT is recursively split into the BT or TT structure.

[0071] As another example, when splitting a CTU using the QTBTTT structure, the CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and if the block is split, the first flag (QT_split_flag) may be extracted. During the splitting process, each node may undergo zero or more repeated QT splits followed by zero or more repeated MTT splits. For example, a CTU may undergo an MTT split right away, or conversely, may undergo only multiple QT splits.

[0072] As another example, when splitting a CTU using the QTBT structure, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Furthermore, for nodes corresponding to leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and splitting direction information are extracted.

[0073] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded through division of the tree structure, it extracts information on the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-prediction, the entropy decoding unit (510) extracts a syntax element for intra-prediction information (intra-prediction mode) of the current block. If the prediction type information indicates inter-prediction, the entropy decoding unit (510) extracts a syntax element for inter-prediction information, i.e., information indicating a motion vector and a reference picture referenced by the motion vector.

[0074] Additionally, the entropy decoding unit (510) extracts information about the quantized transform coefficients of the current block as information related to quantization and information about the residual signal.

[0075] The rearrangement unit (515) can change the sequence of one-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the image encoding device.

[0076] The inverse quantization unit (520) inversely quantizes the quantized transform coefficients using quantization parameters. The inverse quantization unit (520) may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantized coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.

[0077] The inverse transform unit (530) inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a restored residual block for the current block. In addition, when MTS is applied, the inverse transform unit (530) determines a transform function or a transform matrix to be applied in the horizontal and vertical directions, respectively, using the MTS information (mts_idx) signaled from the image encoding device, and performs inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.

[0078] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is inter prediction.

[0079] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference samples around the current block according to the intra prediction mode.

[0080] The inter prediction unit (544) uses the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510) to determine the motion vector of the current block and the reference picture referenced by the motion vector, and predicts the current block using the motion vector and the reference picture.

[0081] The inter prediction unit (544), like the inter prediction unit (124) of the video encoding device, may also generate a prediction block of the current block using bidirectional prediction. When weighted bidirectional prediction is applied, the entropy decoding unit (510) extracts weight information applied to two reference pictures to be used for bidirectional prediction of the current block from the bitstream. The weight information may include weight information to be applied to a luma component and weight information to be applied to a chroma component. The inter prediction unit (544) generates a prediction block for the luma component of the current block and a prediction block for the chroma component using the weight information, respectively.

[0082] An adder (550) restores the current block by adding the residual block output from the inverse transform unit and the predicted block output from the inter prediction unit or the intra prediction unit. Samples within the restored current block are used as reference samples when intra-predicting a block to be decoded later.

[0083] The loop filter unit (560) may include at least one of a deblocking filter (562), an SAO filter (564), and an ALF (566). The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter (564) performs filtering by adding an offset corresponding to the restored block after deblocking filtering to compensate for differences between restored samples and original samples caused by lossy coding. The ALF (566) applies filter coefficients to a target sample to be filtered and surrounding samples of the target sample to perform filtering on the target sample. The ALF (566) divides samples included in an image into predetermined groups and determines one filter to be applied to each group to perform differential filtering for each group. The filter coefficients of the ALF are determined using information on filter coefficients decoded from the non-stream.

[0084] The restored block filtered through the loop filter unit (560) is stored in the memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within the picture to be encoded later.

[0085]

[0086] The following disclosure relates to a coding tool for improving the compression performance of inter prediction, and can be performed in an inter prediction unit (124) of an image encoding device and an inter prediction unit (544) of an image decoding device. The term 'target block' used below can be used with the same meaning as the current block or coding unit (CU) used above, or can also mean a portion of a coding unit.

[0087] I. Combined Inter-intra Prediction

[0088] As described above, the target block is predicted through either inter prediction or intra prediction. The inter-intra merge prediction described in the present disclosure is a technique that complements the inter prediction signal using the intra prediction signal. When the inter-intra merge prediction is applied, the inter prediction unit (124) of the video encoding device determines the motion vector of the target block and predicts the target block using the determined motion vector to generate an inter prediction block. Meanwhile, the intra prediction unit (124) of the video encoding device predicts the target block using reference samples around the target block and generates an intra prediction block. The intra prediction mode used to generate the intra prediction block may be any one of the multiple intra prediction modes described above. For example, the planar mode or the DC mode may be used as the prediction mode for generating the intra prediction block. The final prediction block is generated by the average or weighted average between the inter prediction block and the intra prediction block. The mathematical formula for calculating the final prediction block in the inter-intra merge prediction is as follows.

[0089]

[0090] Here, P inter is an inter prediction block, P intra represents the intra prediction block. wt represents the weight. +2 is an offset for rounding operation.

[0091] The weight can be determined based on whether the previously encoded / decoded neighboring blocks adjacent to the target block were predicted using an inter or intra prediction method. For example, if both the left block and the upper block of the target block were intra-predicted, the intra-predicted block (P intra ) is given a greater weight. For example, set wt = 3. If only one of the left block and the upper block is intra-predicted, the inter-prediction block (P inter ) and intra prediction block (P intra) are given the same weight. For example, set wt = 2. If both the left block and the upper block are not intra-predicted, the inter-prediction block (P inter ) is given a greater weight. For example, set wt = 1.

[0092] If the target block is predicted through inter-intra merge prediction, the inter prediction unit (544) of the video decoding device extracts information about the motion vector of the target block from the bitstream to determine the motion vector of the target block. Then, the target block is predicted in the same manner as the video encoding device.

[0093] Meanwhile, since inter-intra merge prediction is a technique that supplements the inter-prediction signal with the intra-prediction signal, it can be efficient in cases where inter-prediction is somewhat inaccurate, such as when the motion vector of the target block is determined by the merge mode. Therefore, inter-intra merge prediction can be applied only when the motion vector of the target block is determined by the merge mode.

[0094] II. Bidirectional Optical Flow

[0095] Bidirectional optical flow is a technique that additionally compensates for the motion of predicted samples using bidirectional motion prediction, based on the assumption that the samples or objects that make up an image move at a constant speed and there is little change in the sample values.

[0096] Figure 6 is a reference diagram to explain the basic concept of BIO.

[0097] Let us assume that bidirectional motion vectors (MV0, MV1) are determined by (normal) bidirectional motion prediction for the target block, which point to the corresponding regions (i.e., reference blocks) most similar to the target block to be encoded in the current picture in the reference pictures (Ref0 and Ref1). These two motion vectors represent the motion of the entire target block. In the example of Fig. 6, the sample in the reference picture (Ref0) pointed to by the motion vector (MV0) and corresponding to the sample P in the target block is P0, and the sample in the reference picture (Ref1) pointed to by the motion vector (MV1) and corresponding to the sample P in the target block is P1. In addition, let us assume that the motion for sample P in Fig. 6 is slightly different from the overall motion of the target block. For example, if an object located at sample A in Ref0 of Fig. 6 moves past sample P in the target block of the current picture to sample B in Ref1, samples A and B will have quite similar values. Also, in this case, the point within Ref0 that is most similar to the sample P within the target block is not P0 pointed to by the bidirectional motion vector (MV0), but P0 is the point within Ref0 that is most similar to the sample P within the target block by the given displacement vector (v x τ0, v y A is moved by τ0), and the point in Ref1 that is most similar to the sample P in the target block is not P1 pointed to by the bidirectional motion vector (MV1), but P1 is moved by a given displacement vector (-v x τ1, -v y B is moved by τ1). τ0 and τ1 represent the time axis distances for Ref0 and Ref1, respectively, based on the current picture, and are calculated based on POC (Picture Order Count). (v x , v y ) is referred to as “optical flow” or “motion offset.”

[0098] In predicting the sample (P) value of the target block in the current picture, using the values ​​of two reference samples (A, B) enables more accurate prediction than using the reference samples (P0, P1) pointed to by the bidirectional motion vectors (MV0, MV1).

[0099] The value of the sample in the reference picture (Ref0) pointed to by the motion vector (MV0) corresponding to the sample (i,j) in the target block is I (0) Define (i,j) as I, and the value of the sample in the reference picture (Ref1) pointed to by the motion vector (MV1) corresponding to the sample (i,j) in the target block is (1) Define (i,j).

[0100] BIO motion vector (v x , v y ) is the value of sample A in the reference picture Ref0 that corresponds to the sample in the target block. (0) (i + v x τ0, j + v y τ0) can be defined as, and the value of sample B in the reference picture Ref1 is I (1) (i - v x τ1, j - v y τ1) can be defined. Here, if a linear approximation is performed using only the first term of the Taylor series, A and B can be expressed as in mathematical expression 2.

[0101]

[0102] Here, I x (k) Wow I y (k)(k = 0, 1) represents the horizontal and vertical gradient values ​​at the (i, j) positions of Ref0 and Ref1. τ0 and τ1 represent the temporal distances of Ref0 and Ref1 from the current picture, respectively, and are calculated based on POC (Picture Order Count). The formulas are τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current).

[0103] Bidirectional optical flow (v) of each sample within the block x , v y ) is determined as the solution that minimizes △, which is defined as the difference between sample A and sample B. Using the linear approximation of A and B derived from mathematical expression 2, △ can be defined as mathematical expression 3.

[0104]

[0105] For brevity, the sample positions (i, j) in each term of the above equation (3) are omitted.

[0106] For more robust optical flow estimation, we introduce the assumption that the motion is locally consistent with the surrounding samples. The BIO motion vector for the current sample (i, j) to be predicted considers the difference values ​​△ of Equation 3 for all samples (i', j') existing within a mask Ω of a constant size centered on the sample (i, j). That is, the optical flow for the current sample (i, j) is the objective function Φ(v), which is the sum of the squares of the difference values ​​△[i', j'] obtained for each sample within the mask Ω, as shown in Equation 4 below. x , v y ) can be determined as a vector that minimizes .

[0107]

[0108] The bidirectional optical flow of the present disclosure can be applied when one of two reference pictures used for bidirectional prediction is displayed before the current picture and the other is displayed after the current picture, and the distances from the two reference pictures to the current picture are equal, i.e., the difference in picture order count (POC) between each reference picture and the current picture is equal. Accordingly, τ0 and τ1 can be ignored.

[0109] Additionally, the bidirectional optical flow of the present disclosure may be applied only to the luma component.

[0110] The bidirectional optical flow of the present disclosure is performed on a subblock basis, rather than a pixel basis, for the target block to which bidirectional prediction is applied. The subblock size may vary, such as 2x2, 4x4, or 8x8, but for convenience of explanation, the subblock size is assumed to be 4x4 below.

[0111] Before performing optical flow, the inter prediction unit (124) of the video encoding device generates two reference blocks for the target block using the bidirectional prediction described above. Among the two reference blocks, the first reference block refers to a block composed of prediction samples generated from a reference picture (Ref0) using a first motion vector (MV0) of the target block, and the second reference block refers to a block composed of prediction samples generated from a reference picture (Ref1) using a second motion vector (MV1) of the target block.

[0112] The inter prediction unit (124) uses the horizontal and vertical gradient values ​​of the prediction samples in the first reference block and the second reference block to generate (v) optical flow for each of the 4x4 sub-blocks constituting the target block. x , v y ) is computed. Optical flow (v x , v y) is determined so that the difference between the predicted samples from the reference picture (Ref0) and the predicted samples from the reference picture (Ref1) is minimized. The inter prediction unit (124) calculates (v) for the 4x4 sub-block. x , v y ) and derives a sample offset for modifying bidirectional prediction samples for subblock using the gradients of prediction samples within a 4x4 subblock.

[0113] Specifically, the inter prediction unit (124) calculates horizontal and vertical gradients of sample values ​​at positions (i, j) using mathematical expression 5.

[0114]

[0115] k is 0 or 1, and I (0) (i,j) and I (1) (i,j) represent the sample values ​​at positions (i,j) in the first and second reference blocks, respectively. Shift1 is a value derived from the bit-depth for the luma component, such as shift1 = max(6, bitDepth-6).

[0116] To derive the gradient of the samples located at the boundary of each reference block, samples outside the boundary of the first and second reference blocks are required. Therefore, as shown in Fig. 7, each reference block is expanded by one column to the left and right and one row to the top and bottom. To reduce the amount of computation, each sample in the expanded portion can be padded with the sample or integer sample at the nearest location within the reference block. In addition, the gradients at the sample locations outside the boundary of each reference block can also be padded with the gradients corresponding to the sample at the nearest location.

[0117] The inter prediction unit (124) uses horizontal and vertical gradients within a 6x6 window covering a 4x4 sub-block as illustrated in FIG. 7 to calculate S1, S2, S3, S5, and S6 corresponding to the auto-correlation and cross-correlation of the gradients.

[0118]

[0119] Here, Ω represents a window covering the subblock. Also, as shown in Equation 7 below, Ψ x (i,j) is the sum of the horizontal gradient values ​​at the (i,j) position within the first and second reference blocks, Ψ y (i,j) means the sum of the vertical gradient values ​​at the (i,j) position in the first reference block and the second reference block, and θ(i,j) means the difference between the sample value at the (i,j) position in the second reference block and the sample value at the (i,j) position in the first reference block.

[0120]

[0121] Here, na and nb are values ​​derived from bit depth, and have values ​​of min(1, bitDepth-11) and min(4, bitDepth-8).

[0122] The inter prediction unit (124) uses S1, S2, S3, S5, and S6 to calculate the optical flow (v) for a 4x4 sub-block using mathematical expression 8. x , v y ) is calculated.

[0123]

[0124] Here, , , am. is a descending function. am.

[0125] The sample offset for modifying the prediction sample at each sample location (x, y) within the 4x4 subblock of the target block can be calculated using the optical flow calculated for the 4x4 subblock and the gradient values ​​at the sample location (x, y), as in Equation 9. In Equation 9, rnd() means a round-off operation.

[0126]

[0127] The inter prediction unit (124) is a sample offset b(x, y) at the (x, y) location and a prediction sample I within the first reference block and the second reference block. (0) (x,y) and I (1) Using (x,y), the final prediction sample pred(x,y) is generated as in Equation 10.

[0128]

[0129] Here, shift is Max(3, 15 - BitDepth), and O offset is half of the shift as a value for rounding operations.

[0130] As described above, the bidirectional optical flow technique uses the values ​​of prediction samples predicted using the motion information (two motion vectors and two reference pictures) used for bidirectional prediction. Accordingly, the inter prediction unit (544) of the video decoding device can also perform bidirectional optical flow in the same manner as the video encoding device by using the motion information (motion vectors, reference pictures) used for bidirectional prediction received from the video encoding device. No additional signaling of information for the bidirectional optical flow process is required from the video encoding device to the video decoding device.

[0131] The bidirectional optical flow technique described above can also be extended to chroma components. In this case, to reduce computational complexity, the optical flow for chroma components (v x, v y ) can be used as the optical flow for the chrominance component without recalculating the optical flow for the luma component. Therefore, when applying bidirectional optical flow to the chrominance component, only the horizontal and vertical gradients of the chrominance component of each sample need to be calculated.

[0132] The bidirectional optical flow described above requires a significant amount of computation on its own. Furthermore, when used in conjunction with other coding tools, bidirectional optical flow can further increase computational complexity and delay the encoding / decoding process. Furthermore, bidirectional optical flow combined with some coding tools may not contribute to improved coding efficiency. Considering these factors, the execution of bidirectional optical flow may be restricted under certain conditions. Video encoding and decoding devices skip bidirectional optical flow by checking in advance whether predefined conditions are satisfied before executing bidirectional optical flow. The conditions for restricting the execution of bidirectional optical flow are described below. If one or more of the conditions are satisfied, the bidirectional optical flow is skipped and not executed.

[0133] In some embodiments, bidirectional optical flow is not used in conjunction with affine motion prediction, as described below. Since both bidirectional optical flow and affine motion prediction are computationally intensive, combining the two coding tools not only increases computational complexity but also delays the encoding / decoding process. Therefore, if affine motion prediction is used for a target block, bidirectional optical flow is not performed for that block.

[0134] In some other embodiments, bidirectional optical flow may be restricted from being used with inter-intra merge prediction techniques. When inter-intra merge prediction techniques are applied to a target block, bidirectional optical flow for the target block is not applied. Since intra prediction utilizes previously restored samples surrounding the target block, prediction of the target block is possible only after decoding (sample restoration) of all surrounding blocks is completed. Therefore, when applying both inter-intra merge prediction and bidirectional optical flow, the bidirectional optical flow processor must be stopped until intra prediction for the target block is performed after decoding of all surrounding blocks of the target block is completed, which may cause significant delays in the decoding process. Therefore, bidirectional optical flow may not be applied to blocks to which inter-intra merge prediction is applied.

[0135] In some other embodiments, bidirectional optical flow may be restricted from being used in conjunction with local illumination compensation, as described below. For example, bidirectional optical flow is not applied when local illumination compensation is applied.

[0136] In some other embodiments, if the current picture containing the target block or the reference picture referenced by the target block is a picture divided into subpictures, bidirectional optical flow is not applied.

[0137] In some other embodiments, bidirectional optical flow is not performed when weighted bidirectional prediction is performed on a target block and different weights are applied to two reference blocks (a first reference block and a second reference block). That is, when the weights to be applied to the two reference blocks (luma components within the two reference blocks) for predicting the luma component are different, bidirectional optical flow is not performed for the target block. Also, when the weights to be applied to the two reference blocks (chroma components within the two reference blocks) for predicting the chroma component are different, bidirectional optical flow is not performed for the target block. As described above, bidirectional optical flow is based on the assumption that there is little change in sample values ​​between pictures. In contrast, weighted bidirectional prediction assumes that there is a change in sample values ​​between pictures. Therefore, when weighted bidirectional prediction with different weights is applied, bidirectional optical flow for the target block is skipped.

[0138] Alternatively, for bidirectional prediction with different weights applied, bidirectional optical flow can be performed by applying the different weights. The first reference block (I (0) (x,y)) and the second reference block (I (1) When the weights applied to (x,y) are w0 and w1, respectively, I in Equations 5 to 10 (0) (x,y) and I (1) (x,y) is w0I (0) (x,y) and w1I (0) (x,y) is replaced by (x,y). That is, instead of the sample values ​​within the two reference blocks, the sample values ​​multiplied by the weights corresponding to the two reference blocks are used as the optical flow (v x ,v y ), sample offset b(x,y), and final prediction sample pred(x,y) are used to compute.

[0139] III. Affine Motion Prediction

[0140] The inter prediction described so far is motion prediction that reflects a translational motion model. That is, it is a technique for predicting motion in the horizontal direction (x-axis direction) and vertical direction (y-axis direction). However, in reality, various forms of motion, such as rotation, zoom-in, or zoom-out, may exist in addition to translational motion. One aspect of the present disclosure provides affine motion prediction that can cover these various forms of motion.

[0141] Figure 8 is an example diagram for explaining affine motion prediction.

[0142] There are two types of models for affine motion prediction. One model, as shown in Fig. 8(A), uses four parameters, i.e., the motion vectors of two control points, the top-left corner and the top-right corner, of the target block to be encoded. The other model, as shown in Fig. 8(B), uses six parameters, i.e., the motion vectors of three control points, the top-left corner, the top-right corner, and the bottom-left corner of the target block.

[0143] The four-parameter affine model is expressed as in Equation 11. The movement at the sample location (x, y) within the target block can be calculated using Equation 11. Here, the location of the upper left sample of the target block is assumed to be (0,0).

[0144]

[0145] The 6-parameter affine model is expressed as in Equation 12. The movement at the sample location (x, y) within the target block can be calculated by Equation 12.

[0146]

[0147] Here, (mv 0x , mv 0y ) is the motion vector of the upper left control point, (mv 1x , mv 1y ) is the motion vector of the upper right control point, (mv 2x , mv 2y ) is the motion vector of the lower left control point. W is a constant determined from the horizontal length of the target block, and H is a constant determined from the vertical length of the target block.

[0148] Affine motion prediction can be performed using a motion vector calculated through Equation 11 or Equation 12 for each sample within the target block.

[0149] Alternatively, to reduce computational complexity, the process may be performed on a sub-block basis, as illustrated in Figure 9, divided from the target block. For example, the sub-blocks may be 4x4 in size, or 2x2 or 8x8 in size. Below, affine motion prediction is performed on a 4x4 sub-block basis for the target block as an example. This example is provided for convenience of explanation and does not limit the present invention.

[0150] In subblock-level affine motion prediction, the motion vector (affine motion vector) of each subblock is calculated by substituting the center position of each subblock into (x, y) of Equation 11 or 12. Here, the center position may be the actual center point of the subblock or the lower right sample position of the center point. For example, in the case of a 4x4 subblock where the coordinates of the lower left sample are (0,0), the center position of the subblock may be (1.5, 1.5) or (2,2). The predicted block for each subblock is the affine motion vector (mv) of the corresponding subblock. x , mv y ) is generated using.

[0151] Motion vector (mv x , mv y ) can be set to have 1 / 16 sample precision. In this case, the motion vector (mv) calculated through mathematical expression 11 or 12 x , mv y ) can be rounded to 1 / 16 sample units. Meanwhile, adaptive motion vector resolution can be applied in affine motion prediction as in general inter prediction. In this case, the motion vector resolution of the target block, i.e., information about the precision of the motion vector, is signaled for each target block.

[0152] Affine motion prediction can be performed not only on the luma component but also on the chroma component. In the case of the 4:2:0 video format, if affine motion prediction is performed on the luma component in units of 4x4 subblocks, affine motion prediction on the chroma component can be performed in units of 2x2 subblocks. The motion vector (mv) of each subblock of the chroma component x , mv y ) can be derived from the motion vector of the corresponding luminance component. Alternatively, the size of the subblock for affine motion prediction of the chrominance component may be the same as that of the luminance component. If affine motion prediction is performed in units of 4x4 subblocks for the luminance component, affine motion prediction is also performed in units of 4x4 subblocks for the chrominance component. In this case, since a 4x4-sized subblock for the chrominance component corresponds to four 4x4 subblocks for the luminance component, the motion vector (mv) for the subblock of the chrominance component x , mv y ) can be calculated by averaging the motion vectors of four sub-blocks of corresponding luminance components.

[0153] The video encoding device performs intra prediction, inter prediction (translational motion prediction), affine motion prediction, etc., and calculates the RD (rate-distortion) cost to select the optimal prediction method. In order to perform affine motion prediction, the inter prediction unit (124) of the video encoding device determines which type of two types of models to use, and determines two or three control points according to the determined type. The inter prediction unit (1214) uses the motion vectors of the control points to generate a motion vector (mv) for each of the 4x4 sub-blocks in the target block. x , mv y ) is calculated. And, the motion vector (mv) of each sub-block x , mv y ) is used to perform motion compensation within a reference picture in units of subblocks, thereby generating a prediction block for each subblock within the target block.

[0154] The entropy encoding unit (155) of the video encoding device encodes affine-related syntax elements including a flag indicating whether affine motion prediction is applied to a target block, type information indicating a type of an affine model, and motion information indicating a motion vector of each control point, and transmits the encoded affine-related syntax elements to the video decoding device. The type information and the motion information of the control points may be signaled when affine motion prediction is performed, and the motion vectors of the control points may be signaled in a number determined according to the type information. In addition, when adaptive motion vector resolution is applied, motion vector resolution information for the affine motion vector of the target block is signaled.

[0155] The video decoding device determines the type of the affine model and control point motion vectors using the signaled syntaxes, and uses Equation 11 or 12 to derive the motion vector (mv) for each 4x4 sub-block in the target block. x , mv y) is calculated. If the motion vector resolution information for the affine motion vector of the target block is signaled, the motion vector (mv x , mv y ) is corrected to the precision identified by the motion vector resolution information through operations such as rounding.

[0156] The video decoding device generates a motion vector (mv) for each subblock x , mv y ) to perform motion compensation within the reference picture, thereby generating a prediction block for each sub-block.

[0157] To reduce the amount of bits required to encode the motion vectors of control points, a method similar to the general intra prediction (translational motion prediction) described above can be applied.

[0158] As an example, in the case of merge mode, the inter prediction unit (124) of the video encoding device derives the motion vector of each control point from the surrounding blocks of the target block. The inter prediction unit (124) generates a merge candidate list by deriving a predefined number of merge candidates from, for example, the surrounding samples (L, BL, A, AR, AL) of the target block illustrated in FIG. 4. Each of the merge candidates included in the list corresponds to a pair of motion vectors of two or three control points.

[0159] First, the inter prediction unit (124) derives merge candidates from the control point motion vectors of the surrounding blocks predicted in the affine mode among the surrounding blocks. In some embodiments, the number of merge candidates derived from the surrounding blocks predicted in the affine mode may be limited. For example, the inter prediction unit (124) may derive a total of two merge candidates, one from L and BL, and one from A, AR, and AL, from the surrounding blocks predicted in the affine mode. The priority may be in the order of L and BL, and then in the order of A, AR, and AL.

[0160] Meanwhile, when the total number of merge candidates is three or more, the inter prediction unit (124) can derive the number of merge candidates equal to the insufficient number from the translational motion vector of the surrounding blocks.

[0161] Figure 10 is an example diagram illustrating a method for deriving merge candidates for affine motion prediction from translation motion vectors of surrounding blocks.

[0162] The inter prediction unit (124) derives one control point motion vector CPMV1, CPMV2, and CPMV3 from the neighboring block group {B2, B3, A2}, the neighboring block group {B1, B0}, and the neighboring block group {A1, A0}, respectively. As an example, the priorities within each neighboring block group may be the order of B2, B3, and A2, the order of B1, B0, and the order of A1, A0. In addition, another control point motion vector CPMV4 is derived from the collocated block T within the reference picture. The inter prediction unit (124) generates the missing number of merge candidates by combining two or three of the four control point motion vectors. The priorities of the combinations are as follows. The elements within each group are listed in the order of the upper left, upper right, and lower left control point motion vectors.

[0163] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4},

[0164] {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0165] The inter prediction unit (124) selects a merge candidate from the merge candidate list and performs affine motion prediction on the target block. If the selected candidate consists of two control point motion vectors, affine motion prediction is performed using a 4-parameter model. On the other hand, if the selected candidate consists of three control point motion vectors, affine motion prediction is performed using a 6-parameter model. The entropy encoding unit (155) of the video encoding device encodes index information indicating a selected merge candidate among the merge candidates in the merge candidate list and signals the encoded information to the video decoding device.

[0166] The entropy decoding unit (510) of the video decoding device decodes the index information signaled from the video encoding device. The inter prediction unit (544) of the video decoding device constructs a merge candidate list in the same manner as the video encoding device, and performs affine motion prediction using control point motion vectors corresponding to the merge candidates indicated by the index information.

[0167] As another example, in the AMVP mode, the inter prediction unit (124) of the video encoding device determines the type of the affine model and control point motion vectors for the target block. Then, the differential motion vector (motion vector difference), which is the difference between the actual control point motion vectors of the target block and the predicted motion vectors of each control point, is calculated, and the differential motion vector for each control point is encoded and transmitted. To this end, the inter prediction unit (124) of the video encoding device constructs a predefined number of affine AMVP lists. When the target block is of the 4-parameter type, the candidates included in the list each consist of a pair of two control point motion vectors. On the other hand, when the target block is of the 6-parameter type, the candidates included in the list each consist of a pair of three control point motion vectors. The affine AMVP list can be derived using the control point motion vectors or the translational motion vectors of the neighboring blocks, similarly to the method of constructing the merge candidate list described above.

[0168] However, in order to derive candidates to be included in the affine AMVP list, there may be a constraint that only neighboring blocks that refer to the same reference picture as the target block among the neighboring blocks in Fig. 4 are considered.

[0169] In addition, in AMVP mode, the affine model type of the target block must be considered. If the affine model type of the target block is a 4-parameter type, the video encoding device derives two control point motion vectors (upper-left and upper-right control point motion vectors of the target block) using the affine model of the neighboring block. If the affine model type of the target block is a 6-parameter type, the video encoding device derives three control point motion vectors (upper-left, upper-right, and lower-left control point motion vectors of the target block) using the affine model of the neighboring block.

[0170] When the surrounding block is of the 4-parameter type, two or three control point motion vectors are predicted according to the affine model type of the target block using two control point motion vectors of the surrounding block. For example, the affine model of the surrounding block expressed by Equation 11 can be used. In Equation 11, (mv 0x , mv 0y ) and (mv 1x , mv 1y ) are replaced by the motion vectors of the top-left corner and top-right corner control points of the surrounding blocks, respectively. W is replaced by the horizontal length of the surrounding block. The predicted motion vector for each control point of the target block can be derived by inputting the difference between the position of the corresponding control point of the target block and the position of the top-left corner of the surrounding block in (x, y).

[0171] When the surrounding block is of the 6-parameter type, two or three control point motion vectors are predicted according to the affine model type of the target block using the three control point motion vectors of the surrounding block. For example, the affine model of the surrounding block expressed by Equation 12 can be used. In Equation 12, (mv 0x , mv 0y ), (mv 1x , mv 1y ), and (mv 2x , mv 2y ) are replaced by the motion vectors of the control points of the upper left, upper right, and lower left corners of the surrounding blocks, respectively. W and H are replaced by the width and height of the surrounding blocks, respectively. The predicted motion vector for each control point of the target block can be derived by inputting the difference between the position of the corresponding control point of the target block and the position of the upper left corner of the surrounding block in (x, y).

[0172] The inter prediction unit (124) of the video encoding device selects one candidate from the affine AMVP list and generates a differential motion vector between the motion vectors of each actual control point and the predicted motion vector of the control point corresponding to the selected candidate. The entropy encoding unit (155) of the video encoding device encodes type information indicating the affine model type of the target block, index information indicating a candidate selected from among the candidates in the affine AMVP list, and a differential motion vector corresponding to each control point, and transmits the encoded information to the video decoding device.

[0173] The inter prediction unit (544) of the video decoding device determines the affine model type using the information signaled from the video encoding device and generates a differential motion vector of each control point. Then, in the same manner as the video encoding device, an affine AMVP list is generated and a candidate indicated by the signaled index information within the affine AMVP list is selected. The inter prediction unit (544) of the video decoding device adds the differential motion vector corresponding to the predicted motion vector of each control point of the selected candidate to calculate the motion vector of each control point.

[0174] IV. Sample-by-sample adjustment of affine motion prediction samples

[0175] Subblock-level affine motion prediction for a target block has been described above. Another aspect of the present disclosure relates to adjusting the sample values ​​of prediction samples generated from subblock-level affine motion prediction on a sample-by-sample basis. Motion according to the position of each sample within a subblock, which is the unit where affine motion prediction is performed, is additionally compensated.

[0176] When the sample values ​​within a sub-block generated as a result of sub-block-level affine motion prediction for the target block are I(x, y), the image encoding device calculates the horizontal and vertical gradients g at each sample location. x (i,j) and g y(i,j) are calculated. Equation 13 can be used for gradient calculation.

[0177]

[0178] The sample offset ΔI(i,j) for adjusting the prediction sample is calculated using the following mathematical formula.

[0179]

[0180] Here, Δmv(i,j) means the difference between the affine motion vector at sample (i,j) and the affine motion vector at the center position of the subblock, i.e., the motion offset, and can be calculated by applying mathematical expression 11 or mathematical expression 12 depending on the affine model type of the target block. That is, it can be calculated by subtracting the motion vector when the subblock center position is input to (x,y) from the motion vector when (x,y)=(i,j) of mathematical expression 11 or 12 is input. In other words, Δmv(i,j) is calculated by replacing (x,y) of mathematical expressions 11 and 12 with the offsets in the horizontal and vertical directions from the subblock center position to the sample position (i,j), and adding the last term "+ mv 0x "Wow "+ mv 0y ", can be calculated through a mathematical formula. The center position may be the actual center point of the sub-block or the lower right sample position of the center point.

[0181] Meanwhile, the motion vector of each control point of the target block used for calculating Δmv(i,j) and the difference between the sample position (i,j) and the subblock center position are the same for all subblocks. Therefore, the Δmv(i,j) values ​​are calculated only for one subblock, e.g., the first subblock, and can be reused for other subblocks.

[0182] The technique of the present disclosure is based on the assumption that the object moves at a constant speed and the variation of the sample values ​​is smooth. Therefore, the x component (Δmv x ) and y component (Δmv y ) are multiplied by the sample gradient values ​​in the horizontal and vertical directions, respectively, to obtain the sample change in the horizontal direction and the sample change in the vertical direction, and the two change amounts are added to calculate the sample offset ΔI(i,j).

[0183] Finally, the values ​​of the predicted samples are calculated as follows:

[0184]

[0185] When sample-by-sample adjustment is applied to affine motion prediction samples, the inter prediction unit (124) of the video encoding device and the inter prediction unit (544) of the video decoding device perform the process as described above to modify the sample values ​​of the prediction samples generated through affine motion prediction. The gradient values ​​are derived from the prediction samples generated through affine motion prediction, and Δmv(i,j) is derived from the control point motion vectors of the target block. Therefore, no additional information signaling for the process of the present technique is required from the video encoding device to the video decoding device.

[0186] The sample-by-sample adjustment technique for affine motion prediction samples described above is applied to the luma component. Additionally, this technique can be extended to the chrominance component. In the extended application, Δmv for the chrominance component x and Δmv y Δmv was calculated for the luma component without separate operation. x and Δmv ycan be used as is. That is, the image encoding device and the image decoding device calculate gradient values ​​for the predicted samples of the chrominance component generated through affine motion prediction. And the gradient values ​​of the chrominance component and the Δmv calculated for the luma component x and Δmv y By substituting into Equations 14 and 15, the predicted samples of the chrominance components generated through affine motion prediction can be adjusted.

[0187] In the case of bidirectional prediction, mathematical expressions 14 and 15 are applied to each of the two reference pictures. The video encoding device and the video decoding device perform sample-by-sample adjustment on the affine prediction samples for each of the reference pictures of the reference picture list 0 and the reference pictures of the reference picture list 1, thereby generating two reference blocks. The two reference blocks are generated by mathematical expressions 14 and 15. The final prediction block for the target block can be generated by an average operation between the two reference blocks. When the bit depth is 10, the process of generating the final prediction block can be expressed by a mathematical expression as follows.

[0188]

[0189] In mathematical expression 16, “I0(i,j) + clip3( )” are prediction samples within a reference block generated by applying this technique to a reference picture of reference picture list 0, and “I1(i,j) + clip3( )” are prediction samples within a reference block generated by applying this technique to a reference picture of reference picture list 1.

[0190] Meanwhile, in order to prevent delay due to execution of the sample-by-sample adjustment technique for affine motion prediction samples, it is possible to preemptively determine whether the application of this technique is appropriate before executing this technique and thereby exclude the execution of this technique.

[0191] As an example, a video encoding device can determine whether to apply the present technique in units of a predefined image area, and signal a flag indicating whether to apply the present technique to a video decoding device. Here, the predefined image area can be a sequence of pictures, a picture, or a slice. When whether to apply the present technique is determined in units of a sequence, a picture, or a slice, the flag can be included in a header of a sequence (SPS), a header of a picture (PPS), or a slice header, respectively. The video decoding device can extract the corresponding flag included in a bitstream, and determine whether the present technique is applied to blocks within an image area corresponding to the extracted flag.

[0192] As another example, it is possible to determine in advance whether to apply this technique to a target block based on the control point motion vectors of the target block. If the values ​​of the control point motion vectors of the target block are all the same, this technique is not applied. If the affine type of the target block is a 4-parameter model, this technique is not executed if the control point motion vectors of the upper left and upper right are the same. If it is a 6-parameter model, this technique is not executed if the control point motion vectors of the upper left, upper right, and lower left are the same.

[0193] As another example, the application of this technique can be determined based on the angle between the control point motion vectors. For example, if the angle between the control point motion vectors is obtuse (i.e., if the inner product between the vectors is negative), this technique may not be applied. Alternatively, if the control point motion vectors are at an acute angle (i.e., if the inner product between the vectors is positive), this technique may be restricted from being applied.

[0194] As another example, the application of this technique may be excluded when the control point motion vectors of the target block refer to reference pictures in different reference picture lists.

[0195] As another example, to minimize delay issues, this technique may be restricted from use with inter-intra merge prediction techniques, as is the case with bidirectional optical flow. Furthermore, this technique may be excluded when local illumination compensation, as described below, is applied or when bidirectional prediction is used.

[0196] As another example, in unidirectional or bidirectional prediction, this technique does not work if the reference picture referred to by the target block is not a short-term reference picture.

[0197] As another example, this technique is not applicable when the current picture containing the target block or the reference picture referenced by the target block is a picture divided into subpictures.

[0198] As another example, if bidirectional prediction is performed and different weights are applied to the two predicted blocks (the first reference block and the second reference block) generated by affine motion prediction, the present technique is not implemented. That is, if the luma weights to be applied to the two reference blocks (the luma components within the two reference blocks) for predicting the luma component are different, the present technique is not applied to the target block. Also, if the chroma weights to be applied to the two reference blocks (the chroma components within the two reference blocks) for predicting the chroma component are different, the present technique is not applied to the target block.

[0199] Alternatively, in the case of bidirectional prediction where different weights are applied, the present technique can be implemented by applying the different weights. For example, a video encoding device or a video decoding device generates two reference blocks by performing sample-wise adjustment on affine prediction samples for each of the reference pictures in reference picture list 0 and the reference pictures in reference picture list 1. Thereafter, the weights are applied to the two reference blocks to generate a final prediction block.

[0200] Meanwhile, even after the sample-wise adjustment technique for affine motion prediction samples is executed, Δmv x and Δmv y If the value is less than a predetermined threshold, the technique may be stopped from executing.

[0201] V. Local Illumination Compensation

[0202] Local illumination compensation technology is an encoding technology that compensates for variations using a linear model regarding illumination changes between a current block and a predicted block. The inter prediction unit (124) of an image encoding device determines a reference block within a reference picture using a motion vector (translational motion vector) of a target block, and obtains parameters of a linear model for illumination compensation using pre-restored samples around the reference block (upper and left) and pre-restored samples around the target block (upper and left).

[0203] When the restored samples around the reference block are x and the corresponding restored samples around the target block are y, parameters A and b that minimize the sum of the squares of the differences between y and (Ax + b) are derived as in Equation 17.

[0204]

[0205] The final prediction samples are generated by applying the weight of A and the offset of b to the samples in the prediction block (reference block) generated by the motion vector of the target block, as in Equation 18. In Equation 18, pred[x][y] is the prediction sample at the (x, y) position generated by the motion vector of the target block, and pred LIC [x][y] are the final predicted samples after lighting compensation.

[0206]

[0207] Another aspect of the present disclosure relates to techniques for combining these lighting compensation techniques with affine motion prediction.

[0208] As described above, when applying subblock-level affine motion prediction to a target block, motion vectors are generated for each subblock. If lighting compensation parameters are derived and lighting compensation is performed for each subblock using the corresponding motion vector, not only does the computational complexity increase, but a serious delay problem also occurs. This is because the reconstructed samples within each subblock are required for lighting compensation of the next subblock, so the lighting compensation process for the next subblock must be postponed until the adjacent subblock is reconstructed (i.e., until both the prediction block and the residual block for the subblock are reconstructed). The present disclosure is directed to improving these problems.

[0209] Figure 11 illustrates various examples of determining the location of a reference block to derive lighting compensation parameters according to one embodiment of the present disclosure. This embodiment derives a single lighting compensation parameter set (A, b) for a target block and applies the same parameters to all subblocks within the target block. In other words, the entire target block is modified through a single lighting compensation parameter.

[0210] As shown in Fig. 11(A), the inter prediction unit (124) of the video encoding device can determine the position of the reference block within the reference picture using the affine motion vector of the sub-block located at the upper left within the target block or the motion vector of the upper left control point of the target block. The pre-restored samples around the determined reference block are used for parameter derivation. Alternatively, as shown in Fig. 11(B), the position of the reference block can be determined using the affine motion vector of the central sub-block within the target block. Once the position of the reference block is determined, the illumination compensation parameter is derived using the pre-restored samples adjacent to the upper and left sides of the reference block and the corresponding pre-restored samples adjacent to the upper and left sides of the target block.

[0211] As another example, multiple sub-blocks within a target block may be used. As shown in Fig. 11(C), the inter prediction unit (124) determines reference sub-blocks corresponding to each of the boundary sub-blocks using affine motion vectors of the sub-blocks (boundary sub-blocks) located at the boundary within the target block. Samples for deriving the illumination compensation parameter are extracted from the reconstructed samples adjacent to the boundary sub-blocks within the target block and the corresponding reference sub-blocks, respectively. For the sub-blocks located at the upper boundary within the target block and the corresponding reference sub-blocks, the samples are extracted from the reconstructed samples adjacent to the upper boundary, and for the sub-blocks located at the left boundary within the target block and the corresponding reference sub-blocks, the samples are extracted from the reconstructed samples adjacent to the left boundary.

[0212]

[0213] One or more of the coding tools described above may be used to improve the prediction performance of inter prediction, and the application of some coding tools may have limitations that exclude the application of other coding tools to address issues such as complexity or delay.

[0214] Meanwhile, both sample-wise adjustment for affine prediction samples and bidirectional optical flow are techniques for modifying predicted samples after prediction, and the gradients of the samples are used for modification. Therefore, to reduce computational complexity and hardware complexity, the equation for bidirectional optical flow can be modified into the form of an equation for sample-wise adjustment for affine prediction samples in bidirectional prediction. Alternatively, conversely, the equation for sample-wise adjustment for affine prediction samples in bidirectional prediction can also be expressed in the form of an equation for bidirectional optical flow.

[0215] By substituting Equation 9 into Equation 10, the equation for obtaining the final prediction sample in bidirectional optical flow can be expressed as follows.

[0216]

[0217] If mathematical expression 19 is expressed in the form of mathematical expression 16, it is as follows.

[0218]

[0219] That is, the final prediction sample to which the bidirectional optical flow is applied can be calculated using Equation 20 instead of Equation 19. Since Equation 20 is expressed in a form similar to the sample-by-sample adjustment technique for affine prediction samples, there is no need to design separate hardware for mathematical calculation according to the bidirectional optical flow technique. In addition, since Equation 20 is expressed in the form of the average of the prediction block from the reference picture of the reference picture list 0 and the prediction block from the reference picture of the reference picture list 1, the hardware design is simplified.

[0220] Additionally, the motion offset (optical flow) of the bidirectional optical flow technique (v x , v y ) motion vector precision and motion offset (Δmv) of the sample-by-sample adjustment technique for affine prediction samples x , Δmv y ) can also match the motion vector precision of each other. For example, the motion offsets of both techniques can be expressed with a precision of 1 / 32 subsample units. It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to particularly emphasize their implementation independence.

[0221] Meanwhile, the various functions or methods described in the present disclosure may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

[0222] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0223] CROSS-REFERENCE TO RELATED APPLICATION

[0224] This patent application claims priority to Korean Patent Application No. 10-2019-0123491, filed in Korea on October 6, 2019, Korean Patent Application No. 10-2019-0158564, filed in Korea on December 2, 2019, and Korean Patent Application No. 10-2020-0123619, filed in Korea on September 24, 2020, the entire contents of which are incorporated herein by reference.

Claims

1. In a video decoding device for predicting a target block in a current picture to be decoded, By decoding the bitstream, the first and second reference pictures and the first and second motion vectors for bi-prediction are determined, Generating a first reference block from the first reference picture referenced by the first motion vector and generating a second reference block from the second reference picture referenced by the second motion vector, A prediction unit is included that predicts the target block using the first and second reference blocks, The above prediction part, A first coding tool for generating a prediction block of the target block by performing a bi-directional optical flow process using the first and second reference blocks, When the luma weights assigned to the first and second reference pictures are different from each other for predicting the luma component of the target block, the first coding tool is not executed. An image decoding device, characterized in that when the chroma weights assigned to the first and second reference pictures are different from each other for predicting the chroma component of the target block, the first coding tool is not executed.

2. In paragraph 1, The above prediction part, When the luma weights assigned to the first and second reference pictures are different from each other, the luma component of the target block is predicted by applying the luma weights to the first reference block and the second reference block, An image decoding device characterized in that the chroma components of the target block are predicted by applying the chroma weights to the first reference block and the second reference block when the chroma weights assigned to the first and second reference pictures are different from each other.

3. In paragraph 1, When the above first coding tool is executed, the prediction unit, for each of the sub-blocks divided from the target block, Generating first horizontal gradients and first vertical gradients for each of the luma samples of the sub-blocks in the first reference block corresponding to the sub-blocks of the target block, Generating second horizontal gradients and second vertical gradients for each of the luma samples of the sub-blocks in the second reference block corresponding to the sub-blocks of the target block, Computing a motion offset corresponding to a sub-block of the target block using the first and second horizontal gradients and the first and second vertical gradients for the luma samples, An image decoding device characterized by predicting luma samples in a subblock of the target block using values ​​of luma samples of a subblock of the first reference block, values ​​of samples of a subblock of the second reference block, and the motion offset.

4. In paragraph 3, The above prediction part, Computing a sample offset of a luma sample position by using a difference between a first and a second horizontal gradient corresponding to a luma sample position within a subblock of the target block, a difference between a first and a second vertical gradient corresponding to the luma sample position, and the motion offset corresponding to the subblock of the target block, By predicting a luma sample at the luma sample location using the luma sample values ​​in the first and second reference blocks corresponding to the luma sample location and the sample offset of the luma sample location, An image decoding device characterized by predicting luma samples within a subblock of the target block.

5. In paragraph 4, The above prediction part, Generating third horizontal gradients and third vertical gradients for each of the chroma samples of the sub-blocks within the first reference block, Generating fourth horizontal gradients and fourth vertical gradients for each of the chroma samples of the sub-blocks within the second reference block, Computing a sample offset for a chroma sample position using a difference between a third and a fourth horizontal gradient corresponding to a chroma sample position within a subblock of the target block, a difference between a third and a fourth vertical gradient corresponding to the chroma sample position, and the motion offset corresponding to the subblock of the target block, An image decoding device characterized in that the chroma sample of the chroma sample position is predicted by using the chroma sample values ​​in the first and second reference blocks corresponding to the chroma sample position and the sample offset of the chroma sample position.

6. In paragraph 1, The above prediction part, A second coding tool is included that generates an inter prediction block using the first reference block and the second reference block, intra-predicts the target block to generate an intra prediction block, and weighted-averages the inter prediction block and the intra prediction block to generate a prediction block of the target block. An image decoding device, characterized in that execution of the second coding tool restricts execution of the first coding tool.

7. In paragraph 6, When the above second coding tool is executed, the prediction unit, An image decoding device characterized by generating the intra prediction block using the PLANAR mode among a plurality of intra prediction modes.

8. In paragraph 6, An image decoding device, characterized in that the weights for the weighted average are determined by the number of intra-predicted blocks among surrounding blocks including the left and upper blocks of the target block.

9. In a video encoding device for inter-predicting a target block within a current picture to be encoded, Determine the first motion vector and the second motion vector for bi-prediction, Generating a first reference block from a first reference picture referenced by the first motion vector and generating a second reference block from a second reference picture referenced by the second motion vector, A prediction unit is included that predicts the target block using the first and second reference blocks, The above prediction part, A first coding tool for generating a prediction block of the target block by performing a bi-directional optical flow process using the first and second reference blocks, When the luma weights assigned to the first and second reference pictures are different from each other for predicting the luma component of the target block, the first coding tool is not executed. An image encoding device, characterized in that the first coding tool is not executed when the chroma weights assigned to the first and second reference pictures are different from each other for predicting the chroma component of the target block.

10. In paragraph 9, When the above first coding tool is executed, the prediction unit, for each of the sub-blocks divided from the target block, Generating first horizontal gradients and first vertical gradients for each of the luma samples of the sub-blocks in the first reference block corresponding to the sub-blocks of the target block, Generating second horizontal gradients and second vertical gradients for each of the luma samples of the sub-blocks in the second reference block corresponding to the sub-blocks of the target block, Computing a motion offset corresponding to a sub-block of the target block using the first and second horizontal gradients and the first and second vertical gradients for the luma samples, An image encoding device characterized in that the luma samples in the subblocks of the target block are predicted using the values ​​of the luma samples of the subblocks of the first reference block, the values ​​of the samples of the subblocks of the second reference block, and the motion offset.

11. In paragraph 10, The above prediction part, Computing a sample offset of a luma sample position by using a difference between a first and a second horizontal gradient corresponding to a luma sample position within a subblock of the target block, a difference between a first and a second vertical gradient corresponding to the luma sample position, and the motion offset corresponding to the subblock of the target block, By predicting a luma sample at the luma sample location using the luma sample values ​​in the first and second reference blocks corresponding to the luma sample location and the sample offset of the luma sample location, An image encoding device characterized by predicting luma samples within a subblock of the target block.

12. In paragraph 11, The above prediction part, Generating third horizontal gradients and third vertical gradients for each of the chroma samples of the sub-blocks within the first reference block, Generating fourth horizontal gradients and fourth vertical gradients for each of the chroma samples of the sub-blocks within the second reference block, Computing a sample offset of a chroma sample position using a difference between a third and a fourth horizontal gradient corresponding to a chroma sample position in a subblock of the target block, a difference between a third and a fourth vertical gradient corresponding to the sample position, and the motion offset corresponding to the subblock of the target block, An image encoding device characterized in that the chroma sample of the chroma sample position is predicted by using the chroma sample values ​​in the first and second reference blocks corresponding to the chroma sample position and the sample offset of the chroma sample position.

13. In paragraph 9, The above prediction part, A second coding tool is included that generates an inter prediction block using the first reference block and the second reference block, intra-predicts the target block to generate an intra prediction block, and weighted-averages the inter prediction block and the intra prediction block to generate a prediction block of the target block. An image encoding device, characterized in that execution of the second coding tool restricts execution of the first coding tool.

14. In paragraph 6, An image encoding device, characterized in that the weights for the weighted average are determined by the number of intra-predicted blocks among the surrounding blocks including the left and upper blocks of the target block.

15. A method for predicting a target block within a current picture, A step of determining a first motion vector and a second motion vector for bi-prediction; A step of generating a first reference block from a first reference picture referenced by the first motion vector and generating a second reference block from a second reference picture referenced by the second motion vector; and A step of predicting the target block using the first and second reference blocks, The step of predicting the above target block is: A step of executing a first coding tool for generating a prediction block of the target block by performing a bi-directional optical flow process using the first and second reference blocks, When the luma weights assigned to the first and second reference pictures are different from each other for predicting the luma component of the target block, the first coding tool is not executed. A method characterized in that the first coding tool is not executed when the chroma weights assigned to the first and second reference pictures are different from each other for predicting the chroma component of the target block.

Images