Method and device for encoding and decoding image by using inter-prediction mode

Abstract

Disclosed are a method and device for encoding and decoding an image by using block-based motion prediction and sub-block-based motion prediction. According to the disclosed method, the current block is encoded / decoded by using block-based motion information and sub-block-based motion information.

Description

Method and device for encoding and decoding images using inter prediction mode

[0001] The present disclosure relates to a video encoding / decoding method, device, and recording medium for storing a bitstream, and more particularly, to a video encoding and decoding method and device using an inter prediction mode.

[0002] The content described below merely provides background information related to the present embodiment and does not constitute prior art.

[0003] Since video data has a large amount of data compared to voice data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.

[0004] Therefore, when storing or transmitting video data, the encoder compresses the video data and stores or transmits it, and the decoder receives the compressed video data, decompresses it, and plays it back. These video compression technologies include H.264 / AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves encoding efficiency by about 30% compared to HEVC.

[0005] However, as the size, resolution, and frame rate of images are gradually increasing, and the amount of data that needs to be encoded is also increasing, a new compression technology that has better encoding efficiency and better image quality improvement than existing compression technologies is required.

[0006] In particular, the demand for high-resolution, high-quality images, such as HD (High Definition) images and UHD (Ultra High Definition) images, has been increasing in various fields. As image data becomes higher in resolution and quality, the amount of information or bits transmitted increases relatively compared to existing image data. This increase in the amount of information or bits transmitted leads to an increase in transmission and storage costs. In addition, interest in and demand for immersive media, such as VR (Virtual Reality), AR (Artificial Reality) content, and holograms, has been increasing recently, and the broadcasting of videos / images with different image characteristics from real images, such as game images, is increasing. Accordingly, a highly efficient image compression technology is required to effectively compress, transmit, store, and play information of high-resolution, high-quality images with various characteristics as described above.

[0007] The present disclosure provides a method and device for image encoding or decoding for efficiently encoding or decoding an image and improving objective and subjective image quality of a restored image, and a recording medium for storing a bitstream generated by the image encoding method / device.

[0008] One aspect of the present disclosure provides an image decoding method, comprising: a step of deriving block-based motion information of a current block; a step of generating a first prediction block of the current block using the block-based motion information; a step of deriving sub-block-based motion information of the current block; a step of generating a second prediction block of the current block by predicting the current block in sub-block units using the sub-block-based motion information; and a step of generating a prediction block of the current block by performing a weighted sum of the first prediction block and the second prediction block.

[0009] One aspect of the present disclosure provides a video encoding method, comprising: a step of deriving block-based motion information of a current block; a step of generating a first prediction block of the current block using the block-based motion information; a step of deriving sub-block-based motion information of the current block; a step of generating a second prediction block of the current block by predicting the current block in sub-block units using the sub-block-based motion information; and a step of generating a prediction block of the current block by performing a weighted sum of the first prediction block and the second prediction block.

[0010] One aspect of the present disclosure provides a non-transitory computer-readable recording medium storing a bitstream generated by an image encoding device. The bitstream is generated by an image encoding method. The image encoding method includes the steps of: deriving block-based motion information of a current block; generating a first prediction block of the current block using the block-based motion information; deriving sub-block-based motion information of the current block; generating a second prediction block of the current block by predicting the current block in sub-block units using the sub-block-based motion information; and generating a prediction block of the current block by performing a weighted sum of the first prediction block and the second prediction block.

[0011] One aspect of the present disclosure provides a method, comprising: generating a bitstream for an image, the method being performed by at least one processor; and transmitting data including the bitstream. The step of generating the bitstream includes: deriving block-based motion information of a current block; generating a first prediction block of the current block using the block-based motion information; deriving sub-block-based motion information of the current block; generating a second prediction block of the current block by predicting the current block in sub-block units using the sub-block-based motion information; and generating a prediction block of the current block by performing a weighted sum of the first prediction block and the second prediction block.

[0012] According to the present disclosure, a video encoding / decoding device can efficiently encode / decode an image and improve the objective / subjective image quality of a restored image by encoding and decoding a current block using a block-based inter-prediction mode and a sub-block-based inter-prediction mode.

[0013] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure.

[0014] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.

[0015] FIGS. 3A and 3B are diagrams illustrating multiple intra prediction modes, including wide-angle intra prediction modes.

[0016] Figure 4 is an example diagram of the surrounding blocks of the current block.

[0017] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure.

[0018] FIG. 6 is a flowchart of a method for predicting a current block according to one embodiment of the present disclosure.

[0019] FIG. 7 is a diagram for explaining prediction of SbTMVP mode according to one embodiment of the present disclosure.

[0020] FIG. 8A and FIG. 8B are diagrams for explaining affine parameter models according to one embodiment of the present disclosure.

[0021] Figure 9 is an example diagram illustrating a method for deriving combinational affine merge candidates for affine motion prediction.

[0022] FIG. 10 is a diagram for explaining prediction of AMVP-subblock based merge mode according to one embodiment of the present disclosure.

[0023] FIG. 11 is a flowchart of a method for generating a sub-block-based candidate list according to one embodiment of the present disclosure.

[0024] FIG. 12 is a diagram for explaining a sub-block-based template matching cost according to one embodiment of the present disclosure.

[0025] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.

[0026] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image encoding device and its subcomponents will be described with reference to the illustration in FIG. 1.

[0027] The video encoding device may be configured to include a picture segmentation unit (110), a prediction unit (120), a subtractor (130), a transformation unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse transformation unit (165), an adder (170), a loop filter unit (180), and a memory (190).

[0028] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0029] A single image (video) is composed of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). Each CTU is then divided into one or more Coding Units (CUs) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks within a single slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). And, information commonly referenced by one or more SPS is encoded in a Video Parameter Set (VPS). In addition, information commonly applied to one tile or tile group may be encoded as syntax of a tile or tile group header. Syntaxes included in an SPS, PPS, slice header, tile or tile group header may be referred to as high level syntax.

[0030] The picture segmentation unit (110) determines the size of the CTU. Information about the size of the CTU (CTU size) is encoded as the syntax of SPS or PPS and transmitted to the image decoding device.

[0031] The picture segmentation unit (110) divides each picture constituting an image into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a CU, which is a basic unit of encoding.

[0032] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, or a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that mixes two or more of the QT structures, BT structures, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be combined and referred to as a MTT (Multiple-Type Tree).

[0033] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT structure.

[0034] As illustrated in FIG. 2, a CTU may first be split into a QT structure. The quadtree splitting may be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in the QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoding unit (155) and signaled to the image decoding device. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further split into one or more of the BT structure or the TT structure. There may be multiple splitting directions in the BT structure and / or the TT structure. For example, there may be two directions in which the block of the corresponding node is split horizontally and two directions in which the block is split vertically. As illustrated in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (Binary or Ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0035] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the CU split flag (split_cu_flag) value indicates that the node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is a basic unit of encoding. If the CU split flag (split_cu_flag) value indicates that the node is split, the video encoding device starts encoding from the first flag in the above-described manner.

[0036] As another example of a tree structure, when QTBT is used, there may be two types: a type that horizontally splits the block of the corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that vertically splits it (i.e., symmetric vertical splitting). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating the type of split are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may additionally be a type that splits the block of the corresponding node into two blocks of an asymmetrical shape. The asymmetric shape may include a shape that splits the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that splits the block of the corresponding node in a diagonal direction.

[0037] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." Depending on the QTBTTT partitioning employed, the current block may be rectangular as well as square.

[0038] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

[0039] In general, each current block within a picture can be predictively coded. Prediction of the current block can typically be performed using either intra-prediction (using data from the picture containing the current block) or inter-prediction (using data from a picture coded before the picture containing the current block). Inter-prediction encompasses both unidirectional and bidirectional prediction.

[0040] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in Fig. 3a, the multiple intra prediction modes may include two non-directional modes including the Planar mode and the DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently depending on each prediction mode.

[0041] For efficient directional prediction for a rectangular current block, directional modes (intra prediction modes 67 to 80 and -1 to -14) indicated by dotted arrows in Fig. 3b may be additionally used. These may be referred to as "wide-angle intra-prediction modes." In Fig. 3b, the arrows point to corresponding reference samples used for prediction, and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide-angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. At this time, among the wide-angle intra-prediction modes, some wide-angle intra-prediction modes available for the current block may be determined based on the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes (intra prediction modes 67 to 80) having an angle less than 45 degrees are available when the current block is a rectangular shape whose height is smaller than its width, and wide-angle intra prediction modes (intra prediction modes -1 to -14) having an angle greater than -135 degrees are available when the current block is a rectangular shape whose width is larger than its height.

[0042] The intra prediction unit (122) can determine the intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit (122) can encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra prediction unit (122) can calculate bit-rate distortion values ​​using rate-distortion analysis for multiple tested intra prediction modes and select the intra prediction mode with the best bit-rate distortion characteristics among the tested modes.

[0043] The intra prediction unit (122) selects one intra prediction mode from among multiple intra prediction modes and predicts the current block using surrounding pixels (reference pixels) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0044] The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for a block most similar to the current block within reference pictures that were encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture is generated. Generally, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma component and the chroma component. The motion information including information on the reference picture used to predict the current block and information on the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0045] The inter prediction unit (124) may perform interpolation on a reference picture or a reference block to improve prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed on the interpolated reference picture, the motion vector can be expressed up to a precision in decimal units rather than a precision in integer sample units. The precision or resolution of the motion vector can be set differently for each target region to be encoded, such as a slice, tile, CTU, CU, etc. When such adaptive motion vector resolution (AMVR) is applied, information on the motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on the motion vector resolution applied to each CU is signaled. Information on the motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.

[0046] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors indicating the block position most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture to generate a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged or weighted averaged to generate a prediction block for the current block. Then, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is transmitted to the entropy encoding unit (155). Here, reference picture list 0 may be composed of pictures that are before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures that are after the current picture in display order among the restored pictures. However, this is not necessarily limited to this, and restored pictures that are after the current picture in display order may be additionally included in reference picture list 0, and conversely, restored pictures that are before the current picture may be additionally included in reference picture list 1.

[0047] Various methods can be used to minimize the number of bits required to encode motion information.

[0048] For example, if the reference picture and motion vector of the current block are identical to those of a neighboring block, the motion information of the current block can be transmitted to the image decoding device by encoding information that can identify the neighboring block. This method is called 'merge mode'.

[0049] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.

[0050] As the surrounding blocks for deriving merge candidates, all or part of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block within the current picture may be used, as illustrated in FIG. 4. In addition, a block located within a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block within the reference picture or blocks adjacent to the block at the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.

[0051] The inter prediction unit (124) uses these surrounding blocks to construct a merge list containing a predetermined number of merge candidates. Among the merge candidates included in the merge list, the merge candidate to be used as motion information of the current block is selected and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the video decoding device.

[0052] Merge Skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy encoding are close to zero, only neighboring block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high encoding efficiency can be achieved for low-motion images, still images, and screen content images.

[0053] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0054] Another method for encoding motion information is Advanced Motion Vector Prediction (AMVP) mode.

[0055] In AMVP mode, the inter prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive predicted motion vector candidates, all or some of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block in the current picture as shown in FIG. 4 may be used. In addition, a block located in a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as the neighboring block used to derive predicted motion vector candidates. For example, a block co-located with the current block in the reference picture, blocks adjacent to the block in the co-located block, or a block having the center pixel as the upper left pixel in the block in the co-located block may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.

[0056] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of these surrounding blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to produce a differential motion vector.

[0057] The predicted motion vector can be obtained by applying a predefined function (e.g., median, mean, etc.) to the predicted motion vector candidates. In this case, the image decoding device also knows the predefined function. In addition, since the surrounding blocks used to derive the predicted motion vector candidates are blocks that have already been encoded and decoded, the image decoding device also already knows the motion vectors of the surrounding blocks. Therefore, the image encoding device does not need to encode information to identify the predicted motion vector candidates. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.

[0058] Alternatively, the predicted motion vector can be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the differential motion vector and the reference picture used to predict the current block.

[0059] The subtractor (130) subtracts the prediction block generated by the intra prediction unit (122) or inter prediction unit (124) from the current block to generate a residual block.

[0060] The transformation unit (140) transforms residual signals within a residual block having pixel values ​​in a spatial domain into transform coefficients in a frequency domain. The transformation unit (140) may transform the residual signals within the residual block using the entire size of the residual block as a transformation unit, or may divide the residual block into a plurality of sub-blocks and use the sub-blocks as transformation units to perform the transformation. Alternatively, the residual signals may be transformed using only the transformation domain sub-block as a transformation unit by dividing the sub-blocks into two sub-blocks, that is, a transformation domain and a non-transform domain. Here, the transformation domain sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only a sub-block has been converted, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. In addition, the size of the conversion area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0061] Meanwhile, the transformation unit (140) can individually perform transformations on the residual block in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices can be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one transformation function pair with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) on the transformation function pair selected among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0062] The quantization unit (145) quantizes the transform coefficients output from the transform unit (140) using quantization parameters and outputs the quantized transform coefficients to the entropy encoding unit (155). The quantization unit (145) may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit (145) may also apply different quantization coefficients (scaling values) according to the positions of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to an image decoding device.

[0063] The rearrangement unit (150) can perform rearrangement of coefficient values ​​for quantized residual values.

[0064] The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from the DC coefficient to the coefficients of the high-frequency region by using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, the scanning method to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan.

[0065] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transform coefficients output from the rearrangement unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

[0066] In addition, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoding device can divide the block in the same manner as the image encoding device. In addition, the entropy encoding unit (155) encodes information about a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., information about an intra prediction mode) or inter prediction information (information about an encoding mode of motion information (merge mode or AMVP mode), a merge index in the case of a merge mode, and a reference picture index and a differential motion vector in the case of an AMVP mode) according to the prediction type. In addition, the entropy encoding unit (155) encodes information related to quantization, that is, information about a quantization parameter and information about a quantization matrix.

[0067] The inverse quantization unit (160) inversely quantizes the quantized transform coefficients output from the quantization unit (145) to generate transform coefficients. The inverse transform unit (165) transforms the transform coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

[0068] An adder (170) adds the restored residual block and the predicted block generated by the prediction unit (120) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting the next block.

[0069] The loop filter unit (180) performs filtering on restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation / quantization. The loop filter unit (180) may include all or part of a deblocking filter (182), a sample adaptive offset (SAO) filter (184), and an adaptive loop filter (ALF, 186) as an in-loop filter.

[0070] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-based encoding / decoding, and the SAO filter (184) and the ALF (186) perform additional filtering on the deblocking-filtered image. The SAO filter (184) and the ALF (186) are filters used to compensate for the differences between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in units of CTUs. In contrast, the ALF (186) performs block-based filtering, and compensates for distortion by applying different filters by distinguishing the edges and degrees of variation of the corresponding block. Information on filter coefficients to be used in the ALF can be encoded and signaled to an image decoding device.

[0071] The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in the memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0072] The video encoding device can store the bitstream of encoded video data on a non-transitory storage medium or transmit it to the video decoding device using a communication network.

[0073] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image decoding device and its subcomponents will be described with reference to FIG. 5.

[0074] The video decoding device may be configured to include an entropy decoding unit (510), a rearrangement unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).

[0075] Similar to the video encoding device of FIG. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0076] The entropy decoding unit (510) decodes the bitstream generated by the image encoding device to extract information related to block division, thereby determining the current block to be decoded, and extracts prediction information, information on residual signals, etc. required to restore the current block.

[0077] The entropy decoding unit (510) extracts information about the CTU size from the Sequence Parameter Set (SPS) or the Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and the CTU is divided using the tree structure by extracting division information about the CTU.

[0078] For example, when splitting a CTU using the QTBTTT structure, first, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, the second flag (mtt_split_flag) related to the splitting of MTT and the split direction (vertical / horizontal) and / or split type (binary / ternary) information are extracted, and the corresponding leaf nodes are split into the MTT structure. Accordingly, each node below the leaf nodes of QT are split recursively into the BT or TT structure.

[0079] As another example, when splitting a CTU using the QTBTTT structure, the CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and if the block is split, the first flag (QT_split_flag) may be extracted. During the splitting process, each node may undergo zero or more repeated QT splits followed by zero or more repeated MTT splits. For example, a CTU may undergo an MTT split right away, or conversely, may undergo only multiple QT splits.

[0080] As another example, when splitting a CTU using the QTBT structure, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Furthermore, for nodes corresponding to leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and splitting direction information are extracted.

[0081] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded by using the division of the tree structure, it extracts information on the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-prediction, the entropy decoding unit (510) extracts syntax elements for intra-prediction information (intra-prediction mode) of the current block. If the prediction type information indicates inter-prediction, the entropy decoding unit (510) extracts syntax elements for inter-prediction information, i.e., information indicating a motion vector and a reference picture referenced by the motion vector.

[0082] Additionally, the entropy decoding unit (510) extracts information about the quantized transform coefficients of the current block as information related to quantization and information about residual signals.

[0083] The rearrangement unit (515) can change the sequence of one-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the image encoding device.

[0084] The inverse quantization unit (520) inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantization unit (520) may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantized coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.

[0085] The inverse transform unit (530) inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a residual block for the current block.

[0086] In addition, when the inverse transform unit (530) inversely transforms only a portion of a transform block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only a sub-block of the transform block has been transformed, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block, and inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to restore residual signals, and fills “0” values ​​with residual signals for areas that have not been inversely transformed, thereby generating a final residual block for the current block.

[0087] In addition, when MTS is applied, the inverse transform unit (530) determines a transform function or a transform matrix to be applied in the horizontal and vertical directions using MTS information (mts_idx) signaled from the image encoding device, and performs inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.

[0088] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is inter prediction.

[0089] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.

[0090] The inter prediction unit (544) uses the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510) to determine the motion vector of the current block and the reference picture referenced by the motion vector, and predicts the current block using the motion vector and the reference picture.

[0091] An adder (550) adds the residual block output from the inverse transform unit (530) and the predicted block output from the inter prediction unit (544) or the intra prediction unit (542) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting a block to be decoded later.

[0092] The loop filter unit (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as in-loop filters. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter (564) and the ALF (566) perform additional filtering on restored blocks after deblocking filtering to compensate for differences between restored pixels and original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about filter coefficients decoded from the non-stream.

[0093] The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in the memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0094] Hereinafter, an improved coding tool performed by the aforementioned video encoding device or video decoding device is disclosed. The following embodiments may be performed by the intra prediction unit (122) within the video encoding device. Additionally, the following embodiments may be performed by the intra prediction unit (542) within the video decoding device. In the present disclosure, the video processing device refers to at least one of the video encoding device or the video decoding device.

[0095] The present disclosure relates to a combined mode of a block-based inter-prediction mode and a sub-block-based inter-prediction mode. The current block is predicted using a weighted sum of a prediction block according to the block-based inter-prediction mode and a prediction block according to the sub-block-based inter-prediction mode.

[0096] In the present disclosure, the block-based inter-prediction mode may be a prediction mode that refers to a block having the same size as the current block, such as the Advanced Motion Vector Prediction (AMVP) mode, the template matching-based inter-prediction mode, or the merge mode. The sub-block-based inter-prediction mode may be a prediction mode that refers to sub-blocks of a reference block, such as the sub-block-based merge mode, or the merge mode.

[0097] FIG. 6 is a flowchart of a method for predicting a current block according to one embodiment of the present disclosure.

[0098] Referring to FIG. 6, in step S610, the image processing device derives block-based motion information of the current block.

[0099] The image processing device derives block-based motion information of the current block based on pre-encoded blocks, pre-decoded blocks, or pre-restored blocks.

[0100] In one embodiment, a video encoding device can determine block-based motion information of a current block and signal the block-based motion information of the current block. A video decoding device can derive block-based motion information of the current block based on the signaling information.

[0101] First, the image processing device generates a block-based motion candidate list of the current block based on the previously restored blocks.

[0102] The block-based motion candidate list may be an AMVP candidate list. Each block-based motion candidate includes motion information or zero motion vectors of spatial and temporal neighboring blocks of the current block, and may also include a motion vector difference (MVD) and a reference picture index.

[0103] Alternatively, the block-based motion candidate list may be a merge candidate list. The block-based motion candidate list may include motion information of spatial neighboring blocks, motion information of temporal neighboring blocks, motion information of history-based neighboring blocks, or zero motion vectors as block-based motion candidates.

[0104] An image processing device can signal an index indicating a block-based motion candidate within a block-based motion candidate list. An image encoding device can determine a block-based motion candidate for motion prediction of a current block, and signal an index of the determined block-based motion candidate to an image decoding device. The image decoding device can determine a block-based motion candidate from the block-based motion candidate list based on the signaled block-based motion candidate index, and determine block-based motion information of the current block using motion information of the block-based motion candidate. The motion information of the block-based motion candidate can be used as block-based motion information of the current block.

[0105] In another embodiment, the image processing device can search for a reference block based on a template matching cost within a previously restored area, and determine block-based motion information of a current block using motion information of the reference block.

[0106] Here, template matching involves searching for template regions similar to the template containing the surrounding samples of the current block within the restored search region of the current picture or reference picture. The block that uses the searched template region as its template is determined as the reference block of the current block. The template matching cost indicates the difference between the two templates.

[0107] Template matching cost is a value based on the difference between two templates, and can be used as the sum of absolute differences (SAD), the sum of absolute transformed differences (SATD), the mean-removed sum of absolute differences (MR-SAD), the mean squared error (MSE), or the sum of squared errors (SSE).

[0108] The motion information of the reference block searched based on the template matching cost can be used as the block-based motion information of the current block. For example, the block-based motion vector and reference picture index of the current block may be the same as the motion vector and reference picture index of the reference block. As another example, the block-based motion vector of the current block may be the sum of the motion vector of the reference block and the MVD. In another example, the motion information of a candidate with a low template matching cost among AMVP candidates in the AMVP candidate list may be used as the block-based motion information of the current block, and the signaling of the AMVP candidate index may be skipped.

[0109] In step S620, the image processing device generates a first prediction block of the current block using block-based motion information. A block indicated by the block-based motion information of the current block can be used as the first prediction block of the current block.

[0110] In step S630, the image processing device derives sub-block-based motion information of the current block.

[0111] First, the image processing device generates a sub-block-based motion candidate list for the current block based on the previously restored blocks containing sub-blocks. Furthermore, the image processing device can rearrange the candidates within the sub-block-based motion candidate list. The generation of the sub-block-based motion candidate list is illustrated in Fig. 11.

[0112] The subblock-based motion candidate list may be a subblock-based merge candidate list. The subblock-based merge candidate list includes at least one of subblock-based temporal motion vector prediction (SbTMVP) candidates and affine candidates. One of the reconstructed blocks is a neighboring block that refers to a collocated picture of the current picture including the current block, and may be used as an SbTMVP candidate. The SbTMVP candidate and the affine candidate are described in FIGS. 7, 8A, 8B, and 9.

[0113] The sub-block-based motion candidate list may be a merge candidate list. The previously reconstructed blocks may point to at least one of the spatial neighboring blocks, the temporal neighboring blocks, or the history-based neighboring blocks of the current block. The sub-block-based motion candidate list may include motion information of spatial neighboring blocks, motion information of temporal neighboring blocks, motion information of history-based neighboring blocks, or a zero motion vector as sub-block-based motion candidates. Here, the motion information refers to motion information at the sub-block level.

[0114] Thereafter, the image processing device derives sub-block-based motion information of the current block using the sub-block-based motion candidate list.

[0115] In one embodiment, when block-based motion information of a current block is first derived, the image processing device can derive sub-block-based motion information of the current block based on a bilateral matching cost.

[0116] Sub-block-based motion information of the current block can be selected from among the sub-block-based motion candidates based on a bilateral matching cost between blocks predicted using the sub-block-based motion candidates in the sub-block-based motion candidate list and the first prediction block. The sub-block-based motion candidate with the smallest bilateral matching cost can be determined as the sub-block-based motion information of the current block.

[0117] If the motion information of a sub-block-based motion candidate has an L0 direction, a candidate with an L1 direction motion information and a low bidirectional matching cost among the sub-block-based motion candidates may be selected. Alternatively, if the motion information of a sub-block-based motion candidate has an L1 direction, a candidate with an L0 direction motion information and a low bidirectional matching cost among the sub-block-based motion candidates may be selected.

[0118] The two-way matching cost is a value based on the difference between the two blocks, and can be used as the sum of absolute differences (SAD), the sum of absolute transformed differences (SATD), the mean-removed sum of absolute differences (MR-SAD), the mean squared error (MSE), or the sum of squared errors (SSE).

[0119] In another embodiment, the image processing device can derive sub-block-based motion information of the current block based on signaling information.

[0120] An image processing device can signal an index indicating a sub-block-based motion candidate within a sub-block-based motion candidate list. An image encoding device can determine a sub-block-based motion candidate for motion prediction of a current block, and signal an index of the determined sub-block-based motion candidate to an image decoding device. The image decoding device can determine a sub-block-based motion candidate from the sub-block-based motion candidate list based on the signaled sub-block-based motion candidate index, and determine sub-block-based motion information of the current block using motion information of the sub-block-based motion candidate. The motion information of the sub-block-based motion candidate can be used as sub-block-based motion information of the current block.

[0121] Subblock-based motion candidates contain motion information at the subblock level. Subblocks of the current block are predicted using the motion information at the subblock level.

[0122] In step S640, the image processing device predicts the current block on a sub-block basis using sub-block-based motion information, thereby generating a second prediction block of the current block. Specifically, the image processing device predicts each sub-block using motion information of each sub-block of the current block.

[0123] In step S650, the image processing device generates a prediction block of the current block by performing a weighted sum of the first prediction block and the second prediction block of the current block.

[0124] The image processing device can generate a residual block based on the difference between the predicted block of the current block and the original block, and perform transformation / inverse transformation and quantization / inverse quantization on the residual block.

[0125] Meanwhile, the first reference picture used to generate the first prediction block and the second reference picture used to generate the second prediction block may have different prediction directions. In other words, the first prediction block may be generated by referring to a block in a reference picture located in either the L0 direction or the L1 direction with respect to the current picture, and the second prediction block may be generated by referring to sub-blocks in a reference picture located in the other direction. As the prediction direction, the L0 direction refers to reference pictures that are earlier than the current picture in the playback order, and the L1 direction refers to reference pictures that are later than the current picture in the playback order. When the reference picture of the block-based motion information of the current block is included in the LX (where X is 0 or 1) reference picture list, the reference picture of the sub-block-based motion information is included in the L(1-X) reference picture list.

[0126] In another embodiment, the first reference picture used to generate the first prediction block and the second reference picture used to generate the second prediction block may have the same prediction direction.

[0127] Meanwhile, the weights for the weighted sum of the first prediction block and the second prediction block are explained in Fig. 10.

[0128] Meanwhile, block-based motion information and sub-block-based motion information can be corrected based on bidirectional optical flow (BDOF) in both cases where the distances from the current picture to the two reference pictures are equal and when they are not. Alternatively, motion information correction based on BDOF can be performed only when the distances from the current picture to the two reference pictures are equal.

[0129] In the present disclosure, the block-based inter-prediction mode may be an AMVP mode, and the sub-block-based inter-prediction mode may be a sub-block-based merge mode. That is, the block-based inter-prediction mode may be a combined mode of the AMVP (Advanced Motion Vector Prediction) mode and the sub-block-based merge mode. The combined mode may be referred to as a sub-block-based AMVP-merge mode or an AMVP-subblock based merge mode.

[0130] AMVP-subblock-based merge mode can be applied regardless of the distance between the current picture and the reference pictures in both directions. If the distance from the current picture to the L0 reference picture is equal to the distance from the current picture to the L1 reference picture, AMVP-subblock-based merge mode is allowed for the prediction of the current block. Even if the distance from the current picture to the L0 reference picture is not equal to the distance from the current picture to the L1 reference picture, AMVP-subblock-based merge mode is allowed for the prediction of the current block.

[0131] AMVP-subblock-based merge mode can be restricted based on the size of the current block. For example, subblock-based merge mode, including affine mode or SbTMVP mode, can be applied to the current block only if the current block size is 8x8 or larger.

[0132] Below, AMVP mode is described.

[0133] AMVP mode derives the motion vector of the current block using the motion vectors of the spatial and temporal neighboring blocks of the current block, the zero motion vector, and the differential motion vector (MVD), and predicts the current block using the block indicated by the motion vector within the reference picture. Here, the MVD represents the difference between the motion vector indicating the location of the optimal reference block in the reference picture and the motion vectors of the neighboring blocks.

[0134] In AMVP mode, motion information of surrounding blocks may include motion vector candidate information and motion vector difference information.

[0135] In AMVP mode, motion information of the current block is encoded / decoded based on an AMVP candidate list. An image processing device can construct an AMVP candidate list using motion vectors and zero motion vectors of spatial and temporal neighboring blocks, and encode / decode the motion vector of the current block based on the AMVP candidate list. The image processing device can construct an AMVP candidate list including a preset number of candidates (e.g., 2).

[0136] Specifically, the image processing device searches for spatial candidates. The image processing device searches for spatial candidates from surrounding blocks, as illustrated in FIG. 4. Up to two spatial candidates can be selected.

[0137] An image processing device searches for a temporal candidate. The image processing device may determine a corresponding block in a reference picture other than the current picture containing the current block as a temporal candidate. Here, the reference picture may be a co-located picture, and the corresponding block may be a block in the co-located picture that is at the same relative position as the current block in the current picture. Alternatively, the corresponding block may be a block located at the center or bottom right of a block in the same relative position as the current block. One temporal candidate may be selected.

[0138] Despite searching for spatial and temporal candidates, if the number of motion vector predictor (MVP) candidates is less than a preset number, the image processing device adds a zero motion vector as a candidate.

[0139] Once the AMVP candidate list is constructed, the video encoding device determines the optimal motion vector for the current block in terms of optimizing encoding efficiency, and derives the MVD by subtracting the motion vector of any AMVP candidate from the optimal motion vector. The motion vector of the AMVP candidate used to derive the MVD may be referred to as the MVP.

[0140] In AMVP mode, the video encoding device signals a reference picture index and an MVD, which indicate a reference picture in the reference picture list, to the video decoding device. Furthermore, the video encoding device may signal an MVP index, which indicates an AMVP candidate.

[0141] The video decoding device can decode the motion vector of the current block by configuring an AMVP candidate list in the same manner as the AMVP candidate list of the video encoding device, deriving an MVP from the AMVP candidate list using the MVP index, and adding the MVP and the MVD. The video decoding device determines a reference picture from the reference picture list using the reference picture index, and predicts the current block using a reference block indicated by the motion vector of the current block within the reference picture.

[0142] In another embodiment, the MVP index can be derived using template matching in a video decoding device without signaling. The video decoding device can select the candidate among the AMVP candidates that has the template with the smallest difference from the template of the current block.

[0143] Meanwhile, the video encoding device can transmit information for determining the spatial resolution of the MVD together with the MVD. When using AMVR technology, the video encoding device can determine the adaptive spatial resolution of the MVD in terms of bit rate distortion optimization. At this time, the spatial resolution of the MVD and the spatial resolution of the motion vector may be the same. The video encoding device informs the video decoding device of the spatial resolution of the MVD by signaling amvr_flag and amvr_precision_idx to the video decoding device. If amvr_flag is signaled as 0, the video decoding device sets the MVD to a 1 / 4-pel spatial resolution. On the other hand, if amvr_flag is not 0, the video decoding device can determine the spatial resolution of the MVD according to amvr_precision_idx. At this time, the selectable spatial resolution of the MVD may vary depending on the prediction method to which AMVR is applied. The prediction methods applicable to AMVR include general AMVP mode, affine AMVP mode, and IBC (Intra Block Copy) AMVP mode.

[0144] Below, the subblock-based merge mode is described.

[0145] Subblock-based merge mode divides the current block into subblocks, derives motion information for each subblock, and predicts the current block using the motion information of the subblocks. The motion information of the subblocks is derived from motion information according to SbTMVP mode or motion information according to affine mode.

[0146] Below, the SbTMVP mode is described.

[0147] SbTMVP mode is a mode that derives a motion vector field (MVF) representing motion vectors of subblock units from the co-position picture of the current block.

[0148] In SbTMVP mode, a motion vector field is derived from a reference block located at a position equal to the motion vector of a neighboring block of the current block from the co-located block of the current block. Here, the motion vector of the neighboring block may be referred to as a motion shift.

[0149] Referring to FIG. 4, peripheral blocks for inducing motion shift in SbTMVP mode are illustrated.

[0150] Motion shift can be induced by scanning at least one of blocks A0, A1, B0, B1, and B2. The scanning order of the surrounding blocks can be arbitrarily determined. Alternatively, the scanning order of the surrounding blocks can be predefined between the image decoding device and the image encoding device. For example, the scanning order of the surrounding blocks can be scanned in the order of A0, B0, B1, A1, and B2.

[0151] The motion shift of the current block is derived from the motion vectors of the surrounding blocks.

[0152] As an example, the motion vector of a surrounding block at a preset position among surrounding blocks can be determined as a motion shift of the current block.

[0153] As another example, if a neighboring block has a motion vector referencing a co-located picture of the current picture, the motion vector of that neighboring block can be determined as a motion shift. If no motion shift is derived from the neighboring blocks, i.e., if none of the neighboring blocks refer to a co-located picture of the current picture, the motion shift can be set to 0.

[0154] In subblock-based merge mode, blocks available for motion shift among blocks A0, A1, B0, B1, and B2 can be SbTMVP candidates. Blocks available for motion shift can be listed as SbTMVP candidates in scan order.

[0155] When adding SbTMVP candidates to a subblock-based merge candidate list, the image processing device may perform a duplication check between the candidates in the subblock-based merge candidate list and the candidates to be added. If the candidates to be added do not overlap, the image processing device adds the candidates to the subblock-based merge candidate list. If the candidates to be added overlap, the image processing device does not add the candidates to the subblock-based merge candidate list.

[0156] For example, if the block indicated by the motion shift of the lower left block of the current block and the block indicated by the motion shift of the upper left block of the current block are the same, only one of the two blocks can be added as an SbTMVP candidate.

[0157] FIG. 7 is a diagram for explaining prediction of SbTMVP mode according to one embodiment of the present disclosure.

[0158] Referring to Fig. 7, the A1 block located at the lower left of the current block can be used to derive the motion vector field of the current block.

[0159] Block A1 is predicted by referencing block A1' within the co-located picture. Since block A1 uses the co-located picture as a reference picture, the motion vector of block A1 is determined to be a motion shift.

[0160] By applying a motion shift to the co-located block of the current block, a reference block within the co-located picture is selected. The reference block is a block within the co-located picture that is located at a position equal to the motion shift from the relative position of the current block within the current picture.

[0161] The current block and reference block are divided equally. Each subblock of the current block corresponds to each subblock of the reference block. The subblocks of the current block may be referred to as current subblocks, and the subblocks of the reference block may be referred to as reference subblocks.

[0162] Motion information is derived for each subblock of a reference block. Specifically, motion information is derived for the center position of each reference subblock. Here, the center position may be the position of the lower right sample among four adjacent samples located at the center of the reference subblock. In another embodiment, the center position may be one of the four adjacent samples.

[0163] If motion information of a reference subblock is unavailable, motion information of the center position of the reference block may be used as motion information of the reference subblock. Here, the center position may be the position of the lower right reference subblock among four adjacent reference subblocks located at the center of the reference block.

[0164] The motion information of each subblock of the current block is derived from the motion information of each subblock of the reference block. Each current subblock uses the motion vectors of each reference subblock at the corresponding position. If the reference subblock has motion vectors in both the L0 direction and the L1 direction, the current subblock may also have motion vectors in the L0 direction and the L1 direction. The motion information of all subblocks of the current block becomes a motion vector field.

[0165] As an example, the motion information of each reference subblock can be used as the motion information of each current subblock. The L0 motion vector and L1 motion vector of each current subblock can be identical to the motion information of each corresponding reference subblock.

[0166] As another example, the motion vector of each current subblock can be derived by applying temporal motion scaling to the motion vector of each reference subblock. Specifically, the motion vector of each reference subblock can be scaled by considering the picture order count (POC) of the reference pictures referenced by the reference subblocks. The POC is a numbered value that lists the order of pictures in chronological order. A reference picture having a POC value smaller than the POC value of the current picture may indicate a picture relatively past the current picture, and a reference picture having a POC value larger than the POC value of the current picture may indicate a picture relatively future the current picture. The reference picture may be the first reference picture in the reference picture list. The motion vector of each reference subblock can be scaled based on a first difference value between the POC of the current picture and the POC of the reference picture of the current block, and a second difference value between the POC of the co-located picture and the POC of the reference picture of the reference block. As the second difference value is larger than the first difference value, the motion vector of the reference subblock is scaled down. The scaled motion vector of the reference subblock is used as the motion vector of the current subblock.

[0167] As the L0 reference picture and L1 reference picture of the current block, the first reference picture in the L0 reference picture list of the current picture and the first reference picture in the L1 reference picture list of the current picture may be used. Alternatively, the reference pictures of the reference subblocks may be used as the reference pictures of the current block.

[0168] Each subblock of the current block is predicted using the derived motion information. In other words, the current block is predicted using the motion vector field. If the motion information of the reference subblock includes L0 motion information and L1 motion information, the current subblock is predicted by performing a weighted sum on the block indicated by the L0 motion information in the L0 reference picture and the block indicated by the L1 motion information in the L1 reference picture. The bidirectional weights of the current subblock may be the same as the bidirectional weights of the reference subblock.

[0169] Meanwhile, an SbTMVP mode flag may be defined to indicate whether the SbTMVP mode can be applied to the current block, and this may be signaled at at least one of the upper levels of the current block, such as a sequence, a picture, a slice, a tile, a tile group, or a brick. For example, the SbTMVP mode flag may be named sps_sbtmvp_enabled_flag.

[0170] Below, the affine mode is described.

[0171] Affine motion compensation prediction is a prediction mode used to efficiently code video signals that include motion such as zoom, rotation, and shear. While conventional video encoding / decoding systems use only a single motion vector to represent the motion information of the current block to represent optimal motion information for each block, affine mode defines the motion information of a block at the pixel or subblock level. Considering the complexity of deriving a motion vector at the pixel level, motion information can be derived at the subblock level.

[0172] According to the affine mode, the motion vector for each pixel and / or subblock of the current block can be determined using two or more motion vectors associated with the current block. The size of the subblock is NxN, which is smaller than that of the current block. Here, N is a positive integer greater than or equal to 1. For example, N can be 4 or 8.

[0173] In affine mode, motion information at the pixel or subblock level within the current block can be derived and expressed using two or more control point motion vectors (CPMVs). The motion vector of a subblock at a specific location within the current block can be derived using CPMVs. The set of motion vectors of the subblocks within the current block can be referred to as an affine motion vector field (Affine MVF).

[0174] Prediction pixels and / or prediction blocks for pixels and / or subblocks of the current block are generated by performing motion compensation within a reference picture using the motion vectors for each pixel and / or subblock of the current block. Here, the reference picture of the current block can be signaled for the current block or derived from an affine-based MVP candidate.

[0175] Meanwhile, a 4-parameter model or a 6-parameter model can be used as a model to express motion information in affine mode. The 4-parameter model is advantageous in predicting motions such as zoom and rotation. The 6-parameter model is advantageous in predicting motions such as zoom, rotation, and shear.

[0176] FIG. 8A and FIG. 8B are diagrams for explaining affine parameter models according to one embodiment of the present disclosure.

[0177] Figure 8a illustrates a four-parameter model. The four-parameter model utilizes two control point motion vectors (CPMVs) located at the top-left corner and the top-right corner. Each CPMV contains two parameters corresponding to the x-axis and the y-axis, and the two CPMVs contain four parameters.

[0178] The 4-parameter model is expressed as shown in Equation 1. The motion vector (mv) at the location (x, y) of the sample or sub-block within the current block x , mv y ) can be derived based on mathematical expression 1. The position of the upper left sample of the current block can be expressed as (0,0). The position (x, y) of the subblock can indicate the position of the lower right sample among the four samples located at the center of the subblock.

[0179]

[0180] In mathematical expression 1, mv0={mv 0x , mv 0y} is the CPMV of the upper left corner of the current block, and mv1={mv 1x , mv 1y} represents the CPMV of the upper right position of the current block. W and H represent the width and height of the current block, respectively, and mv={mvx , mv y} represents the motion vector at pixel location (x, y).

[0181] Figure 8b illustrates a six-parameter model. The six-parameter model is a model that utilizes three CPMVs located at the top-left corner, the top-right corner, and the bottom-left corner.

[0182] The 6-parameter model is expressed as shown in Equation 2. The motion at the position (x, y) of a sample or sub-block within the current block can be derived based on Equation 2.

[0183]

[0184] In Equation 2, mv0={mv 0x , mv 0y} is the CPMV of the upper left corner of the current block, and mv1={mv 1x , mv 1y} is the CPMV of the upper right position of the current block, and mv2={mv 2x , mv 2y} represents the CPMV of the lower left position of the current block. W and H represent the width and height of the current block, respectively, and mv={mv x , mv y} represents the motion vector at pixel location (x, y).

[0185] In mathematical expressions 1 and 2, the motion vector can be set to have 1 / 16 sample precision. Motion vector (mv x , mv y ) can be rounded to the nearest 1 / 16 sample unit.

[0186] Meanwhile, an affine mode flag may be defined to indicate whether the affine mode can be applied to the current block, and this may be signaled at at least one of the upper levels of the current block, such as a sequence, picture, slice, tile, tile group, or brick. For example, the affine mode flag may be named sps_affine_enabled_flag.

[0187] An image encoding device encodes affine-related syntax elements including a flag indicating whether affine motion prediction has been applied to a target block, type information indicating the type of an affine model, and motion information indicating the motion vector of each control point, and transmits the encoded affine-related syntax elements to an image decoding device. The type information and the motion information of the control points may be signaled when affine motion prediction is performed, and the motion vectors of the control points may be signaled in a number determined according to the type information.

[0188] The video decoding device determines the type of affine model and CPMVs using the signaled syntaxes, and calculates the motion vector (mv) for each 4×4 sub-block in the target block using Equation 1 or 2. x ,mv y ) is calculated. If the motion vector resolution information for the affine motion vector of the target block is signaled, the motion vector (mv x ,mv y ) is modified to a precision identified by the motion vector resolution information using operations such as rounding. The video decoding device calculates the motion vector (mv) for each sub-block x ,mv y ) to perform motion compensation within the reference picture, thereby generating a prediction block for each sub-block.

[0189] Meanwhile, affine mode can be further subdivided into sub-modes based on how motion information is encoded / decoded. For example, affine mode can be further subdivided into affine MVP mode and affine merge mode. Affine merge mode can be utilized in subblock-based merge mode.

[0190] In affine MVP mode, the affine MVP index, MVD and index to the reference picture can be signaled.

[0191] In affine merge mode, an affine merge candidate list may be used to reduce the number of bits required to signal CPMVs.

[0192] The affine merge candidate list may include at least one of an inherited affine merge candidate, a constructed affine merge candidate, a history-based parameter inheritance candidate, a pairwise average affine merge candidate, and a zero merge candidate. The affine merge candidate list may be constructed in the order of the candidates listed above. The affine merge candidate list may include a predefined number of affine merge candidates.

[0193] An inherited affine merge candidate may mean a candidate derived using the CPMV of a surrounding block encoded / decoded in affine mode.

[0194] A combination affine merge candidate can be a candidate derived from CPMVs derived from the motion vectors of the surrounding blocks of each CP (Control Point), and derived using CPMVs. Here, a CP can refer to a specific location within a block used to derive a CPMV. For example, a CP can be the location of each vertex within a block.

[0195] A history-based parameter inheritance candidate is a candidate that stores affine parameters based on history and derives them based on the stored affine parameters. The motion vectors of the current subblock can be derived using the affine parameters stored prior to the current block.

[0196] A pair-average affine merge candidate derives the CPMVs of the current block by averaging the CPMVs of two or more candidates within a subblock-based merge candidate list, and refers to a candidate derived using the CPMVs of the current block. Before a pair-average affine merge candidate is added, the subblock-based merge candidate list contains two or more candidates.

[0197] A zero merge candidate may mean a candidate consisting of CPMVs of size 0.

[0198] The list of affine merge candidates can be added in the following order: inheritance affine merge candidates, combination affine merge candidates, and zero merge candidates. If the number of inheritance affine merge candidates and combination affine merge candidates is less than a predefined number, a zero merge candidate may be added.

[0199] First, the image processing device derives inherited affine merge candidates from CPMVs of neighboring blocks predicted by affine mode among neighboring blocks of the current block. For example, a predetermined number of inherited affine merge candidates may be derived from neighboring blocks (A0, A1, B0, B1, B2) of the current block as shown in FIG. 4. In some embodiments, the number of merge candidates derived from neighboring blocks predicted by affine mode may be limited. For example, a total of two inherited affine merge candidates may be derived from one of A0 and A1, and one of B0, B1, and B2. The priorities may be in the order of A0, A1, and then in the order of B0, B1, and B2. Each of the inherited affine merge candidates included in the affine merge candidate list corresponds to a combination of two or three CPMVs.

[0200] When the total number of inherited affine merge candidates is three or less, the image processing device can derive the missing number of combinatorial affine merge candidates from the translational motion vectors of the surrounding blocks.

[0201] Figure 9 is an example diagram illustrating a method for deriving combinational affine merge candidates for affine motion prediction.

[0202] The image processing device derives CPMV1, CPMV2, and CPMV3 from the neighboring block group {B2, B3, A2}, the neighboring block group {B1, B0}, and the neighboring block group {A1, A0}, respectively. As an example, the priorities within each neighboring block group may be in the order of B2, B3, and A2, the order of B1, B0, and the order of A1, A0. In addition, the image processing device derives CPMV4 from the co-located block T in the reference picture. The image processing device generates the missing number of combined affine merge candidates by combining two or three CPMVs among the four CPMVs. The priorities of the combinations are as follows. The elements within each group are listed in the order of upper-left, upper-right, and lower-left CPMVs.

[0203] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4},

[0204] {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0205] If the affine merge candidate list cannot be filled using the inherited affine merge candidates and the combined affine merge candidates, the image processing device can add a zero motion vector as a candidate.

[0206] Meanwhile, the video encoding device selects an affine merge candidate from the affine merge candidate list in order to optimize encoding efficiency, and performs affine motion prediction for the current block using the selected affine merge candidate. When the affine merge candidate consists of two CPMVs, affine motion prediction is performed using a 4-parameter model. When the affine merge candidate consists of three CPMVs, affine motion prediction is performed using a 6-parameter model. The video encoding device encodes a merge index indicating the selected affine merge candidate and signals it to the video decoding device.

[0207] The video decoding device receives a merge index, constructs an affine merge candidate list in the same manner as the video encoding device, selects an affine merge candidate indicated by the merge index in the affine merge candidate list, and performs affine motion prediction using control point motion vectors corresponding to the selected affine merge candidate.

[0208] FIG. 10 is a diagram for explaining prediction of AMVP-subblock based merge mode according to one embodiment of the present disclosure.

[0209] Referring to FIG. 10, the reference picture in the L0 reference picture list, the current picture, and the reference picture in the L1 reference picture list are sequentially depicted. The L0 reference picture, the current picture, and the L1 reference picture are played at points t-t0, t, and t+t0 in the playback order. Here, t0 and t1 are arbitrary positive integers.

[0210] In Fig. 10, the prediction block in the L0 direction of the current block is generated using the AMVP mode, and the prediction block in the L1 direction of the current block is generated using the SbTMVP mode. AMVP motion vector MV Merge the motion vector of SbTMVP with the L0 direction for the current block MVhas an L1 direction for the current block. That is, the prediction direction according to the AMVP mode is different from the prediction direction according to the SbTMVP mode.

[0211] In another embodiment, the L1-direction prediction block of the current block can be generated using the affine mode. In another embodiment, the L0-direction prediction block can be generated using the subblock-based merge mode, and the L1-direction prediction block of the current block can be generated using the AMVP mode. That is, the AMVP motion vector MV Merge the motion vector of SbTMVP with L1 direction for the current block MV has the L0 direction for the current block.

[0212] For the current block, the predicted block in the L0 direction is the motion vector AMVP of AMVP MV Reference block P pointed to by L0 It is generated using AMVP MV is encoded or decoded using the motion information and MVD of the spatial and temporal surrounding blocks of the current block.

[0213] For the current block, a prediction block in the L1 direction is generated using motion information of the subblock unit of the block indicated by the motion shift. Specifically, a neighboring block referencing a co-located picture among neighboring blocks of the current block is selected, and the motion information of the selected neighboring block becomes a motion shift. The block indicated by the motion shift includes subblocks, and the motion information of each subblock is used as the motion information of the subblocks of the current block. The motion information of the subblocks of the current block includes at least one of a scaled motion vector in the L0 direction or a scaled motion vector in the L1 direction of the subblocks in the block indicated by the motion shift. The subblocks of the current block are reference subblocks C indicated by the motion information of the subblocks. L1are predicted using. The subblocks of the current block can be predicted using the subblocks indicated by the L1 direction motion vectors of the subblocks of the current block on the L1 reference picture. Here, the L1 reference picture is the first picture in the L1 reference picture list.

[0214] The prediction block of the current block is generated by weighting the prediction block in the L0 direction and the prediction block in the L1 direction. The prediction block of the current block is equal to the weighted sum of the prediction block in the L0 direction and the prediction block in the L1 direction.

[0215] Here, the weights for the L0-direction prediction block and the L1-direction prediction block can be determined based on various factors. Both weights are real numbers less than or equal to 1, and the sum of the two weights can be 1.

[0216] In one embodiment, the weights may be arbitrarily determined. The weights may be pre-agreed between the image encoding device and the image decoding device.

[0217] In another embodiment, the weights may be determined based on the distance between the current picture and two reference pictures. The weights may be determined based on the ratio of the distance from the current picture to the L0 reference picture and the distance from the current picture to the L1 reference picture. For example, if the difference between the POC of the current picture and the POC of the LO reference picture is smaller than the difference between the POC of the current picture and the POC of the L1 reference picture, the weight of the prediction block in the L0 direction may be greater than the weight of the prediction block in the L1 direction. The above example can also be applied to the opposite situation.

[0218] In other embodiments, the weights may be determined based on information accessible to the video decoding device, such as derivable information or signaling information. For example, the accessible information may include quantization parameters (QPs), previously encoded / decoded syntax elements, or picture types.

[0219] In another embodiment, the weights may be determined from predefined weight sets based on signaled index information. The video encoding device may define a plurality of weight sets and signal an index indicating one of the weight sets to the video decoding device. The weight set includes a weight of a prediction block in the L0 direction and a weight of a prediction block in the L1 direction. The video decoding device selects one of the weight sets using the signaled index information and uses the selected weight set to predict the current block.

[0220] In another embodiment, the weights may be determined based on an L0 distortion value between the template of the prediction block in the L0 direction and the template of the current block, and an L1 distortion value between the template of the prediction block in the L1 direction and the template of the current block. Here, the template may refer to an area composed of neighboring samples or non-adjacent samples adjacent to a block, and the distortion value may refer to a difference. For example, if the L0 distortion value indicating the difference between the template of the current block and the template of the prediction block in the L0 direction is greater than the L1 distortion value indicating the difference between the template of the current block and the template of the prediction block in the L1 direction, the weight of the prediction block in the L1 direction may be set to be greater than the weight of the prediction block in the L0 direction. Alternatively, if the ratio of the L1 distortion value to the L0 distortion value is greater than 1, the weight of the prediction block in the L0 direction may be set to be greater than the weight of the prediction block in the L1 direction.

[0221] In one embodiment, the weights may be determined based on a combination including at least one of a predefined value, a value based on a distance between the current picture and two reference pictures, a value based on information accessible to the video decoding device, a value based on signaled index information, or a distortion value.

[0222] Meanwhile, signaling information for the AMVP-subblock-based merge mode may include a constraint flag (constraint_flag) and an AMVP-subblock-based merge mode flag. The AMVP-subblock-based merge mode flag indicates whether the AMVP-subblock-based merge mode can be applied to the current block. The AMVP-subblock-based merge mode flag may be an AMVP-subblock-based merge mode enable flag (enabled_flag) or an AMVP-subblock-based merge mode disable flag (disabled_flag). Setting the constraint flag to 0 indicates that there is no constraint on applying the AMVP-subblock-based merge mode. The AMVP-subblock-based merge mode enable flag and the AMVP-subblock-based merge mode disable flag may be set to 0 or 1. On the other hand, if the constraint flag is set to 1, the AMVP-subblock-based merge mode enable flag is set to a value of 0 indicating disablement, and the AMVP-subblock-based merge mode disable flag is set to 1. The video decoding device can determine that the prediction mode of the current block is the AMVP-subblock-based merge mode based on the constraint flag and the AMVP-subblock-based merge mode flag.

[0223] Signaling information for AMVP-subblock based merge mode can be signaled at a higher level, such as a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a picture header (PH), or a slice header (SH).

[0224] As another example, signaling information for AMVP-subblock based merge mode can be signaled at lower levels, such as coding tree unit (CTU), coding unit (CU), prediction unit (PU), or transform unit (TU).

[0225] As another example, signaling information for the AMVP-subblock-based merge mode may not be signaled by an explicit flag, but may be signaled dependently based on flags of other prediction modes or the size of the current block, etc. For example, the AMVP-subblock-based merge mode flag may be signaled dependently on an inter prediction-related syntax element, or may be signaled dependently on an AMVP-merge mode-related syntax element. The video decoding device may determine the AMVP-subblock-based merge mode based on flags of other prediction modes or the size of the current block, etc.

[0226] FIG. 11 is a flowchart of a method for generating a sub-block-based candidate list according to one embodiment of the present disclosure.

[0227] In step S1110 of FIG. 11, the image processing device adds sub-block-based motion candidates to a sub-block-based candidate list.

[0228] Sub-block-based motion candidates can be classified into multiple types. Sub-block-based motion candidates can include first sub-block-based motion candidates and second sub-block-based motion candidates.

[0229] In the sub-block-based merge mode, the first sub-block-based motion candidates may be SbTMVP candidates, and the second sub-block-based motion candidates may be affine candidates.

[0230] For example, candidates may be added to the subblock-based merge candidate list in the following order: SbTMVP candidates, affine-based MVP candidates, and so on. As another example, candidates may be added to the subblock-based merge candidate list in the following order: affine-based MVP candidates, SbTMVP candidates.

[0231] SbTMVP candidates can be added in the scan order of blocks available as motion shifts among the neighboring blocks of the current block. Affine-based MVP candidates can be added in the following order: inherited affine merge candidate, combined affine merge candidate, history-based parameter inheritance candidate, pair-average affine merge candidate, and zero motion vector. In another embodiment, SbTMVP candidates can be added in the reverse scan order, and the order of addition of affine-based MVP candidates can be changed.

[0232] The number of candidates in the subblock-based candidate list can be preset or derived based on signaling information. The subblock-based merge candidate list can consist of K SbTMVP candidates and L affine-based MVP candidates. Here, K and L are positive integers, and K can be less than or equal to the number of neighboring blocks being scanned. The maximum number of subblock-based merge candidates can be 5.

[0233] In step S1120, the image processing device reorders sub-block-based motion candidates within the sub-block-based candidate list.

[0234] An image processing device can reorder candidates within a subblock-based merge candidate list using a subblock-based template matching cost. Hereinafter, the template matching cost may refer to a subblock-based template matching cost.

[0235] The subblock-based template matching cost refers to the difference between the subtemplates of the current block and the templates of the reference subblocks.

[0236] FIG. 12 is a diagram for explaining a sub-block-based template matching cost according to one embodiment of the present disclosure.

[0237] In Fig. 12, the current block is divided into multiple current sub-blocks, and the current sub-blocks are unidirectionally predicted.

[0238] The template of the current block may be an area including neighboring samples adjacent to the left and top of the current block. The template of the current block may be divided into multiple sub-templates based on the sizes of the sub-blocks of the current block.

[0239] Sub-templates may be assigned to some of the sub-blocks of the current block. Sub-templates may be assigned to the upper sub-blocks, left sub-blocks, and upper-left sub-blocks within the current block. For example, a sub-template located at the upper end of sub-block D among the sub-templates of the current block may be used as a template for sub-block D. As another example, sub-templates located at the left and upper end of sub-block A among the sub-templates of the current block may be used as a template for sub-block A. Sub-templates may not be assigned to sub-blocks of the current block that are not adjacent to the upper boundary or the left boundary of the current block.

[0240] Meanwhile, reference subblocks referenced by current subblocks may also have subtemplates corresponding to the templates of the current subblocks. Reference subblocks A, B, C, D, E, F, and G are subblocks located at positions corresponding to the respective motion vectors from the current subblocks A, B, C, D, E, F, and G in the reference picture.

[0241] The subblock-based template matching cost can be calculated based on the difference between the sub-template of the current sub-block and the sub-template of the corresponding reference sub-block. For example, the difference between the upper sub-template of the current sub-block D and the upper sub-template of the reference sub-block D ref can be calculated as the sub-block-based template matching cost.

[0242] The cost of subblock-based template matching, i.e., the difference between templates, can be defined as a cost function. Examples of cost functions that can be used include the Sum of Absolute Differences (SAD), the Sum of Absolute Transformed Differences (SATD), the Mean-Removed Sum of Absolute Differences (MR-SAD), the Mean Squared Error (MSE), or the Sum of Squared Error (SSE).

[0243] The image processing device can rearrange candidates in a sub-block-based candidate list using various methods based on sub-block-based template matching costs.

[0244] As an example, an image processing device may reorder K SbTMVP candidates based on a sub-block-based template matching cost. The K SbTMVP candidates may be listed in descending order of the sub-block-based template matching cost.

[0245] As another example, the image processing device can reorder L affine-based MVP candidates based on the sub-block-based template matching cost. The L affine-based MVP candidates can be listed in descending order of the sub-block-based template matching cost.

[0246] As another example, the image processing device can individually reorder the K SbTMVP candidates and the L affine-based MVP candidates based on the sub-block-based template matching costs. The K SbTMVP candidates can be reordered in descending order of their sub-block-based template matching costs, and the L affine-based MVP candidates can be listed in descending order of their sub-block-based template matching costs.

[0247] As another example, the image processing device can reconstruct the subblock-based merge candidate list by selecting T candidates from the candidates in the subblock-based merge candidate list based on subblock-based template matching costs and reordering the selected candidates based on the subblock-based template matching costs. Here, T is a positive integer less than or equal to K+L. Among the subblock-based merge candidates, T candidates having a small subblock-based template matching cost are selected, and the T candidates can be input into the subblock-based merge candidate list in the order of decreasing subblock-based template matching costs.

[0248] As another example, the image processing device can select U candidates from K SbTMVP candidates based on subblock-based template matching costs, select V candidates from L affine-based MVP candidates based on subblock-based template matching costs, and use the selected candidates as subblock-based merge candidates. Here, U is a positive integer less than or equal to K, and V is a positive integer less than or equal to L. The subblock-based merge candidate list includes U SbTMVP candidates and V affine-based MVP candidates.

[0249] In another embodiment, the reordering of candidates may be omitted. The image processing device may select a candidate at a specific position or order within the subblock-based merge candidate list and use the selected candidate to predict the current block.

[0250] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0251] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.

[0252] Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

[0253] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0254] CROSS-REFERENCE TO RELATED APPLICATION

[0255] This patent application claims priority to Korean Patent Application No. 10-2024-0045767, filed April 4, 2024, which is incorporated herein by reference in its entirety.

Claims

1. In the video decryption method, A step of deriving block-based movement information of the current block; A step of generating a first prediction block of the current block using the block-based motion information; A step of deriving sub-block-based motion information of the current block; A step of generating a second prediction block of the current block by predicting the current block in sub-block units using the sub-block-based motion information; and A step of generating a prediction block of the current block by performing a weighted sum of the first prediction block and the second prediction block. A method for decrypting an image including:

2. In paragraph 1, The weights of the first prediction block and the second prediction block are, A video decoding method, wherein the distance between the first reference picture used to generate the first prediction block and the current picture is determined based on the distance between the second reference picture used to generate the second prediction block and the current picture.

3. In paragraph 1, The weights of the first prediction block and the second prediction block are, A method for decoding an image, wherein a weight set is selected from predefined weight sets based on a signaled weight set index.

4. In paragraph 1, The weights of the first prediction block and the second prediction block are, An image decoding method determined based on a template difference between the first prediction block and the current block, and a template difference between the second prediction block and the current block.

5. In paragraph 1, A method for decoding an image, wherein a first reference picture used to generate the first prediction block and a second reference picture used to generate the second prediction block have different prediction directions.

6. In paragraph 1, The step of deriving block-based movement information of the current block above is: A step of generating a block-based motion candidate list of the current block based on the restored blocks; A step of determining a block-based motion candidate from the block-based motion candidate list based on a signaled block-based motion candidate index; and A step of determining block-based motion information of the current block using motion information of the block-based motion candidate. A method for decrypting an image including:

7. In paragraph 1, The step of deriving block-based movement information of the current block above is: A step of searching for a reference block based on a template matching cost within the restored area; and A step of determining block-based motion information of the current block using motion information of the reference block. A method for decrypting an image including:

8. In paragraph 1, The step of deriving sub-block-based motion information of the current block above is: A step of generating a sub-block-based motion candidate list of the current block based on blocks including the restored sub-blocks; and A step of determining sub-block-based motion information of the current block from the sub-block-based motion candidates based on a bilateral matching cost between blocks predicted using the sub-block-based motion candidates in the sub-block-based motion candidate list and the first prediction block. A method for decrypting an image including:

9. In paragraph 1, The step of deriving sub-block-based motion information of the current block above is: A step of generating a sub-block-based motion candidate list of the current block based on the restored blocks including sub-blocks; and A step of determining sub-block-based motion information of the current block from sub-block-based motion candidates in the sub-block-based motion candidate list based on a signaled sub-block-based motion candidate index. A method for decrypting an image including:

10. In paragraph 9, The above restored blocks are, An image decoding method comprising at least one of a spatial surrounding block, a temporal surrounding block, or a history-based surrounding block of the current block.

11. In paragraph 9, The above restored blocks are, A method for decoding an image, the method comprising at least one surrounding block referencing a collocated picture of the current picture including the current block.

12. In paragraph 9, The step of deriving sub-block-based motion information of the current block above is: A step of reordering the order of the sub-block-based motion candidates based on the template matching cost between the sub-templates of the current block and the sub-templates of the sub-block-based motion candidates. A method for decrypting an image further comprising:

13. In paragraph 12, The step of rearranging the order of the above sub-block-based motion candidates is: A step of rearranging the first sub-block-based motion candidates among the above sub-block-based motion candidates; and A step of rearranging the second sub-block-based motion candidates among the above sub-block-based motion candidates. A method for decrypting an image including:

14. In paragraph 12, The step of rearranging the order of the above sub-block-based motion candidates is: A step of selecting a preset number of candidates from the sub-block-based motion candidates based on a template matching cost between the sub-templates of the current block and the sub-templates of the sub-block-based motion candidates; and A step of reconstructing the sub-block-based motion candidate list including the preset number of candidates in order according to the template matching cost.

15. In paragraph 1, A step of correcting the block-based motion information and the sub-block-based motion information based on a bi-directional optical flow (BDOF) technique when the distances from the current picture to the first reference picture used to generate the first prediction block and the second reference picture used to generate the second prediction block are different. A method for decrypting an image further comprising:

16. In the video encoding method, A step of deriving block-based movement information of the current block; A step of generating a first prediction block of the current block using the block-based motion information; A step of deriving sub-block-based motion information of the current block; A step of generating a second prediction block of the current block by predicting the current block in sub-block units using the sub-block-based motion information; and A step of generating a prediction block of the current block by performing a weighted sum of the first prediction block and the second prediction block. A method of encoding an image including:

17. A method for transmitting data including a bitstream for an image, A step of generating a bitstream for the above image; and A step of transmitting data including the above bitstream Including, The step of generating the above bitstream is: A step of deriving block-based movement information of the current block; A step of generating a first prediction block of the current block using the block-based motion information; A step of deriving sub-block-based motion information of the current block; A step of generating a second prediction block of the current block by predicting the current block in sub-block units using the sub-block-based motion information; and A step of generating a prediction block of the current block by performing a weighted sum of the first prediction block and the second prediction block. How to include.

Images