Method for configuring candidate list for motion information prediction and apparatus using same

Abstract

Disclosed are a method for configuring a candidate list for motion information prediction and an apparatus using same. According to the disclosed method, in order to reduce the use of a zero motion vector candidate, a video encoder and a video decoder may fill at least some of the remaining slots of the candidate list by generating new artificial candidates from candidates already added to a candidate list.

Description

Method for constructing a candidate list for motion information prediction and device using the same

[0001] The present disclosure relates to encoding / decoding of images, and more particularly, to a method for constructing a candidate list for motion information prediction.

[0002] The content described below merely provides background information related to the present invention and does not constitute prior art.

[0003] Since video data has a large amount of data compared to voice data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.

[0004] Therefore, when storing or transmitting video data, an encoder is used to compress the video data and store or transmit it, and a decoder receives the compressed video data, decompresses it, and plays it back. Examples of such video compression technologies include H.264 / AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves encoding efficiency by about 30% compared to HEVC.

[0005] However, as the size, resolution, and frame rate of images are gradually increasing, and the amount of data that needs to be encoded is also increasing, a new compression technology that has better encoding efficiency and better image quality improvement than existing compression technologies is required.

[0006] In HEVC, VVC, or other video coding standards, if the candidate list is not filled with candidates of different types, such as spatial / temporal candidates, zero motion vector candidates may be added redundantly to the candidate list of Merge or AMVP. Redundant zero motion vector candidates reduce the diversity of candidates within the candidate list, which can affect the motion vector encoding efficiency.

[0007] The present disclosure presents several techniques for filling at least some of the remaining slots in the candidate list by generating new artificial candidates from candidates already added to the candidate list, in order to reduce the use of zero motion vector candidates.

[0008] One aspect of the present disclosure provides a method for coding video data. The method comprises the steps of: constructing a merge candidate list of a current block using at least one of a spatial candidate, a temporal candidate, and a history-based candidate; generating at least one artificial merge candidate based on motion information of merge candidates included in the merge candidate list when the merge candidate list is not full; and adding the at least one artificial merge candidate to the merge candidate list.

[0009] The method may include, prior to the step of generating at least one artificial merge candidate, a step of calculating, for each merge candidate in the merge candidate list, a matching cost between a set of surrounding pixels of a current block in a current picture and a set of surrounding pixels of a reference block in a reference picture, and a step of rearranging the order of each merge candidate based on the matching cost of the merge candidates in the merge candidate list.

[0010] Another aspect of the present disclosure provides a device for coding video data, comprising a memory for storing video data and at least one processor. The at least one processor may be configured to construct a merge candidate list of a current block using at least one of a spatial candidate, a temporal candidate, and a history-based candidate, and, if the merge candidate list is not full, generate at least one artificial merge candidate based on motion information of merge candidates included in the merge candidate list, and add the at least one artificial merge candidate to the merge candidate list.

[0011] Another aspect of the present disclosure discloses a method for providing video data to a video decoding device. The method includes the steps of encoding the video data into a bitstream and transmitting the bitstream to the video decoding device. The step of encoding the video data into a bitstream includes the steps of constructing a merge candidate list of a current block using at least one of a spatial candidate, a temporal candidate, and a history-based candidate; generating at least one artificial merge candidate based on motion information of merge candidates included in the merge candidate list when the merge candidate list is not full; and adding the at least one artificial merge candidate to the merge candidate list.

[0012] According to embodiments of the present disclosure, the encoding efficiency of motion vectors can be improved by reducing or eliminating duplication of zero motion vector candidates within a candidate list and improving the diversity of candidates within the candidate list.

[0013] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure.

[0014] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.

[0015] FIGS. 3A and 3B are diagrams illustrating multiple intra prediction modes, including wide-angle intra prediction modes.

[0016] Figure 4a is a conceptual diagram illustrating spatial surrounding blocks for the current block.

[0017] Figure 4b is a conceptual diagram illustrating positions for identifying temporal co-location blocks for the current block.

[0018] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure.

[0019] Figure 6 is a conceptual diagram illustrating a method for adding zero motion vectors to a merge candidate list.

[0020] FIGS. 7A and 7B are conceptual diagrams showing a method for constructing a merge candidate list including a new artificial candidate according to a first embodiment of the present disclosure.

[0021] FIGS. 8A and 8B are conceptual diagrams showing a method for constructing a merge candidate list involving candidate list reordering according to a first embodiment of the present disclosure.

[0022] FIG. 9 is a conceptual diagram showing a method for generating a new artificial candidate to be used in constructing a merge candidate list according to a second embodiment of the present disclosure.

[0023] FIGS. 10A and 10B are conceptual diagrams showing a method for generating new artificial candidates to be used in constructing a merge candidate list according to a third embodiment of the present disclosure.

[0024] FIGS. 11a, 11b and 11c are conceptual diagrams showing a method for generating new artificial candidates to be used in constructing a merge candidate list according to a fourth embodiment of the present disclosure.

[0025] FIGS. 12a and 12b are conceptual diagrams showing a method for generating new artificial candidates to be used in constructing a merge candidate list according to a fifth embodiment of the present disclosure.

[0026] FIGS. 13a, 13b and 13c are conceptual diagrams showing a method for generating new artificial candidates to be used in constructing a merge candidate list according to a sixth embodiment of the present disclosure.

[0027] FIG. 14 is a flowchart illustrating a method by which a video encoder or a video decoder codes (i.e., encodes or decodes) video data according to one aspect of the present disclosure.

[0028] Hereinafter, embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.

[0029] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image encoding device and its subcomponents will be described with reference to the illustration in FIG. 1.

[0030] The video encoding device may be configured to include a picture segmentation unit (110), a prediction unit (120), a subtractor (130), a transformation unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse transformation unit (165), an adder (170), a loop filter unit (180), and a memory (190).

[0031] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0032] A single image (video) is composed of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). Each CTU is then divided into one or more Coding Units (CUs) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks within a single slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). And, information commonly referenced by one or more SPS is encoded in a Video Parameter Set (VPS). In addition, information commonly applied to one tile or tile group may be encoded as syntax of a tile or tile group header. Syntaxes included in an SPS, PPS, slice header, tile or tile group header may be referred to as high level syntax.

[0033] The picture segmentation unit (110) determines the size of the CTU. Information about the size of the CTU (CTU size) is encoded as the syntax of SPS or PPS and transmitted to the image decoding device.

[0034] The picture segmentation unit (110) divides each picture constituting an image into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a CU, which is a basic unit of encoding.

[0035] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, or a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that mixes two or more of the QT structures, BT structures, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be combined and referred to as a MTT (Multiple-Type Tree).

[0036] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT structure.

[0037] As illustrated in FIG. 2, a CTU may first be split into a QT structure. The quadtree splitting may be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in the QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoding unit (155) and signaled to the image decoding device. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further split into one or more of the BT structure or the TT structure. There may be multiple splitting directions in the BT structure and / or the TT structure. For example, there may be two directions in which the block of the corresponding node is split horizontally and two directions in which the block is split vertically. As illustrated in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (Binary or Ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0038] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the CU split flag (split_cu_flag) value indicates that the node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is a basic unit of encoding. If the CU split flag (split_cu_flag) value indicates that the node is split, the video encoding device starts encoding from the first flag in the above-described manner.

[0039] As another example of a tree structure, when QTBT is used, there may be two types: a type that horizontally splits the block of the corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that vertically splits it (i.e., symmetric vertical splitting). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating the type of split are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may additionally be a type that splits the block of the corresponding node into two blocks of an asymmetrical shape. The asymmetric shape may include a shape that splits the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that splits the block of the corresponding node in a diagonal direction.

[0040] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." Depending on the QTBTTT partitioning employed, the current block may be rectangular as well as square.

[0041] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

[0042] In general, each current block within a picture can be predictively coded. Prediction of the current block can typically be performed using either intra-prediction (using data from the picture containing the current block) or inter-prediction (using data from a picture coded before the picture containing the current block). Inter-prediction encompasses both unidirectional and bidirectional prediction.

[0043] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in Fig. 3a, the multiple intra prediction modes may include two non-directional modes including the Planar mode and the DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently depending on each prediction mode.

[0044] For efficient directional prediction for a rectangular current block, directional modes (intra prediction modes 67 to 80 and -1 to -14) indicated by dotted arrows in Fig. 3b may be additionally used. These may be referred to as "wide-angle intra-prediction modes." In Fig. 3b, the arrows point to corresponding reference samples used for prediction, and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide-angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. At this time, among the wide-angle intra-prediction modes, some wide-angle intra-prediction modes available for the current block may be determined based on the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes (intra prediction modes 67 to 80) having an angle less than 45 degrees are available when the current block is a rectangular shape whose height is smaller than its width, and wide-angle intra prediction modes (intra prediction modes -1 to -14) having an angle greater than -135 degrees are available when the current block is a rectangular shape whose width is larger than its height.

[0045] The intra prediction unit (122) can determine an intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit (122) can encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra prediction unit (122) can calculate bit-rate distortion values ​​using rate-distortion analysis for multiple tested intra prediction modes and select an intra prediction mode with the best bit-rate distortion characteristics among the tested modes.

[0046] The intra prediction unit (122) selects one intra prediction mode from among multiple intra prediction modes and predicts the current block using surrounding pixels (reference pixels) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0047] The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for a block most similar to the current block within reference pictures that were encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture is generated. Generally, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma component and the chroma component. The motion information including information on the reference picture used to predict the current block and information on the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0048] The inter prediction unit (124) may perform interpolation on a reference picture or a reference block to improve prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed on the interpolated reference picture, the motion vector can be expressed up to a precision in decimal units rather than a precision in integer sample units. The precision or resolution of the motion vector can be set differently for each target region to be encoded, such as a slice, tile, CTU, CU, etc. When such adaptive motion vector resolution (AMVR) is applied, information on the motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on the motion vector resolution applied to each CU is signaled. Information on the motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.

[0049] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors indicating the block position most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture to generate a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged or weighted averaged to generate a prediction block for the current block. Then, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is transmitted to the entropy encoding unit (155). Here, reference picture list 0 may be composed of pictures that are before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures that are after the current picture in display order among the restored pictures. However, this is not necessarily limited to this, and restored pictures that are after the current picture in display order may be additionally included in reference picture list 0, and conversely, restored pictures that are before the current picture may be additionally included in reference picture list 1.

[0050] Various methods can be used to minimize the number of bits required to encode motion information.

[0051] For example, if the reference picture and motion vector of the current block are identical to those of a neighboring block, the motion information of the current block can be transmitted to the image decoding device by encoding information that can identify the neighboring block. This method is called 'merge mode'.

[0052] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.

[0053] As the surrounding blocks for deriving merge candidates, all or part of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block within the current picture may be used, as illustrated in FIG. 4a. In addition, a block located within a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, as illustrated in FIG. 4b, motion information of a block co-located with the current block within the reference picture (C0) or a block adjacent to the block at the co-located block (C1) may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.

[0054] The inter prediction unit (124) uses these surrounding blocks to construct a merge list containing a predetermined number of merge candidates. Among the merge candidates included in the merge list, the merge candidate to be used as motion information of the current block is selected and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the video decoding device.

[0055] Merge Skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy encoding are close to zero, only neighboring block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high encoding efficiency can be achieved for low-motion images, still images, and screen content images.

[0056] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0057] Another method for encoding motion information is Advanced Motion Vector Prediction (AMVP) mode.

[0058] In AMVP mode, the inter prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive predicted motion vector candidates, all or some of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block in the current picture as shown in FIG. 4A may be used. In addition, a block located in a reference picture other than the current picture in which the current block is located (which may or may not be the same as the reference picture used to predict the current block) may be used as the neighboring block used to derive predicted motion vector candidates. For example, a block co-located with the current block in the reference picture or blocks adjacent to the block in the co-located block may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.

[0059] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of these surrounding blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to produce a differential motion vector.

[0060] The predicted motion vector can be obtained by applying a predefined function (e.g., median, mean, etc.) to the predicted motion vector candidates. In this case, the image decoding device also knows the predefined function. In addition, since the surrounding blocks used to derive the predicted motion vector candidates are blocks that have already been encoded and decoded, the image decoding device also already knows the motion vectors of the surrounding blocks. Therefore, the image encoding device does not need to encode information to identify the predicted motion vector candidates. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.

[0061] Alternatively, the predicted motion vector can be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the differential motion vector and the reference picture used to predict the current block.

[0062] The subtractor (130) subtracts the prediction block generated by the intra prediction unit (122) or inter prediction unit (124) from the current block to generate a residual block.

[0063] The transformation unit (140) transforms residual signals within a residual block having pixel values ​​in a spatial domain into transform coefficients in a frequency domain. The transformation unit (140) may transform the residual signals within the residual block using the entire size of the residual block as a transformation unit, or may divide the residual block into a plurality of sub-blocks and use the sub-blocks as transformation units to perform the transformation. Alternatively, the residual signals may be transformed using only the transformation domain sub-block as a transformation unit by dividing the sub-blocks into two sub-blocks, that is, a transformation domain and a non-transform domain. Here, the transformation domain sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only a sub-block has been converted, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. In addition, the size of the conversion area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0064] Meanwhile, the transformation unit (140) can individually perform transformations on the residual block in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices can be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one transformation function pair with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) on the transformation function pair selected among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0065] The quantization unit (145) quantizes the transform coefficients output from the transform unit (140) using quantization parameters and outputs the quantized transform coefficients to the entropy encoding unit (155). The quantization unit (145) may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit (145) may also apply different quantization coefficients (scaling values) according to the positions of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to an image decoding device.

[0066] The rearrangement unit (150) can perform rearrangement of coefficient values ​​for quantized residual values.

[0067] The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from the DC coefficient to the coefficients of the high-frequency region by using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, the scanning method to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan.

[0068] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transform coefficients output from the rearrangement unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

[0069] In addition, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoding device can divide the block in the same manner as the image encoding device. In addition, the entropy encoding unit (155) encodes information about a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., information about an intra prediction mode) or inter prediction information (information about an encoding mode of motion information (merge mode or AMVP mode), a merge index in the case of a merge mode, and a reference picture index and a differential motion vector in the case of an AMVP mode) according to the prediction type. In addition, the entropy encoding unit (155) encodes information related to quantization, that is, information about a quantization parameter and information about a quantization matrix.

[0070] The inverse quantization unit (160) inversely quantizes the quantized transform coefficients output from the quantization unit (145) to generate transform coefficients. The inverse transform unit (165) transforms the transform coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

[0071] An adder (170) adds the restored residual block and the predicted block generated by the prediction unit (120) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting the next block.

[0072] The loop filter unit (180) performs filtering on restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation / quantization. The loop filter unit (180) may include all or part of a deblocking filter (182), a sample adaptive offset (SAO) filter (184), and an adaptive loop filter (ALF, 186) as an in-loop filter.

[0073] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-based encoding / decoding, and the SAO filter (184) and the ALF (186) perform additional filtering on the deblocking-filtered image. The SAO filter (184) and the ALF (186) are filters used to compensate for the differences between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in units of CTUs. In contrast, the ALF (186) performs block-based filtering, and compensates for distortion by applying different filters by distinguishing the edges and degrees of variation of the corresponding block. Information on filter coefficients to be used in the ALF can be encoded and signaled to an image decoding device.

[0074] The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in the memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0075] The video encoding device can store the bitstream of encoded video data on a non-transitory storage medium or transmit it to the video decoding device using a communication network.

[0076] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image decoding device and its subcomponents will be described with reference to FIG. 5.

[0077] The video decoding device may be configured to include an entropy decoding unit (510), a rearrangement unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).

[0078] Similar to the video encoding device of FIG. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0079] The entropy decoding unit (510) decodes the bitstream generated by the image encoding device to extract information related to block division, thereby determining the current block to be decoded, and extracts prediction information, information on residual signals, etc. required to restore the current block.

[0080] The entropy decoding unit (510) extracts information about the CTU size from the Sequence Parameter Set (SPS) or the Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and the CTU is divided using the tree structure by extracting division information about the CTU.

[0081] For example, when splitting a CTU using the QTBTTT structure, first, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, the second flag (mtt_split_flag) related to the splitting of MTT and the split direction (vertical / horizontal) and / or split type (binary / ternary) information are extracted, and the corresponding leaf nodes are split into the MTT structure. Accordingly, each node below the leaf nodes of QT are split recursively into the BT or TT structure.

[0082] As another example, when splitting a CTU using the QTBTTT structure, the CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and if the block is split, the first flag (QT_split_flag) may be extracted. During the splitting process, each node may undergo zero or more repeated QT splits followed by zero or more repeated MTT splits. For example, a CTU may undergo an MTT split right away, or conversely, may undergo only multiple QT splits.

[0083] As another example, when splitting a CTU using the QTBT structure, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Furthermore, for nodes corresponding to leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and splitting direction information are extracted.

[0084] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded by using the division of the tree structure, it extracts information on the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-prediction, the entropy decoding unit (510) extracts syntax elements for intra-prediction information (intra-prediction mode) of the current block. If the prediction type information indicates inter-prediction, the entropy decoding unit (510) extracts syntax elements for inter-prediction information, i.e., information indicating a motion vector and a reference picture referenced by the motion vector.

[0085] Additionally, the entropy decoding unit (510) extracts information about the quantized transform coefficients of the current block as information related to quantization and information about residual signals.

[0086] The rearrangement unit (515) can change the sequence of one-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the image encoding device.

[0087] The inverse quantization unit (520) inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantization unit (520) may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantized coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.

[0088] The inverse transform unit (530) inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a residual block for the current block.

[0089] In addition, when the inverse transform unit (530) inversely transforms only a portion of a transform block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only a sub-block of the transform block has been transformed, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block, and inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to restore residual signals, and fills “0” values ​​with residual signals for areas that have not been inversely transformed, thereby generating a final residual block for the current block.

[0090] In addition, when MTS is applied, the inverse transform unit (530) determines a transform function or a transform matrix to be applied in the horizontal and vertical directions using MTS information (mts_idx) signaled from the image encoding device, and performs inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.

[0091] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is inter prediction.

[0092] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.

[0093] The inter prediction unit (544) uses the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510) to determine the motion vector of the current block and the reference picture referenced by the motion vector, and predicts the current block using the motion vector and the reference picture.

[0094] An adder (550) adds the residual block output from the inverse transform unit (530) and the predicted block output from the inter prediction unit (544) or the intra prediction unit (542) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting a block to be decoded later.

[0095] The loop filter unit (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as in-loop filters. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter (564) and the ALF (566) perform additional filtering on restored blocks after deblocking filtering to compensate for differences between restored pixels and original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about filter coefficients decoded from the non-stream.

[0096] The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in the memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0097] Hereinafter, improved coding techniques that can be performed by the aforementioned video encoder (e.g., the video encoding device illustrated in FIG. 1) or video decoder (e.g., the video decoding device illustrated in FIG. 5) are disclosed. In particular, the techniques of the present disclosure relate to inter prediction. As described above, in HEVC, VVC, or other video coding standards, motion information can be predictively encoded before being stored in a bitstream. Motion information for a current block can be predictively encoded based on motion information of one or more blocks neighboring the current block. Motion information prediction techniques can include advanced motion vector prediction (AMVP) and merge.

[0098] When encoding a motion vector using AMVP, a video encoder can encode the motion vector as the difference between the motion vector of the current block and a motion vector predictor (MVP). The encoder and the decoder can generate or determine a list of MVP candidates. The encoder can determine or select an MVP from the list of MVP candidates. The MVP candidates can include previously decoded motion vectors of neighboring blocks of the current block in the current picture and / or previously decoded motion vectors of blocks at or near the same position as the current block in other reference pictures.

[0099] The encoder can transmit an indication of the selected MVP and / or motion vector difference (MVD) through the bitstream. The encoder can use an index to indicate the selected MVP in the bitstream. The MVD can be determined or calculated based on the difference between the motion vector of the current block and the selected MVP. For example, for a motion vector indicating a relative position with respect to the position of the current block (e.g., represented by a horizontal component MVx and a vertical component MVy), the MVD can be expressed by two components MVDx and MVDy. MVDx and MVDy can be determined or calculated as follows.

[0100]

[0101] MVDx and MVDy represent the horizontal and vertical components of the MVD, respectively. MVPx and MVPy represent the horizontal and vertical components of the MVP, respectively. A video decoder can decode a motion vector by adding the MVD to the MVP indicated through the bitstream.

[0102] The decoder can decode the current block by determining a reference block. The decoder can determine the reference block based on the decoded motion vector. The reference block is considered a prediction of the current block or corresponds to the prediction of the current block. The decoder can decode the current block by combining the prediction and the residual (sample by sample).

[0103] In HEVC, VVC, or other video coding standards, the MVP candidate list of AMVP may include two or more candidates (e.g., candidates A and B). For example, candidates A and B may include the following types of candidates:

[0104] (1) At most two (or other number) spatial MVP candidates derived from five (or other number) spatial neighboring blocks of the current block being coded.

[0105] (2) (If both spatial MVP candidates are not available or are identical,) one (or other number of) temporal MVP candidates derived from two (or other number of) temporal co-located blocks.

[0106] (3) Zero motion vector candidates (if either or both spatial MVP candidates or temporal MVP candidates are not available).

[0107] As illustrated in Fig. 4a, for example, five spatial candidate neighboring blocks can be positioned relative to the current block. The five spatial candidate neighboring blocks can be A0, A1, B0, B1, and B2. As illustrated in Fig. 4b, two temporal co-located blocks can be positioned relative to the current block. The two temporal co-located blocks can be C0 and C1. The two temporal co-located blocks can be in a reference picture different from the current picture of the current block. The motion vector of the temporal MVP candidate can be obtained by scaling the motion vector of the co-located block based on the POC distances. The reference picture index of the temporal MVP candidate can be set equal to 0.

[0108] The encoder may encode the motion vector of the current block using merge mode. In merge mode, the encoder may reuse motion information of a neighboring block (e.g., one of neighboring blocks A0, A1, B0, B1, and B2) for inter prediction of the current block. In merge mode, the encoder may also reuse motion information of a temporally co-located block (e.g., one of temporally co-located blocks C0 and C1) for inter prediction of the current block. The reference picture list to be used for deriving the temporally co-located blocks may be explicitly signaled in the slice header. Since the motion information of the neighboring blocks or the temporally co-located blocks can be used for the current block, there is no need to indicate an MVD for the current block. Since there is no need to indicate an MVD for the current block, the signaling overhead for sending / signaling the motion information of the current block is reduced.

[0109] The encoder and decoder can generate a candidate list of motion information from neighboring blocks or temporally co-located blocks of the current block, in a manner similar to AMVP. The encoder can decide to predict the motion information of the current block being coded using the motion information of one neighboring block or temporally co-located block from the candidate list. The encoder can signal an index indicating the determined motion information within the candidate list.

[0110] In HEVC, VVC, or other video coding standards, a candidate list for merge mode may be constructed by including, in order, the following different types of candidates:

[0111] (1) At most 4 (or another number) spatial merge candidates derived from 5 (or another number) spatial surrounding blocks (e.g., as shown in Fig. 4a).

[0112] (2) One (or other number) of temporal merge candidates derived from two (or other number) of temporally co-located blocks (e.g., as shown in Fig. 4b).

[0113] (3) Up to four (or other number) history-based MVP (HMVP) merge candidates from the FIFO table.

[0114] (4) Pairwise Average MVP (PA-MVP) merge candidates

[0115] (5) Zero motion vector candidates

[0116] The spatial neighboring blocks and temporal co-located blocks used in merge mode may be identical to the spatial neighboring blocks and temporal co-located blocks used in AMVP.

[0117] HMVP merge candidates can be added to the merge candidate list after the spatial merge candidates and temporal merge candidates. For example, the motion information of previously coded blocks (i.e., blocks adjacent or not adjacent to the current block) can be stored in a first-in-first-out (FIFO) table and used as the MVP for the current block. A FIFO table containing multiple HMVP candidates is maintained during the encoding and decoding processes. Whenever an inter-coded block exists, the associated motion information is added to the last entry of the FIFO table as a new HMVP candidate. The motion vectors of the last four (or other number) HMVP candidates in the table can be inserted into the candidate list after the temporal merge candidates.

[0118] Pairwise average candidates can be generated by averaging predefined pairs of candidates in an existing merge candidate list. The predefined pairs can be defined as {(0, 1), (0, 2), (1, 2), (0, 3), (1, 3), (2, 3)}, where the numbers represent merge indices for the merge candidate list. The averaged motion vectors are computed separately for each reference picture list.

[0119] If the merge candidate list is not complete after the pairwise mean candidates are added, the encoder and decoder insert zero motion vector candidates at the end of the candidate list to fill the remaining slots in the candidate list.

[0120] In the VVC standard, we further explore how to add zero motion vector candidates to the candidate list. Figure 6 is a conceptual diagram illustrating a method for adding zero motion vector candidates to the merge candidate list.

[0121] First, as in Step 1 of Fig. 6, the video encoder and the video decoder repeatedly add zero motion vectors (ZR0, ZR1, ZR2, ZR3 in Fig. 6) to the merge candidate list while increasing the reference picture index of the zero motion vector by 1 each time until the number of added zero motion vectors does not exceed the minimum number of reference pictures of the current slice (4 in the example of Fig. 6). The minimum number of reference pictures of the current slice, numRefIdx, is determined by the following equation.

[0122]

[0123] Here, numRef_L0 is the number of reference pictures in the reference picture list L0, and numRef_L1 is the number of reference pictures in the reference picture list L1. As in Equation 2, if the current slice is a B-slice, numRefIdx is the minimum value between numRef_L0 and numRef_L1, and if it is not a B-slice, numRefIdx is numRef_L0.

[0124] Next, if the number of added zero motion vectors exceeds the minimum number of reference pictures of the current slice, as in Step 2 of FIG. 6, the encoder and decoder add zero motion vectors (ZR4, ZR5 in the example of FIG. 6) to the merge candidate list until the merge candidate list is full, while fixing the reference picture index of the zero motion vector to 0.

[0125] Likewise, in HEVC, VVC, or other video coding standards, if the candidate list is not filled with candidates of different types, such as spatial / temporal candidates, zero motion vector candidates may be added redundantly to the candidate list of Merge or AMVP. That is, the candidate list may have multiple zero motion vector candidates with different or identical reference picture indices. Redundant zero motion vector candidates reduce the diversity of candidates within the candidate list, which may affect the motion vector encoding efficiency.

[0126] The present disclosure presents several techniques for generating new artificial candidates from candidates already included in the candidate list (e.g., spatial / temporal candidates, HMVP candidates, and / or PA-MVP candidates) to fill remaining slots in the candidate list. These techniques can improve motion vector encoding efficiency by reducing or eliminating the redundancy of zero motion vector candidates in the candidate list and increasing the diversity of candidates within the candidate list. The new artificial candidates can be used in addition to candidates of different types, and in some cases, they can be used as substitutes for PA-MVP candidates.

[0127] Although the following description is given from the perspective of generating artificial merge candidates to be added to the merge candidate list, it should be understood that the same or equivalent methods can be applied to generating artificial MVP candidates to be added to the MVP candidate list in AMVP mode.

[0128] In a first embodiment, when an available spatial / temporal candidate (or HMVP candidate, or PA-MVP candidate) in the merge candidate list uses bidirectional prediction, two unidirectional prediction merge candidates can be generated from such candidates. In other words, the encoder and decoder can split the motion information of the bidirectional prediction merge candidate in the current merge candidate list into L0 motion information and L1 motion information, and add each piece of motion information as a new unidirectional prediction merge candidate to the merge candidate list.

[0129] FIGS. 7A and 7B are conceptual diagrams showing a method for constructing a merge candidate list including a new artificial candidate according to a first embodiment of the present disclosure.

[0130] The encoder and decoder can insert or add various types of merge candidates, such as spatial MVP candidates, temporal MVP candidates, HMVP candidates, or PA-MVP candidates, to the (initial) merge candidate list (step 1). In the examples of Figures 7a and 7b, the candidates inserted into the (initial) merge candidate list in step 1 are m0, m1, m2, m3, m4, and m5.

[0131] The encoder and decoder can identify one or more bidirectional prediction merge candidates among the candidates inserted in step 1, and generate two unidirectional prediction merge candidates by splitting the L0 motion information and L1 motion information of the identified bidirectional prediction merge candidates (step 2). In the examples of Figs. 7a and 7b, the merge candidate list includes the bidirectional prediction merge candidate m0, from which two new unidirectional prediction merge candidates are generated.

[0132] As illustrated in FIG. 7a, the generated unidirectional prediction merge candidates can be added to the merge candidate list before adding zero motion vector candidates (step 3). The zero motion vector candidates can be inserted at the end of the candidate list to fill the remaining slots in the candidate list. In some embodiments, as illustrated in FIG. 7b, the generated unidirectional prediction merge candidates can be added to the merge candidate list after adding (e.g., a fixed number of zero motion vector candidates, such as 1, 2, or 3) zero motion vector candidates (step 3). Notwithstanding the relative addition order between the unidirectional prediction merge candidates and the zero motion vector candidates, the zero motion vector candidates can be used as an auxiliary when the bidirectional prediction merge candidates do not exist or when artificial candidates generated therefrom alone cannot fill the remaining slots in the candidate list.

[0133] During the process of constructing the merge candidate list, the merge candidates may be reordered based on predefined criteria. For example, the merge candidates may be reordered based on whether the candidate is a bidirectional prediction candidate. In this example, bidirectional prediction candidates may be reordered to be placed before or after unidirectional prediction candidates. The merge candidates may be reordered in ascending order based on their template matching costs. The template matching cost may be calculated by the sum of absolute differences (SAD) or the sum of absolute transformed differences (SATD) between the template samples of the current block and their corresponding reference samples.

[0134] The template of the current block may include a set of reconstructed (top and left) samples neighboring the current block. The reference samples (or simply "templates of the reference block") in the reference picture of the template may be identified by the motion vector of the merge candidate relative to the template of the current block. When the merge candidate uses bidirectional prediction, the template of the reference block for reference picture list 0 may be identified by the motion vector of the merge candidate in reference picture list 0 relative to the template of the current block in the current picture. Similarly, the template of the reference block for reference picture list 1 may be identified by the motion vector of the merge candidate in reference picture list 1 relative to the template of the current block in the current picture.

[0135] The candidate list reordering process may be performed before and / or after generating new artificial candidates in the candidate list.

[0136] For example, as illustrated in FIG. 8a, the encoder and decoder can add merge candidates of types such as spatial candidates, temporal candidates, HMVP, or PA-MVP to the merge candidate list (step 1), and then perform a candidate list reordering process (step 2). After the candidate list reordering process, the encoder and decoder can identify one or more bidirectional prediction merge candidates among the candidates present in the merge candidate list, and generate unidirectional prediction merge candidates from each bidirectional prediction merge candidate (step 3). In the example of FIG. 8a, new unidirectional prediction merge candidates n0, n1 are generated from the bidirectional prediction merge candidate m0, and four new unidirectional prediction merge candidates n2, n3, n4, n5 are generated from the other two bidirectional prediction merge candidates. After generating the unidirectional prediction merge candidates, the encoder and decoder can also perform the candidate list reordering process again. For example, the encoder and decoder can calculate the template matching cost of new candidates and add the new candidates to the merge candidate list in an order based on the template matching cost (step 4).

[0137] In the method illustrated in Fig. 8a, the zero motion vector candidate is excluded in the candidate list reordering process, and thus the zero motion vector candidate is positioned at the end of the candidate list. In contrast, in the method illustrated in Fig. 8b, even the zero motion vector candidate is reordered in the candidate list reordering process, and the candidate with the highest template matching cost is positioned at the end of the candidate list.

[0138] The candidate list reordering process described in connection with the first embodiment and the additional method of inserting new artificial candidates into the candidate list can also be applied to other embodiments described below. One or more merge candidates to be used to generate new artificial candidates can be selected according to a predefined rule or order based on the indices assigned to the merge candidates through the reordering process.

[0139] FIG. 9 is a conceptual diagram showing a method for generating a new artificial candidate to be used in constructing a merge candidate list according to a second embodiment of the present disclosure.

[0140] In a second embodiment, the encoder and decoder can generate a bidirectional prediction merge candidate from available unidirectional prediction merge candidates in the merge candidate list for the current block belonging to slice B. The generated bidirectional prediction merge candidate can be added to the remaining slots in the merge candidate list as a new merge candidate.

[0141] For example, if the merge candidate is a unidirectional prediction with L0, a bidirectional prediction candidate is generated based on the combination of the motion vector mv0 of the unidirectional prediction merge candidate and a new vector mv1 in the opposite direction to mv0. The new vector mv1 of L1 is obtained by scaling the motion vector mv0 of L0. The scaling factor is calculated based on the POC distance of the reference pictures of L0 and L1 to the current picture. The new vector mv1 of L1 can be calculated using the following mathematical formula.

[0142]

[0143] Here, POC curr is the current picture's POC, POC L0 is the POC of the L0 reference picture, POC L1 is the POC of the L1 reference picture, mv 0_x Wow mv 0_y is the x-component and y-component of the L0 motion vector, mv1_x Wow mv 1_y represents the x-component and y-component of the L1 motion vector.

[0144] The reference picture in L1 for a new bidirectional prediction candidate can be selected or determined in one of the following ways: (1) using a predefined value (e.g., 0, 1, 2, etc.) as the reference picture index in L1 (refIdxL1); (2) using the same value as the reference picture index (refIdxL0) of the unidirectional prediction candidate as the reference picture index in L1; (3) using a reference picture in L1 that is mirrored (i.e., has the same POC distance to the current picture) for the reference picture of the unidirectional prediction candidate; (4) using a reference picture in L1 that is closest to the current picture.

[0145] As another example, if the unidirectional prediction merge candidate is a unidirectional prediction with L1, the L1 motion information of the bidirectional prediction merge candidate is identical to the motion information of the unidirectional prediction merge candidate, and the L0 motion information of the bidirectional prediction merge candidate is composed of a scaled vector obtained from a reference picture selected or determined from the reference picture list L0 and the motion vector of the unidirectional unidirectional prediction merge candidate.

[0146] FIGS. 10A and 10B are conceptual diagrams showing a method for generating new artificial candidates to be used in constructing a merge candidate list according to a third embodiment of the present disclosure.

[0147] In a third embodiment, the encoder and decoder can generate a bidirectional prediction merge candidate from a pair of available unidirectional prediction merge candidates in a merge candidate list for a current block belonging to a B slice, and add the generated bidirectional prediction merge candidate as a new merge candidate to the merge candidate list. The pair of available unidirectional prediction merge candidates can be sequentially selected according to the order of the merge index (after a candidate list reordering process).

[0148] As illustrated in Fig. 10a, if the first merge candidate is a unidirectional prediction using L0 and the second merge candidate is a unidirectional prediction using L1, the first merge candidate can be used as the L0 motion vector of the new bidirectional prediction candidate, and the second merge candidate can be used as the L1 motion vector of the new bidirectional prediction candidate.

[0149] When a pair of unidirectional predictive merge candidates use the same reference picture list, a method of reversing the direction of the motion vector of one of the two merge candidates (e.g., the candidate with the larger merge index) can be used.

[0150] For example, as illustrated in FIG. 10b, if both the first merge candidate and the second merge candidate are unidirectional predictions using L0, the motion information of the first merge candidate is directly used as the L0 motion information of the new bidirectional prediction candidate. Then, the modified motion information of the second merge candidate, which is obtained by changing the sign of the motion vector of the second merge candidate and changing the reference picture list from L0 to L1, is used as the L1 motion information of the new bidirectional prediction candidate. Here, the reference picture in L1 mirrored for the L0 reference picture of the second merge candidate can be used as the reference picture of the L1 motion vector of the new bidirectional prediction candidate. If the reference picture in the mirrored L1 does not exist, the reference picture in L1 closest to the current picture can be used, in which case the L1 motion vector of the bidirectional prediction candidate can be derived by scaling the motion vector (mv1) of the second merge candidate based on the POC distances.

[0151] FIGS. 11a, 11b and 11c are conceptual diagrams showing a method for generating new artificial candidates to be used in constructing a merge candidate list according to a fourth embodiment of the present disclosure.

[0152] In a fourth embodiment, the encoder and decoder can generate a new merge candidate by changing the reference picture of an available merge candidate in the merge candidate list. The reference picture index of the new merge candidate can be, for example, (1) a predefined value (e.g., 0, 1, 2, etc.), or (2) a value obtained by adding or multiplying a predefined value (e.g., 0, 1, 2, etc.) to the reference picture index of the original merge candidate.

[0153] In one embodiment, as illustrated in FIG. 11a, the new merge candidate may use the motion vector of the original merge candidate as is, despite the reference picture change.

[0154] In another embodiment, as illustrated in FIG. 11b, the motion vector of the new merge candidate can be derived by scaling the motion vector of the original merge candidate based on the POC distances between the current picture and the (original and new) reference pictures.

[0155] In another embodiment, a motion vector corresponding to the POC of a reference picture that has been changed can be derived in the following manner, as described with reference to FIG. 11c.

[0156] Step 1: From at least one merge candidate already existing in the merge candidate list, linear or quadratic equations representing or approximating the relationship between a position ([x], [y]) and a POC can be derived. Fig. 11c shows graphs of linear and quadratic functions passing through points representing the position of a current block in a current picture, the position of a reference block in an L0 reference picture pointed to by an L0 merge candidate, and the position of a reference block in an L1 reference picture pointed to by an L1 merge candidate in the [x]-POC plane. In Fig. 11c, (a) is a graph illustrating a case where the relationship between the [x] component of a position and the POC is linear, and (b) is a graph illustrating a case where the relationship between the [x] component of a position and the POC is quadratic. Similarly, one can consider a graph of a linear or quadratic function passing through three points in the [y]-POC plane.

[0157] Step 2: From the derived relations, the position ([x1], [y1]) corresponding to the POC of the changed reference picture (indicated as L0' in Fig. 11c) can be predicted or calculated.

[0158] Step 3: The difference between the position ([x1], [y1]) predicted in Step 2 and the position ([x0], [y0]) of the current block in the current picture, i.e., the displacement vector (horizontal components and vertical components) representing the displacement from ([x0], [y0]) to ([x1], [y1]), is calculated, and this displacement vector can be used as a motion vector of a new merge candidate corresponding to the POC of the changed reference picture.

[0159] FIGS. 12a and 12b are conceptual diagrams showing a method for generating new artificial candidates to be used in constructing a merge candidate list according to a fifth embodiment of the present disclosure.

[0160] In the fifth embodiment, the encoder and decoder can generate pairwise merge candidates by applying weights in various ways to the original merge candidate pairs that have already been inserted, and add the generated pairwise merge candidates to the merge candidate list as new merge candidates.

[0161] In one embodiment, the encoder and decoder can generate a new merge candidate as a position vector of a point that internally or externally divides two motion vectors from a pair of merge candidates using the same reference picture by a predefined ratio (e.g., 3:1, 1:3, 5:3, 7:1, etc.). FIG. 12A illustrates a position vector of a point that internally divides a first motion vector and a second motion vector by 3:1 and a position vector of a point that internally divides a first motion vector and a second motion vector by 1:3 (represented by a dotted arrow).

[0162] In another embodiment, the ratio of internalization or externalization (i.e., internalization / externalization) can be determined or selected from a plurality of predefined ratios based on information of two motion vectors. The information of the vectors can be the magnitude of the vector (L1 norm, L2 norm, x component, y component, etc.), the ratio of vector components, the difference between the two vectors (L1 norm, L2 norm, etc.), the angle between the two vectors (cosθ, sinθ, etc.), the template cost (SAD, SATD, etc.), etc.

[0163] For example, the ratio of internal or external division may be determined based on whether the angle between two motion vectors is less than a threshold value. If the angle between two motion vectors is less than the threshold value, it can be seen that the two motion vectors are facing a similar direction. Therefore, if the angle is less than the threshold value, it is advantageous to increase the diversity of the merge candidate list to determine the position vector (represented by the dotted arrow) of the point that "externally divides" the two motion vectors into an m:n ratio as a new merge candidate, as illustrated in (a) of Fig. 12b. On the other hand, if the angle is greater than or equal to the threshold value, a new merge candidate can be generated based on the position vector (represented by the dotted arrow) of the point that "internally divides" the two motion vectors, as illustrated in (b) of Fig. 12b.

[0164] In another embodiment, the encoder and decoder can add a new motion vector obtained by weighting two motion vectors corresponding to a pair of merge candidates using the same reference picture to the merge candidate list as a new merge candidate. Here, the weights used in the weighted-sum operation can be determined or selected from a plurality of predefined weights (based on the aforementioned information of the two motion vectors).

[0165] FIGS. 13a, 13b and 13c are conceptual diagrams showing a method for generating new artificial candidates to be used in constructing a merge candidate list according to a sixth embodiment of the present disclosure.

[0166] In a sixth embodiment, the encoder and decoder can generate a new merge candidate by adding or subtracting an offset to a horizontal component or a vertical component of a motion vector of an available merge candidate in the merge candidate list.

[0167] In one embodiment, a predefined offset (e.g., having a value of 4, 8, 16, or 32) may be used. Figure 13a illustrates four new motion vectors (represented by dashed arrows) generated from the motion vector of the original merge candidate. The new motion vectors are generated by adding 8 to the horizontal component of the original motion vector, subtracting 8 from the horizontal component, adding 8 to the vertical component, and subtracting 8 from the horizontal component.

[0168] In another embodiment, the offset can be selected or determined based on the information of the current block, for example, from a list of predefined available offsets, such as {4, 8, 16, 32}. The information of the current block can be width, width or height, aspect ratio, position, etc. Different offsets can also be added to or subtracted from the horizontal and vertical components of the motion vector, respectively. For example, as illustrated in (a) and (b) of FIG. 13b, the motion vectors of artificial candidates (represented by dotted arrows) can be obtained by adding or subtracting a horizontal offset equal to the width of the current block to the horizontal component of the motion vector, and by adding or subtracting a vertical offset equal to the height of the current block to the vertical component of the motion vector. (c) of FIG. 13b shows an example in which the offset is determined according to the position of the current block. In (c) of FIG. 13b, W and H represent the width and height of the current picture, respectively, and (x, y) represents the position of the current block. At this time, the offset can be determined as the simplest integer ratio of x:Wx and y:Hy. For example, in the case of the position (104,120) in a 416x480 image, the offset can be determined as (-1,0), (3,0), (0,1), (0,-1).

[0169] In another embodiment, the offset can be determined based on information of the motion vector of the original merge candidate. Here, the information of the motion vector can be a magnitude (L1 norm, L2 norm, x component, y component, etc.), a ratio of vector components, etc. For example, as illustrated in (a) and (b) of FIG. 13c, by adding or subtracting a horizontal offset having the same magnitude as (or proportional to) the horizontal component of the motion vector of the original merge candidate to the horizontal component of the original motion vector, and by adding or subtracting a vertical offset having the same magnitude as (or proportional to) the vertical component of the original motion vector to the vertical component of the original motion vector, motion vectors of artificial candidates (represented by dotted arrows) can be obtained.

[0170] FIG. 14 is a flowchart illustrating a method by which a video encoder or a video decoder codes (i.e., encodes or decodes) video data according to one aspect of the present disclosure.

[0171] The video encoder and video decoder can construct a merge candidate list of the current block using at least one of a spatial candidate, a temporal candidate, or a history-based candidate (S1410).

[0172] The video encoder and the video decoder may generate at least one artificial merge candidate based on motion information of merge candidates included in the merge candidate list, if the merge candidate list is not full (S1430). The video encoder and the video decoder may add the generated artificial merge candidate to the merge candidate list (S1440).

[0173] The video encoder can encode an indication indicating a merge candidate selected from a candidate list, and encode the current block using the selected merge candidate. The video decoder can decode an indication indicating a merge candidate selected from the candidate list, and decode the current block using the selected merge candidate (S1450).

[0174] The video encoder and the video decoder may perform a merge candidate reordering process based on a template matching cost before generating at least one artificial merge candidate (S1420). For example, the video encoder and the video decoder may calculate, for each merge candidate in the merge candidate list, a matching cost between a set of surrounding pixels of a current block in a current picture (i.e., a template of the current block) and a set of surrounding pixels of a reference block in a reference picture (i.e., a template of the reference block). The video encoder and the video decoder may reorder the order of each merge candidate based on the matching cost of the merge candidates in the merge candidate list. One or more original merge candidates to be used for generating new artificial candidates may be selected according to a predefined rule or order based on indices assigned to the original merge candidates through the reordering process.

[0175] In S1430, at least one of the various methods described above may be used to generate at least one artificial merge candidate.

[0176] For example, the video encoder and the video decoder can identify a bidirectional prediction candidate among the merge candidates included in the merge candidate list, and generate two artificial merge candidates including a unidirectional prediction candidate having L0 motion information of the identified bidirectional prediction candidate and a unidirectional prediction candidate having L1 motion information of the identified bidirectional prediction candidate.

[0177] As another example, the video encoder and the video decoder may select a unidirectional prediction merge candidate from among the merge candidates included in the merge candidate list, and generate a new bidirectional prediction merge candidate from the selected unidirectional prediction merge candidate. One of the L0 motion information and the L1 motion information of the new bidirectional prediction merge candidate may be identical to the motion information of the selected unidirectional prediction merge candidate, and the other may be obtained by mirroring the motion information of the selected unidirectional prediction merge candidate (i.e., by changing the reference picture list and scaling the motion vector if necessary).

[0178] As another example, the video encoder and the video decoder may select a pair of unidirectional prediction merge candidates in a predefined order from among the merge candidates included in the merge candidate list, and generate a new bidirectional prediction merge candidate from the pair of selected unidirectional prediction merge candidates. Here, if the pair of selected unidirectional prediction merge candidates use different reference picture lists, the motion information of each of the pair of selected unidirectional prediction merge candidates may be used as L0 motion information and L1 motion information of the new bidirectional prediction merge candidate. If the pair of selected unidirectional prediction merge candidates use the same reference picture list, the L0 motion information and the L1 motion information of the new bidirectional prediction merge candidate may be composed of (1) mirrored motion information of the motion information of one of the pair of selected unidirectional prediction merge candidates and (2) the motion information of the other.

[0179] As another example, the video encoder and the video decoder may select a unidirectional prediction merge candidate from among the merge candidates included in the merge candidate list, and modify motion information of the selected unidirectional prediction merge candidate to generate a new unidirectional prediction merge candidate that uses a different reference picture from the selected unidirectional prediction merge candidate.

[0180] As another example, the video encoder and the video decoder may select a pair of unidirectional prediction merge candidates that use the same reference picture from among the merge candidates included in the merge candidate list, derive a position vector of a point that internally or externally divides two motion vectors from the pair of selected unidirectional prediction merge candidates by a predefined ratio, and generate a new unidirectional prediction merge candidate that uses the same reference picture as the pair of unidirectional prediction merge candidates but has the derived position vector as a motion vector.

[0181] As another example, the video encoder and the video decoder may select a unidirectional prediction merge candidate from among the merge candidates included in the merge candidate list, and modify motion information of the selected unidirectional prediction merge candidate to generate a new unidirectional prediction merge candidate that uses a different motion vector from the selected unidirectional prediction merge candidate. Here, the motion vector of the new unidirectional prediction merge candidate may be derived by adding or subtracting a predefined offset to a horizontal component or a vertical component of the motion vector of the unidirectional prediction merge candidate.

[0182] Although the method of FIG. 15 is illustrated in terms of generating artificial merge candidates to be added to the merge candidate list, it should be understood that the same or corresponding methods can be applied to generating artificial MVP candidates to be added to the MVP candidate list when coding motion vectors in the AMVP mode. That is, according to another aspect of the present disclosure, a video encoder and a video decoder can construct an MVP candidate list of a current block using at least one of a spatial candidate and a temporal candidate. The video encoder and the video decoder can generate at least one artificial MVP candidate based on motion information of the MVP candidates included in the MVP candidate list, when the MVP candidate list is not full. The video encoder and the video decoder can add the generated artificial MVP candidate to the MVP candidate list.

[0183] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0184] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.

[0185] Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

[0186] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0187]

[0188]

[0189] CROSS-REFERENCE TO RELATED APPLICATION

[0190] This patent application claims priority to Korean patent application No. 10-2023-0173199, filed in Korea on December 4, 2023, and Korean patent application No. 10-2024-0161098, filed in Korea on November 13, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. A method for encoding video data, A step of constructing a merge candidate list of the current block using at least one of a spatial candidate, a temporal candidate, or a history-based candidate; If the above merge candidate list is not full, a step of generating at least one artificial merge candidate based on the movement information of the merge candidates included in the above merge candidate list; and A step of adding at least one artificial merge candidate to the merge candidate list. A method comprising:

2. In paragraph 1, Prior to the step of generating at least one artificial merge candidate, For each merge candidate in the above merge candidate list, a step of calculating a matching cost between a set of surrounding pixels of a current block in a current picture and a set of surrounding pixels of a reference block in a reference picture; A step of rearranging the order of each merge candidate based on the matching cost of the merge candidates in the above merge candidate list; A method comprising:

3. In paragraph 1, The step of generating at least one artificial merge candidate comprises: A step of identifying a bidirectional prediction candidate among the merge candidates included in the above merge candidate list; and A step of generating a new unidirectional prediction candidate having L0 motion information of the identified bidirectional prediction candidate and a new unidirectional prediction candidate having L1 motion information of the identified bidirectional prediction candidate. A method comprising:

4. In paragraph 1, The step of generating at least one artificial merge candidate comprises: A step of selecting a unidirectional prediction merge candidate from among the merge candidates included in the above merge candidate list; Step of generating a new bidirectional prediction merge candidate from the selected unidirectional prediction merge candidate. A method, characterized in that one of the L0 motion information and the L1 motion information of the new bidirectional prediction merge candidate is identical to the motion information of the selected unidirectional prediction merge candidate, and the other is obtained by mirroring the motion information of the selected unidirectional prediction merge candidate.

5. In paragraph 1, The step of generating at least one artificial merge candidate comprises: A step of selecting a pair of unidirectional predictive merge candidates in a predefined order from among the merge candidates included in the above merge candidate list; and A step of generating a new bidirectional prediction merge candidate from the above pair of selected unidirectional prediction merge candidates. A method, characterized by including:

6. In paragraph 5, A method, characterized in that when the above pair of selected unidirectional prediction merge candidates use different reference picture lists, the motion information of each of the above pair of selected unidirectional prediction merge candidates is used as L0 motion information and L1 motion information of the new bidirectional prediction merge candidate.

7. In paragraph 5, A method wherein, when the above pair of selected unidirectional prediction merge candidates use the same reference picture list, the L0 motion information and the L1 motion information of the new bidirectional prediction merge candidate are composed of (1) mirrored motion information of one of the above pair of selected unidirectional prediction merge candidates and (2) motion information of the other.

8. In paragraph 1, The step of generating at least one artificial merge candidate comprises: A step of selecting a unidirectional prediction merge candidate from among the merge candidates included in the above merge candidate list; and A step of modifying the motion information of the above-mentioned selected unidirectional prediction merge candidate to generate a new unidirectional prediction merge candidate that uses a different reference picture from the above-mentioned selected unidirectional prediction merge candidate. A method comprising:

9. In paragraph 8, The step of generating the new unidirectional prediction merge candidate is as follows: A step of deriving a relational expression representing the relationship between the position ([x], [y]) and the POC based on the position of the current block in the current picture and the position of the reference block in the reference picture indicated by at least one merge candidate included in the merge candidate list; A step of deriving the location of a reference block in a reference picture of the new unidirectional prediction merge candidate based on the above derived relational expression; and A step of determining a motion vector of the new unidirectional prediction merge candidate based on a difference between the position of a reference block in the reference picture of the new unidirectional prediction merge candidate and the position of the current block in the current picture. A method comprising:

10. In paragraph 1, The step of generating at least one artificial merge candidate comprises: A step of selecting a pair of unidirectional prediction merge candidates using the same reference picture from among the merge candidates included in the above merge candidate list; A step of deriving a position vector of a point that divides two motion vectors from the above pair of selected unidirectional prediction merge candidates inward or outward by a predefined ratio; and A step of generating a new unidirectional prediction merge candidate using the same reference picture as the above pair of unidirectional prediction merge candidates, but having the position vector as a motion vector. A method comprising:

11. In paragraph 1, The step of generating at least one artificial merge candidate comprises: A step of selecting a unidirectional prediction merge candidate from among the merge candidates included in the above merge candidate list; and A step of modifying the motion information of the above-mentioned selected unidirectional prediction merge candidate to generate a new unidirectional prediction merge candidate that uses a different motion vector from the above-mentioned selected unidirectional prediction merge candidate. A method comprising: including a motion vector of the new unidirectional prediction merge candidate; wherein the motion vector of the new unidirectional prediction merge candidate is derived by adding or subtracting a predefined offset to a horizontal component or a vertical component of the motion vector of the unidirectional prediction merge candidate.

12. In paragraph 1, A step of encoding a mark indicating a merge candidate selected from the above merge candidate list. A step of encoding the current block using the selected merge candidate. How to include more.

13. A device for decrypting video data, Memory for storing the bitstream; and comprising at least one processor, At least one processor of the above, A step of constructing a merge candidate list of the current block using at least one of a spatial candidate, a temporal candidate, or a history-based candidate; If the above merge candidate list is not full, a step of generating at least one artificial merge candidate based on the movement information of the merge candidates included in the above merge candidate list; and A step of adding at least one artificial merge candidate to the merge candidate list. A device configured to perform:

14. In paragraph 13, At least one processor of the above, A step of decoding an indication representing a merge candidate selected from the merge candidate list from the bitstream; and A step of decrypting the current block using the selected merge candidate. A device configured to perform further.

15. A method for providing video data to a video decoding device. A step of encoding the above video data into a bitstream; and A step of transmitting the above bitstream to the video decoding device. and a step of encoding the video data into a bitstream, A step of constructing a merge candidate list of the current block using at least one of a spatial candidate, a temporal candidate, or a history-based candidate; If the above merge candidate list is not full, a step of generating at least one artificial merge candidate based on the movement information of the merge candidates included in the above merge candidate list; and A step of adding at least one artificial merge candidate to the merge candidate list. A method comprising:

16. In paragraph 15, The step of encoding the above video data into a bitstream is: A step of encoding a mark indicating a merge candidate selected from the above merge candidate list. A step of encoding the current block using the selected merge candidate. How to include more.

Images