Method and apparatus for encoding and decoding an image using an inter prediction mode

Abstract

A method and apparatus for video encoding and decoding using block-based motion prediction and sub-block-based motion prediction are disclosed. According to the disclosed method, a current block is encoded / decoded using block-based motion information and sub-block-based motion information.

Description

Technical Field

[0001] This disclosure relates to a video encoding / decoding method, an apparatus, and a recording medium for storing bitstreams, and more specifically, to a matrix-based intra-frame prediction method and an apparatus using an inter-frame prediction mode. Background Technology

[0002] The statements in this section provide background information in connection with this disclosure only and do not necessarily constitute prior art.

[0003] Assuming a massive amount of data compared to audio or still image data, unprocessed or uncompressed video data requires substantial hardware resources, including memory, to be stored or transmitted in its raw form.

[0004] Therefore, when storing or transmitting video data, encoders are typically used to compress the video data for storage or transmission, while decoders receive the compressed video data and perform its decompression and reconstruction. Such video compression techniques include H.264 / AVC, High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), with VVC improving the coding efficiency of HEVC by approximately 30% or more.

[0005] However, as video size, resolution, and frame rate continue to increase, and the amount of data that needs to be encoded increases accordingly, new compression techniques are needed that are more efficient in encoding than existing compression techniques and more effective in improving video quality.

[0006] In particular, there has been a recent surge in demand for high-resolution, high-quality video (such as high-definition (HD) and ultra-high-definition (UHD) video) across various sectors. As image data becomes higher resolution and higher quality, the amount of information or bits transmitted increases compared to existing image data. This increase in the amount of information or bits transmitted leads to higher transmission and storage costs. Furthermore, there has been a recent increase in interest and demand for immersive media (such as virtual reality (VR), artificial reality (AR) content, and holograms), and the broadcasting of videos / images with characteristics different from real-world images (such as game footage) is on the rise. Therefore, a highly efficient video compression technology is needed to effectively compress, transmit, store, and play information from high-resolution, high-quality video with all of the aforementioned characteristics. Summary of the Invention

[0007] Technical issues

[0008] This disclosure provides a method and apparatus for video encoding or decoding to efficiently encode or decode video and improve the objective and subjective quality of reconstructed video, as well as a recording medium for storing the bitstream generated by the video encoding method / apparatus.

[0009] Technical solution

[0010] At least one aspect of this disclosure provides a video decoding method, including deriving block-based motion information of a current block and generating a first prediction block of the current block using the block-based motion information. The video decoding method further includes deriving sub-block-based motion information of the current block and generating a second prediction block of the current block by predicting the current block on a sub-block basis using the sub-block-based motion information. The video decoding method further includes generating a prediction block of the current block by performing a weighted summation on the first prediction block and the second prediction block.

[0011] Another aspect of this disclosure provides a video coding method, including deriving block-based motion information for a current block and generating a first prediction block for the current block using the block-based motion information. The video coding method also includes deriving sub-block-based motion information for the current block and generating a second prediction block for the current block by predicting the current block on a sub-block basis using the sub-block-based motion information. The video coding method further includes generating a prediction block for the current block by performing a weighted summation on the first and second prediction blocks.

[0012] Another aspect of this disclosure provides a non-transitory computer-readable recording medium for storing a bitstream generated by a video coding apparatus. The bitstream is generated by a video coding method. The video coding method includes deriving block-based motion information for a current block and generating a first prediction block for the current block using the block-based motion information. The video coding method also includes deriving sub-block-based motion information for the current block and generating a second prediction block for the current block by predicting the current block on a sub-block basis using the sub-block-based motion information. The video coding method further includes generating a prediction block for the current block by performing a weighted summation on the first and second prediction blocks.

[0013] Another aspect of this disclosure provides a method executed by at least one processor. The method includes generating a bitstream of video and transmitting data including the bitstream. Generating the bitstream includes: deriving block-based motion information for a current block, and generating a first prediction block for the current block using the block-based motion information. Generating the bitstream also includes deriving sub-block-based motion information for the current block, and generating a second prediction block for the current block by predicting the current block on a sub-block basis using the sub-block-based motion information. Generating the bitstream further includes generating a prediction block for the current block by performing a weighted summation on the first and second prediction blocks.

[0014] Beneficial effects

[0015] According to embodiments of this disclosure, a video encoding / decoding apparatus can encode and decode the current block using block-based inter-frame prediction modes and sub-block-based inter-frame prediction modes, thereby efficiently encoding / decoding the video and improving the objective / subjective quality of the reconstructed video. Attached Figure Description

[0016] Figure 1 This is a block diagram of a video encoding apparatus that can implement the technology disclosed herein.

[0017] Figure 2 This demonstrates a method for segmenting blocks using a quadtree plus binary tree ternary tree (QTBTTT) structure.

[0018] Figure 3a and Figure 3b Several intra-frame prediction modes, including the wide-angle intra-frame prediction mode, are shown.

[0019] Figure 4 The adjacent blocks of the current block are shown.

[0020] Figure 5 This is a block diagram of a video decoding apparatus that can implement the technology disclosed herein.

[0021] Figure 6 This is a flowchart of a method for predicting the current block according to an embodiment of the present disclosure.

[0022] Figure 7 A prediction of the SbTMVP pattern according to an embodiment of this disclosure is shown.

[0023] Figure 8a and Figure 8b An affine parameter model according to an embodiment of the present disclosure is shown.

[0024] Figure 9 This is an example view showing the method for deriving constructed affine merging candidates for affine motion prediction.

[0025] Figure 10 The prediction of the merge pattern based on AMVP-sub-blocks according to an embodiment of the present disclosure is shown.

[0026] Figure 11 This is a flowchart of a method for generating a candidate list based on sub-blocks according to an embodiment of the present disclosure.

[0027] Figure 12 The sub-block-based template matching cost according to an embodiment of the present disclosure is illustrated. Detailed Implementation

[0028] In the following description, some embodiments of the present disclosure are described in detail with reference to the accompanying drawings. In the following description, the same reference numerals denote the same elements, even if these elements are shown in different drawings. Furthermore, in the following description of some embodiments, detailed descriptions of relevant known components and functions may be omitted for clarity and brevity when it is thought that this might obscure the subject matter of the present disclosure.

[0029] Figure 1 This is a block diagram of a video encoding apparatus capable of implementing the technology disclosed herein. In the following, reference is made to... Figure 1 The diagram illustrates the video encoding device and its components.

[0030] The encoding device may include an image segmenter 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a rearrangement unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filter unit 180, and a memory 190.

[0031] Each component of the encoding device can be implemented as hardware or software, or a combination of hardware and software. Furthermore, the function of each component can be implemented as software, and a microprocessor can be implemented to execute the software function corresponding to each component.

[0032] A video consists of one or more sequences of images. Each image is segmented into multiple regions, and encoding is performed on each region. For example, an image is segmented into one or more tiles and / or stripes. Here, one or more tiles can be defined as a tile group. Each tile and / or strip is segmented into one or more coding tree units (CTUs). Furthermore, each CTU is segmented into one or more coding units (CUs) through a tree structure. Information applied to each coding unit (CU) is encoded as the syntax of the CU, and information commonly applied to the CUs included in a CTU is encoded as the syntax of the CTU. Additionally, information commonly applied to all blocks in a strip is encoded as the syntax of the strip header, and information applied to all blocks constituting one or more images is encoded as a picture parameter set (PPS) or picture header. Furthermore, information commonly referenced by multiple images is encoded as a sequence parameter set (SPS). Additionally, information commonly referenced by one or more SPSs is encoded as a video parameter set (VPS). Furthermore, information commonly applied to a tile or tile group can also be encoded as the syntax of the tile or tile group header. The syntax included in SPS, PPS, strip headers, tile or tile group headers can be referred to as advanced syntax.

[0033] Image splitter 110 determines the size of the coding tree unit (CTU). Information about the size of the CTU (CTU size) is encoded into SPS or PPS syntax and passed to the video decoding device.

[0034] Image segmenter 110 segments each image that constitutes the video into multiple coding tree units (CTUs) of predetermined size, and then recursively segments the CTUs using a tree structure. The leaf nodes in the tree structure are called coding units (CUs), which are the basic units of coding.

[0035] A tree structure can be a quadtree (QT), where a higher node (or parent node) is divided into four lower nodes (or child nodes) of the same size. A tree structure can also be a binary tree (BT), where a higher node is divided into two lower nodes. A tree structure can also be a ternary tree (TT), where a higher node is divided into three lower nodes in a 1:2:1 ratio. A tree structure can also be a mixture of two or more of the QT, BT, and TT structures. For example, a quadtree plus binary tree (QTBT) structure can be used, or a quadtree plus binary ternary tree (QTBTTT) structure can be used. Here, a binary ternary tree (BTTT) is added to the tree structure to form a multi-type tree (MTT).

[0036] Figure 2 This is a diagram used to describe the method of segmenting blocks using the QTBTTT structure.

[0037] like Figure 2 As shown, the CTU can first be segmented into a QT structure. Quadtree segmentation can be recursive until the size of the segmented block reaches the minimum block size (MinQTSize) allowed for a leaf node in the QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is segmented into four nodes of the next layer is encoded by the entropy encoder 155 and signaled to the video decoding device. When a leaf node of the QT is not larger than the maximum block size (MaxBTSize) allowed for a root node in the BT, the leaf node can be further segmented into at least one of a BT structure and a TT structure. Multiple segmentation directions can exist in the BT structure and / or the TT structure. For example, there can be two directions: the direction in which the block of the corresponding node is horizontally segmented and the direction in which the block of the corresponding node is vertically segmented. Figure 2 As shown, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether these nodes are split, a flag additionally indicating the splitting direction (vertical or horizontal), and / or (if these nodes are split) a flag indicating the splitting type (binary tree or ternary tree) are encoded by the entropy encoder 155 and signaled to the video decoding device.

[0038] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into the next layer of four nodes, the CU splitting flag (split_cu_flag) indicating whether the node is split can also be encoded. When the value of the CU splitting flag (split_cu_flag) indicates that each node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU as the basic unit of encoding. When the value of the CU splitting flag (split_cu_flag) indicates that each node is split, the video encoding device first begins encoding the first flag according to the above scheme.

[0039] When QTBT is used as another example of a tree structure, two types can exist: the type where the corresponding node's block is horizontally split into two blocks of the same size (i.e., symmetrical horizontal split), and the type where the corresponding node's block is vertically split into two blocks of the same size (i.e., symmetrical vertical split). A split flag (split_flag) indicating whether each node in the BT structure is split into blocks of the next level, and split type information indicating the split type, are encoded by the entropy encoder 155 and transmitted to the video decoding device. Additionally, there can be a type where the corresponding node's block is split into two blocks that are asymmetrically divided. Asymmetrical forms can include the form where the corresponding node's block is split into two rectangular blocks with a size ratio of 1:3, or it can also include the form where the corresponding node's block is split along a diagonal direction.

[0040] The CU can have various sizes depending on the QTBT or QTBTTT segmentation performed from the CTU. In the following text, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block". Due to the QTBTTT segmentation, the current block can be rectangular in addition to square.

[0041] Predictor 120 predicts the current block to generate a prediction block. Predictor 120 includes an intra-frame predictor 122 and an inter-frame predictor 124.

[0042] Typically, each current block in an image can be predictively encoded. This prediction can usually be performed using either intra-frame prediction (using data from the image containing the current block) or inter-frame prediction (using data from an image encoded before the image containing the current block). Inter-frame prediction includes both one-way and two-way prediction.

[0043] Intra-predictor 122 predicts pixels in the current block using pixels that are neighbors of the current block in the current image, including the current block (reference pixels). Multiple intra-prediction modes exist depending on the prediction direction. For example, such as... Figure 3aAs shown, multiple intra-frame prediction modes can include two non-directional modes (including a planar mode and a DC mode) and 65 directional modes. The neighboring pixels and arithmetic equations to be used are defined differently for each prediction mode.

[0044] To perform efficient directional prediction for the current block with a rectangular shape, it is possible to add the following: Figure 3b The directional modes (intra-prediction modes #67 to #80, #-1 to #-14) are shown as dashed arrows. These directional modes can be referred to as "wide-angle intra-prediction modes." Figure 3b In the diagram, the arrow indicates the corresponding reference sample used for prediction and does not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrow. When the current block has a rectangular shape, the wide-angle intra-frame prediction mode is a mode that performs prediction in the opposite direction to the specific directional mode without using additional bit transmission. In this case, in the wide-angle intra-frame prediction mode, some wide-angle intra-frame prediction modes that can be used for the current block can be determined by the ratio of the width to the height of the current block with a rectangular shape. For example, when the current block has a rectangular shape with a height less than its width, wide-angle intra-frame prediction modes with an angle less than 45 degrees can be used (intra-frame prediction modes #67 to #80). When the current block has a rectangular shape with a width greater than its height, wide-angle intra-frame prediction modes with an angle greater than -135 degrees can be used.

[0045] Intra-predictor 122 can determine the intra-prediction to be used for encoding the current block. In some examples, intra-predictor 122 can encode the current block using multiple intra-prediction modes, and can also select the appropriate intra-prediction mode to use from the tested modes. For example, intra-predictor 122 can calculate rate-distortion values ​​using rate-distortion analysis for multiple tested intra-prediction modes, and can also select the intra-prediction mode with the best rate-distortion characteristics from the tested modes.

[0046] Intra-predictor 122 selects one of multiple intra-prediction modes and predicts the current block using neighboring pixels (reference pixels) determined according to the selected intra-prediction mode and an arithmetic equation. Information about the selected intra-prediction mode is encoded by entropy encoder 155 and transmitted to the video decoding device.

[0047] Inter-frame predictor 124 generates a predicted block for the current block using a motion compensation process. Inter-frame predictor 124 searches for the most similar block to the current block in a reference image that was encoded and decoded earlier than the current image, and generates the predicted block for the current block using the searched block. Furthermore, a motion vector (MV) is generated, corresponding to the displacement between the current block in the current image and the predicted block in the reference image. Typically, motion estimation is performed for the luma component, and the motion vector calculated based on the luma component is used for both the luma and chroma components. Motion information, including information about the reference image and information for predicting the motion vector for the current block, is encoded by entropy encoder 155 and transmitted to the video decoding device.

[0048] The inter-frame predictor 124 can also perform interpolation on a reference image or reference block to improve prediction accuracy. In other words, subsamples are interpolated between two adjacent integer samples by applying filter coefficients to multiple consecutive integer samples, including two integer samples. When performing a search for the most similar block to the current block against the interpolated reference image, the motion vector can be represented with fractional-unit precision rather than integer-sample-unit precision. The precision or resolution of the motion vector can be set differently for each target region to be encoded (e.g., slice, tile, CTU, CU, etc.). When applying this adaptive motion vector resolution (AMVR), information on the motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, information on the applied motion vector resolution is signaled for each CU. The motion vector resolution information can be information representing the precision of the motion vector difference, which will be described below.

[0049] Simultaneously, the inter-frame predictor 124 can perform inter-frame prediction using bidirectional prediction. In the case of bidirectional prediction, two reference images and two motion vectors representing the location of the most similar block to the current block in each reference image are used. The inter-frame predictor 124 selects a first reference image and a second reference image from reference image list 0 (RefPicList0) and reference image list 1 (RefPicList1), respectively. The inter-frame predictor 124 also searches for the most similar block to the current block in the respective reference images to generate a first reference block and a second reference block. Furthermore, the predicted block for the current block is generated by averaging or weighted averaging the first reference block and the second reference block. In addition, motion information containing information about the two reference images used to predict the current block and including information about the two motion vectors is passed to the entropy encoder 155. Here, reference image list 0 may consist of images from the pre-reconstructed images that are displayed before the current image, and reference image list 1 may consist of images from the pre-reconstructed images that are displayed after the current image. However, while not specifically limited to this, pre-reconstructed images that are displayed after the current image may be additionally included in reference image list 0. Conversely, pre-reconstructed images that are displayed before the current image may also be additionally included in reference image list 1.

[0050] Various methods can be used to minimize the number of bits consumed in encoding motion information.

[0051] For example, when the reference image and motion vector of the current block are the same as those of the neighboring blocks, the information that can identify the neighboring blocks is encoded to transmit the motion information of the current block to the video decoding device. This method is called merging mode.

[0052] In merge mode, the inter-frame predictor 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as "merge candidate") from the neighboring blocks of the current block.

[0053] like Figure 4 As shown, some or all of the following adjacent blocks in the current image—left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2—can be used as neighboring blocks for deriving merge candidates. Additionally, blocks located in a reference image (which may be the same as or different from the reference image used to predict the current block) outside the current image can also be used as merge candidates. For example, blocks in the reference image that are co-located with the current block or adjacent to that co-located block can be additionally used as merge candidates. If the number of merge candidates selected by the above method is less than a preset number, a zero vector is added to the merge candidates.

[0054] The inter-frame predictor 124 configures a merge list including a predetermined number of merge candidates by using neighboring blocks. It selects merge candidates from the merge list to be used as motion information for the current block and generates merge index information to identify the selected candidates. The generated merge index information is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0055] Merge skip mode is a special case of merge mode. After quantization, when all transform coefficients used for entropy coding are close to zero, only adjacent block selection information is transmitted, and the residual signal is not transmitted. By using merge skip mode, relatively high coding efficiency can be achieved for images with slight motion, still images, and screen content images.

[0056] In the following text, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0057] Another method for encoding motion information is the Advanced Motion Vector Prediction (AMVP) pattern.

[0058] In AMVP mode, the inter-frame predictor 124 derives motion vector predictor candidates for the current block by using neighboring blocks of the current block. As neighboring blocks used to derive the motion vector predictor candidates, [the following can be used]. Figure 4 The diagram shows all or some of the left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2 adjacent to the current block in the current image. Additionally, blocks located in a reference image (which may be the same as or different from the reference image used to predict the current block) outside the current image can also be used as neighboring blocks for deriving motion vector predictor candidates. For example, blocks in the reference image that are in the same position as the current block or adjacent to that in-position block can be used. If the number of motion vector candidates selected by the above method is less than a preset number, a zero vector is added to the motion vector candidates.

[0059] Inter-frame predictor 124 derives motion vector predictor candidates using motion vectors from adjacent blocks, and determines a motion vector predictor for the current block's motion vector using these candidates. Furthermore, the motion vector difference is calculated by subtracting the motion vector predictor from the motion vector of the current block.

[0060] Motion vector predictors can be obtained by applying predefined functions (e.g., center value and average value calculations) to motion vector predictor candidates. In this case, the video decoding device is also aware of the predefined functions. Furthermore, since the adjacent blocks used to derive the motion vector predictor candidates are blocks that have already been encoded and decoded, the video decoding device may already know the motion vectors of the adjacent blocks. Therefore, the video encoding device does not need to encode the information used to identify the motion vector predictor candidates. Therefore, in this case, the information of the motion vector difference and the information of the reference image used to predict the current block are encoded.

[0061] Alternatively, a motion vector predictor can be determined by selecting any one of the motion vector predictor candidates. In this case, the information used to identify the selected motion vector predictor candidate is additionally encoded along with the information on the motion vector difference and the information on the reference image used to predict the current block.

[0062] Subtractor 130 generates a residual block by subtracting the prediction block generated by intra-predictor 122 or inter-predictor 124 from the current block.

[0063] Transformer 140 transforms the residual signal in the residual block, which has pixel values ​​in the spatial domain, into transform coefficients in the frequency domain. Transformer 140 can use the total size of the residual block as a transform unit to transform the residual signal in the residual block, or it can divide the residual block into multiple sub-blocks and use the sub-blocks as transform units to perform the transformation. Optionally, the residual block can be divided into two sub-blocks (i.e., a transformed region and a non-transformed region) so that only the transformed region sub-block is used as a transform unit to transform the residual signal. Here, the transformed region sub-block can be one of two rectangular blocks with a 1:1 size ratio based on the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicates that only the sub-block is transformed, and the orientation (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag) are encoded by entropy encoder 155 and signaled to the video decoding device. Furthermore, the size of the transformed region sub-block can have a 1:3 size ratio based on the horizontal axis (or vertical axis). In this case, the flag (cu_sbt_quad_flag) corresponding to the segment is additionally encoded by the entropy encoder 155 and signaled to the video decoding device.

[0064] Simultaneously, the transformer 140 can perform transformations on the residual block independently in the horizontal and vertical directions. Various types of transformation functions or transformation matrices can be used for the transformation. For example, a pair of transformation functions for horizontal and vertical transformations can be defined as a multiple transform set (MTS). The transformer 140 can select the transformation function pair with the highest transformation efficiency from the MTS and can transform the residual block in each of the horizontal and vertical directions. Information about the transformation function pair in the MTS (mts_idx) is encoded by the entropy encoder 155 and signaled to the video decoding device.

[0065] Quantizer 145 uses quantization parameters to quantize the transform coefficients output from transformer 140 and outputs the quantized transform coefficients to entropy encoder 155. Quantizer 145 can also directly quantize the associated residual blocks of any block or frame without performing a transform. Quantizer 145 can also apply different quantization coefficients (scaling values) based on the position of the transform coefficients in the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions can be encoded and passed to the video decoding device.

[0066] The rearrangement unit 150 can perform coefficient realignment on the quantized residual values.

[0067] Rearrangement unit 150 can transform a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, rearrangement unit 150 can output a one-dimensional coefficient sequence by scanning the DC coefficients to high-frequency domain coefficients using zigzag scanning or diagonal scanning. Depending on the size of the transform unit and the intra-frame prediction mode, vertical scanning along the column direction and horizontal scanning along the row direction of the two-dimensional block-type coefficient array can also be used instead of zigzag scanning. In other words, the scanning method to be used can be determined from zigzag scanning, diagonal scanning, vertical scanning, and horizontal scanning, depending on the size of the transform unit and the intra-frame prediction mode.

[0068] The entropy encoder 155 generates a bitstream by encoding the one-dimensional quantized transform coefficient sequence output from the rearrangement unit 150 using various encoding schemes, including context-based adaptive binary arithmetic coding (CABAC), exponential Golomb coding, or similar schemes.

[0069] Furthermore, the entropy encoder 155 encodes block-segmentation related information (such as CTU size, CTU segmentation flag, QT segmentation flag, MTT segmentation type, MTT segmentation direction, etc.) to enable the video decoding device to segment blocks in the same way as the video encoding device. Additionally, the entropy encoder 155 encodes prediction type information indicating whether the current block is encoded using intra-frame prediction or inter-frame prediction. The entropy encoder 155 encodes intra-frame prediction information (i.e., information about the intra-frame prediction mode) or inter-frame prediction information (the merge index in merge mode, and information about the reference image index and motion vector difference in AMVP mode) based on the prediction type. Furthermore, the entropy encoder 155 encodes quantization-related information (i.e., information about quantization parameters and information about the quantization matrix).

[0070] Inverse quantizer 160 inverse quantizes the quantized transform coefficients output from quantizer 145 to generate transform coefficients. Inverse transformer 165 transforms the transform coefficients output from inverse quantizer 160 from the frequency domain to the spatial domain to reconstruct the residual block.

[0071] Adder 170 adds the reconstructed residual block to the predicted block generated by predictor 120 to reconstruct the current block. Pixels in the reconstructed current block can be used as reference pixels when performing intra-frame prediction for the next sequential block.

[0072] The loop filtering unit 180 performs filtering on the reconstructed pixels to reduce block artifacts, ringing artifacts, blurring artifacts, etc., caused by block-based prediction and transform / quantization. The loop filtering unit 180, as an in-loop filter, may include all or some of the following: a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.

[0073] Deblocking filter 182 filters the boundaries between reconstructed blocks to remove block artifacts caused by block unit encoding / decoding. SAO filter 184 and ALF 186 perform additional filtering on the deblocked video. SAO filter 184 and ALF 186 are filters used to compensate for differences between reconstructed pixels and original pixels caused by lossy coding. SAO filter 184 applies an offset in units of CTU to enhance subjective image quality and coding efficiency. On the other hand, ALF 186 performs block unit filtering and applies different filters to compensate for distortion by dividing the boundaries of the corresponding blocks and the degree of variation. The filter coefficient information for ALF can be encoded and passed to the video decoding device.

[0074] The reconstructed blocks, filtered by deblocking filter 182, SAO filter 184, and ALF 186, are stored in memory 190. When all blocks in an image have been reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.

[0075] Video encoding devices can store the bit stream of encoded video data in a non-transitory storage medium, or transmit the bit stream to a video decoding device via a communication network.

[0076] Figure 5 This is a functional block diagram of a video decoding apparatus capable of implementing the technology of this disclosure. Hereinafter, reference will be made to... Figure 5 Describe the video decoding device and its components.

[0077] The video decoding device may include an entropy decoder 510, a rearrangement unit 515, an inverse quantizer 520, an inverse transformer 530, a predictor 540, an adder 550, a loop filter unit 560, and a memory 570.

[0078] and Figure 1 Similar to video encoding devices, each component of a video decoding device can be implemented as hardware or software, or a combination of both. Furthermore, the function of each component can be implemented as software, and a microprocessor can be implemented to execute the software function corresponding to each component.

[0079] The entropy decoder 510 extracts information related to block segmentation by decoding the bitstream generated by the video encoding device to determine the current block to be decoded, and extracts the prediction information and information about the residual signal needed to reconstruct the current block.

[0080] The entropy decoder 510 determines the size of the CTU by extracting information about the CTU size from the sequence parameter set (SPS) or picture parameter set (PPS) and segments the image into CTUs with the determined size. Furthermore, the CTU is identified as the highest level (i.e., the root node) of the tree structure, and segmentation information for that CTU can be extracted to segment the CTU using the tree structure.

[0081] For example, when using the QTBTTT structure to segment the CTU, the first flag (QT_split_flag) related to the QT split is first extracted to divide each node into four lower-level nodes. Furthermore, for nodes corresponding to leaf nodes of the QT, the second flag (mtt_split_flag), split direction (vertical / horizontal), and / or split type (binary / ternary) related to the MTT split are extracted to segment the corresponding leaf nodes into MTT structures. As a result, each node below a leaf node of the QT is recursively segmented into either a BT or TT structure.

[0082] As another example, when splitting a CTU using a QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU has been split is extracted. A first flag (QT_split_flag) can also be extracted when the corresponding block is split. During the splitting process, for each node, zero or more recursive MTT splits may occur after zero or more recursive QT splits. For example, for a CTU, an MTT split may occur immediately, or conversely, it may only occur after multiple QT splits.

[0083] As another example, when using a QTBT structure to segment a CTU, the first flag (QT_split_flag) associated with the segmentation of the QT is extracted to split each node into the four nodes of the lower layer. Additionally, a split flag (split_flag) indicating whether the node corresponding to the leaf node of the QT is further segmented into a BT, as well as the segmentation direction information, is extracted.

[0084] Simultaneously, when the entropy decoder 510 determines the current block to be decoded using tree-structured segmentation, it extracts prediction type information indicating whether the current block is intra-prediction or inter-prediction. When the prediction type information indicates intra-prediction, the entropy decoder 510 extracts syntax elements for the intra-prediction information (intra-prediction mode) used for the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 extracts information representing syntax elements used for inter-prediction information (i.e., motion vectors and the reference image to which these motion vectors are referenced).

[0085] In addition, the entropy decoder 510 extracts quantization-related information and extracts information about the quantized transform coefficients of the current block as information about the residual signal.

[0086] The rearrangement unit 515 can transform the one-dimensional quantized transform coefficient sequence entropy decoded by the entropy decoder 510 back into a two-dimensional coefficient array (i.e., a block) in the reverse order of the coefficient scan performed by the video encoding device.

[0087] The inverse quantizer 520 performs inverse quantization on the quantized transform coefficients and uses quantization parameters to perform inverse quantization on the quantized transform coefficients. The inverse quantizer 520 can also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantizer 520 can perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from the video encoding device to a 2D array of quantized transform coefficients.

[0088] The inverse transformer 530 reconstructs the residual signal by inversely transforming the inversely quantized transform coefficients from the frequency domain to the spatial domain, thereby generating the residual block of the current block.

[0089] Furthermore, when the inverse transformer 530 performs an inverse transform on a portion (sub-block) of the transform block, it extracts a flag (cu_sbt_flag) indicating that only the sub-blocks of the transform block are transformed, the orientation (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-blocks, and / or the position information (cu_sbt_pos_flag) of the sub-blocks. The inverse transformer 530 also inversely transforms the transform coefficients of the corresponding sub-blocks from the frequency domain to the spatial domain to reconstruct the residual signal, and fills the regions not inversely transformed with "0" values ​​as the residual signal to generate the final residual block of the current block.

[0090] Furthermore, when applying MTS, the inverse transformer 530 uses the MTS information (mts_idx) transmitted from the video encoding device to determine the transform function or transform matrix to be applied in each of the horizontal and vertical directions. The inverse transformer 530 also performs inverse transform on the transform coefficients in the transform block in the horizontal and vertical directions using the determined transform function.

[0091] Predictor 540 may include intra-predictor 542 and inter-predictor 544. Intra-predictor 542 is activated when the prediction type of the current block is intra-prediction, and inter-predictor 544 is activated when the prediction type of the current block is inter-prediction.

[0092] Based on the syntax elements for the intra prediction mode extracted from the entropy decoder 510, the intra predictor 542 determines the intra prediction mode for the current block among multiple intra prediction modes. The intra predictor 542 also predicts the current block by using neighboring reference pixels of the current block, based on the intra prediction mode.

[0093] The inter-frame predictor 544 determines the motion vector of the current block and the reference picture to which the motion vector is referenced by using the syntax elements for the inter-frame prediction mode extracted from the entropy decoder 510, and uses the motion vector and the reference picture to predict the current block.

[0094] Adder 550 reconstructs the current block by adding the residual block output from inverse transform 530 to the prediction block output from inter-frame predictor 544 or intra-frame predictor 542. The pixels in the reconstructed current block are used as reference pixels when performing intra-frame prediction on the block to be decoded later.

[0095] The loop filtering unit 560, serving as an in-loop filter, may include a deblocking filter 562, a SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundaries between reconstructed blocks to remove blocking artifacts caused by block unit decoding. The SAO filter 564 and ALF 566 perform additional filtering on the reconstructed blocks after deblocking to compensate for differences between reconstructed pixels and original pixels caused by lossy encoding. The filter coefficients of the ALF are determined using filter coefficient information decoded from the bitstream.

[0096] The reconstructed blocks, filtered by deblocking filter 562, SAO filter 564, and ALF 566, are stored in memory 570. When all blocks in an image have been reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.

[0097] Hereinafter, an improved encoding tool executed by the aforementioned video encoding apparatus or video decoding apparatus is disclosed. The following embodiments can be executed by the intra-frame predictor 122 in the video encoding apparatus. Furthermore, the following embodiments can be executed by the intra-frame predictor 542 in the video decoding apparatus. In this disclosure, the image processing apparatus refers to at least one of the video encoding apparatus and the video decoding apparatus.

[0098] This disclosure relates to a combination of block-based inter-frame prediction modes and sub-block-based inter-frame prediction modes. The current block is predicted using a weighted sum of predicted blocks based on the block-based inter-frame prediction mode and predicted blocks based on the sub-block-based inter-frame prediction mode.

[0099] In this disclosure, the block-based inter-frame prediction mode can be a prediction mode that references a block with the same size as the current block, such as an Advanced Motion Vector Prediction (AMVP) mode, a template-matching-based inter-frame prediction mode, or a merging mode. The sub-block-based inter-frame prediction mode can be a prediction mode that references a sub-block of a reference block, such as a sub-block-based merging mode or a merging mode.

[0100] Figure 6 This is a flowchart of a method for predicting the current block according to an embodiment of the present disclosure.

[0101] Reference Figure 6 In step S610, the image processing device exports block-based motion information of the current block.

[0102] The image processing device derives block-based motion information of the current block based on pre-encoded blocks and pre-decoded blocks (i.e., pre-reconstructed blocks).

[0103] In one implementation, the video encoding apparatus can determine the block-based motion information of the current block and can signal the block-based motion information of the current block. The video decoding apparatus can derive the block-based motion information of the current block based on the signaling information.

[0104] First, the image processing device generates a block-based motion candidate list for the current block based on the pre-reconstructed blocks.

[0105] The block-based motion candidate list can be an AMVP candidate list. Each block-based motion candidate can include motion information or zero motion vectors of the spatial and temporal neighboring blocks of the current block, and can include motion vector difference (MVD) and reference image index.

[0106] Alternatively, the block-based motion candidate list can be a merged candidate list. The block-based motion candidate list can include motion information of spatially adjacent blocks, motion information of temporally adjacent blocks, motion information of historically adjacent blocks, zero motion vectors, etc., as block-based motion candidates.

[0107] The image processing apparatus can signal the index of a block-based motion candidate within a block-based motion candidate list. The video encoding apparatus can determine block-based motion candidates for motion prediction of the current block and can signal the determined index of the block-based motion candidate to the video decoding apparatus. The video decoding apparatus can determine the block-based motion candidate from the block-based motion candidate list based on the signaled block-based motion candidate index and can use the motion information of the block-based motion candidate to determine the block-based motion information of the current block. The motion information of the block-based motion candidate can be used as the block-based motion information of the current block.

[0108] In another embodiment, the image processing apparatus can search for reference blocks based on template matching costs within a pre-reconstructed region, and can use the motion information of the reference blocks to determine the block-based motion information of the current block.

[0109] Here, template matching involves searching for template regions similar to those of neighboring samples that include the current block within the search region of the pre-reconstructed image in the current or reference image. The block with the searched template region as its template is designated as the reference block for the current block. The template matching cost refers to the difference between the two templates.

[0110] Template matching cost is based on the difference between two templates and may include the sum of absolute differences (SAD), the sum of absolute transformation differences (SATD), the sum of absolute differences with mean removed (MR-SAD), the mean squared error (MSE), or the sum of squared errors (SSE).

[0111] Motion information of a reference block found based on template matching cost can be used as block-based motion information for the current block. As an example, the block-based motion vector and reference image index of the current block can be equivalent to the motion vector and reference image index of the reference block. As another example, the block-based motion vector of the current block can be equivalent to the sum of the MVD and the motion vector of the reference block. In yet another example, the motion information of the candidate with the lowest template matching cost among the AMVP candidates in the AMVP candidate list can be used as the block-based motion information for the current block, and signaling notification for the AMVP candidate index can be skipped.

[0112] In step S620, the image processing device generates a first predicted block for the current block using block-based motion information. The block indicated by the block-based motion information of the current block can be used as the first predicted block for the current block.

[0113] In step S630, the image processing device derives motion information of the current block based on its sub-blocks.

[0114] First, the image processing device generates a sub-block-based motion candidate list for the current block based on pre-reconstructed blocks, including sub-blocks. Furthermore, the image processing device can reorder the candidates within the sub-block-based motion candidate list. The generation of the sub-block-based motion candidate list... Figure 11 As described in the text.

[0115] The sub-block-based motion candidate list can be a sub-block-based merged candidate list. The sub-block-based merged candidate list includes at least one of sub-block-based temporal motion vector prediction (SbTMVP) candidates and affine candidates. One of the pre-reconstructed blocks is a neighboring block referencing a co-image of the current image including the current block, and can be used as an SbTMVP candidate. SbTMVP candidates and affine candidates in... Figure 7 , Figure 8a , Figure 8b and Figure 9 As described in the text.

[0116] The sub-block-based motion candidate list can be a merged candidate list. The pre-reconstructed block can refer to at least one of the current block's spatial neighboring blocks, temporal neighboring blocks, and history-based neighboring blocks. The sub-block-based motion candidate list can include motion information of spatial neighboring blocks, motion information of temporal neighboring blocks, motion information of history-based neighboring blocks, zero motion vectors, etc., as sub-block-based motion candidates. Here, motion information refers to motion information at the sub-block level.

[0117] Subsequently, the image processing device uses a sub-block-based motion candidate list to derive sub-block-based motion information for the current block.

[0118] In one implementation, when the block-based motion information of the current block is first exported, the image processing device can export the sub-block-based motion information of the current block based on the bidirectional matching cost.

[0119] The sub-block motion information of the current block can be selected from the sub-block motion candidates based on the bidirectional matching cost between the block predicted by the sub-block motion candidate in the sub-block motion candidate list and the first predicted block. The sub-block motion candidate with the minimum bidirectional matching cost can be determined as the sub-block motion information of the current block.

[0120] If the motion information of a sub-block-based motion candidate has an L0 direction, then a candidate with L1 direction motion information and lower bidirectional matching cost can be selected from the sub-block-based motion candidate pool. Alternatively, if the motion information of a sub-block-based motion candidate has an L1 direction, then a candidate with L0 direction motion information and lower bidirectional matching cost can be selected from the sub-block-based motion candidate pool.

[0121] The cost of bidirectional matching is based on the difference between the two blocks and may include the sum of absolute differences (SAD), the sum of absolute transformation differences (SATD), the sum of mean-removed absolute differences (MR-SAD), the mean squared error (MSE), or the sum of squared errors (SSE).

[0122] In another embodiment, the image processing device can derive sub-block-based motion information of the current block based on signaling information.

[0123] The image processing apparatus can signal the index of a sub-block-based motion candidate within a sub-block-based motion candidate list. The video encoding apparatus can determine sub-block-based motion candidates for motion prediction of the current block and can signal the determined sub-block-based motion candidate indexes to the video decoding apparatus. The video decoding apparatus can determine sub-block-based motion candidates from the sub-block-based motion candidate list based on the signaled sub-block motion candidate indexes and can use the motion information of the sub-block-based motion candidates to determine the sub-block-based motion information for the current block. The motion information of the sub-block-based motion candidates can be used as the sub-block-based motion information for the current block.

[0124] Sub-block-based motion candidates include motion information at the sub-block level. Sub-blocks of the current block are predicted using this sub-block-level motion information.

[0125] In step S640, the image processing device predicts the current block on a sub-block basis using motion information based on sub-blocks, thereby generating a second predicted block for the current block. Specifically, the image processing device predicts each sub-block using motion information of each sub-block of the current block.

[0126] In step S650, the image processing device generates a prediction block for the current block by performing a weighted summation on the first and second prediction blocks of the current block.

[0127] The image processing device can generate a residual block based on the difference between the predicted block of the current block and the original block, and perform a transform / inverse transform and a quantization / inverse quantization on the residual block.

[0128] Simultaneously, the first reference image used to generate the first prediction block and the second reference image used to generate the second prediction block can have different prediction directions. In other words, the first prediction block can be generated by referencing a block within a reference image located in either the L0 or L1 direction relative to the current image, and the second prediction block can be generated by referencing a sub-block within a reference image located in the other direction. As prediction directions, the L0 direction refers to a reference image that is earlier in the playback order than the current image, and the L1 direction refers to a reference image that is later in the playback order than the current image. If the reference image based on the block's motion information for the current block is included in the LX (where X is 0 or 1) reference image list, then the reference image based on the sub-block's motion information is included in the L(1-X) reference image list.

[0129] In another embodiment, the first reference image used to generate the first prediction block and the second reference image used to generate the second prediction block may have the same prediction direction.

[0130] Meanwhile, the weights of the weighted summation of the first and second prediction blocks are in Figure 10 As described in the text.

[0131] Furthermore, in both cases where the distances from the current image to the two reference images are the same and different, block-based motion information and sub-block-based motion information can be corrected based on bidirectional optical flow (BDOF). Alternatively, BDOF-based motion information correction can be performed only when the distances from the current image to the two reference images are the same.

[0132] In this disclosure, the block-based inter-frame prediction mode can be the AMVP mode, and the sub-block-based inter-frame prediction mode can be the sub-block-based merging mode. That is, the block-based inter-frame prediction mode can be a combination of the Advanced Motion Vector Prediction (AMVP) mode and the sub-block-based merging mode. The combined mode can be referred to as the sub-block-based AMVP merging mode or the AMVP-sub-block-based merging mode.

[0133] AMVP's sub-block-based merging pattern can be applied regardless of the distance between the current image and reference images in both directions. AMVP's sub-block-based merging pattern is allowed for the prediction of the current block if the distance from the current image to the L0 reference image is equal to the distance from the current image to the L1 reference image. AMVP's sub-block-based merging pattern is also allowed for the prediction of the current block even if the distance from the current image to the L0 reference image is not equal to the distance from the current image to the L1 reference image.

[0134] AMVP's sub-block-based merge patterns can be restricted based on the size of the current block. For example, sub-block-based merge patterns, including affine or SbTMVP patterns, can only be applied to the current block if the current block size is 8×8 or larger.

[0135] The AMVP pattern is described below.

[0136] The AMVP model derives the motion vector of the current block using the motion vectors of its spatial and temporal neighbors, the zero motion vector, and the motion vector difference (MVD). It then uses the blocks within the reference image indicated by these motion vectors to predict the pattern of the current block. Here, MVD represents the difference between the motion vector indicating the optimal reference block position on the reference image and the motion vectors of its neighbors.

[0137] In AMVP mode, the motion information of adjacent blocks can include motion vector candidate information and motion vector difference information.

[0138] In AMVP mode, motion information of the current block is encoded / decoded based on the AMVP candidate list. The image processing device can construct the AMVP candidate list using motion vectors and zero motion vectors of spatially and temporally adjacent blocks, and encode / decode the motion vector of the current block based on the AMVP candidate list. The image processing device can construct an AMVP candidate list including a preset number of candidates (e.g., 2).

[0139] Specifically, the image processing device searches for candidates in the space. The image processing device selects candidates from, for example... Figure 4 The example shows the neighboring block search space candidates. A maximum of two space candidates can be selected.

[0140] The image processing device searches for time candidates. The image processing device may determine a corresponding block within a reference image other than the current image, including the current block, as a time candidate. Here, the reference image may be a co-located image, and within the co-located image, the corresponding block may be a block that is in the same relative position as the current block in the current image. Alternatively, the corresponding block may be a block located at the center or lower right corner of a block that is in the same relative position as the current block. One time candidate may be selected.

[0141] Even though spatial and temporal candidates have been searched, if the number of motion vector prediction (MVP) candidates is less than a preset number, the image processing device will add a zero motion vector as a candidate.

[0142] Once the AMVP candidate list is constructed, the video coding unit determines the optimal motion vector for the current block based on coding efficiency optimization, and derives the MVD by subtracting the motion vectors of the AMVP candidates from the optimal motion vector. The motion vectors of the AMVP candidates used to derive the MVD are called MVPs.

[0143] In AMVP mode, the video encoding device signals the MVD (Multi-Image Decoding) and the reference image index of the reference image in the reference image list to the video decoding device. Additionally, the video encoding device can signal the MVP index indicating the AMVP candidate.

[0144] The video decoding device can construct the AMVP candidate list in the same way as the video encoding device, derive the MVP from the AMVP candidate list using the MVP index, and decode the motion vector of the current block by adding the MVP to the MVD. The video decoding device uses a reference image index to determine reference images from a list of reference images, and uses reference blocks within those reference images, indicated by the motion vector of the current block, to predict the current block.

[0145] In another implementation, the MVP index can be derived in the video decoding device via template matching without signaling notification. The video decoding device can then select the candidate whose template differs least from the template of the current block from the AMVP candidates.

[0146] Simultaneously, the video encoding device can send information determining the spatial resolution of the MVD along with the MVD. When using AMVR technology, the video encoding device can determine the adaptive spatial resolution of the MVD based on bitrate distortion optimization. In this case, the spatial resolution of the MVD and the spatial resolution of the motion vectors can be the same. The video encoding device informs the video decoding device of the MVD's spatial resolution by signaling `amvr_flag` and `amvr_precision_idx`. If the signaled `amvr_flag` is 0, the video decoding device sets the MVD to a 1 / 4-pixel spatial resolution. On the other hand, if `amvr_flag` is not 0, the video decoding device can determine the MVD's spatial resolution based on `amvr_precision_idx`. In this case, the optional spatial resolution of the MVD can vary depending on the prediction method applied to AMVR. Prediction methods applicable to AMVR include general AMVP mode, affine AMVP mode, intra-block copy (IBC) AMVP mode, etc.

[0147] The following describes a sub-block-based merging pattern.

[0148] The sub-block-based merging pattern divides the current block into sub-blocks, derives motion information for each sub-block, and uses the motion information of the sub-blocks to predict the pattern of the current block. The motion information of the sub-blocks is derived from the motion information of the SbTMVP pattern or from the motion information of the affine pattern.

[0149] The SbTMVP pattern is described below.

[0150] SbTMVP mode is a mode that derives a motion vector field (MVF) representing motion vectors in units of sub-blocks from the co-image of the current block.

[0151] In SbTMVP mode, the motion vector field is derived from a reference block, which is located at the position of the motion vector of the adjacent block relative to the current block's co-position block, offset from the current block's motion vector. Here, the motion vector of the adjacent block can be referred to as motion shift.

[0152] Reference Figure 4 This shows the adjacent blocks used to derive motion offsets in the SbTMVP mode.

[0153] Motion offset can be induced by at least one of scan blocks A0, A1, B0, B1, and B2. The scanning order of adjacent blocks can be arbitrarily determined. Alternatively, the scanning order of adjacent blocks can be predefined between the video decoding device and the video encoding device. For example, the scanning order of adjacent blocks can be scanned in the order of A0, B0, B1, A1, and B2.

[0154] The motion offset of the current block is derived from the motion vectors of the adjacent blocks.

[0155] As an example, the motion vector of a neighboring block at a preset position can be determined as the motion offset of the current block.

[0156] As another example, if a neighboring block has a motion vector that references a co-image of the current image, then the motion vector of that neighboring block can be determined as a motion offset. If no motion offset is derived from a neighboring block, that is, if neither neighboring block references a co-image of the current image, then the motion offset can be set to 0.

[0157] In the sub-block-based merging mode, blocks A0, A1, B0, B1, and B2 that can be used as motion offsets can be SbTMVP candidates. Blocks that can be used as motion offsets can be listed as SbTMVP candidates based on the scan order.

[0158] When adding an SbTMVP candidate to the sub-block-based merge candidate list, the image processing device may perform a duplicate check between the candidates in the sub-block-based merge candidate list and the candidate to be added. If the candidate to be added is not a duplicate, the image processing device adds that candidate to the sub-block-based merge candidate list. If the candidate to be added is a duplicate, the image processing device does not add that candidate to the sub-block-based merge candidate list.

[0159] As an example, if the block indicated by the motion offset of the lower left block of the current block is the same as the block indicated by the motion offset of the upper left block of the current block, then only one of the two blocks can be added as a candidate for SbTMVP.

[0160] Figure 7 A prediction of the SbTMVP pattern according to an embodiment of this disclosure is shown.

[0161] Reference Figure 7 Block A1, located to the lower left of the current block, can be used to derive the motion vector field of the current block.

[0162] Block A1 is predicted by referencing block A1′ within the same image. Since block A1 uses the same image as a reference, the motion vector of block A1 is determined as the motion offset.

[0163] A reference block is selected within the same image by applying a motion offset to the current block's corresponding block. The reference block is the block within the same image located at a distance corresponding to the motion offset relative to the current block within the current image.

[0164] The current block and the reference block are divided equally. Each sub-block of the current block corresponds to each sub-block of the reference block. A sub-block of the current block can be called a current sub-block, and a sub-block of the reference block can be called a reference sub-block.

[0165] Export the motion information of each sub-block of the reference block. Specifically, export the motion information of the center position of each reference sub-block. Here, the center position can be the position of the lower right sample among the four adjacent samples located at the center of the reference sub-block. In another embodiment, the center position can be one of the four adjacent samples.

[0166] If the motion information of the reference sub-block is unavailable, the motion information of the center position of the reference block can be used as the motion information of that reference sub-block. Here, the center position can be the position of the lower right reference sub-block among the four adjacent reference sub-blocks located at the center of the reference block.

[0167] The motion information of each sub-block of the current block is derived from the motion information of each sub-block of the reference block. Each current sub-block uses the motion vector of each reference sub-block located at the corresponding position. If the reference sub-block has motion vectors in both directions L0 and L1, then the current sub-block can also have motion vectors in both directions L0 and L1. The motion information of all sub-blocks of the current block is called the motion vector field.

[0168] As an example, the motion information of each reference sub-block can be used as the motion information of each current sub-block. The L0 motion vector and L1 motion vector of each current sub-block can be the same as the motion information of each corresponding reference sub-block.

[0169] As another example, the motion vector of each current sub-block can be derived by applying temporal motion scaling to the motion vector of each reference sub-block. Specifically, the motion vector of each reference sub-block can be scaled by considering the Picture Order Count (POC) of the reference images referenced by the reference sub-block. POC refers to the value of a number assigned after the images are arranged in chronological order. Reference images with POC values ​​less than the current image's POC value can represent past images relative to the current image, and reference images with POC values ​​greater than the current image's POC value can represent future images relative to the current image. A reference image can be the first reference image in a list of reference images. The motion vector of each reference sub-block can be scaled based on a first difference between the POC of the current image and the POC of the reference images of the current block, and a second difference between the POC of the co-located image and the POC of the reference images of the reference block. The larger the second difference is relative to the first difference, the more the motion vector of the reference sub-block is scaled. The scaled motion vector of the reference sub-block is used as the motion vector of the current sub-block.

[0170] For the L0 and L1 reference images of the current block, the first reference image in the L0 reference image list and the first reference image in the L1 reference image list of the current block can be used. Alternatively, the reference image of a reference sub-block can be used as the reference image of the current block.

[0171] Each sub-block of the current block is predicted using derived motion information. In other words, the current block is predicted using a motion vector field. If the motion information of the reference sub-block includes L0 and L1 motion information, the current sub-block is predicted by weighted summation of the blocks indicated by L0 motion information in the L0 reference image and the blocks indicated by L1 motion information in the L1 reference image. The bidirectional weights of the current sub-block can be equal to the bidirectional weights of the reference sub-block.

[0172] Additionally, an SbTMVP mode flag can be defined to indicate whether the SbTMVP mode applies to the current block, and this can be signaled at at least one level in higher-level sequences (such as sequences, pictures, stripes, tiles, tile groups, and bricks) of the current block. For example, the SbTMVP mode flag can be named sps_sbtmvp_enabled_flag.

[0173] The affine mode is described below.

[0174] Affine motion compensation prediction is a predictive mode used for efficiently encoding video image signals that include motion such as scaling, rotation, and shearing. In conventional video coding / decoding systems, only a single motion vector is used to represent the optimal motion information for the current block on a block-by-block basis. In affine modes, however, the motion information of a block is defined on a pixel-by-pixel or sub-block-by-sub-block basis. Considering the complexity of deriving motion vectors on a pixel-by-pixel basis, motion information can be derived on a sub-block-by-sub-block basis.

[0175] According to the affine pattern, the pixels of the current block and / or the motion vectors per sub-block can be determined using two or more motion vectors associated with the current block. The sub-blocks are N×N in size, smaller than the current block. Here, N is a positive integer greater than or equal to 1. For example, N can be 4 or 8.

[0176] In affine mode, motion information of pixels within the current block or at the sub-block level can be derived and expressed using two or more control point motion vectors (CPMVs). CPMVs can be used to derive motion vectors for sub-blocks at specific locations within the current block. The set of motion vectors for sub-blocks within the current block is called an affine motion vector field (Affine MVF).

[0177] Motion compensation is performed within a reference image using pixels of the current block and / or individual motion vectors at the sub-block level to generate pixels of the current block and / or predicted pixels and / or predicted blocks of the sub-blocks. Here, the reference image for the current block can be the signaling notification for the current block or derived from an affine-based MVP candidate.

[0178] Furthermore, as a model for expressing motion information in affine mode, either a 4-parameter model or a 6-parameter model can be used. The 4-parameter model is advantageous for predicting motions such as scaling and rotation. The 6-parameter model is advantageous for predicting motions such as scaling, rotation, and shearing.

[0179] Figure 8a and Figure 8b An affine parameter model according to an embodiment of the present disclosure is shown.

[0180] Figure 8aThe diagram illustrates a 4-parameter model. A 4-parameter model uses two control point motion vectors (CPMVs) located at the top left and top right corners. Each CPMV includes two parameters corresponding to the x-axis and y-axis, and two CPMVs together comprise four parameters.

[0181] The 4-parameter model is shown in Equation 1. Based on Equation 1, the motion vector (mv) at the position (x, y) of the sample or sub-block within the current block can be derived. x ,mv y The position of the top-left sample of the current block can be represented as (0, 0). The position (x, y) of the sub-block can indicate the position of the bottom-right sample among the four samples at the center of the sub-block.

[0182] Equation 1

[0183] In equation 1, mv0 = {mv 0x ,mv 0y} represents the CPMV at the top left corner of the current block, mv1={mv 1x ,mv 1y} represents the CPMV at the top right position of the current block. W and H represent the width and height of the current block, respectively, and mv={mv x ,mv y} represents the motion vector at pixel position (x, y).

[0184] Figure 8B shows a 6-parameter model. The 6-parameter model uses three CPMVs located in the top left, top right, and bottom left corners.

[0185] The 6-parameter model is expressed as shown in Equation 2. The motion of a sample or sub-block within the current block at position (x, y) can be derived based on Equation 2.

[0186] Equation 2

[0187] In equation 2, mv0 = {mv 0x ,mv 0y} represents the top-left corner CPMV of the current block, mv1={mv 1x ,mv 1y} represents the CPMV at the top right corner of the current block, mv2={mv 2x ,mv 2y} represents the CPMV of the bottom left corner of the current block. W and H represent the width and height of the current block, respectively, and mv={mv x ,mv y} represents the motion vector at pixel position (x, y).

[0188] In Equations 1 and 2, the motion vector can be set to have 1 / 16 sample precision. Motion vector (mv) x ,mv y It can be rounded to the nearest 1 / 16 of the sample.

[0189] Additionally, an affine mode flag can be defined to indicate whether the affine mode applies to the current block, and this flag can be signaled at at least one higher level, such as the sequence, image, stripe, tile, tile group, and brick of the current block. For example, the affine mode flag can be named sps_affine_enabled_flag.

[0190] The video encoding device encodes affine-related syntax elements, including flags indicating whether affine motion prediction has been applied to the target block, type information indicating the type of the affine model, and motion information indicating the motion vector of each control point, and sends these to the video decoding device. When performing affine motion prediction, the type information and motion information of the control points can be signaled, and the motion vectors of the control points can be signaled in the number determined by the type information.

[0191] The video decoding device uses the syntax of the signaling notification to determine the type of affine model and CPMV, and uses Equation 1 or Equation 2 to calculate the motion vector (mv) of each 4×4 sub-block within the target block. x ,mv y If the signaling provides motion vector resolution information for the affine motion vector of the target block, then calculations such as rounding are used to round the motion vector (mv). x ,mv y The resolution is modified to the precision identified by the motion vector resolution information. The video decoding device uses the motion vector (mv) of each sub-block. x ,mv y Motion compensation is performed within the reference image to generate prediction blocks for each sub-block.

[0192] Furthermore, based on the encoding / decoding method of motion information, affine modes can be divided into detailed modes. As an example, affine modes can be further subdivided into affine MVP mode and affine merge mode. Affine merge mode can be used in sub-block-based merge modes.

[0193] In the affine MVP mode, the index of the reference image, MVD, and affine MVP index signaling can be notified.

[0194] In affine merge mode, an affine merge candidate list can be used to reduce the number of bits required for signaling notification to CPMV.

[0195] The affine merge candidate list may include at least one of the following: inherited affine merge candidates, constructed affine merge candidates, history-based parameter inheritance candidates, pairwise average affine merge candidates, and zero merge candidates. The affine merge candidate list may be constructed in the order described above. The affine merge candidate list may include a predetermined number of affine merge candidates.

[0196] Inherited affine merge candidates can refer to candidates derived using the CPMV of adjacent blocks encoded / decoded in affine mode.

[0197] The constructed affine merge candidate can refer to a candidate derived using CPMVs, where each CPMV is derived based on the motion vectors of the neighboring blocks for each control point (CP). Here, CP can refer to a specific location within the block used to derive the CPMV. For example, CP could be the location of each vertex of the block.

[0198] History-based parameter inheritance candidates refer to candidates derived from the stored affine parameters after they have been stored in the past. The motion vector of the current sub-block can be derived using affine parameters stored prior to the current block.

[0199] Pairwise average affine merge candidates refer to candidates derived by averaging the CPMV of two or more candidates in the sub-block-based merge candidate list, and then using the CPMV of the current block. Before adding pairwise average affine merge candidates, the sub-block-based merge candidate list includes two or more candidates.

[0200] Zero-merging candidate can refer to a candidate consisting of CPMVs of size 0.

[0201] Inherited affine merge candidates, constructed affine merge candidates, and zero merge candidates can be added to the affine merge candidate list in this order. If the number of inherited affine merge candidates and constructed affine merge candidates is less than a predetermined number, a zero merge candidate can be added.

[0202] First, the image processing device derives inherited affine merging candidates from the CPMV of neighboring blocks predicted in affine mode from the current block's neighboring blocks. For example, it can be derived from... Figure 4 The adjacent blocks (A0, A1, B0, B1, B2) of the current block shown derive a predetermined number of inherited affine merge candidates. In some implementations, the number of merge candidates derived from adjacent blocks predicted in an affine pattern can be limited. For example, a total of two inherited affine merge candidates can be derived from one of A0 and A1 and one of B0, B1, and B2. The priority order can be A0 and A1, and B0, B1, and B2. Each inherited affine merge candidate included in the affine merge candidate list corresponds to a combination of two or three CPMVs.

[0203] If the total number of inherited affine merge candidates is three or less, the image processing device can derive the missing number of constructed affine merge candidates from the translational motion vectors of adjacent blocks.

[0204] Figure 9 This is an example view showing the method for deriving constructed affine merging candidates for affine motion prediction.

[0205] The image processing device derives CPMV1, CPMV2, and CPMV3 from adjacent block groups {B2, B3, A2}, {B1, B0}, and {A1, A0}, respectively. As an example, the priority order within each adjacent block group can be B2, B3, and A2; B1 and B0; and A1 and A0. Furthermore, the image processing device derives CPMV4 from the corresponding block T within the reference image. The image processing device combines two or three of the four CPMVs to generate the required number of affine merge candidates. The priority of the construction is as follows: elements within each group are arranged in the order of top-left, top-right, and bottom-left CPMVs.

[0206] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}.

[0207] If the affine merge candidate list cannot be filled using inherited affine merge candidates and constructed affine merge candidates, the image processing device may add a zero motion vector as a candidate.

[0208] Simultaneously, the video coding unit selects an affine merge candidate from the affine merge candidate list for coding efficiency optimization, and performs affine motion prediction on the current block using the selected affine merge candidate. If the affine merge candidate consists of two CPMVs, affine motion prediction is performed using a 4-parameter model. If the affine merge candidate consists of three CPMVs, affine motion prediction is performed using a 6-parameter model. The video coding unit encodes the merge index indicating the selected affine merge candidate and signals it to the video decoding unit.

[0209] The video decoding device receives the merging index, constructs an affine merging candidate list in the same manner as the video encoding device, selects the affine merging candidate indicated by the merging index in the affine merging candidate list, and performs affine motion prediction using the control point motion vectors corresponding to the selected affine merging candidate.

[0210] Figure 10 This illustrates a prediction of the AMVP sub-block-based merging pattern according to an embodiment of this disclosure.

[0211] Reference Figure 10 This displays the reference images in the L0 reference image list, the current image, and the reference images in the L1 reference image list, in that order. The L0 reference image, the current image, and the L1 reference image are played at time points t-t0, t, and t+t0 in the playback order. Here, t0 and t1 are any positive integers.

[0212] exist Figure 10 In this model, the predicted blocks in the L0 direction of the current block are generated using the AMVP model, while the predicted blocks in the L1 direction of the current block are generated using the SbTMVP model. The motion vector AMVPMV of AMVP has an L0 direction relative to the current block, while the motion vector MergeMV of SbTMVP has an L1 direction relative to the current block. That is, the prediction directions according to the AMVP model and the prediction directions according to the SbTMVP model are different.

[0213] In another implementation, the predicted block in the L1 direction of the current block can be generated using an affine pattern. In yet another implementation, the predicted block in the L0 direction is generated using a sub-block-based merging pattern, and the predicted block in the L1 direction of the current block can be generated using an AMVP pattern. That is, the motion vector of the AMVP is AMVP. MV The motion vector of SbTMVP has an L1 orientation relative to the current block and is merged. MV It has the L0 direction relative to the current block.

[0214] For the current block, the predicted block in the L0 direction uses the motion vector AMVP generated by AMVP. MV Indicated reference block P L0 Generate. AMVP MV Encoding or decoding is performed using MVD and motion information from spatially and temporally adjacent blocks of the current block.

[0215] For the current block, the predicted block in the L1 direction is generated using the motion information of the block indicated by the motion offset within sub-blocks. Specifically, among the neighboring blocks of the current block, neighboring blocks of the reference copiography are selected, and the motion information of the selected neighboring blocks becomes the motion offset. The block indicated by the motion offset includes sub-blocks, and the motion information of each sub-block is used as the motion information of the sub-blocks of the current block. The motion information of the sub-blocks of the current block includes at least one of the scaled motion vectors in the L0 direction and the scaled motion vectors in the L1 direction of the sub-blocks within the block indicated by the motion offset. The sub-blocks of the current block use the reference sub-block C indicated by the motion information of the sub-blocks. L1 Make predictions. The sub-blocks of the current block can be predicted using the sub-blocks indicated on the L1 reference image by the L1 direction motion vector of the current block's sub-blocks. Here, the L1 reference image is the first image in the L1 reference image list.

[0216] The predicted block for the current block is generated by a weighted sum of the predicted blocks in the L0 direction and the predicted blocks in the L1 direction. The predicted block for the current block is equal to the weighted sum of the predicted blocks in the L0 direction and the predicted blocks in the L1 direction.

[0217] Here, the weights of the prediction blocks in the L0 direction and the prediction blocks in the L1 direction can be determined based on various factors. The two weights are real numbers less than or equal to 1, and the sum of the two weights can be 1.

[0218] In this implementation, the weights can be arbitrarily determined. The weights can be pre-specified between the video encoding device and the video decoding device.

[0219] In another implementation, the weights can be determined based on the distances between the current image and two reference images. The weights can be determined based on the ratio of the distance from the current image to the L0 reference image to the distance from the current image to the L1 reference image. For example, if the difference between the POC of the current image and the POC of the L0 reference image is less than the difference between the POC of the current image and the POC of the L1 reference image, then the weight of the predicted block in the L0 direction can be greater than the weight of the predicted block in the L1 direction. The above example can also be applied to the opposite case.

[0220] In another implementation, the weights can be determined based on information accessible to the video decoding device, such as signaling information or information deriveable by the video decoding device. For example, accessible information may include quantization parameters (QP), previously encoded / decoded syntax elements, image type, etc.

[0221] In another embodiment, weights can be determined from a predefined weight set based on index information provided by signaling notification. The video encoding apparatus can define multiple weight sets and can signal to the video decoding apparatus an index of one of these weight sets. The weight set includes weights for the predicted block in the L0 direction and weights for the predicted block in the L1 direction. The video decoding apparatus uses the index information provided by signaling notification to select one of the weight sets and uses the selected weight set to predict the current block.

[0222] In another implementation, the weights can be determined based on the L0 distortion value between the template of the predicted block in the L0 direction and the template of the current block, and the L1 distortion value between the template of the predicted block in the L1 direction and the template of the current block. Here, the template refers to a region consisting of samples that are adjacent to or not adjacent to the block, and the distortion value can refer to the difference. For example, if the L0 distortion value representing the difference between the template of the current block and the template of the predicted block in the L0 direction is greater than the L1 distortion value representing the difference between the template of the current block and the template of the predicted block in the L1 direction, then the weight of the predicted block in the L1 direction can be set to be greater than the weight of the predicted block in the L0 direction. Alternatively, if the ratio of the L1 distortion value to the L0 distortion value is greater than 1, then the weight of the predicted block in the L0 direction can be set to be greater than the weight of the predicted block in the L1 direction.

[0223] In an implementation, the weight can be determined based on a combination of at least one of the following: a predefined value, a value based on the distance between the current image and two reference images, a value based on information accessible by the video decoding device, a value based on index information from signaling notifications, and a distortion value.

[0224] Simultaneously, the signaling information used for the AMVP subblock merging mode can include a constraint flag and an AMVP subblock merging mode flag. The AMVP subblock merging mode flag indicates whether the AMVP subblock merging mode is applicable to the current block. The AMVP subblock merging mode flag can be an enabled flag (enabled_flag) or a disabled flag (disabled_flag). Setting the constraint flag to 0 indicates that there is no constraint on applying the AMVP subblock merging mode. The enabled and disabled flags can be set to either 0 or 1. Conversely, if the constraint flag is set to 1, the enabled flag is set to 0, indicating it is disabled, and the disabled flag is set to 1. Based on the constraint flag and the AMVP subblock merging mode flag, the video decoding device can determine that the prediction mode for the current block is the AMVP subblock merging mode.

[0225] Signaling information for AMVP subblock merging mode can be signaled at a higher level, such as in the Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), or Strip Header (SH).

[0226] As another example, signaling information for AMVP subblock merging mode can be signaled at a lower level, such as in the coding tree unit (CTU), coding unit (CU), prediction unit (PU), or transform unit (TU).

[0227] As another example, signaling information for the AMVP sub-block merging mode can be signaled not only without explicit flags, but also based on flags of other prediction modes, the size of the current block, etc. For instance, the AMVP sub-block merging mode flag can be signaled based on syntax elements related to inter-frame prediction or on syntax elements related to the AMVP merging mode. The video decoding device can determine the AMVP sub-block merging mode based on flags of other prediction modes, the size of the current block, etc.

[0228] Figure 11 This is a flowchart of a method for generating a candidate list based on sub-blocks according to an embodiment of the present disclosure.

[0229] exist Figure 11 In step S1110, the image processing device adds sub-block-based motion candidates to the sub-block-based candidate list.

[0230] Sub-block-based motion candidates can be classified into several types. Sub-block-based motion candidates can include first sub-block-based motion candidates and second sub-block-based motion candidates.

[0231] In the sub-block-based merging mode, the first sub-block-based motion candidate can be an SbTMVP candidate, and the second sub-block-based motion candidate can be an affine candidate.

[0232] As an example, candidates can be added to the sub-block-based merge candidate list in the order of SbTMVP candidates first, followed by affine-based MVP candidates. As another example, candidates can be added to the sub-block-based merge candidate list in the order of affine-based MVP candidates first, followed by SbTMVP candidates.

[0233] SbTMVP candidates can be added in the scan order of blocks that can be used as motion offsets in the adjacent blocks of the current block. Affine-based MVP candidates can be added in the following order: inherited affine merge candidates, constructed affine merge candidates, history-based parameter inheritance candidates, pairwise average affine merge candidates, and zero motion vector. In another implementation, SbTMVP candidates can be added in the reverse order of the scan order, and the order in which affine-based MVP candidates are added can be different.

[0234] The number of candidates in the sub-block-based candidate list can be preset or derived based on signaling information. The sub-block-based merge candidate list can consist of K SbTMVP candidates and L affine-based MVP candidates. Here, K and L are positive integers, and K can be less than or equal to the number of neighboring blocks being scanned. The maximum number of sub-block-based merge candidates can be 5.

[0235] In step S1120, the image processing device reorders the sub-block-based motion candidates in the sub-block-based candidate list.

[0236] Image processing apparatuses can reorder candidates within a sub-block-based merge candidate list using sub-block-based template matching cost. In the following text, template matching cost may refer to sub-block-based template matching cost.

[0237] The template matching cost based on sub-blocks refers to the difference between the sub-template of the current block and the template of the reference sub-block.

[0238] Figure 12 The cost of sub-block-based template matching according to an embodiment of this disclosure is illustrated.

[0239] exist Figure 12 In this process, the current block is divided into multiple current sub-blocks, and the current sub-blocks are predicted unidirectionally.

[0240] The template for the current block can be a region that includes adjacent samples to the left and top of the current block. The template for the current block can be divided into multiple sub-templates based on the size of the sub-blocks of the current block.

[0241] Sub-templates can be assigned to some of the current block's sub-blocks. Sub-templates can be assigned to the top, left, and top-left sub-blocks within the current block. For example, in the current block's sub-templates, the sub-template at the top of sub-block D can be used as the template for sub-block D. As another example, in the current block's sub-templates, the sub-templates to the left and top of sub-block A can be used as the template for sub-block A. Sub-blocks within the current block that are not adjacent to the top or left boundary of the current block may not be assigned sub-templates.

[0242] Simultaneously, the reference sub-block of the current sub-block can also have a sub-template corresponding to the template of the current sub-block. Reference sub-blocks A, B, C, D, E, F, and G are sub-blocks located on the reference image, respectively, at a distance of one motion vector from the current sub-blocks A, B, C, D, E, F, and G.

[0243] The template matching cost based on a sub-block can be calculated based on the difference between the sub-template of the current sub-block and the sub-template of the corresponding reference sub-block. For example, the difference between the top sub-template of the current sub-block D and the top sub-template of the reference sub-block Dref can be calculated as the template matching cost based on the sub-block.

[0244] The cost of template matching based on sub-blocks, i.e. the difference between templates, can be defined as a cost function. As a cost function, the sum of absolute differences (SAD), the sum of absolute transformation differences (SATD), the sum of mean-absolute differences (MR-SAD), the mean squared error (MSE), or the sum of squared errors (SSE) can be used.

[0245] The image processing device can reorder the candidates in the sub-block-based candidate list using various methods based on the sub-block-based template matching cost.

[0246] As an example, the image processing apparatus can reorder K SbTMVP candidates based on sub-block-based template matching costs. The K SbTMVP candidates can be listed in order of lower sub-block-based template matching costs.

[0247] As another example, the image processing device can reorder L affine-based MVP candidates based on the sub-block-based template matching cost. The L affine-based MVP candidates can be listed in order of lower sub-block-based template matching costs.

[0248] As another example, the image processing apparatus can reorder the K SbTMVP candidates and L affine-based MVP candidates according to the sub-block-based template matching cost. The K SbTMVP candidates are reordered in order of their smaller sub-block-based template matching costs, and the L affine-based MVP candidates can be listed in order of their smaller sub-block-based template matching costs.

[0249] As another example, the image processing device can select T candidates from a sub-block-based merge candidate list based on the sub-block-based template matching cost, and reorder the selected candidates according to the sub-block-based template matching cost, thereby reorganizing the sub-block-based merge candidate list. Here, T is a positive integer less than or equal to K+L. From the sub-block-based merge candidate list, T candidates with smaller sub-block-based template matching costs are selected, and these T candidates can be input into the sub-block-based merge candidate list in order of smaller sub-block-based template matching costs.

[0250] As another example, the image processing apparatus can select U candidates from K SbTMVP candidates based on a sub-block-based template matching cost, and V candidates from L affine-based MVP candidates based on a sub-block-based template matching cost, and use the selected candidates as sub-block-based merge candidates. Here, U is a positive integer less than or equal to K, and V is a positive integer less than or equal to L. The sub-block-based merge candidate list includes U SbTMVP candidates and V affine-based MVP candidates.

[0251] In another implementation, candidate reordering can be omitted. The image processing apparatus can select candidates at specific positions or in a specific order within a list of merged candidates for sub-blocks, and can use the selected candidates to predict the current block.

[0252] Although the steps in the various flowcharts are described as being executed sequentially, these steps merely illustrate the technical ideas of some embodiments of this disclosure. Therefore, those skilled in the art to which this disclosure pertains can perform the steps by changing the sequence described in the various figures or by performing two or more steps in parallel. Thus, the steps in the various flowcharts are not limited to the temporal order shown.

[0253] It should be understood that the above description presents illustrative embodiments that can be implemented in various other ways. The functionality described in some embodiments may be implemented by hardware, software, firmware, and / or combinations thereof. It should also be understood that the functional components described in this disclosure are designated as "...units" to strongly emphasize their potential for independent implementation.

[0254] Furthermore, the various methods or functions described in some embodiments can be implemented as instructions stored in one or more processor-readable and executable non-transitory recording media. Non-transitory recording media can include, for example, various types of recording devices in which data is stored in a computer-system-readable form. For example, non-transitory recording media can include storage media such as erasable programmable read-only memory (EPROM), flash memory drives, optical disc drives, hard disk drives, and solid-state drives (SSDs).

[0255] Although embodiments of this disclosure have been described for illustrative purposes, those skilled in the art to which this disclosure pertains will understand that various modifications, additions, and substitutions can be made without departing from the spirit and scope of this disclosure. Therefore, embodiments of this disclosure have been described for the sake of brevity and clarity. The scope of the technical concept of the embodiments of this disclosure is not limited by these descriptions. Therefore, those skilled in the art to which this disclosure pertains should understand that the scope of this disclosure should not be limited by the embodiments explicitly described above, but rather by the claims and their equivalents.

[0256] Cross-references to related applications

[0257] This application claims priority and benefits to Korean Patent Application No. 10-2024-0045767, filed on April 4, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. A video decoding method, comprising: Export the block-based motion information for the current block; The first prediction block of the current block is generated using the block-based motion information; Export the sub-block-based motion information of the current block; A second predicted block for the current block is generated by predicting the current block on a sub-block basis using the motion information based on sub-blocks; and The prediction block for the current block is generated by performing a weighted summation on the first prediction block and the second prediction block.

2. The video decoding method according to claim 1, wherein, The weights of the first prediction block and the second prediction block are determined based on the distance between the current image and the first reference image used to generate the first prediction block, and the distance between the current image and the second reference image used to generate the second prediction block.

3. The video decoding method according to claim 1, wherein, Based on the weight set index of the signaling notification, the weights of the first prediction block and the second prediction block are selected from the predefined weight set.

4. The video decoding method according to claim 1, wherein, The weights of the first prediction block and the second prediction block are determined based on the template difference between the first prediction block and the current block, and the template difference between the second prediction block and the current block.

5. The video decoding method according to claim 1, wherein, The first reference image used to generate the first prediction block and the second reference image used to generate the second prediction block have different prediction directions.

6. The video decoding method according to claim 1, wherein, The block-based motion information of the current block is derived as follows: Generate a block-based motion candidate list for the current block based on the pre-reconstructed blocks; Based on the signaling notification-based block-based motion candidate index, block-based motion candidates are determined from the block-based motion candidate list; and The block-based motion information of the current block is determined using the motion information of the block-based motion candidates.

7. The video decoding method according to claim 1, wherein, The block-based motion information of the current block is derived as follows: Within the pre-reconstructed region, a reference block is searched based on template matching cost; and The motion information of the reference block is used to determine the block-based motion information of the current block.

8. The video decoding method according to claim 1, wherein, The process of deriving the sub-block-based motion information of the current block includes: Based on blocks including pre-reconstructed sub-blocks, generate a sub-block-based motion candidate list for the current block; and The sub-block motion information of the current block is determined from the sub-block motion candidates based on the bidirectional matching cost (bidirectional matching cost) between the block predicted using the sub-block motion candidate list and the first predicted block.

9. The video decoding method according to claim 1, wherein, The process of deriving the sub-block-based motion information of the current block includes: Based on a pre-reconstructed block including sub-blocks, a sub-block-based motion candidate list is generated for the current block; and Based on the sub-block-based motion candidate index of the signaling notification, the sub-block-based motion information of the current block is determined from the sub-block-based motion candidate list.

10. The video decoding method according to claim 9, wherein, The pre-reconstructed block includes at least one of the spatial neighboring blocks, temporal neighboring blocks, and history-based neighboring blocks of the current block.

11. The video decoding method according to claim 9, wherein, The pre-reconstructed block includes at least one neighboring block, which references a co-located image (co-located image) of the current image including the current block.

12. The video decoding method according to claim 9, wherein, Exporting the sub-block-based motion information of the current block further includes: Based on the template matching cost between the sub-templates of the current block and the sub-templates of the motion candidates based on the sub-block, the order of the motion candidates based on the sub-block is reordered.

13. The video decoding method according to claim 12, wherein, Reordering the sub-block-based motion candidates includes: The first sub-block-based motion candidate in the sub-block-based motion candidate is reordered; and The second sub-block-based motion candidate in the sub-block-based motion candidate is reordered.

14. The video decoding method according to claim 12, wherein, Reordering the sub-block-based motion candidates includes: Based on the template matching cost between the sub-templates of the current block and the sub-templates of the sub-block-based motion candidates, a predetermined number of candidates are selected from the sub-block-based motion candidates; and Based on the template matching cost, the sub-block-based motion candidate list, which includes a predetermined number of candidates, is reorganized in order.

15. The video decoding method according to claim 1, further comprising: When the distance from the current image to the first reference image used to generate the first prediction block is different from the distance to the second reference image used to generate the second prediction block, the block-based motion information and the sub-block-based motion information are corrected based on bidirectional optical flow (BDOF) technology.

16. A video coding method, comprising: Export the block-based motion information for the current block; The first prediction block of the current block is generated using the block-based motion information; Export the sub-block-based motion information of the current block; A second predicted block for the current block is generated by predicting the current block on a sub-block basis using the motion information based on sub-blocks; and The prediction block for the current block is generated by performing a weighted summation on the first prediction block and the second prediction block.

17. A method for transmitting data including a bitstream of video, the method comprising: Generate the bitstream of the video; and Transmitting data including the bit stream, Generating the bitstream includes: Export the block-based motion information for the current block; Using the block-based motion information, generate the first prediction block for the current block; Export the sub-block-based motion information of the current block; A second predicted block for the current block is generated by predicting the current block on a sub-block basis using the motion information based on sub-blocks; and The prediction block for the current block is generated by performing a weighted summation on the first prediction block and the second prediction block.

Images