Image encoding method using merge mode and apparatus therefor

Abstract

The image decoding method according to embodiments of the present disclosure includes the following steps: configuring a merge candidate list for the current block; dividing the merge candidates in the merge candidate list into one or more subgroups; configuring a modified merge candidate list for the current block by rearranging the merge candidate list for each subgroup based on the template matching (TM) cost of the merge candidates; and deriving a prediction sample for the current block based on the modified merge candidate list, wherein the rearranged subgroups are subgroups in which the merge candidates are rearranged based on the TM cost.

Description

Technical Field

[0001] This disclosure relates to image / video coding methods and image / video coding devices. Background Technology

[0002] Video coding is used in a variety of applications, such as digital storage media, television broadcasting, video streaming services, and real-time communications. Recently, the demand for high-resolution and high-quality video has been growing across various sectors.

[0003] As images / videos reach higher resolutions and higher quality, their data size increases, resulting in a relative increase in the amount of information or bits transmitted.

[0004] Furthermore, interest and demand for immersive media, such as virtual reality (VR), artificial reality (AR), mixed reality (MR) content, or holograms, are currently increasing. Additionally, attempts to use immersive media to deliver immersive experiences are growing in various sectors, including gaming, education, medicine, real estate, and marketing.

[0005] Therefore, efficient video coding techniques are needed to effectively compress, transmit, store, and reproduce high-resolution and high-quality video information with the various characteristics described above. Summary of the Invention

[0006] Technical solution

[0007] According to embodiments of this disclosure, methods and apparatus for enhancing video / image coding efficiency are provided.

[0008] According to embodiments of this disclosure, a method and apparatus for inter-frame predictive video coding are provided.

[0009] According to embodiments of this disclosure, an image decoding method performed by a decoding device is provided. The method includes the following steps: constructing a merge candidate list for a current block; dividing the merge candidates in the merge candidate list into at least one subgroup; constructing a modified merge candidate list for the current block by reordering the merge candidate list for each subgroup based on the template matching (TM) cost of the merge candidates; and deriving a prediction sample for the current block based on the modified merge candidate list, wherein the reordered subgroups are subgroups in which the merge candidates within the subgroup are reordered based on the TM cost.

[0010] According to embodiments of this disclosure, an image encoding method performed by an encoding device is provided. The method includes the following steps: constructing a merge candidate list for a current block; dividing the merge candidates in the merge candidate list into at least one subgroup; constructing a modified merge candidate list for the current block by reordering the merge candidate list for each subgroup based on the template matching (TM) cost of the merge candidates; deriving a prediction sample for the current block based on the modified merge candidate list; and encoding image information including prediction-related information of the current block, wherein the reordered subgroups are subgroups in which the merge candidates within the subgroup are reordered based on the TM cost.

[0011] According to embodiments of the present disclosure, a decoding apparatus for image decoding is provided. The decoding apparatus includes a memory and at least one processor connected to the memory, wherein the at least one processor is configured to perform the following operations: constructing a merge candidate list for a current block; dividing the merge candidates in the merge candidate list into at least one subgroup; constructing a modified merge candidate list for the current block by reordering the merge candidate list for each subgroup based on the template matching (TM) cost of the merge candidates; and deriving a prediction sample for the current block based on the modified merge candidate list, wherein the reordered subgroups are subgroups in which the merge candidates within the subgroup are reordered based on the TM cost.

[0012] According to embodiments of the present disclosure, an encoding apparatus for image encoding is provided. The encoding apparatus includes a memory and at least one processor connected to the memory, wherein the at least one processor is configured to perform the following operations: constructing a merge candidate list for a current block; dividing the merge candidates in the merge candidate list into at least one subgroup; constructing a modified merge candidate list for the current block by reordering the merge candidate list for each subgroup based on the template matching (TM) cost of the merge candidates; deriving a prediction sample for the current block based on the modified merge candidate list; and encoding image information including prediction-related information of the current block, wherein the reordered subgroups are subgroups in which the merge candidates within the subgroup are reordered based on the TM cost.

[0013] According to embodiments of the present disclosure, a method is provided for transmitting video / video data comprising a bitstream generated by a video / image encoding method according to at least one embodiment of the present disclosure.

[0014] According to embodiments of the present disclosure, an apparatus is provided for transmitting video / video data comprising a bitstream generated by a video / image encoding method according to at least one embodiment of the present disclosure.

[0015] According to embodiments of the present disclosure, a computer-readable storage medium is provided that stores a program for performing a method according to at least one embodiment of the present disclosure.

[0016] According to embodiments of the present disclosure, a computer-readable digital storage medium is provided that stores encoded video / image information generated by a video / image encoding method according to at least one embodiment of the present disclosure.

[0017] According to embodiments of the present disclosure, a computer-readable digital storage medium is provided that stores encoded information or encoded video / image information, the encoded information or encoded video / image information causing a decoding device to perform a video / image decoding method according to at least one embodiment of the present disclosure.

[0018] Technical effect

[0019] According to the embodiments of this disclosure, the overall video / image compression efficiency can be improved.

[0020] According to embodiments of this disclosure, the prediction performance of the current block can be improved.

[0021] According to the embodiments of this disclosure, the merge candidate list of the current block can be divided into subgroups, and the merge candidates can be reordered for each subgroup based on template matching (TM) cost, thereby improving the inter-frame prediction accuracy of the current block and improving the overall coding efficiency. Attached Figure Description

[0022] Figure 1 Examples of video / image coding systems to which embodiments of the present disclosure are applicable are illustrated schematically.

[0023] Figure 2 This is a diagram that schematically illustrates the configuration of a video / image encoding device to which embodiments of the present disclosure are applicable.

[0024] Figure 3 This is a diagram that schematically illustrates the configuration of a video / image decoding device to which embodiments of the present disclosure are applicable.

[0025] Figure 4 An example of the inter-frame prediction process is illustrated.

[0026] Figure 5 An example of a video / image coding method based on inter-frame prediction is shown.

[0027] Figure 6 An example of a video / image decoding method based on inter-frame prediction is shown.

[0028] Figure 7A reference template for deriving template matching (TM) cost is shown, where TM cost is the MVD prediction cost of the MV candidate.

[0029] Figure 8 A reference template for deriving template matching (TM) cost is illustrated, where TM cost is the MVD prediction cost of MV candidates in affine AMVP pattern or affine MMVD pattern.

[0030] Figure 9 An example of an MMVD candidate is shown, which is a combination of available signs and magnitudes.

[0031] Figure 10 An example is provided illustrating a partition shape supported by the Geometric Partitioning Mode (GPM).

[0032] Figure 11 Intra-prediction modes that can be used as IPM candidates are illustrated by example.

[0033] Figure 12 A flowchart of an affine motion prediction method according to an embodiment of the present disclosure is illustrated by way of example.

[0034] Figure 13 An example of constructing an affine merge candidate list for the current block is shown.

[0035] Figure 14 An example is provided illustrating the adjacent blocks of the current block used to derive the affine candidate for inheritance.

[0036] Figure 15 An example is provided of the adjacent blocks of the current block used to derive the affine candidate for construction.

[0037] Figure 16 An example is shown of adjacent sub-blocks input into a linear regression process used to derive the parameter set of a linear model.

[0038] Figure 17 An implementation of a reference sample for generating a reference template for merging candidates through bidirectional prediction is illustrated.

[0039] Figure 18 The template for the current block and the reference template for merging candidates based on sub-blocks are shown.

[0040] Figure 19 An implementation of determining whether to perform a reordering is performed based on the execution criteria of Adaptive Reordering (ARMC-TM) with template matching merge candidates is illustrated.

[0041] Figure 20 A video / image coding method according to an embodiment of the present disclosure is illustrated schematically.

[0042] Figure 21A video / image decoding method according to an embodiment of the present disclosure is illustrated schematically. Detailed Implementation

[0043] Because this disclosure can have various modifications and implementations, specific implementations are illustrated in the accompanying drawings and will be described in detail. However, it should be understood that there is no intention to limit the implementations of this disclosure to that particular implementation. The terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the technical scope of this disclosure. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise. As used herein, the term “and / or” includes any and all combinations of the associated listed items. As used herein, the terms “comprising,” “including,” and “having” specify the presence of the stated features, quantities, operations, elements, components, and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, elements, components, and / or combinations thereof. In this disclosure, the use of the term “may” (e.g., regarding what the example or implementation may include or implement) in conjunction with an example or implementation indicates the existence of at least one example or implementation that includes or implements that feature, but all examples are not limited thereto, and the corresponding feature or configuration may be omitted.

[0044] Each component in the accompanying drawings described in this disclosure is shown independently to facilitate the explanation of its different features and functions, which does not imply that each component is implemented as separate hardware or separate software. For example, two or more of these components may be combined to form a single component, or a single component may be divided into multiple components. Embodiments in which components are integrated and / or separated are also included within the scope of this disclosure without departing from its spirit.

[0045] In this disclosure, “A or B” can mean “A only”, “B only”, or “both A and B”. In other words, “A or B” can be interpreted in this disclosure as “A and / or B”. For example, “A, B or C” in this disclosure can mean “A only”, “B only”, “C only”, or “any and all combinations of A, B and C”.

[0046] As used in this article, a forward slash ( / ) or a comma can mean "and / or". For example, "A / B" can mean "A and / or B". Therefore, "A / B" can mean "A only", "B only", or "both A and B". For example, "A, B, C" can mean "A, B, or C".

[0047] In this disclosure, "at least one of A and B" can mean "only A", "only B" or "both A and B". Furthermore, the expressions "at least one of A or B" or "at least one of A and / or B" can be interpreted in the same way as "at least one of A and B".

[0048] In this disclosure, "at least one of A, B, and C" can mean "A only", "B only", "C only", or "any and all combinations of A, B, and C". Furthermore, "at least one of A, B, or C" or "at least one of A, B, and / or C" can mean "at least one of A, B, and C".

[0049] The brackets used in this disclosure may mean "for example". Specifically, when indicated as "prediction (intra-frame prediction)", "intra-frame prediction" can be presented as an example of "prediction". In other words, "prediction" in this disclosure is not limited to "intra-frame prediction", and "intra-frame prediction" can be presented as an example of "prediction". Furthermore, even when indicated as "prediction (i.e., intra-frame prediction)", "intra-frame prediction" can be presented as an example of "prediction".

[0050] In this disclosure, the technical features described in a single figure may be implemented independently or simultaneously.

[0051] This disclosure relates to video / image coding. For example, the methods / implementations described in this disclosure can be applied to enhanced compression models or methods disclosed in the H.267 standard. Furthermore, the methods / implementations disclosed herein can be applied to methods disclosed in the AOMedia Video 2 (AV2) standard or next-generation video / image coding standards (e.g., H.268 and H.269).

[0052] In this disclosure, encoding may include encoding and / or decoding. In this disclosure, image encoding may be used interchangeably with video encoding.

[0053] In this disclosure, video can refer to a collection of images over time. An image typically refers to a unit representing a single image at a particular time, and a slice / tile refers to a unit that forms part of an image in encoding. A slice / tile may include one or more coding tree units (CTUs). A single image may include one or more slices / tiles. A tile may represent a rectangular area of ​​a CTU within a specific tile row and a specific tile column of an image.

[0054] A single image can be divided into two or more sub-images. A sub-image can be a rectangular region of one or more slices of the image.

[0055] A pixel or cell can refer to the smallest unit that makes up a picture (or image). A "sample" can be used as the term corresponding to a pixel. A sample can generally represent a pixel or pixel value, and can represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component.

[0056] A unit can represent the basic unit of image processing. A unit may include a specific region of an image and at least one piece of information associated with that region. A single unit may include a luminance block and two chrominance (e.g., Cb and Cr) blocks. The term "unit" may be used interchangeably with terms such as "block" or "region" in some cases. Typically, an M×N block may include a set (or array) of samples or transform coefficients in M ​​columns and N rows.

[0057] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Furthermore, the same reference numerals may be used throughout the drawings to indicate the same elements, and redundant descriptions of the same elements may be omitted.

[0058] Figure 1 Examples of video / image coding systems to which embodiments of the present disclosure may be applied are illustrated schematically.

[0059] Reference Figure 1 A video / image encoding system may include a first device (encoding device) and a second device (decoding device). The first device may deliver encoded video / image information or data to the second device in the form of a file or stream via a digital storage medium or network.

[0060] A video / image encoding system may also include a video / image acquisition device and a video / image renderer. The video / image acquisition device may be included in the encoding device, or it may be configured as a separate device or an external component. The video / image renderer may be included in the decoding device, or it may be configured as a separate device or an external component.

[0061] The first device may include a transmitter as an internal component, or a transmitter as a separate device or an external component.

[0062] The second device may include a receiver as an internal component, or as a receiver as a separate device or an external component.

[0063] An encoding device may be called an encoder, and a decoding device may be called a decoder. A transmitter may be included in the encoding device. A receiver may be included in the decoding device. A renderer may include a display, and the display may be configured as a separate device or an external component.

[0064] The decoding and encoding devices applied in one or more embodiments of this disclosure can be included in multimedia broadcasting transmitters / receivers, mobile communication terminals, home theater video devices, digital cinema video devices, surveillance cameras, video conferencing devices, real-time communication devices such as video communication, mobile streaming devices, storage media, cameras, video-on-demand (VoD) service providers, over-the-air (OTT) video devices, internet streaming service providers, three-dimensional (3D) video devices, virtual reality (VR) devices, augmented reality (AR) devices, video telephony devices, transportation terminals (e.g., vehicle terminals (including autonomous vehicle terminals), aircraft terminals, and ship terminals), and medical video devices, and can be used to process video signals or data signals. For example, over-the-air (OTT) video devices can include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, and digital video recorders (DVRs).

[0065] A video / image acquisition device can acquire video / image sources. The video / image acquisition device can acquire video / images through processes of capturing, compositing, or generating video / images. The video / image acquisition device may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras and a video / image archive including previously captured video / images. The video / image generation device may include, for example, a camera, a computer, a tablet PC, and a smartphone, and can (electronically) generate video / images. For example, virtual video / images can be generated by a computer, in which case the video / image capture process can be replaced by a process of generating relevant data. The video / image source can perform a video / image preprocessing process to input the optimized video / image into the encoder.

[0066] Encoding devices can encode input video / images. They can perform a series of processes, such as prediction, transformation, and quantization, to achieve compression and encoding efficiency. The encoded data (encoded video / image information) can be output as a bitstream.

[0067] The transmitter can send encoded images / image information or data, output as a bitstream, to a receiver of a receiving device via a digital storage medium or network, either as a file or a stream. The encoded images / image information or data output as a bitstream can be sent to the receiver via a streaming server. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter can include elements for generating media files according to a predetermined file format and may include elements for transmission via a broadcast / communication network. The receiver can receive / extract the bitstream and send the received bitstream to a decoding device.

[0068] A streaming server can temporarily store bitstreams during the sending or receiving of bitstreams. Based on user requests via a web server, the streaming server sends multimedia data to the user's device, with the web server acting as an intermediary to notify the user of available services. When a user requests a desired service from the web server, the web server forwards the request to the streaming server, and the streaming server sends the multimedia data to the user. The content streaming system may include a separate control server, in which case the control server manages the commands / responses between devices within the content streaming system.

[0069] A streaming server can receive content from media storage devices and / or encoding devices. For example, when receiving content from an encoding device, the content can be received in real time. In this case, the streaming server can store the bitstream for a certain period of time to provide a smooth streaming service.

[0070] Decoding devices can decode video / images by performing a series of processes such as dequantization, inverse transform, and prediction, which correspond to the operations of encoding devices.

[0071] The renderer can render the decoded video / images. The rendered video / images can then be displayed on the monitor.

[0072] Figure 2 This diagram schematically illustrates the configuration of a video / image encoding apparatus to which embodiments of the present disclosure may be applied. Hereinafter, the encoding apparatus may include image encoding apparatus and / or video encoding apparatus.

[0073] Reference Figure 2 The encoding device 200 may include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-frame predictor and an intra-frame predictor. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstruction block generator. According to embodiments, the image partitioner 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 may be configured as at least one hardware component (e.g., an encoder chipset or processor). The memory 270 may include a decoded picture buffer (DPB) or may be configured as a digital storage medium. The hardware component may also include the memory 270 as an internal / external component.

[0074] Image partitioner 210 can partition an input image (or picture or frame) input to encoding device 200 into one or more processing units. For example, a processing unit may be called a coding unit (CU). In this case, the coding unit can be recursively partitioned from a coding tree unit (CTU) or a maximum coding unit (LCU) according to a quadtree-binary-tritree (QTBTTT) structure. For example, a single coding unit can be partitioned into multiple coding units of greater depth based on a quadtree structure, a binary tree structure, and / or a ternary tree structure. In this case, for example, a quadtree structure can be applied first, and a binary tree structure and / or a ternary tree structure can be applied later. Alternatively, a binary tree structure can be applied first. The encoding process according to this disclosure can be performed based on the final coding unit that is no longer partitioned. In this case, based on the encoding efficiency according to the image characteristics, the maximum coding unit can be used as the final coding unit, or, if necessary, the coding unit can be recursively partitioned into deeper coding units, thereby using the coding unit with the optimal size as the final coding unit. Here, the encoding process may include prediction, transformation, and reconstruction processes, which will be described below. In another example, the processing unit may also include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit can be split or partitioned from the aforementioned final encoding unit. The prediction unit may be a unit for sample prediction, and the transformation unit may be a unit for deriving the transform coefficients and / or a unit for deriving the residual signal from the transform coefficients.

[0075] The term "unit" may be used interchangeably with the terms "cell" or "region" depending on the context. Typically, an M×N block can represent an array of samples or transform coefficients arranged in M ​​rows and N columns. A sample can typically represent a pixel or pixel value, and may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component. "Sample" can be used as a term corresponding to a pixel or cell in a single picture (or image).

[0076] Encoding device 200 generates a residual signal (residual signal, residual block, or residual sample array) by subtracting the prediction signal (prediction block or prediction sample array) output from the predictor from the input image signal (original block or original sample array), and the generated residual signal is sent to converter 232. In this case, as illustrated, the component in encoder 200 used to subtract the prediction signal (prediction block or prediction sample array) from the input image signal (original block or original sample array) can be called subtractor 231. The predictor can perform prediction on the processing target block (hereinafter referred to as the current block) and can generate a prediction block including prediction samples for the current block. The predictor can determine whether intra-frame prediction or inter-frame prediction is applied based on the current block or CU. The predictor can generate various information about the prediction, such as prediction mode information, and can send the generated information to entropy encoder 240, as described below in the description of each prediction mode. The information about the prediction can be encoded by entropy encoder 240 and output as a bitstream.

[0077] An intra-frame predictor can refer to samples within the current image to predict the current block. Depending on the prediction mode, the referenced samples can be located either adjacent to the current block or far from it. In intra-frame prediction, prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC modes and planar modes. Directional modes can include, for example, 33 or 65 directional prediction modes based on the granularity of the prediction direction. However, this example is for illustration only, and more or fewer directional prediction modes can be used depending on the configuration. The intra-frame predictor can determine the prediction mode applied to the current block based on the prediction modes applied to neighboring blocks.

[0078] Inter-frame predictors can derive predicted blocks for the current block based on reference blocks (reference sample arrays) specified by motion vectors on a reference image. Here, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted based on the correlation between the motion information of neighboring blocks and the current block, on a block, sub-block, or sample basis. Motion information can include motion vectors and reference image indices. Motion information can also include inter-frame prediction direction (L0 prediction, L1 prediction, and bidirectional prediction) information. In inter-frame prediction, neighboring blocks can include spatially adjacent blocks existing within the current image and temporally adjacent blocks existing in the reference image. The reference image including the reference block and the reference image including the temporally adjacent block can be the same or different. A temporally adjacent block can be called a collinear reference block or collinear CU (colCU), and a reference image including a temporally adjacent block can also be called a collinear image (colPic). For example, the inter-frame predictor can configure a motion information candidate list based on neighboring blocks and can generate information indicating candidates for deriving the motion vector and / or reference image index for the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip and merge modes, the inter-frame predictor can use motion information about neighboring blocks as motion information about the current block. In skip mode, unlike merge mode, the residual signal may not be sent. In motion vector prediction (MVP) mode, the motion vector of the current block can be indicated by using the motion vectors of neighboring blocks as the motion vector predictor's concurrent signal to notify the motion vector difference.

[0079] Predictor 220 can generate a prediction signal based on various prediction methods described below. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but can also apply intra-frame prediction and inter-frame prediction simultaneously, which can be called combined inter-frame and intra-frame prediction (CIIP). Furthermore, the predictor can predict blocks based on an intra-block copy (IBC) prediction mode or a palette mode. The IBC prediction mode or palette mode can be used, for example, for screen content coding (SCC). IBC essentially performs prediction within the current frame, but can be performed similarly to inter-frame prediction in that it derives a reference block within the current frame based on a block vector. That is, IBC can utilize at least one of the inter-frame prediction techniques described in this disclosure.

[0080] The predicted signal generated by predictor 220 can be used to generate a reconstructed signal or a residual signal. Transformer 232 can generate transform coefficients by applying transform techniques to the residual signal. For example, the transform techniques may include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loève Transform (KLT), Graph-Based Transform (GBT), or Conditional Nonlinear Transform (CNT).

[0081] Quantizer 233 can quantize the transform coefficients and send the quantized transform coefficients to entropy encoder 240, which can encode the quantized signal (information about the quantized transform coefficients) and output the encoded signal as a bitstream. The information about the quantized transform coefficients can be called residual information. Quantizer 233 can rearrange the block-form quantized transform coefficients into a one-dimensional vector form based on the coefficient scan order, and can generate information about the transform coefficients based on the one-dimensional vector form of the quantized transform coefficients. Entropy encoder 240 can perform various encoding methods, such as Exponential Golomb, Context Adaptive Variable Length Coding (CAVLC), and Context Adaptive Binary Arithmetic Coding (CABAC). Entropy encoder 240 can encode information necessary for video / image reconstruction (e.g., values ​​of syntax elements) other than the quantized transform coefficients, either together or separately. The encoded information (e.g., encoded video / image information) can be sent or stored as a bitstream based on a Network Abstraction Layer (NAL). The video / image information may also include information about various parameter sets, such as Adaptive Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). Furthermore, the video / image information may also include general constraint information. In this disclosure, information and / or syntax elements that transmit / signal notifications from the encoding device to the decoding device may be included in the video / image information. The video / image information may be encoded by the aforementioned encoding process and included in a bitstream. The bitstream may be transmitted over a network or stored in a digital storage medium. The network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmitter (not shown) and / or storage device (not shown) for transmitting and / or storing signals output from the entropy encoder 240 may be configured as internal / external components of the encoding device 200, or the transmitter may be included in the entropy encoder 240.

[0082] The quantized transform coefficients output from quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients using dequantizer 234 and inverse transform unit 235. Adder 250 can add the reconstructed residual signal to the prediction signal output from the predictor to generate a reconstructed signal (reconstructed image, reconstructed block, or array of reconstructed samples). When there is no residual for the processing target block, such as when a skip mode is applied, the prediction block can be used as a reconstructed block. Adder 250 can be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current image, or it can be used for inter-frame prediction of the next image after being filtered as follows.

[0083] Luminance mapping with chroma scaling (LMCS) can be applied in image encoding and / or reconstruction processes.

[0084] Filter 260 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 260 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and the modified reconstructed image can be stored in memory 270, specifically in the DPB of memory 270. Various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering. Filter 260 can generate filtering-related information and can send the generated information to entropy encoder 240. The filtering-related information can be encoded by entropy encoder 240 and output as a bitstream.

[0085] The modified reconstructed image sent to memory 270 can be used as a reference image in the inter-frame predictor. When inter-frame prediction is applied via the modified reconstructed image, the encoding device can avoid prediction mismatch between the encoding device 200 and the decoding device, and can improve encoding efficiency.

[0086] The DPB of memory 270 can store modified reconstructed images for use as reference images in the inter-frame predictor. Memory 270 can store motion information about blocks in the current image from which motion information is derived (or encoded) and / or about blocks in the reconstructed image. The stored motion information can be sent to the inter-frame predictor to be used as motion information about spatially adjacent blocks or about temporally adjacent blocks. Memory 270 can store reconstructed samples of reconstructed blocks in the current image and can send these reconstructed samples to the intra-frame predictor.

[0087] Figure 3 This diagram schematically illustrates the configuration of a video / image decoding device to which embodiments of the present disclosure can be applied. Hereinafter, the decoding device may include an image decoding device and / or a video decoding device.

[0088] Reference Figure 3The decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter-frame predictor and an intra-frame predictor. The residual processor 320 may include a dequantizer 321 and an inverse transformer 322. According to embodiments, the entropy decoder 310, residual processor 320, predictor 330, adder 340, and filter 350 may be configured as a single hardware component (e.g., a decoder chipset or processor). The memory 360 may include a decoded picture buffer (DPB) and may be configured as a digital storage medium. The hardware component may also include the memory 360 as an internal / external component.

[0089] When a bitstream including video / image information is input, the decoding device 300 can, based on the... Figure 2 The process of processing video / image information in an encoding device reconstructs the image. For example, decoding device 300 can deduce units / blocks based on block partitioning information obtained from the bitstream. Decoding device 300 can perform decoding using processing units applied to the encoding device. Therefore, the processing unit used for decoding can be, for example, an encoding unit, and the encoding unit can be partitioned from encoding tree units or maximum encoding units according to a quadtree structure, binary tree structure, and / or ternary tree structure. One or more transform units can be derived from the encoding unit. The reconstructed image signal decoded and output by decoding device 300 can be reproduced via a reproduction device.

[0090] Decoding device 300 can receive signals output from encoding device in the form of a bitstream, and the received signals can be decoded by entropy decoder 310. For example, entropy decoder 310 can parse the bitstream to derive information (e.g., video / image information) necessary for image reconstruction (or picture reconstruction). Video / image information may also include information about various parameter sets, such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). Furthermore, video / image information may also include general constraint information. Decoding device can further decode the picture based on the information about the parameter sets and / or general constraint information. In this disclosure, the transmitted / received information and / or syntax elements, which will be described below, can be decoded through a decoding process and can be obtained from the bitstream. For example, entropy decoder 310 can decode information in the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, or CABAC, and can output the values ​​of the syntax elements required for image reconstruction and the quantized values ​​of the transform coefficients for the residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element in the bitstream. It can determine a context model using information about the target syntax element and decoding information about neighbors and the target block, or information about symbols / bins decoded in previous stages. It can generate symbols corresponding to the values ​​of each syntax element by predicting the occurrence probability of bins based on the determined context model and performing arithmetic decoding on the bins. Here, after determining the context model, the CABAC entropy decoding method can update the context model using information about the decoded symbols / bins for use in the context model of the next symbol / bin. Prediction-related information from the information decoded by the entropy decoder 310 can be provided to the predictor 330, and the residual values ​​(i.e., quantized transform coefficients and related parameter information) obtained through entropy decoding in the entropy decoder 310 can be input to the residual processor 320. The residual processor 320 can derive residual signals (residual blocks, residual samples, or arrays of residual samples). Filtering-related information from the information decoded by the entropy decoder 310 can be provided to the filter 350. A receiver (not shown) for receiving signals output from the encoding device may be further configured as an internal / external element of the decoding device 300, or the receiver may be a component of the entropy decoder 310. The decoding device according to this disclosure may be referred to as a video / image / picture decoding device and may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoder 310, and the sample decoder may include at least one of a dequantizer 321, an inverse transformer 322, an adder 340, a filter 350, a memory 360, and a predictor 330.

[0091] Dequantizer 321 can dequantize the quantized transform coefficients to output transform coefficients. Dequantizer 321 can rearrange the quantized transform coefficients into two-dimensional blocks. In this case, the rearrangement can be performed based on the coefficient scan order performed in the encoding device. Dequantizer 321 can perform dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain the transform coefficients.

[0092] The inverse transformer 322 performs an inverse transformation on the transformation coefficients to obtain the residual signal (residual block or residual sample array).

[0093] The predictor can perform prediction for the current block and generate a prediction block that includes prediction samples of the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on prediction-related information output from the entropy decoder 310, and determine the specific intra-frame / inter-frame prediction mode.

[0094] Predictor 330 can generate a prediction signal based on various prediction methods described below. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but also apply intra-frame prediction and inter-frame prediction simultaneously, which can be called combined inter-frame and intra-frame prediction (CIIP). Furthermore, the predictor can predict blocks based on an intra-block copy (IBC) prediction mode or a palette mode. The IBC prediction mode or palette mode can be used, for example, for screen content coding (SCC). IBC essentially performs prediction within the current frame, but can be performed similarly to inter-frame prediction in that it derives a reference block within the current frame based on a block vector. That is, IBC can utilize at least one of the inter-frame prediction techniques described in this disclosure.

[0095] Intra-frame predictors can refer to samples within the current image to predict the current block. Depending on the prediction mode, the referenced samples can be located as being near (adjacent) to the current block, or as being far from the current block. In intra-frame prediction, prediction modes can include multiple non-directional modes and multiple directional modes. The intra-frame predictor can determine the prediction mode applied to the current block based on the prediction modes applied to neighboring blocks.

[0096] An inter-frame predictor can deduce a predicted block for the current block based on reference blocks (reference sample arrays) specified by motion vectors on a reference image. Here, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between the motion information of neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, and bidirectional prediction) information. In inter-frame prediction, neighboring blocks may include spatially adjacent blocks existing within the current image and temporally adjacent blocks existing in the reference image. For example, inter-frame predictor 332 can configure a motion information candidate list based on neighboring blocks and can deduce the motion vector and / or reference image index of the current block based on received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and information about the prediction may include information indicating the inter-frame prediction mode for the current block.

[0097] Adder 340 can add the obtained residual signal to the prediction signal (prediction block or prediction sample array) output from predictor 330 to generate a reconstruction signal (reconstructed image, reconstruction block, or reconstruction sample array). When there is no residual for the processing target block, such as when a skip mode is applied, the prediction block can be used as a reconstruction block.

[0098] Adder 340 can be referred to as a reconstructor or reconstruction block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current image, or it can be output after being filtered as described below, or used for inter-frame prediction of the next image.

[0099] Luminance mapping with chroma scaling (LMCS) can be applied in the image decoding process.

[0100] Filter 350 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 350 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and the modified reconstructed image can be sent to memory 360, specifically to the DPB in memory 360. Various filtering methods can include, for example, unblocking filtering, sample adaptive shifting, adaptive loop filtering, and bilateral filtering.

[0101] The (modified) reconstructed image stored in the DPB of memory 360 can be used as a reference image in the inter-frame predictor. Memory 360 can store motion information about blocks in the current image from which motion information is derived (or decoded) and / or about blocks in the reconstructed image. The stored motion information can be sent to the inter-frame predictor to be used as motion information about spatially adjacent blocks or about temporally adjacent blocks. Memory 360 can store reconstructed samples of reconstructed blocks in the current image and can send the reconstructed samples to the intra-frame predictor.

[0102] The embodiments described in this specification for the filter 260 and predictor 220 of the encoding device 200 can also be applied equally or correspondingly to the filter 350 and predictor 330 of the decoding device 300.

[0103] As described above, during video encoding, prediction is performed to increase compression efficiency. Prediction generates prediction blocks that include predicted samples for the current block, which is the target block for encoding. Prediction blocks consist of predicted samples in the spatial domain (or pixel domain). Prediction blocks are derived identically in both the encoding and decoding devices, and the encoding device can enhance image encoding efficiency by signaling information about the residuals between the original block and the prediction blocks (residual information) to the decoding device, rather than the original sample values ​​of the original block. The decoding device can derive residual blocks including residual samples based on the residual information, generate reconstructed blocks including reconstructed samples by combining the residual blocks with the prediction blocks, and generate a reconstructed image including the reconstructed blocks.

[0104] Residual information can be generated through transformation and quantization processes. For example, an encoding device can derive a residual block between the original block and the prediction block, perform a transformation process on the residual samples (residual sample array) included in the residual block to derive transform coefficients, perform a quantization process on the transform coefficients to derive quantized transform coefficients, and signal the relevant residual information to a decoding device (via a bitstream). Residual information may include information such as the values ​​and locations of the quantized transform coefficients, the transform technique, the transform kernel, and the quantization parameters. The decoding device can perform dequantization / inverse transform processes based on the residual information to derive residual samples (or residual blocks). The decoding device can generate a reconstructed image based on the prediction block and the residual block. The encoding device can perform dequantization / inverse transform on the quantized transform coefficients to derive residual blocks used for reference in inter-frame prediction of subsequent images, and can generate a reconstructed image based on the residual blocks.

[0105] In this disclosure, at least one of quantization / dequantization and / or transformation / inverse transformation may be omitted. When quantization / dequantization is omitted, the quantized transformation coefficients may be referred to as transformation coefficients. When transformation / inverse transformation is omitted, the transformation coefficients may be referred to as coefficients or residual coefficients, or, for the sake of consistency, may still be referred to as transformation coefficients.

[0106] Furthermore, in this disclosure, quantized transform coefficients and transform coefficients can be referred to as transform coefficients and scaled transform coefficients, respectively. In this case, residual information may include information about one or more transform coefficients, and information about one or more transform coefficients may be signaled via residual coding syntax. Transform coefficients can be derived based on residual information (or information about one or more transform coefficients), and scaled transform coefficients can be derived by inverse transforming (scaling) the transform coefficients. Residual samples can be derived based on inverse transforming (scaling) the scaled transform coefficients. These details can be equally applied to other parts of this disclosure or described in other parts of this disclosure.

[0107] As described above, the predictor of an encoding / decoding device can derive predicted samples by performing inter-frame prediction on a block-by-block basis. Inter-frame prediction can be represented as a prediction derived in a manner that depends on data elements (i.e., sample values, motion information, etc.) of images other than the current image. When inter-frame prediction is applied to the current block, the predicted block (predicted sample array) of the current block can be derived based on the reference block (reference sample array) on the reference image indicated by the reference image index, specified by the motion vector. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, the motion information of the current block can be predicted on a block-by-block, sub-block, or sample-by-sample basis based on the correlation between the motion information of neighboring blocks and the current block. Motion information can include motion vectors and reference image indices. Motion information can also include inter-frame prediction type (L0 prediction, L1 prediction, bidirectional prediction, etc.) information. When applying inter-frame prediction, neighboring blocks can include spatially adjacent blocks existing in the previous image and temporally adjacent blocks existing in the reference image. The reference image including the reference block and the reference image including the temporally adjacent block can be the same or different. Temporally adjacent blocks can be referred to by names such as co-located reference blocks or co-located CUs (colCU), and the reference picture including the temporally adjacent blocks can be called a co-located picture (colPic). For example, a candidate list of motion information can be constructed based on the current block's neighboring blocks, and a flag or index information indicating which candidate to select (use) to derive the motion vector of the current block and / or the reference picture index can be signaled. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the motion information of the current block can be the same as the motion information of the selected neighboring blocks. In skip mode, unlike merge mode, residual signaling may not be sent. In motion vector prediction (MVP) mode, the motion vectors of the selected neighboring blocks can be used as motion vector predictors, and the motion vector difference can be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

[0108] Motion information can include L0 motion information and / or L1 motion information, depending on the inter-frame prediction type (L0 prediction, L1 prediction, bidirectional prediction, etc.). A motion vector in the L0 direction can be called an L0 motion vector or MVL0, and a motion vector in the L1 direction can be called an L1 motion vector or MVL1. Prediction based on the L0 motion vector can be called L0 prediction, prediction based on the L1 motion vector can be called L1 prediction, and prediction based on both L0 and L1 motion vectors can be called bidirectional prediction. Here, the L0 motion vector can represent the motion vector associated with a reference image list L0 (L0), and the L1 motion vector can represent the motion vector associated with a reference image list L1 (L1). The reference image list L0 can include images preceding the current image in terms of output order as reference images, and the reference image list L1 can include images following the current image in terms of output order. The preceding images can be called forward (reference) images, and the following images can be called backward (reference) images. The reference image list L0 can also include images following the current image in terms of output order as reference images. In this case, within the reference image list L0, previous images can be indexed first, followed by subsequent images. The reference image list L1 can also include images that precede the current image in terms of output order. In this case, within the reference image list L1, subsequent images can be indexed first, followed by previous images. Here, the output order can correspond to the Image Order Count (POC) order.

[0109] For inter-frame prediction, the predictor within the encoding device and the video / image encoding process based on inter-frame prediction can schematically perform operations such as the following.

[0110] Figure 4 An example of the inter-frame prediction process is illustrated.

[0111] Reference Figure 4 As described above, the inter-frame prediction process may include an inter-frame prediction mode / type determination step, a motion vector derivation / refinement step, and an inter-frame prediction execution (prediction sample generation) step. The inter-frame prediction process can be performed in the encoding and decoding devices described above. In this document, the encoding apparatus may include an encoding apparatus and / or a decoding apparatus.

[0112] The coding device determines the inter-frame prediction mode / type (S400).

[0113] The encoding device can determine the inter-frame prediction mode / type to be applied to the current block from the various inter-frame prediction modes / types disclosed in this document, and can generate prediction-related information. The prediction-related information may include inter-frame prediction mode information indicating the inter-frame prediction mode applied to the current block and / or inter-frame prediction type information indicating the inter-frame prediction type applied to the current block. The decoding device can determine the inter-frame prediction mode / type to be applied to the current block based on the prediction-related information.

[0114] The coding device derives / refines the motion vector of the current block (S410). The coding device can derive / refine the motion vector of the current block based on the determined inter-frame prediction mode / type. Here, motion information of neighboring blocks of the current block can be used to derive / refine the motion vector.

[0115] For example, when a skip mode or merge mode is applied to the current block, the encoding device can construct a merge candidate list and select one merge candidate from the merge candidates included in the list. Information indicating the selected merge candidate (e.g., merge index) can be included in the prediction-related information.

[0116] The decoding device can construct an (A)MVP candidate list, and can use the motion vector of the MVP candidate selected from the MVP (Motion Vector Predictor) candidates included in the (A)MVP candidate list as the MVP of the current block. The selection can be indicated based on selection information (MVP flag or MVP index). In this case, not only the selection information but also information about MVD can be included in the prediction-related information.

[0117] Furthermore, as described below, the motion information of the current block can be derived without constructing a candidate list, and in this case, the motion information of the current block can be derived according to the process disclosed in the prediction pattern / type described below. In this case, the candidate list construction as described above can be omitted.

[0118] The encoding device predicts the current block based on the derived / refined motion vectors (generating prediction samples) (S420). The encoding device can use samples of a reference block indicated by motion vectors on a reference image to derive prediction samples for the current block.

[0119] The encoding process based on inter-frame prediction can schematically include, for example, the following operations.

[0120] Figure 5 An example of a video / image coding method based on inter-frame prediction is shown.

[0121] Reference Figure 5S500 can be executed by the predictor of the encoding device, S505 can be executed by the residual processor of the encoding device, and S510 or S515 can be executed by the entropy encoder of the encoding device. Specifically, prediction-related information can be derived by the predictor and encoded by the entropy encoder. Residual information can be derived by the residual processor and encoded by the entropy encoder. Residual information is information about residual samples. Residual information can include information about the quantized transform coefficients for the residual samples. As mentioned above, residual samples can be derived into transform coefficients by the transformer of the encoding device, and transform coefficients can be derived into quantized transform coefficients by the quantizer. Information about the quantized transform coefficients can be encoded in the entropy encoder through the residual encoding process.

[0122] The encoding device performs inter-frame prediction on the current block (S500). The encoding device can deduce the inter-frame prediction mode / type and motion information of the current block, and can generate prediction samples for the current block. Here, the processes of determining the inter-frame prediction mode / type, deduce motion information, and generate prediction samples can be performed simultaneously, or any one of these processes can be performed before the others. For example, the encoder's predictor can search for blocks similar to the current block within a predetermined region (search region) of a reference image using motion estimation, and can deduce the reference block whose difference from the current block is the smallest or equal to or less than a predetermined criterion. Based on this, a reference image index indicating the reference image in which the reference block is located can be derived, and a motion vector can be derived based on the positional difference between the reference block and the current block. The encoding device can determine the mode applicable to the current block from various prediction modes. The encoding device can compare the RD costs for various prediction modes and determine the optimal prediction mode for the current block.

[0123] For example, when a skip mode or merge mode is applied to the current block, the encoding device can construct a merge candidate list (described later) and deduce a reference block from the reference images indicated by the merge candidates included in the merge candidate list that has the smallest difference from the current block, or is equal to or less than a predetermined criterion. In this case, the merge candidate associated with the deduced reference block is selected, and merge index information indicating the selected merge candidate can be generated and concurrently signaled to the decoding device. The motion information of the current block can be deduced using the motion information of the selected merge candidate.

[0124] As another example, when the (A)MVP mode is applied to the current block, the encoding device can construct an (A)MVP candidate list (described later), and the motion vector of the MVP candidate selected from the MVP (Motion Vector Predictor) candidates included in the (A)MVP candidate list can be used as the MVP of the current block. In this case, for example, the motion vector of the reference block derived through the motion estimation described above can be used as the motion vector of the current block, and the MVP candidate with the motion vector having the smallest difference from the motion vector of the current block can be the selected MVP candidate. The MVD (Motion Vector Difference) can be derived, which is the difference obtained by subtracting the MVP from the motion vector of the current block. In this case, information about the MVD can be signaled to the decoding device. Furthermore, when the (A)MVP mode is applied, the value of the reference image index can be configured as reference image index information and can be signaled to the decoding device separately.

[0125] The encoding device can perform residual processing based on the predicted samples (S505). The encoding device can derive residual samples based on the predicted samples. The encoding device can derive residual samples by comparing the original samples of the current block with the predicted samples. Residual information can be generated based on the residual samples. The residual information may include information about the quantized transform coefficients as described above.

[0126] The encoding device encodes video information, including prediction-related information and / or residual information (S510 or S515). The encoding device can output the encoded video information as a bitstream. Prediction-related information is information related to the prediction process and may include prediction mode information (e.g., skip flag, merge flag, or mode index, etc.) and information about motion information. Information about motion information may include candidate selection information (e.g., merge index, MVP flag, or MVP index), which is information used to derive motion vectors. Furthermore, information about motion information may include information about the aforementioned MVD and / or reference image index information. Additionally, information about motion information may include information indicating whether L0 prediction, L1 prediction, or bidirectional prediction is applied. Residual information is information about residual samples. Residual information may include information about the quantized transform coefficients for the residual samples.

[0127] The output bitstream can be stored in a (digital) storage medium and sent to a decoding device, or it can be sent to a decoding device via a network.

[0128] Furthermore, as mentioned above, the encoding device can generate a reconstructed image (including reconstructed samples and reconstructed blocks) based on reference samples and residual samples. This is to derive the same prediction result in the encoding device as the prediction result performed in the decoding device, thereby improving encoding efficiency. Therefore, the encoding device can store the reconstructed image (or reconstructed samples, reconstructed blocks) in memory and use it as a reference image for inter-frame prediction. As mentioned above, a loop filtering process or a similar process can be further applied to the reconstructed image.

[0129] Decoding devices can perform operations corresponding to those performed in encoding devices. Video / image decoding processes based on inter-frame prediction may include, for example, the following operations.

[0130] Figure 6 An example of a video / image decoding method based on inter-frame prediction is shown.

[0131] Reference Figure 6 S600 can be executed by the entropy decoder of the decoding device, S610 can be executed by the predictor of the decoding device, S615 can be executed by the residual processor of the decoding device, and S620 can be executed by the adder or reconstructor of the decoding device.

[0132] Specifically, the decoding device obtains image / video information from the bitstream (S600). The image / video information may include prediction-related information and / or residual information.

[0133] The decoding device performs inter-frame prediction based on prediction-related information (S610). The decoding device can deduce the inter-frame prediction mode / type for the current block, deduce / refine the motion information of the current block, and generate prediction samples within the current block based on the intra-frame prediction mode / type and / or motion information. In this case, the decoding device can perform a prediction sample filtering process. Prediction sample filtering can be called post-filtering. Some or all of the prediction samples can be filtered through the prediction sample filtering process. Depending on the situation, the prediction sample filtering process can be omitted.

[0134] The decoding device performs residual processing based on the residual information (S615). The decoding device can derive residual samples for the current block based on the residual information. Specifically, the dequantizer of the residual processor can perform dequantization based on the quantized transform coefficients derived from the residual information to derive the transform coefficients, and the inverse transformer of the residual processor can perform an inverse transform on the transform coefficients to derive residual samples for the current block.

[0135] The decoding device generates a reconstructed block / image (S620). The decoding device can generate reconstructed samples for the current block based on predicted samples and / or residual samples, and can derive a reconstructed block including the reconstructed samples. A reconstructed image of the current image can be generated based on the reconstructed block. As described above, a loop filtering process or a similar process can be further applied to the reconstructed image.

[0136] Prediction-related information can be encoded / decoded using the binarization and encoding methods described in this disclosure. For example, prediction-related information can be binarized using fixed-length binarization, truncated Rice binarization, truncated unary binarization, etc. For example, prediction-related information can be encoded / decoded using entropy coding (e.g., CABAC, CAVLC).

[0137] Furthermore, as described above, when the (A)MVP mode is applied to the current block, the decoding device can construct an (A)MVP candidate list, deduce the motion vector of the MVP candidate selected from the MVP (motion vector predictor) candidates included in the (A)MVP candidate list as the MVP of the current block, deduce the MVD of the current block based on the signaled notification of MVD, and deduce the motion information of the current block based on the MVP and MVD.

[0138] Furthermore, according to this disclosure, MVD prediction can be applied. Based on MVD prediction, the syntax for the MVD prediction index indicating the MVD and the syntax for the MVD magnitude can be parsed. MVD candidates can be derived by combining the MVD magnitude derived based on the syntax and possible signs. Template matching (TM) cost can be used to reorder the MVs combined with the derived MVD candidates, and the candidates indicated by the MVD prediction index among the reordered candidates can be derived as the MVD of the current block.

[0139] Specifically, based on MVD prediction, the decoding device can derive MVD as follows.

[0140] For example, 1) the decoding device can parse the amplitude of the MVD component. That is, it can signal the MVD amplitude syntax indicating the size of the MVD component. Then, 2) the decoding device can parse the context-coded MVD prediction index. The MVD prediction index can point to one of the MVD candidates. 3) the decoding device can generate MVD candidates (which are combinations of available amplitudes and available symbols derived based on the MVD amplitude syntax) and can construct MV candidates by adding them to the MV predictor (MVP) of the current block. 4) the decoding device can derive the MVD prediction cost for each MV candidate and can sort the MV candidates based on their MVD prediction costs. Here, the MVD prediction cost of an MV candidate can be the template matching (TM) cost of the MV candidate. 5) the decoding device can select the true MVD from the sorted MV candidates based on the signaled MVD prediction index.

[0141] Furthermore, for example, bilateral filters can be used for reference template generation, which is used to derive the MVD prediction cost of MV candidates.

[0142] Figure 7 A reference template for deriving the TM cost, which is the MVD prediction cost of the MV candidate, is illustrated.

[0143] like Figure 7 As illustrated, the TM cost of an MV candidate can be derived as the sum of absolute differences (SAD) between the current template, which includes the neighboring samples of the current block, and the reference template, which includes the neighboring reference samples of the reference block indicated by the graph candidate. For example, the TM cost of an MV candidate can be derived based on the following formula.

[0144] [Formula 1]

[0145] Here, i and j represent the positions (i, j) of the samples within the template, and Cost distortion Indicates cost, Temp ref This represents the sample value of the reference template of the reference block indicated by the MV candidate, and Temp cur This represents the sample value of the current template in the current block. The difference between corresponding samples between the reference template and the current template can be accumulated, and the accumulated difference can be used as a cost function to rank the MV candidates of the current block.

[0146] Furthermore, for example, MVD prediction can be applied not only to AMVP mode, but also to affine AMVP mode, MMVD mode, and affine MMVD mode. Additionally, when surround motion compensation is available, surround offset can be considered to clip MV candidates.

[0147] For example, when MVD prediction is applied to affine AMVP or affine MMVD patterns, sub-block-based templates can be used. For instance, the final cost of the MV candidates for the current block can be derived by accumulating the template matching cost for each sub-block of the current block.

[0148] Figure 8 A reference template for deriving the TM cost is shown, which is the MVD prediction cost of MV candidates in the affine AMVP pattern or the affine MMVD pattern.

[0149] like Figure 8 As illustrated, in the affine AMVP mode or the affine MMVD mode, the TM cost for each sub-block of the current block can be derived, and the final cost of the MV candidate for the current block can be derived by accumulating the TM cost for each sub-block.

[0150] Furthermore, for example, when encoding the MVD magnitude syntax for MVD magnitudes, the first six valid suffixes of the MVD magnitude syntax (bin) can be context-encoded. Valid suffixes (bin) can include symbol bins.

[0151] Furthermore, for example, the number of valid suffix bins in the context-encoded MVD magnitude syntax can be derived based on the block size. For example, for blocks with a width and height greater than N, up to 6 valid suffix bins, including the symbol bin, can be context-encoded, and for blocks with a width or height equal to or less than N, up to 2 valid suffix bins can be encoded. For example, N can be 4. Alternatively, for example, for blocks with a width and height greater than N, up to 6 valid suffix bins, including the symbol bin, can be context-encoded, and for blocks with a width or height equal to or less than N, up to 4 valid suffix bins can be encoded. For example, N can be 4.

[0152] Furthermore, for example, the number of combinations of signs and amplitudes in MMVD can be 16.

[0153] Figure 9 An example of an MMVD candidate is shown, which is a combination of available signs and magnitudes.

[0154] like Figure 9As shown, 16 MMVD candidates can be predefined. Furthermore, for example, when MVD prediction is applied to the 16 MMVDs, the TM cost for each MMVD candidate can be derived, and by sorting the MMVD candidates according to the order of their derived TM costs, only the first 8 candidates can be derived as MMVD candidates for the current block. The TM cost for each MMVD candidate can be derived as the sum of absolute differences (SAD) between the template of the current block and the reference template of the reference block indicated by the MV candidates derived from the MMVD candidates and the current block's MVP.

[0155] Furthermore, for example, according to this disclosure, GPM (Geometric Partitioning Mode) can be applied as an implementation of inter-frame prediction. GPM can be considered as one of the prediction schemes or prediction types. When GPM is applied to the current block, the current block can be partitioned into two partitions, motion information for each of the two partitions can be derived, and prediction samples for the current block can be derived by performing inter-frame prediction on each partition based on the motion information for each of the two partitions.

[0156] Figure 10 An example of a partition shape supported by GPM is illustrated.

[0157] Reference Figure 10 GPM can support 64 partition shapes from combinations of 20 angles and 4 distances. For example, the 64 partition shapes can include 32 partition shapes as combinations of 8 angles and 4 distances, 24 partition shapes as combinations of 8 angles and 3 distances, and 8 partition shapes as combinations of 4 angles and 2 distances.

[0158] For example, a GPM partition index indicating the partition shape of the current block can be signaled. For instance, the partition shape of the current block can be derived from the partition shape indicated by the GPM partition index. The following table can represent the partition shapes indicated by the GPM partition index.

[0159] [Table 1]

[0160] Here, gpm_partition_idx can represent the GPM partition index, angleIdx can represent the angle index, and distanceIdx can represent the distance index. The partition shape of the current block can be derived based on the GPM partition index signaled, and the current block can be partitioned into multiple partitions based on the derived partition shape.

[0161] Furthermore, for example, GPM can also be applied to non-square blocks. For example, Figure 10(b) can illustrate the partition shape of a GPM applied to a non-square block with a W / H ratio of 2. Here, W can represent the width of the block, and H can represent the height of the block. For example, when the current block is a non-square block with a W / H ratio of 2, the GPM partition index that signals the notification can represent the partition shape shown in the table below.

[0162] [Table 2]

[0163] For example, when the current block is a non-square block with a W / H ratio of 2, the partition shape of the current block can be derived based on Table 2 and the GPM partition index instead of Table 1.

[0164] In addition, for example, Figure 10 (c) can represent the partition shape of a GPM applied to a non-square block with a W / H of 4. For example, when the current block is a non-square block with a W / H of 2, the GPM partition index that signals the notification can represent the partition shape shown in the table below.

[0165] [Table 3]

[0166] For example, when the current block is a non-square block with a W / H of 4, the partition shape of the current block can be derived based on Table 3 and the GPM partition index.

[0167] For example, the partition shape of a table can be derived based on the size of the current block, and the partition shape of the current block can be derived based on the derived table and the GPM partition index. Alternatively, for example, the partition shape of a table can be derived based on the width and / or height of the current block, and the partition shape of the current block can be derived based on the derived table and the GPM partition index. Or, for example, the partition shape of a table can be derived based on the ratio of the width to the height of the current block, and the partition shape of the current block can be derived based on the derived table and the GPM partition index.

[0168] Specifically, for example, when the W / H of the current block is 1, Table 1 can be derived as a table with the partition shape of the current block, and the partition shape of the current block can be derived based on Table 1 and the GPM partition index of the current block. Alternatively, for example, when the W / H of the current block is 2, Table 2 can be derived as a table with the partition shape of the current block, and the partition shape of the current block can be derived based on Table 2 and the GPM partition index of the current block. Alternatively, for example, when the W / H of the current block is 4, Table 3 can be derived as a table with the partition shape of the current block, and the partition shape of the current block can be derived based on Table 3 and the GPM partition index of the current block.

[0169] Furthermore, for example, there may be constraints on the size of the block to which GPM is applied. That is, for example, whether to apply GPM can be determined based on the current block size. For example, the minimum width or minimum height of the block to which GPM is applied can be 8 or 4. That is, when the width or height of the current block is less than 8 or 4, GPM may not be applied. Furthermore, for example, the maximum width or maximum height of the block to which GPM is applied can be 64 or 128. That is, when the width or height of the current block is greater than 64 or 128, GPM may not be applied. Furthermore, for example, the maximum width-to-height ratio of the block to which GPM is applied can be 1:4 or 4:1. That is, when the maximum width-to-height ratio of the current block is greater than 1:4 or 4:1, GPM may not be applied. Alternatively, for example, the maximum width-to-height ratio of the block to which GPM is applied can be 1:4 or 4:1, or 1:8 or 8:1. That is, when the maximum width-to-height ratio of the current block is greater than 1:8 or 8:1, GPM may not be applied.

[0170] Furthermore, for example, according to this disclosure, a GPM with inter-frame and intra-frame prediction can be applied. A GPM with inter-frame and intra-frame prediction can be considered one of the prediction schemes or prediction types. A GPM with inter-frame and intra-frame prediction can be called an inter-frame-intra-prediction GPM. When a GPM with inter-frame and intra-frame prediction is applied to the current block, the current block can be partitioned into two partitions, and inter-frame prediction can be applied to one of the two partitions, while intra-frame prediction can be applied to the other partition.

[0171] For example, in GPM that includes inter-frame and intra-frame prediction, the final prediction sample can be generated by applying weights to the inter-frame prediction sample and the intra-frame prediction sample for each individual partition. That is, each weight in the inter-frame prediction sample (i.e., the prediction sample of the partition to which inter-frame prediction is applied) and the intra-frame prediction sample (i.e., the prediction sample of the partition to which intra-frame prediction is applied) can be derived, and the final prediction sample can be generated based on the weighted sum of the inter-frame prediction sample and the intra-frame prediction sample. The inter-frame prediction sample can be derived based on the inter-frame GPM, and the intra-frame prediction sample can be derived based on the IPM (Intra-Frame Prediction Mode) candidate list and the index signaled from the coding device. The inter-frame prediction mode applied to one partition of the current block can be derived from the inter-frame GPM, and the intra-frame prediction mode applied to another partition of the current block can be derived from the IPM candidate indicated by the signaled index in the configured IPM candidate list. For example, the size of the IPM candidate list can be predefined as 3.

[0172] Figure 11 Intra-prediction modes that can be used as IPM candidates are illustrated by example.

[0173] Reference Figure 11The intra-prediction modes that can be used as IPM candidates (i.e., available IPM candidates) can be parallel angle modes (parallel modes) about GPM block boundaries, vertical angle modes (vertical modes) about GPM block boundaries, and / or planar modes. Figure 11 (a) can be illustrated in parallel mode. Figure 11 (b) can be illustrated in a vertical pattern, and Figure 11 (c) can be used to illustrate a planar pattern.

[0174] Furthermore, for example, during the construction of the IPM candidate list, intra-prediction modes derived from neighboring blocks and / or decoder-side intra-prediction mode (DIMD) methods can be derived as IPM candidates. When intra-prediction modes and / or DIMD derived from neighboring blocks are derived as IPM candidates, the aforementioned parallel modes can preferably be derived as IPM candidates. For example, when the size of the IPM candidate list is 3, after parallel modes are derived as IPM candidates, intra-prediction modes and / or DIMD derived from neighboring blocks can be derived as IPM candidates; correspondingly, when there are equal IPM candidates within an IPM candidate list, up to two IPM candidates derived from DIMD and / or neighboring blocks can be derived. When deriving neighboring intra-prediction modes (i.e., intra-prediction modes derived from neighboring blocks), the number of available positions in neighboring blocks can be up to 5, but this can be limited by the GPM block boundary angle, as shown in the table below.

[0175] [Table 4]

[0176] Here, the GPM angle can refer to the GPM block boundary angle indicated by the index, the first partition can refer to the location of available neighboring blocks in the first partition, and the second partition can represent the location of available neighboring blocks in the second partition. For example, when the current block is partitioned by the GPM block boundary angle of index 0, the available neighboring blocks of the first partition may include the upper neighboring block, and the available neighboring blocks of the second partition may include the left neighboring block and the upper neighboring block. Alternatively, for example, when the current block is partitioned by the GPM block boundary angle of index 5, the available neighboring blocks of the first partition may include the left neighboring block and the upper neighboring block, and the available neighboring blocks of the second partition may include the left neighboring block.

[0177] For example, in GPM that includes inter-frame prediction and intra-frame prediction, the motion information of the partition to which inter-frame prediction is applied can be derived based on the regular GPM MV candidate list. For example, the regular GPM MV candidate list can be a merge candidate list derived based on the neighboring blocks of the partition to which inter-frame prediction is applied. For example, the merge candidate list can be derived based on the neighboring blocks of the partition to which inter-frame prediction is applied, and the motion information of the partition to which inter-frame prediction is applied can be derived based on the merge candidate pointed to by the merge index of the merge candidate in the merge candidate list for the partition to which inter-frame prediction is applied.

[0178] Furthermore, for example, GPM can be combined with MMVD (Merge with Motion Vector Difference). That is, in GPM which includes inter-frame prediction and intra-frame prediction, GPM-MMVD can be applied to partitions where inter-frame prediction is applied.

[0179] For example, MMVD information for partitions using inter-frame prediction can be signaled, the MMVD of the partitions using inter-frame prediction can be derived from the MMVD information, and the motion information of the partitions using inter-frame prediction can be derived from the merge candidates and MMVD of the partitions using inter-frame prediction. The MMVD information may include an MMVD distance index and an MMVD direction index. The MMVD distance index indicates the distance to the MMVD, and the MMVD direction index indicates the direction of the MMVD.

[0180] Furthermore, regression-based GPM can be applied to GPM that includes inter-frame and intra-frame prediction. For example, a pairing list can be constructed for partitions of the current block to which GPM including inter-frame and intra-frame prediction is applied, and pairing indices pointing to selected pairing candidates can be signaled. Pairing candidates can include intra-prediction mode candidates for partitions of the current block to which intra-frame prediction is applied, and MV candidates for partitions of the current block to which inter-frame prediction is applied.

[0181] For example, a flag indicating whether regression-based GPM, including inter-frame prediction and intra-frame prediction, is applied can be signaled. This flag can be signaled at the CU level. Here, regression-based GPM including inter-frame prediction and intra-frame prediction can be called regression-based inter-intra-frame prediction GPM, and the flag can be called the regression-based inter-intra-frame prediction GPM flag. Furthermore, for example, a flag indicating whether regression-based GPM including inter-frame prediction and intra-frame prediction is available can be signaled in a high-level syntax (e.g., SPS, PPS). When the value of the available flag is 0, the signaling of the flag indicating whether regression-based GPM including inter-frame prediction and intra-frame prediction is applied can be omitted. The available flag can be called the regression-based inter-intra-frame prediction GPM available flag.

[0182] When a regression-based GPM with inter-frame and intra-frame predictions is applied to the current block, a pairing list of partitions for the current block can be constructed. For example, an intra-frame prediction mode candidate could be one of the top six intra-frame prediction modes in the MPM list, and an MV candidate could be one of the regular GPM MV candidates. The pairing of intra-frame prediction mode candidates and MV candidates can be derived based on two integer mixing matrices derived from the regression model of the template for the current block. Here, the integer mixing matrices can be as follows.

[0183] [Equation 2]

[0184] Here, parameters a, b, and c can be derived as parameters that minimize the mean squared error (MSE) of the template for the current block.

[0185] A pairing index pointing to one of the pairing candidates in the pairing list can be signaled, and the pairing candidates for the current block can be derived based on the pairing index. Motion information for the partitions for which inter-frame predictions of the current block are applied can be derived based on the MV candidates of the derived pairing candidates.

[0186] Furthermore, to further improve coding performance, in GPM that includes inter-frame prediction and intra-frame prediction, TIMD can be used as an IPM candidate for the partitions applying intra-frame prediction. For example, an IPM candidate list for partitions applying intra-frame prediction can be constructed such that parallel modes can preferably be derived as IPM candidates, and subsequently, TIMD, DIMD, and neighboring blocks can be derived as IPM candidates in sequence. That is, for example, when constructing the IPM candidate list, IPM candidates can be derived in the order of parallel mode, TIMD, DIMD, and intra-frame prediction modes derived from neighboring blocks.

[0187] Alternatively, for example, in a GPM that includes inter-frame prediction and intra-frame prediction, the IPM candidate for the partition to which intra-frame prediction is applied may include at least one of vertical intra-frame prediction mode, horizontal intra-frame prediction mode, planar mode, DC mode, or directional planar mode.

[0188] Furthermore, for example, signaling for GPM including inter-frame prediction and intra-frame prediction can be performed as follows.

[0189] [Table 5]

[0190] Referring to Table 5, signals can be sent to notify the GPM partition index, inter-frame-intra-frame prediction GPM flag, and / or inter-frame-intra-frame prediction GPM mode flag for the current block.

[0191] For example, `gpm_partition_idx` can indicate the GPM partition index. For example, the GPM partition index can indicate the partition shape of the current block to which GPM is applied. For example, the block boundary angle and the distance between the center point of the current block and the block boundary can be derived based on `gpm_partition_idx`.

[0192] For example, `gpm_partition_interintra_flag` can indicate whether inter-frame intra-frame prediction (GPM) is applied. Specifically, when `gpm_partition_interintra_flag` is 1, it indicates that inter-frame intra-frame prediction GPM is applied to the current block; and when it is 0, it indicates that inter-frame intra-frame prediction GPM is not applied to the current block. Here, `gpm_partition_interintra_flag` can be referred to as the inter-frame intra-frame prediction GPM flag.

[0193] Furthermore, for example, if `gpm_partition_interintra_flag` indicates that inter-frame / intra-frame prediction GPM is applied to the current block, `gpm_prediction_interintra_mode_flag` can be signaled. `gpm_prediction_interintra_mode_flag` can indicate the partitions for which inter-frame prediction is applied and the partitions for which intra-frame prediction is applied. Here, the partitions for which inter-frame prediction is applied can be called inter-frame partitions, and the partitions for which intra-frame prediction is applied can be called intra-frame partitions. Furthermore, `gpm_prediction_interintra_mode_flag` can be called the inter-frame / intra-frame prediction GPM mode flag.

[0194] For example, when the value of `gpm_prediction_interintra_mode_flag` is 1, it indicates that inter-frame prediction is applied to the first partition of the current block and intra-frame prediction is applied to the second partition of the current block. Similarly, when the value of `gpm_prediction_interintra_mode_flag` is 0, it indicates that intra-frame prediction is applied to the first partition of the current block and inter-frame prediction is applied to the second partition of the current block. In other words, when the value of `gpm_prediction_interintra_mode_flag` is 1, it indicates that the first partition of the current block is an inter-frame partition and the second partition of the current block is an intra-frame partition; and when the value of `gpm_prediction_interintra_mode_flag` is 0, it indicates that the first partition of the current block is an intra-frame partition and the second partition of the current block is an inter-frame partition. Alternatively, for example, when the value of gpm_prediction_interintra_mode_flag is 0, gpm_prediction_interintra_mode_flag can indicate that inter-frame prediction is applied to the first partition of the current block and intra-frame prediction is applied to the second partition of the current block, and when the value of gpm_prediction_interintra_mode_flag is 1, gpm_prediction_interintra_mode_flag can indicate that intra-frame prediction is applied to the first partition of the current block and inter-frame prediction is applied to the second partition of the current block. That is, when the value of gpm_prediction_interintra_mode_flag is 0, gpm_prediction_interintra_mode_flag can indicate that the first partition of the current block is an inter-frame partition and the second partition of the current block is an intra-frame partition, and when the value of gpm_prediction_interintra_mode_flag is 1, gpm_prediction_interintra_mode_flag can indicate that the first partition of the current block is an intra-frame partition and the second partition of the current block is an inter-frame partition. Here, gpm_prediction_interintra_mode_flag can be called the inter-frame-intra prediction GPM mode flag.

[0195] Furthermore, for example, for a partition of the current block, an index indicating a motion information candidate for inter-frame prediction and / or an index indicating an IPM candidate can be signaled. For example, when the first partition of the current block is an inter-frame partition and the second partition of the current block is an intra-frame partition, an index indicating a motion information candidate for the first partition, gpm_inter_idx0, can be signaled, and an index indicating an IPM candidate for the second partition, gpm_intra_idx1, can be signaled. Similarly, when the first partition of the current block is an intra-frame partition and the second partition of the current block is an inter-frame partition, an index indicating an IPM candidate for the first partition, gpm_intra_idx0, can be signaled, and an index indicating a motion information candidate for the second partition, gpm_inter_idx1, can be signaled.

[0196] Alternatively, for example, signaling for GPM with inter-frame and intra-frame prediction can be performed as follows.

[0197] [Table 6]

[0198] Referring to Table 6, the GPM partition index and inter-frame / intra-frame prediction GPM indicator for the current block can be signaled.

[0199] For example, `gpm_partition_interintra_idc` can indicate whether the first partition of the current block is an inter-frame partition and the second partition of the current block is an intra-frame partition, the first partition of the current block is an intra-frame partition and the second partition of the current block is an inter-frame partition, or whether a GPM with both inter-frame and intra-frame predictions is not applied to the current block. `gpm_partition_interintra_idc` can be binarized based on a truncated Rice (or truncated unary) scheme. Here, `gpm_partition_interintra_idc` can indicate an inter-frame-intra-frame prediction GPM indicator.

[0200] For example, the inter-frame-intra prediction GPM mode and binarization indicated by gpm_partition_interintra_idc can be as follows.

[0201] [Table 7]

[0202] Referring to Table 7, when the value of gpm_partition_interintra_idc is 0, gpm_partition_interintra_idc indicates that the inter-frame-intra prediction GPM is not applied; when the value of gpm_partition_interintra_idc is 1, gpm_partition_interintra_idc indicates that the GPM for the first partition of the current block being an inter-frame partition and the second partition of the current block being an intra-frame partition is applied; and when the value of gpm_partition_interintra_idc is 2, gpm_partition_interintra_idc indicates that the GPM for the first partition of the current block being an intra-frame partition and the second partition of the current block being an inter-frame partition is applied.

[0203] Furthermore, for example, for a partition of the current block, an index indicating a motion information candidate for inter-frame prediction and / or an index indicating an IPM candidate can be signaled. For example, when the first partition of the current block is an inter-frame partition and the second partition of the current block is an intra-frame partition, an index indicating a motion information candidate for the first partition, gpm_inter_idx0, can be signaled, and an index indicating an IPM candidate for the second partition, gpm_intra_idx1, can be signaled. Similarly, when the first partition of the current block is an intra-frame partition and the second partition of the current block is an inter-frame partition, an index indicating an IPM candidate for the first partition, gpm_intra_idx0, can be signaled, and an index indicating a motion information candidate for the second partition, gpm_inter_idx1, can be signaled.

[0204] Furthermore, for example, partitions for inter-frame prediction and intra-frame prediction can be determined based on the partition type (i.e., partition shape) of the current block. That is, for example, partitions for inter-frame prediction and intra-frame prediction can be determined based on the partition shape of the current block. For example, when one of the partitions of the current block is partitioned as a partition not adjacent to neighboring samples, the partition not adjacent to neighboring samples can be determined as a partition for inter-frame prediction, and the other partition can be determined as a partition for intra-frame prediction. In this case, the syntax indicating the partitions for inter-frame prediction and intra-frame prediction (e.g., the gpm_prediction_interintra_mode_flag mentioned above) can be omitted. Neighboring samples can include upper neighboring samples and left neighboring samples. Partitions not adjacent to neighboring samples have low dependence on neighboring blocks and are therefore highly likely not to be intra-frame predicted. Therefore, determining the partitions for inter-frame prediction and intra-frame prediction based on the partition shape can improve coding efficiency and reduce the number of bits used for inter-frame-intra-frame prediction GPM.

[0205] Furthermore, this document proposes an affine motion model that effectively derives motion vectors for sub-blocks or sample points of the current block and improves the accuracy of inter-frame prediction despite variations such as image rotation, scaling, or reduction. In other words, an affine motion model can be proposed for deriving motion vectors for sub-blocks or sample points of the current block. Predictions using the affine motion model can be called affine inter-frame prediction or affine motion prediction.

[0206] Encoding / decoding devices can predict the deformable shape of an image by affine inter-frame prediction based on the motion vector at the control point (CP) of the current block, thereby improving image compression performance by increasing prediction accuracy. Furthermore, motion vectors for at least one control point of the current block can be derived using motion vectors from neighboring blocks, reducing the data burden of additional side information and significantly improving inter-frame prediction efficiency.

[0207] For example, for inter-frame prediction using the affine motion model described above (i.e., affine motion prediction), there can be an affine merging mode (AF_MERGE) and an affine inter-frame mode (AF_INTER). Here, the affine inter-frame mode can also be called the affine motion vector prediction mode (AF_MVP).

[0208] Affine merging mode is similar to conventional merging mode in that it does not send the MVD (Motion Vector Difference) for the control points. That is, similar to conventional skip / merge mode, affine merging mode can instruct an encoding / decoding method for performing prediction by deriving the CPMV for each of two or three control points from neighboring blocks of the current block without encoding the MVD. Here, when the top-left sample position within the current block is specified as (0,0), sample positions (0,0), (w, 0), and (0, h) can be identified as control points. In the following text, the control point at sample position (0,0) can be referred to as CP0, the control point at sample position (w, 0) as CP1, and the control point at sample position (0, h) as CP2.

[0209] For example, when the AF_MRG mode is applied to the current block, the MVs (i.e., CPMV0, CPMV1 or CPMV0, CPMV1, CPMV2) of CP0 and CP1 (or CP0, CP1, and CP2) of the adjacent blocks in which the affine mode is applied can be derived from the adjacent blocks in which the affine mode is applied. In other words, the CPMV0 and CPMV1 (or CPMV0, CPMV1, and CPMV2) of the adjacent blocks in which the affine mode is applied can be derived as merge candidates, and the merge candidates can be derived as the CPMV0 and CPMV1 (or CPMV0, CPMV1, and CPMV2) of the current block.

[0210] Here, when the affine merging mode is applied to the current block, the encoding / decoding device can construct an affine merging candidate list based on the adjacent blocks of the current block.

[0211] In addition, each affine merging candidate can refer to the combination of CP0 and CP1's CPMV in a four-parameter affine motion model, and can refer to the combination of CP0, CP1 and CP2's CPMV in a six-parameter affine motion model.

[0212] Affine inter-frame mode can instruct the derivation of inter-frame prediction for the MVP (Motion Vector Predictor) of control points, derive the motion vector of the control point based on the received MVD (Motion Vector Difference) and MVP, derive the affine MVF of the current block based on the motion vector of the control point, and perform prediction based on the affine MVF. Here, the motion vector of the control point can be called CPMV (Control Point Motion Vector), the MVP of the control point can be called CPMVP (Control Point Motion Vector Predictor), and the MVD of the control point can be called CPMVD (Control Point Motion Vector Difference). Specifically, for example, the encoding device can derive CPMVP (Control Point Motion Vector Predictor) and CPMV (Control Point Motion Vector) for each of CP0 and CP1 (or CP0, CP1, and CP2), and can send or store information about CPMVP and / or CPMVD (i.e., the difference between CPMVP and CPMV).

[0213] Here, when the affine inter-frame mode is applied to the current block, the encoding / decoding device can construct an affine MVP candidate list based on the adjacent blocks of the current block. The affine MVP candidates can be called CPMVP pairing candidates, and the affine MVP candidate list can also be called the CPMVP candidate list.

[0214] In addition, each affine MVP candidate can refer to the combination of CP0 and CP1 CPMVPs in a four-parameter affine motion model, and can refer to the combination of CP0, CP1 and CP2 CPMVPs in a six-parameter affine motion model.

[0215] Figure 12 A flowchart of an affine motion prediction method according to an embodiment of the present disclosure is illustrated by way of example.

[0216] Reference Figure 12 The affine motion prediction method can be roughly represented as follows. When the affine motion prediction method begins, CPMV (S1200) can be obtained first. Here, when using a four-parameter affine model, CPMV can include CPMV0 and CPMV1, and when using a six-parameter affine model, CPMV can include CPMV0, CPMV1, and CPMV2.

[0217] After that, affine motion compensation can be performed based on CPMV (S1210), and affine motion prediction can be terminated.

[0218] Furthermore, for example, two affine prediction modes can exist to determine the CPMV. These two affine prediction modes can include an inter-affine prediction mode and an affine merging mode. The inter-affine prediction mode can explicitly determine the CPMV by signaling the motion vector difference (MVD) information for the CPMV. On the other hand, the affine merging mode can derive the CPMV without signaling the MVD information.

[0219] In other words, the affine merging pattern can derive the CPMV of the current block by using the CPMV of the neighboring blocks encoded in the affine pattern, and the affine merging pattern can also be called the sub-block merging pattern when the motion vector is determined on a sub-block basis.

[0220] In affine merging mode, the encoding device can signal the decoding device with the index of adjacent blocks encoded in affine mode for deriving the CPMV of the current block, and can also signal the difference between the CPMV of adjacent blocks and the CPMV of the current block. Here, affine merging mode can construct an affine merging candidate list based on adjacent blocks, and the index of the adjacent block can indicate the adjacent block to be referenced for deriving the CPMV of the current block from the affine merging candidate list. The affine merging candidate list can also be called a sub-block merging candidate list.

[0221] Affine inter-frame mode can also be called affine MVP mode. In affine MVP mode, the CPMV of the current block can be derived based on the CPMVP (Control Point Motion Vector Predictor) and CPMVD (Control Point Motion Vector Difference). In other words, the encoding device can determine the CPMVP for the CPMV of the current block, derive the CPMVD as the difference between the CPMV and CPMVP of the current block, and signal the decoding device with information about the CPMVP and CPMVD. Here, affine MVP mode can construct an affine MVP candidate list based on neighboring blocks, and the information about the CPMVP can indicate which neighboring blocks to refer to for deriving the CPMVP for the current block from the affine MVP candidate list. The affine MVP candidate list can also be called the Control Point Motion Vector Predictor candidate list.

[0222] For example, when the affine merge mode is applied to the current block, the current block can be encoded as described below.

[0223] The encoding / decoding device can construct an affine merge candidate list including affine merge candidates for the current block, and can derive the CPMV (control point motion vector) for the current block based on one of the affine merge candidates in the affine merge candidate list. The encoding / decoding device can derive a prediction sample for the current block based on the CPMV, and can generate a reconstructed image for the current block based on the derived prediction sample.

[0224] Specifically, the affine merging candidate list can be constructed as follows.

[0225] Figure 13 An example of constructing an affine merge candidate list for the current block is shown.

[0226] Reference Figure 13 The encoding device can add sub-block-based temporal merging candidates to the affine merging candidate list (S1300). Specifically, the encoding / decoding device can derive sub-block-based temporal merging candidates based on the co-occurring sub-blocks of the co-occurring block within the reference image. For example, sub-block-based temporal merging candidates can include sub-block unit motion information derived from the motion information of the co-occurring sub-blocks. Sub-block-based temporal merging candidates can also be called SbTMVP (sub-block-based temporal motion vector prediction candidate). Furthermore, the reference image including the co-occurring block can also be called a co-occurring image (colPic). A specific method for deriving sub-block-based temporal merging candidates will be described later.

[0227] Subsequently, the encoding / decoding device can add the inherited affine candidate to the affine merging candidate list (S1310).

[0228] Specifically, the encoding / decoding device can deduce inherited affine candidates based on the neighboring blocks of the current block. Here, neighboring blocks can include the lower-left neighboring block A0, the left neighboring block A1, the upper neighboring block B0, the upper-right neighboring block B1, and the upper-left neighboring block B2 of the current block.

[0229] Figure 14 An exemplary illustration shows the adjacent blocks of the current block used to derive affine candidates for inheritance. (Refer to...) Figure 14 The adjacent blocks of the current block can include the lower left adjacent block A0, the left adjacent block A1, the upper adjacent block B0, the upper right adjacent block B1, and the upper left adjacent block B2.

[0230] For example, when the size of the current block is W×H, and the x-component of the upper left sample position of the current block is 0 and its y-component is 0, the left adjacent block can be a block that includes a sample with coordinates (-1, H-1), the upper adjacent block can be a block that includes a sample with coordinates (W-1, -1), the upper right adjacent block can be a block that includes a sample with coordinates (W, -1), the lower left adjacent block can be a block that includes a sample with coordinates (-1, H), and the upper left adjacent block can be a block that includes a sample with coordinates (-1, -1).

[0231] Inherited affine candidates can be derived based on valid adjacent reconstructed blocks encoded in affine mode. For example, an encoding / decoding device can sequentially examine adjacent blocks A0, A1, B0, B1, and B2. When adjacent blocks are encoded in affine mode (i.e., when adjacent blocks are validly reconstructed using an affine motion model), two or three CPMVs for the current block can be derived based on the affine motion models of the adjacent blocks, and these CPMVs can be derived as affine candidates for inheritance of the current block. As an example, up to five inherited affine candidates can be added to the affine merge candidate list. That is, up to five inherited affine candidates can be derived based on adjacent blocks.

[0232] Subsequently, the encoding / decoding device can add the constructed affine candidate to the affine merging candidate list (S1320).

[0233] For example, when the number of affine candidates in the affine merging candidate list is less than 5, a constructed affine candidate can be added to the affine merging candidate list. The constructed affine candidate can represent an affine candidate generated by using the neighboring motion information (i.e., motion vectors of neighboring blocks and reference image indices) for each CP in the current block. Motion information for each CP can be derived based on its spatial or temporal neighboring blocks. The motion information for each CP can be called a candidate motion vector for that CP.

[0234] Figure 15 An example is shown of the adjacent blocks of the current block used to derive the affine candidate for construction.

[0235] Reference Figure 15 Adjacent blocks can include spatially adjacent blocks and temporally adjacent blocks. Spatially adjacent blocks can include adjacent block A0, adjacent block A1, adjacent block A2, adjacent block B0, adjacent block B1, adjacent block B2, and adjacent block B3. Figure 15 The adjacent block T shown in the diagram can represent a time-adjacent block.

[0236] Here, adjacent block B2 can represent the adjacent block located to the upper left of the current block's top-left sample position, adjacent block B3 can represent the adjacent block located above the current block's top-left sample position, and adjacent block A2 can represent the adjacent block located to the left of the current block's top-left sample position. Additionally, adjacent block B1 can represent the adjacent block located above the current block's top-right sample position, and adjacent block B0 can represent the adjacent block located to the upper right of the current block's top-right sample position. Furthermore, for example, adjacent block A1 can represent the adjacent block located to the left of the current block's bottom-left sample position, and adjacent block A0 can represent the adjacent block located to the lower left of the current block's bottom-left sample position.

[0237] In addition, refer to Figure 15 The current block's CP can include CP0, CP1, CP2, and / or CP3. CP0 can represent the top-left position of the current block, CP1 can represent the top-right position of the current block, CP2 can represent the bottom-left position of the current block, and CP3 can represent the bottom-right position of the current block. For example, when the size of the current block is W×H and the x-component of the top-left sample position of the current block is 0 and its y-component is 0, CP0 can represent the position at coordinates (0, 0), CP1 can represent the position at coordinates (W, 0), CP2 can represent the position at coordinates (0, H), and CP3 can represent the position at coordinates (W, H).

[0238] The candidate motion vectors for each of the above CPs can be derived as follows.

[0239] For example, the encoding / decoding device can check whether neighboring blocks in the first group are available according to a first order, and can deduce the motion vector of the neighboring block that is first identified as available during the check as a candidate motion vector for CP1. That is, the candidate motion vector for CP1 can be the motion vector of the neighboring block that is first identified as available by checking the neighboring blocks in the first group according to the first order. Availability can indicate the existence of motion vectors of neighboring blocks. That is, available neighboring blocks can be blocks coded with inter-frame prediction (i.e., blocks with inter-frame prediction applied). Here, for example, the first group may include neighboring block B2, neighboring block B3, and neighboring block A2. The first order can be the order in the first group from neighboring block B2 to neighboring block B3 and then to neighboring block A2. As an example, when neighboring block B2 is available, the motion vector of neighboring block B2 can be deduced as a candidate motion vector for CP1. When neighboring block B2 is unavailable and neighboring block B3 is available, the motion vector of neighboring block B3 can be deduced as a candidate motion vector for CP1. When neighboring blocks B2 and B3 are unavailable and neighboring block A2 is available, the motion vector of neighboring block A2 can be derived as a candidate motion vector for CP1.

[0240] Furthermore, for example, the encoding / decoding device can check whether neighboring blocks in the second group are available according to a second order, and can deduce the motion vector of the neighboring block that is first identified as available during the check as a candidate motion vector for CP2. That is, the candidate motion vector for CP2 can be the motion vector of the neighboring block that is first identified as available by checking the neighboring blocks in the second group according to the second order. Availability can indicate the existence of motion vectors of neighboring blocks. That is, available neighboring blocks can be blocks coded with inter-frame prediction (i.e., blocks with inter-frame prediction applied). Here, for example, the second group can include neighboring block B1 and neighboring block B0. The second order can be the order from neighboring block B1 to neighboring block B0 in the second group. As an example, when neighboring block B1 is available, the motion vector of neighboring block B1 can be deduced as a candidate motion vector for CP2. When neighboring block B1 is unavailable and neighboring block B0 is available, the motion vector of neighboring block B0 can be deduced as a candidate motion vector for CP2.

[0241] Here, for example, the second group may include adjacent block B1 and adjacent block B0. The second order may be the order from adjacent block B1 to adjacent block B0 in the second group. As an example, when adjacent block B1 is available, the motion vector of adjacent block B1 can be derived as a candidate motion vector for CP2. When adjacent block B1 is unavailable and adjacent block B0 is available, the motion vector of adjacent block B0 can be derived as a candidate motion vector for CP2. Here, for example, the third group may include adjacent block A1 and adjacent block A0. The third order may be the order from adjacent block A1 to adjacent block A0 in the third group. As an example, when adjacent block A1 is available, the motion vector of adjacent block A1 can be derived as a candidate motion vector for CP3. When adjacent block A1 is unavailable and adjacent block A0 is available, the motion vector of adjacent block A0 can be derived as a candidate motion vector for CP3.

[0242] Furthermore, for example, the encoding / decoding device can check whether a temporally adjacent block (i.e., adjacent block T) is available, and when a temporally adjacent block (i.e., adjacent block T) is available, the motion vector of the temporally adjacent block (i.e., adjacent block T) can be derived as a candidate motion vector for CP4.

[0243] The combination of candidate motion vectors for CP0, candidate motion vectors for CP1, candidate motion vectors for CP2, and / or candidate motion vectors for CP3 can be derived as the constructed affine candidate.

[0244] For example, as mentioned above, a 6-affine model requires motion vectors from three CPs. The three CPs for the 6-affine model can be chosen from CP0, CP1, CP2, and CP3. For example, CPs can be selected as one of {CP0, CP1, CP3}, {CP0, CP1, CP2}, {CP1, CP2, CP3}, and {CP0, CP2, CP3}. As an example, a 6-affine model can be constructed using CP0, CP1, and CP2. In this case, CPs can be represented as {CP0, CP1, CP2}.

[0245] Furthermore, for example, as mentioned above, a 4-affine model requires motion vectors from two CPs. Two CPs can be chosen from CP0, CP1, CP2, and CP3 for the 4-affine model. For example, CPs can be chosen as one of {CP0, CP3}, {CP1, CP2}, {CP0, CP1}, {CP1, CP3}, {CP0, CP2}, and {CP2, CP3}. As an example, a 4-affine model can be constructed using CP0 and CP1. In this case, CP can be represented as {CP0, CP1}.

[0246] The constructed affine candidates, as combinations of candidate motion vectors, can be added to the affine merging candidate list in the following order. That is, after deriving the candidate motion vectors for CP, the constructed affine candidates can be derived in the following order.

[0247] {CP0, CP1, CP2}, {CP0, CP1, CP3}, {CP0, CP2, CP3}, {CP1, CP2, CP3}, {CP0,CP1}, {CP0, CP2}, {CP1, CP2}, {CP0, CP3}, {CP1, CP3}, {CP2, CP3}

[0248] In other words, for example, affine candidates constructed for candidate motion vectors of CP0, CP1, and CP2; affine candidates constructed for candidate motion vectors of CP0, CP1, and CP3; affine candidates constructed for candidate motion vectors of CP0, CP2, and CP3; affine candidates constructed for candidate motion vectors of CP1, CP2, and CP3; affine candidates constructed for candidate motion vectors of CP0 and CP2; affine candidates constructed for candidate motion vectors of CP1, CP2, and CP3; affine candidates constructed for candidate motion vectors of CP0 and CP2; affine candidates constructed for candidate motion vectors of CP0 and CP2; affine candidates constructed for candidate motion vectors of CP0 and CP2; affine candidates constructed for candidate motion vectors of CP3 ...3; affine candidates constructed for candidate motion vectors of CP0 and CP2; affine candidates constructed for candidate motion vectors of CP3; affine candidates constructed for candidate motion vectors of CP0 and CP2; affine candidates constructed for candidate motion vectors of CP3; affine candidates constructed for candidate motion vectors of CP0 and CP2; affine candidates constructed for candidate motion vectors of CP Affine candidates constructed for the candidate motion vectors of CP1, including affine candidates constructed for the candidate motion vectors of CP0 and CP2, including affine candidates constructed for the candidate motion vectors of CP1 and CP2, including affine candidates constructed for the candidate motion vectors of CP0 and CP3, including affine candidates constructed for the candidate motion vectors of CP1 and CP3, including affine candidates constructed for the candidate motion vectors of CP2 and CP3, can be added to the affine merge candidate list in this order.

[0249] Subsequently, the encoding / decoding device can add the zero motion vector to the affine merging candidate list as an affine candidate (S1330).

[0250] For example, when the number of affine candidates in the affine merging candidate list is less than 5, affine candidates including those with zero motion vectors can be added to the affine merging candidate list until the affine merging candidate list is configured with the maximum number of affine candidates. The maximum number of affine candidates can be 5. Additionally, a zero motion vector can represent a motion vector with a vector value of 0.

[0251] Furthermore, according to this disclosure, a method for deriving regression-based affine candidates as affine merging candidates or affine MVP candidates is proposed. For example, the derivation of regression-based affine candidates can be considered as one of the techniques or types used to derive affine candidates in inter-frame prediction.

[0252] For example, when applying a regression-based affine candidate derivation method, the center positions and motion vectors of adjacent sub-blocks can be input into a linear regression process used to derive the parameter set of the linear model, so that the predicted CPMV (control point motion vector) can be derived as a regression-based affine candidate.

[0253] Figure 16 The diagram illustrates adjacent sub-blocks input to the linear regression process used to derive the parameter set of the linear model.

[0254] have Figure 16The 4×4 neighboring sub-blocks of the current block shown can be input into a linear regression process to derive the linear model parameter set. For example, the motion center position and motion vector of the 4×4 neighboring sub-blocks can be input into the linear regression process. The linear regression process can be called a regression model. Alternatively, the linear model parameter set can be the parameters of an affine motion model.

[0255] For example, the parameter set of a linear model used for a six-parameter affine motion model can be as follows.

[0256] [Formula 3]

[0257] Referring to Equation 3, the parameter set of the linear model may include the linear model parameter a. xx a xy a yx a yy b x and b y The set of parameters for a linear model can be called the set of parameters for an affine model, and the parameters of the linear model can be called the parameters of the affine model. The regression model can be a linear regression model based on the mean squared error (MSE) scheme. That is, for example, the affine model parameters can be calculated by solving a regression model based on the mean squared error (MSE) to find the center positions and motion vectors of adjacent sub-blocks, which are then input into the model.

[0258] Reference Figure 16 W can represent the width of the current block, and H can represent the height of the current block. The input to the regression model can include the center positions (x, y) of the available adjacent 4×4 sub-blocks and the motion vectors (mvx and mvy) as described above. Furthermore, for example, as... Figure 16 As shown, the sub-blocks used as input can include the bottom-left adjacent sub-block, the left adjacent sub-block, the top-left adjacent sub-block, the top adjacent sub-block, and / or the top-right adjacent sub-block. Furthermore, as... Figure 16 As shown, the reference area including the upper right adjacent sub-block can be an area with a width of half the width of the current block (W / 2), the reference area including the upper left adjacent sub-block can be an area with a width of half the width of the current block (W / 2), and the reference area including the lower left adjacent sub-block can be an area with a height of half the height of the current block (H / 2).

[0259] Furthermore, for example, a method can be proposed for deriving neighboring sub-blocks based on the shape of the current block, whereby neighboring sub-blocks are used to derive regression-based affine candidates. For instance, when the current block is a non-square block, regression-based affine candidates can be derived based solely on neighboring sub-blocks within a subset of the current block's lower-left, left-side, upper-left, upper, and upper-right neighboring sub-blocks.

[0260] For example, when the current block is a non-square block whose width is greater than its height, the motion vectors and center positions of the available sub-blocks among the top-left, top-upper, and top-right adjacent sub-blocks of the current block can be input into the linear regression process, thereby deriving the predicted CPMV as a regression-based affine candidate. In other words, for example, when the current block is a non-square block whose width is greater than its height, the motion vectors and center positions of the available sub-blocks among the top-left, top-upper, and top-right adjacent sub-blocks of the current block are input into the regression model, and the affine model parameters can be calculated based on the mean squared error (MSE). Furthermore, the CPMV derived based on the affine model parameters can be derived as a regression-based affine candidate.

[0261] For example, when the current block is a non-square block whose width is greater than twice its height, the motion vectors and center positions of the available sub-blocks among the top-left, top-upper, and top-right adjacent sub-blocks can be input into a linear regression process. This allows the predicted CPMV to be derived as a regression-based affine candidate. In other words, for example, when the current block is a non-square block whose width is greater than twice its height, affine model parameters can be calculated by solving for the motion vectors and center positions of the available sub-blocks among the top-left, top-upper, and top-right adjacent sub-blocks and inputting them into a regression model. The affine model parameters can then be calculated based on the mean squared error (MSE), and the CPMV derived from these affine model parameters can be used to derive a regression-based affine candidate.

[0262] For example, when the current block is a non-square block whose height is greater than its width, the motion vectors and center positions of the available sub-blocks among the lower-left, left-side, and upper-left adjacent sub-blocks can be input into a linear regression process. This allows the predicted CPMV to be derived as a regression-based affine candidate. In other words, for example, when the current block is a non-square block whose height is greater than its width, the affine model parameters can be calculated by solving for the motion vectors and center positions of the available sub-blocks among the lower-left, left-side, and upper-left adjacent sub-blocks based on the mean squared error (MSE). The CPMV derived based on these affine model parameters can then be derived as a regression-based affine candidate.

[0263] For example, when the current block is a non-square block whose height is greater than twice its width, the motion vectors and center positions of the available sub-blocks among the lower-left, left-side, and upper-left-side adjacent sub-blocks can be input into a linear regression process. This allows the predicted CPMV to be derived as a regression-based affine candidate. In other words, for example, when the current block is a non-square block whose height is greater than twice its width, the affine model parameters can be calculated by solving for the motion vectors and center positions of the available sub-blocks among the lower-left, left-side, and upper-left-side adjacent sub-blocks based on the mean squared error (MSE). The CPMV derived based on these affine model parameters can then be derived as a regression-based affine candidate.

[0264] Furthermore, for example, by inputting the sub-block motion fields of previously encoded affine blocks (e.g., previously encoded affine CUs) and / or the motion vectors of neighboring sub-blocks adjacent to the current block into a linear regression process, the predicted CPMV (Control Point Motion Vector) can be derived as a regression-based affine candidate. The derived regression-based affine candidate can be added to the affine merge candidate list. Previously encoded affine blocks can be derived by scanning locations not adjacent to the current block and the affine HMVP (History-Based Motion Vector Prediction) table.

[0265] Furthermore, for example, up to two regression-based affine candidates can be derived. For instance, one regression-based affine candidate can be derived based on the motion information of neighboring sub-blocks adjacent to the current block, and another can be derived based on the motion information of blocks other than neighboring sub-blocks. Moreover, when applying Adaptive Reordering with Template Matching Merge Candidates (ARMC-TM), which will be described later, linear regression affine candidates can be included in a single subgroup to which ARMC-TM can be applied. Furthermore, for example, the number of affine candidates to which ARMC-TM is applied can be 30, and a reordered candidate list including 15 candidates can be derived by reordering based on TM cost.

[0266] Furthermore, for example, this disclosure proposes an adaptive reordering of merge candidates with template matching (ARMC-TM). ARMC-TM can refer to a method for reordering merge candidates within a merge candidate list based on TM cost. For example, merge candidates for the current block can be derived based on the adjacent blocks of the current block, and merge candidates can be adaptively reordered based on TM cost. ARMC-TM can be applied to regular merge mode, TM merge mode, and affine merge mode (excluding SbTMVP candidates). For example, in the case of TM merge mode, merge candidates can be reordered before the refinement process. Here, for example, the TM cost can be, for example, the sum of absolute differences (SAD) or the sum of absolute transformation differences (SATD). That is, for example, the SAD or SATD between the template of the current block and the template of the reference block indicated by the merge candidate can be derived as the TM cost of the merge candidate.

[0267] Furthermore, for example, when applying ARMC-TM, the merge candidates in the merge candidate list can be divided into subgroups, and the merge candidates can be reordered for each subgroup, thereby deriving a reordered merge candidate list. Here, for example, when applying the regular merge mode and the TM merge mode, the size of the subgroup can be 5. Furthermore, for example, when applying the affine merge mode, the size of the subgroup can be 3. Furthermore, for example, the merge candidates of the last subgroup that is not the first subgroup may not be reordered. Furthermore, for example, zero candidates in the merge candidate list can be excluded from ARMC-TM. Furthermore, for example, a method for reordering only the merge candidates of the first subgroup can be proposed. For example, when applying the regular merge mode and the TM merge mode, only the 5 merge candidates in the previous order can be reordered, that is, the ARMC-TM process can be applied to them, and when applying the affine merge mode, only the 3 merge candidates in the previous order can be reordered, that is, the ARMC-TM process can be applied to them.

[0268] Specifically, for example, ARMC-TM can be executed as follows.

[0269] First, an initial merge candidate list can be constructed, including spatial merge candidates, TMVPs, non-neighboring spatial merge candidates, HMVPs, paired merge candidates, and / or virtual merge candidates. Then, the merge candidates in the initial merge candidate list can be divided into subgroups.

[0270] Subsequently, when applying Template Matching (TM) merge mode or Adaptive DMVR mode to the current block, each merge candidate in the initial merge candidate list can be refined based on TM or multi-round DMVR. That is, the refinement process can be performed before the reordering process. The merge candidates for each subgroup can be reordered based on the TM cost, thereby generating a reordered merge candidate list. The merge indices pointing to the merge candidates in the reordered merge candidate list can be signaled, and the motion information of the current block can be derived based on the merge candidates pointed to by the merge indices in the reordered merge candidate list.

[0271] Furthermore, for example, when applying ARMC-TM and multi-round DMVR, a reordering process can be applied after the modified merge candidates are derived through the first round of DMVR application. The first round can refer to PU-level DMVR. Furthermore, for example, when deriving modified motion information based on template matching (i.e., when deriving modified merge candidates based on template matching), the template size can be set to 1 (i.e., W×1 and / or 1×H). Here, the width of the current block can be represented by W, and its height can be represented by H. Furthermore, for example, when the width of the current block is greater than twice its height, only the upper template can be used to calculate the TM cost for the refinement process, and when the height of the current block is greater than twice its width, only the left template can be used to calculate the TM cost for the refinement process. Furthermore, for example, 1 / 16-pixel MVD precision can be performed during template matching. Furthermore, for example, in TM merging mode, the first four merge candidates can be reordered using modified motion information.

[0272] For example, the TM cost of merging candidates during the reordering process can be calculated as the SAD or SATD between samples within the template of the current block and samples within the template of the reference block (reference template). The template of the current block can consist of reconstructed samples adjacent to the current block, and the template of the reference block can consist of reconstructed samples adjacent to the reference block within the reference image indicated by the motion information of the merging candidates of the current block.

[0273] Furthermore, for example, when the motion information of the current block's merging candidate is bidirectional motion information, a reference sample of the merging candidate's reference template can be generated through bidirectional prediction, which will be described later.

[0274] Figure 17 An implementation of a reference sample for generating a reference template for merging candidates through bidirectional prediction is illustrated.

[0275] Reference Figure 17For example, the predicted sample of the reference template can be derived by applying a 12-tap interpolation filter to the reference samples of the template (RT0) within the reference images of reference image list 0 and the template (RT1) within the reference images of reference image list 1, and the TM cost of merging candidates can be calculated as the SAD between the predicted sample of the reference template and the reconstructed sample of the current template (T) adjacent to the current block. Alternatively, for example, the predicted sample of the reference template can be derived by applying an N-tap interpolation filter to the reference samples of the template (RT0) within the reference images of reference image list 0 and the template (RT1) within the reference images of reference image list 1, and the TM cost of merging candidates can be calculated as the SAD between the predicted sample of the reference template and the reconstructed sample of the current template (T) adjacent to the current block. Here, N can be 2, 4, 8, 12, or 16. Alternatively, for example, the predicted sample of the reference template can be derived by applying a bilinear filter to the reference samples of the template (RT0) in the reference images of reference image list 0 and the reference samples of the template (RT1) in the reference images of reference image list 1, and the TM cost of merging candidates can be calculated as the SAD between the predicted sample of the reference template and the reconstructed sample of the current template (T) adjacent to the current block.

[0276] Furthermore, for example, when the size of the sub-blocks of the current block is Wsub×Hsub, the upper template used for merging candidates based on sub-blocks can be composed of multiple sub-templates of size Wsub×1, and its left template can be composed of multiple sub-templates of size 1×Hsub.

[0277] Figure 18 The template for the current block and the reference template for merging candidates based on sub-blocks are shown.

[0278] For example, such as Figure 18 As shown, the reference sample for each sub-template can be derived based on the motion information of the sub-blocks in the first row and the first column of the current block.

[0279] Furthermore, for example, criteria can be set for executing ARMC-TM, and the decision on whether to execute ARMC-TM can be made based on these criteria. Specifically, the determination process can be performed as follows.

[0280] Figure 19 An implementation method for determining whether to perform a reordering based on the ARMC-TM execution standard is illustrated.

[0281] For example, first, among all merge candidates in the merge candidate list for the current block, the minimum cost difference between the merge candidate and the previous merge candidate (i.e., the merge candidate in the previous order of the merge candidate) can be derived (determining |D1-D2|).

[0282] Subsequently, when the determined minimum value is greater than or equal to λ, the diversity of the merge candidate list can be considered sufficient, and the reordering process can be stopped. On the other hand, when the determined minimum value is less than λ, the corresponding merge candidate (i.e., the merge candidate with the minimum cost difference from the previous merge candidate) can be considered redundant and can be moved to the next position in the order of the corresponding merge candidate in the merge candidate list. This algorithm can be repeated until the minimum value is not less than λ. λ can be called a threshold or a specific value.

[0283] Reference Figure 19 The minimum cost between the merge candidate and the merge candidate of the previous order can be derived as 5, which is the difference between the cost of the merge candidate at index 4 and the cost of the merge candidate at index 3, which was the previous order. Since 5 is less than 157 (i.e., the value of λ), the merge candidate at index 4 can be reordered to index 5 (i.e., the next order).

[0284] The algorithm described above can be applied to conventional merge patterns, TM merge patterns, BM merge patterns, and / or affine merge patterns. Additionally, for example, similar algorithms can be applied to merge MMVD patterns and / or MVD prediction methods that use ARMC for reordering processes.

[0285] Furthermore, for example, the value of λ can be set to be the same as the rate distortion criterion used to select the best merging candidate for the low-latency configuration on the encoder side. Alternatively, for example, the value of λ can be set to be the same as the value λ corresponding to another QP used for the random access configuration. Alternatively, for example, the set of λ values ​​corresponding to each signaled QP offset can be provided in the SPS, or for QP offsets that are not present in the SPS, they can be provided in the slice header.

[0286] Figure 20 A video / image encoding method according to an embodiment of the present disclosure is illustrated schematically. Figure 20 The method disclosed in the article can be derived from Figure 2 The encoding device disclosed in the document executes the code. Specifically, for example, Figure 20 S2000 to S2030 can be executed by the predictor 220 of the encoding device 200, and Figure 20 S2040 can be executed by the entropy encoder 240 of the encoding device 200. Figure 20 The methods disclosed herein may include the embodiments described above.

[0287] Reference Figure 20The encoding device configures the merge candidate list for the current block (S2000). Based on the RD cost, the encoding device can derive from various prediction modes / types the prediction mode applied to the current block as a regular merge mode, a template matching (TM) merge mode, or an affine merge mode. Here, a regular merge mode can refer to an inter-frame prediction mode that uses the motion information of the adjacent blocks as the motion information of the current block, and a TM merge mode can refer to an inter-frame prediction mode that derives refined merge candidates by performing a template matching-based refinement process on the merge candidates derived from the adjacent blocks and uses the refined merge candidates. Furthermore, an affine merge mode can refer to an inter-frame prediction mode that uses the affine motion model of the adjacent blocks (i.e., the affine candidate (CPMV) derived from the adjacent blocks) as the CPMV of the current block.

[0288] When the prediction mode of the current block is deduced to be a regular merge mode, a template matching (TM) merge mode, or an affine merge mode, the encoding device can configure a merge candidate list based on the adjacent blocks of the current block.

[0289] For example, when applying a regular merge mode or a TM merge mode to the current block, the merge candidate list may include spatial merge candidates, temporal merge candidates, non-proximity spatial merge candidates, HMVP, pairwise merge candidates, and / or virtual merge candidates. Here, spatial merge candidates may be merge candidates derived based on neighboring blocks adjacent to the current block, temporal merge candidates may be merge candidates derived based on temporally adjacent blocks of the current block, non-proximity spatial merge candidates may be merge candidates derived based on neighboring blocks not adjacent to the current block, and HMVP may be merge candidates derived based on motion information of previously encoded reference blocks included in the HMVP table.

[0290] Furthermore, for example, when the affine merging pattern is applied to the current block, the merge candidate list can be the aforementioned affine merging candidate list. That is, for example, when the affine merging pattern is applied to the current block, the merge candidate list can include sub-block-based temporal merge candidates, inherited affine candidates, constructed affine candidates, zero affine candidates, and / or regression-based affine candidates. Here, a zero affine candidate can be an affine candidate that includes the aforementioned zero motion vector.

[0291] The encoding device divides the merge candidates in the merge candidate list into at least one subgroup (S2010). For example, the encoding device may divide the merge candidates in the merge candidate list into at least one subgroup.

[0292] For example, when a regular merge pattern or a TM merge pattern is applied to the current block (i.e., when the prediction pattern for the current block is derived as a regular merge pattern or a TM merge pattern), the size of the subgroup can be 5. That is, for example, when a regular merge pattern or a TM merge pattern is applied to the current block, the merge candidate list can be divided into subgroups containing 5 merge candidates.

[0293] Furthermore, for example, when an affine merge pattern is applied to the current block (i.e., when the prediction pattern of the current block is derived as an affine merge pattern), the size of the subgroup can be 3. That is, for example, when an affine merge pattern is applied to the current block, the merge candidate list can be divided into subgroups containing 3 merge candidates.

[0294] The encoding device reorders the merge candidate list for each subgroup based on the template matching (TM) cost of the merge candidates to configure the modified merge candidate list for the current block (S2020).

[0295] The encoding device can reorder the list of merge candidates for each subgroup based on the template matching (TM) cost of the merge candidates. That is, for example, the encoding device can reorder the merge candidates within a specific subgroup based on their TM costs. For instance, the encoding device can reorder the merge candidates within a specific subgroup in order of lower TM costs.

[0296] In addition, specific subgroups within at least one subgroup of the merge candidate list can be reordered.

[0297] For example, the last subgroup that is not the first subgroup may not be reordered. That is, if the merge candidate list includes one subgroup, the subgroup may be reordered because it is the last subgroup that is the first subgroup, and if the merge candidate list includes multiple subgroups, the last subgroup may not be reordered because it is not the first subgroup.

[0298] Alternatively, for example, only the first subgroup within at least one subgroup can be reordered. That is, subgroups within at least one subgroup that are not the first subgroup may not be reordered.

[0299] Furthermore, for example, the encoding device can determine whether to reorder the merge candidate list based on TM cost. For instance, the merge candidate list may not be reordered if the minimum difference between the TM cost of a merge candidate and the TM cost of a previous merge candidate is greater than or equal to a specific value, and the merge candidate list may be reordered for each subgroup if the minimum difference is less than a specific value. That is, for example, if the minimum difference between the TM cost of a merge candidate and the TM cost of a previous merge candidate is greater than or equal to a specific value, the subgroups of the merge candidate list may not be reordered, and the merge candidate list may be reordered for each subgroup if the minimum difference is less than a specific value. Here, the minimum difference may represent the minimum TM cost difference between a merge candidate and a previous merge candidate (i.e., a merge candidate in the previous order of that merge candidate).

[0300] Furthermore, for example, the specific value can be a preset value. For instance, the specific value could be set to be the same as the rate distortion criterion used by the encoding device to select the best merge candidate for a low-latency configuration. Alternatively, for instance, the specific value could be set to be the same as the value of another QP corresponding to the random access configuration. Or, for instance, the specific value could be set to be the same as the QP offset used for the current block. Information about the QP offset can be communicated via the SPS or slice hair signal.

[0301] Furthermore, for example, the TM cost of merging candidates can be derived as the sum of absolute differences (SAD) or the sum of absolute transformation differences (SATD) between the template of the reference block to which the motion information of the merging candidate points and the template of the current block. The template of the reference block may include the upper neighbor and left neighbor samples adjacent to the reference block, and the template of the current block may include the upper neighbor and left neighbor samples adjacent to the current block.

[0302] Furthermore, for example, when the motion information of the merge candidates in the merge candidate list is bidirectional motion information, the L0 template of the L0 reference block pointed to by the L0 motion information of the merge candidate and the L1 template of the L1 reference block pointed to by the L1 motion information can be derived. The prediction template can be derived by interpolating the L0 template and the L1 template, and the TM cost of the merge candidate can be derived based on the template of the current block and the prediction template. For example, the TM cost can be derived as the sum of absolute differences (SAD) or the sum of absolute transformation differences (SATD) between the template of the current block and the prediction template. Here, the bidirectional motion information can be motion information that includes L0 (list 0) motion information and L1 (list 1) motion information.

[0303] Furthermore, for example, the prediction template can be derived by interpolating the L0 and L1 templates using a 12-tap interpolation filter. Alternatively, for example, the prediction template can be derived by interpolating the L0 and L1 templates using an N-tap interpolation filter, where N can be 2, 4, 8, 12, or 16. Alternatively, for example, the prediction template can be derived by interpolating the L0 and L1 templates using a bilinear filter.

[0304] Furthermore, for example, when the merge candidate is a sub-block-based merge candidate, the template of the merge candidate can include an upper template and a left template, and the upper template can be W sub It consists of multiple sub-templates of size ×1, and the left template can be composed of 1×H. sub It consists of multiple sub-templates of different sizes. Here, W sub It can represent the width of the child blocks of the current block, and H sub It can represent the height of the child blocks of the current block. Affine merge candidates can be merge candidates based on child blocks.

[0305] The encoding device derives the predicted sample for the current block based on the modified merge candidate list (S2030).

[0306] The encoding device can deduce motion information for the current block based on a modified merge candidate list. For example, the encoding device can select a merge candidate from the merge candidates in the modified merge candidate list and deduce motion information for the current block based on the selected merge candidate.

[0307] The encoding device can derive prediction samples for the current block based on motion information. The encoding device can derive prediction samples for the current block by performing prediction based on motion information. That is, the encoding device can derive a reference region within a reference image based on motion information, and can generate prediction samples for the current block based on reconstructed samples within the reference region. In this case, as described above, a prediction sample filtering process for all or some of the prediction samples for the current block can be further performed as needed.

[0308] The encoding device encodes image information including prediction-related information for the current block (S2040). The encoding device can encode image information including prediction-related information for the current block and can signal the image information via a bit stream. That is, the encoding device can output image information including prediction-related information for the current block in the form of a bit stream.

[0309] For example, prediction-related information may include a merge flag indicating whether a regular merge pattern should be applied to the current block and / or an affine merge flag indicating whether an affine merge pattern should be applied to the current block. The affine merge flag may also be called the merge sub-block flag or the sub-block merge flag.

[0310] As an example, the affine merge flag of the current block can be signaled. When the affine merge flag is 1, it indicates that the affine merge mode is applied to the current block, and no signaling is required. Alternatively, the affine merge flag of the current block can be signaled. When the affine merge flag is 0, it indicates that the merge mode is applied to the current block, and when the merge flag is 1, it indicates that the regular merge mode is applied. Furthermore, a flag indicating whether to apply a template matching-based refinement process can be signaled. When this flag is 1, it indicates that the TM merge mode is applied to the current block.

[0311] Alternatively, as another example, the merge flag of the current block can be signaled, and when the merge flag's value is 1, the merge flag can indicate that the regular merge mode should be applied to the current block, and the affine merge flag can be left unsigned. Furthermore, a flag indicating whether a template-matching refinement process should be applied can be signaled, and when the flag's value is 1, the flag can indicate that the TM merge mode is applied to the current block. Additionally, the merge flag of the current block can be signaled, and when the merge flag's value is 0, the affine merge flag can be signaled, and when the affine merge flag's value is 1, the affine merge mode can be indicated to be applied to the current block.

[0312] Furthermore, for example, prediction-related information may include the merge candidate index of the current block. The merge candidate index can indicate one of the merge candidates included in the merge candidate list. That is, the merge candidate index can indicate the selected merge candidate among the merge candidates included in the merge candidate list.

[0313] Furthermore, according to embodiments of this disclosure, image information may include various types of information. For example, image information may include information disclosed in at least one of the tables above.

[0314] In addition, image information may include residual information. Residual information is information about the residual samples. The residual information may include information about the quantized transform coefficients for the residual samples.

[0315] Image information can be sent to the decoding device via a network or storage medium.

[0316] Furthermore, as mentioned above, the encoding device can generate a reconstructed image (including reconstructed samples and reconstructed blocks) based on reference samples and residual samples. This is so that the decoding device can derive the same prediction result as the prediction result performed in the encoding device, thereby improving encoding efficiency. Therefore, the encoding device can store the reconstructed image (or reconstructed samples, reconstructed blocks) in memory and can use the reconstructed image as a reference image for inter-frame prediction. As mentioned above, loop filtering processes, etc., can be further applied to the reconstructed image.

[0317] According to the above implementation, the merge candidate list of the current block can be divided into subgroups, and the merge candidates can be reordered for each subgroup based on TM cost. Thus, only merge candidates with high probability for inter-frame prediction can be effectively reordered, thereby improving the accuracy and efficiency of inter-frame prediction.

[0318] Furthermore, based on TM cost, the merge candidates in the merge candidate list of the current block can be reordered only for subgroups that meet specific conditions. Thus, only merge candidates with a high probability of being used for inter-frame prediction can be effectively reordered, thereby improving the accuracy and efficiency of inter-frame prediction.

[0319] Figure 21 A video / image decoding method according to an embodiment of the present disclosure is illustrated schematically. Figure 21 The method disclosed in the article can be derived from Figure 3 The decoding device disclosed in the document performs the operation. Specifically, for example, Figure 21 S2100 to S2130 can be executed by the predictor 330 of the decoding device 300. Figure 21 The methods disclosed herein may include the embodiments described above.

[0320] Reference Figure 21 The decoding device configures the current block's merge candidate list (S2100).

[0321] For example, a decoding device can obtain prediction-related information for the current block from the bitstream. Based on this information, the decoding device can deduce the prediction mode of the current block as a regular merging mode, a template matching (TM) merging mode, or an affine merging mode. Here, a regular merging mode can refer to an inter-frame prediction mode that uses the motion information of the adjacent blocks as the motion information of the current block, and a TM merging mode can refer to performing a template matching-based refinement process on the merging candidates derived from the adjacent blocks to derive modified merging candidates and using the modified merging candidates as the inter-frame prediction mode. Furthermore, an affine merging mode can refer to an inter-frame prediction mode that uses the affine motion model of the adjacent blocks (i.e., the affine candidate (CPMV) derived from the adjacent blocks) as the CPMV of the current block.

[0322] For example, prediction-related information may include a merge flag indicating whether a regular merge pattern should be applied to the current block and / or an affine merge flag indicating whether an affine merge pattern should be applied to the current block. The affine merge flag may also be called the merge sub-block flag or the sub-block merge flag.

[0323] As an example, the affine merge flag of the current block can be signaled. When the affine merge flag is 1, it indicates that the affine merge mode is applied to the current block, and no signaling is required. Alternatively, the affine merge flag of the current block can be signaled. When the affine merge flag is 0, it indicates that the merge mode is applied to the current block, and when the merge flag is 1, it indicates that the regular merge mode is applied. Furthermore, a flag indicating whether to apply a template matching-based refinement process can be signaled. When this flag is 1, it indicates that the TM merge mode is applied to the current block.

[0324] Alternatively, as another example, the merge flag of the current block can be signaled, and when the merge flag's value is 1, the merge flag can indicate that the regular merge mode is applied to the current block, and the affine merge flag can be left unsigned. Furthermore, a flag indicating whether a template-matching refinement process is applied can be signaled, and when this flag's value is 1, it can indicate that the TM merge mode is applied to the current block. Additionally, the merge flag of the current block can be signaled, and when the merge flag's value is 0, the affine merge flag can be signaled, and when the affine merge flag's value is 1, the affine merge mode is applied to the current block.

[0325] When the prediction mode of the current block is deduced to be a regular merge mode, a template matching (TM) merge mode, or an affine merge mode, the decoding device can configure a merge candidate list based on the adjacent blocks of the current block.

[0326] For example, when a regular merge mode or a TM merge mode is applied to the current block, the merge candidate list may include spatial merge candidates, temporal merge candidates, non-proximity spatial merge candidates, HMVP, pairwise merge candidates, and / or virtual merge candidates. Here, spatial merge candidates may be merge candidates derived based on neighboring blocks adjacent to the current block, temporal merge candidates may be merge candidates derived based on temporally adjacent blocks of the current block, non-proximity spatial merge candidates may be merge candidates derived based on neighboring blocks not adjacent to the current block, and HMVP may be merge candidates derived based on motion information of previously encoded reference blocks included in the HMVP table.

[0327] Furthermore, for example, when the affine merging pattern is applied to the current block, the merge candidate list can be the aforementioned affine merging candidate list. That is, for example, when the affine merging pattern is applied to the current block, the merge candidate list can include sub-block-based temporal merge candidates, inherited affine candidates, constructed affine candidates, zero affine candidates, and / or regression-based affine candidates. Here, a zero affine candidate can be an affine candidate that includes the aforementioned zero motion vector.

[0328] The decoding device divides the merge candidates in the merge candidate list into at least one subgroup (S2110). For example, the decoding device may divide the merge candidates in the merge candidate list into at least one subgroup.

[0329] For example, when a regular merge pattern or a TM merge pattern is applied to the current block (i.e., when the prediction pattern for the current block is derived as a regular merge pattern or a TM merge pattern), the size of the subgroup can be 5. That is, for example, when a regular merge pattern or a TM merge pattern is applied to the current block, the merge candidate list can be divided into subgroups containing 5 merge candidates.

[0330] Furthermore, for example, when an affine merge pattern is applied to the current block (i.e., when the prediction pattern of the current block is derived as an affine merge pattern), the size of the subgroup can be 3. That is, for example, when an affine merge pattern is applied to the current block, the merge candidate list can be divided into subgroups containing 3 merge candidates.

[0331] The decoding device reorders the list of merge candidates for each subgroup based on the template matching (TM) cost of the merge candidates to configure the modified list of merge candidates for the current block (S2120).

[0332] The decoding device can reorder the list of merge candidates for each subgroup based on the template matching (TM) cost of the merge candidates. That is, for example, the decoding device can reorder the merge candidates within a specific subgroup based on their TM cost. For instance, the decoding device can reorder the merge candidates within a specific subgroup in order of lower TM cost.

[0333] In addition, specific subgroups within at least one subgroup of the merge candidate list can be reordered.

[0334] For example, the last subgroup that is not the first subgroup may not be reordered. That is, if the merge candidate list includes one subgroup, the subgroup may be reordered because it is the last subgroup that is the first subgroup, and if the merge candidate list includes multiple subgroups, the last subgroup may not be reordered because it is not the first subgroup.

[0335] Alternatively, for example, only the first subgroup within at least one subgroup can be reordered. That is, subgroups within at least one subgroup that are not the first subgroup may not be reordered.

[0336] Furthermore, for example, the decoding device can determine whether to reorder the merge candidate list based on TM cost. For instance, the merge candidate list may not be reordered if the minimum difference between the TM cost of a merge candidate and the TM cost of a previous merge candidate is greater than or equal to a specific value, and the merge candidate list may be reordered for each subgroup if the minimum difference is less than a specific value. That is, for example, if the minimum difference between the TM cost of a merge candidate and the TM cost of a previous merge candidate is greater than or equal to a specific value, the subgroups of the merge candidate list may not be reordered, and the merge candidate list may be reordered for each subgroup if the minimum difference is less than a specific value. Here, the minimum difference may represent the minimum TM cost difference between a merge candidate and a previous merge candidate (i.e., a merge candidate in the previous order of that merge candidate).

[0337] Furthermore, for example, the specific value can be a preset value. For instance, the specific value can be set to be the same as the rate distortion criterion used by the encoding device to select the best merge candidate for a low-latency configuration. Alternatively, for instance, the specific value can be set to be the same as the value corresponding to another QP used for a random access configuration. Or, for instance, the specific value can be set to be the same as the QP offset for the current block. Information about the QP offset can be communicated via the SPS or slice hair signal.

[0338] Furthermore, for example, the TM cost of merging candidates can be derived as the sum of absolute differences (SAD) or the sum of absolute transformation differences (SATD) between the template of the reference block to which the motion information of the merging candidate points and the template of the current block. The template of the reference block may include the upper neighbor and left neighbor samples adjacent to the reference block, and the template of the current block may include the upper neighbor and left neighbor samples adjacent to the current block.

[0339] Furthermore, for example, when the motion information of the merge candidates in the merge candidate list is bidirectional motion information, the L0 template of the L0 reference block pointed to by the L0 motion information of the merge candidate and the L1 template of the L1 reference block pointed to by the L1 motion information can be derived. The prediction template can be derived by interpolating the L0 template and the L1 template, and the TM cost of the merge candidate can be derived based on the template of the current block and the prediction template. For example, the TM cost can be derived as the sum of absolute differences (SAD) or the sum of absolute transformation differences (SATD) between the template of the current block and the prediction template. Here, the bidirectional motion information can be motion information that includes L0 (list 0) motion information and L1 (list 1) motion information.

[0340] Furthermore, for example, the prediction template can be derived by interpolating the L0 and L1 templates using a 12-tap interpolation filter. Alternatively, for example, the prediction template can be derived by interpolating the L0 and L1 templates using an N-tap interpolation filter, where N can be 2, 4, 8, 12, or 16. Alternatively, for example, the prediction template can be derived by interpolating the L0 and L1 templates using a bilinear filter.

[0341] Furthermore, for example, when the merge candidate is a sub-block-based merge candidate, the template of the merge candidate can include an upper template and a left template, and the upper template can be W sub It consists of multiple sub-templates of size ×1, and the left template can be composed of 1×H. sub It consists of multiple sub-templates of different sizes. Here, W sub It can represent the width of the child blocks of the current block, and H sub It can represent the height of the child blocks of the current block. Affine merge candidates can be merge candidates based on child blocks.

[0342] The decoding device derives the prediction sample for the current block based on the modified merge candidate list (S2130).

[0343] The decoding device can deduce the motion information of the current block based on a modified merge candidate list. For example, the decoding device can select a merge candidate from the merge candidates in the modified merge candidate list and deduce the motion information of the current block based on the selected merge candidate.

[0344] Furthermore, for example, the decoding device can obtain a merge candidate index for the current block from the bitstream, and can select the merge candidate pointed to by the merge candidate index from among the merge candidates included in the modified merge candidate list. The decoding device can obtain image information from the bitstream, and the image information may include prediction-related information for the current block. The prediction-related information may include the merge candidate index.

[0345] The decoding device can derive predicted samples for the current block based on motion information. The decoding device can derive predicted samples for the current block by performing predictions based on motion information. That is, the decoding device can derive a reference region within a reference image based on motion information, and can generate predicted samples for the current block based on reconstructed samples within the reference region. In this case, as described above, depending on the circumstances, a predicted sample filtering process for all or some of the predicted samples for the current block can be further performed.

[0346] The decoding device can generate reconstructed samples based on the predicted samples of the current block. For example, the decoding device can generate reconstructed samples for the current block based on the residual samples and predicted samples for the current block. The residual samples for the current block can be generated based on the received residual information. Furthermore, as an example, the decoding device can generate a reconstructed image including the reconstructed samples. As described above, loop filtering processes, etc., can then be further applied to the reconstructed image.

[0347] According to the above implementation, the merge candidate list of the current block can be divided into subgroups, and the merge candidates can be reordered for each subgroup based on TM cost. Thus, only merge candidates with high probability for inter-frame prediction can be effectively reordered, thereby improving the accuracy and efficiency of inter-frame prediction.

[0348] Furthermore, based on TM cost, the merge candidates in the merge candidate list of the current block can be reordered only for subgroups that meet specific conditions. Thus, only merge candidates with a high probability of being used for inter-frame prediction can be effectively reordered, thereby improving the accuracy and efficiency of inter-frame prediction.

[0349] Although the method is described as a series of steps or blocks based on the flowchart in the above embodiments, the embodiments are not limited to the order of the steps, and a step may occur in a different order than another step described above or simultaneously with another step described above. Furthermore, those skilled in the art will understand that the steps shown in the flowchart are not exclusive, and other steps may be included or one or more steps in the flowchart may be deleted without affecting the scope of the embodiments of this disclosure.

[0350] The methods described above according to embodiments of the present disclosure can be implemented in software, and the encoding and / or decoding devices according to the present disclosure can be included in devices for performing image processing, such as televisions, computers, smartphones, set-top boxes, and display devices.

[0351] The embodiments described above can be implemented in the form of a recording medium including computer-executable (program) instructions, such as a program module executed by a computer. The module can be stored in memory and executed by a processor. The memory can be located inside or outside the processor and can be connected to the processor by various known means. The computer-readable medium can be any available medium accessible to a computer and can include both volatile and non-volatile media, as well as removable and non-removable media. Furthermore, the computer-readable medium can include both computer storage media and communication media. Computer storage media can include both volatile and non-volatile media, as well as removable and non-removable media, implemented using any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Communication media generally include computer-readable instructions, data structures, program modules, other data in modulated data signals (such as carrier waves), or other transmission mechanisms, and include any information delivery medium.

[0352] Furthermore, the embodiments described above in this disclosure can be implemented as a computer program (or computer program product) including computer-executable instructions. The computer program may include programmable machine instructions processed by a processor and may be implemented in a high-level programming language, an object-oriented programming language, assembly language, or machine language. Additionally, the computer program may be recorded on a tangible computer-readable recording medium (e.g., memory, hard disk, magnetic / optical media, or solid-state drive (SSD)).

[0353] Therefore, embodiments of this disclosure can be implemented by executing the computer program described above using a computing device. The computing device may include at least some of a processor, memory, storage devices, high-speed interfaces connected to the memory and high-speed expansion ports, and low-speed interfaces connected to low-speed buses and storage devices. These components may be interconnected via various buses and may be mounted on a common motherboard or otherwise suitably configured.

[0354] A processor can process instructions within a computing device. These instructions may include those stored in memory or storage devices to display graphical information on an external input / output device (such as a display) connected to a high-speed interface, providing a graphical user interface (GUI). In another embodiment, multiple processors and / or multiple buses may be appropriately utilized along with multiple memories and memory types. Furthermore, the processor may be implemented as a chipset comprising multiple independent analog and / or digital processors.

[0355] Memory stores information within a computing device. For example, memory may include volatile memory cells or a collection of volatile memory cells. In another example, memory may include non-volatile memory cells or a collection of non-volatile memory cells. Memory may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0356] Storage devices can provide large-capacity storage space for computing devices. Storage devices can be computer-readable media or components that include computer-readable media. For example, storage devices can include devices or other components within a storage area network (SAN), and can be floppy disk devices, hard disk devices, optical disk devices, magnetic tape devices, flash memory, other similar semiconductor storage devices, or device arrays.

[0357] The network can be implemented as a wired network, such as a local area network (LAN), a wide area network (WAN), or a value-added network (VAN), or various types of wireless networks, such as mobile radio communication networks or satellite communication networks.

[0358] Although this disclosure has been described with reference to embodiments illustrated in the accompanying drawings, these embodiments are merely exemplary. Those skilled in the art will understand that various modifications and variations are possible. That is, the scope of this disclosure is not limited to the described embodiments, and various modifications and alterations made by those skilled in the art based on the basic concepts defined in the appended claims also fall within the scope of the claims. Therefore, the true technical scope of this disclosure should be determined by the technical spirit of the appended claims.

Claims

1. An image decoding method performed by a decoding device, the image decoding method comprising the following steps: Construct a list of candidate blocks to merge; Divide the merge candidates in the merge candidate list into at least one subgroup; The modified list of merge candidates for the current block is constructed by reordering the list of merge candidates for each subgroup based on the template matching TM cost of the merge candidates. as well as The predicted sample for the current block is derived based on the modified merge candidate list. The reordered subgroups are those in which the merge candidates within the subgroups are reordered based on the TM cost.

2. The image decoding method according to claim 1, wherein, The last subgroup that is not the first subgroup is not reordered.

3. The image decoding method according to claim 1, wherein, The first subgroup of at least one subgroup is reordered.

4. The image decoding method according to claim 1, wherein, The TM cost of the merge candidate in the merge candidate list is derived as the sum of absolute differences (SAD) or absolute transformation differences (SATD) between the template of the reference block indicated by the motion information of the merge candidate and the template of the current block.

5. The image decoding method according to claim 1, wherein, An affine merge pattern is applied to the current block, and the size of the subgroup is 3.

6. The image decoding method according to claim 1, wherein, When the motion information of the merging candidate in the merging candidate list is bidirectional predicted motion information, the L0 template of the L0 reference block indicated by the L0 motion information of the merging candidate and the L1 template of the L1 reference block indicated by the L1 motion information are derived. The prediction template is derived by interpolating the L0 template and the L1 template. The TM cost of the merge candidate is derived based on the template of the current block and the prediction template.

7. The image decoding method according to claim 6, wherein, The prediction template is derived by interpolating the L0 template and the L1 template using a 12-tap interpolation filter.

8. The image decoding method according to claim 6, wherein, The prediction template is derived by interpolating the L0 template and the L1 template using an 8-tap interpolation filter.

9. The image decoding method according to claim 1, wherein, The step of constructing the modified merge candidate list for the current block by reordering the merge candidate list for each subgroup based on the TM cost of the merge candidates includes: determining whether to reorder the merge candidate list based on the TM cost. Wherein, when the minimum difference between the TM cost of a merging candidate in the merging candidate list and the TM cost of a previous merging candidate is equal to or greater than a specific value, the merging candidate list is not reordered.

10. An image encoding method performed by an encoding device, the image encoding method comprising the following steps: Construct a list of candidate blocks to merge; Divide the merge candidates in the merge candidate list into at least one subgroup; The modified list of merge candidates for the current block is constructed by reordering the list of merge candidates for each subgroup based on the template matching TM cost of the merge candidates. The predicted sample for the current block is derived based on the modified list of merge candidates; as well as The image information, including prediction-related information for the current block, is encoded. The reordered subgroups are those in which the merge candidates within the subgroups are reordered based on the TM cost.

11. The image encoding method according to claim 10, wherein, The last subgroup that is not the first subgroup is not reordered.

12. The image encoding method according to claim 10, wherein, The first subgroup of at least one subgroup is reordered.

13. The image encoding method according to claim 10, wherein, The TM cost of the merge candidate in the merge candidate list is derived as the sum of absolute differences (SAD) or absolute transformation differences (SATD) between the template of the reference block indicated by the motion information of the merge candidate and the template of the current block.

14. The image encoding method according to claim 10, wherein, An affine merge pattern is applied to the current block, and the size of the subgroup is 3.

15. The image encoding method according to claim 10, wherein, When the motion information of the merging candidate in the merging candidate list is bidirectional predicted motion information, the L0 template of the L0 reference block indicated by the L0 motion information of the merging candidate and the L1 template of the L1 reference block indicated by the L1 motion information are derived. The prediction template is derived by interpolating the L0 template and the L1 template. The TM cost of the merge candidate is derived based on the template of the current block and the prediction template.

16. The image encoding method according to claim 15, wherein, The prediction template is derived by interpolating the L0 template and the L1 template using a 12-tap interpolation filter.

17. The image encoding method according to claim 15, wherein, The prediction template is derived by interpolating the L0 template and the L1 template using an 8-tap interpolation filter.

18. The image encoding method according to claim 10, wherein, The step of constructing the modified merge candidate list for the current block by reordering the merge candidate list for each subgroup based on the TM cost of the merge candidates includes: determining whether to reorder the merge candidate list based on the TM cost. Wherein, when the minimum difference between the TM cost of a merging candidate in the merging candidate list and the TM cost of a previous merging candidate is equal to or greater than a specific value, the merging candidate list is not reordered.

19. A method for transmitting image data, the method comprising the following steps: Obtain a bitstream generated by an encoding method, wherein the encoding method includes the following steps: constructing a merge candidate list for the current block; dividing the merge candidates in the merge candidate list into at least one subgroup; constructing a modified merge candidate list for the current block by reordering the merge candidate list for each subgroup based on the template matching TM cost of the merge candidates; deriving a prediction sample for the current block based on the modified merge candidate list; and encoding image information including prediction-related information of the current block; and Send the image data including the bitstream; and The reordered subgroups are those in which the merge candidates within the subgroups are reordered based on the TM cost.

20. The method according to claim 19, wherein, The first subgroup of at least one subgroup is reordered.

Images