History-based image coding method and apparatus

Abstract

To provide a method and apparatus for improving image coding efficiency.SOLUTION: The method includes deriving a history-basedMotionVectorPrediction (HMVP) buffer for a current block, constructing a motion information candidate list based on HMVP candidates included in the HMVP buffer, deriving motion information of the current block based on the motion information candidate list, deriving a reference picture index of the current block based on the motion information, deriving a motion vector of the current block based on the motion information, generating prediction samples for the current block based on the reference picture index and the motion vector, and generating reconstructed samples based on the prediction samples. The current picture includes one or more tiles and includes a plurality of tile columns and tile rows, and a tile is a rectangular region of coding tree units (CTUs) within a specific tile column and a specific tile row in the current picture.SELECTED DRAWING: Figure 27

Description

[Technical Field]

[0001] The present technology relates to image coding, and more particularly to a history-based image coding method and apparatus. [Background technology]

[0002] Recently, demand for high-resolution, high-quality images / videos, such as 4K or 8K or higher UHD (Ultra High Definition) images / videos, is increasing in various fields. As the resolution and quality of image / video data increases, the amount of information or bits to be transmitted increases relative to existing image / video data. Therefore, when image data is transmitted using existing media such as wired or wireless broadband lines, or when image / video data is stored using existing storage media, transmission costs and storage costs increase.

[0003] In addition, interest and demand for immersive media such as VR (Virtual Reality), AR (Artificial Reality) content and holograms has been increasing recently, and broadcasts of images / videos with different image characteristics from real images, such as game images, are on the rise.

[0004] Therefore, a highly efficient image / video compression technique is required to effectively compress, transmit, store, and play back high-resolution, high-quality image / video information having the above-mentioned various characteristics. Summary of the Invention [Problem to be solved by the invention]

[0005] The technical problem of this document is to provide a method and apparatus for increasing image coding efficiency.

[0006] Another technical problem of this document is to provide an efficient inter-prediction method and apparatus.

[0007] Another technical problem of this document is to provide a method and apparatus for deriving history-based motion vectors.

[0008] Another technical problem of this document is to provide a method and apparatus for efficiently deriving HMVP (History-based Motion Vector Prediction) candidates.

[0009] Another technical problem of this document is to provide a method and apparatus for efficiently updating the HMVP buffer.

[0010] Another technical problem of this document is to provide a method and apparatus for efficiently initializing an HMVP buffer. [Means for solving the problem]

[0011] According to an embodiment of the present document, there is provided an image decoding method performed by a decoding device, the method comprising the steps of: deriving an HMVP (History-based Motion Vector Prediction) buffer for a current block; constructing a motion information candidate list based on HMVP candidates included in the HMVP buffer; deriving motion information for the current block based on the motion information candidate list; generating predicted samples for the current block based on the motion information; and generating reconstructed samples based on the predicted samples, wherein one or more tiles exist in the current picture, and the HMVP buffer is initialized with a first CTU in a CTU row that includes the current block in the current tile.

[0012] Another embodiment of the present document provides a decoding device for performing image decoding, the decoding device including: a prediction unit that derives an HMVP (History-based Motion Vector Prediction) buffer for a current block, constructs a motion information candidate list based on HMVP candidates stored in the HMVP buffer, derives motion information for the current block based on the motion information candidate list, and generates predicted samples for the current block based on the motion information; and an adder that generates reconstructed samples based on the predicted samples, wherein one or more tiles exist in the current picture, and the HMVP buffer is initialized with the first CTU of a CTU row that includes the current block in the current tile.

[0013] According to yet another embodiment of the present document, there is provided an image encoding method performed by an encoding device, the method comprising the steps of: deriving a History-based Motion Vector Prediction (HMVP) buffer for a current block; constructing a motion information candidate list based on HMVP candidates included in the HMVP buffer; deriving motion information for the current block based on the motion information candidate list; generating prediction samples for the current block based on the motion information; deriving residual samples based on the prediction samples; and encoding image information including information on the residual samples, wherein one or more tiles exist in the current picture, and the HMVP buffer is initialized with a first CTU in a CTU row including the current block in the current tile.

[0014] According to yet another embodiment of the present document, there is provided an encoding device for performing image encoding, the encoding device including: a prediction unit that derives an HMVP (History-based Motion Vector Prediction) buffer for a current block, constructs a motion information candidate list based on HMVP candidates stored in the HMVP buffer, derives motion information for the current block based on the motion information candidate list, and generates predicted samples for the current block based on the motion information; a residual processing unit that derives residual samples based on the predicted samples; and an entropy encoding unit that encodes image information including information on the residual samples, wherein the current picture includes one or more tiles, and the HMVP buffer is initialized with a first CTU in a CTU row that includes the current block in the current tile.

[0015] According to yet another embodiment of the present document, there is provided a digital storage medium having stored thereon image data having encoded image information generated by an image encoding method performed by an encoding device.

[0016] According to yet another embodiment of the present document, there is provided a digital storage medium having stored thereon image data having encoded image information that causes a decoding device to perform an image decoding method. [Effects of the Invention]

[0017] According to one embodiment of this document, the overall image / video compression efficiency can be increased.

[0018] According to one embodiment of this document, the amount of transmitted data required for residual processing can be reduced through efficient inter prediction.

[0019] According to one embodiment of this document, the HMVP buffer can be managed efficiently.

[0020] According to one embodiment of this document, parallel processing can be supported through efficient HMVP buffer management.

[0021] According to one embodiment of this document, motion vectors for inter prediction can be derived efficiently. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 illustrates a schematic diagram of an example of a video / image coding system to which embodiments of the present document may be applied. [Figure 2] 1 is a diagram illustrating a schematic configuration of a video / image encoding device to which embodiments of the present document can be applied; [Figure 3] 1 is a diagram illustrating the configuration of a video / image decoding device to which the embodiments of the present document can be applied; [Figure 4] FIG. 1 illustrates an example of an inter-prediction based video / image encoding method. [Figure 5] FIG. 1 illustrates an example of an inter-prediction based video / image decoding method. [Figure 6] FIG. 10 is a diagram illustrating an example of an inter-prediction procedure. [Figure 7] 10 is a diagram illustrating an example of spatially adjacent blocks used for motion information candidate derivation in conventional merge or AMVP mode. [Figure 8] FIG. 1 illustrates a schematic diagram of an example of an HMVP candidate-based decoding procedure. [Figure 9] FIG. 10 is a diagram illustrating an example of HMVP table updating according to the FIFO rule. [Figure 10] FIG. 10 is a diagram illustrating an example of an HMVP table update according to the restricted FIFO rule. [Figure 11] FIG. 1 is a diagram illustrating an example of WPP (Wavefront Parallel Processing), which is one of the techniques for parallel processing. [Figure 12] FIG. 1 is a diagram illustrating an example of a problem that occurs when applying a general HMVP method in consideration of parallel processing. [Figure 13]FIG. 10 is an exemplary diagram illustrating a method for initializing a history management buffer (HMVP buffer) according to one embodiment of the present document. [Figure 14] 1 is a diagram illustrating an exemplary HMVP buffer management method according to an embodiment. [Figure 15] 10 is a diagram illustrating an example of an HMVP buffer management method according to another embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of an HMVP buffer initialization method in a tile structure. [Figure 17] FIG. 10 is a diagram illustrating an example of an HMVP buffer initialization method for the first CTU of a tile according to another embodiment. [Figure 18] FIG. 10 is a diagram illustrating an example of an HMVP management buffer initialization method for the first CTU in a CTU row in each tile according to yet another embodiment. [Figure 19] FIG. 10 is a diagram showing an example of a structure in which tiles and slices coexist. [Figure 20] FIG. 10 is a diagram illustrating an example of a method for initializing an HMVP buffer for the first CTU in each tile. [Figure 21] FIG. 10 is a diagram illustrating an example of a method for initializing an HMVP buffer for each slice in a tile. [Figure 22] FIG. 10 is a diagram illustrating an example of initializing an HMVP buffer for the first CTU of the first tile in a tile group. [Figure 23] FIG. 10 is a diagram illustrating an example of initializing an HMVP buffer for the first CTU of each tile in a tile group. [Figure 24] FIG. 10 is a diagram illustrating an example of initializing an HMVP buffer for the CTU rows of each tile in a tile group. [Figure 25] 1 illustrates a schematic diagram of an example of a video / image encoding method and associated components including an inter-prediction method according to an embodiment(s) of the present document. [Figure 26]1 illustrates a schematic diagram of an example of a video / image encoding method including an inter-prediction method and related components according to one or more embodiments of the present document; [Figure 27] 1 is a diagram illustrating an example of an image decoding method including an inter-prediction method and related components according to an embodiment of the present document; [Figure 28] 1 is a diagram illustrating an example of an image decoding method including an inter-prediction method and related components according to an embodiment of the present document; [Figure 29] FIG. 1 illustrates an example of a content streaming system in which embodiments such as those disclosed herein may be applied. DETAILED DESCRIPTION OF THE INVENTION

[0023] The methods presented in this document may be modified in various ways and may have various embodiments. Specific embodiments will be illustrated in the drawings and described in detail. The terms used in this document are merely used to describe specific embodiments and are not intended to limit the technical ideas of the methods presented in this document. The singular expressions include the expression "at least one" unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood not to preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0024] Meanwhile, each component in the drawings described in this document is illustrated independently for the convenience of explaining the different characteristic functions, etc., and does not mean that each component is realized by separate hardware or software. For example, two or more components may be combined to form a single component, or a single component may be divided into multiple components. Embodiments in which each component is integrated and / or separated are also included within the scope of this document as long as they do not deviate from the essence of the method disclosed herein.

[0025] This document relates to video / image coding. For example, the methods / embodiments disclosed in this document may be applied to methods disclosed in the Versatile Video Coding (VVC) standard. The methods / embodiments disclosed in this document may also be applied to methods disclosed in the Essential Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the 2nd Generation Of Audio Video Coding Standard (AVS2), or next-generation video / image coding standards (e.g., H.267 or H.268).

[0026] This document presents various embodiments for video / image coding, and unless otherwise stated, the embodiments may also be implemented in combination with each other.

[0027] In this document, video can refer to a collection of a series of images over time. A picture generally refers to a unit that shows one image at a specific time, and a slice / tile is a unit that constitutes part of a picture in coding. A slice / tile can include one or more coding tree units (CTUs). A picture can be composed of one or more slices / tiles. A picture can be composed of one or more tile groups. A tile group can include one or more tiles. A brick can represent a rectangular region of CTU rows within a tile in a picture. A tile can be partitioned into multiple bricks, and each brick can consist of one or more CTU rows within the tile. A tile that is not partitioned into multiple bricks may also be referred to as a brick.A brick scan can refer to a specific sequential ordering of CTUs partitioning a picture, where the CTUs can be ordered consecutively in a CTU raster scan within a brick, the bricks within a tile can be ordered consecutively in a raster scan of the bricks of the tile, and the tiles within a picture can be ordered consecutively in a raster scan of the tiles of the picture. A tile is a rectangular region of CTUs within a particular tile column and a particular tile row in a picture. The tile column is a rectangular region of CTUs having a height equal to the height of the picture and a width specified by syntax elements in the picture parameter set.The tile row is a rectangular region of CTUs having a height specified by syntax elements in the picture parameter set and a width equal to the height of the picture. A tile scan can indicate a specific sequential ordering of CTUs partitioning a picture, where the CTUs can be consecutively ordered in a CTU raster scan in a tile, and tiles in a picture can be consecutively ordered in a raster scan of the tiles of the picture. A slice includes an integer number of bricks of a picture that may be exclusively contained in a single NAL unit. A slice may consist of either a number of complete tiles or only a consecutive sequence of complete bricks of one tile.In this document, the terms tile group and slice can be used interchangeably, for example, tile group / tile group header is also called slice / slice header in this document.

[0028] A pixel or a pel may refer to the smallest unit that constitutes one picture (or image). A "sample" may also be used as a term corresponding to a pixel. A sample generally refers to a pixel or a pixel value, and may refer to only the value of a pixel / pixel of a luma component, or may refer to only the value of a pixel / pixel of a chroma component.

[0029] A unit may refer to a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. The term unit may be mixed with terms such as block or area in some cases. In a general case, an M×N block may include a set (or array) of samples or transform coefficients consisting of M columns and N rows.

[0030] In this document, the terms " / " and "," should be interpreted to mean "and / or." For example, "A / B" means "A and / or B," and "A, B" means "A and / or B." Additionally, "A / B / C" means "at least one of A, B, and / or C." Also, "A, B, C" means "at least one of A, B, and / or C." (In this document, the terms " / " and "," should be interpreted to indicate "and / or." For instance, the expression "A / B" may mean "A and / or B." Further, "A, B" may mean "A and / or B." Further, "A / B / C" may mean "at least one of A, B, and / or C." Also, "A / B / C" may mean "at least one of A, B, and / or C.")

[0031] Additionally, in this document, "or" should be interpreted as "and / or." For example, "A or B" can mean 1) only "A," 2) only "B," or 3) "A and B." Furthermore, in this document, the term "or" can mean "additionally or alternatively." For instance, the expression "A or B" may comprise 1) only A, 2) only B, and / or 3) both A and B. In other words, the term "or" in this document should be interpreted to indicate "additionally or alternatively."

[0032] Hereinafter, the embodiments of the present document will be described in more detail with reference to the accompanying drawings. Hereinafter, the same components in the drawings will be designated by the same reference numerals, and duplicate descriptions of the same components may be omitted.

[0033] FIG. 1 illustrates schematically an example of a video / image coding system in which embodiments of the present document may be applied.

[0034] As shown in Figure 1, a video / image coding system can include a first device (source device) and a second device (receiving device). The source device can transmit encoded video / image information or data to the receiving device in file or streaming form via a digital storage medium or a network.

[0035] The source device may include a video source, an encoding device, and a transmitting unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may also be called a video / image encoding device, and the decoding device may also be called a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may also include a display unit, which may be a separate device or an external component.

[0036] A video source can acquire video / images through a video / image capture, synthesis, or generation process, etc. A video source can include a video / image capture device and / or a video / image generation device. A video / image capture device can include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device can include, for example, a computer, a tablet, a smartphone, etc., and can (electronically) generate video / images. For example, a virtual video / image can be generated via a computer, etc., in which case the video / image capture process can be substituted for the process by which the associated data is generated.

[0037] An encoding device can encode input video / images. The encoding device can perform a series of steps, such as prediction, transformation, and quantization, for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0038] The transmitter can transmit the encoded video / image information or data output in the form of a bitstream to a receiver of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter can include elements for generating a media file in a predetermined file format and elements for transmission via a broadcasting / communication network. The receiver can receive / extract the bitstream and transmit it to a decoding device.

[0039] The decoding device can decode the video / image by performing a series of steps such as inverse quantization, inverse transform, prediction, etc., which correspond to the operations of the encoding device.

[0040] The renderer can render the decoded video / images, which can then be displayed via the display unit.

[0041] 2 is a diagram for explaining the configuration of a video / image encoding device to which the embodiments of this document can be applied. Hereinafter, the video encoding device may include an image encoding device.

[0042] As shown in FIG. 2, the encoding device 200 may include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter predictor 221 and an intra predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may further include a subtractor 231. The adder 250 may also be referred to as a reconstructor or a reconstructed block generator. The image dividing unit 210, the predicting unit 220, the residual processing unit 230, the entropy encoding unit 240, the adding unit 250, and the filtering unit 260 may be configured by one or more hardware components (e.g., an encoder chipset or a processor) depending on the embodiment. The memory 270 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium. The hardware components may further include the memory 270 as an internal / external component.

[0043] The image division unit 210 may divide an input image (or picture, frame) input to the encoding device 200 into one or more processing units. For example, the processing units may also be called coding units (CUs). In this case, the coding units may be recursively divided into coding tree units (CTUs) or largest coding units (LCUs) using a quad-tree, binary-tree, and ternary-tree (QTBTTT) structure. For example, one coding unit may be divided into multiple coding units of deeper depths based on a quad-tree structure, a binary tree structure, and / or a ternary tree structure. In this case, for example, the quad-tree structure may be applied first, and then the binary tree structure and / or the ternary tree structure may be applied. Alternatively, the binary tree structure may be applied first. The coding procedure according to this document may be performed based on a final coding unit that is not further divided. In this case, the largest coding unit may be used as the final coding unit based on coding efficiency according to image characteristics, or, if necessary, the coding unit may be recursively divided into coding units of lower depths, and a coding unit of an optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be divided or partitioned from the final coding unit.The prediction unit is a unit of sample prediction, or the transform unit is a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0044] The term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample generally represents a pixel or a pixel value, and may represent only a pixel / pixel value of a luma component, or only a pixel / pixel value of a chroma component. A sample may be used as a term corresponding to one picture (or image) for a pixel or a pel.

[0045] The encoding apparatus 200 may generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from the inter prediction unit 221 or the intra prediction unit 222 from an input image signal (original block, original sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, as shown in the figure, a unit in the encoder 200 that subtracts the prediction signal (predicted block, prediction sample array) from the input image signal (original block, original sample array) is also referred to as the subtraction unit 231. The prediction unit may perform prediction on a current block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied on a current block or CU basis. The prediction unit may generate various information related to prediction, such as prediction mode information, and transmit the information to the entropy encoding unit 240, as will be described later in the description of each prediction mode. The prediction information can be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0046] The intra prediction unit 222 may predict the current block by referring to samples in the current picture. The referenced samples may be located adjacent to or distant from the current block depending on the prediction mode. In intra prediction, prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, DC mode and planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the granularity of the prediction direction. However, this is merely an example, and more or less directional prediction modes may be used depending on the settings. The intra prediction unit 222 may also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0047] The inter prediction unit 221 may derive a predicted block for a current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. To reduce the amount of motion information transmitted in inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on an inter prediction direction (e.g., L0 prediction, L1 prediction, or Bi prediction). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may also be called a collocated reference block, a collocated CU (colCU), or the like, and the reference picture including the temporal neighboring block may also be called a collocated picture (colPic). For example, the inter predictor 221 may construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive a motion vector and / or a reference picture index for the current block. Inter prediction may be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter predictor 221 may use motion information of neighboring blocks as motion information for the current block. In the case of skip mode, unlike in merge mode, a residual signal is not transmitted.In the case of the Motion Vector Prediction (MVP) mode, the motion vector of the neighboring block is used as a motion vector predictor, and the motion vector difference can be signaled to indicate the motion vector of the current block.

[0048] The predictor 220 may generate a prediction signal based on various prediction methods, which will be described later. For example, the predictor may apply intra prediction or inter prediction for prediction of a block, or may simultaneously apply intra prediction and inter prediction. This is also called combined inter and intra prediction (CIIP). The predictor may also use an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content image / video coding, such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but may be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may utilize at least one of the inter prediction techniques described herein. The palette mode may be considered an example of intra coding or intra prediction. When the palette mode is applied, sample values ​​within the picture may be signaled based on information about a palette table and a palette index.

[0049] The prediction signal generated by the prediction unit (including the inter prediction unit 221 and / or the intra prediction unit 222) may be used to generate a reconstructed signal or a residual signal. The transform unit 232 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). Here, GBT refers to a transform obtained from a graph representing relationship information between pixels. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. The transform process may be applied to pixel blocks having the same square size or to blocks of variable sizes that are not square.

[0050] The quantization unit 233 quantizes the transform coefficients and transmits them to the entropy encoding unit 240. The entropy encoding unit 240 encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. The information about the quantized transform coefficients is also called residual information. The quantization unit 233 can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit 240 can perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoding unit 240 can also encode information required for video / image restoration (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in Network Abstraction Layer (NAL) units. The video / image information may further include information on various parameter sets, such as an Adaptation Parameter Set (APS), a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), or a Video Parameter Set (VPS). The video / image information may also include general constraint information. In this document, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded through the above-described encoding procedure and included in the bitstream.The bitstream can be transmitted via a network or stored in a digital storage medium. Here, the network can include a broadcast network and / or a communication network, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) that transmits the signal output from the entropy encoding unit 240 and / or a storage unit (not shown) that stores the signal can be configured as an internal / external element of the encoding device 200, or the transmitter can be included in the entropy encoding unit 240.

[0051] The quantized transform coefficients output from the quantizer 233 may be used to generate a prediction signal. For example, a residual signal (residual block or residual samples) may be reconstructed by applying inverse quantization and inverse transform to the quantized transform coefficients via the inverse quantizer 234 and the inverse transformer 235. The adder 250 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding a prediction signal output from the inter predictor 221 or the intra predictor 222 to the reconstructed residual signal. When there is no residual for the current block, such as when a skip mode is applied, a predicted block may be used as the reconstructed block. The adder 250 may also be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next current block in the current picture, or may be used for inter prediction of the next picture after filtering, as described below.

[0052] Meanwhile, Luma Mapping with Chroma Scaling (LMCS) may be applied during picture encoding and / or restoration.

[0053] The filtering unit 260 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 260 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may store the modified reconstructed picture in the memory 270, specifically, in the DPB of the memory 270. The various filtering methods may include, for example, deblock filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc. The filtering unit 260 may generate various information for filtering and transmit it to the entropy encoding unit 240, as will be described later in connection with each filtering method. The filtering information may be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0054] The modified reconstructed picture transmitted to the memory 270 can be used as a reference picture in the inter prediction unit 221. When inter prediction is applied through this, the encoding device can avoid prediction mismatch between the encoding device 200 and the decoding device, and can also improve coding efficiency.

[0055] The memory 270DPB can store the modified reconstructed picture for use as a reference picture in the inter predictor 221. The memory 270 can store motion information of blocks in the current picture from which motion information is derived (or encoded) and / or motion information of blocks in already reconstructed pictures. The stored motion information can be transmitted to the inter predictor 221 to be used as motion information of spatially neighboring blocks or temporally neighboring blocks. The memory 270 can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra predictor 222.

[0056] FIG. 3 is a diagram illustrating the configuration of a video / image decoding device to which the embodiments of this document can be applied.

[0057] As shown in FIG. 3, the decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter predictor 331 and an intra predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. Depending on the embodiment, the entropy decoder 310, the residual processor 320, the predictor 330, the adder 340, and the filter 350 may be configured as a single hardware component (e.g., a decoder chipset or processor). In addition, the memory 360 may include a decoded picture buffer (DPB) or may be configured as a digital storage medium. The above hardware components may further include a memory 360 as an internal / external component.

[0058] When a bitstream including video / image information is input, the decoding device 300 can reconstruct an image corresponding to the process by which the video / image information was processed by the encoding device of FIG. 2. For example, the decoding device 300 can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device 300 can perform decoding using the processing units applied by the encoding device. Accordingly, the processing unit for decoding is, for example, a coding unit, and the coding unit can be divided from a coding tree unit or a maximal coding unit according to a quadtree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units can be derived from the coding unit. The reconstructed image signal decoded and output by the decoding device 300 can be reproduced via a playback device.

[0059] The decoding device 300 may receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal may be decoded via the entropy decoding unit 310. For example, the entropy decoding unit 310 may parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). The video / image information may also include general constraint information. The decoding device may also decode pictures based on the information on the parameter sets and / or the general constraint information. Signaling / received information and / or syntax elements, which will be described later in this document, may be decoded and obtained from the bitstream via the decoding procedure. For example, the entropy decoding unit 310 may decode information in a bitstream based on a coding method such as Exponential-Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients for residuals. More specifically, the CABAC entropy decoding method receives BINs corresponding to each syntax element in the bitstream, determines a context model using information on the syntax element to be decoded and decode information on adjacent and current blocks or information on symbols / BINs decoded in previous steps, predicts the occurrence probability of BINs based on the determined context model, and performs arithmetic decoding of the BINs to generate symbols corresponding to the values ​​of each syntax element.In this case, after determining a context model, the CABAC entropy decoding method may update the context model using information about the decoded symbol / BIN for the context model of the next symbol / BIN. Prediction-related information from the information decoded by the entropy decoding unit 310 is provided to a prediction unit (inter prediction unit 332 and intra prediction unit 331), and residual values ​​entropy-decoded by the entropy decoding unit 310, i.e., quantized transform coefficients and related parameter information, may be input to the residual processing unit 320. The residual processing unit 320 may derive a residual signal (residual block, residual sample, residual sample array). Furthermore, filtering-related information from the information decoded by the entropy decoding unit 310 may be provided to the filtering unit 350. Meanwhile, a receiving unit (not shown) that receives a signal output from the encoding device may be further configured as an internal / external element of the decoding device 300, or the receiving unit may be a component of the entropy decoding unit 310. Meanwhile, the decoding device according to this document may also be referred to as a video / image / picture decoding device, and the decoding device may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit 310, and the sample decoder may include at least one of the inverse quantization unit 321, the inverse transform unit 322, the addition unit 340, the filtering unit 350, the memory 360, the inter prediction unit 332, and the intra prediction unit 331.

[0060] The inverse quantization unit 321 may inverse quantize the quantized transform coefficients to output transform coefficients. The inverse quantization unit 321 may rearrange the quantized transform coefficients in a two-dimensional block format. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit 321 may perform inverse quantization on the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0061] The inverse transform unit 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0062] The prediction unit may perform prediction on the current block and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on the prediction information output from the entropy decoding unit 310, and may determine a specific intra / inter prediction mode.

[0063] The predictor 330 may generate a prediction signal based on various prediction methods, which will be described later. For example, the predictor may apply intra prediction or inter prediction for predicting a block, or may simultaneously apply intra prediction and inter prediction. This is also called combined inter and intra prediction (CIIP). The predictor may also use an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content image / video coding, such as games, such as screen content coding (SCC). IBC basically performs prediction within a current picture, but may be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may utilize at least one of the inter prediction techniques described herein. Palette mode may be considered an example of intra coding or intra prediction. When the palette mode is applied, information about the palette table and palette index can be signaled in the video / image information.

[0064] The intra prediction unit 331 may predict the current block by referring to samples in the current picture. The referenced samples may be located adjacent to or distant from the current block depending on the prediction mode. In intra prediction, prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 331 may also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0065] The inter prediction unit 332 may derive a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. To reduce the amount of motion information transmitted in inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. For example, the inter prediction unit 332 may construct a motion information candidate list based on the neighboring blocks and derive a motion vector and / or a reference picture index for the current block based on received candidate selection information. Inter prediction may be performed based on various prediction modes, and the prediction information may include information indicating the inter prediction mode for the current block.

[0066] The adder 340 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding a prediction signal (predicted block, predicted sample array) output from a prediction unit (including the inter prediction unit 332 and / or the intra prediction unit 331) to the obtained residual signal. When there is no residual for the current block, such as when a skip mode is applied, the predicted block may be used as the reconstructed block.

[0067] The adder 340 is also referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of a next block to be processed in the current picture, may be output after filtering as described below, or may be used for inter prediction of a next picture.

[0068] Meanwhile, Luma Mapping with Chroma Scaling (LMCS) can be applied during the picture decoding process.

[0069] The filtering unit 350 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 350 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may transmit the modified reconstructed picture to the memory 360, specifically, to the DPB of the memory 360. The various filtering methods may include, for example, deblock filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc.

[0070] The (modified) reconstructed picture stored in the DPB of the memory 360 can be used as a reference picture in the inter predictor 332. The memory 360 can store motion information of a block from which motion information in the current picture is derived (or decoded) and / or motion information of a block in an already reconstructed picture. The stored motion information can be transmitted to the inter predictor 332 to be used as motion information of a spatially neighboring block or a temporally neighboring block. The memory 360 can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra predictor 331.

[0071] In this specification, the embodiments described for the filtering unit 260, inter prediction unit 221, and intra prediction unit 222 of the encoding device 200 can also be applied identically or correspondingly to the filtering unit 350, inter prediction unit 332, and intra prediction unit 331 of the decoding device 300, respectively.

[0072] As described above, prediction is performed to improve compression efficiency when performing video coding. Accordingly, a predicted block including predicted samples for a current block, which is a block to be coded, can be generated. Here, the predicted block includes predicted samples in the spatial domain (or pixel domain). The predicted block is derived in the same way by an encoding device and a decoding device. The encoding device can improve image coding efficiency by signaling to a decoding device information (residual information) regarding the residual between the original block and the predicted block, rather than the original sample values ​​of the original block themselves. The decoding device can derive a residual block including residual samples based on the residual information, combine the residual block with the predicted block to generate a reconstructed block including reconstructed samples, and generate a reconstructed picture including the reconstructed block.

[0073] The residual information may be generated through a transform and quantization procedure. For example, an encoding device may derive a residual block between the original block and the predicted block, perform a transform procedure on residual samples (residual sample array) included in the residual block to derive transform coefficients, and perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, and then signal the related residual information to a decoding device (via a bitstream). Here, the residual information may include information such as value information, position information, transform technique, transform kernel, and quantization parameter of the quantized transform coefficients. The decoding device may perform an inverse quantization / inverse transform procedure based on the residual information to derive residual samples (or a residual block). The decoding device may generate a reconstructed picture based on the predicted block and the residual block. The encoding device may also derive a residual block by inverse quantizing / inverse transforming the quantized transform coefficients for reference for inter-prediction of a future picture, and generate a reconstructed picture based on the residual block.

[0074] When inter prediction is applied, a prediction unit of an encoding device / decoding device may perform inter prediction on a block-by-block basis to derive predicted samples. Inter prediction may refer to a prediction derived in a manner that is dependent on data elements (e.g., sample values ​​or motion information) of picture(s) other than the current picture. When inter prediction is applied to a current block, a predicted block (prediction sample array) for the current block may be derived based on a reference block (reference sample array) identified by a motion vector in a reference picture indicated by a reference picture index. In this case, to reduce the amount of motion information transmitted in the inter prediction mode, motion information of the current block may be predicted on a block-by-block, sub-block-by-sample basis based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction type information (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). When inter-prediction is applied, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may also be called a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block is also called a collocated picture (colPic).For example, a motion information candidate list may be constructed based on neighboring blocks of a current block, and flag or index information indicating which candidate is selected (used) to derive a motion vector and / or a reference picture index for the current block may be signaled. Inter prediction may be performed based on various prediction modes. For example, in skip mode and (normal) merge mode, the motion information of the current block is the same as that of a selected neighboring block. Unlike merge mode, in skip mode, no residual signal is transmitted. In motion vector prediction (MVP) mode, the motion vector of a selected neighboring block may be used as a motion vector predictor, and a motion vector difference may be signaled. In this case, the motion vector of the current block may be derived using the sum of the motion vector predictor and the motion vector difference.

[0075] A video / image encoding procedure based on inter prediction may generally include, for example, the following.

[0076] FIG. 4 shows an example of an inter-prediction based video / image encoding method.

[0077] The encoding apparatus performs inter prediction on a current block (S400). The encoding apparatus may derive an inter prediction mode and motion information for the current block and generate a predicted sample for the current block. Here, the inter prediction mode determination, motion information derivation, and predicted sample generation procedures may be performed simultaneously, or one procedure may be performed before the other procedures. For example, the inter prediction unit of the encoding apparatus may include a prediction mode determination unit, a motion information derivation unit, and a predicted sample derivation unit, in which the prediction mode determination unit may determine a prediction mode for the current block, the motion information derivation unit may derive motion information for the current block, and the predicted sample derivation unit may derive a predicted sample for the current block. For example, the inter prediction unit of the encoding apparatus may search for a block similar to the current block within a certain region (search region) of a reference picture through motion estimation and derive a reference block whose difference from the current block is minimum or equal to or less than a certain criterion. Based on this, a reference picture index indicating a reference picture in which the reference block is located can be derived, and a motion vector can be derived based on a position difference between the reference block and the current block. The encoding apparatus can determine a mode to be applied to the current block from various prediction modes. The encoding apparatus can compare RD costs for the various prediction modes to determine an optimal prediction mode for the current block.

[0078] For example, when a skip mode or a merge mode is applied to the current block, the encoding apparatus may construct a merge candidate list (described later) and derive a reference block, among reference blocks indicated by merge candidates included in the merge candidate list, whose difference between the current block and the current block is minimum or equal to or less than a certain criterion. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information indicating the selected merge candidate may be generated and signaled to the decoding apparatus. Motion information of the current block may be derived using motion information of the selected merge candidate.

[0079] As another example, when the (A)MVP mode is applied to the current block, the encoding apparatus may construct an (A)MVP candidate list (described later) and use a motion vector of an MVP (motion vector predictor) candidate selected from the MVP candidates included in the (A)MVP candidate list as the MVP of the current block. In this case, for example, a motion vector pointing to a reference block derived by the motion estimation described above may be used as the motion vector of the current block, and the MVP candidate having the smallest difference from the motion vector of the current block among the MVP candidates may become the selected MVP candidate. A motion vector difference (MVD), which is the difference obtained by subtracting the MVP from the motion vector of the current block, may be derived. In this case, information regarding the MVD may be signaled to the decoding apparatus. Furthermore, when the (A)MVP mode is applied, the value of the reference picture index may be configured as reference picture index information and separately signaled to the decoding apparatus.

[0080] The encoding apparatus may derive residual samples based on the predicted samples (S410) by comparing the original samples of the current block with the predicted samples.

[0081] The encoding apparatus encodes image information including prediction information and residual information (S420). The encoding apparatus can output the encoded image information in the form of a bitstream. The prediction information is information related to the prediction procedure and can include prediction mode information (e.g., a skip flag, a merge flag, or a mode index) and information on motion information. The information on the motion information can include candidate selection information (e.g., a merge index, e.g., an MVP flag or an MVP index) for deriving a motion vector. The information on the motion information can also include information on the above-mentioned MVD and / or reference picture index information. The information on the motion information can also include information indicating whether L0 prediction, L1 prediction, or bi-prediction is applied. The residual information is information on the residual samples. The residual information can also include information on quantized transform coefficients for the residual samples.

[0082] The output bitstream can be stored on a (digital) storage medium and transmitted to the decoding device, or can be transmitted to the decoding device via a network.

[0083] Meanwhile, as described above, the encoding apparatus can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on the reference samples and the residual samples. This is because the encoding apparatus derives the same prediction result as that performed by the decoding apparatus, thereby improving coding efficiency. Therefore, the encoding apparatus can store the reconstructed picture (or reconstructed samples, reconstructed blocks) in a memory and use it as a reference picture for inter prediction. As described above, an in-loop filtering procedure or the like can be further applied to the reconstructed picture.

[0084] A video / image decoding procedure based on inter prediction may generally include, for example, the following.

[0085] FIG. 5 shows an example of an inter-prediction based video / image decoding method.

[0086] As shown in Figure 5, the decoding device may perform operations corresponding to those performed by the encoding device. The decoding device may perform prediction on the current block based on the received prediction information to derive a predicted sample.

[0087] Specifically, the decoding device may determine a prediction mode for the current block based on received prediction information (S500). The decoding device may determine which inter prediction mode is applied to the current block based on prediction mode information in the prediction information.

[0088] For example, it may determine whether the merge mode or the (A)MVP mode is applied to the current block based on the merge flag. Alternatively, it may select one from various inter prediction mode candidates based on the mode index. The inter prediction mode candidates may include skip mode, merge mode, and / or (A)MVP mode, or various inter prediction modes described below.

[0089] The decoding apparatus derives motion information of the current block based on the determined inter prediction mode (S510). For example, when a skip mode or a merge mode is applied to the current block, the decoding apparatus may construct a merge candidate list (described below) and select one merge candidate from among the merge candidates included in the merge candidate list. The selection may be performed based on the selection information (merge index) described above. Motion information of the selected merge candidate may be used to derive motion information of the current block. The motion information of the selected merge candidate may be used as motion information of the current block.

[0090] As another example, when the (A)MVP mode is applied to the current block, the decoding apparatus may construct an (A)MVP candidate list (described later) and use a motion vector of an MVP (motion vector predictor) candidate selected from among the MVP candidates included in the (A)MVP candidate list as the MVP of the current block. The selection may be performed based on the selection information (MVP flag or MVP index). In this case, the MVD of the current block may be derived based on information related to the MVD, and the motion vector of the current block may be derived based on the MVP of the current block and the MVD. Furthermore, the decoding apparatus may derive a reference picture index of the current block based on the reference picture index information. A picture pointed to by the reference picture index in the reference picture list for the current block may be derived as a reference picture referenced for inter-prediction of the current block.

[0091] On the other hand, as will be described later, the motion information of the current block may be derived without constructing a candidate list, and in this case, the motion information of the current block may be derived according to a procedure disclosed in the prediction mode section, which will be described later. In this case, the candidate list construction as described above may be omitted.

[0092] The decoding apparatus may generate predictive samples for the current block based on the motion information of the current block (S520). In this case, the reference picture may be derived based on a reference picture index of the current block, and the predictive samples of the current block may be derived using samples of a reference block pointed to in the reference picture by the motion vector of the current block. In this case, as described below, a predictive sample filtering procedure may further be performed on all or some of the predictive samples of the current block, depending on the case.

[0093] For example, the inter-prediction unit of the decoding device may include a prediction mode determination unit, a motion information derivation unit, and a prediction sample derivation unit, and may determine a prediction mode for the current block based on the prediction mode information received by the prediction mode determination unit, derive motion information (such as a motion vector and / or a reference picture index) for the current block based on information regarding the motion information received by the motion information derivation unit, and derive a prediction sample for the current block by the prediction sample derivation unit.

[0094] The decoding apparatus generates residual samples for the current block based on the received residual information (S530). The decoding apparatus generates reconstructed samples for the current block based on the predicted samples and the residual samples, and can generate a reconstructed picture based on the reconstructed samples (S540). Thereafter, as described above, an in-loop filtering procedure can be further applied to the reconstructed picture.

[0095] FIG. 6 exemplarily illustrates an inter prediction procedure.

[0096] As shown in Figure 6, as described above, the inter prediction procedure may include an inter prediction mode determination step, a motion information derivation step according to the determined prediction mode, and a prediction (prediction sample generation) step based on the derived motion information. As described above, the inter prediction procedure may be performed by an encoding device and a decoding device. In this document, a coding device may include an encoding device and / or a decoding device.

[0097] As shown in FIG. 6, a coding apparatus determines an inter prediction mode for a current block (S600). Various inter prediction modes can be used for predicting a current block in a picture. For example, merge mode, skip mode, motion vector prediction (MVP) mode, affine mode, sub-block merge mode, merge with MVD (MMVD) mode, etc. can be used. Decoder-side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, bi-prediction with CU-level weight (BCW), bi-directional optical flow (BDOF), etc. can be used as additional modes or instead of them. Affine mode is also called affine motion prediction mode. MVP mode is also called advanced motion vector prediction mode. In this document, some modes and / or motion information candidates derived by some modes may be included as one of the motion information-related candidates of other modes. For example, an HMVP candidate may be added as a merge candidate in the merge / skip mode, or as an MVP candidate in the MVP mode. When the HMVP candidate is used as a motion information candidate in the merge mode or skip mode, the HMVP candidate is also called an HMVP merge candidate.

[0098] Prediction mode information indicating the inter prediction mode of the current block may be signaled from the encoding device to the decoding device. The prediction mode information may be included in a bitstream and received by the decoding device. The prediction mode information may include index information indicating one of a plurality of candidate modes. Alternatively, the inter prediction mode may be indicated through hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags. For example, a skip flag may be signaled to indicate whether the skip mode can be applied, a merge flag may be signaled to indicate whether the merge mode can be applied when the skip mode is not applied, or an MVP mode may be applied when the merge mode is not applied, or a flag for additional division may be further signaled. The affine mode may be signaled in an independent mode or in a mode dependent on the merge mode or MVP mode. For example, the affine mode may include an affine merge mode and an affine MVP mode.

[0099] The coding apparatus derives motion information for the current block (S610). The motion information may be derived based on the inter prediction mode.

[0100] A coding apparatus may perform inter-prediction using motion information of a current block. An encoding apparatus may derive optimal motion information for a current block through a motion estimation procedure. For example, the encoding apparatus may use an original block in an original picture for the current block to search for a similar reference block with high correlation in fractional pixel units within a predetermined search range in the reference picture, thereby deriving motion information. Block similarity may be derived based on a phase-based sample value difference. For example, block similarity may be calculated based on the SAD between the current block (or a template of the current block) and a reference block (or a template of the reference block). In this case, motion information may be derived based on the reference block with the smallest SAD within the search range. The derived motion information may be signaled to a decoding apparatus in various ways based on the inter-prediction mode.

[0101] The coding apparatus performs inter prediction based on motion information for the current block (S620). The coding apparatus may derive predictive sample(s) for the current block based on the motion information. The current block including the predictive sample(s) is also referred to as a predicted block.

[0102] Meanwhile, in inter prediction, according to the conventional merge or AMVP mode, a method of reducing the amount of motion information is used by using motion vectors of spatially / temporally neighboring blocks of the current block as motion information candidates. For example, the neighboring blocks used to derive motion information candidates of the current block may include the lower left corner neighboring block, left neighboring block, upper right corner neighboring block, upper neighboring block, and upper left corner neighboring block of the current block.

[0103] FIG. 7 exemplarily shows spatially neighboring blocks used for motion information candidate derivation in conventional merge or AMVP mode.

[0104] Basically, the spatially neighboring blocks are limited to blocks adjacent to the current block. This is to improve hardware feasibility, because deriving information on blocks far from the current block causes problems such as an increase in line buffer space. However, using motion information on non-neighboring blocks to derive motion information candidates for the current block improves performance by enabling the construction of various candidates. A history-based motion vector prediction (HMVP) method can be used to use motion information on non-neighboring blocks without increasing the line buffer space. In this document, HMVP may refer to history-based motion vector prediction or history-based motion vector predictor. According to this document, inter prediction can be performed efficiently using HMVP and parallel processing can be supported. For example, embodiments of this document propose various methods for managing history buffers for parallel processing, and parallel processing can be supported based on these methods. However, supporting parallel processing does not necessarily mean that parallel processing should be performed; rather, a coding device may or may not perform parallel processing depending on hardware performance and service type. For example, if the coding device has a multi-core processor, the coding device may process some of the slices, bricks, and / or tiles in parallel, whereas if the coding device has a single-core processor or a multi-core processor, the coding device may perform sequential processing while reducing computational and memory burdens.

[0105] An HMVP candidate according to the above-described HMVP method may include motion information of a previously coded block. For example, motion information of a previously coded block in a current picture according to a block coding order is not considered as motion information of the current block if the previously coded block is not adjacent to the current block. However, an HMVP candidate may be considered as a motion information candidate (e.g., a merge candidate or an MVP candidate) for the current block without considering whether the previously coded block is adjacent to the current block. In this case, multiple HMVP candidates may be stored in a buffer. For example, if a merge mode is applied to the current block, an HMVP candidate (HMVP merge candidate) may be added to a merge candidate list. In this case, the HMVP candidate may be added next to the spatial merge candidate and the temporal merge candidate included in the merge candidate list.

[0106] According to the HMVP method, motion information of previously coded blocks can be stored in the form of a table and used as motion information candidates (e.g., merge candidates) for the current block. A table (or buffer, list) containing multiple HMVP candidates can be maintained during the encoding / decoding procedure. The table (or buffer, list) is also referred to as an HMVP table (or buffer, list). According to an embodiment of this document, the table (or buffer, list) can be initialized when encountering a new slice. Alternatively, according to an embodiment of this document, the table (or buffer, list) can be initialized when encountering a new CTU row. When the table is initialized, the number of HMVP candidates included in the table can be set to 0. The size of the table (or buffer, list) can be fixed to a specific value (e.g., 5). For example, if there is an inter-coded block, the associated motion information can be added as a new HMVP candidate in the last entry of the table. The (HMVP) table is also called an (HMVP) buffer or an (HMVP) list.

[0107] 8 illustrates an example of an HMVP candidate-based decoding procedure, where the HMVP candidate-based decoding procedure may include an HMVP candidate-based inter-prediction procedure.

[0108] As shown in FIG. 8, the decoding device loads an HMVP table including one or more HMVP candidates and decodes a block based on at least one of the HMVP candidates. Specifically, for example, the decoding device may derive motion information of a current block based on at least one of the HMVP candidates, perform inter prediction on the current block based on the motion information, and derive a predicted block (including a predicted sample). As described above, a reconstructed block may be generated based on the predicted block. The derived motion information of the current block may be updated in the table. In this case, the motion information may be added as a new HMVP candidate as the last entry in the table. If the number of HMVP candidates already included in the table is the same as the size of the table, the first candidate in the table may be deleted, and the derived motion information may be added as a new HMVP candidate as the last entry in the table.

[0109] FIG. 9 exemplarily shows an HMVP table update according to the FIFO rule, and FIG. 10 exemplarily shows an HMVP table update according to the limited FIFO rule.

[0110] A first-in, first-out (FIFO) rule may be applied to the table. For example, if the table size S is 16, this indicates that 16 HMVP candidates may be included in the table. If more than 16 HMVP candidates are generated from previously coded blocks, the FIFO rule may be applied, thereby allowing the table to include up to 16 of the most recently coded motion information candidates. In this case, as shown in FIG. 9, the FIFO rule may be applied to remove the oldest HMVP candidate and add a new HMVP candidate.

[0111] Meanwhile, to further improve coding efficiency, a limited FIFO rule may be applied as shown in Figure 10. As shown in Figure 10, when an HMVP candidate is inserted into the table, a redundancy check may be applied first. This may determine whether an HMVP candidate with the same motion information already exists in the table. If an HMVP candidate with the same motion information exists in the table, the HMVP candidate with the same motion information is removed from the table, and the HMVP candidates after the removed HMVP candidate are moved by one interval (i.e., each index -1) before a new HMVP candidate is inserted.

[0112] As described above, HMVP candidates can be used in the merge candidate list construction procedure. In this case, for example, all HMVP candidates that can be inserted from the last entry to the first entry in the table can be inserted after the spatial merge candidate and the temporal merge candidate. In this case, a pruning check can be applied to the HMVP candidates. The maximum number of merge candidates allowed can be signaled, and if the total number of available merge candidates reaches the maximum number of merge candidates, the merge candidate list construction procedure can be terminated.

[0113] Similarly, HMVP candidates can also be used in the (A) MVP candidate list construction procedure. In this case, the motion vectors of the last k HMVP candidates in the HMVP table can be added next to the TMVP candidates that constitute the MVP candidate list. In this case, for example, an HMVP candidate having the same reference picture as the MVP target reference picture can be used to construct the MVP candidate list. Here, the MVP target reference picture can indicate a reference picture for inter-prediction of the current block to which the MVP mode is applied. In this case, a pruning check can be applied to the HMVP candidate. For example, k is 4. However, this is merely an example, and k can have various values ​​such as 1, 2, 3, and 4.

[0114] On the other hand, if the total number of merge candidates is equal to or greater than 15, a truncated unary plus fixed length (with 3 bits) binarization method can be applied for merge index coding as shown in Table 1 below.

[0115] [Table 1]

[0116] The above table assumes that Nmrg=15, where Nmrg indicates the total number of merge candidates.

[0117] Meanwhile, when developing a solution that applies a video codec, parallel processing can be supported in image / video coding for implementation optimization.

[0118] FIG. 11 exemplarily shows WPP (Wavefront Parallel Processing), which is one of the techniques for parallel processing.

[0119] As shown in FIG. 11, when WPP is applied, parallel processing can be performed in units of CTU rows. In this case, when coding (encoding / decoding) a block indicated by an X, there is a dependency on the position indicated by the arrow. Therefore, it is necessary to wait for the completion of coding of the CTU to the upper right of the block to be currently coded. In addition, when WPP is applied, initialization of the CABAC probability table (or context information) can be performed in units of slices. To perform parallel processing including entropy encoding / decoding, the CABAC probability table (or context information) must be initialized in units of CTU rows. WPP can be considered a technology proposed for determining an efficient initialization position. When WPP is applied, each LCT row can be called a substream, and parallel processing can be supported if the coding device has multiple processing cores. For example, when WPP is applied, if three processing cores process decoding in parallel, the first processing core may decode substream 0, the second processing core may decode substream 1, and the third processing core may decode substream 2. When WPP is applied, after coding is performed (progressed) on the nth substream (n is an integer), coding of the second CTU or LCU of the nth substream may be completed, and then coding of the n+1th substream may be performed. For example, in the case of entropy coding, once entropy coding of the second LCU of the nth substream is completed, the first LCU of the n+1th substream may be entropy coded based on context information for the second LCU of the nth substream. In this case, the number of substreams within a slice may be the same as the number of LCU rows. In addition, the number of substreams within a slice may be the same as the number of entry points. In this case, the number of entry points may be determined by the number of entry point offsets.For example, the number of entry points may be one greater than the number of entry point offsets. Information regarding the number of entry point offsets and / or information regarding the offset values ​​may be included in the above-mentioned video / image information and encoded, and may be signaled to a decoding device via a bitstream. Meanwhile, if a coding device has one processing core, it may perform coding processing on a sub-stream basis, thereby reducing memory load and coding dependency.

[0120] The HMVP method described above stores motion information derived from the coding procedure for each block as candidates by the size of a predetermined buffer (HMVP table). In this case, as many candidates as the number of buffers can be filled, as shown without additional conditions in FIG. 9, or candidates can be filled without overlapping by checking whether newly added candidates overlap with candidates already in the buffer (HMVP table). This allows for a variety of candidates to be configured. However, when developing a solution using a video codec, it is generally impossible to know when the buffer will be filled with HMVP candidates, making it impossible to implement parallel processing with or without WPP.

[0121] FIG. 12 exemplarily illustrates the problems encountered when applying the general HMVP method in consideration of parallel processing.

[0122] As shown in FIG. 12, when parallelization is performed for each CTU row, as in WPP, a dependency problem of the HMVP buffer may occur. For example, the HMVP buffer for the first CTU in the Nth (N>=1)th CTU row is not filled until the coding (encoding / decoding) of a block in the N-1th CTU row, for example, the block in the last CTU in the N-1th CTU row, is completed. That is, when parallel processing is applied under the current structure, the decoding device cannot know whether the HMVP candidate stored in the current HMVP buffer matches the HMVP buffer used for decoding the current (target) block. This is because a difference may occur between the HMVP buffer derived at the time of coding of the current block when sequential processing is applied and the HMVP buffer derived at the time of coding of the current block when parallel processing is applied.

[0123] In one embodiment of this document, in order to solve the above-mentioned problems, when applying HMVP, parallel processing is supported by initializing a history management buffer (HMVP buffer).

[0124] FIG. 13 exemplarily illustrates a method for initializing a history management buffer (HMVP buffer) according to one embodiment of this document.

[0125] As shown in FIG. 13, an HMVP buffer may be initialized for each first CTU in a CTU row. That is, when coding the first CTU in a CTU row, the HMVP buffer may be initialized so that the number of HMVP candidates included in the HMVP buffer is set to 0. By initializing the HMVP buffer for each CTU row as described above, HMVP candidates derived during the coding of CTUs located to the left of the current block may be used without restriction even when parallel processing is supported. In this case, for example, if a current CU, which is the current block, is located in the first CTU in a CTU row and the current CU corresponds to the first CU in the first CTU, the number of HMVP candidates included in the HMVP buffer is 0. Also, for example, if a CU coded before the current CU in the CTU row is coded in inter mode, an HMVP candidate may be derived based on motion information of the previously coded CU and included in the HMVP buffer.

[0126] FIG. 14 illustrates an exemplary HMVP buffer management method according to one embodiment.

[0127] As shown in FIG. 14, the HMVP buffer can be initialized in units of slices, and it can be determined whether the CTU to be coded (current CTU) is the first CTU in each CTU row for each CTU in the slice. In FIG. 14, as an example, it is described that if (ctu_idx % Num) is 0, it is determined to be the first CTU. Here, Num means the number of CTUs in each CTU row. As another example, when the above-mentioned brick concept is used, if (ctu_idx_in_brick % BrickWidth) is 0, it can be determined to be the first CTU in the CTU row (in the corresponding brick). Here, ctu_idx_in_brick indicates the index of the corresponding CTU in the brick, and BrickWidth indicates the width of the corresponding brick in CTU units. In other words, BrickWidth can indicate the number of CTU columns in the corresponding brick. If the current CTU is the first CTU in the CTU row, the HMVP buffer is initialized (i.e., the number of candidates in the HMVP buffer is set to 0); otherwise, the HMVP buffer is maintained. Then, a prediction process for each CU in the CTU (e.g., based on merge or MVP mode) is performed, and the candidates stored in the HMVP buffer can be included as motion information candidates for merge mode or MVP mode (e.g., merge or MVP candidates). The motion information of the target block (current block) derived in the inter prediction process based on merge mode or MVP mode is stored (updated) in the HMVP buffer as a new HMVP candidate. In this case, the above-described overlap check process can also be performed. Then, the above-described procedure can be repeated for the CU and CTU.

[0128] As another example, when applying HMVP, the dependency on a CTU basis can be removed by initializing the HMVP buffer for each CTU.

[0129] FIG. 15 illustrates an exemplary HMVP buffer management method according to another embodiment.

[0130] As shown in Figure 15, HMVP buffer initialization can be performed for each CTU without determining whether the current CTU is the first CTU in each CTU row. In this case, since the HMVP buffer is initialized on a CTU-by-CTU basis, motion information of blocks present in the CTU is stored in the HMVP table. In this case, HMVP candidates can be derived based on motion information of blocks (e.g., CUs) within the same CTU, and HMVP buffer initialization is possible without determining whether the current CTU is the first CTU in each CTU row, as described below.

[0131] As described above, the HMVP buffer can be initialized in units of slices, thereby making it possible to use motion vectors of blocks spatially separated from the current block. However, in this case, parallel processing cannot be supported within a slice, and therefore, in the above-described embodiments, a method of initializing the buffer in units of CTU rows or CTUs has been proposed. That is, according to the embodiments of this document, the HMVP buffer can be initialized in units of slices, and within a slice, it can be initialized in units of CTU rows.

[0132] On the other hand, when coding (encoding / decoding) a picture, the picture can be divided into slices and / or tiles. For example, the picture can be divided into slices in consideration of error resilience, or the picture can be divided into tiles in order to encode / decode a partial area of ​​the picture. When a picture is divided into multiple tiles, if an HMVP management buffer is applied, initialization is performed on a CTU row basis within the picture, i.e., initializing the HMVP buffer with the first CTU of each CTU row within the picture, which is not suitable for a tile structure for encoding / decoding a portion of a picture.

[0133] FIG. 16 exemplarily illustrates an HMVP buffer initialization method in a tile structure.

[0134] As shown in Figure 16, in the case of tile 1 and tile 3, the HMVP management buffer is not initialized for each tile, so there is an (HMVP) dependency with tile 0 and tile 2. Therefore, when tiles exist, the HMVP buffer can be initialized in the following way.

[0135] For example, the HMVP buffer can be initialized in units of CTUs. This can be applied without dividing it into tiles, slices, etc.

[0136] As another example, the HMVP buffer can be initialized for the first CTU of each tile.

[0137] FIG. 17 illustrates an example of an HMVP buffer initialization method for the first CTU of a tile according to another embodiment.

[0138] As shown in Figure 17, when coding the first CTU of each tile, the HMVP buffer is initialized. That is, when coding tile 0, HMVP buffer 0 is initialized and used, and when coding tile 1, HMVP buffer 1 is initialized and used.

[0139] As yet another example, the HMVP buffer can be initialized for the first CTU in a row of CTUs in each tile.

[0140] FIG. 18 illustrates an example of an HMVP management buffer initialization method for the first CTU in a row of CTUs in each tile according to yet another embodiment.

[0141] As shown in FIG. 18, an HMVP buffer may be initialized for each CTU row in each tile. For example, an HMVP buffer may be initialized with the first CTU of the first CTU row in tile n, the HMVP buffer may be initialized with the first CTU of the second CTU row in tile n, and the HVMP buffer may be initialized with the first CTU of the third CTU row in tile n. In this case, if the coding device has a multi-core processor, the coding device may initialize and use HMVP buffer 0 for the first CTU row in tile n, HVMP buffer 1 for the second CTU row in tile n, and HMVP buffer 2 for the third CTU row in tile n, thereby supporting parallel processing. On the other hand, if the coding device has a single-core processor, the coding device may initialize and reuse HVMP buffers with the first CTU of each CTU row in each tile according to the coding order.

[0142] On the other hand, the tile division structure and the slice division structure allow tiles and slices to coexist within one picture.

[0143] FIG. 19 shows an example of a structure in which tiles and slices coexist.

[0144] Figure 19 exemplarily shows a case where one picture is divided into four tiles and each tile contains two slices. As shown in Figure 19, there may be cases where one picture contains both slices and tiles, and the HMVP buffer can be initialized as follows:

[0145] For example, the HMVP buffer can be initialized in units of CTUs. This method can be applied regardless of whether the CTU is located in a tile or a slice.

[0146] As another example, the HMVP buffer can be initialized for the first CTU in each tile.

[0147] FIG. 20 shows an example of a method for initializing the HMVP buffer for the first CTU in each tile.

[0148] As shown in Figure 20, the HMVP buffer may be initialized in the first CTU of each tile. Even if there are multiple slices in one tile, HMVP buffer initialization may be performed in the first CTU in the tile.

[0149] As yet another example, HMVP buffer initialization can be performed for each slice in a tile.

[0150] FIG. 21 shows an example of how to initialize the HMVP buffer for each slice in a tile.

[0151] As shown in Figure 21, the HMVP buffer can be initialized in the first CTU of each slice in a tile. Therefore, if there are multiple slices in one tile, HMVP buffer initialization can be performed for each of the multiple slices. In this case, the HMVP buffer initialization can be performed when processing the first CTU of each slice.

[0152] On the other hand, there can be multiple tiles without slices in one picture, or multiple tiles in one slice. In such cases, you can perform HMVP buffer initialization as follows:

[0153] For example, an HMVP buffer may be initialized for each tile group.

[0154] FIG. 22 shows an example of initializing an HMVP buffer for the first CTU of the first tile in a tile group.

[0155] As shown in Figure 22, one picture can be divided into two tile groups, and each tile group (TileGroup0, TileGroup1) can be divided into multiple tiles. In this case, the HMVP buffer can be initialized for the first CTU of the first tile in one tile group.

[0156] As another example, the HMVP buffer may be initialized on a tile-by-tile basis within a tile group.

[0157] FIG. 23 shows an example of initializing an HMVP buffer for the first CTU of each tile in a tile group.

[0158] As shown in Figure 23, one picture can be divided into two tile groups, and each tile group (TileGroup0, TileGroup1) can be divided into multiple tiles. In this case, the HMVP buffer can be initialized for the first CTU of each tile in one tile group.

[0159] As yet another example, an HMVP buffer can be initialized for the CTU rows of each tile in the tile group.

[0160] FIG. 24 shows an example of initializing the HMVP buffer for the CTU rows of each tile in a tile group.

[0161] As shown in Figure 24, one picture can be divided into two tile groups, and each tile group (TileGroup0, TileGroup1) can be divided into multiple tiles. In this case, the HMVP buffer can be initialized with the first CTU in the CTU row of each tile in one tile group.

[0162] Alternatively, in this case, the HMVP management buffer can be initialized in units of CTUs. Naturally, this can be applied regardless of the division into tiles, slices, tile groups, etc.

[0163] 25 and 26 schematically illustrate an example of a video / image encoding method and related components including an inter-prediction method according to one or more embodiments of the present document. The method disclosed in FIG. 25 may be performed by the encoding device disclosed in FIG. 2. Specifically, for example, steps S2500 to S2530 of FIG. 25 may be performed by the prediction unit 220 of the encoding device, step S2540 of FIG. 25 may be performed by the residual processing unit 230 of the encoding device, and step S2550 of FIG. 25 may be performed by the entropy encoding unit 240 of the encoding device. The method disclosed in FIG. 25 may include the embodiments described above in the present document.

[0164] As shown in FIG. 25, the encoding apparatus derives an HMVP buffer for a current block (S2500). The encoding apparatus may perform the HMVP buffer management method described above in the embodiments of this document. For example, the HMVP buffer may be initialized in units of slices, tiles, or tile groups. And / or the HMVP buffer may be initialized in units of CTU rows. In this case, the HMVP buffer may be initialized in units of CTU rows within the slice, tile, or tile group. Here, a tile may refer to a rectangular region, such as a CTU, within a picture. A tile may be specified based on a specific tile row and a specific tile column within the picture. For example, there may be one or more tiles within the current picture. In this case, the HMVP buffer may be initialized with the first CTU of the CTU row including the current block in the current tile. Alternatively, there may be one or more slices within the current picture. In this case, the HMVP buffer may be initialized with the first CTU of the CTU row including the current block in the current slice. Alternatively, there may be one or more tile groups in the current picture, in which case the HMVP buffer may be initialized with the first CTU of the CTU row that includes the current block in the current tile group.

[0165] The encoding device may determine whether the current CTU is the first CTU in the CTU row. In this case, the HMVP buffer may be initialized with the first CTU in the CTU row in which the current CTU including the current block is located. In other words, the HMVP buffer may be initialized when processing the first CTU in the CTU row in which the current CTU including the current block is located. If it is determined that the current CTU including the current block is the first CTU in the CTU row in the current tile, the HMVP buffer may include HMVP candidates derived based on motion information of blocks processed earlier than the current block in the current CTU. If it is determined that the current CTU is not the first CTU in the CTU row in the current tile, the HMVP buffer may include HMVP candidates derived based on motion information of blocks processed earlier than the current block in the CTU row in the current tile. Also, for example, if the current CU, which is the current block, is located in the first CTU of the CTU row in the current tile and the current CU corresponds to the first CU of the first CTU, the number of HMVP candidates included in the HMVP buffer is 0. Also, for example, if a CU coded before the current CU in the CTU row in the current tile (e.g., a CU coded before the current CU in the current CTU and / or a CU in a CTU coded before the current CTU in the current CTU row) is coded in inter mode, an HMVP candidate may be derived based on motion information of the previously coded CU and included in the HMVP buffer.

[0166] If the current picture is divided into multiple tiles, the HMVP buffer may be initialized in units of CTU rows within each tile.

[0167] The HMVP buffer may be initialized in units of CTU rows within a tile or slice. For example, if a specific CTU in the CTU row is not the first CTU in the CTU row in the current picture, but is the first CTU in the CTU row in the current tile or slice, the HMVP buffer may be initialized with the specific CTU.

[0168] When the HMVP buffer is initialized, the number of HMVP candidates included in the HMVP buffer may be set to zero.

[0169] The encoding apparatus constructs a motion information candidate list based on the HMVP buffer (S2510). The HMVP buffer may include HMVP candidates, and the motion information candidate list may be constructed including the HMVP candidates.

[0170] As an example, when a merge mode is applied to the current block, the motion information candidate list may be a merge candidate list. As another example, when an (A)MVP mode is applied to the current block, the motion information candidate list may be an MVP candidate list. When a merge mode is applied to the current block, the HMVP candidate may be added to the merge candidate list if the number of available merge candidates (e.g., including spatial merge candidates and temporal merge candidates) in the merge candidate list for the current block is less than a predetermined maximum number of merge candidates. In this case, the HMVP candidate may be inserted after the spatial candidate and the temporal candidate in the merge candidate list. In other words, the HMVP candidate may be assigned an index value greater than the indexes assigned to the spatial candidate and the temporal candidate in the merge candidate list. When an (A)MVP mode is applied to the current block, the HMVP candidate may be added to the MVP candidate list if the number of available MVP candidates (derived based on spatial and temporal neighboring blocks) in the MVP candidate list for the current block is less than two.

[0171] The encoding apparatus may derive motion information of the current block based on the motion information candidate list (S2520).

[0172] The encoding apparatus may derive motion information of the current block based on the motion information candidate list. For example, when a merge mode or an MVP mode is applied to the current block, the HMVP candidate included in the HMVP buffer may be used as a merge candidate or an MVP candidate. For example, when a merge mode is applied to the current block, the HMVP candidate included in the HMVP buffer may be included as a candidate in a merge candidate list, and the HMVP candidate may be indicated among the candidates included in the merge candidate list based on a merge index. The merge index is prediction-related information and may be included in image / video information, which will be described later. In this case, the HMVP candidate may be assigned an index in the merge candidate list with a lower priority than the spatial merge candidates and temporal merge candidates included in the merge candidate list. That is, the index value assigned to the HMVP candidate may be assigned a higher value than the index values ​​of the spatial merge candidates and temporal merge candidates. As another example, if the MVP mode is applied to the current block, the HMVP candidates included in the HMVP buffer may be included as candidates in a merge candidate list, and the HMVP candidate among the candidates included in the MVP candidate list may be indicated based on an MVP flag (or an MVP index). The MVP flag (or an MVP index) may be prediction-related information and may be included in image / video information, which will be described later.

[0173] The encoding apparatus generates a prediction sample for the current block based on the derived motion information (S2530). The encoding apparatus can perform inter prediction (motion compensation) based on the motion information and derive the prediction sample using a reference sample in a reference picture indicated by the motion information.

[0174] The encoding device generates residual samples based on the predicted samples (S2540). The encoding device can generate residual samples based on the original samples for the current block and the predicted samples for the current block.

[0175] The encoding device derives information about the residual samples based on the residual samples, and encodes image / video information including the information about the residual samples (S2550). The information about the residual samples may be referred to as residual information and may include information about quantized transform coefficients. The encoding device may perform a transform / quantization procedure on the residual samples to derive the quantized transform coefficients.

[0176] The encoded image / video information may be output in the form of a bitstream. The bitstream may be transmitted to a decoding device via a network or a storage medium. The image / video information may further include prediction-related information, which may further include information on various prediction modes (e.g., merge mode, MVP mode, etc.), MVD information, etc.

[0177] Figures 27 and 28 schematically illustrate an example of an image decoding method including an inter prediction method and related components according to an embodiment of the present document. The method disclosed in Figure 27 may be performed by the decoding device disclosed in Figure 3. Specifically, for example, steps S2700 to S2730 in Figure 27 may be performed by the prediction unit 330 of the decoding device, and step S2740 may be performed by the addition unit 340 of the decoding device. The method disclosed in Figure 27 may include the embodiments described above in this document.

[0178] As shown in FIG. 27, the decoding device derives an HMVP buffer for a current block (S2700). The decoding device may perform the HMVP buffer management method described above in the embodiments of this document. For example, the HMVP buffer may be initialized in units of slices, tiles, or tile groups. And / or the HMVP buffer may be initialized in units of CTU rows. In this case, the HMVP buffer may be initialized in units of CTU rows within the slice, tile, or tile group. Here, a tile may indicate a rectangular region, such as a CTU, within a picture. A tile may be specified based on a specific tile row and a specific tile column within the picture. For example, there may be one or more tiles within the current picture. In this case, the HMVP buffer may be initialized with the first CTU of the CTU row including the current block in the current tile. Alternatively, there may be one or more slices within the current picture. In this case, the HMVP buffer may be initialized with the first CTU of the CTU row including the current block in the current slice. Alternatively, there may be one or more tile groups in the current picture, in which case the HMVP buffer may be initialized with the first CTU of the CTU row that includes the current block in the current tile group.

[0179] The decoding device may determine whether the current CTU is the first CTU in the CTU row. In this case, the HMVP buffer may be initialized with the first CTU in the CTU row in which the current CTU including the current block is located. In other words, the HMVP buffer may be initialized when processing the first CTU in the CTU row in which the current CTU including the current block is located. If it is determined that the current CTU including the current block is the first CTU in the CTU row in the current tile, the HMVP buffer may include HMVP candidates derived based on motion information of blocks processed earlier than the current block in the current CTU. If it is determined that the current CTU is not the first CTU in the CTU row in the current tile, the HMVP buffer may include HMVP candidates derived based on motion information of blocks processed earlier than the current block in the CTU row in the current tile. Also, for example, if the current CU, which is the current block, is located in the first CTU of the CTU row in the current tile and the current CU corresponds to the first CU of the first CTU, the number of HMVP candidates included in the HMVP buffer is 0. Also, for example, if a CU coded earlier than the current CU in the CTU row in the current tile (e.g., a CU coded earlier than the current CU in the current CTU and / or a CU in a CTU coded earlier than the current CTU in the current CTU row) is coded in inter mode, an HMVP candidate may be derived based on motion information of the earlier coded CU and included in the HMVP buffer.

[0180] If the current picture is divided into multiple tiles, the HMVP buffer may be initialized in units of CTU rows within each tile.

[0181] The HMVP buffer may be initialized in units of CTU rows within a tile or slice. For example, if a specific CTU in the CTU row is not the first CTU in the CTU row in the current picture, but is the first CTU in the CTU row in the current tile or slice, the HMVP buffer may be initialized with the specific CTU.

[0182] When the HMVP buffer is initialized, the number of HMVP candidates included in the HMVP buffer may be set to zero.

[0183] The decoding apparatus constructs a motion information candidate list based on the HMVP buffer (S2710). The HMVP buffer may include HMVP candidates, and the motion information candidate list may be constructed including the HMVP candidates.

[0184] As an example, when a merge mode is applied to the current block, the motion information candidate list may be a merge candidate list. As another example, when an (A)MVP mode is applied to the current block, the motion information candidate list may be an MVP candidate list. When a merge mode is applied to the current block, the HMVP candidate may be added to the merge candidate list if the number of available merge candidates (e.g., including spatial merge candidates and temporal merge candidates) in the merge candidate list for the current block is less than a predetermined maximum number of merge candidates. In this case, the HMVP candidate may be inserted after the spatial candidate and the temporal candidate in the merge candidate list. In other words, the HMVP candidate may be assigned an index value greater than the indexes assigned to the spatial candidate and the temporal candidate in the merge candidate list. When an (A)MVP mode is applied to the current block, the HMVP candidate may be added to the MVP candidate list if the number of available MVP candidates (derived based on spatial and temporal neighboring blocks) in the MVP candidate list for the current block is less than two.

[0185] The decoding apparatus can derive motion information of the current block based on the motion information candidate list (S2720).

[0186] The encoding apparatus may derive motion information of the current block based on the motion information candidate list. For example, when a merge mode or an MVP mode is applied to the current block, the HMVP candidate included in the HMVP buffer may be used as a merge candidate or an MVP candidate. For example, when a merge mode is applied to the current block, the HMVP candidate included in the HMVP buffer may be included as a candidate in a merge candidate list, and the HMVP candidate among the candidates included in the merge candidate list may be indicated based on a merge index obtained from a bitstream. In this case, the HMVP candidate may be assigned an index in the merge candidate list with a lower priority than the spatial merge candidates and temporal merge candidates included in the merge candidate list. That is, the index value assigned to the HMVP candidate may be assigned a higher value than the index values ​​of the spatial merge candidates and temporal merge candidates. As another example, if MVP mode is applied to the current block, the HMVP candidate included in the HMVP buffer may be included as a candidate in the merge candidate list, and the HMVP candidate may be indicated among the candidates included in the MVP candidate list based on an MVP flag (or MVP index) obtained from the bitstream.

[0187] The decoding device generates a prediction sample for the current block based on the derived motion information (S2730). The decoding device can derive the prediction sample using the reference sample indicated by the motion information in the reference picture by performing inter prediction (motion compensation) based on the motion information. The current block including the prediction sample is also called a predicted block.

[0188] The decoding apparatus generates reconstructed samples based on the predicted samples (S2740). As described above, a reconstructed block / picture can be generated based on the reconstructed samples. As described above, the decoding apparatus can obtain residual information (including information about quantized transform coefficients) from the bitstream, derive residual samples based on the residual information, and generate the reconstructed samples based on the predicted samples and the residual samples. As described above, an in-loop filtering procedure such as deblock filtering, SAO, and / or ALF procedure can be applied to the reconstructed picture as needed to improve subjective / objective image quality.

[0189] In the above-described embodiments, the method is described based on a flow chart with a series of steps or blocks, but the embodiment is not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps than those described above. Also, those skilled in the art will understand that the steps shown in the flow chart are not exclusive, and other steps may be included, or one or more steps in the flow chart may be deleted without affecting the scope of the embodiments herein.

[0190] The methods according to the embodiments of this document described above can be implemented in software form, and the encoding device and / or decoding device according to this document can be included in devices that perform image processing, such as TVs, computers, smartphones, set-top boxes, display devices, etc.

[0191] In this document, when an embodiment is implemented in software, the method described above may be implemented with modules (processes, functions, etc.) that perform the functions described above. The modules may be stored in memory and executed by a processor. The memory may be internal or external to the processor and may be coupled to the processor in various well-known ways. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, and / or data processing devices. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, a memory card, a storage medium, and / or other storage devices. That is, the embodiments described herein may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units illustrated in each figure may be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, information (e.g., information on instructions) or algorithms for implementation may be stored on a digital storage medium.

[0192] In addition, the decoding device and encoding device to which the embodiment(s) of this document are applied may be included in a multimedia broadcast transmitting / receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video interaction device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camcorder, a video-on-demand (VoD) service providing device, an over-the-top (OTT) video device, an internet streaming service providing device, a three-dimensional (3D) video device, a virtual reality (VR) device, an augmented reality (AR) device, an image telephone video device, a transportation terminal (e.g., a vehicle terminal (including an autonomous vehicle), an airplane terminal, a ship terminal, etc.), a medical video device, etc., and may be used to process a video signal or a data signal. For example, an over-the-top (OTT) video device may include a game console, a Blu-ray player, an internet-connected TV, a home theater system, a smartphone, a tablet PC, a digital video recorder (DVR), etc.

[0193] Furthermore, a processing method to which one or more embodiments of this document are applied can be produced in the form of a computer-executable program and stored in a computer-readable recording medium. Multimedia data having a data structure according to one or more embodiments of this document can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium can include, for example, Blu-ray Discs (BDs), Universal Serial Buses (USBs), ROMs, PROMs, EPROMs, EEPROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. The computer-readable recording medium can also include media embodied in the form of carrier waves (e.g., transmission via the Internet). The bitstream generated by the encoding method can be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0194] Furthermore, the embodiment(s) of this document may be implemented in a computer program product by program code, which may be executed by a computer in accordance with the embodiment(s) of this document, and which may be stored on a computer-readable carrier.

[0195] FIG. 29 illustrates an example of a content streaming system to which the embodiments disclosed herein may be applied.

[0196] As shown in FIG. 29, a content streaming system to which the embodiments of this document are applied may generally include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0197] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, camcorder, etc. directly generates a bitstream, the encoding server may be omitted.

[0198] The bitstream can be generated by an encoding method or a bitstream generation method to which an embodiment of this document is applied, and the streaming server can temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0199] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary for informing the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, which then transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, which controls commands and responses between devices in the content streaming system.

[0200] The streaming server may receive content from a media storage device and / or an encoding server. For example, when receiving content from the encoding server, the content may be received in real time. In this case, the streaming server may store the bitstream for a certain period of time to provide a smooth streaming service.

[0201] Examples of the user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), navigation systems, slate PCs, tablet PCs, ULTRABOOK (registered trademark), wearable devices (e.g., smartwatches, smart glasses, and HMDs (Head Mounted Displays)), digital TVs, desktop computers, and digital signatures.

[0202] Each server in the content streaming system can be operated as a distributed server, and in this case, data received by each server can be processed in a distributed manner.

Claims

1. An image decoding method performed by a decoding device, comprising: deriving a History-based Motion Vector Prediction (HMVP) buffer for the current block; deriving merge candidates for constructing a merge candidate list for the current block, the merging candidates include spatial candidates derived based on spatial candidate blocks and temporal candidates derived based on temporal candidate blocks; the HMVP candidates included in the HMVP buffer are inserted as merge candidates in the merge candidate list; the HMVP candidate is inserted after the temporal candidate in the merge candidate list; deriving motion information of the current block based on the HMVP candidates in the merge candidate list; generating a prediction sample for the current block based on the motion information; generating reconstructed samples based on the predicted samples; The current picture is partitioned into one or more tiles; The current block corresponds to one of coding units (CUs) divided from a coding tree unit (CTU), The HMVP buffer is updated based on the motion information of the previous block; The HMVP buffer is initialized with the first CTU for each CTU row of each tile; The method further includes determining whether the CTU is a first CTU in a CTU row of a current tile; the HMVP buffer for the current block is derived based on the result of the determination; The method, wherein the HMVP buffer is initialized for the CTU based on the result of the determination that the CTU is the first CTU in the CTU row of the current tile.

2. An image encoding method performed by an encoding device, comprising: deriving a History-based Motion Vector Prediction (HMVP) buffer for the current block; deriving merge candidates for constructing a merge candidate list for the current block, the merging candidates include spatial candidates derived based on spatial candidate blocks and temporal candidates derived based on temporal candidate blocks; the HMVP candidates included in the HMVP buffer are inserted as merge candidates in the merge candidate list; the HMVP candidate is inserted after the temporal candidate in the merge candidate list; deriving motion information of the current block based on the HMVP candidates in the merge candidate list; generating a prediction sample for the current block based on the motion information; generating residual samples based on the prediction samples; encoding image information including information about the residual samples; The current picture is partitioned into one or more tiles; The current block corresponds to one of coding units (CUs) divided from a coding tree unit (CTU), The HMVP buffer is updated based on the motion information of the previous block; The HMVP buffer is initialized with the first CTU for each CTU row of each tile; The method further includes determining whether the CTU is a first CTU in a CTU row of a current tile; the HMVP buffer for the current block is derived based on the result of the determination; The method, wherein the HMVP buffer is initialized for the CTU based on the result of the determination that the CTU is the first CTU in the CTU row of the current tile.

3. A method for transmitting data relating to an image, comprising: obtaining a bitstream for the image, the bitstream comprising: deriving a History-based Motion Vector Prediction (HMVP) buffer for the current block; deriving merge candidates for constructing a merge candidate list for the current block, the merging candidates include spatial candidates derived based on spatial candidate blocks and temporal candidates derived based on temporal candidate blocks; the HMVP candidates included in the HMVP buffer are inserted as merge candidates in the merge candidate list; the HMVP candidate is inserted after the temporal candidate in the merge candidate list; deriving motion information of the current block based on the HMVP candidates in the merge candidate list; generating a prediction sample for the current block based on the motion information; generating residual samples based on the prediction samples; encoding image information including information about the residual samples; transmitting the data including the bitstream; The current picture is partitioned into one or more tiles; The current block corresponds to one of coding units (CUs) divided from a coding tree unit (CTU), The HMVP buffer is updated based on the motion information of the previous block; The HMVP buffer is initialized with the first CTU for each CTU row of each tile; The method further includes determining whether the CTU is a first CTU in a CTU row of a current tile; the HMVP buffer for the current block is derived based on the result of the determination; The method, wherein the HMVP buffer is initialized for the CTU based on the result of the determination that the CTU is the first CTU in the CTU row of the current tile.

Images