Video or image coding based on intra block coding
By using high-efficiency filtering and intra-block compilation technology in image/video encoding and decoding, signaling to inform the merged candidate list of information, solving the problem of high-resolution image/video data transmission and storage costs, achieving more efficient Codec efficiency and visual quality.
Patent Information
- Application Number
- CN202411063945.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2019-03-04
- Filing Date
- 2020-03-04
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art is difficult to efficiently compress and transmit high-resolution, high-quality image/video data, especially in the applications of virtual reality, augmented reality and immersive media, with high transmission and storage costs and existing intra-block compilation Inadequate technical efficiency.
Using a method and device, through efficient filtering applications and intra-block compilation technology, including signaling block vectors and motion vector information in merge candidate lists in inter-prediction and intra-prediction, to improve image/video codec efficiency.
Improve image/video compression efficiency, improve subjective and objective visual quality, enhance the freedom of the codec, and reduce transmission and storage costs.
Smart Images

Figure CN119094754A_ABST
Abstract
Description
[0001] This application is a divisional application of the patent application with application number 202080018333.0 (PCT / KR2020 / 003045), which was submitted to the China Patent Office on September 2, 2021, with an international application date of March 4, 2020, and the invention name is “Video or image coding based on intra-frame block coding”. Technical Field
[0002] This document relates to video or image coding based on intra block coding (IBC). Background Art
[0003] Recently, there is an increasing demand for high-resolution, high-quality images / videos such as 4K or 8K ultra-high definition (UHD) images / videos in various fields. As the image / video resolution or quality becomes higher, relatively more information or bits are transmitted compared to conventional image / video data. Therefore, if the image / video data is transmitted via a medium such as an existing wired / wireless broadband line or is stored in a conventional storage medium, the cost of transmission and storage is easily increased.
[0004] In addition, there is growing interest and demand for virtual reality (VR) and artificial reality (AR) content and immersive media such as holograms; and there is also growing broadcasting of images / videos that exhibit image / video characteristics different from actual images / videos (e.g., game images / videos).
[0005] Therefore, highly efficient image / video compression technology is required to effectively compress and transmit, store, or play high-resolution, high-quality images / videos showing various characteristics as described above.
[0006] In addition, intra-block coding (IBC) techniques are also discussed to improve compression efficiency and improve coding efficiency of screen content. In order to effectively apply these techniques, a method of effectively signaling relevant information is required. Summary of the invention
[0007] Technical Solution
[0008] According to the embodiments of this document, a method and apparatus for improving image coding efficiency are provided.
[0009] According to the embodiments of this document, an efficient filtering application method and device are provided.
[0010] According to the embodiments of this invention, an efficient IBC application method and device are provided.
[0011] According to an embodiment of this document, a method and apparatus for signaling information about a maximum number of block vector candidates included in a candidate list for IBC are provided.
[0012] According to an embodiment of the present invention, a method and apparatus are provided for separately signaling information about a maximum number of block vector candidates included in a merge candidate list for IBC merge and information about a maximum number of motion vector candidates included in a merge candidate list for a merge mode for inter prediction.
[0013] According to an embodiment of this document, a video / image decoding method performed by a decoding device is provided.
[0014] According to an embodiment of this document, a decoding apparatus for performing video / image decoding is provided.
[0015] According to an embodiment of this document, a video / image encoding method performed by an encoding device is provided.
[0016] According to an embodiment of this document, there is provided an encoding apparatus for performing video / image encoding.
[0017] According to one embodiment of the present invention, a computer-readable digital storage medium is provided, in which encoded video / image information generated according to the video / image encoding method disclosed in at least one embodiment of the present invention is stored.
[0018] According to an embodiment of this document, a computer-readable digital storage medium is provided, in which encoding information or encoded video / image information is stored for enabling a decoding device to perform a video / image decoding method disclosed in at least one embodiment of this document.
[0019] Beneficial Effects
[0020] According to the embodiments of this document, the overall image / video compression efficiency may be improved.
[0021] According to the embodiments of this document, the subjective / objective visual quality can be improved through efficient filtering.
[0022] According to the embodiments of this document, a merge candidate list for IBC may be constructed.
[0023] According to an embodiment of this document, information about the maximum number of candidates included in a merge candidate list for IBC may be signaled.
[0024] According to the embodiments of this invention, the degree of freedom of the codec can be increased and the coding efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 An example of a video / image coding system to which embodiments of this document may be applied is illustrated.
[0026] Figure 2is a diagram schematically illustrating a configuration of a video / image encoding device to which the embodiments of this document can be applied.
[0027] Figure 3 is a diagram schematically illustrating a configuration of a video / image decoding device to which the embodiments of this document can be applied.
[0028] Figure 4 This is a diagram for explaining a method of deriving motion information through an inter-frame prediction mode.
[0029] Figure 5 and Figure 6 The diagram illustrates an image encoding method based on the IBC prediction mode and a predictor of an encoding device that executes the image encoding method.
[0030] Figure 7 and Figure 8 The diagram illustrates an image decoding method based on the IBC prediction mode and a predictor of a decoding device that executes the image decoding method.
[0031] Fig. 9 and Fig.10 An example of a video / image encoding method and related components according to embodiments of the present invention is schematically shown.
[0032] Fig.11 and Fig.12 An example of an image / video decoding method and related components according to embodiments of the present invention is schematically shown.
[0033] Fig.13 An example of a content streaming system to which the embodiments disclosed herein may be applied is shown. DETAILED DESCRIPTION
[0034] This article can be modified in various forms, and its specific embodiments will be described and shown in the drawings. However, these embodiments are not intended to limit this article. The terms used in the following description are only used to describe specific embodiments and are not intended to limit this article. Singular expressions include plural expressions unless they can be clearly understood differently. Terms such as "including" and "having" are intended to indicate the presence of features, quantities, steps, operations, elements, components, or combinations thereof used in the following description, so it should be understood that the possibility of the presence or addition of one or more different features, quantities, steps, operations, elements, components, or combinations thereof is not excluded.
[0035] At the same time, for the convenience of describing different characteristic functions, each configuration in the drawings described in this article is shown independently, which does not mean that each configuration is implemented as separate hardware or separate software. For example, two or more components in various components can be combined to form one component, or one component can be divided into multiple components. Embodiments that integrate and / or separate each component are also included in the scope of the disclosure of this article.
[0036] Hereinafter, an example of the present embodiment will be described in detail with reference to the accompanying drawings. In addition, the same reference numerals are used throughout the drawings to indicate the same elements, and the same description of the same elements will be omitted.
[0037] Figure 1 An example of a video / image coding system to which embodiments of this document may be applied is illustrated.
[0038] Reference Figure 1 The video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit the encoded video / image information or data in the form of a file or stream transmission to the receiving device via a digital storage medium or a network.
[0039] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.
[0040] The video source can obtain the video / image by the process of capturing, synthesizing or generating the video / image. The video source may include a video / image capturing device and / or a video / image generating device. For example, the video / image capturing device may include one or more cameras, a video / image archive including previously captured videos / images, etc. For example, the video / image generating device may include a computer, a tablet computer, and a smart phone, and may (electronically) generate the video / image. For example, a virtual video / image may be generated by a computer, etc. In this case, the video / image capturing process may be replaced by a process of generating relevant data.
[0041] The encoding device can encode the input video / image. For compression and coding efficiency, the encoding device can perform a series of processes such as prediction, transformation and quantization. The encoded data (encoded video / image information) can be output in the form of a bit stream.
[0042] The transmitter may transmit the encoded image / image information or data output in the form of a bit stream to a receiver of a receiving device in the form of a file or stream transmission through a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for transmission through a broadcast / communication network. The receiver may receive / extract a bit stream and transmit the received bit stream to a decoding device.
[0043] The decoding device may decode a video / image by performing a series of processes such as dequantization, inverse transformation, and prediction corresponding to the operations of the encoding device.
[0044] The renderer can render the decoded video / image. The rendered video / image can be displayed by a display.
[0045] This article relates to video / image coding. For example, the methods / embodiments disclosed herein can be applied to methods disclosed in the Versatile Video Coding (VVC) standard, the Essential Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the 2nd generation audio and video coding standard (AVS2), or the next generation video / image coding standard (e.g., H.267 or H.268, etc.).
[0046] Various embodiments of video / image coding are suggested herein, and unless otherwise mentioned, the embodiments may be performed in combination with each other.
[0047] In this document, video may refer to a series of images over time. A picture generally refers to a unit representing an image at a specific time frame, and a slice / tile refers to a unit that constitutes a part of a picture in terms of coding. A slice / tile may include one or more coding tree units (CTUs). A picture may consist of one or more slices / tiles. A picture may consist of one or more tile groups. A tile group may include one or more tiles. A tile may represent a rectangular area of a CTU row within a tile in a picture. A tile may be divided into a plurality of tiles, each tile consisting of one or more CTU rows within a tile. Tiles that are not divided into a plurality of tiles may also be referred to as tiles. Tile scanning may represent a specific ordering of CTUs of a partitioned picture, wherein CTUs are continuously sorted with a CTU raster scan within a tile, and tiles within a tile are continuously sorted with a raster scan of tiles of a tile, and tiles in a tile are continuously sorted with a raster scan of tiles of a tile, and tiles in a picture are continuously sorted with a raster scan of tiles of a tile. A tile is a rectangular area of a CTU within a specific tile column and a specific tile row in a picture. A tile column is a rectangular area of a CTU whose height is equal to the height of the picture and whose width is specified by a syntax element in the picture parameter set. A tile row is a rectangular area of a CTU whose height is specified by a syntax element in the picture parameter set and whose width is equal to the width of the picture. A tile scan is a specific sequential ordering of the CTUs of a partitioned picture, wherein a raster scan of the CTUs in a tile continuously orders the CTUs, while a raster scan of the tiles of a picture continuously orders the tiles in a picture. A slice includes an integer number of tiles of a picture that can be exclusively contained in a single NAL unit. A slice can consist of a continuous sequence of multiple complete tiles or only complete tiles of one tile. In this document, tile groups and slices can be used interchangeably. For example, in this document, a tile group / tile group header can be referred to as a slice / slice header.
[0048] At the same time, a picture can be divided into two or more sub-pictures. A sub-picture can be a rectangular area of one or more slices within a picture.
[0049] A pixel or a picture element (pel) may refer to the smallest unit constituting a picture (or image). In addition, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luminance component or only a pixel / pixel value of a chrominance component.
[0050] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of a picture and information related to the area. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. In some cases, a unit may be used interchangeably with terms such as a block or an area. In general, an M×N block may include M columns and N rows of samples (or sample arrays) or a set (or array) of transform coefficients. Alternatively, a sample may refer to a pixel value in the spatial domain, and when such a pixel value is transformed to the frequency domain, it may refer to a transform coefficient in the frequency domain.
[0051] In this document, "A or B" may mean "only A", "only B", or "A and B". In other words, "A or B" in this document may be interpreted as "A and / or B". For example, in this document, "A, B or C (A, B or C)" means "only A", "only B", "only C", or "any combination of A, B, and C".
[0052] As used herein, a slash ( / ) or a comma (comma) may mean "and / or". For example, "A / B" may mean "A and / or B". Thus, "A / B" may mean "only A", "only B", or "A and B". For example, "A,B,C" may mean "A, B, or C".
[0053] Herein, "at least one of A and B" may mean "only A", "only B" or "A and B". In addition, herein, the expression "at least one of A or B" or "at least one of A and / or B" may be interpreted as "at least one of A and B".
[0054] Furthermore, herein, "at least one of A, B, and C" means "only A", "only B", "only C", or "any combination of A, B, and C". Furthermore, "at least one of A, B, or C" or "at least one of A, B and / or C" may mean "at least one of A, B, and C".
[0055] In addition, the brackets used in this document may refer to "for example". Specifically, when "prediction (intra-frame prediction)" is indicated, "intra-frame prediction" may be proposed as an example of "prediction". In other words, "prediction" in this document is not limited to "intra-frame prediction", and "intra-frame prediction" may be proposed as an example of "prediction". In addition, even when "prediction (ie, intra-frame prediction)" is indicated, "intra-frame prediction" may be proposed as an example of "prediction".
[0056] The technical features described separately in one of the figures of this document can be implemented separately or simultaneously.
[0057] Figure 2is a diagram schematically illustrating a configuration of a video / image encoding device to which the embodiments of this document can be applied. Hereinafter, the so-called video encoding device may include an image encoding device.
[0058] Reference Figure 2 , the encoding device 200 includes an image segmenter 210, a predictor 220, a residual processor 230 and an entropy encoder 240, an adder 250, a filter 260 and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234 and an inverse transformer 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. According to an embodiment, the image segmenter 210, the predictor 220, the residual processor 230, the entropy encoder 240, the adder 250 and the filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). In addition, the memory 270 may include a decoded picture buffer (DPB), or may be configured by a digital storage medium. The hardware component may also include the memory 270 as an internal / external component.
[0059] The image divider 210 may divide the input image (or picture or frame) input to the encoding device 200 into one or more processors. For example, the processor may be referred to as a coding unit (CU). In this case, the coding unit may be recursively divided from a coding tree unit (CTU) or a maximum coding unit (LCU) according to a quadtree binary tree ternary tree (QTBTTT) structure. For example, one coding unit may be divided into a plurality of coding units with a deeper depth based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, a quadtree structure may be applied first, and a binary tree structure and / or a ternary structure may be applied later. Alternatively, a binary tree structure may be applied first. The coding process according to the present disclosure may be performed based on a final coding unit that is no longer divided. In this case, based on coding efficiency according to image characteristics, the maximum coding unit may be used as the final coding unit, or if necessary, the coding unit may be recursively divided into coding units with a deeper depth and a coding unit with an optimal size may be used as the final coding unit. Here, the coding process may include a process of prediction, transformation, and reconstruction (to be described later). As another example, the processor may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be split or partitioned from the above-mentioned final coding unit. The prediction unit may be a unit for sample prediction, and the transform unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from the transform coefficient.
[0060] In some cases, a unit may be used interchangeably with terms such as a block or region. In general, an M×N block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luminance component or only a pixel / pixel value of a chrominance component. A sample may be used as a term corresponding to a picture (or image) of a pixel or a picture element.
[0061] In the encoding device 200, a prediction signal (prediction block, prediction sample array) output from the inter predictor 221 or the intra predictor 222 is subtracted from an input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is sent to the transformer 232. In this case, as shown, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input image signal (original block, original sample array) in the encoder 200 may be referred to as a subtractor 231. The predictor may perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a prediction block including prediction samples of the current block. The predictor may determine whether intra prediction or inter prediction is applied based on the current block or CU. As described later in the description of each prediction mode, the predictor may generate various types of information related to prediction (e.g., prediction mode information) and send the generated information to the entropy encoder 240. Information about prediction may be encoded in the entropy encoder 240 and output in the form of a bitstream.
[0062] The intra-frame predictor 222 may predict the current block with reference to samples in the current picture. Depending on the prediction mode, the referenced samples may be located near the current block or may be spaced apart. In intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. For example, the non-directional mode may include a DC mode and a plane mode. For example, depending on the level of detail of the prediction direction, the directional mode may include 33 directional prediction modes or 65 directional prediction modes. However, this is only an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-frame predictor 222 may use the prediction mode applied to the neighboring blocks to determine the prediction mode applied to the current block.
[0063] The inter-frame predictor 221 may derive a prediction block of the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. Here, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be referred to as a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block may be referred to as a collocated picture (colPic). For example, the inter-frame predictor 221 may configure a motion information candidate list based on the neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter-frame prediction may be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter-frame predictor 221 may use the motion information of the neighboring block as the motion information of the current block. In the skip mode, unlike the merge mode, the residual signal may not be sent. In the case of the motion vector prediction (MVP) mode, the motion vector of the neighboring block may be used as a motion vector predictor, and the motion vector of the current block may be indicated by signaling the motion vector difference.
[0064] The predictor 220 may generate a prediction signal based on various prediction methods described below. For example, the predictor may not only apply intra prediction or inter prediction to predict a block, but also apply both intra prediction and inter prediction at the same time. This may be referred to as combined inter and intra prediction (CIIP). In addition, the predictor may predict a block based on an intra block copy (IBC) prediction mode or a palette mode. The IBC prediction mode or the palette mode may be used for content image / video coding of games, such as screen content coding (SCC). IBC basically performs prediction in the current picture, but may be performed similarly to inter prediction so that a reference block is derived in the current picture. That is, IBC may use at least one inter prediction technique described herein. The palette mode may be considered as an example of intra coding or intra prediction. When the palette mode is applied, the sample values within the picture may be signaled based on information about the palette table and the palette index.
[0065] The prediction signal generated by the predictor (including the inter-frame predictor 221 and / or the intra-frame predictor 222) can be used to generate a reconstruction signal or to generate a residual signal. The transformer 232 can generate a transform coefficient by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loève transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, when the relationship information between pixels is represented by a graph, GBT means a transform obtained from a graph. CNT refers to a transform generated based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process can be applied to square pixel blocks of the same size or to blocks of variable sizes other than square.
[0066] The quantizer 233 may quantize the transform coefficients and send them to the entropy encoder 240, and the entropy encoder 240 may encode the quantized signal (information about the quantized transform coefficients) and output a bitstream. The information about the quantized transform coefficients may be referred to as residual information. The quantizer 233 may rearrange the block type quantized transform coefficients into a one-dimensional vector form based on the coefficient scanning order, and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. Information about the transform coefficients may be generated. The entropy encoder 240 may perform various encoding methods such as exponential Golomb, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), etc. The entropy encoder 240 may encode information required for video / image reconstruction other than the quantized transform coefficients (e.g., the value of a syntax element, etc.) together or separately. The encoded information (e.g., encoded video / image information) may be transmitted or stored in units of NAL (Network Abstraction Layer) in the form of a bitstream. The video / image information may also include information about various parameter sets, such as an adaptive parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may also include general constraint information. In this article, information and / or syntax elements sent / signaled from the encoding device to the decoding device may be included in the video / image information. The video / image information may be encoded by the above-mentioned encoding process and included in the bitstream. The bitstream may be sent via a network or may be stored in a digital storage medium. The network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) that sends a signal output from the entropy encoder 240 and / or a storage unit (not shown) that stores the signal may be included as an internal / external element of the encoding device 200, and alternatively, the transmitter may be included in the entropy encoder 240.
[0067] The quantized transform coefficients output from the quantizer 233 can be used to generate a prediction signal. For example, a residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transformation to the quantized transform coefficients via a dequantizer 234 and an inverse transformer 235. The adder 250 adds the reconstructed residual signal to the prediction signal output from the inter-frame predictor 221 or the intra-frame predictor 222 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array). If there is no residual in the block to be processed (for example, in the case of applying a skip mode), the prediction block can be used as a reconstructed block. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. As described below, the generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current picture and can be used for inter-frame prediction of the next picture through filtering.
[0068] Furthermore, luma mapping with chroma scaling (LMCS) may be applied during picture encoding and / or reconstruction.
[0069] The filter 260 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 260 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture and store the modified reconstructed picture in the memory 270 (specifically, the DPB of the memory 270). For example, various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filter 260 can generate various types of information related to filtering and send the generated information to the entropy encoder 240, as described later in the description of each filtering method. The information related to filtering can be encoded by the entropy encoder 240 and output in the form of a bitstream.
[0070] The modified reconstructed picture transmitted to the memory 270 may be used as a reference picture in the inter predictor 221. When inter prediction is applied by the encoding device, prediction mismatch between the encoding device 200 and the decoding device may be avoided and encoding efficiency may be improved.
[0071] The DPB of the memory 270 DPB may store a modified reconstructed picture used as a reference picture in the inter-frame predictor 221. The memory 270 may store motion information of a block for deriving (or encoding) motion information in the current picture and / or motion information of a reconstructed block in the picture. The stored motion information may be sent to the inter-frame predictor 221 and used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory 270 may store reconstructed samples of a reconstructed block in the current picture and may transmit the reconstructed samples to the intra-frame predictor 222.
[0072] Figure 3 is a schematic diagram illustrating a configuration of a video / image decoding device to which an embodiment of the present disclosure can be applied.
[0073] Reference Figure 3 , the decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter-frame predictor 331 and an intra-frame predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. According to an embodiment, the entropy decoder 310, the residual processor 320, the predictor 330, the adder 340, and the filter 350 may be configured by a hardware component (e.g., a decoder chipset or a processor). In addition, the memory 360 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium. The hardware component may also include the memory 360 as an internal / external component.
[0074] When a bit stream including video / image information is input, the decoding apparatus 300 can reconstruct the bit stream corresponding to the bit stream in the video / image format. Figure 2 The image corresponding to the processing of the video / image information in the encoding device. For example, the decoding device 300 may derive the unit / block based on the block segmentation related information obtained from the bit stream. The decoding device 300 may use a processor applied in the encoding device to perform decoding. Therefore, for example, the decoding processor may be a coding unit, and the coding unit may be divided from the coding tree unit or the maximum coding unit according to a quadtree structure, a binary tree structure and / or a ternary tree structure. One or more transform units may be derived from the coding unit. The reconstructed image signal decoded and output by the decoding device 300 may be reproduced by a reproduction device.
[0075] The decoding device 300 may receive Figure 2The received signal may be decoded by the entropy decoder 310. For example, the entropy decoder 310 may parse the bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). The video / image information may also include information about various parameter sets, such as an adaptive parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may also include general constraint information. The decoding device may also decode the picture based on the information about the parameter set and / or the general constraint information. The signaled / received information and / or syntax elements described later in this document may be decoded and obtained from the bitstream by a decoding process. For example, the entropy decoder 310 may decode the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the syntax elements required for image reconstruction and the quantized values of the transform coefficients of the residual. More specifically, the CABAC entropy decoding method may receive bins corresponding to each syntax element in the bitstream, use the decoding target syntax element information, the decoding information of the decoding target block, or the information of the symbol / bin decoded in the previous stage to determine the context model, and perform arithmetic decoding on the bin by predicting the probability of the bin appearing according to the determined context model, and generate symbols corresponding to the values of each syntax element. In this case, the CABAC entropy decoding method may update the context model by using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information related to prediction among the information decoded by the entropy decoder 310 may be provided to the predictor (inter-frame predictor 332 and intra-frame predictor 331), and the residual value (ie, quantized transform coefficients and related parameter information) performed entropy decoding in the entropy decoder 310 may be input to the residual processor 320. The residual processor 320 may derive a residual signal (residual block, residual sample, residual sample array). In addition, information about filtering among the information decoded by the entropy decoder 310 may be provided to the filter 350. In addition, a receiver (not shown) for receiving a signal output from the encoding device may also be configured as an internal / external element of the decoding device 300, or the receiver may be a component of the entropy decoder 310. In addition, the decoding device according to this document may be referred to as a video / image / picture decoding device, and the decoding device may be classified into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoder 310, and the sample decoder may include at least one of the dequantizer 321, the inverse transformer 322, the adder 340, the filter 350, the memory 360, the inter-frame predictor 332, and the intra-frame predictor 331.
[0076] The dequantizer 321 may dequantize the quantized transform coefficient and output the transform coefficient. The dequantizer 321 may rearrange the quantized transform coefficient in a two-dimensional block form. In this case, the rearrangement may be performed based on a coefficient scanning order performed in the encoding device. The dequantizer 321 may perform dequantization on the quantized transform coefficient using a quantization parameter (e.g., quantization step size information) and obtain the transform coefficient.
[0077] The inverse transformer 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).
[0078] The predictor may perform prediction on the current block and generate a prediction block including prediction samples of the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on information on prediction output from the entropy decoder 310 and may determine a specific intra / inter prediction mode.
[0079] The predictor 320 may generate a prediction signal based on various prediction methods described below. For example, the predictor may not only apply intra prediction or inter prediction to predict a block, but may also apply intra prediction and inter prediction at the same time. This may be referred to as combined inter and intra prediction (CIIP). In addition, the predictor may predict a block based on an intra block copy (IBC) prediction mode or a palette mode. The IBC prediction mode or the palette mode may be used for content image / video coding of games, etc., such as screen content coding (SCC). IBC basically performs a prediction in the current picture, but may be performed similarly to inter prediction so that a reference block is derived in the current picture. That is, IBC may use at least one inter prediction technique described herein. The palette mode may be considered an example of intra coding or intra prediction. When the palette mode is applied, the sample values within the picture may be signaled based on information about the palette table and the palette index.
[0080] The intra-frame predictor 331 may predict the current block with reference to samples in the current picture. Depending on the prediction mode, the referenced samples may be located near the current block or may be spaced apart. In intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. The intra-frame predictor 331 may use the prediction mode applied to the neighboring blocks to determine the prediction mode applied to the current block. The intra-frame predictor 331 may predict the current block with reference to samples in the current picture. Depending on the prediction mode, the referenced samples may be located near the current block or may be spaced apart. In intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. The intra-frame predictor 331 may use the prediction mode applied to the neighboring blocks to determine the prediction mode applied to the current block.
[0081] The inter-frame predictor 332 may derive a prediction block of the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter-frame predictor 332 may configure a motion information candidate list based on neighboring blocks and derive a motion vector and / or a reference picture index of the current block based on the received candidate selection information. Inter-frame prediction may be performed based on various prediction modes, and information about the prediction may include information indicating the inter-frame prediction mode of the current block.
[0082] The adder 340 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictor (including the inter-frame predictor 332 and / or the intra-frame predictor 331). If there is no residual in the block to be processed, for example, when the skip mode is applied, the prediction block can be used as the reconstructed block.
[0083] The adder 340 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next block to be processed in the current picture, may be output through a filter as described below, or may be used for inter prediction of the next picture.
[0084] The filter 350 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 350 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture and store the modified reconstructed picture in the memory 360 (specifically, the DPB of the memory 360). For example, the various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.
[0085] The (modified) reconstructed picture stored in the DPB of the memory 360 may be used as a reference picture in the inter-frame predictor 332. The memory 360 may store the motion information of the block from which the motion information in the current picture is derived (or decoded) and / or the motion information of the reconstructed block in the picture. The stored motion information may be sent to the inter-frame predictor 260 to be used as the motion information of the spatial neighboring block or the motion information of the temporal neighboring block. The memory 360 may store the reconstructed samples of the reconstructed block in the current picture and transmit the reconstructed samples to the intra-frame predictor 331.
[0086] In the present disclosure, the embodiments described in the filter 260, the inter-frame predictor 221, and the intra-frame predictor 222 of the encoding device 200 may be the same as or respectively applied to the filter 350, the inter-frame predictor 332, and the intra-frame predictor 331 of the decoding device 300. This may also be applied to the unit 332 and the intra-frame predictor 331.
[0087] As described above, when performing video coding, prediction is performed to enhance compression efficiency. A prediction block including prediction samples of a current block (ie, a target coding block) may be generated by prediction. In this case, the prediction block includes prediction samples in a spatial domain (or a pixel domain). The prediction block is derived identically in the encoding device and the decoding device. The encoding device may enhance image coding efficiency by notifying the decoding device with a signal of information (residual information) about the residual between the original block (rather than the original sample value of the original block itself) and the prediction block. The decoding device may derive a residual block including residual samples based on the residual information, may generate a reconstructed block including reconstructed samples by adding the residual block and the prediction block, and may generate a reconstructed picture including the reconstructed block.
[0088] The residual information may be generated by a transform process and a quantization process. For example, the encoding device may derive a residual block between the original block and the prediction block, may derive a transform coefficient by performing a transform process on a residual sample (residual sample array) included in the residual block, may derive a quantized transform coefficient by performing a quantization process on the transform coefficient, and may signal the relevant residual information (through a bitstream) to the decoding device. In this case, the residual information may include information such as value information, position information, transform scheme, transform core, and quantization parameter of the quantized transform coefficient. The decoding device may perform a dequantization / inverse transform process based on the residual information and may derive a residual sample (or residual block). The decoding device may generate a reconstructed picture based on the prediction block and the residual block. In addition, the encoding device may derive a residual block by dequantizing / inverse transforming the quantized transform coefficient for reference to inter-frame prediction of subsequent pictures, and may generate a reconstructed picture. In this article, at least one of quantization / dequantization and / or transform / inverse transform may be omitted. When quantization / dequantization is omitted, the quantized transform coefficient may be referred to as a transform coefficient. When the transform / inverse transform is omitted, the transform coefficient may be referred to as a coefficient or a residual coefficient, or may still be referred to as a transform coefficient for the sake of uniform expression. In this article, the quantized transform coefficient and the transform coefficient may be referred to as a transform coefficient and a scaled transform coefficient, respectively. In this case, the residual information may include information about the transform coefficient, and the information about the transform coefficient may be signaled by the residual coding syntax. The transform coefficient may be derived based on the residual information (or information about the transform coefficient), and the scaled transform coefficient may be derived by inverse transforming (scaling) the transform coefficient. The residual sample may be derived based on the inverse transform (transform) of the scaled transform coefficient. This may also be applied / expressed in other parts of this article.
[0089] Intra-frame prediction may refer to a prediction of a prediction sample generated for a current block based on a reference sample in a picture to which the current block belongs (hereinafter referred to as the current picture). When intra-frame prediction is applied to the current block, neighboring reference samples to be used for intra-frame prediction of the current block may be derived. The neighboring reference samples of the current block may include samples adjacent to the left boundary of the current block having a size of nWxnH and a total of 2xnH samples adjacent to the lower left, samples adjacent to the upper boundary of the current block and a total of 2xnW samples adjacent to the upper right, and one sample adjacent to the upper left of the current block. Alternatively, the neighboring reference samples of the current block may include multiple upper neighboring samples and multiple left neighboring samples. In addition, the neighboring reference samples of the current block may include a total of nH samples adjacent to the right boundary of the current block having a size of nWxnH, a total of nW samples adjacent to the lower boundary of the current block, and one sample adjacent to the lower right of the current block.
[0090] However, some neighboring reference samples of the current block may not yet be decoded or available. In this case, the decoder can configure neighboring reference samples for prediction by replacing unavailable samples with available samples. Alternatively, the neighboring reference samples to be used for prediction can be configured by interpolation of available samples.
[0091] When deriving neighboring reference samples, (i) the prediction sample may be derived based on an average or interpolation of neighboring reference samples of the current block, and (ii) the prediction sample may be derived based on reference samples existing in a specific (prediction) direction for the prediction sample among peripheral reference samples of the current block. Case (i) may be referred to as a non-directional mode or a non-angular mode, and case (ii) may be referred to as a directional mode or an angular mode.
[0092] Further, based on the prediction samples of the current block in the neighboring reference samples, the prediction samples can be generated by interpolating between the first neighboring samples located in the prediction direction of the intra prediction mode of the current block and the second neighboring samples located in the opposite direction of the prediction direction. The above situation can be called linear interpolation intra prediction (LIP). In addition, a linear model can be used to generate chrominance prediction samples based on luma samples. This situation can be called LM mode.
[0093] In addition, the temporary prediction sample of the current block can be derived based on the filtered neighboring reference samples, and the prediction sample of the current block can be derived by weighted summing at least one reference sample derived according to the intra prediction mode among the existing neighboring reference samples (i.e., unfiltered neighboring reference samples) and the temporary prediction sample. The above situation can be called position-dependent intra prediction (PDPC).
[0094] In addition, a reference sample line with the highest prediction accuracy among the neighboring multiple reference sample lines of the current block can be selected to derive a prediction sample by using a reference sample in a prediction direction located on the corresponding line, and then the reference sample line used herein can be indicated (signaled) to a decoding device, thereby performing intra-frame prediction encoding. The above situation can be referred to as multi-reference line (MRL) intra-frame prediction or MRL-based intra-frame prediction.
[0095] In addition, intra prediction can be performed by dividing the current block into vertical or horizontal sub-partitions based on the same intra prediction mode, and neighboring reference samples can be derived and used in units of sub-partitions. That is, in this case, the intra prediction mode of the current block is also applicable to the sub-partitions, and in some cases, the intra prediction performance can be improved by deriving and using neighboring reference samples in units of sub-partitions. This prediction method can be called intra sub-partitioning (ISP) or intra prediction based on ISP.
[0096] The above-mentioned intra-frame prediction method may be referred to as an intra-frame prediction type separate from the intra-frame prediction mode. The intra-frame prediction type may be referred to in various terms, such as an intra-frame prediction technique or an additional intra-frame prediction mode. For example, the intra-frame prediction type (or additional intra-frame prediction mode) may include at least one of the above-mentioned LIP, PDPC, MRL, and ISP. General intra-frame prediction methods other than specific intra-frame prediction types such as LIP, PDPC, MRL, or ISP may be referred to as normal intra-frame prediction types. When a specific intra-frame prediction type is not applied, a normal intra-frame prediction type may generally be applied, and prediction may be performed based on the above-mentioned intra-frame prediction mode. At the same time, post-filtering may be performed on the derived prediction samples as needed.
[0097] Specifically, the intra prediction process may include an intra prediction mode / type determination step, a neighboring reference sample derivation step, and a prediction sample derivation step based on the intra prediction mode / type. In addition, a post-filtering step may be performed on the derived prediction samples as needed.
[0098] When intra prediction is applied, the intra prediction mode of the neighboring block can be used to determine the intra prediction mode applied to the current block. For example, the decoding device can select one of the mpm candidates in the mpm list derived from the intra prediction mode of the neighboring block (e.g., the left and / or upper neighboring block) of the current block based on the received most probable mode (mpm) index, and select one of the other remaining intra prediction modes not included in the mpm candidate (and the plane mode) based on the remaining intra prediction mode information. The mpm list can be configured to include or exclude the plane mode as a candidate. For example, if the mpm list includes the plane mode as a candidate, the mpm list may have six candidates. If the mpm list does not include the plane mode as a candidate, the mpm list may have five candidates. When the mpm list does not include the plane mode as a candidate, a non-planar flag (e.g., intra_luma_not_planar_flag) indicating whether the intra prediction mode of the current block is not a plane mode can be signaled. For example, the mpm flag can be signaled first, and when the value of the mpm flag is 1, the mpm index and the non-planar flag can be signaled. In addition, the mpm index may be signaled when the value of the non-planar flag is 1. Here, the mpm list is configured to not include the planar mode as a candidate does not intend to indicate that the planar mode is not mpm, but rather to first signal the non-planar flag to first check whether it is the planar mode, because the planar mode is always considered to be mpm.
[0099] For example, it can be indicated based on an mpm flag (e.g., Intra_luma_mpm_flag) whether the intra prediction mode applied to the current block is in the mpm candidate (and planar mode) or in the remaining mode. A value of 1 for the mpm flag can indicate that the intra prediction mode for the current block is within the mpm candidate (and planar mode), and a value of 0 for the mpm flag can indicate that the intra prediction mode for the current block is not in the mpm candidate (and planar mode). A value of 0 for a non-planar flag (e.g., Intra_luma_not_planar_flag) can indicate that the intra prediction mode of the current block is a planar mode, and a value of 1 for the non-planar flag value can indicate that the intra prediction mode of the current block is not a planar mode. The mpm index can be signaled in the form of an mpm_idx or intra_luma_mpm_idx syntax element, and the remaining intra prediction mode information can be signaled in the form of a rem_intra_luma_pred_mode or intra_luma_mpm_remainder syntax element. For example, the remaining intra prediction mode information may index the remaining intra prediction modes not included in the mpm candidates (and planar modes) in all intra prediction modes in the order of the prediction mode numbers to indicate one of them. The intra prediction mode may be an intra prediction mode for a luma component (sample). Hereinafter, the intra prediction mode information may include at least one of an mpm flag (e.g., Intra_luma_mpm_flag), a non-planar flag (e.g., Intra_luma_not_planar_flag), an mpm index (e.g., mpm_idx or intra_luma_mpm_idx), and remaining intra prediction mode information (rem_intra_luma_pred_mode or intra_luma_mpm_remainder). In this document, the MPM list may be referred to by a variety of terms, such as MPM candidate list and candModeList. When a MIP is applied to the current block, a separate mpm flag (eg, intra_mip_mpm_flag), mpm index (eg, intra_mip_mpm_idx), and remaining intra prediction mode information (eg, intra_mip_mpm_remainder) for the MIP may be signaled and the non-planar flag may not be signaled.
[0100] In other words, in general, when block segmentation is performed on an image, the current block to be coded and the neighboring blocks have similar image characteristics. Therefore, the probability that the current block and the neighboring blocks have the same or similar intra-frame prediction mode is high. Therefore, the encoder can use the intra-frame prediction mode of the neighboring block to encode the intra-frame prediction mode of the current block.
[0101] For example, the encoder / decoder may configure a list of most probable modes (MPMs) for the current block. The MPM list may also be referred to as an MPM candidate list. Here, MPM may refer to a mode that considers the similarity between the current block and the neighboring blocks in intra-frame prediction mode coding to improve coding efficiency. As described above, the MPM list may be configured to include a planar mode or may be configured to exclude a planar mode. For example, when the MPM list includes a planar mode, the number of candidates in the MPM list may be 6. If the MPM list does not include a planar mode, the number of candidates in the MPM list may be 5.
[0102] The encoder / decoder can be configured with an MPM list consisting of 5 or 6 MPMs.
[0103] To configure the MPM list, three modes can be considered: default intra mode, neighborhood intra mode and derived intra mode.
[0104] For the neighborhood intra mode, two neighboring blocks may be considered, namely the left neighboring block and the upper neighboring block.
[0105] As described above, if the MPM list is configured not to include the planar mode, the planar mode is excluded from the list and the number of MPM list candidates may be set to five.
[0106] Furthermore, the non-directional mode (or non-angular mode) among the intra prediction modes may include a DC mode based on an average of neighboring reference samples of the current block or a planar mode based on interpolation.
[0107] When inter-frame prediction is applied, the predictor of the encoding device / decoding device can derive prediction samples by performing inter-frame prediction in units of blocks. Inter-frame prediction can be a prediction derived in a manner that depends on data elements (e.g., sample values or motion information) of pictures other than the current picture. When inter-frame prediction is applied to the current block, the prediction block (prediction sample array) of the current block can be derived based on the reference block (reference sample array) specified by the motion vector on the reference picture indicated by the reference picture index. Here, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information of the current block can be predicted in units of blocks, sub-blocks or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter-frame prediction type (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. A temporally neighboring block may be referred to as a collocated reference block, a collocated CU (colCU), etc., and a reference picture including a temporally neighboring block may be referred to as a collocated picture (colPic). For example, a motion information candidate list may be configured based on a neighboring block of the current block, and a flag or index information indicating which candidate is selected (used) to derive a motion vector and / or a reference picture index of the current block may be signaled. Inter-frame prediction may be performed based on various prediction modes. For example, in the case of a skip mode and a merge mode, the motion information of the current block may be the same as the motion information of the neighboring block. In the skip mode, unlike the merge mode, a residual signal may not be sent. In the case of a motion vector prediction (MVP) mode, the motion vector of the selected neighboring block may be used as a motion vector predictor and the motion vector of the current block may be signaled. In this case, the motion vector of the current block may be derived using the sum of the motion vector predictor and the motion vector difference.
[0108] According to the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.), the motion information may include L0 motion information and / or L1 motion information. The motion vector in the L0 direction may be referred to as an L0 motion vector or MVL0, and the motion vector in the L1 direction may be referred to as an L1 motion vector or MVL1. Prediction based on the L0 motion vector may be referred to as an L0 prediction, prediction based on the L1 motion vector may be referred to as an L1 prediction, and prediction based on both the L0 motion vector and the L1 motion vector may be referred to as a dual prediction. Here, the L0 motion vector may indicate a motion vector associated with the reference picture list L0 (L0), and the L1 motion vector may indicate a motion vector associated with the reference picture list L1 (L1). The reference picture list L0 may include a picture earlier than the current picture in output order as a reference picture, and the reference picture list L1 may include a picture later than the current picture in output order as a reference picture. The previous picture may be referred to as a forward (reference) picture, and the subsequent picture may be referred to as a backward (reference) picture. The reference picture list L0 may further include a picture later than the current picture in output order as a reference picture. In this case, the previous picture may be indexed first in the reference picture list L0, and the subsequent picture may be indexed later. The reference picture list L1 may further include a picture before the current picture in the output order as a reference picture. In this case, the subsequent picture may be indexed first in the reference picture list 1, and the previous picture may be indexed later. The output order may correspond to a picture order count (POC) order.
[0109] Figure 4 : is a diagram for explaining a method of deriving motion information by an inter prediction mode. Here, in the inter prediction mode, a merge mode, a motion vector prediction (MVP) mode (or an advanced motion vector prediction (AMVP)), a pairwise average merge mode, and a history-based MVP (HMVP) mode are described as follows.
[0110] When the merge mode is applied, the motion information of the current prediction block is not sent directly, but the motion information of the neighboring prediction blocks is used to derive the motion information of the current prediction block. Therefore, the motion information of the current prediction block can be indicated by sending flag information indicating the use of the merge mode and a merge index indicating which neighboring prediction blocks are used.
[0111] The encoder must search for a merge candidate block for deriving motion information of the current prediction block to perform merge mode. For example, up to five merge candidate blocks can be used, but this document is not limited to this. The maximum number of merge candidate blocks can be sent in a slice header, a patch group header, or a parameter set (i.e., a sequence parameter set (SPS)), but this document is not limited to this. After finding the merge candidate block, the encoder can generate a merge candidate list and select the merge candidate block with the smallest cost among them as the final merge candidate block.
[0112] Various embodiments of configuring a merge candidate block of a merge candidate list are provided herein.
[0113] The merge candidate list may use, for example, five merge candidate blocks. For example, four spatial merge candidates and one temporal merge candidate may be used. As a specific example, in the case of spatial merge candidates, Figure 4 The blocks shown may be used as spatial merging candidates.
[0114] The merge candidate list for the current block may be configured based on, for example, the following process.
[0115] The coding device (encoder / decoder) inserts the spatial merge candidate derived by searching the spatial neighboring blocks of the current block into the merge candidate list (S310). For example, the spatial neighboring blocks may include the lower left neighboring block, the left neighboring block, the upper right neighboring block, the upper neighboring block, and the upper left neighboring block of the current block. However, this is merely an example, and in addition to the above-mentioned spatial neighboring blocks, additional neighboring blocks such as the right neighboring block, the lower neighboring block, and the lower right neighboring block may be used as spatial neighboring blocks. The coding device may search the spatial neighboring blocks based on priority to detect available blocks and derive the motion information of the detected blocks as spatial merge candidates. For example, the encoder and the decoder may search in the order of A1, B1, B0, A0, B2. Figure 4 The five blocks shown are indexed in turn and the available candidates are configured to configure a merge candidate list.
[0116] The compilation device inserts the time merge candidates derived by searching for temporally adjacent blocks of the current block into the merge candidate list (S320). The temporally adjacent blocks may be located on a reference picture that is a picture different from the current picture in which the current block is located. The reference picture on which the temporally adjacent blocks are located may be referred to as a collocated picture or a col picture. For the current block on the col picture, the temporally adjacent blocks may be searched in the order of the bottom-right adjacent block and the bottom-right center block of the collocated block. Meanwhile, when motion data compression is applied, specific motion information may be stored as representative motion information for each predetermined storage unit in the col picture. In this case, there is no need to store the motion information for all blocks in the predetermined storage unit, thereby achieving a motion data compression effect. In this case, the predetermined storage unit may be determined in advance, for example, as a 16×16 sample unit, an 8×8 sample unit, etc., or the size information about the predetermined storage unit may be signaled from the encoder to the decoder. When motion data compression is applied, the motion information of the temporally adjacent blocks may be replaced with the representative motion information of the predetermined storage unit in which the temporally adjacent blocks are located. That is to say, in this case, from an implementation perspective, based on the coordinates (top-left sample position) of the temporally adjacent blocks, a predetermined value is arithmetically shifted to the right, and then, based on the motion information of the prediction block covering the arithmetically left-shifted position, the time merge candidates may be derived. For example, in the case where the predetermined storage unit is a 2n×2n sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block located at the modified position ((xTnb>>n)<<n), (yTnb>>n)<<n)) may be used for the time merge candidates. Specifically, for example, in the case where the predetermined storage unit is a 16×16 sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block located at the modified position ((xTnb>>4)<<4), (yTnb>>4)<<4)) may be used for the time merge candidates. Or, for example, in the case where the predetermined storage unit is an 8×8 sample unit, if the coordinates of the temporally adjacent block are (xTnb, yTnb), the motion information of the prediction block located at the modified position ((xTnb>>3)<<3), (yTnb>>3)<<3)) may be used for the time merge candidates.
[0117] The coding device may determine whether the number of current merge candidates is less than the maximum number of merge candidates (the maximum number of candidates included in the merge candidate list) (S330). The maximum number of merge candidates may be predefined or signaled to the decoder from the encoder (i.e., through the patch group header or SPS). For example, the encoder may generate information about the maximum number of merge candidates (the maximum number of candidates included in the merge candidate list), encode the information, and send the encoded information to the decoder in the form of a bitstream. If the maximum number of merge candidates is filled, the subsequent candidate addition process may not be performed.
[0118] As a result of the check, if the number of current merge candidates is less than the maximum number of merge candidates, the compiling device inserts an additional merge candidate into the merge candidate list (S340). The additional merge candidate may include, for example, ATMVP and combined bidirectional prediction merge candidates (when the slice / patch type of the current slice / patch group is B) and / or a zero vector merge candidate.
[0119] As a result of the check, if the number of current merge candidates is not less than the number of maximum merge candidates, the coding device may terminate the construction of the merge candidate list. In this case, the encoder may select the best merge candidate among the merge candidates configuring the merge candidate list based on the rate-distortion (RD) cost, and signal the decoder with selection information (e.g., a merge index) indicating the selected merge candidate. The decoder may select the best merge candidate based on the merge candidate list and the selection information.
[0120] As described above, the motion information of the selected merge candidate may be used as the motion information of the current block, and the prediction sample of the current block may be derived based on the motion information of the current block. The encoder may derive the residual sample of the current block based on the prediction sample, and may signal the residual information about the residual sample to the decoder. The decoder may generate a reconstructed sample based on the residual sample and the prediction sample derived based on the residual information, and generate a reconstructed picture based thereon as described above.
[0121] When the skip mode is applied, the motion information of the current block may be derived in the same manner as the case where the merge mode is applied. However, when the skip mode is applied, the residual signal for the corresponding block is omitted, and thus the prediction sample may be used as the reconstructed sample.
[0122] When the motion vector prediction (MVP) mode is applied, the reconstructed spatial neighboring blocks (which may be Figure 4The motion vector predictor (MVP) candidate list is generated based on the motion vector of the reconstructed spatial neighboring block) and / or the motion vector of the temporal neighboring block (or Col block). That is, the motion vector corresponding to the reconstructed spatial neighboring block and / or the motion vector corresponding to the temporal neighboring block can be used as a motion vector predictor candidate. When bidirectional prediction is applied, an MVP candidate list for deriving L0 motion information and an MVP candidate list for deriving L1 motion information can be generated and used respectively. The above-mentioned prediction information (or information about prediction) may include selection information (such as an MVP flag or an MVP index) indicating the best motion vector predictor candidate selected from the motion vector predictor candidates included in the list. In this case, the predictor may use the selection information to select the motion vector predictor of the current block from the motion vector predictor candidates included in the motion vector candidate list. The predictor of the encoding device may obtain the motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, encode it, and output it in the form of a bitstream. That is, the MVD may be obtained as a value obtained by subtracting the motion vector predictor from the motion vector of the current block. In this case, the predictor of the decoding device may obtain the motion vector difference included in the information about the prediction and derive the motion vector of the current block by adding the motion vector difference and the motion vector predictor. The predictor of the decoding device may obtain or derive a reference picture index indicating the reference picture from the information about the prediction.
[0123] A history-based MVP (HMVP) merge candidate can be added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previous coded block is stored in a table and used as the MVP of the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a non-sub-block inter-coded CU, the relevant motion information is added to the last entry of the table as a new HMVP candidate.
[0124] The HMVP table size S is set to 6, which indicates that up to 6 history-based MVP (HMVP) candidates can be added to the table. When inserting a new motion candidate into the table, a constrained first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to find if there is an identical HMVP in the table. If found, the identical HMVP is deleted from the table, and then all subsequent HMVP candidates are moved forward.
[0125] HMVP candidates can be used in the merge candidate list construction process. Check the latest HMVP candidates in the table in turn and insert them into the candidate list after the TMVP candidate. Apply redundancy check on the HMVP candidate for spatial or temporal merge candidates.
[0126] In order to reduce the number of redundant check operations, the following simplifications are introduced:
[0127] 1) The number of HMPV candidates for merge list generation is set to (N<=4)?M:(8-N), where N indicates the number of existing candidates in the merge list and M indicates the number of HMVP candidates available in the table.
[0128] 2) Once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, the merge candidate list construction process from HMVP is terminated.
[0129] The pairwise average candidates are generated by averaging the predefined candidate pairs in the existing merge candidate list, and the predefined pairs are defined as {(0,1),(0,2),(1,2),(0,3),(1,3),(2,3)}, where the number represents the merge index of the merge candidate list. The average motion vector is calculated separately for each reference list. If both motion vectors are available in one list, they are averaged even if the two motion vectors point to different reference pictures; if only one motion vector is available, the one motion vector is used directly; if no motion vector is available, the list is kept invalid.
[0130] When the merge list is not full after adding pairwise average merge candidates, zero MVPs are inserted into the list until the maximum number of merge candidates is encountered.
[0131] In the following, it will be described that Figure 2 or Figure 3 A detailed description of IBC performed by a predictor of a device. IBC can be used for content image / video coding of games, etc., such as screen content coding (SCC). IBC basically performs prediction in the current picture, but in terms of deriving reference blocks in the current picture, it can be performed similarly to inter-frame prediction. That is, IBC can use at least one of the inter-frame prediction techniques described in this document. For example, IBC can apply at least one of the above methods to derive motion information (motion vector). IBC can refer to the current picture and can therefore be referred to as a current picture reference (CPR). When the above prediction method is applied in IBC, the motion vector (motion information) can generally be referred to as (replaced or mixed with) a block vector (block information).
[0132] For IBC, the encoding device may perform block matching (BM) to derive the best block vector (or motion vector) for the current block (e.g., CU). The derived block vector (or motion vector) may be signaled to the decoding device through the bitstream using a method similar to the block information (motion vector) signaling in the above-mentioned inter-frame prediction. The decoding device may derive a reference block for the current block in the current picture by the signaled block vector (motion vector), thereby driving the prediction signal (prediction block or prediction sample) for the current block. Here, the block vector (or motion vector) may represent the displacement from the current block to the reference block located in the reconstructed area in the current picture. Therefore, the block vector (or motion vector) may be referred to as a displacement vector. In the following, in IBC, a motion vector may correspond to a block vector or a displacement vector. The motion vector of the current block may include a motion vector for a luminance component (luminance motion vector) or a motion vector for a chrominance component (chrominance motion vector). For example, the luminance motion vector of a CU for IBC coding may be an integer sample unit (i.e., integer precision). The chrominance motion vector may also be clipped in integer sample units. As described above, IBC may use at least one of the inter-frame prediction techniques, for example, when IBC is applied as AMVR, 1-pel and 4-pel motion vector precision may be switched.
[0133] To reduce memory consumption and decoder complexity, only the reconstructed portion of the predefined area including the current CTU may be used. This restriction allows the IBC mode to be implemented using local on-chip memory for hardware implementation.
[0134] On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD checks on blocks with a width or height no greater than 16 luma samples. For non-merge mode, block vector search is first performed using a hash-based search. If the hash search does not return a valid candidate, a local search based on block matching is performed.
[0135] In hash-based search, hash key matching (32-bit CRC) between the current block and the reference blocks is extended to all allowed block sizes. The hash key calculation for each position in the current picture is based on 4x4 sub-blocks. For larger-sized current blocks, a hash key is determined to match the hash key of a reference block when all hash keys of all 4×4 sub-blocks match the hash key of the corresponding reference position. If multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference is calculated and the one with the smallest cost is selected.
[0136] In the block match search, the search range is set to N samples to the left and above the current block within the current CTU. At the beginning of the CTU, the value of N is initialized to 128 if there is no temporal reference picture, and to 64 if there is at least one temporal reference picture. The hash hit rate is defined as the percentage of samples in the CTU that find a match using a hash-based search. When encoding the current CTU, if the hash hit rate is less than 5%, N is halved.
[0137] At the CU level, the IBC mode is signaled using flag information (ie, flag information specifying whether IBC is applied, pred_mode_ibc_flag), and may be signaled as either the IBC AMVP mode or the IBC skip / merge mode, as shown below.
[0138] - IBC skip / merge mode: The merge candidate index is used to indicate which block vector from the list of neighboring candidate IBC coding blocks is used to predict the current block. The merge list consists of spatial, HMVP and pairwise candidates.
[0139] - IBC AMVP mode: Block vector differences are coded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the upper neighbor (if IBC coding is done). When either neighbor is not available, the default block vector will be used as the predictor. A flag is signaled to indicate the block vector predictor index.
[0140] Figure 5 and Figure 6 The diagram illustrates an image encoding method based on the IBC prediction mode and a predictor of an encoding device that executes the image encoding method.
[0141] The encoding device may derive a prediction mode and a motion vector of the current block, and generate a prediction sample of the current block (S500). The prediction mode may include at least one of the above-mentioned inter-frame prediction modes. Here, the prediction mode determination, motion vector derivation, and prediction sample generation processes may be performed simultaneously, or one process may be performed before another. For example, the predictor of the encoding device may include a prediction mode determiner, a motion vector deriver, and a prediction sample deriver. The prediction mode determiner may determine the prediction mode of the current block, the motion vector deriver may derive the motion vector of the current block, and the prediction sample deriver 183 may derive the prediction sample of the current block. As described above, the motion vector may be referred to as a block vector. For example, the predictor of the encoding device may search for a block similar to the current block in a reconstruction area (or a predetermined area (search area)) by block matching (BM), and derive a reference block whose difference with the current block is a minimum value or a predetermined reference or less. The predictor may derive a motion vector based on the displacement difference between the reference block and the current block. The encoding device may determine a mode applied to the current block in various prediction modes. The encoding apparatus may compare RD costs based on various prediction modes and determine an optimal prediction mode for the current block.
[0142] For example, when the skip mode or merge mode is applied to the current block, the encoding device may configure the above-mentioned merge candidate list and derive a reference block whose difference with the current block is equal to or less than a minimum value or a predetermined standard. In this case, a merge candidate associated with the derived reference block may be selected, and merge index information indicating the selected merge candidate may be generated and signaled to the decoding device. The motion vector of the current block may be derived using the motion vector of the selected merge candidate.
[0143] As another example, when the (A)MVP mode is applied to the current block, the encoding device may configure the above-mentioned (A)MVP candidate list and use the motion vector of the MVP candidate selected from the MVP (motion vector predictor) candidates included in the (A)MVP candidate list as the MVP of the current block. In this case, for example, the motion vector indicating the reference block derived by the above-mentioned motion estimation may be used as the motion vector of the current block, and among the MVP candidates, the MVP candidate whose motion vector has the smallest difference with the motion vector of the current block may be the selected MVP candidate. A motion vector difference (MVD) as a difference from which the MVP is subtracted may be derived from the motion vector of the current block. In this case, information about the MVD may be signaled to the decoding device.
[0144] The encoding apparatus may derive residual samples based on the prediction samples (S510). The encoding apparatus may derive residual samples by comparing original samples of the current block with the prediction samples.
[0145] The encoding device encodes the image information including prediction information and residual information (S520). The encoding device can output the encoded image information in the form of a bitstream. The prediction information may include a prediction mode vector (i.e., a skip flag, a merge flag, or a mode index) and information about a motion vector as information related to the prediction process. The information about the motion vector may include candidate selection information (e.g., a merge index, an mvp flag, or an mvp index) as information for deriving the motion vector. In addition, the information about the motion vector may include information about the above-mentioned MVD. The information about the motion vector may include information indicating whether L0 prediction, L1 prediction, or bi prediction is applied. The residual information is information about the residual sample. The residual information may include information about the quantized transform coefficients of the residual sample.
[0146] The output bitstream may be stored in a (digital) storage medium and delivered to the decoding device, or may be delivered to the decoding device via a network.
[0147] At the same time, as described above, the encoding device can generate a reconstructed picture (including a reconstructed sample and a reconstructed block) based on the reference sample and the residual sample. This is because the encoding device is to derive the same prediction result as the prediction result performed in the decoding device, so the coding efficiency can be improved. Therefore, the encoding device can store the reconstructed picture (or reconstructed sample, reconstructed block) in a memory and use it as a reference picture for inter-frame prediction.
[0148] Figure 7 and Figure 8 The diagram illustrates an image decoding method based on the IBC prediction mode and a predictor of a decoding device that performs the image decoding method.
[0149] The decoding device may perform operations corresponding to those performed in the encoding device. The decoding device may perform IBC prediction on the current block and derive a prediction sample based on the received prediction information.
[0150] Specifically, the decoding apparatus may determine a prediction mode of the current block based on the received prediction information (S700). The decoding apparatus may determine which inter prediction mode to apply to the current block based on prediction mode information in the prediction information.
[0151] For example, the decoding device may determine whether to apply the merge mode to the current block or whether the (A)MVP mode is determined based on the merge flag. Alternatively, one of the various inter-frame prediction mode candidates may be selected based on the mode index. The inter-frame prediction mode candidates may include skip mode, merge mode, and / or (A)MVP mode, or may include a combination of Figure 4 Describes the various inter prediction modes.
[0152] The decoding device derives the motion vector of the current block based on the determined prediction mode (S710). As described above, the motion vector may be referred to as a block vector. For example, when the skip mode or merge mode is applied to the current block, the decoding device may configure the merge candidate list as described above and select one of the merge candidates included in the merge candidate list. The selection may be performed based on the above selection information (merge index). The motion vector of the selected merge candidate may be used to derive the motion vector of the current block. The motion vector of the selected merge candidate may be used as the motion vector of the current block.
[0153] As another example, when the (A)MVP mode is applied to the current block, the decoding device may configure the (A)MVP candidate list as described above and use the motion vector of the mvp candidate selected from the mvp candidates included in the (A)MVP candidate list as the mvp of the current block. The selection may be performed based on the above-mentioned selection information (mvp flag or mvp index). In this case, the MVD of the current block may be derived based on the information about the MVD, and the motion vector of the current block may be derived based on the mvp and MVD of the current block. In addition, the reference picture index of the current block may be derived based on the reference picture index information. The picture indicated by the reference picture index in the reference picture list of the current block may be derived as the reference picture referenced by the inter-frame prediction of the current block.
[0154] At the same time, as described above, the motion vector of the current block can be derived without configuring the candidate list, and in this case, the motion vector of the current block can be derived according to the process disclosed in the corresponding prediction mode. In this case, the configuration of the candidate list as described above can be omitted.
[0155] The decoding device may generate a prediction sample of the current block based on the motion vector of the current block (S720). The decoding device may derive the prediction sample of the current block using the sample of the reference block indicated by the motion vector of the current block on the current picture. In this case, a prediction sample filtering process may be further performed on all or some of the prediction samples of the current block.
[0156] For example, the predictor of the decoding device may include a prediction mode determiner, a motion vector deriver, and a prediction sample deriver. The prediction mode of the current block may be determined based on the prediction mode vector received from the prediction mode determiner, the motion vector of the current block may be derived based on the information about the motion vector received from the motion vector deriver 262, and the prediction sample deriver may derive the prediction sample of the current block.
[0157] The decoding device generates residual samples of the current block based on the received residual information (S730). The decoding device can generate reconstructed samples for the current block based on the predicted samples and the residual samples, and generate a reconstructed picture based on this (S740). Thereafter, an in-loop filtering process or the like can be further applied to the reconstructed picture as described above.
[0158] Embodiments related to merge mode and skip mode in IBC prediction mode will be described together with the following table. Specifically, a method for determining the maximum number of merge mode candidates derived for finding a motion vector of a current block when using IBC is shown.
[0159] The following table is an exemplary syntax for a tile group header.
[0160] [Table 1]
[0161]
[0162] In the next generation video coding technology, when the tile group type (tile_group_type) is P (unidirectional prediction) or B (bidirectional prediction), the maximum number of general merge mode candidates is determined by the syntax element six_minus_max_num_merge_cand, and the maximum number of merge mode candidates for IBC can be determined by the same syntax element.
[0163] When the tile group type (tile_group_type) is I (intra prediction), there is no general merge mode, so the same syntax six_minus_max_num_merge_cand is sent to determine the maximum number of merge mode candidates for IBC.
[0164] The IBC technology is a technology for performing motion compensation within a picture, and may have different characteristics from the existing inter-prediction merge mode, and the method for constructing merge mode candidates may be different. Therefore, using the same number of merge candidates may be inefficient in terms of compression performance.
[0165] In one example, the syntax of a tile group header may be as shown in the following table.
[0166] [Table 2]
[0167]
[0168] The semantics of the syntax elements included in the syntax of Table 2 may be expressed, for example, as shown in the following table.
[0169] [Table 3]
[0170]
[0171] In another example, the syntax of a tile group header may be as shown in the following table.
[0172] [Table 4]
[0173]
[0174] The semantics of the syntax elements included in the syntax of Table 4 may be expressed, for example, as shown in the following table.
[0175] [Table 5]
[0176]
[0177] In another example, the syntax of a tile group header may be as shown in the following table.
[0178] [Table 6]
[0179]
[0180] The semantics of the syntax elements included in the syntax of Table 6 may be expressed, for example, as shown in the following table.
[0181] [Table 7]
[0182]
[0183] In another example, the syntax of a tile group header may be as shown in the following table.
[0184] [Table 8]
[0185]
[0186] The semantics of the syntax elements included in the syntax of Table 8 may be expressed, for example, as shown in the following table.
[0187] [Table 9]
[0188]
[0189] In another example, the syntax of a tile group header may be as shown in the following table.
[0190] [Table 10]
[0191]
[0192] The semantics of the syntax elements included in the syntax of Table 10 may be expressed, for example, as shown in the following table.
[0193] [Table 11]
[0194]
[0195] When the above example is described in a standard document format, it can be expressed as follows, the details of which will be apparent to those skilled in the art.
[0196] [Table 12]
[0197]
[0198]
[0199]
[0200] The accompanying drawings are created to explain the specific examples of this document. Since the names of specific devices or the names of specific signals / messages / fields described in the accompanying drawings are presented by way of example, the technical features of this document are not limited to the specific names used in the accompanying drawings.
[0201] Fig. 9 and Fig.10 An example of a video / image encoding method and related components according to embodiments of the present invention is schematically shown. Fig. 9 The method disclosed in can be Figure 2 Specifically, for example, Fig. 9 S900 and S910 may be performed by the predictor 220 of the encoding device, and Fig. 9 S920 and S930 may be performed by the entropy encoder 240 of the encoding device. Fig. 9 The method disclosed in may include the above-mentioned embodiments in this article.
[0202] Reference Fig. 9 , the encoding device may derive the prediction mode of the current block in the current picture as an IBC prediction mode (S900). The IBC prediction mode may include an IBC merge mode, an IBC (A) MVP mode, an IBC HMVP mode, and an IBC pairwise average merge mode. In an example, the IBC prediction mode may be an IBC merge mode (merge mode for IBC).
[0203] The encoding device may derive a candidate list for the IBC prediction mode (S910). The candidate list for the IBC prediction mode may include block vector candidates indicating a reference block in the current picture. In one example, the candidate list for the IBC prediction mode may be a merged candidate list for the IBC prediction mode. Table 12 above shows mvL[0] and mvL[1] as exemplary candidate lists.
[0204] The encoding device may generate index information indicating (indicating, specifying) a block vector of the current block based on the candidate list for the IBC prediction mode (S920). For example, the index information may be merge_idx[xCb][yCb] included in Table 12 above.
[0205] The encoding device may encode the video / image information (S930). The image / video information may include index information indicating a block vector of the current block. The information used to generate the luma sample may include, for example, prediction-related information (prediction mode information) and residual information. The prediction-related information includes information about various prediction modes (i.e., merge mode of inter-frame prediction, MVP mode of inter-frame prediction, IBC prediction mode (IBC merge mode, IBC (A) MVP mode, IBC HMVP mode), etc.), MVD information, information about the candidate list, information about the maximum number of block vectors included in the candidate list for the IBC prediction mode, information about the maximum number of block vectors included in the candidate list for the IBC merge mode, inter-frame prediction mode information about the maximum number of block vectors included in the candidate list, information about the maximum number of motion vectors included in the candidate list for the merge mode for inter-frame prediction, information about block vectors (motion vectors), etc. In addition, the image information may include flag information (i.e., pred_mode_ibc_flag) indicating whether the IBC prediction mode is applied.
[0206] In one embodiment, the image information may include a sequence parameter set (SPS), and the SPS includes an IBC enable flag that specifies whether the IBC prediction mode is enabled. For example, when the value of the IBC enable flag is 1, information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode may be generated.
[0207] In one embodiment, the maximum number of block vector candidates included in the candidate list for the IBC prediction mode may be derived based on a difference between 6 and a value of information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode. For example, the maximum number of block vector candidates included in the candidate list for the IBC prediction mode may be derived based on an equation (MaxNumibcMergeCand=6-six_minus_max_num_ibc_merge_cand) included in Table 3. In the equation, MaxNumibcMergeCand may represent the maximum number of block vector candidates included in the candidate list for the IBC prediction mode, and six_minus_max_num_ibc_merge_cand may represent the value of information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode.
[0208] In one embodiment, the image information may include information about the maximum number of motion vector candidates included in the candidate list for the inter-frame prediction mode, and the information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode (e.g., six_minus_max_num_merge_cand) may be different from the information about the maximum number of motion vector candidates included in the candidate list for the inter-frame prediction mode (e.g., six_minus_max_num_ibc_merge_cand).
[0209] In one embodiment, when the number of block vector candidates included in the candidate list for the IBC prediction mode is less than the maximum number and greater than 1, a derivation process of a pairwise average merge candidate may be performed. Alternatively, when the number of block vector candidates included in the candidate list for the IBC prediction mode is less than the maximum number (or the maximum number - 1, MaxNumMergeCand-1) and the number of history-based motion vector prediction (HMVP) candidates is greater than 0, a derivation process of a history-based merge candidate may be performed.
[0210] The coded image / video information may be output in the form of a bit stream. The bit stream may be transmitted to a decoding device via a network or a (digital) storage medium.
[0211] According to an embodiment of the present invention, the image information may include various information. For example, the image information may include at least one information or at least one syntax element disclosed in at least one of Tables 1, 2, 4, 6, 8, and 10 above.
[0212] In one example, the encoding device may derive residual samples based on the predicted samples and the original samples. In this case, residual information may be derived based on the residual samples. Reconstructed samples may be generated based on the residual information. Reconstructed blocks and reconstructed pictures may be derived based on the reconstructed samples.
[0213] Fig.11 and Fig.12 An example of an image / video decoding method and related components according to embodiments of the present invention is schematically shown. Fig.11 The method disclosed in can be Figure 3 The decoding device shown is executed. Specifically, for example, Fig.11 S1100 may be performed by the entropy decoder 310 of the decoding device, S1110 to S1140 may be performed by the predictor 330 of the decoding device, and S1150 may be performed by the adder 340 of the decoding device. Fig.11 The method disclosed in may include the above-mentioned embodiments in this article.
[0214] Reference Fig.11, the decoding device can receive / obtain image / video information. For example, the decoding device can receive / obtain image / video information through a bitstream. The decoding device can obtain index information for deriving a block vector through a bitstream (S1100). For example, the index information can be merge_idx[xCb][yCb] included in Table 12 above. In addition, the decoding device can further obtain flag information (i.e., pred_mode_ibc_flag) indicating whether the IBC prediction mode is applied through the bitstream.
[0215] According to an embodiment of the present invention, the image / video information may include various information. For example, the image / video information may include the information disclosed in at least one of Tables 1, 2, 4, 6, 8, and 10 above.
[0216] The decoding device may derive the prediction mode of the current block in the current picture as an IBC prediction mode (S1110). The IBC prediction mode may include an IBC merge mode, an IBC (A) MVP mode, an IBC HMVP mode, and an IBC pairwise average merge mode. In one example, the IBC prediction mode may be an IBC merge mode (merge mode for IBC).
[0217] The decoding device may derive a candidate list for the IBC prediction mode (S1120). The candidate list for the IBC prediction mode may include block vector candidates indicating a reference block in the current picture. In one example, the candidate list for the IBC prediction mode may be a merged candidate list for the IBC prediction mode. Table 12 above shows mvL[0] and mvL[1] as exemplary candidate lists.
[0218] The decoding apparatus may derive a block vector for the current block based on the candidate list for the IBC prediction mode and the index information ( S1130 ).
[0219] The decoding device may generate a prediction sample of the current block based on the block vector (S1140). The prediction sample of the current block may be generated based on a reference sample in a reference block indicated by the block vector. Here, the reference block may be included in the current picture.
[0220] The decoding device may generate a reconstructed sample based on the predicted sample (S1150). The decoding device may obtain residual information from the image information and may derive the residual sample based on the residual information. The decoding device may generate (derive) the reconstructed sample based on the residual sample and the predicted sample. For example, the decoding device may generate (derive) the reconstructed sample by adding the residual sample to the predicted sample.
[0221] In one embodiment, the image information may include a sequence parameter set (SPS), and the SPS includes an IBC enable flag that specifies whether the IBC prediction mode is enabled. For example, when the value of the IBC enable flag is 1, information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode may be signaled / parsed.
[0222] In one embodiment, the maximum number of block vector candidates included in the candidate list for the IBC prediction mode may be derived based on a difference between 6 and a value of information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode. For example, the maximum number of block vector candidates included in the candidate list for the IBC prediction mode may be derived based on the equation (MaxNumibcMergeCand=6-six_minus_max_num_ibc_merge_cand) included in Table 3. In the equation, MaxNumibcMergeCand may represent the maximum number of block vector candidates included in the candidate list for the IBC prediction mode, and six_minus_max_num_ibc_merge_cand may represent the value of information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode.
[0223] In one embodiment, the image information may include information about the maximum number of motion vector candidates included in the candidate list for the inter-frame prediction mode, and the information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode (e.g., six_minus_max_num_merge_cand) may be different from the information about the maximum number of motion vector candidates included in the candidate list for the inter-frame prediction mode (e.g., six_minus_max_num_ibc_merge_cand).
[0224] In one embodiment, when the number of block vector candidates included in the candidate list for the IBC prediction mode is less than the maximum number and greater than 1, a derivation process of a pairwise average merge candidate may be performed. Alternatively, when the number of block vector candidates included in the candidate list for the IBC prediction mode is less than the maximum number (or the maximum number - 1, MaxNumMergeCand-1) and the number of history-based motion vector prediction (HMVP) candidates is greater than 0, a derivation process of a history-based merge candidate may be performed.
[0225] In the above-mentioned embodiment, the method is described based on a flow chart with a series of steps or boxes. The present disclosure is not limited to the order of the above-mentioned steps or boxes. Some steps or boxes may occur simultaneously with other steps or boxes as described above or in a different order. In addition, it will be understood by those skilled in the art that the steps shown in the above-mentioned flow chart are not exclusive, and further steps may be included, or one or more steps in the flow chart may be deleted without affecting the scope of the present disclosure.
[0226] The methods according to the above-described embodiments of the present invention may be implemented in software form, and the encoding device and / or decoding device according to the present invention may be included in a device that performs image processing for a TV, computer, smartphone, set-top box, display device, etc., for example.
[0227] When the embodiments herein are implemented in software, the above methods can be implemented as modules (processes, functions, etc.) that perform the above functions. The modules can be stored in a memory and executed by a processor. The memory can be inside or outside the processor and can be coupled to the processor in various well-known ways. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, and / or data processing devices. The memory may include a read-only memory (ROM), a random access memory (RAM), a flash memory, a memory card, a storage medium, and / or other storage devices. That is, the embodiments described herein can be implemented and executed on a processor, a microprocessor, a controller, or a chip. For example, the functional units shown in the figures can be implemented and executed on a computer, a processor, a microprocessor, a controller, or a chip. In this case, information about instructions or algorithms for implementation can be stored in a digital storage medium.
[0228] In addition, the decoding device and the encoding device to which the present disclosure is applied may be included in a multimedia broadcast transmission / reception device, a mobile communication terminal, a home theater video device, a digital theater video device, a surveillance camera, a video chat device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camera, a VoD service provider, an over-the-top (OTT) video device, an Internet streaming service provider, a three-dimensional (3D) video device, a teleconferencing video device, a traffic user device (i.e., a vehicle user device, an aircraft user device, a ship user device, etc.) and a medical video device, and may be used to process video signals and data signals. For example, an over-the-top (OTT) video device may include a game console, a Blu-ray player, an Internet access TV, a home theater system, a smart phone, a tablet computer, a digital video recorder (DVR), etc.
[0229] In addition, the processing method of the present invention can be generated in the form of a program to be executed by a computer and can be stored in a computer-readable recording medium. Multimedia data with a data structure according to the present disclosure can also be stored in a computer-readable recording medium. Computer-readable recording media include all types of storage devices in which data that can be read by a computer system is stored. For example, computer-readable recording media may include BD, universal serial bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. In addition, computer-readable recording media include media implemented in carrier form (i.e., transmitted over the Internet). In addition, the bit stream generated by the encoding method can be stored in a computer-readable recording medium or can be sent via a wired / wireless communication network.
[0230] In addition, the embodiments of the present invention may be implemented using a computer program product according to a program code, and the program code may be executed in a computer through the embodiments of the present invention. The program code may be stored on a carrier that can be read by a computer.
[0231] Fig.13 An example of a content streaming system to which the embodiments disclosed herein may be applied is shown.
[0232] Reference Fig.13 The content streaming transmission system to which the embodiments of this document are applied may mainly include an encoding server, a streaming transmission server, a network server, a media storage, a user device, and a multimedia input device.
[0233] The encoding server compresses the content input from a multimedia input device such as a smartphone, a camera, or a camcorder into digital data to generate a bitstream and sends the bitstream to the streaming server. For another example, when a multimedia input device such as a smartphone, a camera, or a camcorder directly generates a bitstream, the encoding server can be omitted.
[0234] A bitstream may be generated by applying the encoding method or the bitstream generating method of the embodiment of the present disclosure, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.
[0235] The streaming server sends multimedia data to the user device based on the user's request through the network server, and the network server is used as a medium to inform the user of the service. When the user requests the required service from the network server, the network server passes it to the streaming server, and the streaming server sends the multimedia data to the user. In this case, the content streaming system may include a separate control server. In this case, the control server is used to control the command / response between the devices in the content streaming system.
[0236] The streaming server may receive content from a media storage and / or encoding server. For example, when content is received from an encoding server, the content may be received in real time. In this case, in order to provide a smooth streaming service, the streaming server may store the bitstream for a predetermined time.
[0237] Examples of user devices may include mobile phones, smart phones, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, tablet PCs, tablet PCs, ultrabooks, wearable devices (e.g., smart watches, smart glasses, head-mounted displays), digital televisions, desktop computers, digital signage, etc. Each server in the content streaming system may be operated as a distributed server, in which case data received from each server may be distributed.
[0238] Each server in the content streaming system may be operated as a distributed server, in which case data received from each server may be distributed and processed.
[0239] The claims described herein may be combined in various ways. For example, the technical features of the method claims herein may be combined and implemented as a device, and the technical features of the device claims herein may be combined and implemented as a method. In addition, the technical features of the method claims herein and the technical features of the device claims herein may be combined to be implemented as a device, and the technical features of the method claims herein and the technical features of the device claims herein may be combined and implemented as a method.
Claims
1. A method for decoding an image performed by a decoding device, the method include: Obtaining image information through a bitstream, the image information including flag information for an intra block copy (IBC) prediction mode and index information for deriving a block vector; deriving a prediction mode for a current block in a current picture as the IBC prediction mode based on the flag information; deriving a candidate list for the IBC prediction mode, wherein the candidate list includes block vector candidates; deriving the block vector for the current block based on the candidate list for the IBC prediction mode and the index information; Based on the block vector, generating a prediction sample of the current block; and Based on the predicted samples, reconstructed samples are generated, wherein the image information includes information about a maximum number of block vector candidates included in the candidate list for the IBC prediction mode and information about a maximum number of motion vector candidates included in a candidate list for an inter prediction mode, wherein the information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode is different from the information about the maximum number of motion vector candidates included in the candidate list for the inter prediction mode, and Therein, a maximum value for the maximum number of block vector candidates included in the candidate list for the IBC prediction mode is predetermined to be 6.
2. The method according to claim 1, in, Based on a difference between 6 and a value of the information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode, the maximum number of block vector candidates included in the candidate list for the IBC prediction mode is derived.
3. The method according to claim 2, in, The maximum number of the block vector candidates included in the candidate list for the IBC prediction mode is derived based on the following equation, MaxNumibcMergeCand=6-six_minus_max_num_ibc_merge_candwhere, MaxNumibcMergeCand represents the maximum number of the block vector candidates included in the candidate list for the IBC prediction mode, and six_minus_max_num_ibc_merge_cand represents the value of information on the maximum number of the block vector candidates included in the candidate list for the IBC prediction mode.
4. The method according to claim 1, in, Based on a case where the number of block vector candidates included in the candidate list for the IBC prediction mode is less than the maximum number and greater than 1, a derivation process of pairwise average merging candidates is performed.
5. The method according to claim 1, in, Based on the case where the number of block vector candidates included in the candidate list for the IBC prediction mode is less than the maximum number and the number of history-based motion vector prediction (HMVP) candidates is greater than 0, a history-based merge candidate derivation process is performed.
6. A method for encoding an image performed by an encoding device, the method include: deriving a prediction mode for a current block in a current picture as an intra block coding (IBC) prediction mode; generating flag information for the IBC prediction mode; deriving a candidate list for the IBC prediction mode, wherein the candidate list includes block vector candidates for a reference block in the current picture; generating index information of a block vector for the current block based on the candidate list for the IBC prediction mode; as well as encoding the image information including the flag information and the index information, wherein the image information includes information about a maximum number of block vector candidates included in the candidate list for the IBC prediction mode and information about a maximum number of motion vector candidates included in a candidate list for an inter prediction mode, wherein the information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode is different from the information about the maximum number of motion vector candidates included in the candidate list for the inter prediction mode, and Therein, a maximum value for the maximum number of block vector candidates included in the candidate list for the IBC prediction mode is predetermined to be 6.
7. The method according to claim 6, in, Based on a difference between 6 and a value of the information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode, the maximum number of block vector candidates included in the candidate list for the IBC prediction mode is derived.
8. The method according to claim 7, in, The maximum number of the block vector candidates included in the candidate list for the IBC prediction mode is derived based on the following equation, MaxNumibcMergeCand=6-six_minus_max_num_ibc_merge_candwhere, MaxNumibcMergeCand represents the maximum number of the block vector candidates included in the candidate list for the IBC prediction mode, and six_minus_max_num_ibc_merge_cand represents the value of information on the maximum number of the block vector candidates included in the candidate list for the IBC prediction mode.
9. The method according to claim 6, in, Based on a case where the number of block vector candidates included in the candidate list for the IBC prediction mode is less than the maximum number and greater than 1, a derivation process of pairwise average merging candidates is performed.
10. The method according to claim 6, in, Based on the case where the number of block vector candidates included in the candidate list for the IBC prediction mode is less than the maximum number and the number of history-based motion vector prediction (HMVP) candidates is greater than 0, a history-based merge candidate derivation process is performed.
11. A non-transitory computer-readable digital storage medium storing a bit stream of image information generated by a method, the method include: deriving a prediction mode for a current block in a current picture as an intra block coding (IBC) prediction mode; generating flag information for the IBC prediction mode; deriving a candidate list for the IBC prediction mode, wherein the candidate list includes block vector candidates for a reference block in the current picture; generating index information of a block vector for the current block based on the candidate list for the IBC prediction mode; as well as encoding image information to generate the bit stream, wherein the image information includes the flag information and the index information, wherein the image information includes information about a maximum number of block vector candidates included in the candidate list for the IBC prediction mode and information about a maximum number of motion vector candidates included in a candidate list for an inter prediction mode, wherein the information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode is different from the information about the maximum number of motion vector candidates included in the candidate list for the inter prediction mode, and Therein, a maximum value for the maximum number of block vector candidates included in the candidate list for the IBC prediction mode is predetermined to be 6.
12. A method for transmitting data for image information, include: Obtaining a bitstream of the image information, wherein the bitstream is generated based on: deriving a prediction mode for a current block in a current picture as an intra block coding (IBC) prediction mode; generating flag information for the IBC prediction mode; deriving a candidate list for the IBC prediction mode, wherein the candidate list includes block vector candidates for a reference block in the current picture; generating index information of a block vector for the current block based on the candidate list for the IBC prediction mode; and encoding the image information including the flag information and the index information; and sending said data comprising said bit stream, wherein the image information includes information about a maximum number of block vector candidates included in the candidate list for the IBC prediction mode and information about a maximum number of motion vector candidates included in a candidate list for an inter prediction mode, wherein the information about the maximum number of block vector candidates included in the candidate list for the IBC prediction mode is different from the information about the maximum number of motion vector candidates included in the candidate list for the inter prediction mode, and Therein, a maximum value for the maximum number of block vector candidates included in the candidate list for the IBC prediction mode is predetermined to be 6.
