Intra prediction-based image coding method and apparatus using unified MPM list
By using an intra-frame prediction method with a unified MPM list, the problem of high transmission and storage costs for high-resolution images and videos is solved, coding efficiency is improved and complexity is reduced, and it is applicable to various image compilation techniques.
Patent Information
- Application Number
- CN202411130592.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2019-02-26
- Filing Date
- 2020-02-26
- Publication Date
- 2024-11-12
AI Technical Summary
Existing technologies suffer from high transmission and storage costs due to increased information volume when transmitting and storing high-resolution, high-quality images and videos. This is especially true when transmitting immersive media such as virtual reality and holograms, where the image characteristics differ from real images, resulting in low compression efficiency.
An intra-frame prediction method using a unified most probable mode (MPM) list is proposed. By constructing a candidate intra-frame prediction mode list, prediction samples are generated and image encoding is performed. This method supports normal intra-frame prediction, multi-reference line intra-frame prediction, and sub-partition intra-frame prediction, thereby improving coding efficiency.
By simplifying the intra-frame prediction structure and improving prediction capabilities, the implementation complexity is reduced, thereby enhancing the compression efficiency of images and videos and reducing transmission and storage costs.
Smart Images

Figure CN118945332A_ABST
Abstract
Description
[0001] This application is a divisional application of the patent application with application number 202080016567.1 (PCT / KR2020 / 002764) filed on August 25, 2021, application date on February 26, 2020, and titled "Image coding method and device based on intra-frame prediction using a unified MPM list". Technical Field
[0002] The present disclosure relates to an image coding technology, and more particularly, to an intra-prediction based image coding method and apparatus using a unified most probable mode (MPM) list. Background Art
[0003] Recently, there is an increasing demand for high-resolution and high-quality images and videos such as ultra-high-definition (HUD) images and 4K or 8K or larger videos in various fields. As image and video data become high-resolution and high-quality, the amount of information or the number of bits transmitted relative to existing image and video data increases. Therefore, if a medium such as an existing wired or wireless broadband line is used to transmit image data or an existing storage medium is used to store image and video data, the transmission cost and storage cost increase.
[0004] In addition, there has been an increasing interest and demand for immersive media such as virtual reality (VR), artificial reality (AR) content, or holograms recently. Broadcasting of images and videos such as game images whose image characteristics are different from those of real images is increasing.
[0005] Therefore, in order to effectively compress and transmit or store and play back information of high-resolution and high-quality images and videos having such various characteristics, efficient image and video compression technology is required. Summary of the invention
[0006] The technical purpose of this specification is to provide a method and device for enhancing image coding efficiency.
[0007] Another technical purpose of the present specification is to provide a method and apparatus for performing efficient intra-frame prediction.
[0008] Another technical objective of the present specification is to provide a method and device for performing image coding, which can derive a unified MPM list for efficient intra-frame prediction.
[0009] Another technical purpose of the present specification is to provide a method and device for performing image coding, which can derive a unified MPM list for normal intra prediction, multi-reference line intra prediction, and sub-partition intra prediction.
[0010] According to an embodiment of the present document, provided herein is an image decoding method performed by a decoding device. The method may include the following steps: obtaining reference line index information representing a reference line for intra prediction of a current block from a bitstream, constructing a most probable mode (MPM) list by deriving a candidate intra prediction mode of the current block based on the value of the reference line index information, deriving an intra prediction mode for the current block based on the MPM list, generating a prediction sample for the current block based on the intra prediction mode, and generating a reconstructed sample for the current block based on the prediction sample. Here, based on the value of the reference line index information not being equal to 0, one of the candidate intra prediction modes may be a DC mode, and the number of candidate intra prediction modes including the DC mode may be equal to 5.
[0011] According to another embodiment of the present document, provided herein is an image encoding method performed by an encoding device. The method may include the following steps: generating reference line index information representing a reference line for intra prediction of a current block, constructing a most probable mode (MPM) list by deriving a candidate intra prediction mode of the current block based on the value of the reference line index information, deriving an intra prediction mode for the current block based on the MPM list, generating a prediction sample for the current block based on the intra prediction mode, generating residual information for the current block based on the prediction sample, and encoding image information including the reference line index information and the residual information. Here, based on the value of the reference line index information not being equal to 0, one of the candidate intra prediction modes may be a DC mode, and the number of candidate intra prediction modes including the DC mode may be equal to 5.
[0012] According to another embodiment of the present document, provided herein is a digital storage medium storing encoded image information, which enables a decoding device to perform an image decoding method. The image decoding method may include the following steps: obtaining reference line index information representing a reference line for intra-frame prediction of a current block from a bitstream, constructing a most probable mode (MPM) list by deriving a candidate intra-frame prediction mode of the current block based on the value of the reference line index information, deriving an intra-frame prediction mode for the current block based on the MPM list, generating a prediction sample for the current block based on the intra-frame prediction mode, and generating a reconstructed sample for the current block based on the prediction sample. Here, based on the value of the reference line index information not being equal to 0, one of the candidate intra-frame prediction modes may be a DC mode, and the number of candidate intra-frame prediction modes including the DC mode may be equal to 5.
[0013] According to the present specification, the overall image / video compression efficiency can be enhanced.
[0014] According to the present specification, overall coding efficiency may be enhanced by reducing implementation complexity by performing efficient intra prediction and by enhancing prediction capability (or performance).
[0015] According to the present specification, the intra prediction structure can be simplified by constructing (or configuring) a unified MPM list for normal intra prediction, multi-reference line intra prediction, and sub-partition intra prediction, and the coding efficiency can be enhanced by efficiently coding the intra prediction mode. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 An example of a video / image coding system to which embodiments of this document may be applied is schematically illustrated.
[0017] Figure 2 is a schematic diagram illustrating a configuration of a video / image encoding device to which embodiments of this document can be applied.
[0018] Figure 3 is a schematic diagram illustrating a configuration of a video / image decoding device to which embodiments of this document can be applied.
[0019] Figure 4 An example of an intra-frame prediction-based image encoding method to which the exemplary embodiments of this document are applicable is schematically illustrated, and Figure 5 An intra predictor in an encoding device is schematically illustrated.
[0020] Figure 6 An example of an intra-frame prediction-based image decoding method to which the exemplary embodiments of this document are applicable is schematically illustrated, and Figure 7 An intra predictor in a decoding device is schematically illustrated.
[0021] Figure 8 An example of an intra prediction method based on the MPM mode in an encoding device to which exemplary embodiments of this document are applicable is illustrated.
[0022] Fig. 9 An example of an intra prediction method based on the MPM mode in a decoding device to which exemplary embodiments of this document are applicable is illustrated.
[0023] Fig.10 An example of an intra prediction mode to which embodiments of this document can be applied is illustrated.
[0024] Fig.11 Examples of reference sample lines for intra prediction using multiple reference lines are illustrated.
[0025] Fig.12 An example of sub-partitioning according to intra sub-partitioning (ISP) partitioning is illustrated.
[0026] Fig.13 is a diagram illustrating an implementation of a method for generating a unified MPM list according to the present document.
[0027] Fig.14 is a diagram illustrating another embodiment of a method for generating a unified MPM list according to the present document.
[0028] Fig.15 is a flowchart generally showing an encoding method that can be performed by an encoding device according to an embodiment of the present specification.
[0029] Fig.16 is a flowchart generally showing a decoding method that can be performed by a decoding device according to an embodiment of the present specification.
[0030] Fig.17 An example of a content streaming system to which the embodiments disclosed in this document can be applied is illustrated. DETAILED DESCRIPTION
[0031] This document can be modified in various ways and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit this document to specific embodiments. The terms commonly used in this specification are used to describe specific embodiments rather than to limit the technical spirit of this document. Unless otherwise clearly indicated in the context, singular expressions include plural expressions. Terms such as "including" or "having" in this specification should be understood to indicate the presence of characteristics, numbers, steps, operations, elements, components, or combinations thereof described in this specification, without excluding the possibility of the presence or addition of one or more characteristics, numbers, steps, operations, elements, components, or combinations thereof.
[0032] In addition, in order to facilitate the description related to different feature functions, the elements in the drawings described in this document are illustrated independently. This does not mean that each element is implemented as separate hardware or separate software. For example, at least two elements can be combined to form a single element, or a single element can be divided into multiple elements. Embodiments in which elements are combined and / or separated are also included in the scope of the rights of this document, unless it deviates from the essence of this document.
[0033] In this document, the term "A or B" may mean "only A", "only B", or "both A and B". In other words, in this document, the term "A or B" may be interpreted as indicating "A and / or B". For example, in this document, the term "A, B, or C" may mean "only A", "only B", "only C", or "any combination of A, B, and C".
[0034] A slash “ / ” or a comma used in this document may mean “and / or”. For example, “A / B” may mean “A and / or B”. Thus, “A / B” may mean “only A”, “only B”, or “both A and B”. For example, “A, B, C” may mean “A, B, or C”.
[0035] In this document, "at least one of A and B" may mean "only A", "only B", or "both A and B". In addition, in this document, the expression "at least one of A or B" or "at least one of A and / or B" may be interpreted as the same as "at least one of A and B".
[0036] Furthermore, in this document, “at least one of A, B, and C” may mean “only A,” “only B,” “only C,” or “any combination of A, B, and C.” Furthermore, “at least one of A, B, or C” or “at least one of A, B and / or C” may mean “at least one of A, B, and C.”
[0037] In addition, brackets used in this document may mean "for example". Specifically, in the case of expressing "prediction (intra-frame prediction)", it may be indicated that "intra-frame prediction" is proposed as an example of "prediction". In other words, the term "prediction" in this document is not limited to "intra-frame prediction", and it may be indicated that "intra-frame prediction" is proposed as an example of "prediction". In addition, even in the case of expressing "prediction (ie, intra-frame prediction)", it may be indicated that "intra-frame prediction" is proposed as an example of "prediction".
[0038] In this document, technical features individually described in one drawing may be implemented individually, or may be implemented simultaneously.
[0039] Hereinafter, preferred embodiments of the present document are described in more detail with reference to the accompanying drawings. Hereinafter, in the accompanying drawings, the same reference numerals are used for the same elements, and redundant descriptions of the same elements may be omitted.
[0040] Figure 1 An example of a video / image coding system to which embodiments of this document may be applied is schematically illustrated.
[0041] Reference Figure 1 The video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit the encoded video / image information or data to the receiving device in the form of a file or stream transmission via a digital storage medium or a network.
[0042] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.
[0043] The video source can obtain the video / image by capturing, synthesizing or generating the video / image process. The video source may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive including previously captured videos / images, etc. The video / image generation device may include, for example, a computer, a tablet computer, and a smart phone, and may (electronically) generate the video / image. For example, a virtual video / image may be generated by a computer, etc. In this case, the video / image capture process may be replaced by a process that generates relevant data.
[0044] The encoding device can encode the input video / image. The encoding device can perform a series of processes such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.
[0045] The transmitter may transmit the encoded video / image information or data output in the form of a bit stream to a receiver of a receiving device in the form of a file or stream transmission through a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for transmitting through a broadcast / communication network. The receiver may receive / extract a bit stream and transmit the received / extracted bit stream to a decoding device.
[0046] The decoding device may decode a video / image by performing a series of processes such as dequantization, inverse transformation, prediction, etc. corresponding to the operation of the encoding device.
[0047] The renderer can render the decoded video / image, and the rendered video / image can be displayed through a display.
[0048] This document relates to video / image coding. For example, the methods / implementations disclosed in this document may be applied to methods disclosed in the Versatile Video Coding (VVC) standard, the Essential Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second generation audio video coding standard (AVS2), or the next generation video / image coding standard (e.g., H.267 or H.268, etc.).
[0049] In this document, various embodiments related to video / image coding may be provided, and unless otherwise specified, the embodiments may be combined with each other and performed.
[0050] In this document, video may mean a collection of a series of images over time. Generally, a picture means a unit representing an image of a specific time region, and a slice / tile is a unit that constitutes a part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles. A picture may be composed of one or more tile groups. A tile group may include one or more tiles. A block may represent a rectangular area of a CTU row within a tile in a picture. A tile may be divided into a plurality of blocks, each of which is composed of one or more CTU rows within a tile. A tile that is not divided into a plurality of blocks may also be referred to as a block. Block scanning may be a specific ordering of CTUs of a partitioned picture as follows: CTUs are continuously sorted in a block by a CTU raster scan, blocks within a tile are continuously sorted by a raster scan of a block of a tile, and tiles in a picture are continuously sorted by a raster scan of a tile of a picture. A tile is a rectangular area of a CTU within a specific tile column and a specific tile row in a picture. A tile column is a rectangular area whose height of the CTU is equal to the height of the picture and whose width is specified by a syntax element in a picture parameter set. A tile row is a rectangular area whose height of the CTU is specified by a syntax element in a picture parameter set and whose width is equal to the width of the picture. Tile scanning can be a specific order sorting of CTUs of the following partitioned pictures: CTUs can be sorted continuously in tiles by CTU raster scanning, and tiles in pictures can be sorted continuously by raster scanning of tiles of pictures. A slice includes an integer number of blocks of a picture that can be exclusively included in a single NAL unit. A slice can consist of multiple complete tiles or only of a complete block portion of a continuous sequence of tiles. In this document, tile groups and slices can be used interchangeably with each other. For example, in this document, a tile group / tile group header may be referred to as a slice / slice header.
[0051] Pixel or pel may mean the smallest unit constituting a picture (or image). In addition, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luminance component, or only a pixel / pixel value of a chrominance component.
[0052] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region and information related to the region. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. Depending on the situation, terms such as unit, block, region, etc. may be used interchangeably. In general, an M×N block may include a set (or array) of samples (or sample arrays) or transform coefficients consisting of M columns and N rows.
[0053] Figure 2 is a diagram schematically describing a configuration of a video / image encoding device to which the present document can be applied. Hereinafter, the so-called video encoding device may include an image encoding device.
[0054] Reference Figure 2 , the encoding device 200 may include an image segmenter (210), a predictor (220), a residual processor (230), an entropy encoder (240), an adder (250), a filter (260) and a memory (270). The predictor (220) may include an inter-frame predictor (221) and an intra-frame predictor (222). The residual processor (230) may include a transformer (232), a quantizer (233), a dequantizer (234), and an inverse transformer (235). The residual processor (230) may also include a subtractor (231). The adder (250) may be referred to as a reconstructor or a reconstructed block generator. According to an embodiment, the image segmenter (210), the predictor (220), the residual processor (230), the entropy encoder (240), the adder (250) and the filter (260) described above may be composed of one or more hardware components (e.g., an encoder chipset or processor). In addition, the memory (270) may include a decoded picture buffer (DPB) and may be composed of a digital storage medium. The hardware component may also include the memory (270) as an internal / external component.
[0055] The image splitter (210) splits the input image (or picture or frame) input to the encoding device (200) into one or more processing units. As an example, the processing unit may be referred to as a coding unit (CU). In this case, starting from a coding tree unit (CTU) or a maximum coding unit (LCU), the coding unit may be recursively split according to a quadtree binary tree ternary tree (QTBTTT) structure. For example, one coding unit may be divided into a plurality of coding units with a deeper depth based on a quadtree structure, a binary tree structure, and / or a ternary tree structure. In this case, for example, the quadtree structure may be applied first, and then the binary tree structure and / or the ternary tree structure may be applied. Alternatively, the binary tree structure may be applied first. The coding process according to the present document may be performed based on the final coding unit that is not further split. In this case, based on the coding efficiency according to the image characteristics, the maximum coding unit may be directly used as the final coding unit. Alternatively, the coding unit may be recursively split into coding units with a further deeper depth as needed, so that the coding unit of the optimal size may be used as the final coding unit. Here, the coding process may include processes such as prediction, transformation, and reconstruction, which will be described later. As another example, the processing unit may also include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may be divided or partitioned from the above-mentioned final coding unit. The prediction unit may be a unit for sample prediction, and the transformation unit may be a unit for deriving a transformation coefficient and / or a unit for deriving a residual signal based on the transformation coefficient.
[0056] Depending on the situation, the unit and terms such as block, region, etc. may be used interchangeably. In general, an M×N block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally represent a pixel or a value of a pixel, and may represent only a pixel / pixel value of a luminance component, or only a pixel / pixel value of a chrominance component. A sample may be used as a term corresponding to a pixel or pel of a picture (or image).
[0057] In the encoding device (200), a prediction signal (prediction block, prediction sample array) output from an inter-frame predictor (221) or an intra-frame predictor (222) is subtracted from an input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is sent to a transformer (232). In this case, as shown, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input image signal (original block, original sample array) in the encoder (200) may be referred to as a subtractor (231). The predictor may perform prediction on a processing target block (hereinafter, referred to as a "current block"), and may generate a prediction block including prediction samples for the current block. The predictor may determine whether to apply intra-frame prediction or inter-frame prediction based on the current block or CU. As discussed later in the description of each prediction mode, the predictor may generate various information related to prediction such as prediction mode information, and send the generated information to an entropy encoder (240). Information about the prediction may be encoded in the entropy encoder (240) and output in the form of a bitstream.
[0058] The intra-frame predictor (222) can predict the current block by referring to samples in the current picture. Depending on the prediction mode, the reference sample can be located near the current block or separated from the current block. In intra-frame prediction, the prediction mode may include a plurality of non-directional modes and a plurality of directional modes. The non-directional mode may include, for example, a DC mode and a planar mode. Depending on the level of detail of the prediction direction, the directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes. However, this is merely an example, and more or fewer directional prediction modes may be used depending on the setting. The intra-frame predictor (222) may determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring block.
[0059] The inter-frame predictor (221) may derive a prediction block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information may be predicted on a block, sub-block or sample basis based on the correlation of motion information between a neighboring block and a current block. The motion information may include a motion vector and a reference picture index. The motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same as or different from each other. The temporal neighboring block may be referred to as a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block may be referred to as a collocated picture (colPic). For example, the inter-frame predictor (221) may configure a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter-frame prediction may be performed based on various prediction modes. For example, in the case of the skip mode and the merge mode, the inter-frame predictor (221) can use the motion information of the neighboring block as the motion information of the current block. In the skip mode, unlike the merge mode, the residual signal cannot be sent. In the case of the motion information prediction (motion vector prediction, MVP) mode, the motion vector of the neighboring block can be used as a motion vector prediction item, and the motion vector of the current block can be indicated by signaling the motion vector difference.
[0060] The predictor (220) may generate a prediction signal based on various prediction methods. For example, the predictor may apply intra prediction or inter prediction to a prediction of a block, and may also apply intra prediction and inter prediction at the same time. This may be referred to as combined inter and intra prediction (CIIP). In addition, the predictor may be based on an intra-block copy (IBC) prediction mode or a palette mode to perform prediction on a block. The IBC prediction mode or the palette mode may be used for content image / video coding such as games such as screen content coding (SCC). Although IBC basically performs prediction in the current picture, its execution is similar to inter prediction in that it derives a reference block in the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. The palette mode may be considered as an example of intra coding or intra prediction. When the palette mode is applied, the sample values in the picture may be signaled based on information about the palette index and the palette table.
[0061] The prediction signal generated by the predictor (including the inter-frame predictor (221) and / or the intra-frame predictor (222)) can be used to generate a reconstructed signal or to generate a residual signal. The transformer (232) can generate a transform coefficient by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loève transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, GBT means a transform obtained from a curve graph when the relationship information between pixels is represented by a curve graph. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process can be applied to square pixel blocks of the same size, or can be applied to blocks of variable size rather than square blocks.
[0062] The quantizer (233) may quantize the transform coefficients and send them to the entropy encoder (240), and the entropy encoder (240) may encode the quantized signal (information about the quantized transform coefficients) and output the encoded signal in the bitstream. The information about the quantized transform coefficients may be referred to as residual information. The quantizer (233) may rearrange the quantized transform coefficients of the block type into a one-dimensional vector form based on the coefficient scanning order, and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoder (240) may perform various encoding methods such as, for example, exponential Golomb, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), etc. The entropy encoder (240) may encode information required for video / image reconstruction other than the quantized transform coefficients (e.g., the value of the syntax element, etc.) together or separately. The encoded information (e.g., the encoded video / image information) may be transmitted or stored on a unit basis of a network abstraction layer (NAL) in the form of a bitstream. The video / image information may further include information about various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), etc. In addition, the video / image information may further include conventional constraint information. In this document, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded by the above-mentioned encoding process and included in a bitstream. The bitstream may be transmitted through a network or stored in a digital storage medium. Here, the network may include a broadcast network, a communication network, and / or the like, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) that transmits a signal output from the entropy encoder (240) or a memory (not shown) that stores it may be configured as an internal / external element of the encoding device (200), or the transmitter may be included in the entropy encoder (240).
[0063] The quantized transform coefficients output from the quantizer (233) can be used to generate a prediction signal. For example, by applying dequantization and inverse transformation to the quantized transform coefficients using a dequantizer (234) and an inverse transformer (235), a residual signal (residual block or residual sample) can be reconstructed. The adder (250) adds the reconstructed residual signal to the prediction signal output from the inter-frame predictor (221) or the intra-frame predictor (222), so that a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) can be generated. When there is no residual for the processing target block as in the case of applying the jump mode, the prediction block can be used as a reconstructed block. The adder (250) can be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current picture, and as described later, can be used for inter-frame prediction of the next picture performed by filtering.
[0064] Furthermore, in the picture encoding and / or reconstruction process, luma mapping with chroma scaling (LMCS) may be applied.
[0065] The filter (260) can improve the subjective / objective video quality by applying filtering to the reconstructed signal. For example, the filter (260) can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and the modified reconstructed picture can be stored in the memory (270), especially in the DPB of the memory (270). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive ring filter, bilateral filter, etc. As discussed later in the description of each filtering method, the filter (260) can generate various information related to filtering and send the generated information to the entropy encoder (240). The information about filtering can be encoded in the entropy encoder 240 and output in the form of a bitstream.
[0066] The modified reconstructed picture sent to the memory (270) can be used as a reference picture in the inter-frame predictor (221). Accordingly, the encoding device can avoid prediction mismatch in the encoding device (200) and the decoding device when applying inter-frame prediction, and can also improve encoding efficiency.
[0067] The DPB of the memory (270) may store the modified reconstructed picture so as to use it as a reference picture in the inter-frame predictor (221). The memory (270) may store motion information of a block in the current picture from which motion information has been derived (or encoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be sent to the inter-frame predictor (221) to be used as motion information of a neighboring block or motion information of a temporally neighboring block. The memory (270) may store reconstructed samples of the reconstructed block in the current picture and send them to the intra-frame predictor (222).
[0068] Figure 3 is a diagram schematically describing a configuration of a video / image decoding device to which this document can be applied.
[0069] Reference Figure 3 , the video decoding device (300) may include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350) and a memory (360). The predictor (330) may include an inter-frame predictor (331) and an intra-frame predictor (332). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321). According to an embodiment, the entropy decoder (310), the residual processor (320), the predictor (330), the adder (340) and the filter (350) described above may be composed of one or more hardware components (e.g., a decoder chipset or a processor). In addition, the memory (360) may include a decoded picture buffer (DPB) and may be composed of a digital storage medium. The hardware component may also include the memory (360) as an internal / external component.
[0070] When a bit stream including video / image information is input, the decoding device (300) can be used with the bit stream already in the Figure 2 The processing of processing video / image information in the encoding device reconstructs the image accordingly. For example, the decoding device (300) can derive the unit / block based on the information related to the block segmentation obtained from the bit stream. The decoding device (300) can perform decoding by using the processing unit applied in the encoding device. Therefore, the decoding processing unit can be, for example, a coding unit, which can be divided into a quadtree structure, a binary tree structure and / or a ternary tree structure using a coding tree unit or a maximum coding unit. One or more transform units can be derived from the coding unit. And, the reconstructed image signal decoded and output by the decoding device (300) can be reproduced by a reproducer.
[0071] The decoding device (300) may receive the Figure 2The signal output by the encoding device of the encoding device, and the received signal can be decoded by the entropy decoder (310). For example, the entropy decoder (310) can parse the bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). The video / image information may also include information about various parameter sets such as adaptive parameter set (APS), picture parameter set (PPS), sequence parameter set (SPS), video parameter set (VPS), etc. In addition, the video / image information may also include conventional constraint information. The decoding device may further decode the picture based on the information about the parameter set and / or the conventional constraint information. In this document, the signaled / received information and / or syntax elements to be described later can be decoded and obtained from the bitstream through a decoding process. For example, the entropy decoder (310) can decode the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, CABAC, etc., and can output the values of syntax elements required for image reconstruction and the quantized values of the transform coefficients of the residual. More specifically, the CABAC entropy decoding method may receive a bin corresponding to each syntax element in a bitstream, determine a context model using decoding target syntax element information and decoding information of neighboring and decoding target blocks or information of symbols / bins decoded in a previous step, predict bin generation probabilities according to the determined context model, and perform arithmetic decoding on the bins to generate symbols corresponding to each syntax element value. Here, the CABAC entropy decoding method may update the context model using information of symbols / bins decoded by the context model for the next symbol / bin after determining the context model. Information about prediction among the information decoded in the entropy decoder (310) may be provided to a predictor (inter-frame predictor (332) and intra-frame predictor (331)), and residual values (i.e., quantized transform coefficients) and associated parameter information on which entropy decoding has been performed in the entropy decoder (310) may be input to a residual processor (320). The residual processor (320) may derive a residual signal (residual block, residual sample, residual sample array). In addition, information about filtering among the information decoded in the entropy decoder (310) may be provided to the filter (350). In addition, a receiver (not shown) receiving a signal output from the encoding device may also constitute the decoding device (300) as an internal / external element, and the receiver may be a component of the entropy decoder (310). In addition, the decoding device according to the present document may be referred to as a video / image / picture decoding device, and the decoding device may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder).The information decoder may include an entropy decoder (310), and the sample decoder may include at least one of a dequantizer (321), an inverse transformer (322), an adder (340), a filter (350), a memory (360), an inter-frame predictor (332), and an intra-frame predictor (331).
[0072] The dequantizer (321) may output a transform coefficient by dequantizing the quantized transform coefficient. The dequantizer (321) may rearrange the quantized transform coefficient into a two-dimensional block form. In this case, the rearrangement may be performed based on the order of coefficient scanning that has been performed in the encoding device. The dequantizer (321) may perform dequantization on the quantized transform coefficient using a quantization parameter (e.g., quantization step size information) and obtain the transform coefficient.
[0073] The inverse transformer (322) obtains a residual signal (residual block, residual sample array) by inversely transforming the transform coefficients.
[0074] The predictor may perform prediction on the current block and generate a prediction block including prediction samples for the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on information about prediction output from the entropy decoder (310), and specifically may determine an intra / inter prediction mode.
[0075] The predictor (320) may generate a prediction signal based on various prediction methods. For example, the predictor may apply intra prediction or inter prediction to the prediction of a block, and may also apply intra prediction and inter prediction at the same time. This may be referred to as combined inter and intra prediction (CIIP). In addition, the predictor may be based on an intra-block copy (IBC) prediction mode or a palette mode to perform prediction on a block. The IBC prediction mode or the palette mode may be used for content image / video coding such as games such as screen content coding (SCC). Although IBC basically performs prediction in the current picture, its execution is similar to inter prediction in that it derives a reference block in the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. The palette mode may be considered as an example of intra coding or intra prediction. When the palette mode is applied, information about the palette table and the palette index may be included in the video / image information and signaled.
[0076] The intra-frame predictor (331) can predict the current block by referring to samples in the current picture. Depending on the prediction mode, the reference sample can be located near the current block or separated from the current block. In intra-frame prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. The intra-frame predictor (331) can determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring block.
[0077] The inter-frame predictor (332) can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information can be predicted on a block, sub-block or sample basis based on the correlation of the motion information between the neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter-frame predictor (332) may configure a motion information candidate list based on the neighboring blocks, and derive a motion vector and / or a reference picture index of the current block based on the received candidate selection information. Inter-frame prediction may be performed based on various prediction modes, and information about the prediction may include information indicating a mode of inter-frame prediction for the current block.
[0078] The adder (340) adds the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictor (inter-frame predictor (332) or intra-frame predictor (331)), so that a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) can be generated. When there is no residual for the processing target block as in the case of applying the skip mode, the prediction block can be used as the reconstructed block.
[0079] The adder (340) may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next block to be processed in the current picture, may be output through filtering as described below, or may be used for inter prediction of the next picture.
[0080] In addition, luma mapping with chroma scaling (LMCS) may be applied to the picture decoding process.
[0081] The filter (350) can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter (350) can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory (360), specifically, in the DPB of the memory (360). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.
[0082] The (modified) reconstructed picture stored in the DPB of the memory (360) can be used as a reference picture in the inter-frame predictor (332). The memory (360) can store the motion information of the block from which the motion information in the current picture is derived (or decoded) and / or the motion information of the block in the reconstructed picture. The stored motion information can be sent to the inter-frame predictor (332) to be used as the motion information of the spatial neighboring block or the motion information of the temporal neighboring block. The memory (360) can store the reconstructed samples of the reconstructed block in the current picture and transmit the reconstructed samples to the intra-frame predictor (331).
[0083] In the present disclosure, the implementation described in the filter (260), inter-frame predictor (221) and intra-frame predictor (222) of the encoding device (200) can be the same as the filter (350), inter-frame predictor (332) and intra-frame predictor (331) of the decoding device (300) or can be applied corresponding to the filter (350), inter-frame predictor (332) and intra-frame predictor (331) of the decoding device (300), respectively.
[0084] As described above, when performing video coding, prediction is performed to improve compression efficiency. A prediction block including prediction samples of the current block, that is, a target coding block, can be generated by prediction. In this case, the prediction block includes prediction samples in the spatial domain (or pixel domain). The prediction block is derived similarly in the encoding device and the decoding device. The encoding device can improve image coding efficiency by signaling information (residual information) about the original block rather than the original sample value of the original block itself and the residual between the prediction block. The decoding device can derive a residual block including residual samples based on the residual information, can generate a reconstructed block including reconstructed samples by adding the residual block to the prediction block, and can generate a reconstructed picture including the reconstructed block.
[0085] Residual information can be generated through a transformation and quantization process. For example, the encoding device can derive a residual block between the original block and the prediction block, can derive a transform coefficient by performing a transformation process on the residual samples (residual sample array) included in the residual block, can derive a quantized transform coefficient by performing a quantization process on the transform coefficient, and can signal the relevant residual information to the decoding device (through a bitstream). In this case, the residual information may include information such as value information, position information, a transformation scheme, a transform kernel, and a quantization parameter of the quantized transform coefficient. The decoding device can perform a dequantization / inverse transformation process based on the residual information, and can derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the prediction block and the residual block. In addition, the encoding device can derive a residual block for inter-frame prediction reference of subsequent pictures by dequantizing / inverse transforming the quantized transform coefficients, and can generate a reconstructed picture.
[0086] In addition, if intra prediction is performed, the correlation between samples can be used, and the difference between the original block and the predicted block, i.e., the residual, can be obtained. The above-mentioned transformation and quantization can be applied to the residual. Therefore, spatial redundancy can be reduced. Hereinafter, the encoding method and decoding method using intra prediction are specifically described.
[0087] Intra prediction refers to prediction of a prediction sample of a current block based on a reference sample located outside the current block within a picture including the current block (hereinafter, the current picture). In this case, the reference sample located outside the current block may refer to a sample adjacent to the current block. If intra prediction is applied to the current block, neighboring reference samples to be used for intra prediction of the current block may be derived.
[0088] For example, when the size (width×height) of the current block is nW×nH, the neighboring reference samples of the current block may include a sample adjacent to the left boundary and a total of 2×nH samples adjacent to the lower left of the current block, a sample adjacent to the upper boundary and a total of 2×nW samples adjacent to the upper right of the current block, and one sample adjacent to the upper left of the current block. Alternatively, the neighboring reference samples of the current block may also include multiple columns of upper neighboring samples and multiple rows of left neighboring samples. In addition, the neighboring reference samples of the current block may also include a total of nH samples adjacent to the right boundary of the current block of size nW×nH, a total of nW samples adjacent to the lower boundary of the current block, and one sample adjacent to the lower right of the current block.
[0089] In this case, some of the neighboring reference samples of the current block have not been decoded or may not be available. In this case, the decoding device can configure the neighboring reference samples to be used for prediction by replacing the unavailable samples with available samples. Alternatively, the neighboring reference samples to be used for prediction can be constructed by interpolation of available samples.
[0090] If the neighboring reference samples are derived, (i) the prediction samples may be derived based on an average or interpolation of the neighboring reference samples of the current block, and (ii) the prediction samples may be derived based on reference samples existing in a specific (prediction) direction for the prediction samples among the neighboring reference samples of the current block. (i) may be applied when the intra prediction mode is a non-directional mode or a non-angular mode. (ii) may be applied when the intra prediction mode is a directional mode or an angular mode.
[0091] In addition, the prediction sample can also be generated by interpolating between the first neighboring sample located in the prediction direction of the intra prediction mode of the current block based on the prediction sample of the current block among the neighboring reference samples and the second neighboring sample located in the opposite direction of the prediction direction. The foregoing case may be referred to as linear interpolation intra prediction (LIP). In addition, the chrominance prediction sample may also be generated based on the luma sample using a linear model. This case may be referred to as LM mode.
[0092] In addition, a temporary prediction sample of the current block may be derived based on the filtered neighboring reference sample, and the prediction sample of the current block may also be derived by weighted summing at least one reference sample derived from among the normal neighboring reference samples (i.e., the unfiltered neighboring reference samples) according to the intra prediction mode and the temporary prediction sample. The foregoing may be referred to as position-dependent intra prediction (PDPC).
[0093] In addition, a prediction sample can be derived using a reference sample in a prediction direction located in a corresponding line by selecting a reference sample line with the highest prediction accuracy among neighboring multiple reference sample lines of the current block, and intra-frame prediction encoding can be performed by a method for indicating (signaling) the reference sample line used at this time to a decoding device. The foregoing case may be referred to as multi-reference line (MRL) intra-frame prediction or MRL-based intra-frame prediction.
[0094] In addition, intra prediction can be performed based on the same intra prediction mode by dividing the current block into vertical or horizontal sub-partitions, and neighboring reference samples can be derived and used in units of sub-partitions. That is, in this case, the intra prediction mode of the current block is equally applied to the sub-partitions, and neighboring reference samples can be derived and used in units of sub-partitions, thereby enhancing intra prediction performance in some cases. This prediction method may be referred to as intra sub-partition (ISP) intra prediction or ISP-based intra prediction.
[0095] The aforementioned intra-frame prediction method may be referred to as an intra-frame prediction type separate from the intra-frame prediction mode. The intra-frame prediction type may be referred to as various terms such as intra-frame prediction technology or additional intra-frame prediction mode. For example, the intra-frame prediction type (or additional intra-frame prediction mode, etc.) may include at least one of the aforementioned LIP, PDPC, MRL and ISP. A general (or normal) intra-frame prediction method different from a specific intra-frame prediction type such as LIP, PDPC, MRL and ISP may be referred to as a normal intra-frame prediction type. If a specific intra-frame prediction type is not applied, a normal intra-frame prediction type may generally be applied, and prediction may be performed based on the aforementioned intra-frame prediction mode. In addition, if necessary, post-processing filtering may also be performed on the derived prediction samples.
[0096] The following drawings are prepared to illustrate specific examples of this document. Since the names of specific devices or specific terms or names (eg, grammatical names, etc.) described in the drawings are presented exemplarily, the technical features of this document are not limited to the specific names used in the following drawings.
[0097] Figure 4 An example of an intra-frame prediction-based image encoding method to which the exemplary embodiments of this document are applicable is schematically illustrated, and Figure 5 An intra predictor in an encoding device is schematically illustrated. Figure 5 The intra-frame predictor in the encoding device shown in FIG. 1 may also be applied equally or correspondingly to Figure 2 The intra-frame predictor (222) of the encoding device (200) shown in FIG.
[0098] Reference Figure 4 and Figure 5 , S400 may be performed by an intra-frame predictor (222) of the encoding device, and S410 may be performed by a residual processor (230) of the encoding device. Specifically, S410 may be performed by a subtractor (231) of the encoding device. In S420, prediction information may be derived by the intra-frame predictor (222) and encoded by an entropy encoder (240). In S420, residual information may be derived by the residual processor (230) and encoded by the entropy encoder (240). The residual information indicates information about residual samples. The residual information may include information about quantized transform coefficients of the residual samples. As described above, the residual samples may be derived by transform coefficients via the transformer (232) of the encoding device, and the transform coefficients may be derived by quantized transform coefficients via the quantizer (233). The information about the quantized transform coefficients may be encoded by the entropy encoder (240) through a residual coding process.
[0099] The encoding device performs intra prediction of the current block (S400). The encoding device may derive the intra prediction mode / type of the current block, derive neighboring reference samples of the current block, and generate prediction samples in the current block based on the intra prediction mode / type and the neighboring reference samples. Here, the processes of determining the intra prediction mode / type, deriving neighboring reference samples, and generating prediction samples may also be performed simultaneously, and any one process may also be performed earlier than the other processes.
[0100] For example, the intra-frame predictor (222) of the encoding device may include an intra-frame prediction mode / type determiner (222-1), a reference sample deriver (222-2), and a prediction sample deriver (222-3), wherein the intra-frame prediction mode / type determiner (222-1) may determine the intra-frame prediction mode / type of the current block, the reference sample deriver (222-2) may derive the neighboring reference samples of the current block, and the prediction sample deriver (222-3) may derive the prediction sample of the current block. In addition, although not shown, if a prediction sample filtering process is performed, the intra-frame predictor (222) may also further include a prediction sample filter (not shown). The encoding device may determine a mode / type applied to the current block among multiple intra-frame prediction modes / types. The encoding device may compare the RD costs of the intra-frame prediction modes / types and determine the optimal intra-frame prediction mode / type for the current block.
[0101] As described above, the encoding device may also perform a prediction sample filtering process. Prediction sample filtering may be referred to as post-filtering. Some or all of the prediction samples may be filtered by the prediction sample filtering process. In some cases, the prediction sample filtering process may be omitted.
[0102] The encoding apparatus generates residual samples of the current block based on the (filtered) prediction samples (S410). The encoding apparatus may compare the prediction samples based on phases in the original samples of the current block and derive the residual samples.
[0103] The encoding device may encode image information including information about intra prediction (prediction information) and residual information about residual samples (S420). The prediction information may include intra prediction mode information and intra prediction type information. The residual information may include residual coding syntax. The encoding device may derive quantized transform coefficients by transforming / quantizing the residual samples. The residual information may include information about the quantized transform coefficients.
[0104] The encoding device can output the encoded image information in the form of a bit stream. The output bit stream can be delivered to the decoding device via a storage medium or a network.
[0105] As described above, the encoding device may generate a reconstructed picture (including reconstructed samples and reconstructed blocks). To this end, the encoding device may derive (modified) residual samples by dequantizing / inverse transforming the quantized transform coefficients again. As described above, the reason for transforming / quantizing the residual samples and then dequantizing / inverse transforming them again is to derive the same residual samples as the residual samples derived by the decoding device as described above. The encoding device may generate a reconstructed block including the reconstructed samples of the current block based on the predicted samples and the (modified) residual samples. A reconstructed picture of the current picture may be generated based on the reconstructed block. As described above, an in-loop filtering process or the like may be further applied to the reconstructed picture.
[0106] Figure 6An example of an intra-frame prediction-based image decoding method to which the exemplary embodiments of this document are applicable is schematically illustrated, and Figure 7 An intra predictor in a decoding device is schematically illustrated. Figure 7 The intra-frame predictor in the decoding device shown in FIG. 1 may also be applied equally or correspondingly to Figure 3 The intra predictor 331 of the decoding device 300 shown in FIG.
[0107] Reference Figure 6 and Figure 7 , the decoding device may perform operations corresponding to the above operations performed by the encoding device. S600 to S620 may be performed by the intra-frame predictor 331 of the decoding device, and the prediction information in S600 and the residual information in S630 may be obtained from the bit stream by the entropy decoder 310 of the decoding device. The residual processor 320 of the decoding device may derive residual samples of the current block based on the residual information. Specifically, the dequantizer 321 of the residual processor 320 may derive transform coefficients by performing dequantization based on the quantized transform coefficients derived based on the residual information, and the inverse transformer 322 of the residual processor derives the residual samples of the current block by inversely transforming the transform coefficients. S640 may be performed by the reconstructor or adder 340 of the decoding device.
[0108] The decoding device may derive the intra prediction mode / type of the current block based on the received prediction information (intra prediction mode / type information) (S600). The decoding device may derive the neighboring reference samples of the current block (S610). The decoding device generates the prediction samples in the current block based on the intra prediction mode / type and the neighboring reference samples (S620). In this case, the decoding device may perform a prediction sample filtering process. Prediction sample filtering may be referred to as post filtering. Some or all of the prediction samples may be filtered by the prediction sample filtering process. In some cases, the prediction sample filtering process may be omitted.
[0109] The decoding device generates residual samples for the current block based on the received residual information (S630). The decoding device may generate reconstructed samples for the current block based on the prediction samples and the residual samples, and derive a reconstructed block including the reconstructed samples (S640). A reconstructed picture of the current picture may be generated based on the reconstructed block. As described above, an in-loop filtering process or the like may be further applied to the reconstructed picture.
[0110] Here, the intra-frame predictor (331) of the decoding device may include an intra-frame prediction mode / type determiner (331-1), a reference sample deriver (331-2), and a prediction sample deriver (331-3), wherein the intra-frame prediction mode / type determiner (331-1) may determine the intra-frame prediction mode / type of the current block based on the intra-frame prediction mode / type information obtained by the entropy decoder (310), the reference sample deriver (331-2) may derive the neighboring reference samples of the current block, and the prediction sample deriver (331-3) may derive the prediction sample of the current block. In addition, although not shown, if the above-mentioned prediction sample filtering process is performed, the intra-frame predictor (331) may also further include a prediction sample filter (not shown).
[0111] The intra-frame prediction mode information may include, for example, flag information (e.g., intra_luma_mpm_flag) indicating whether the most probable mode (MPM) is applied to the current block or whether the remaining mode is applied thereto. At this time, if the MPM is applied to the current block, the prediction mode information may also include index information (e.g., intra_luma_mpm_idx) indicating one of the intra-frame prediction mode candidates (MPM candidates). The intra-frame prediction mode candidate (MPM candidate) may be composed of an MPM candidate list or an MPM list. In addition, if the MPM is not applied to the current block, the intra-frame prediction mode information may also include remaining mode information (e.g., intra_luma_mpm_remainder) indicating one of the remaining intra-frame prediction modes other than the intra-frame prediction mode candidate (MPM candidate). The decoding device may determine the intra-frame prediction mode of the current block based on the intra-frame prediction mode information.
[0112] In addition, the intra prediction type information may be implemented in various forms. As an example, the intra prediction type information may include intra prediction type index information indicating one of the intra prediction types. As another example, the intra prediction type information may include at least one of the following: reference sample line information (e.g., intra_luma_ref_idx) indicating whether MRL is applied to the current block, and if MRL is applied, which reference sample line is used; ISP flag information (e.g., intra_subpartitions_mode_flag) indicating whether ISP is applied to the current block; ISP type information (e.g., intra_subpartitions_split_flag) indicating the type of division of the sub-partition when ISP is applied; flag information indicating whether PDCP is applied; or flag information indicating whether LIP is applied. In addition, the intra prediction type information may include a MIP flag indicating whether MIP is applied to the current block.
[0113] The above-mentioned intra-frame prediction mode information and / or intra-frame prediction type information can be encoded / decoded by the coding method described in this document. For example, the above-mentioned intra-frame prediction mode information and / or intra-frame prediction type information can be encoded / decoded by entropy coding (e.g., CABAC, CAVLC) based on segmented (Rice) binary code.
[0114] Meanwhile, in the case of applying intra prediction, the intra prediction mode of the neighboring block can be used to determine the intra prediction mode applied to the current block. For example, the decoding device can select one of the mpm candidates in the most likely mode (mpm) list derived based on the intra prediction mode of the neighboring blocks (e.g., left neighboring blocks and / or upper neighboring blocks) of the current block and the additional candidate mode based on the received mpm index, or can select one of the remaining intra prediction modes not included in the mpm candidates (and plane mode) based on the remaining intra prediction mode information. The mpm list can be constructed to include or exclude the plane mode as a candidate. For example, if the mpm list includes the plane mode as a candidate, the mpm list can have 6 candidates, but if the mpm list does not include the plane mode as a candidate, the mpm list can have 5 candidates. If the mpm list does not include the plane mode as a candidate, a non-planar flag (e.g., intra_luma_not_planar_flag) indicating whether the intra prediction mode of the current block is not a plane mode can be signaled. For example, the mpm flag can be signaled first, and the mpm index and the non-planar flag can be signaled when the value of the mpm flag is equal to 1. In addition, the mpm index may be signaled when the value of the non-planar flag is equal to 1. Here, the mpm list is constructed to not include the planar mode as a candidate because the planar mode is always considered to be mpm, so in order to first identify whether the intra prediction mode is the planar mode by first signaling the flag (non-planar flag), rather than saying that the planar mode is not mpm.
[0115] For example, it may be indicated based on an mpm flag (e.g., intra_luma_mpm_flag) whether the intra prediction mode applied to the current block is in the mpm candidates (and planar mode) or in the remaining mode. An mpm flag value of 1 may indicate that the intra prediction mode of the current block is in the mpm candidates (and planar mode), while an mpm flag value of 0 may indicate that the intra prediction mode used for the current block is not in the mpm candidates (and planar mode). A non-planar flag (e.g., intra_luma_not_planar_flag) value of 0 may indicate that the intra prediction mode used for the current block is the planar mode, while a non-planar flag value of 1 may indicate that the intra prediction mode used for the current block is not the planar mode. The mpm index may be signaled in the form of an mpm_idx or intra_luma_mpm_idx syntax element, and the remaining intra prediction mode information may be signaled in the form of a rem_intra_luma_pred_mode or intra_luma_mpm_remainder syntax element. For example, the remaining intra prediction mode information may index the remaining intra prediction modes that are not included in the mpm candidates (and planar modes) among all intra prediction modes in the order of their prediction mode numbers, and may indicate one of them. The intra prediction mode may be an intra prediction mode of a luminance component (sample). In the following, the intra prediction mode information may include at least one of the following items: an mpm flag (e.g., intra_luma_mpm_flag), a non-planar flag (e.g., intra_luma_not_planar_flag), an mpm index (e.g., mpm_idx or intra_luma_mpm_idx), and remaining intra prediction mode information (rem_intra_luma_pred_mode or intra_luma_mpm_remainder). In this document, the mpm list may be referred to by various terms, such as an mpm candidate list, a candidate mode list (candModelList), and a candidate intra prediction mode list.
[0116] Typically, when blocks for an image are divided, the current block and neighboring blocks to be coded have similar image characteristics. Therefore, the current block and the neighboring blocks are more likely to have the same or similar intra prediction modes. Therefore, the encoder may use the intra prediction mode of the neighboring blocks to encode the intra prediction mode of the current block. For example, the encoder / decoder may construct a most probable mode (MPM) list for the current block. The MPM list may also be referred to as an MPM candidate list. Here, MPM may represent a mode for improving coding efficiency by taking into account the similarity between the current block and the neighboring blocks when coding the intra prediction mode.
[0117] Figure 8An example of an intra prediction method based on the MPM mode in an encoding device to which exemplary embodiments of this document are applicable is illustrated.
[0118] Reference Figure 8 , the encoding device constructs an MPM list of the current block (S800). The MPM list may include candidate intra prediction modes (MPM candidates) that are more likely to be applied to the current block. The MPM list may also include intra prediction modes of neighboring blocks, and further include specific intra prediction modes according to a predetermined method. A specific method for constructing the MPM list will be described later.
[0119] The encoding device determines an intra prediction mode of the current block (S810). The encoding device may perform prediction based on various intra prediction modes, and determine an optimal intra prediction mode based on rate-distortion optimization (RDO) based on the above prediction. In this case, the encoding device may also determine the optimal intra prediction mode using only the MPM candidates and the plane mode configured in the MPM list, or further use the remaining intra prediction modes and the MPM candidates and the plane mode configured in the MPM list to determine the optimal intra prediction mode.
[0120] Specifically, for example, if the intra prediction type of the current block is a specific type other than the normal intra prediction type (e.g., LIP, MRL, or ISP), the encoding device may determine the optimal intra prediction mode by considering only the MPM candidates and the plane mode as intra prediction mode candidates of the current block. That is, in this case, the intra prediction mode of the current block may be determined only among the MPM candidates and the plane mode, and in this case, the MPM flag may not be encoded / signaled. In this case, even without separate signaling of the MPM flag, the decoding device may estimate that the MPM flag is 1.
[0121] In general, if the intra prediction mode of the current block is not the planar mode but one of the MPM candidates in the MPM list, the encoding device generates an mpm index (mpm idx) indicating one of the MPM candidates. If the intra prediction mode of the current block does not even exist in the MPM list, the encoding device generates remaining intra prediction mode information indicating a mode such as the intra prediction mode of the current block among the remaining intra prediction modes not included in the MPM list (and the planar mode).
[0122] The encoding device may encode the intra-frame prediction mode information and output it in the form of a bitstream (S820). The intra-frame prediction mode information may include the aforementioned MPM flag, non-plane flag, MPM index and / or remaining intra-frame prediction mode information. In general, the MPM index and the remaining intra-frame prediction mode information have an alternating relationship and are not signaled at the same time when indicating the intra-frame prediction mode of a block. That is, the value 1 of the MPM flag is signaled together with the non-plane flag or the MPM index, or the value 0 of the MPM flag is signaled together with the remaining intra-frame prediction mode information. However, as described above, if a specific intra-frame prediction type is applied to the current block, the MPM flag is not signaled and only the non-plane flag and / or the MPM index may also be signaled. That is, in this case, the intra-frame prediction mode information may also include only the non-plane flag and / or the MPM index.
[0123] Fig. 9 An example of an intra prediction method based on the MPM mode in a decoding device to which exemplary embodiments of this document are applicable is illustrated. Fig. 9 The decoding device shown in FIG. 1 can determine the Figure 8 The encoding apparatus shown in determines and signals the intra-prediction mode corresponding to the intra-prediction mode information.
[0124] Reference Fig. 9 , the decoding apparatus obtains intra prediction mode information from the bitstream (S900). As described above, the intra prediction mode information may include at least one of an MPM flag, a non-plane flag, an MPM index, and a remaining intra prediction mode.
[0125] The decoding device constructs an MPM list (S910). The MPM list is composed of the same MPM list configured in the encoding device. That is, the MPM list may also include intra prediction modes of neighboring blocks, and further include a specific intra prediction mode according to a predetermined method. A specific method for constructing the MPM list will be described later.
[0126] Although it is illustrated that S910 is executed later than S900 , this is illustrative, and S910 may be executed earlier than S900 , and S900 and S910 may be executed at the same time.
[0127] The decoding apparatus determines an intra prediction mode of a current block based on the MPM list and the intra prediction mode information (S920).
[0128] As an example, if the value of the MPM flag is 1, the decoding device may derive the plane mode as the intra prediction mode of the current block or (based on the non-plane flag) derive the candidate indicated by the MPM index among the MPM candidates in the MPM list as the intra prediction mode of the current block. Here, the MPM candidate may also indicate only the candidates included in the MPM list, or also include the plane mode applicable to the case where the value of the MPM flag is 1 and the candidates included in the MPM list.
[0129] As another example, if the value of the MPM flag is 0, the decoding apparatus may derive, as the intra prediction mode of the current block, an intra prediction mode indicated by the remaining intra prediction mode information among the remaining intra prediction modes not included in the MPM list and the planar mode.
[0130] As another example, if the intra prediction type of the current block is a specific type (e.g., LIP, MRL, or ISP), the decoding device may derive a candidate or planar mode indicated by an MPM index in the MPM list as the intra prediction mode of the current block even without confirmation of the MPM flag.
[0131] When constructing the MPM list, in an embodiment, the encoding device / decoding device may derive a left mode as a candidate intra prediction mode of a left neighboring block of the current block, and may derive an upper mode as a candidate intra prediction mode of an upper neighboring block of the current block. Here, the left neighboring block may refer to a neighboring block located at the bottom among the left neighboring blocks adjacent to the left side of the current block, and the upper neighboring block may refer to a neighboring block located at the rightmost among the upper neighboring blocks adjacent to the current block. For example, if the size of the current block is W×H, and the x component and y component of the upper left sample position of the current block are xN and yN, respectively, the left neighboring block may be a block including samples in coordinates (xN-1, yN+H-1), and the upper neighboring block may be a block including samples in coordinates (xN+W-1, yN-1).
[0132] For example, when the left neighboring block is available and intra-frame prediction is applied to the left neighboring block, the encoding device / decoding device can derive the intra-frame prediction mode of the left neighboring block as the left candidate intra-frame prediction mode (i.e., left mode). When the upper neighboring block is available, intra-frame prediction is applied to the upper neighboring block, and the upper neighboring block is included in the current CTU, the decoding device can derive the intra-frame prediction mode of the upper neighboring block as the upper candidate intra-frame prediction mode (i.e., upper mode). Alternatively, when the left neighboring block is not available or intra-frame prediction is not applied to the left neighboring block, the encoding device / decoding device can derive the plane mode as the left mode. When the upper neighboring block is not available or intra-frame prediction is not applied to the upper neighboring block or the upper neighboring block is not included in the current CTU, the decoding device can derive the plane mode as the upper mode.
[0133] The encoding device / decoding device can construct an MPM list by deriving a candidate intra-frame prediction mode of the current block based on the left mode derived from the left neighboring block and the upper mode derived from the upper neighboring block. In this case, the MPM list may include the left mode and the upper mode, and may also include a specific intra-frame prediction mode according to a predetermined method.
[0134] In addition, when constructing the MPM list, a normal intra prediction mode (normal intra prediction type) may be applied, or a specific intra prediction type (eg, MRL or ISP) may be applied. This document proposes a method for constructing an MPM list when not only a normal intra prediction method (normal intra prediction type) but also a specific intra prediction type (eg, MRL or ISP) is applied, and this will be described later.
[0135] In addition, the intra prediction mode may include a non-directional (or non-angle) intra prediction mode and a directional (or angle) intra prediction mode. For example, in the HEVC standard, intra prediction modes including 2 non-directional prediction modes and 33 directional prediction modes are used. The non-directional prediction mode may include a plane intra prediction mode (ie, No. 0) and a DC intra prediction mode (ie, No. 1). The directional prediction mode may include intra prediction modes No. 2 to No. 34. The plane intra prediction mode may be referred to as a plane mode, and the DC intra prediction mode may be referred to as a DC mode.
[0136] Alternatively, to obtain a given edge direction proposed in a natural video, such as Fig.10 As in, the directional intra prediction mode can be expanded from the existing 33 modes to 65 modes. In this case, the intra prediction mode may include 2 non-directional intra prediction modes and 65 directional intra prediction modes. The non-directional intra prediction mode may include a plane intra prediction mode (ie, No. 0) and a DC intra prediction mode (ie, No. 1). The directional intra prediction mode may include intra prediction modes No. 2 to No. 66. The extended directional intra prediction mode may be applied to blocks of all sizes and may be applied to both luminance components and chrominance components. However, this is an example, and the embodiments of this document may be applied to situations where the number of intra prediction modes is different. Intra prediction mode No. 67 may also be used depending on the situation. Intra prediction mode No. 67 may indicate a linear model (LM) mode.
[0137] Fig.10 An example of an intra prediction mode to which embodiments of this document can be applied is illustrated.
[0138] Reference Fig.10 , based on the intra prediction mode No. 34 with the upper left diagonal prediction direction, the mode can be divided into an intra prediction mode with horizontal directionality and an intra prediction mode with vertical directionality. Fig.10, H and V mean horizontal directivity and vertical directivity, respectively. Each of numbers -32 to 32 indicates a displacement of 1 / 32 unit at a sample grid position. Intra-prediction modes No. 2 to 33 have horizontal directivity, and intra-prediction modes No. 34 to 66 have vertical directivity. Intra-prediction mode No. 18 and intra-prediction mode No. 50 indicate a horizontal intra-prediction mode and a vertical intra-prediction mode, respectively. Intra-prediction mode No. 2 may be referred to as a lower left diagonal intra-prediction mode, intra-prediction mode No. 34 may be referred to as an upper left diagonal intra-prediction mode, and intra-prediction mode No. 66 may be referred to as an upper right diagonal intra-prediction mode.
[0139] In addition, intra prediction may use MRL using multiple reference lines. The MRL method may perform intra prediction using neighboring samples located on a sample line that is one to three samples away from the upper side and / or left side of the current block as reference samples.
[0140] Fig.11 An example of reference sample lines for intra prediction using multiple reference lines is illustrated. Fig.11 The block unit shown in may indicate a current block.
[0141] According to an exemplary embodiment, intra prediction may use a reference sample (or a reference sample that is first closest to the current block, i.e., a reference sample that is located at a sample distance of zero from the current block) as a reference sample for prediction. According to another exemplary embodiment, multi-reference line (MRL) intra prediction is a method of using reference samples that are located at a sample distance of K (K is an integer of 1 or greater) from the left and upper boundaries of the current block, and may have more options for reference samples and more accurate prediction performance than intra prediction using a reference sample that is closest to the current block (i.e., located at a sample distance of zero from the current block). Reference samples of the current block may also be referred to as neighboring samples of the current block or reference line samples of the current block, and reference line samples may also be referred to as samples on a reference line.
[0142] Reference Fig.11 , the positions of neighboring reference samples located at 0, 1, 2, and 3 sample distances from the current block may be referred to as reference lines 0, 1, 2, and 3, respectively. Reference lines may be referred to as reference sample lines, reference sample rows, or reference sample columns, or may also be simply referred to as lines, rows, or columns. Reference lines 0, 1, 2, and 3 may be positioned in an order close to the current block. As an example, multi-reference line intra prediction may be performed based on reference lines 1 and 2. As another example, multi-reference line intra prediction may be performed based on reference lines 1 and 3. However, multi-reference line intra prediction according to the present document is not necessarily limited to these examples.
[0143] In addition, intra prediction based on multiple reference lines (MRL) can signal reference line information indicating which reference line is used. For example, reference line information can be signaled in the form of an intra_luma_ref_idx syntax element. If the intra_luma_ref_idx value is equal to 0, it can indicate that the first reference sample closest to the current block (i.e., located at zero sample distance) is used to perform intra prediction. If the intra_luma_ref_idx value is equal to 1, it can indicate that the second reference sample closest to the current block (i.e., located at one sample distance) is used to perform intra prediction. If the intra_luma_ref_idx value is equal to 2, it can indicate that the third or fourth reference sample closest to the current block (i.e., located at 2 or 3 sample distances) is used to perform intra prediction.
[0144] At the same time, the block currently intended to be coded (encoded / decoded) is regarded as a coding unit, so that intra prediction can perform encoding / decoding without segmentation. In addition, intra prediction can also be performed by dividing the block currently intended to be coded into sub-partitions. Such an intra prediction method may be referred to as intra sub-partition (ISP) intra prediction or ISP-based intra prediction. That is, the ISP method can perform intra prediction by segmenting the block currently intended to be coded in the horizontal direction or in the vertical direction. In this case, a reconstructed block can be generated by performing encoding / decoding in units of segmented blocks, and the reconstructed block can be used as a reference block for the next segmented block. The current intra sub-partition (ISP) can be segmented according to the size of the block as in Table 1. Table 1 below shows the number of sub-partitions according to the block size when the intra sub-partition (ISP) mode is applied to the current block.
[0145] [Table 1]
[0146]
[0147] Fig.12 An example of splitting sub-partitions according to intra sub-partitioning (ISP) is illustrated.
[0148] exist Fig.12 , (a) represents an example of partitioning a current block (original HxW partition, ie, HxW-sized CU) in a horizontal direction and in a vertical direction when the current block is a 4x8 block or an 8x4 block.
[0149] like Fig.12As illustrated in (a), a 4x8 block or an 8x4 block may be partitioned in the horizontal direction or in the vertical direction. When the block is partitioned in the horizontal direction, the block may be partitioned into two sub-partition blocks of each (H / 2)xW size, and when the block is partitioned in the vertical direction, the block may be partitioned into two sub-partition blocks of each Hx(W / 2) size.
[0150] exist Fig.12 , (b) represents an example of partitioning the current block (original HxW partition, ie, HxW-sized CU) in the horizontal direction and in the vertical direction when the current block is a residual block excluding 4x4, 4x8, and 8x4 blocks.
[0151] like Fig.12 As illustrated in (b), the HxW block excluding 4x4, 4x8 and 8x4 blocks can be partitioned in the horizontal direction or in the vertical direction. In the case where the block is partitioned in the horizontal direction, the block can be partitioned into four sub-partition blocks each having a size of (H / 4)xW, and in the case where the block is partitioned in the vertical direction, the block can be partitioned into four sub-partition blocks each having a size of Hx(W / 4).
[0152] In order to reduce the coding complexity, the intra-frame sub-partitioning method generates an MPM list according to the corresponding partitioning method (horizontal partitioning and vertical partitioning), and generates the best mode by comparing the appropriate prediction mode among the prediction modes in the generated MPM list from the perspective of rate-distortion optimization (RDO). In addition, when the above-mentioned multi-reference line (MRL) intra-frame prediction is used, the intra-frame sub-partitioning method cannot be used. That is, the intra-frame sub-partitioning method can only be applied when the zeroth reference line (ie, the intra_luma_ref_idx value is equal to 0) is used. In addition, when the above-mentioned intra-frame sub-partitioning method is used, the above-mentioned PDPC cannot be used.
[0153] According to the intra subpartitioning method, whether intra subpartitioning is applied is first transmitted in units of blocks, and if the current block uses intra subpartitioning (intra_subpartitions_mode_flag), information on whether the partitioning method is horizontal partitioning or vertical partitioning (intra_subpartitions_split_flag) is encoded / decoded.
[0154] Next, Table 2 represents an example of signaling information (eg, intra_subpartitions_mode_flag or intra_subpartitions_split_flag) for applying subpartition intra prediction through a coding unit syntax.
[0155] [Table 2]
[0156]
[0157] In the case of applying the intra-frame sub-partitioning method, the intra-frame prediction mode for the current block is equally applied to the sub-partition, and the neighboring reference samples are derived and used in units of sub-partitions to enhance the intra-frame prediction performance. That is, in the case of applying the intra-frame sub-partitioning method, the residual sample processing process is performed in units of sub-partitions. In other words, the intra-frame prediction samples are derived with respect to each sub-partition, and the residual signal (residual sample) of the corresponding sub-partition is added to the derived intra-frame prediction sample to obtain the reconstructed sample. The residual signal (residual sample) can be derived through the dequantization / inverse transform process based on the residual information (quantized transform coefficient information or residual coding syntax) in the above-mentioned bitstream. That is, the derivation of the prediction sample for the first sub-partition and the derivation of the residual sample can be performed, and based on this, the reconstructed sample of the first sub-partition can be derived. In this case, in the case of deriving the prediction sample of the second sub-partition, some of the reconstructed samples in the first sub-partition (for example, the upper or left neighboring reference sample of the second sub-partition) can be used as the neighboring reference sample of the second sub-partition. In the same manner, the derivation of the prediction samples and the derivation of the residual samples for the second sub-partition may be performed, and based on this, the reconstructed samples of the second sub-partition may be derived. In this case, in the case of deriving the prediction samples of the third sub-partition, some of the reconstructed samples in the second sub-partition (e.g., the upper or left neighboring reference samples of the third sub-partition) may be used as neighboring reference samples for the third sub-partition. The intra-sub-partitioning method may be applied in the same manner even with respect to the remaining sub-partitions.
[0158] As described above, as intra prediction, a multi-reference line (MRL)-based intra prediction method and a sub-partition (ISP)-based intra prediction method may be applied, or a normal intra prediction method excluding a specific intra prediction method such as MRL, ISP, etc. may be applied. In this case, normal intra prediction that is not a specific intra prediction type (e.g., MRL or ISP) uses 67 intra prediction modes to perform intra prediction encoding / decoding, and multi-reference line intra prediction uses 65 intra prediction modes excluding plane mode and DC mode to perform intra prediction encoding / decoding. In addition, sub-partition intra prediction uses 66 intra prediction modes excluding DC mode to perform intra prediction encoding / decoding. Since the above three types of intra prediction (existing intra prediction, multi-reference line intra prediction, and sub-partition intra prediction) use different numbers of intra prediction modes to perform intra prediction encoding / decoding, they have different MPM list generation methods for their predictions.
[0159] More specifically, normal intra prediction uses all 67 intra prediction modes to construct an MPM list including 6 MPM candidates. Multi-reference line intra prediction does not use planar mode and DC mode, and uses 65 intra prediction modes excluding planar mode and DC mode to construct an MPM list including 6 MPM candidates. Sub-partition intra prediction does not use DC mode, but uses 66 intra prediction modes excluding DC mode to construct an MPM list including 6 MPM candidates. In this case, sub-partition intra prediction constructs the MPM list in different methods depending on the horizontal partition and the vertical partition. As described above, for one intra prediction, different methods are used to construct an MPM list including 6 MPM candidates.
[0160] Therefore, in order to improve the coding efficiency of intra prediction, a unified MPM list generation method can be used. Therefore, this document proposes a scheme that can construct the MPM list used in normal intra prediction, multi-reference line intra prediction and sub-partition intra prediction in a unified method. In an embodiment, it is possible to generate an MPM list for normal intra prediction, multi-reference line intra prediction and sub-partition intra prediction after generating a unified temporary MPM list. In another embodiment, after generating a unified temporary MPM list, a specific prediction mode (e.g., DC mode) can be added according to a specific intra prediction type (i.e., normal intra prediction, multi-reference line intra prediction and sub-partition intra prediction), and taking this into account, an MPM list for normal intra prediction, multi-reference line intra prediction and sub-partition intra prediction can be generated.
[0161] By using the unified MPM list generation method according to the embodiment of this document, the encoding / decoding structure of intra-frame prediction can be simplified, and the video encoding / decoding efficiency can be improved by improving the intra-frame mode encoding / decoding efficiency.
[0162] Fig.13 is a diagram illustrating an implementation of a method for generating a unified MPM list according to the present document.
[0163] In this embodiment, a method for constructing a unified MPM list considering unused specific prediction modes (e.g., planar mode or DC mode) according to specific intra prediction types (i.e., normal intra prediction, multi-reference line intra prediction, and subpartition intra prediction) will be described.
[0164] As an embodiment, the method for generating an MPM list including 6 MPM candidates used in normal intra prediction may be equally applied to the MPM list generation method for multi-reference line intra prediction and sub-partition intra prediction. In this case, the MPM list generation method used in normal intra prediction may be an existing MPM list generation method, or a method by improving the existing MPM list generation method. For example, Figure 8 and Fig. 9 The above method is used to construct the MPM list used in normal intra prediction.
[0165] Here, since normal intra prediction takes into account all 67 intra prediction modes to generate the MPM list, it includes planar mode and DC mode. However, since multi-reference line intra prediction does not use planar mode and DC mode, and sub-partition intra prediction does not use DC mode, they can consider this to generate their corresponding MPM lists.
[0166] Reference Fig.13 , an MPM list including 6 MPM candidates used in normal intra prediction may be temporarily generated. For convenience of explanation, the MPM list temporarily generated first is referred to as a temporary MPM list. In other words, the temporary MPM list is an MPM list including 6 MPM candidates used in normal intra prediction (or an MPM list improved by various improvement methods), and may be generated according to Figure 8 and Fig. 9 The above method is used to construct the temporary MPM list. Even when generating the MPM list for multi-reference line intra prediction and sub-partition intra prediction, such a temporary MPM list construction method can be equally applied. Therefore, a temporary MPM list including equal 6 MPM candidates can be generated for all normal intra prediction, multi-reference line intra prediction and sub-partition intra prediction.
[0167] In this case, in the case of multi-reference line intra prediction, the planar mode and the DC mode may not be used, and in the case of sub-partition intra prediction, the DC mode may not be used. In this case, a specific mode not used in the corresponding prediction method can be removed from the equally generated temporary MPM list, and a suitable MPM list can be reconstructed according to the corresponding prediction method.
[0168] As an example, Fig.13 As illustrated in (a), (b) and (c) of FIG. 1 , a temporary MPM list can be generated with respect to normal intra prediction, multi-reference line intra prediction and sub-partition intra prediction. In this case, each of the temporary MPM lists equally includes 6 MPM candidates. In addition, since multi-reference line intra prediction does not use planar mode and DC mode, as shown in FIG. Fig.13As illustrated in (b), the plane mode of MPM index 0 and the DC mode of MPM index 2 can be removed from the temporary MPM list, and the MPM candidates in the temporary MPM list can be rearranged. Therefore, an MPM list including 4 MPM candidates can be finally generated. In addition, since the DC mode is not used in the sub-partition intra prediction, as shown in FIG. Fig.13 As illustrated in (c), the DC mode of MPM index 2 may be removed from the temporary MPM list, and the MPM candidates in the temporary MPM list may be rearranged. Therefore, an MPM list including 5 MPM candidates may be finally generated.
[0169] Fig.14 is a diagram illustrating another embodiment of a method for generating a unified MPM list according to the present document.
[0170] In this embodiment, a method will be described in which a specific prediction mode (e.g., DC mode) is added according to a specific intra prediction type (i.e., normal intra prediction, multi-reference line intra prediction, and sub-partition intra prediction), and a unified MPM list is constructed in consideration of this. As an example, a method for constructing a unified MPM list in the case where the DC mode is additionally used in multi-reference line intra prediction will be described.
[0171] As an embodiment, the method for generating an MPM list including 6 MPM candidates used in normal intra prediction may be equally applied to the MPM list generation method for multiple reference line intra prediction and sub-partition intra prediction. In this case, the MPM list generation method used in normal intra prediction may be an existing MPM list generation method, or a method by improving the existing MPM list generation method. For example, the above-mentioned Figure 8 and Fig. 9 Method to construct the MPM list used in normal intra prediction.
[0172] In this case, since prediction is performed by adding the DC mode in multi-reference line intra prediction, the DC mode is used instead of the planar mode. In addition, the DC mode is not used in the sub-partition intra prediction. In this embodiment, this can be considered to generate the corresponding MPM list.
[0173] Reference Fig.14 , an MPM list including 6 MPM candidates used in normal intra prediction may be temporarily generated. For convenience of explanation, the MPM list temporarily generated first is referred to as a temporary MPM list. In other words, the temporary MPM list is an MPM list including 6 MPM candidates used in normal intra prediction (or an MPM list improved by various improvement methods), and may be generated according to Figure 8 and Fig. 9The above method is used to construct the temporary MPM list. Even when generating the MPM list for multi-reference line intra prediction and sub-partition intra prediction, such a temporary MPM list construction method can be equally applied. Therefore, a temporary MPM list including equal 6 MPM candidates can be generated for all normal intra prediction, multi-reference line intra prediction and sub-partition intra prediction.
[0174] In this case, in the case of multi-reference line intra prediction, the DC mode may be used and the planar mode may not be used, whereas in the case of sub-partition intra prediction, the DC mode may not be used. In this case, a specific mode not used in the corresponding prediction method may be removed from the equally generated temporary MPM list, and a suitable MPM list may be reconstructed according to the corresponding prediction method.
[0175] As an example, Fig.14 As illustrated in (a), (b) and (c) of FIG. 1 , a temporary MPM list may be generated with respect to normal intra prediction, multi-reference line intra prediction and sub-partition intra prediction. In this case, each of the temporary MPM lists equally includes 6 MPM candidates. In addition, since multi-reference line intra prediction additionally uses the DC mode instead of the planar mode, as shown in FIG. Fig.14 As illustrated in (b), the plane mode of MPM index 0 can be removed from the temporary MPM list, and the MPM candidates in the temporary MPM list can be rearranged. Therefore, an MPM list including 5 MPM candidates can be finally generated. In addition, since the sub-partition intra prediction does not use the DC mode, as shown in Fig.14 As illustrated in (c), the DC mode of MPM index 2 may be removed from the temporary MPM list, and the MPM candidates in the temporary MPM list may be rearranged. Therefore, an MPM list including 5 MPM candidates may be finally generated.
[0176] Fig.13 and Fig.14An example of generating an MPM list will be described, and the basic concept of the proposed method is to generate an MPM list including multiple (e.g., 6) MPM candidates in normal intra prediction, and then use the generated MPM list equally even in multi-reference line intra prediction and sub-partition intra prediction. However, since a specific mode (e.g., plane mode or DC mode) is not used in multi-reference line intra prediction and sub-partition intra prediction, the MPM candidates can be configured in consideration of this. If there is a specific mode (e.g., plane mode or DC mode) that is not used in the corresponding prediction method in the MPM list, the specific mode can be removed, and the MPM candidates can be rearranged so that the MPM list for the corresponding prediction method can be constructed. In addition, in the above-mentioned embodiment, it has been described that an MPM list (i.e., a temporary MPM list) including multiple (e.g., 6) MPM candidates for the corresponding prediction method is generated, and then the MPM list is finally constructed by removing the specific mode (e.g., plane mode or DC mode) that is not used in the corresponding prediction method. However, this is merely exemplary, and the process of generating a temporary MPM list can be omitted. For example, after deriving 6 MPM candidates (without generating a temporary MPM list), an MPM list suitable for the corresponding prediction method can be constructed by excluding a specific mode (e.g., planar mode or DC mode) not used in the corresponding prediction method. In this case, the planar mode is included in the 6 MPM candidates, and in the case where the planar mode is not used like multi-reference line intra prediction, 5 MPM candidates excluding this mode can be derived, and an MPM list including the 5 MPM candidates can be constructed.
[0177] In addition, the above-mentioned embodiment ( Fig.13 and Fig.14 The embodiments described above are merely exemplary to help understand the basic concept of the unified MPM list generation method proposed in this document. The basic concept of the method proposed in this document is to generate a unified MPM list without distinguishing between normal intra prediction, multi-reference line intra prediction, and sub-partition intra prediction when generating the MPM list. Therefore, even in multi-reference line intra prediction and sub-partition intra prediction, the MPM list generated in normal intra prediction can be used equally. In this case, the intra prediction modes that are not used in multi-reference line intra prediction and sub-partition intra prediction can be considered to finally construct an MPM list suitable for the corresponding intra prediction.
[0178] That is, according to the implementation scheme proposed in this document, since a unified MPM list can be constructed, the encoding / decoding structure of intra-frame prediction can be simplified. In addition, the overall video encoding / decoding performance can be improved by improving the encoding / decoding efficiency of the intra-frame prediction mode.
[0179] Additionally, when generating the above unified MPM list, this document proposes a method of constructing the same number of MPM candidates for normal intra prediction and specific types of intra prediction (ie, multi-reference line intra prediction, subpartition intra prediction) in the MPM list.
[0180] According to the implementation mode, as described above in Fig.13 and Fig.14 The method described in , for generating an MPM list including 6 MPM candidates used in normal intra prediction can be similarly applied to the MPM list generation method for multi-reference line intra prediction and sub-partition intra prediction. At this time, the MPM lists generated for normal intra prediction, multi-reference line intra prediction, and sub-partition intra prediction can all include the same number of MPM candidates (e.g., 6 MPM candidates). However, since multi-reference line intra prediction does not use planar mode and / or DC mode, and since sub-partition intra prediction does not use DC mode, specific modes that are not used in each prediction method can be removed from the MPM list including the 6 MPM candidates generated in the same way, and the MPM list suitable for the corresponding prediction method can be reconstructed.
[0181] When the MPM list is generated by using the above method, the MPM list for normal intra prediction may include 6 MPM candidates, the MPM list for multi-reference line intra prediction may include 4 or 5 MPM candidates, and the MPM list for subpartition intra prediction may include 5 MPM candidates.
[0182] That is, the number of MPM candidates for each of normal intra prediction, multi-reference line intra prediction, and sub-partition intra prediction may be constructed differently. Therefore, in order to construct the number of MPM candidates in the same way for each prediction method, a random intra prediction mode may be added to each MPM list corresponding to multi-reference line intra prediction and sub-partition intra prediction, so as to generate an MPM list that includes the same number of MPM candidates (e.g., 6 MPM candidates). For example, in the case of constructing an MPM list for multi-reference line intra prediction by including 4 MPM candidates, 2 random intra prediction modes may be added. Alternatively, in the case of constructing an MPM list for sub-partition intra prediction by including 5 MPM candidates, 1 random intra prediction mode may be added.
[0183] Alternatively, as another example, an MPM list having the same number of MPM candidates as normal intra prediction (e.g., 6 MPM candidates) may be generated by adding only a random intra prediction mode to the MPM list for multi-reference line intra prediction. Alternatively, an MPM list having the same number of MPM candidates as normal intra prediction (e.g., 6 MPM candidates) may be generated by adding only a random intra prediction mode to the MPM list for subpartition intra prediction. Alternatively, as described above, an MPM list each having the same number of MPM candidates as normal intra prediction (e.g., 6 MPM candidates) may be generated by adding a random intra prediction mode to the MPM list for each of multi-reference line intra prediction and subpartition intra prediction.
[0184] Additionally, as described above, the present document proposes a method for constructing the number of MPM candidates in the same manner in the MPM list for normal intra prediction and a specific intra prediction type (i.e., multi-reference line intra prediction, sub-partition intra prediction). At this time, the number of MPM candidates may be predetermined (e.g., 6 MPM candidates), as described above, or may be variably (or adjustably) determined. In the following, the present document proposes a method for variably adjusting the number of MPM candidates in the MPM list for normal intra prediction and a specific intra prediction type (i.e., multi-reference line intra prediction, sub-partition intra prediction).
[0185] According to an embodiment, in the case where the MPM list for each of normal intra prediction, multi-reference line intra prediction, and subpartition intra prediction is constructed to all include the same number of MPM candidates (e.g., 6 MPM candidates), the number of candidates may be variably adjusted to the same random number, such as 5 MPM candidates or 4 MPM candidates, etc. And, therefore, an MPM list including MPM candidates adjusted to a random number (e.g., 4 or 5 MPM candidates, etc.) may be finally generated.
[0186] Fig.15 is a flowchart schematically illustrating an encoding method that can be performed by an encoding device according to an embodiment of the present disclosure.
[0187] Fig.15 The method disclosed in can be Figure 2 The encoding device (200) disclosed in the embodiment of the present invention is implemented. More specifically, Fig.15 Steps S1500 to S1530 may be performed by Figure 2 The predictor (220) shown (more specifically, the intra predictor (222)) performs, Fig.15 Step S1540 may be performed by Figure 2 The residual processor (230) shown performs, and Fig.15 Step S1550 may be performed by Figure 2The entropy encoder (240) shown is implemented. Additionally, Fig.15 The method disclosed in the disclosure may include the above embodiments described in this document. Therefore, the detailed description of the contents of this document overlapping with the above embodiments shall be omitted or simplified.
[0188] Reference Fig.15 , the encoding apparatus may generate reference line index information related to a reference line representing a reference line used for intra prediction of a current block ( S1500 ).
[0189] The reference line index information represents a reference line used for multi-reference line (or multi-reference line, MRL) based intra prediction, and may be information indicating neighboring reference samples located at 0, 1, 2, and 3 sample distances from the current block.
[0190] For example, the reference line index information may be represented in the format of the above-described intra_luma_ref_idx syntax element, and based on the intra_luma_ref_idx value, the reference line index information may be an index value related to indicating any one of reference lines 0, 1, 2, and 3. As an example, in the case where the value of the reference line index information (e.g., intra_luma_ref_idx) is equal to 0, this may indicate that the reference line is referenced by using the first closest reference line ( Fig.11 Intra prediction is performed using samples of reference line 0 of the current block. And, in the case where the value of the reference line index information (e.g., intra_luma_ref_idx) is not equal to 0 (i.e., in the case where the value is equal to 1 to 3), this may indicate that the second to fourth closest reference lines ( Fig.11 Intra prediction is performed on samples of reference lines 1 to 3 of the reference line index information (eg, intra_luma_ref_idx). That is, when the value of the reference line index information (eg, intra_luma_ref_idx) is not equal to 0 (ie, when the value is equal to 1 to 3), this may mean that a multi-reference line (MRL)-based intra prediction method is used.
[0191] According to an embodiment, the encoding device may determine whether to perform intra prediction by applying a plurality of reference lines to the current block, and based on this determination, may generate reference line index information, and may signal this to the decoding device.
[0192] The encoding apparatus may construct a most probable mode (MPM) list by deriving candidate intra prediction modes of the current block based on a value of the reference line index information ( S1510 ).
[0193] According to an embodiment, the encoding device may construct an MPM list based on whether a specific intra prediction method (e.g., multi-reference line intra prediction, sub-partition intra prediction, etc.) is applied. At this time, the above embodiment may be applied to the process of constructing the MPM list. For example, Fig.13 and Fig.14 The method shown is to build the MPM list.
[0194] For example, the encoding device may generate an MPM list based on whether multi-reference line intra prediction is applied. For example, based on the case where the encoding device performs intra prediction by applying multiple reference lines to the current block, that is, the case where the reference line index information value is not equal to 0, the encoding device may derive a candidate intra prediction mode and then construct an MPM list. At this time, the encoding device may derive a DC mode as one of the candidate intra prediction modes and may construct an MPM list by including the derived DC mode therein. Additionally, the encoding device may derive a candidate intra prediction mode based on a predetermined number of MPM candidates. For example, the predetermined number of MPM candidates may be equal to 5, and in this case, the encoding device may construct an MPM list by deriving 5 candidate intra prediction modes including the DC mode. In other words, when the reference line index information value is not equal to 0 (when the reference line index information value is equal to 1 or 2), one of the candidate intra prediction modes in the MPM list may be a DC mode, and the number of candidate intra prediction modes including the DC mode may be equal to 5.
[0195] For example, when the number of candidate intra prediction modes in the MPM list is less than 5, the encoding device may add a specific intra prediction mode to the MPM list. That is, a specific intra prediction mode may be added to the MPM list until the MPM list includes 5 candidate intra prediction modes. In this article, the specific intra prediction mode may be a predetermined intra prediction mode. For example, the specific intra prediction mode may be a DC mode, or may be an intra prediction mode with a specific directionality.
[0196] Additionally, as described above, in the case where intra prediction is performed by applying multiple reference lines, the plane mode may not be available. Therefore, based on the case where the value of the reference line index information is not equal to 0, the encoding device may not include the plane mode in the candidate intra prediction modes in the MPM list. For example, for multi-reference line intra prediction, the encoding device may construct a temporary MPM list. At this time, in the case where the plane mode is included among the candidate intra prediction modes in the temporary MPM list, the encoding device may remove the plane mode from the temporary MPM list and may reconstruct this list into an MPM list. In this article, the process of constructing a temporary MPM list may be skipped (or omitted) in accordance with the MPM list implementation method. For example, since the plane mode is not used when the value of the reference line index information is not equal to 0, it is first determined based on the value of the reference line index information that the plane mode is not used as the intra prediction mode, and then, 5 candidate intra prediction modes (here, including the DC mode) excluding the plane mode are derived from the 6 candidate intra prediction modes. Thereafter, the MPM list may be composed of the derived 5 candidate intra prediction modes. Therefore, according to the algorithm for implementing the MPM list, the MPM list can be implemented without an intermediate process of constructing a temporary MPM list.
[0197] Alternatively, as another example, the encoding device may generate an MPM list based on subpartition mode information related to indicating whether subpartition intra prediction is used for the current block. For example, the subpartition mode information may use the above-mentioned intra_subpartitions_mode_flag syntax element. And, when the value of intra_subpartitions_mode_flag is equal to 1, this indicates that subpartition intra prediction is used for the current block. And, when the value of intra_subpartitions_mode_flag is equal to 0, this indicates that subpartition intra prediction is not used for the current block. As described above, since subpartition intra prediction cannot be performed when using multi-reference line intra prediction, when the value of the reference line index information is equal to 0, an MPM list may be generated based on the subpartition mode information. Additionally, when subpartition intra prediction is used, DC mode may not be available. Therefore, when the value of the reference line index information is equal to 0, and when the sub-partition mode information indicates that sub-partition intra prediction is used for the current block (for example, when the value of intra_subpartitions_mode_flag is equal to 1), the encoding device may not include the DC mode in the candidate intra prediction modes of the MPM list. At this time, the encoding device may generate the MPM list based on a predetermined number of candidates in the MPM list. For example, the predetermined number of candidates may be equal to 5, and in this case, the encoding device may construct the MPM list by deriving 5 candidate intra prediction modes that do not include the DC mode.
[0198] In other words, when the value of the reference line index information is equal to 0 and the value of the sub-partition mode information is equal to 1, the candidate intra-frame prediction modes in the MPM list do not include the DC mode, and the number of candidate intra-frame prediction modes that do not include the DC mode can be equal to 5.
[0199] For example, for sub-partition intra prediction, the encoding device can construct a temporary MPM list. At this time, in the case where the DC mode is included among the candidate intra prediction modes in the temporary MPM list, the encoding device can remove the DC mode from the temporary MPM list and can reconstruct this list into an MPM list. Here, the process of constructing a temporary MPM list can be skipped (or omitted) according to the MPM list implementation method. For example, since the DC mode is not used when sub-partition intra prediction is used, by not including the DC mode in the candidate intra prediction mode, an MPM list including 5 candidate intra prediction modes can be finally constructed. Therefore, the MPM list can be implemented without the intermediate process of constructing a temporary MPM list.
[0200] Additionally, in the case where the sub-partition mode information indicates that the sub-partition intra prediction mode is used for the current block, and in the case where the number of candidate intra prediction modes in the MPM list is less than 5, the encoding device may add a specific intra prediction mode to the MPM list. That is, a specific intra prediction mode may be added to the MPM list until the MPM list includes 5 candidate intra prediction modes. Here, the specific intra prediction mode may be a predetermined intra prediction mode. For example, the specific intra prediction mode may be a planar mode, or may be an intra prediction mode with a specific directionality.
[0201] Additionally, in the case where the value of the reference line index information is equal to 0, the encoding device may generate sub-partition mode information and may signal this information to the decoding device. In other words, in the case where multiple reference lines are not applied (i.e., in the case where intra prediction is performed by using the sample of the first closest current block reference line), the encoding device determines whether to perform sub-partition intra prediction, and, based on this determination, the encoding device may generate sub-partition mode information. At this time, the sub-partition mode information may be encoded as image information, and the encoded information may be output in a bitstream format.
[0202] The encoding apparatus may derive an intra prediction mode for the current block based on the MPM list ( S1520 ).
[0203] According to an embodiment, the encoding device may derive an intra prediction mode with an optimal rate-distortion (RD) cost by performing various intra prediction modes on the current block and may determine the derived intra prediction mode as the intra prediction mode of the current block. At this time, the encoding device may derive the optimal intra prediction mode of the current block based on the intra prediction modes including 2 non-directional intra prediction modes and 65 directional intra prediction modes. Alternatively, the encoding device may determine the optimal intra prediction mode by using only the MPM candidates configuring the MPM list.
[0204] For example, in a case where the value of the reference line index information is not equal to 0, the encoding device may derive the optimal intra prediction mode of the current block by using the MPM candidate intra prediction mode included in the MPM list. That is, in this case, the intra prediction mode of the current block can be determined only from the candidate intra prediction modes including the DC mode in the MPM list. Additionally, in a case where the value of the reference line index information is not equal to 0, the encoding device may not encode / signal the MPM flag information. As described above, in a case where the MPM flag information is not encoded / signaled, the value of the MPM flag information may be inferred to be 1. As described above, the MPM flag information may be represented in the format of the intra_luma_mpm_flag syntax element. For example, in a case where the value of intra_luma_mpm_flag is equal to 1, this may indicate that the intra prediction mode of the current block is selected from the MPM candidate intra prediction mode. And, in a case where the value of intra_luma_mpm_flag is equal to 0, this may indicate that the intra prediction mode of the current block is not selected from the MPM candidate intra prediction mode.
[0205] Based on the case where the value of the reference line index information is not equal to 0, as described above, the encoding device may generate MPM index information related to a candidate intra prediction mode among the candidate intra prediction modes indicated in the MPM list, the candidate intra prediction mode being derived as the intra prediction mode of the current block. That is, in the case where the value of the reference line index information is not equal to 0 and the value of the MPM flag information is inferred to be 1, the MPM index information may be encoded / signaled.
[0206] The encoding apparatus may generate a prediction sample of the current block based on the intra prediction mode of the current block ( S1530 ).
[0207] According to an embodiment, the encoding device may derive at least one neighboring reference sample among neighboring reference samples of the current block based on the intra prediction mode, and then the encoding device may generate a prediction sample based on the neighboring reference sample. Here, the neighboring reference sample may be derived based on the reference line index information. And, here, for example, the neighboring reference sample may include a neighboring reference sample included in a reference line, the reference line being indicated by the reference line index information.
[0208] The encoding apparatus may generate residual information for the current block based on the prediction sample of the current block ( S1540 ).
[0209] According to an embodiment, the encoding device may derive a residual sample of the current block based on the predicted sample of the current block and the original sample of the current block. And, the encoding device may generate residual information for the current block based on the residual sample. Here, the residual information may include information of a quantized transform coefficient derived by performing transformation and quantization on the residual sample, such as value information, position information, transformation scheme, transformation kernel, and quantization parameter.
[0210] The encoding apparatus may encode the image information including the reference line index information and the residual information (S1550).
[0211] According to an embodiment, the encoding device may encode image information including reference line index information and information related to residual samples of the current block, the reference line index information being determined based on whether intra prediction based on multiple reference lines (MRL) is applied as described above, and then the encoding device may output the encoded information in a bitstream format. Additionally, the encoding device may include sub-partition mode information determined based on whether sub-partition intra prediction is applied to the current block as described above, and MPM index information related to an intra prediction mode indicating the current block, etc. in the image information. And, then, the encoding device may encode the image information and then output the image information as a bitstream.
[0212] The bitstream can be sent to the decoding device via a network or a (digital) storage medium. In this article, the network may include a broadcast network and / or a communication network, etc., and the digital storage medium may include various types of storage media, such as USB, SD, CD, DVD, Blu-ray disc, HDD, SSD, etc.
[0213] Fig.16 is a flowchart schematically illustrating a decoding method that can be performed by a decoding device according to an embodiment of the present disclosure.
[0214] Fig.16 The method disclosed in can be Figure 3 The decoding device (300) disclosed in the embodiment of the present invention is executed. More specifically, Fig.16Steps S1600 to S1630 may be performed by Figure 3 The entropy decoder (310) and / or the predictor (330) shown (more specifically, the intra-frame predictor (331)) are executed. And, Fig.16 Step S1640 may be performed by Figure 3 The adder (340) disclosed in is executed. Additionally, Fig.16 The method disclosed in the disclosure may include the above embodiments described in this document. Therefore, the detailed description of the contents of this document overlapping with the above embodiments shall be omitted or simplified.
[0215] Reference Fig.16 , the decoding apparatus may obtain reference line index information related to a reference line representing an intra prediction for a current block from a bitstream ( S1600 ).
[0216] The reference line index information represents a reference line used for multi-reference line (or multi-reference line, MRL) based intra prediction, and may be information indicating neighboring reference samples located at 0, 1, 2, and 3 sample distances from the current block.
[0217] For example, the reference line index information may be represented in the format of the above-described intra_luma_ref_idx syntax element, and based on the intra_luma_ref_idx value, the reference line index information may be an index value related to indicating any one of reference lines 0, 1, 2, and 3. As an example, in the case where the value of the reference line index information (e.g., intra_luma_ref_idx) is equal to 0, this may indicate that the reference line is referenced by using the first closest reference line ( Fig.11 Intra prediction is performed using samples of reference line 0 of the current block. And, in the case where the value of the reference line index information (e.g., intra_luma_ref_idx) is not equal to 0 (i.e., in the case where the value is equal to 1 to 3), this may indicate that the second to fourth closest reference lines ( Fig.11 Intra prediction is performed on samples of reference lines 1 to 3 of the reference line index information (eg, intra_luma_ref_idx). That is, when the value of the reference line index information (eg, intra_luma_ref_idx) is not equal to 0 (ie, when the value is equal to 1 to 3), this may mean that a multi-reference line (MRL)-based intra prediction method is used.
[0218] According to an embodiment, the decoding device may obtain a reference line index information (e.g., intra_luma_ref_idx) syntax element from a bitstream and may then parse (decode) the obtained syntax information. Thereafter, the decoding device may obtain a value of the reference line index information (e.g., intra_luma_ref_idx) as a parsed result. Then, based on this value, the decoding device may determine whether multi-reference line intra prediction is being applied.
[0219] The decoding apparatus may construct a most probable mode (MPM) list by deriving a candidate intra prediction mode of the current block based on a value of the reference line index information ( S1610 ).
[0220] According to an embodiment, based on whether a specific intra prediction method (e.g., multi-reference line intra prediction, sub-partition intra prediction, etc.) is applied, the decoding device may construct an MPM list. At this time, the above embodiment may be applied to the process of constructing the MPM list. For example, Fig.13 and Fig.14 The method shown is to build the MPM list.
[0221] As an example, based on the reference line index information, the decoding device may determine whether multi-reference line intra prediction is being applied. And, according to the determined result, the decoding device may generate an MPM list. For example, based on the case where the decoding device performs intra prediction by applying multiple reference lines to the current block, that is, the case where the value of the reference line index information is not equal to 0, the decoding device may derive a candidate intra prediction mode and then construct an MPM list. At this time, the decoding device may derive a DC mode as one of the candidate intra prediction modes and may construct an MPM list by including the derived DC mode therein. Additionally, the decoding device may derive a candidate intra prediction mode based on a predetermined number of MPM candidates. For example, the predetermined number of MPM candidates may be equal to 5, and in this case, the decoding device may construct an MPM list by deriving 5 candidate intra prediction modes including the DC mode. In other words, when the value of the reference line index information is not equal to 0 (when the value of the reference line index information is equal to 1 or 2), one of the candidate intra prediction modes in the MPM list may be a DC mode, and the number of candidate intra prediction modes including the DC mode may be equal to 5.
[0222] For example, when the number of candidate intra prediction modes in the MPM list is less than 5, the decoding device may add a specific intra prediction mode to the MPM list. That is, a specific intra prediction mode may be added to the MPM list until the MPM list includes 5 candidate intra prediction modes. In this article, the specific intra prediction mode may be a predetermined intra prediction mode. For example, the specific intra prediction mode may be a DC mode, or may be an intra prediction mode with a specific directionality.
[0223] Additionally, as described above, in the case where intra prediction is performed by applying multiple reference lines, the plane mode may not be available. Therefore, based on the case where the value of the reference line index information is not equal to 0, the decoding device may not include the plane mode in the candidate intra prediction modes in the MPM list. For example, for multi-reference line intra prediction, the decoding device may construct a temporary MPM list. At this time, in the case where the plane mode is included among the candidate intra prediction modes in the temporary MPM list, the decoding device may remove the plane mode from the temporary MPM list and may reconstruct this list into an MPM list. In this article, the process of constructing a temporary MPM list may be skipped (or omitted) in accordance with the MPM list implementation method. For example, since the plane mode is not used when the value of the reference line index information is not equal to 0, it is first determined based on the reference line index information value that the plane mode is not used as the intra prediction mode, and then, 5 candidate intra prediction modes (here, including the DC mode) excluding the plane mode are derived from the 6 candidate intra prediction modes. Thereafter, the MPM list may be composed of the derived 5 candidate intra prediction modes. Therefore, according to the algorithm for implementing the MPM list, the MPM list can be implemented without an intermediate process of constructing a temporary MPM list.
[0224] Alternatively, as another example, the decoding device may generate an MPM list based on subpartition mode information related to indicating whether subpartition intra prediction is used for the current block. For example, the subpartition mode information may use the above-mentioned intra_subpartitions_mode_flag syntax element. And, when the value of intra_subpartitions_mode_flag is equal to 1, this indicates that subpartition intra prediction is used for the current block. And, when the value of intra_subpartitions_mode_flag is equal to 0, this indicates that subpartition intra prediction is not used for the current block. As described above, since subpartition intra prediction cannot be performed when using multi-reference line intra prediction, when the value of the reference line index information is equal to 0, an MPM list may be generated based on the subpartition mode information. Additionally, when subpartition intra prediction is used, DC mode may not be available. Therefore, when the value of the reference line index information is equal to 0, and when the sub-partition mode information indicates that sub-partition intra prediction is used for the current block (for example, when the value of intra_subpartitions_mode_flag is equal to 1), the decoding device may not include the DC mode in the candidate intra prediction modes of the MPM list. At this time, the decoding device may generate the MPM list based on the predetermined number of candidates in the MPM list. For example, the predetermined number of candidates may be equal to 5, and in this case, the decoding device may construct the MPM list by deriving 5 candidate intra prediction modes that do not include the DC mode.
[0225] In other words, when the value of the reference line index information is equal to 0 and the value of the sub-partition mode information is equal to 1, the candidate intra-frame prediction modes in the MPM list do not include the DC mode, and the number of candidate intra-frame prediction modes that do not include the DC mode can be equal to 5.
[0226] For example, for sub-partition intra prediction, the decoding device can construct a temporary MPM list. At this time, in the case where the DC mode is included among the candidate intra prediction modes in the temporary MPM list, the decoding device can remove the DC mode from the temporary MPM list and can reconstruct this list into an MPM list. Here, the process of constructing (or configuring) the temporary MPM list can be skipped (or omitted) according to the MPM list implementation method. For example, since the DC mode is not used when sub-partition intra prediction is used, by not including the DC mode in the candidate intra prediction mode, an MPM list including 5 candidate intra prediction modes can be finally constructed thereafter. Therefore, the MPM list can be implemented without the intermediate process of constructing a temporary MPM list.
[0227] Additionally, in the case where the sub-partition mode information indicates that the sub-partition intra prediction mode is used for the current block, and in the case where the number of candidate intra prediction modes in the MPM list is less than 5, the decoding device may add a specific intra prediction mode to the MPM list. That is, a specific intra prediction mode may be added to the MPM list until the MPM list includes 5 candidate intra prediction modes. Here, the specific intra prediction mode may be a predetermined intra prediction mode. For example, the specific intra prediction mode may be a planar mode, or may be an intra prediction mode with a specific directionality.
[0228] Additionally, in the case where the value of the reference line index information is equal to 0, the decoding device can obtain the sub-partition mode information from the bitstream. In other words, only in the case where multiple reference lines are not applied (that is, in the case where intra-frame prediction is performed by using the sample of the first closest reference line for the current block), the decoding device can obtain and decode the sub-partition information related to the indication of whether to perform sub-partition intra-frame prediction. At this time, in the case where the value of the reference line index information is equal to 0, and in the case where the value of the sub-partition mode information is equal to 1, the decoding device may not use the DC mode as the intra-frame prediction mode of the current block. That is, in this case, the decoding device can generate an MPM list by not including the DC mode in the candidate intra-frame prediction mode.
[0229] The decoding apparatus may derive an intra prediction mode of the current block based on the MPM list ( S1620 ).
[0230] According to an embodiment, the decoding device may obtain information about the intra prediction mode of the current block from the bitstream. As information for indicating the intra prediction mode of the current block, the information about the intra prediction mode may include MPM flag information, MPM index information, residual mode information, etc.
[0231] At this time, in the case where the value of the reference line index information is not equal to 0, the MPM flag information may not be signaled from the encoding device. As described above, in the case where the MPM flag information is not signaled, the decoding device may infer the value of the MPM flag information to be 1. As described above, the MPM flag information may be signaled in the format of the intra_luma_mpm_flag syntax element. For example, in the case where the value of intra_luma_mpm_flag is equal to 1, this may indicate that the intra-frame prediction mode of the current block is selected from the MPM candidate intra-frame prediction mode. And, in the case where the value of intra_luma_mpm_flag is equal to 0, this may indicate that the intra-frame prediction mode of the current block is not selected from the MPM candidate intra-frame prediction mode.
[0232] Additionally, in a case where the value of the reference line index information is not equal to 0, and in a case where the value of the MPM flag information is inferred to be 1, the MPM index information may be signaled from the encoding device. That is, the decoding device may obtain the MPM index information from the bitstream, and then, the decoding device may decode the obtained MPM index information. As described above, the MPM index information may include an index value indicating an intra prediction mode of a current block among candidate intra prediction modes included in the MPM list, and may be represented, for example, in the format of an intra_luma_mpm_idx syntax element.
[0233] That is, when the value of the reference line index information is not equal to 0, and when the value of the MPM flag information is inferred to be 1, the decoding device can obtain the MPM index information and decode it, and can select the candidate intra-frame prediction mode indicated by the MPM index information from among the candidate intra-frame prediction modes in the MPM list. Thereafter, the decoding device can derive the selected candidate intra-frame prediction mode as the intra-frame prediction mode of the current block.
[0234] The decoding apparatus may generate a prediction sample of the current block based on the intra prediction mode of the current block ( S1630 ).
[0235] According to an embodiment, the decoding device may derive at least one neighboring reference sample among the neighboring reference samples of the current block based on the intra prediction mode derived as described above. And then, the decoding device may generate a prediction sample based on the neighboring reference sample. Here, the neighboring reference sample may be derived based on the reference line index information. And, here, for example, the neighboring reference sample may include a neighboring reference sample included in a reference line, the reference line being indicated by the reference line index information.
[0236] The decoding apparatus may generate a reconstructed sample of the current block based on the prediction sample ( S1640 ).
[0237] According to an embodiment, the decoding device may directly use the prediction samples as reconstructed samples according to the prediction mode, or the decoding device may generate reconstructed samples by adding residual samples to the prediction samples.
[0238] In the presence of residual samples of the current block, the decoding device may receive information about the residual of the current block. The information about the residual may include the transform coefficients of the residual samples. The decoding device may derive the residual samples (or residual sample arrays) of the current block based on the residual information. The decoding device may generate reconstructed samples based on the predicted samples and the residual samples, and the decoding device may derive a reconstructed block or a reconstructed picture based on the reconstructed samples. Thereafter, as described above, if necessary, the decoding device may apply an in-loop filtering process such as a deblocking filter and / or an SAO process to the reconstructed picture in order to enhance the subjective / objective image quality.
[0239] In the above-mentioned embodiments, although the methods have been described based on flowcharts in the form of a series of steps or units, the embodiments of this document are not limited to the order of these steps, and some of these steps may be performed in an order different from the order of other steps or may be performed simultaneously with other steps. In addition, those skilled in the art will understand that the steps shown in the flowcharts are not exclusive, and without affecting the scope of rights of this document, these steps may include additional steps or one or more steps in the flowcharts may be deleted.
[0240] The above-mentioned method according to the present document may be implemented in software form, and the encoding device and / or decoding device according to the present document may be included in an apparatus for performing image processing, such as a TV, a computer, a smart phone, a set-top box, or a display device.
[0241] In this document, when the implementation is implemented in software form, the above-mentioned method can be implemented as a module (program, function, etc.) for performing the above-mentioned functions. The module can be stored in a memory and executed by a processor. The memory can be arranged inside or outside the processor and connected to the processor by various well-known means. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits and / or data processing devices. The memory may include a read-only memory (ROM), a random access memory (RAM), a flash memory, a memory card, a storage medium and / or other storage devices. That is, the implementation described in this document can be implemented and executed on a processor, a microprocessor, a controller or a chip. For example, the functional units illustrated in the accompanying drawings can be implemented and executed on a computer, a processor, a microprocessor, a controller or a chip. In this case, information (e.g., information about instructions) or algorithms for such implementation can be stored in a digital storage medium.
[0242] In addition, the decoding device and the encoding device to which this document is applied may be included in a multimedia broadcast transmission and reception device, a mobile communication terminal, a home theater video device, a digital theater video device, a surveillance camera, a video chat device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camera, a video on demand (VoD) service providing device, an over-the-top (OTT) video device, an Internet streaming service providing device, a three-dimensional (3D) video device, a virtual reality (VR) device, an augmented reality (AR) device, a video phone video device, a transportation means terminal (e.g., a vehicle (including an autonomous vehicle) terminal, an airplane terminal, and a ship terminal) and a medical video device, and may be used to process a video signal or a data signal. For example, an over-the-top (OTT) video device may include a game console, a Blueray player, an Internet access TV, a home theater system, a smart phone, a tablet PC, and a digital video recorder (DVR).
[0243] In addition, the processing method applying the present document can be generated in the form of a program executed by a computer and can be stored in a computer-readable recording medium. The multimedia data with a data structure according to the present document can also be stored in a computer-readable recording medium. Computer-readable recording media include all kinds of storage devices storing computer-readable data. Computer-readable recording media may include, for example, Blueray discs (BD), universal serial buses (USB), ROMs, PROMs, EPROMs, EEPROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. In addition, computer-readable recording media include media implemented in the form of carrier waves (e.g., transmitted over the Internet). In addition, the bit stream generated using the encoding method can be stored in a computer-readable recording medium or can be transmitted over wired and wireless communication networks.
[0244] In addition, the embodiments of this document may be implemented as a computer program product using program code. The program code may be executed by a computer according to the embodiments of this document. The program code may be stored on a carrier that can be read by a computer.
[0245] Fig.17 An example of a content streaming system to which the embodiments disclosed in this document can be applied is illustrated.
[0246] Reference Fig.17 The content streaming transmission system to which the implementation mode of this document is applied may mainly include an encoding server, a streaming transmission server, a web server, a media storage device, a user device and a multimedia input device.
[0247] The encoding server compresses the content input from the multimedia input device such as a smart phone, a camera, a camcorder, etc. into digital data to generate a bit stream, and transmits the bit stream to the streaming server. As another example, when the multimedia input device such as a smart phone, a camera, a camcorder, etc. directly generates a bit stream, the encoding server can be omitted.
[0248] A bitstream may be generated by applying the encoding method or the bitstream generating method of the embodiment of this document, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.
[0249] The streaming server sends multimedia data to the user device through the web server based on the user's request, and the web server is used as a medium to inform the user of the service. When the user requests the desired service from the web server, the web server transmits it to the streaming server, and the streaming server sends the multimedia data to the user. In this case, the content streaming system may include a separate control server. In this case, the control server is used to control the command / response between the devices in the content streaming system.
[0250] The streaming server can receive content from a media storage device and / or an encoding server. For example, when receiving content from an encoding server, the content can be received in real time. In this case, in order to provide a stable streaming service, the streaming server can store the bit stream for a predetermined time.
[0251] Examples of user equipment may include mobile phones, smart phones, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigators, touch-screen PCs, tablet PCs, ultrabooks, wearable devices (e.g., smart watches, smart glasses, head-mounted displays), digital TVs, desktop computers, digital signage, etc.
[0252] The various servers in the content streaming system may operate as distributed servers, in which case data received from the various servers may be distributed.
[0253] The claims in this specification can be combined in various ways. For example, the technical features in the method claims of this specification can be combined to be implemented or performed in a device, and the technical features in the device claims can be combined to be implemented or performed in a method. In addition, the technical features in the method claims and the device claims can be combined to be implemented or performed in a device. In addition, the technical features in the method claims and the device claims can be combined to be implemented or performed in a method.
Claims
1. An image decoding device, comprising: Memory; as well as at least one processor connected to the memory, wherein the at least one processor is configured to: Obtain reference line index information indicating whether multiple reference lines are used for intra prediction of the current block from a bitstream, Wherein, the value of the reference line index information equal to 0 indicates that the multiple reference lines are not used for intra-frame prediction of the current block, and Wherein, the value of the reference line index information is not equal to 0, indicating that the multiple reference lines are used for intra-frame prediction of the current block; deriving a most probable mode candidate for the current block based on a value of the reference line index information; deriving an intra prediction mode for the current block based on the most probable mode candidate; generating prediction samples for the current block based on the intra prediction mode; and generating reconstructed samples for the current block based on the predicted samples, wherein the number of the most probable mode candidates for the current block is variably determined depending on whether the multiple reference lines are used for intra prediction of the current block, wherein, based on the value of the reference line index information not being equal to 0, the number of the most probable mode candidates for the current block is determined to be 5, and the planar mode is not included in the most probable mode candidates, and Wherein, based on the value of the reference line index information being equal to 0, the number of the most probable mode candidates for the current block is determined to be 6.
2. The device according to claim 1, wherein: The at least one processor is further configured to derive the intra prediction mode for the current block, select the intra prediction mode of the current block from among the most probable mode candidates in the most probable mode list by obtaining the most probable mode index information, The most probable mode index information has a value related to an intra prediction mode for the current block among candidate intra prediction modes in the most probable mode list.
3. The device according to claim 1, wherein: The value of the reference line index information being equal to 0 indicates that intra prediction is performed by using samples of the first closest reference line for the current block, and The fact that the value of the reference line index information is not equal to 0 indicates that intra prediction is performed by using one of the reference lines among the second to fourth closest reference lines for the current block.
4. An image encoding device, comprising: Memory; as well as at least one processor connected to the memory, wherein the at least one processor is configured to: generating reference line index information indicating whether multiple reference lines are used for intra prediction of a current block; Wherein, the value of the reference line index information equal to 0 indicates that the multiple reference lines are not used for intra-frame prediction of the current block, and Wherein, the value of the reference line index information is not equal to 0, indicating that the multiple reference lines are used for intra-frame prediction of the current block; deriving a most probable mode candidate for the current block based on a value of the reference line index information; deriving an intra prediction mode for the current block based on the most probable mode candidate; generating prediction samples for the current block based on the intra prediction mode; generating residual information for the current block based on the prediction samples; and encoding the image information including the reference line index information and the residual information, wherein the number of the most probable mode candidates for the current block is variably determined depending on whether the multiple reference lines are used for intra prediction of the current block, wherein, based on the value of the reference line index information not being equal to 0, the number of the most probable mode candidates for the current block is determined to be 5, and the planar mode is not included in the candidate intra prediction modes in the most probable mode candidates, and Wherein, based on the value of the reference line index information being equal to 0, the number of the most probable mode candidates for the current block is determined to be 6.
5. The device of claim 4, wherein the at least one processor is further configured to: generating most probable mode index information related to the intra prediction mode for the current block among the most probable mode candidates in the most probable mode list, in, The image information is encoded to include the most probable mode index information.
6. A device for transmitting data for image information, comprising: at least one processor configured to generate a bitstream for the image information, wherein the bitstream is generated based on obtaining the bitstream for the image information, The bitstream is generated based on the following steps: generating reference line index information indicating whether multiple reference lines are used for intra prediction of a current block, deriving a most probable mode candidate for the current block based on a value of the reference line index information, deriving an intra prediction mode for the current block based on the most probable mode candidate, generating a prediction sample for the current block based on the intra prediction mode, generating residual information for the current block based on the prediction sample, and encoding the image information including the reference line index information and the residual information; and a transmitter configured to transmit the data including the bit stream, The value of the reference line index information being equal to 0 indicates that the multiple reference lines are not used for intra-frame prediction of the current block. The value of the reference line index information is not equal to 0, indicating that the multiple reference lines are used for intra-frame prediction of the current block. wherein the number of the most probable mode candidates for the current block is variably determined depending on whether the multiple reference lines are used for intra prediction of the current block, wherein, based on the value of the reference line index information not being equal to 0, the number of the most probable mode candidates for the current block is determined to be 5, and the planar mode is not included in the candidate intra prediction modes in the most probable mode candidates, Wherein, based on the value of the reference line index information being equal to 0, the number of the most probable mode candidates for the current block is determined to be 6.
