Image encoding / decoding method based on spatial geometric partition mode, method for transmitting bit stream, and recording medium for storing bit stream

Abstract

An image encoding / decoding method, a bitstream transmission method, and a computer-readable recording medium having a bitstream stored thereon are provided. An image decoding method according to the present disclosure may be an image decoding method performed by an image decoding device, and includes the steps of: configuring candidates for a spatial geometric partition mode (SGPM), each of which includes a prediction mode for an intra prediction mode of each partition partitioned by a partition mode of the SGPM; and predicting the current block based on any one of the candidates, in which a prediction mode of a first candidate among the candidates is determined as information indicating an intra block copy prediction mode of the reference position.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an image encoding / decoding method, a method of transmitting a bitstream, and a recording medium storing a bitstream, and relates to prediction based on a spatial geometric partition mode. BACKGROUND

[0002] Recently, the demand for high-resolution and high-quality images, such as high-definition (HD) images and ultra-high-definition (UHD) images, is increasing in various fields. As the resolution and quality of image data increase, the amount of transmitted information or bits relatively increases compared to existing image data. The increase in the amount of transmitted information or bits results in an increase in transmission and storage costs.

[0003] Therefore, an efficient image compression technique is needed to efficiently transmit, store, and reproduce information about high-resolution and high-quality images. SUMMARY

[0004] TECHNICAL PROBLEM

[0005] An object of the present disclosure is to provide an image encoding / decoding method and apparatus having improved encoding / decoding efficiency.

[0006] In addition, an object of the present disclosure is to propose an efficient intra mode encoding method.

[0007] In addition, an object of the present disclosure is to propose a method for configuring candidates for a spatial geometric partition mode.

[0008] In addition, an object of the present disclosure is to propose a method for utilizing block vector information of a spatial geometric partition mode.

[0009] In addition, an object of the present disclosure is to provide a non-transitory computer-readable recording medium storing a bitstream generated by an image encoding method according to the present disclosure.

[0010] In addition, an object of the present disclosure is to provide a non-transitory computer-readable recording medium storing a bitstream, which is received and decoded by an image decoding apparatus according to the present disclosure, and used for reconstruction of an image.

[0011] In addition, an object of the present disclosure is to provide a method for transmitting a bitstream generated by an image encoding method according to the present disclosure.

[0012] The technical problems to be solved by the present disclosure are not limited to the above-mentioned technical problems, and those of ordinary skill in the art to which the present disclosure pertains will clearly understand other technical problems from the following description.

[0013] TECHNICAL SOLUTION

[0014] The image decoding method according to an aspect of the disclosure is an image decoding method performed by an image decoding apparatus, the image decoding method including: configuring candidates for a spatial geometric partition mode (SGPM) - wherein each of the candidates includes prediction modes of intra prediction modes for each partition partitioned by a partition mode of the SGPM -; and predicting a current block based on one of the candidates, wherein the prediction mode of a first candidate among the candidates is determined based on information indicating an intra block copy prediction mode of a reference location.

[0015] The image encoding method according to another aspect of the disclosure is an image encoding method performed by an image encoding apparatus, the image encoding method including: configuring candidates for a spatial geometric partition mode (SGPM) - wherein each of the candidates includes prediction modes of intra prediction modes for each partition partitioned by a partition mode of the SGPM -; and predicting a current block based on one of the candidates, wherein the prediction mode of a first candidate among the candidates is determined based on information indicating an intra block copy prediction mode of a reference location.

[0016] The computer-readable recording medium according to another aspect of the disclosure can store a bitstream generated by the image encoding method or apparatus of the disclosure.

[0017] The transmission method according to another aspect of the disclosure can transmit a bitstream generated by the image encoding method or apparatus of the disclosure.

[0018] The features briefly summarized above with respect to the disclosure are merely exemplary aspects of the subsequent detailed description of the disclosure, and do not limit the scope of the disclosure.

[0019] Technical Effects

[0020] According to the disclosure, it is possible to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0021] In addition, according to the disclosure, since the candidates for the spatial geometric partition mode can be efficiently configured, it is possible to improve the prediction performance.

[0022] In addition, according to the disclosure, it is possible to improve the encoding performance by diversely utilizing the block vectors of the spatial geometric partition mode.

[0023] In addition, according to the disclosure, it is possible to provide a non-transitory computer-readable recording medium storing a bitstream generated by the image encoding method according to the disclosure.

[0024] In addition, according to the present disclosure, a non-transitory computer-readable recording medium storing a bitstream received and decoded by an image decoding apparatus according to the present disclosure and used for reconstruction of an image can be provided.

[0025] In addition, according to the present disclosure, a method for transmitting a bitstream generated by an image encoding method can be provided.

[0026] Effects obtainable from the present disclosure are not limited to the above-mentioned effects and other effects not mentioned above will be clearly understood from the following description by those having ordinary skill in the art to which the present disclosure pertains. BRIEF DESCRIPTION OF DRAWINGS

[0027] FIG. 1 is a diagram schematically illustrating a video encoding system to which embodiments of the present disclosure can be applied.

[0028] FIG. 2 is a diagram schematically illustrating an image encoding apparatus to which embodiments of the present disclosure can be applied.

[0029] FIG. 3 is a diagram schematically illustrating an image decoding apparatus to which embodiments of the present disclosure can be applied.

[0030] FIG. 4 is a flowchart illustrating an image encoding method based on intra prediction.

[0031] FIG. 5 is a diagram schematically illustrating an intra prediction unit in an image encoding apparatus.

[0032] FIG. 6 is a flowchart illustrating an image encoding method based on intra prediction.

[0033] FIG. 7 is a diagram schematically illustrating an intra prediction unit in an image decoding apparatus.

[0034] FIG. 8 is a flowchart illustrating an intra prediction mode signaling method performed in an image encoding apparatus.

[0035] FIG. 9 is a flowchart illustrating an intra prediction mode determination method performed in an image decoding apparatus.

[0036] FIG. 10 is a diagram for explaining a method of configuring HoG (Histogram of Gradient) in DIMD (Decoder-side Intra Mode Derivation).

[0037] FIG. 11 is a diagram for explaining a method of configuring a prediction block when DIMD is applied.

[0038] FIG. 12 This is an example diagram used to illustrate the search region for intra-frame template matching.

[0039] FIG. 13 This is a diagram illustrating an example of using block vectors derived from intra-frame template matching for intra-frame block copying.

[0040] FIG. 14 This is a diagram illustrating an example of intra-frame block copying.

[0041] FIG. 15 It is a diagram used to illustrate spatial geometric partitioning patterns.

[0042] FIG. 16 This is a diagram illustrating an example of template-based multi-reference line intra-frame prediction modes.

[0043] FIG. 17 This is a diagram illustrating an example of DVB (Direct Block Vector) mode.

[0044] FIG. 18 This is a flowchart illustrating an image encoding method and an image decoding method according to one embodiment of the present disclosure.

[0045] FIG. 19 This is a diagram illustrating an example of a method for deriving errors in a spatial geometric partitioning pattern.

[0046] FIG. 20 This is a diagram used to illustrate the partitioning direction in a spatial geometric partitioning pattern and an example of the adjacent reference position according to that partitioning direction.

[0047] FIG. 21 This is a diagram illustrating examples of adjacent and non-adjacent reference positions in a spatial geometric partitioning pattern.

[0048] FIG. 22 This is an exemplary illustration of a content streaming system that can be applied to embodiments of this disclosure. Detailed Implementation

[0049] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, enabling those skilled in the art to readily practice the present disclosure. However, the present disclosure can be implemented in various different forms and is not limited to the embodiments described herein.

[0050] In describing embodiments of this disclosure, detailed descriptions of well-known structures or functions will be omitted if it is determined that such detailed descriptions may obscure the main points of this disclosure. Furthermore, in the accompanying drawings, portions unrelated to the description of this disclosure are omitted, and similar reference numerals are assigned to similar portions.

[0051] In this disclosure, when a component is described as “connected,” “linked,” or “associated” to another component, this can include not only a direct connection but also an indirect connection in which another component is present. Furthermore, when a component is described as “comprising” or “having” another component, this means that another component may be further included, unless otherwise stated.

[0052] In this disclosure, terms such as "first" and "second" are used only for the purpose of distinguishing one component from another, and unless otherwise stated, do not limit the order or importance of the components. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0053] In this disclosure, components are distinguished from each other to clearly describe their respective features and do not necessarily imply that the components are separate. That is, multiple components may be integrated into one hardware or software unit, or a component may be distributed across multiple hardware or software units. Therefore, such integrated or distributed implementations are included within the scope of this disclosure, even if not specifically mentioned.

[0054] In this disclosure, the components described in various embodiments are not necessarily essential components, and some components may be optional. Therefore, embodiments comprising a subset of the components described in one embodiment are also included within the scope of this disclosure. Furthermore, embodiments that include other components besides those described in the various embodiments are also included within the scope of this disclosure.

[0055] This disclosure relates to image encoding and decoding, and the terms used in this disclosure may have the meanings commonly used in the art to which this disclosure pertains, unless redefined in this disclosure.

[0056] In this disclosure, "picture" generally refers to a unit representing an image at a specific time, and a slice / tile is a coding unit that constitutes part of a picture, and a picture may consist of one or more slices / tiles. Additionally, a slice / tile may include one or more coding tree units (CTUs).

[0057] In this disclosure, "pixel" or "cell" can refer to the smallest unit that constitutes a picture (or image). Additionally, "sample" can be used as the term corresponding to a pixel. A sample can generally represent a pixel or pixel value, and can represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component.

[0058] In this disclosure, "unit" can refer to a basic unit of image processing. A unit may include a specific region of an image and at least one of the information associated with that region. The term "unit" may be used interchangeably with terms such as "sample array," "block," or "region," depending on the context. In general, an M×N block may include a set (or array) of samples (or sample arrays) or transform coefficients arranged in M ​​columns and N rows.

[0059] In this disclosure, "current block" can refer to one of "current coding block," "current coding unit," "block to be encoded," "block to be decoded," or "block to be processed." When performing prediction, "current block" can refer to "current prediction block" or "block to be predicted." When performing transform (inverse transform) / quantization (dequantization), "current block" can refer to "current transform block" or "block to be transformed." When performing filtering, "current block" can refer to "block to be filtered."

[0060] In this disclosure, unless explicitly stated as a chroma block, "current block" may refer to a block that includes both luma component blocks and chroma component blocks, or it may refer to "the luma block of the current block." The luma component block of the current block may be explicitly represented by a description indicating the luma component block, such as "luma block" or "current luma block." Similarly, the chroma component block of the current block may be explicitly represented by a description indicating the chroma component block, such as "chroma block" or "current chroma block."

[0061] In this disclosure, " / " and "," can be interpreted as "and / or". For example, "A / B" and "A, B" can be interpreted as "A and / or B". In addition, "A / B / C" and "A, B, C" can mean at least one of A, B and / or C.

[0062] In this disclosure, the term "or" may be interpreted as "and / or". For example, "A or B" may mean: 1) only A, 2) only B, or 3) both A and B. Alternatively, in this disclosure, the term "or" may mean "additionally or alternatively".

[0063] Overview of a video coding system

[0064] FIG. 1 This is an illustrative diagram illustrating a video coding system that can be applied according to embodiments of the present disclosure.

[0065] A video encoding system according to one embodiment may include an encoding device (10) and a decoding device (20). The encoding device (10) may deliver encoded video and / or image information or data to the decoding device (20) in the form of a file or stream via a digital storage medium or network.

[0066] An encoding device (10) according to one embodiment may include a video source generator (11), an encoder (12), and a transmitter (13). A decoding device (20) according to one embodiment may include a receiver (21), a decoder (22), and a renderer (23). The encoder (12) may be referred to as a video / image encoder, and the decoder (22) may be referred to as a video / image decoder. The transmitter (13) may be included in the encoder (12). The receiver (21) may be included in the decoder (22). The renderer (23) may include a display unit, or the display unit may be configured as a separate device or an external component.

[0067] The video source generator (11) can acquire video / images through video / image capture processing, compositing processing, or generation processing. The video source generator (11) may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive including previously captured video / images, etc. The video / image generation device may include, for example, a computer, tablet, smartphone, etc., and can generate video / images (electronically). For example, virtual video / images can be generated by a computer, etc., and in this case, the video / image capture processing can be replaced by processing that generates related data.

[0068] The encoder (12) can encode the input video / image. The encoder (12) can perform a series of processes such as prediction, transformation and quantization for compression and encoding efficiency. The encoder (12) can output the encoded data (encoded video / image information) in the form of a bitstream.

[0069] The transmitter (13) can acquire encoded video / image information or data output in the form of a bitstream, and can transmit it to the receiver (21) of the decoding device (20) or another external object in the form of a file or stream via a digital storage medium or network. The digital storage medium may include various storage media, such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter (13) may include elements for generating media files in a predetermined file format, and may include elements for transmission via a broadcast / communication network. The transmitter (13) may be configured as a transmission device separate from the encoding device (10), and in this case, the transmission device may include at least one processor and a transmitter, the at least one processor for acquiring encoded video / image information or data output in the form of a bitstream, and the transmitter for transmitting the encoded video / image information or data in the form of a file or stream. The receiver (21) can extract / receive the bitstream from the storage medium or network and transmit the bitstream to the decoder (22).

[0070] The decoder (22) can decode video / images by performing a series of processes such as dequantization, inverse transform and prediction corresponding to the operations of the encoder (12).

[0071] The renderer (23) can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0072] Overview of an image encoding device

[0073] FIG. 2 This is an illustrative diagram illustrating an image encoding device that can be applied according to embodiments of the present disclosure.

[0074] like FIG. 2 As shown, the image coding device (100) may include an image partitioning unit (110), a subtractor (115), a transform unit (120), a quantization unit (130), a dequantization unit (140), an inverse transform unit (150), an adder (155), a filtering unit (160), a memory (170), an inter-frame prediction unit (180), an intra-frame prediction unit (185), and an entropy coding unit (190). The inter-frame prediction unit (180) and the intra-frame prediction unit (185) may be collectively referred to as the "prediction unit". The transform unit (120), the quantization unit (130), the dequantization unit (140), and the inverse transform unit (150) may be included in the residual processing unit. The residual processing unit may further include a subtractor (115).

[0075] According to the implementation, all or at least some of the multiple components constituting the image encoding device (100) can be implemented as a single hardware component (e.g., an encoder or a processor). Additionally, the memory (170) may include a decoded image buffer (DPB) and may be implemented using a digital storage medium.

[0076] An image partitioning unit (110) can partition an input image (or picture or frame) input to an image coding device (100) into one or more processing units. As an example, a processing unit may be referred to as a coding unit (CU). A coding unit can be obtained by recursively partitioning a coding tree unit (CTU) or a largest coding unit (LCU) according to a QT / BT / TT (quadtree / binary tree / truncate tree) structure. For example, a coding unit can be partitioned into multiple deeper coding units based on a quadtree structure, a binary tree structure, and / or a truncate tree structure. For the partitioning of coding units, a quadtree structure can be applied first, and then a binary tree structure and / or a truncate tree structure can be applied. The coding processing according to this disclosure can be performed based on a final coding unit that is no longer partitioned. The largest coding unit can be used as the final coding unit, or a lower-depth coding unit obtained by partitioning the largest coding unit can be used as the final coding unit. Here, the coding processing may include processes such as prediction, transformation, and / or reconstruction, which will be described later. As another example, the processing unit used for encoding can be a prediction unit (PU) or a transform unit (TU). Prediction and transform units can be partitioned or derived separately from the final encoding unit. The prediction unit can be a unit for predicting samples, and the transform unit can be a unit for deriving transform coefficients and / or a unit for deriving the residual signal from the transform coefficients.

[0077] The prediction unit (inter-frame prediction unit (180) or intra-frame prediction unit (185)) can perform prediction on the block to be processed (the current block) and can generate a prediction block including prediction samples for the current block. The prediction unit can determine whether to apply intra-frame prediction or inter-frame prediction based on the current block or CU. The prediction unit can generate various information related to the prediction of the current block and transmit the information to the entropy coding unit (190). The prediction-related information can be encoded by the entropy coding unit (190) and output in the form of a bitstream.

[0078] The intra-prediction unit (185) can predict the current block by referencing samples in the current image. Depending on the intra-prediction mode and / or intra-prediction technique, the referenced samples may be located among the neighbors of the current block or may be spaced apart from the current block. The intra-prediction mode may include multiple non-directional modes and multiple directional modes. Non-directional modes may include, for example, DC mode and planar mode. Directional modes may include, for example, 33 or 65 directional prediction modes depending on the fineness of the prediction direction. However, this is merely an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-prediction unit (185) can determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.

[0079] The inter-frame prediction unit (180) can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference image. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted based on blocks, sub-blocks, or samples based on the correlation between the motion information of neighboring blocks and the current block. The motion information may include motion vectors and reference image indices. The motion information may further include information about the inter-frame prediction direction (L0 prediction, L1 prediction, bidirectional prediction, etc.). In the case of inter-frame prediction, neighboring blocks may include spatial neighboring blocks existing in the current image and temporal neighboring blocks existing in the reference image. The reference image including the reference block and the reference image including the temporal neighboring block may be the same as each other or may be different from each other. The temporal neighboring block may be referred to as a co-located reference block or co-located CU (colCU). The reference image including the temporal neighboring block may be referred to as a co-located image (colPic). For example, the inter-frame prediction unit (180) can configure a motion information candidate list based on the neighboring blocks and can generate information indicating which candidate to use to derive the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes, and for example, in skip mode and merge mode, the inter-frame prediction unit (180) can use the motion information of the neighboring block as the motion information of the current block. In skip mode, unlike merge mode, residual signals may not be sent. In motion vector prediction (MVP) mode, the motion vectors of the neighboring blocks can be used as motion vector predictors, and the motion vectors of the current block can be signaled by encoding the motion vector difference and an indicator for the motion vector predictor. The motion vector difference can refer to the difference between the motion vectors of the current block and the motion vector predictor.

[0080] The prediction unit can generate a prediction signal based on various prediction methods and / or prediction techniques described later. For example, the prediction unit can apply not only intra-frame prediction or inter-frame prediction for the prediction of the current block, but also both intra-frame prediction and inter-frame prediction simultaneously. A prediction method that applies both intra-frame prediction and inter-frame prediction simultaneously for the prediction of the current block can be called combined intra-frame and inter-frame prediction (CIIP). Additionally, the prediction unit can perform intra-block copy (IBC) for the prediction of the current block. Intra-block copy can be used, for example, for encoding content images / videos such as in game content coding (SCC). IBC is a method of predicting the current block by using a reconstructed reference block located at a predetermined distance from the current block in the current image. When IBC is applied, the position of the reference block in the current image can be encoded as a vector (block vector) corresponding to the predetermined distance. IBC essentially performs prediction within the current image, but can be performed similarly to inter-frame prediction in terms of deriving the reference block from the current image. That is, IBC can use at least one of the inter-frame prediction techniques described in this disclosure.

[0081] The prediction signal generated by the prediction unit can be used to generate a reconstructed signal or a residual signal. The subtractor (115) can generate a residual signal (residual signal, residual block, residual sample array) by subtracting the prediction signal (prediction block, prediction sample array) output from the prediction unit from the input image signal (original block, original sample array). The generated residual signal can be sent to the transform unit (120).

[0082] The transform unit (120) can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loeve Transform (KLT), Graph-Based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is represented as a graph. CNT refers to a transform based on generating a prediction signal using all previously reconstructed pixels. The transform process can be applied to pixel blocks of the same square shape or to blocks of non-square, variable size.

[0083] The quantization unit (130) can quantize the transform coefficients and send the quantized transform coefficients to the entropy coding unit (190). The entropy coding unit (190) can encode the quantized signal (information about the quantized transform coefficients) and output the encoded signal as a bit stream. The information about the quantized transform coefficients can be referred to as residual information. The quantization unit (130) can rearrange the block-shaped quantized transform coefficients into a one-dimensional vector form based on the coefficient scan order, and can generate information about the quantized transform coefficients based on the one-dimensional vector form of the quantized transform coefficients.

[0084] The entropy coding unit (190) can perform various coding methods, such as exponential Golomb coding, context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy coding unit (190) can encode the information required for video / image reconstruction (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / image information) can be transmitted or stored as a bitstream in units of Network Abstraction Layer (NAL) units. The video / image information may further include information about various parameter sets, such as Adaptive Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). Additionally, the video / image information may further include general constraint information. The signaling information, transmission information, and / or syntax elements mentioned in this disclosure can be encoded and included in the bitstream through the above coding process.

[0085] The bitstream can be transmitted over a network or stored in a digital storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter (not shown) for transmitting the signal output from the entropy encoding unit (190) and / or the storage unit (not shown) for storing the signal may be configured as internal or external components of the image encoding device (100), or the transmitter may be configured as a component of the entropy encoding unit (190).

[0086] The quantized transform coefficients output from the quantization unit (130) can be used to generate the residual signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients through the dequantization unit (140) and the inverse transform unit (150).

[0087] The adder (155) generates a reconstructed signal (reconstructed image, reconstructed block, or reconstructed sample array) by adding the reconstructed residual signal to a prediction signal output from the inter-frame prediction unit (180) or the intra-frame prediction unit (185). When no residual exists for the block to be processed, such as when a skip mode is applied, the prediction block can be used as a reconstructed block. The adder (155) can be referred to as a reconstruction unit or a reconstructed block generation unit. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current image, or it can be used for inter-frame prediction of the next image after filtering, as described later.

[0088] The filtering unit (160) can improve subjective and / or objective image quality by applying filtering to the reconstructed signal. For example, the filtering unit (160) can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and can store the modified reconstructed image in memory (170), specifically in the DPB of memory (170). Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc. The filtering unit (160) can generate various filtering-related information, as will be described later in the description of each filtering method, and send this information to the entropy coding unit (190). The filtering-related information can be encoded by the entropy coding unit (190) and output as a bitstream.

[0089] The modified reconstructed image sent to the memory (170) can be used as a reference image by the inter-frame prediction unit (180). By doing so, when inter-frame prediction is applied, the image coding device (100) can avoid prediction mismatch between the image coding device (100) and the image decoding device, and can also improve coding efficiency.

[0090] The DPB in memory (170) can store a modified reconstructed image that will be used as a reference image by the inter-frame prediction unit (180). Memory (170) can store motion information of blocks in the current image with deduced (or encoded) motion information and / or motion information of blocks in the reconstructed image. The stored motion information can be transmitted to the inter-frame prediction unit (180) to be used as motion information of spatial neighbor blocks or temporal neighbor blocks. Memory (170) can store reconstructed samples of reconstructed blocks in the current image and transmit the reconstructed samples to the intra-frame prediction unit (185).

[0091] Overview of an image decoding device

[0092] FIG. 3 This is an illustrative diagram illustrating an image decoding device that can be applied according to embodiments of the present disclosure.

[0093] likeFIG. 3 As illustrated, the image decoding device (200) may include an entropy decoding unit (210), a dequantization unit (220), an inverse transform unit (230), an adder (235), a filtering unit (240), a memory (250), an inter-frame prediction unit (260), and an intra-frame prediction unit (265). The inter-frame prediction unit (260) and the intra-frame prediction unit (265) may be collectively referred to as the "prediction unit". The dequantization unit (220) and the inverse transform unit (230) may be included in the residual processing unit.

[0094] According to the implementation, all or at least some of the multiple components constituting the image decoding device (200) can be implemented as a single hardware component (e.g., a decoder or processor). Additionally, the memory (170) may include a decoded image buffer (DPB) and may be implemented using a digital storage medium.

[0095] The image decoding device (200) that receives a bitstream including video / image information can perform operations with... FIG. 2 The image is reconstructed by processing corresponding to the processing performed in the image encoding device (100). For example, the image decoding device (200) can perform decoding by using a processing unit applied in the image encoding device. Therefore, the processing unit for decoding can be, for example, an encoding unit. The encoding unit can be a coding tree unit, or it can be obtained by partitioning the largest coding unit. In addition, the reconstructed image signal decoded and output by the image decoding device (200) can be reproduced by a playback device (not shown).

[0096] Image decoding device (200) can receive data in bitstream form from... FIG. 2The received signal is output by the image encoding device. The received signal can be decoded by the entropy decoding unit (210). For example, the entropy decoding unit (210) can parse the bitstream and derive the information required for image reconstruction (or picture reconstruction) (e.g., video / image information). The video / image information may further include information about various parameter sets, such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). In addition, the video / image information may further include general constraint information. The image decoding device may additionally use the information about the parameter sets and / or general constraint information to decode the image. The signaling information, received information, and / or syntax elements mentioned in this disclosure can be obtained from the bitstream by decoding the process. For example, the entropy decoding unit (210) can decode the information in the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, or CABAC, and can output the values ​​of the syntax elements required for image reconstruction and the quantized values ​​of the transform coefficients for the residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element from the bitstream, determine the context model by using the information of the syntax element to be decoded and the decoding information of neighboring blocks and the block to be decoded or the information of symbols / bins decoded in previous stages, predict the occurrence probability of bins based on the determined context model, and perform arithmetic decoding of bins to generate symbols corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method can update the context model for the next symbol / bin by using the information of the decoded symbols / bins after determining the context model. In the information decoded by the entropy decoding unit (210), prediction-related information can be provided to the prediction unit (including the inter-frame prediction unit (260) and the intra-frame prediction unit (265)), and the residual values ​​(i.e., quantized transform coefficients and related parameter information) entropy decoded by the entropy decoding unit (210) can be input to the dequantization unit (220). In addition, in the information decoded by the entropy decoding unit (210), filtering-related information can be provided to the filtering unit (240). Meanwhile, a receiver (not shown) for receiving signals output from the image encoding device may be additionally configured as an internal or external component of the image decoding device (200), or the receiver may be configured as a component of the entropy decoding unit (210).

[0097] Meanwhile, the image decoding device according to this disclosure can be referred to as a video / image / picture decoding device. The image decoding device may include an information decoder (video / image / picture information decoder) and / or a sample decoder (video / image / picture sample decoder). The information decoder may include an entropy decoding unit (210), and the sample decoder may include at least one of a dequantization unit (220), an inverse transform unit (230), an adder (235), a filtering unit (240), a memory (250), an inter-frame prediction unit (260), and an intra-frame prediction unit (265).

[0098] The dequantization unit (220) can dequantize the quantized transform coefficients and output the transform coefficients. The dequantization unit (220) can rearrange the quantized transform coefficients into two-dimensional blocks. In this case, the rearrangement can be performed based on the coefficient scanning order executed in the image encoding device. The dequantization unit (220) can perform dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain the transform coefficients.

[0099] The inverse transform unit (230) can perform inverse transform on the transform coefficients and obtain the residual signal (residual block, residual sample array).

[0100] The prediction unit can perform prediction on the current block and generate a prediction block that includes prediction samples for the current block. The prediction unit can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on prediction-related information output from the entropy decoding unit (210), and can determine a specific intra-frame / inter-frame prediction mode (prediction technique).

[0101] The prediction unit can generate a prediction signal based on various prediction methods (techniques) that will be described later, which are the same as those described in the description of the prediction unit of the image coding device (100).

[0102] The intra-prediction unit (265) can predict the current block by referring to samples in the current image. The description of the intra-prediction unit (185) can be applied equivalently to the intra-prediction unit (265).

[0103] The inter-frame prediction unit (260) can deduce the prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference image. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted based on blocks, sub-blocks, or samples based on the correlation between the motion information of neighboring blocks and the current block. The motion information may include motion vectors and reference image indices. The motion information may further include information about the inter-frame prediction direction (L0 prediction, L1 prediction, bidirectional prediction, etc.). In the case of inter-frame prediction, neighboring blocks may include spatial neighboring blocks existing in the current image and temporal neighboring blocks existing in the reference image. For example, the inter-frame prediction unit (260) can construct a motion information candidate list based on neighboring blocks and can deduce the motion vector and / or reference image index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes (techniques), and the information related to the prediction may include information indicating the mode (technique) of the inter-frame prediction for the current block.

[0104] Adder (235) generates a reconstruction signal (reconstructed image, reconstruction block, reconstruction sample array) by adding the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the prediction unit (including inter-frame prediction unit (260) and / or intra-frame prediction unit (265)). When there is no residual for the block to be processed, such as when a skip mode is applied, the prediction block can be used as the reconstruction block. The description of adder (155) can be applied equivalently to adder (235). Adder (235) can be referred to as reconstruction unit or reconstruction block generation unit. The generated reconstruction signal can be used for intra-frame prediction of the next block to be processed in the current image, or it can be used for inter-frame prediction of the next image after filtering, as described later.

[0105] The filtering unit (240) can improve subjective and / or objective image quality by applying filtering to the reconstructed signal. For example, the filtering unit (240) can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and can store the modified reconstructed image in memory (250), specifically in the DPB of memory (250). Various filtering methods may include, for example, deblocking filtering, sample adaptive shifting, adaptive loop filtering, bilateral filtering, etc.

[0106] The (modified) reconstructed image stored in the DPB in memory (250) can be used as a reference image by the inter-frame prediction unit (260). Memory (250) can store motion information of blocks in the current image whose motion information has been derived (or decoded) and / or motion information of blocks in the reconstructed image. The stored motion information can be sent to the inter-frame prediction unit (260) to be used as motion information of spatial neighbor blocks or temporal neighbor blocks. Memory (250) can store reconstructed samples of reconstructed blocks in the current image and send the reconstructed samples to the intra-frame prediction unit (265).

[0107] The embodiments described in this specification in the filtering unit (160), inter-frame prediction unit (180) and intra-frame prediction unit (185) of the image encoding device (100) can be applied to the filtering unit (240), inter-frame prediction unit (260) and intra-frame prediction unit (265) of the image decoding device (200) in the same or corresponding manner.

[0108] General intra prediction

[0109] Intra-frame prediction can be represented as generating prediction samples for the current block based on reference samples within the image to which the current block belongs (hereinafter referred to as the current image). When intra-frame prediction is applied to the current block, neighbor reference samples for the intra-frame prediction of the current block can be derived. The neighbor reference samples of the current block may include: a total of 2×nH samples including samples adjacent to the left boundary of the current block of size nW×nH and samples adjacent to the lower left position; a total of 2×nW samples including samples adjacent to the upper boundary of the current block and samples adjacent to the upper right position; and one sample adjacent to the upper left position of the current block. Alternatively, the neighbor reference samples of the current block may include multiple columns of upper neighbor samples and multiple rows of left neighbor samples. In addition, the neighbor reference samples of the current block may include: a total of nH samples adjacent to the right boundary of the current block of size nW×nH; a total of nW samples adjacent to the lower boundary of the current block; and one sample adjacent to the lower right position of the current block.

[0110] However, some of the neighbor reference samples in the current block may not have been decoded or may be unavailable. In this case, the image decoding device (200) can configure the neighbor reference samples for prediction by replacing the unavailable samples with available samples. Alternatively, the neighbor reference samples for prediction can be configured by interpolating the available samples.

[0111] When deriving neighbor reference samples, (i) the predicted sample can be derived based on the average or interpolation of the neighbor reference samples of the current block, and (ii) the predicted sample can be derived based on reference samples among the neighbor reference samples of the current block that exist in a specific (prediction) direction relative to the predicted sample. Case (i) can be referred to as non-directional mode or non-angular mode, and case (ii) can be referred to as directional mode or angular mode. Alternatively, a predicted sample can be generated by interpolating between a second neighbor sample and a first neighbor sample in a direction opposite to the prediction direction of the intra-prediction mode of the current block relative to the predicted sample of the current block. This case can be referred to as linear interpolation intra-prediction (LIP). Additionally, a chroma predicted sample can be generated based on a luminance sample using a linear model. This case can be referred to as LM mode. Furthermore, a provisional predicted sample of the current block can be derived based on filtered neighbor reference samples, and a predicted sample of the current block can be derived by performing a weighted sum on at least one reference sample derived according to the intra-prediction mode (i.e., an unfiltered neighbor reference sample) among the existing neighbor reference samples and the provisional predicted sample. The above situation can be referred to as Position-Dependent Intra-Prediction (PDPC). Alternatively, intra-prediction coding can be performed by selecting the reference sample row with the highest prediction accuracy from among multiple neighboring reference sample rows of the current block, deriving the prediction sample using reference samples in the prediction direction located on the selected row, and signaling the used reference sample row to the image decoding device (200). This situation can be referred to as Multi-Reference-Row (MRL) intra-prediction or MRL-based intra-prediction. Furthermore, the current block can be divided into vertical or horizontal sub-partitions, intra-prediction can be performed based on the same intra-prediction mode, and neighboring reference samples can be derived and used based on the sub-partitions. That is, in this case, the intra-prediction mode used for the current block is applied to the sub-partitions in the same way, but neighboring reference samples are derived and used based on the sub-partitions, thereby improving intra-prediction performance in some cases. This prediction method can be referred to as Intra-Segmentation (ISP) or ISP-based intra-prediction. To distinguish it from the intra-prediction modes in Section 1.2, the above intra-prediction methods can be referred to as intra-prediction types. Intra-prediction types can be referred to as intra-prediction techniques or additional intra-prediction modes, etc. For example, intra-prediction types (or additional intra-prediction modes, etc.) may include at least one of LIP, PDPC, MRL, and ISP mentioned above. General intra-prediction methods that exclude specific intra-prediction types such as LIP, PDPC, MRL, and ISP can be referred to as regular intra-prediction types. When a specific intra-prediction type is not applied, regular intra-prediction types can typically be applied, and prediction can be performed based on the aforementioned intra-prediction modes. Simultaneously, post-processing filtering can be performed on the derived prediction samples as needed.

[0112] Specifically, the intra-frame prediction process may include an intra-frame prediction mode / type determination step, a neighbor reference sample derivation step, and a prediction sample derivation step based on the intra-frame prediction mode / type. Additionally, a post-filtering step for the derivation of prediction samples may be performed as needed.

[0113] In addition to the intra-prediction types mentioned above, affine linear weighted intra-prediction (ALWIP) can be used. ALWIP can also be referred to as LWIP (linear weighted intra-prediction) or MIP (matrix weighted intra-prediction or matrix-based intra-prediction). When applying MIP to the current block, the prediction samples for the current block can be derived by (i) using neighboring reference samples after an averaging process, (ii) performing a matrix-vector multiplication process, and (iii) additionally performing horizontal / vertical interpolation as needed. The intra-prediction mode used for MIP can be configured to differ from the intra-prediction modes used in LIP, PDPC, MRL, ISP intra-prediction, or regular intra-prediction. The intra-prediction mode used for MIP can be referred to as MIP intra-prediction mode, MIP prediction mode, or MIP mode. For example, the matrix and offset used in matrix-vector multiplication can be set differently depending on the intra-prediction mode used for MIP. Here, the matrix can be referred to as the (MIP) weight matrix, and the offset can be referred to as the (MIP) offset vector or (MIP) bias vector. The MIP method will be described in detail later.

[0114] The block reconstruction process based on intra-frame prediction and the intra-frame prediction unit (185) in the image coding device (100) can, for example, schematically include FIG. 4 and FIG. 5 .

[0115] S400 can be performed by the intra-prediction unit (185) of the image coding device (100), and S410 can be performed by the residual processing unit of the image coding device (100). Specifically, S410 can be performed by the subtractor (115) of the image coding device (100). In S420, the prediction information can be derived by the intra-prediction unit (185) and encoded by the entropy coding unit (190). In S420, the residual information can be derived by the residual processing unit and encoded by the entropy coding unit (190). The residual information is information about the residual samples. The residual information may include information about the quantization transform coefficients of the residual samples. As described above, the residual samples can be derived into transform coefficients by the transform unit (120) of the image coding device (100), and the transform coefficients can be derived into quantization transform coefficients by the quantization unit (130). The information about the quantization transform coefficients can be encoded by the entropy coding unit (190) through the residual coding process.

[0116] The image coding device (100) performs intra-prediction (S400) on the current block. The image coding device (100) can derive the intra-prediction mode / type for the current block, derive the neighbor reference samples for the current block, and generate prediction samples in the current block based on the intra-prediction mode / type and the neighbor reference samples. Here, the intra-prediction mode / type determination, the neighbor reference sample derivation, and the prediction sample generation process can be performed simultaneously, or one process can be performed before the other. For example, the intra-prediction unit (185) of the image coding device (100) may include an intra-prediction mode / type determination unit (186), a reference sample derivation unit (187), and a prediction sample derivation unit (188). The intra-prediction mode / type determination unit (186) can determine the intra-prediction mode / type for the current block, the reference sample derivation unit (187) can derive the neighbor reference samples for the current block, and the prediction sample derivation unit (188) can derive the prediction samples for the current block. Meanwhile, although not illustrated, the intra-prediction unit (185) may further include a prediction sample filtering unit (not shown) when performing the prediction sample filtering process described later. The image coding device (100) may determine the mode / type to be applied to the current block from among a plurality of intra-prediction modes / types. The image coding device (100) may compare the RD costs of the intra-prediction modes / types and determine the optimal intra-prediction mode / type for the current block.

[0117] Simultaneously, the image encoding device (100) can perform a predictive sample filtering process. Predictive sample filtering can be referred to as post-filtering. Some or all of the predictive samples can be filtered through the predictive sample filtering process. In some cases, the predictive sample filtering process can be omitted.

[0118] The image coding device (100) generates a residual sample for the current block based on the (filtered) predicted sample (S410). The image coding device (100) can derive the residual sample by comparing the predicted sample with the original sample of the current block based on the phase.

[0119] The image encoding device (100) can encode image information including information about intra-frame prediction (prediction information) and residual information about residual samples (S420). The prediction information may include intra-frame prediction mode information and intra-frame prediction type information. The image encoding device (100) can output the encoded image information in the form of a bitstream. The output bitstream can be transmitted to the image decoding device (200) via a storage medium or a network.

[0120] The residual information may include the residual coding syntax, which will be described later. The image coding device (100) can derive the quantization transform coefficients by transforming / quantizing the residual samples. The residual information may include information about the quantization transform coefficients.

[0121] Simultaneously, as described above, the image encoding device (100) can generate a reconstructed image (including reconstructed samples and reconstructed blocks). For this purpose, the image encoding device (100) can derive (modified) residual samples by performing dequantization / inverse transform on the quantization transform coefficients. The reason for performing dequantization / inverse transform again after the transform / quantization of the residual samples is to derive residual samples that are the same as those derived by the image decoding device (200), as described above. The image encoding device (100) can generate a reconstructed block including reconstructed samples for the current block based on the predicted samples and the (modified) residual samples. The reconstructed image for the current image can be generated based on the reconstructed blocks. As described above, in-loop filtering processes, etc., can be further applied to the reconstructed image.

[0122] The video / image decoding process based on intra-frame prediction and the intra-frame prediction unit in the image decoding device (200) may, for example, schematically include the following.

[0123] The image decoding device (200) can perform operations corresponding to those performed by the image encoding device (100).

[0124] S600 to S620 can be performed by the intra-prediction unit (265) of the image decoding device (200), and the prediction information of S600 and the residual information of S630 can be obtained from the bitstream by the entropy decoding unit (210) of the image decoding device (200). The residual processing unit of the image decoding device (200) can derive residual samples for the current block based on the residual information. Specifically, the dequantization unit (220) of the residual processing unit can derive the transform coefficients by performing dequantization based on the quantization transform coefficients derived from the residual information, and the inverse transform unit (230) of the residual processing unit can derive the residual samples for the current block by performing an inverse transform on the transform coefficients. S640 can be performed by the adder (235) or the reconstruction unit of the image decoding device (200).

[0125] Specifically, the image decoding device (200) can deduce the intra-prediction mode / type for the current block based on the received prediction information (intra-prediction mode / type information) (S600). The image decoding device (200) can deduce the neighbor reference samples of the current block (S610). The image decoding device (200) can generate prediction samples in the current block based on the intra-prediction mode / type and the neighbor reference samples (S620). In this case, the image decoding device (200) can perform a prediction sample filtering process. Prediction sample filtering can be referred to as post-filtering. Some or all of the prediction samples can be filtered by the prediction sample filtering process. In some cases, the prediction sample filtering process can be omitted.

[0126] The image decoding device (200) can generate residual samples for the current block based on the received residual information. The image decoding device (200) can generate reconstruction samples for the current block based on the predicted samples and the residual samples, and can derive a reconstruction block including the reconstruction samples (S630). A reconstructed image for the current image can be generated based on the reconstruction block. As described above, in-loop filtering processes, etc., can be further applied to the reconstructed image.

[0127] Here, the intra-prediction unit (265) of the image decoding device (200) may include an intra-prediction mode / type determination unit (266), a reference sample derivation unit (267), and a prediction sample derivation unit (268). The intra-prediction mode / type determination unit (266) may determine the intra-prediction mode / type for the current block based on the intra-prediction mode / type information generated and signaled by the intra-prediction mode / type determination unit (186) of the image encoding device (100). The reference sample derivation unit (267) may derive the neighboring reference samples of the current block, and the prediction sample derivation unit (268) may derive the prediction samples of the current block. Meanwhile, although not illustrated, when the above-described prediction sample filtering process is performed, the intra-prediction unit (265) may further include a prediction sample filtering unit (not shown).

[0128] Intra-prediction mode information may include, for example, flag information (e.g., intra_luma_mpm_flag) indicating whether the most probable mode (MPM) is applied to the current block or a remaining mode is applied. When the MPM is applied to the current block, the prediction mode information may further include index information (e.g., intra_luma_mpm_idx) indicating one of the intra-prediction mode candidates (MPM candidates). The intra-prediction mode candidates (MPM candidates) may be configured as an MPM candidate list or an MPM list. Additionally, when the MPM is not applied to the current block, the intra-prediction mode information may further include remaining mode information (e.g., intra_luma_mpm_remainder) indicating one of the remaining intra-prediction modes excluding the intra-prediction mode candidates (MPM candidates). The image decoding device (200) may determine the intra-prediction mode of the current block based on the intra-prediction mode information. A separate MPM list may be configured for the aforementioned MIP.

[0129] Furthermore, intra-prediction type information can be implemented in various forms. For example, intra-prediction type information may include intra-prediction type index information indicating one of the intra-prediction types. As another example, intra-prediction type information may include at least one of the following: reference sample line information (e.g., intra_luma_ref_idx) indicating whether the MRL is applied to the current block and which reference sample line is used when applied; ISP flag information (e.g., intra_subpartitions_mode_flag) indicating whether the ISP is applied to the current block; ISP type information (e.g., intra_subpartitions_split_flag) indicating the split type of the subpartition when the ISP is applied; flag information indicating whether the PDPC is applied; or flag information indicating whether the LIP is applied. Additionally, intra-prediction type information may include a MIP flag indicating whether the MIP is applied to the current block.

[0130] Intra-prediction mode information and / or intra-prediction type information can be encoded / decoded using the encoding methods described in this document. For example, intra-prediction mode information and / or intra-prediction type information can be encoded / decoded using entropy coding based on truncated (Rice) binary codes (e.g., CABAC, CAVLC).

[0131] Intra prediction mode / type determination

[0132] When intra-prediction is applied, the intra-prediction mode applied to the current block can be determined by using the intra-prediction modes of neighboring blocks. For example, the image decoding device (200) can select one of the MPM candidates from the list of MPM (most probable modes) derived from the intra-prediction modes of the current block's neighboring blocks (e.g., the left and / or upper neighboring blocks) and additional candidate modes based on the received MPM index, or it can select one of the remaining intra-prediction modes not included in the MPM candidates (and planar modes) based on the remaining intra-prediction mode information. The MPM list can be constructed to include planar modes as candidates or not. For example, when the MPM list includes planar modes as candidates, the MPM list can have six candidates, and when the MPM list does not include planar modes as candidates, the MPM list can have three candidates. When the MPM list is constructed to not include planar modes as candidates, a non-planar flag (e.g., intra_luma_not_planar_flag) can be signaled indicating whether the intra-prediction mode of the current block is not a planar mode. For example, the MPM flag can be signaled first, and when the MPM flag is 1, the MPM index and non-planar flag can be signaled. Additionally, when the non-planar flag is 1, the MPM index can be signaled. Here, the MPM list is configured not to include planar patterns as candidates not because planar patterns are not MPMs, but because planar patterns are always considered MPMs; therefore, the non-planar flag is signaled first to check if the pattern is a planar pattern.

[0133] For example, MPM flags (e.g., `intra_luma_mpm_flag`) can be used to indicate whether the intra prediction mode applied to the current block is included in the MPM candidates (and planar modes) or in the remaining modes. A value of 1 for the MPM flag indicates that the intra prediction mode applied to the current block is included in the MPM candidates (and planar modes), and a value of 0 for the MPM flag indicates that the intra prediction mode applied to the current block is not included in the MPM candidates (and planar modes). A value of 0 for the non-planar flag (e.g., `intra_luma_not_planar_flag`) indicates that the intra prediction mode of the current block is planar, and a value of 1 for the non-planar flag indicates that the intra prediction mode of the current block is not planar. The MPM index can be signaled in the form of `mpm_idx` or `intra_luma_mpm_idx` syntax elements, and the remaining intra prediction mode information can be signaled in the form of `rem_intra_luma_pred_mode` or `intra_luma_mpm_remainder` syntax elements. For example, the remaining intra-prediction mode information can be indexed among all intra-prediction modes in order of prediction mode number to indicate one of the remaining intra-prediction modes not included in the MPM candidates (and planar modes). The intra-prediction mode can be an intra-prediction mode for the luma component (samples). In the following text, the intra-prediction mode information may include at least one of the following: an MPM flag (e.g., intra_luma_mpm_flag), a non-planar flag (e.g., intra_luma_not_planar_flag), an MPM index (e.g., mpm_idx or intra_luma_mpm_idx), and remaining intra-prediction mode information (rem_intra_luma_pred_mode or intra_luma_mpm_remainder). In this document, the MPM list may be referred to by various terms, such as the MPM candidate list, candModeList, etc. When MIP is applied to the current block, the individual MPM flags used for MIP (e.g., intra_mip_mpm_flag), MPM index (e.g., intra_mip_mpm_idx), and remaining intra-prediction mode information (e.g., intra_mip_mpm_remainder) can be signaled, but the non-plane flags are not signaled.

[0134] For example, the intra-prediction mode signaling process in the image coding device (100) and the intra-prediction mode determination process in the image decoding device (200) can be performed as follows.

[0135] Reference FIG. 8The image coding device (100) constructs an MPM list for the current block (S800). The MPM list may include candidate intra-prediction modes (MPM candidates) with a high probability of being applied to the current block. The MPM list may include intra-prediction modes of neighboring blocks, and may further include specific intra-prediction modes according to a predefined method. The detailed method for constructing the MPM list will be described later.

[0136] The image coding device (100) determines the intra-prediction mode for the current block (S810). The image coding device (100) can perform prediction based on various intra-prediction modes and can determine the optimal intra-prediction mode based on the RDO (rate-distortion optimization) based on that intra-prediction mode. In this case, the image coding device (100) can determine the optimal intra-prediction mode by using only the MPM candidates and planar modes constructed in the MPM list, or by further using the remaining intra-prediction modes in addition to the MPM candidates and planar modes constructed in the MPM list. Specifically, for example, when the intra-prediction type of the current block is a specific type other than the regular intra-prediction type (e.g., LIP, MRL, or ISP), the image coding device (100) can determine the optimal intra-prediction mode by considering only the MPM candidates and planar modes as intra-prediction mode candidates for the current block. That is, in this case, the intra-prediction mode for the current block can be determined only among the MPM candidates and planar modes, and in this case, the MPM flag may not be encoded / signaled. The image decoding device (200) can estimate the value of the MPM flag to be 1 without receiving the signaling of the MPM flag separately.

[0137] Simultaneously, typically, when the intra-prediction mode of the current block is not a planar mode and is one of the MPM candidates in the MPM list, the image coding device (100) generates an MPM index (mpm_idx) indicating one of the MPM candidates. When the intra-prediction mode of the current block is not included in the MPM list, residual intra-prediction mode information is generated indicating the mode that is the same as the intra-prediction mode of the current block among the residual intra-prediction modes that are not included in the MPM list (and planar modes).

[0138] The image coding device (100) can encode intra-prediction mode information and output it as a bitstream. The intra-prediction mode information may include the aforementioned MPM flag, non-planar flag, MPM index, and / or remaining intra-prediction mode information. Typically, the MPM index and remaining intra-prediction mode information are interchangeable in indicating the intra-prediction mode used for a block and are therefore not signaled simultaneously. That is, a value of 1 for the MPM flag is signaled along with either the non-planar flag or the MPM index, or a value of 0 for the MPM flag is signaled along with the remaining intra-prediction mode information. However, as described above, when a specific intra-prediction type is applied to the current block, the MPM flag may not be signaled, and only the non-planar flag and / or the MPM index may be signaled. That is, in this case, the intra-prediction mode information may only include the non-planar flag and / or the MPM index.

[0139] The image decoding device (200) can determine the intra prediction mode corresponding to the intra prediction mode information determined by the image encoding device (100) and notified by the concurrent signal.

[0140] Reference FIG. 9 The image decoding device (200) obtains intra-prediction mode information from the bitstream (S900). The intra-prediction mode information may include at least one of the following as described above: MPM flag, non-planar flag, MPM index, and remaining intra-prediction mode information.

[0141] The image decoding device (200) constructs an MPM list (S910). The MPM list is constructed in the same way as the MPM list constructed in the image encoding device (100). That is, the MPM list may include intra-prediction modes of neighboring blocks, and may further include specific intra-prediction modes according to a predefined method. The detailed method for constructing the MPM list will be described later.

[0142] Although S910 is executable after S900, this is just an example, and S910 can be executed before or simultaneously with S900.

[0143] The image decoding device (200) determines the intra-prediction mode of the current block based on the MPM list and intra-prediction mode information (S920). For example, when the value of the MPM flag is 1, the image decoding device (200) can deduce the planar mode as the intra-prediction mode of the current block (based on the non-planar flag), or it can deduce the candidate indicated by the MPM index among the MPM candidates in the MPM list as the intra-prediction mode of the current block. As another example, when the value of the MPM flag is 0, the image decoding device (200) can deduce the intra-prediction mode of the current block from the remaining intra-prediction modes indicated by the remaining intra-prediction mode information that are not included in the MPM list and planar modes as the intra-prediction mode of the current block. At the same time, as yet another example, when the intra-prediction type of the current block is a specific type (e.g., LIP, MRL, or ISP), the image decoding device (200) can deduce the planar mode or the candidate indicated by the MPM index in the MPM list as the intra-prediction mode of the current block without checking the MPM flag.

[0144] Typically, when performing block partitioning on an image, the current block to be encoded and its neighboring blocks have similar image characteristics. Therefore, there is a high probability that the current block and its neighboring blocks have the same or similar intra-prediction modes. Thus, the encoder can use the intra-prediction modes of the neighboring blocks to encode the intra-prediction mode of the current block.

[0145] For example, the image encoding device (100) / image decoding device (200) can construct an MPM (Most Probable Mode) list for the current block. The MPM list can also be referred to as an MPM candidate list. Here, MPM can refer to a mode that improves coding efficiency by considering the similarity between the current block and neighboring blocks when encoding intra-frame prediction modes. As mentioned above, the MPM list can be constructed to include planar modes or to exclude planar modes. For example, when the MPM list includes planar modes, the number of candidates in the MPM list can be six. When the MPM list does not include planar modes, the number of candidates in the MPM list can be five.

[0146] The image encoding device (100) / image decoding device (200) can construct an MPM list including six MPMs.

[0147] To construct an MPM list, consider the following patterns.

[0148] - Default intra-frame mode

[0149] - Neighbor In-Frame Mode

[0150] - Intra-frame mode (IPM mode) derived from neighboring inter-frame modes

[0151] - DIMD (Decoder-Side Intra-Frame Mode Derivation) mode

[0152] - Derived intra-frame mode

[0153] For neighbor-based intra-frame mode, neighboring blocks can be considered, namely, the left neighboring block, the top neighboring block, the bottom left neighboring block, the top right neighboring block, the top left neighboring block, etc. When neighbor-based intra-frame mode is used as MPM, the input order can vary based on the size information of the current block. For example, when the block height is greater than or equal to the block width, the intra-frame mode of the top neighboring block can be considered first, followed by the intra-frame mode of the left neighboring block.

[0154] Even when neighboring blocks are encoded in inter-frame mode rather than intra-frame mode, intra-frame mode information can still be obtained through the IPM buffer. When the position indicated by the motion vector of a neighboring inter-frame block is in intra-frame mode, the corresponding intra-frame mode can be stored in the IPM buffer. The intra-frame mode stored in the IPM buffer can be used as the MPM mode of the neighboring block.

[0155] MPM candidate configuration can be performed from DIMD, as described later. When the current block is not in DIMD mode, the intra-frame mode derived from DIMM can be used as an MPM candidate.

[0156] As mentioned above, when the MPM list is constructed to exclude flat patterns, flat patterns are excluded from the list, and the number of candidates in the MPM list can be set to five.

[0157] Prediction sample derivation based on intra prediction mode / type

[0158] The prediction unit of the image encoding device (100) / image decoding device (200) can derive reference samples based on the intra-prediction mode of the current block from the neighboring reference samples of the current block, and can generate prediction samples of the current block based on the reference samples.

[0159] For example, a predicted sample can be derived based on the average or interpolation of the neighboring reference samples of the current block, or it can be derived based on a reference sample among the neighboring reference samples of the current block that exists in a specific prediction direction relative to the predicted sample. Case (i) can be referred to as a non-directional mode or a non-angular mode, and case (ii) can be referred to as a directional mode or an angular mode. Alternatively, a predicted sample can be generated by interpolating between a second neighbor sample and a first neighbor sample in a direction opposite to the prediction direction of the intra-prediction mode of the current block relative to the predicted sample of the current block. This case can be referred to as Linear Interpolation Intra-Prediction (LIP). Furthermore, a provisional predicted sample of the current block can be derived based on filtered neighboring reference samples, and a predicted sample of the current block can be derived by performing a weighted sum on at least one reference sample derived according to the intra-prediction mode (i.e., an unfiltered neighboring reference sample) among the existing neighboring reference samples and the provisional predicted sample. This case can be referred to as PDPC (Location-Related Intra-Prediction). Alternatively, intra-frame predictive coding can be performed by selecting the reference sample row with the highest prediction accuracy from among multiple neighboring reference sample rows in the current block, deriving the predicted sample using reference samples from the corresponding row located in the prediction direction, and signaling the used reference sample row to the decoding device. This can be called Multi-Reference Row Intra-Frame Prediction (MRL) or MRL-based intra-frame prediction. Furthermore, the current block can be divided into vertical or horizontal sub-partitions, and intra-frame prediction can be performed based on the same intra-frame prediction mode, while neighboring reference samples can be derived and used based on the sub-partitions. That is, in this case, the intra-frame prediction mode used for the current block is applied to the sub-partitions in the same way, but neighboring reference samples are derived and used based on the sub-partitions, thereby improving intra-frame prediction performance in some cases. This prediction method can be called Intra-Frame Sub-Partition (ISP) or ISP-based intra-frame prediction. Additionally, when the prediction direction relative to the predicted sample points between neighboring reference samples, i.e., when the prediction direction points to the fractional sample position, the value of the predicted sample can be derived by interpolating multiple reference samples located around the corresponding prediction direction.

[0160] The aforementioned intra-prediction methods can be referred to as intra-prediction types, distinguishing them from the aforementioned intra-prediction modes. Intra-prediction types can also be referred to by various terms, such as intra-prediction techniques or additional intra-prediction modes. For example, an intra-prediction type or additional intra-prediction mode may include at least one of LIP, PDPC, MRL, and ISP. Information about the intra-prediction type can be encoded in the image encoding device (100), included in the bitstream, and signaled to the image decoding device (200). Information about the intra-prediction type can be implemented in various forms, such as flag information indicating whether each intra-prediction type is applied or index information indicating one of multiple intra-prediction types.

[0161] The MPM list used to derive the above intra-prediction modes can be constructed differently depending on the intra-prediction type. Alternatively, the MPM list can be constructed uniformly regardless of the intra-prediction type.

[0162] Decoder-side intra mode derivation (DIMD)

[0163] The DIMD mode can be used by derivation in the image encoding device (100) and the image decoding device (200) without directly sending intra-frame prediction mode information. First, the horizontal and vertical gradients can be obtained from the second neighbor sample column and sample row, and HoG (gradient histogram) can be constructed from them.

[0164] HoG can be like FIG. 10 The diagram shows the construction. HoG can be obtained by applying a Sobel filter using an L-shaped row and column of three pixels surrounding the current block. HoG is not used for texture analysis when the block boundaries exist in different CTUs.

[0165] After that, as FIG. 11 As shown, two intra-frame modes with the largest histogram magnitudes can be selected, and the final prediction block can be constructed by mixing prediction blocks predicted using these modes with the planar mode. The weights can be derived from the histogram magnitudes. Additionally, DIMD flags can be sent based on the block to check whether DIMD is used.

[0166] Intra template matching prediction (Intra TMP)

[0167] Intra-Template Matching Prediction (IntraTMP) is a special intra-prediction mode in which the optimal prediction block is copied (derived) from the reconstructed portion of the current frame, in which an L-shaped template is matched with the current template. For a predetermined search region, the image coding device (100) can search for the template most similar to the current template in the reconstructed portion of the current frame and can match the corresponding block ( FIG. 12 The matching block is used as the prediction block. The image encoding device (100) can send the IntraTMP usage, and the same prediction operation can be performed at the image decoding device (200).

[0168] A prediction signal can be generated by matching the L-shaped causal neighbors of the current block with another block in a predefined search region, such as... FIG. 12 As shown. In FIG. 12 In the diagram, R1 represents the current CTU, R2 represents the upper left CTU, R3 represents the upper CTU, and R4 represents the left CTU.

[0169] SAD (Sum of Absolute Differences) can be used as a cost function. Within each region, the image decoding device (200) can search for a template with the minimum SAD for the current block, and the block corresponding to the searched template can be used as a prediction block.

[0170] Error can be referred to as cost, error, difference, template cost, etc. That is, in this disclosure, "error," "cost," "difference," "template cost," etc., can have the same meaning. Error can be derived based on at least one of SAD (sum of differences), SATD (sum of transformed differences), SSE (sum of squared errors), MR-SAD (sum of differences to the mean), MR-SSE (sum of squared errors to the mean), and MR-SATD (sum of transformed differences to the mean).

[0171] The dimensions of all search regions (SearchRange_w, SearchRange_h) can be set proportionally to the block size (BlkW, BlkH) to allow for a fixed number of SAD comparisons per pixel. That is, the dimensions of all search regions can be set as shown in Equation 1.

[0172] [Formula 1]

[0173] In Equation 1, α is a constant that controls the trade-off between gain and complexity. For example, α can be equal to 5.

[0174] To improve the speed of template matching processing, the search range of all search regions can be subsampled by a factor of 2. In this case, the template matching search can be reduced by a factor of 4. After finding the optimal match, a refinement process can be performed. The refinement process can be performed by conducting a second template matching search around the optimal match with a reduced range. The reduced range can be defined as min(BlkW, BlkH) / 2.

[0175] Intra-frame template matching can be enabled for CUs with a width and height of 64 or less. The maximum CU size for intra-frame template matching can be variable.

[0176] When DIMD (decoder-side intra-mode derivation) is not used in the current CU, a special flag can be used to signal the intra-template matching prediction mode at the CU level. Here, the special flag can indicate whether the intra-template matching prediction mode is applied.

[0177] Intra TMP derived block vector candidate for IBC

[0178] The block vector (IntraTMP BV) derived from intra-template matching prediction can be used for IBC. The IntraTMP BV of the stored neighboring blocks can be used together with the IBC BV as spatial BV candidates during the construction of the IBC candidate list.

[0179] IntraTMP BV can be stored in the IBC block vector buffer. For example... FIG. 13 As shown, the current IBC block can use both the IBC BV and IntraTMP BV of the neighboring block as BV candidates in the IBC BV candidate list.

[0180] IntraTMP BV can be added to the IBC BV candidate list as a spatial candidate.

[0181] Intra block copy (IBC)

[0182] Reference FIG. 14 In IBC mode, block matching is performed between the current block and the reference block to find the optimal block vector (BV). Additionally, IBC mode can manage multiple block vector candidates in a candidate list. It can signal one of the block vector candidates constructed in the candidate list, and can also signal block vector information.

[0183] Block vectors are used to represent the displacement from the current block to a reference block that has been reconstructed in the current image.

[0184] Spatial geometry partition mode (SGPM)

[0185] Similar to inter-frame GPM, SGPM consists of a partition mode (partition_mode) and two associated intra-prediction modes (intra_pred_mode0, intra_pred_mode1). For example... FIG. 15 As shown in (b), when these patterns are directly signaled in the bitstream, very large overhead bits can be generated. To more efficiently represent basic partitioning and prediction information in the bitstream, such as... FIG. 15 As shown in (c), a candidate list is applied, and only the candidate index (sgpm_cand_idx) is signaled in the bitstream.

[0186] The block size constraints for SGPM can be as follows: 4 ≤ width ≤ 64, 4 ≤ height ≤ 64, width < height × 8, height < width × 8, width × height ≥ 32. That is, SGPM can also be applied to 4×8, 8×4, 4×16, and 16×4 blocks.

[0187] In SGPM, adaptive blending is enabled. Assuming the original blend width of VVC and ECM is τ, adaptive SGPM blending can be described as follows.

[0188] - When min(width, height) == 4, select 1 / 2. τ.

[0189] - Otherwise, when min(width, height) == 8, choose τ.

[0190] Otherwise, when min(width, height) == 16, select 2. τ.

[0191] Otherwise, when min(width, height) == 32, select 4. τ.

[0192] - Otherwise, choose 8 τ.

[0193] In SGPM, 26 of the 64 partition patterns are predefined.

[0194] For each partition mode, an IPM list (candidate list) for each partition is derived. The size of the IPM list is 3. The method for deriving the IPM list reuses the method introduced by inter-frame-intra-frame GPM, and one difference is the exclusion of TIMD derivation.

[0195] The template size is set to 1. That is, the height of the upper template portion is 1, and the width of the left template portion is 1.

[0196] Possible combinations between a partitioned mode and two intra-prediction modes are based on the SAD (Sum of Abilities) of the template's prediction and reconstruction, ordered in ascending order. The template weights are 1 or 0 depending on the partitioned mode. The length of the candidate list for combinations is set to 16.

[0197] Intra-prediction modes (horizontal and vertical) derived from TIMD are added to the IPM list and placed in the first position. The size of the IPM list is maintained at 3.

[0198] Template-based multi-reference line intra prediction (TMRL)

[0199] The TMRL mode combines reference lines and prediction modes, constructing a list of candidate combinations using template matching. The indices of the candidate combination list are encoded to indicate which reference line and prediction mode to use when encoding the current block. The regular multiple reference line (MRL) used for non-TIMD sections is replaced by the TMRL mode.

[0200] The TMRL mode expands the reference line candidate list and the intra-prediction mode candidate list. The expanded reference line candidate list is {1, 3, 5, 7, 12}. Constraints on the upper CTU row remain unchanged. The size of the intra-prediction mode candidate list is 10. The construction of the intra-prediction mode candidate list is similar to that of MPM, but the difference is that planar modes are excluded from the intra-prediction mode candidate list. When DC modes are not included, DC modes are added after the modes of the five neighboring PUs and the DIMD mode, and angle modes with incremental angles of ±1 to ±4 are added compared to the angle modes present in the intra-prediction mode candidate list. The accuracy of angle prediction is expanded from 65 to 129. Additionally, non-adjacent positions are added as candidates for constructing the intra-prediction mode candidate list. When neighboring blocks and non-adjacent blocks are encoded in SGPM or GPM mode, the intra-prediction mode of the block is replaced by the partition angle.

[0201] The TMRL candidate is constructed as follows. There are 5 × 10 = 50 combinations of extended reference lines for the block and allowed intra-prediction modes. Since the extended reference line starts from reference line 1, the region covered by reference line 0 is used for template matching. For example... FIG. 16 As shown, the SAD error is calculated for the template region between the prediction and reconstruction generated from 50 combinations. The twenty combinations with the smallest SAD error are selected in ascending order to construct the TMRL candidate list.

[0202] For TMRL signaling, instead of directly encoding the reference line and intra-frame mode, the index of the TMRL candidate list is encoded to indicate the combination of reference line and prediction mode used to encode the current block.

[0203] Derivation of prediction samples for chroma components

[0204] When performing intra-prediction on the current block, prediction can be performed on the luma component block (luma block) and the chroma component block (chroma block). In this case, the intra-prediction mode for the chroma component (chroma block) can be set independently of the intra-prediction mode for the luma component (luma block).

[0205] For example, the intra-chroma prediction mode for the chroma component can be indicated based on intra-chroma prediction mode information, and this information can be signaled in the form of the `intra_chroma_pred_mode` syntax element. As an example, the intra-chroma prediction mode information can indicate one of several candidate modes, including at least one of the following: planar mode, DC mode, vertical mode, horizontal mode, DM (derived mode), L_CCLM (cross-component linear model), T_CCLM, and LT_CCLM. DM can also be referred to as direct mode. CCLM can be referred to as LM.

[0206] Simultaneously, DM and CCLM predict chroma blocks using information from the luma blocks. DM can indicate a mode in which the same intra-prediction mode used for the luma component is applied as the intra-prediction mode for the chroma component. Additionally, CCLM can indicate an intra-prediction mode in which, during the generation of the prediction block for the chroma block, reconstructed samples of the luma block are subsampled, and samples derived by applying CCLM parameters α and β to the subsampled samples are used as prediction samples for the chroma block.

[0207] Direct block vector (DBV) for chroma blocks

[0208] DBV is used for chroma blocks within a dual-tree slice. Enabling chroma dual-tree and signaling a flag indicating whether the chroma block is encoded using IBC mode. FIG. 17 When one of the luma blocks at the five locations shown is encoded using IBC or IntraTMP, the block vector of the corresponding luma block is scaled and used as the block vector of the chroma block. Template matching is used to perform the block vector scaling.

[0209] This application will be described below. This application relates to intra-frame prediction and to an efficient intra-frame pattern coding method. According to the embodiments proposed in this application, SGPM candidates can be constructed efficiently, and block vectors can be utilized. Therefore, according to this application, prediction blocks can be generated in various ways, and prediction performance can be improved.

[0210] Problems of related art

[0211] As described above, in SGPM, a candidate list can be constructed using a combination of partition information (partition_mode_idx, indicating the partition mode of SGPM), the intra-pred mode of partition 0 (intra_pred_mode0_idx, indicating the intra-prediction mode of partition 0), and the intra-prediction mode of partition 1 (intra_pred_mode1_idx, indicating the intra-prediction mode of partition 1). Furthermore, in SGPM, an index (sgpm_cand_idx) indicating one of these candidates can be signaled, and prediction blocks can be generated based on the candidate indicated by the index sgpm_cand_idx.

[0212] However, in regular SGPM, the candidate list is constructed using only combinations of planar modes, DC modes, and directional modes (intra-prediction modes from 2 to 67) for each partition, without considering combinations of IntraTMP or IBC. That is, in regular SGPM, combinations of IntraTMP or IBC (block vectors of IntraTMP or block vectors of IBC) are not included in the candidate list.

[0213] Embodiments

[0214] This application proposes an implementation method for constructing a candidate list by considering a combination of IntraTMP or IBC. Additionally, this application proposes various implementation methods for constructing SGPM candidates. Furthermore, this application proposes various implementation methods utilizing SGPM candidates (candidate block vectors). According to the implementation methods proposed in this application, coding efficiency can be improved. The implementation methods described below can be performed independently, or two or more implementation methods can be combined and performed together. In the following text, "IntraTMP" is referred to as "IntraTMP mode," and "IBC" is referred to as "IBC prediction mode."

[0215] An example of the operation of the SGPM proposed in this application is illustrated in FIG. 18 middle. FIG. 18 The steps can be performed in the image encoding device (100) and the image decoding device (200).

[0216] Reference FIG. 18 In step S1800, the image encoding device (100) encodes the information (SGPM flag) indicating whether to perform SGPM, and the image decoding device (200) can decode the SGPM information (SGPM flag). When the value of the SGPM flag is true, SGPM can be performed.

[0217] Step S1810 corresponds to the step of constructing the candidate list for SGPM (SGPM candidate list). That is, step S1810 corresponds to the step of constructing the candidate list for SGPM. A combination of partition mode candidate partition_mode_idx, intra_pred_mode0_idx of partition 0, and intra_pred_mode1_idx of partition 1 can be included as a candidate in the candidate list. The partition mode candidate can be called a partition mode, and the intra_pred_mode candidate can be called a prediction mode. Each of partition_mode_idx, intra_pred_mode0_idx, and intra_pred_mode1_idx can have various values. For example, the number of block partitioning methods (partition mode candidates) can be 26. That is, partition_mode_idx can be a value representing 26 partitioning methods. In addition, the intra_pred_mode0_idx and intra_pred_mode1_idx intra_mode candidates can each be three. This is just an example and can vary depending on the conventions between the image encoding device (100) and the image decoding device (200).

[0218] Among the candidates included in the SGPM candidate list, the prediction mode of a candidate (the first candidate) can be determined based on the IntraTMP mode or IBC prediction mode at a reference location used to derive the prediction mode. For example, the IntraTMP mode or IBC prediction mode at the reference location can be determined as the first candidate's prediction mode, or the block vector of the IntraTMP mode or IBC prediction mode at the reference location can be determined as the first candidate's prediction mode. As another example, information indicating the IntraTMP mode or IBC prediction mode at the reference location can be determined as the first candidate's prediction mode. When the intra-prediction type at the reference location is IntraTMP mode or IBC prediction mode, the IntraTMP mode or IBC prediction mode at the reference location (or the block vector of the IntraTMP mode or IBC prediction mode, or the information indicating the IntraTMP mode or IBC prediction mode) can be determined as the first candidate's prediction mode. The reference location can be a reference block.

[0219] The size of the SGPM candidate list can be defined by an agreement between the image encoding device (100) and the image decoding device (200). For example, the size of the SGPM candidate list can be 16. Since the size of the SGPM candidate list can be limited, not all combinations of partition_mode_idx, intra_pred_mode0_idx, and intra_pred_mode1_idx can be included in the SGPM candidate list. For example, partition_mode_idx can represent 26 partitioning methods, and intra_pred_mode0_idx and intra_pred_mode1_idx can each represent three intra-frame mode candidates, thus generating 26 × 3 × 3 = 234 combinations. Therefore, including all candidates in the SGPM candidate list may be inefficient in terms of signaling. Therefore, for all or predefined combinations, such as... FIG. 19 As illustrated, the error between the values ​​of the SGPM predicted samples in the template region of the current block and the values ​​of the reconstructed samples can be obtained. The combinations can be sorted in order of increasing error, and 16 combinations with smaller errors can be included in the SGPM candidate list.

[0220] Errors can be calculated using error calculation methods such as SAD (sum of differences), SATD (sum of transformed differences), SSE (sum of squared errors), MR-SAD (sum of differences to the mean), MR-SSE (sum of squared errors to the mean), and MR-SATD (sum of transformed differences to the mean).

[0221] The template regions can be defined by an agreement between the image encoding device (100) and the image decoding device (200). For example, when the width and height of the current block are W and H respectively, the upper template region of the current block can be defined as an m×W region, and the left template region of the current block can be defined as an n×H region. Here, m and n can be any values ​​equal to or greater than 1.

[0222] Step S1820 corresponds to the step of generating a prediction block. That is, step S1820 corresponds to the step of predicting the current block based on a candidate in the SGPM candidate list. In step S1820, one of the candidates in the SGPM candidate list is selected, and a prediction block can be generated based on it. When selecting a candidate from the SGPM candidate list, index information (sgpm_cand_idx) can be signaled to indicate which candidate is used to generate the prediction block. According to the implementation, without signaling the index information, a candidate for SGPM can be known through an agreement between the image encoding device (100) and the image decoding device (200). For example, without separate signaling, a candidate corresponding to the first index of the SGPM candidate list can be selected.

[0223] Embodiment 1

[0224] Implementation 1 relates to a method for constructing SGPM candidates. Through Implementation 1, various methods are proposed for obtaining intra-mode candidates (intra_pred_mode0_idx or intra_pred_mode1_idx) in step S1810. Here, the intra-mode candidate can be a prediction mode for the intra-prediction mode of each partition divided by the partitioning mode of the SGPM.

[0225] According to one implementation, the intra-mode candidate for each partition can vary according to the partitioning orientation of the block. That is, among the candidates in the SGPM candidate list, the prediction mode of one candidate (the second candidate) can be determined based on the partitioning orientation indicated by the partitioning mode of the second candidate. In SGPM, when a block is partitioned, the block can be divided into two partitions, and the intra-mode candidate for each partition can be derived from a reference block (reference position) closest to each partition. Therefore, the intra-mode candidate can vary according to the partitioning orientation of the block.

[0226] For example, such as FIG. 20 As illustrated in (a), when the left side of a vertical partition is partition 0 and the right side of a vertical partition is partition 1, partition 0 can determine intra-mode candidates by referring to reference positions close to partition 0 (such as TL (top left), L (left), and BL (bottom left)), and partition 1 can determine intra-prediction candidates by referring to reference positions close to partition 1 (such as T (top) and TL (top left)). As another example, such as... FIG. 20As illustrated in (b), when the upper side of a horizontal partition is partition 0 and the lower side of a horizontal partition is partition 1, partition 0 can determine intra-mode candidates by referring to reference positions close to partition 0 (such as TL, T, and TR), and partition 1 can determine intra-mode candidates by referring to reference positions close to partition 1 (such as L and BL). FIG. 20 In the examples, the reference positions TL, T, TR, L, and BL are merely examples, and the reference positions may be defined differently by conventions between the image encoding device (100) and the image decoding device (200).

[0227] According to one implementation, an intra-frame pattern at a specific location (specific reference location) can be included as a candidate for each partition, regardless of the partitioning orientation of the block. That is, a second candidate prediction pattern can be determined as the intra-frame prediction pattern at a predetermined reference location, regardless of the partitioning orientation indicated by the second candidate partitioning pattern. The specific location or predetermined reference location can be defined by an agreement between the image encoding device (100) and the image decoding device (200). For example, the intra-frame prediction pattern at the TL location of the current block can be included as a candidate for each partition, regardless of the partitioning orientation of the block.

[0228] According to one implementation, intra-prediction modes stored at reference locations adjacent to or not adjacent to the current block can be included as intra-prediction mode candidates for each partition. That is, among the candidates in the SGPM candidate list, the prediction mode of one candidate (second candidate) can be determined as the intra-prediction mode stored at a reference location adjacent to or not adjacent to the current block. For example, as... FIG. 21 As illustrated, intra-mode candidates can be intra-modes stored at adjacent or non-adjacent locations. Furthermore, the intra-mode candidates for each partition can vary depending on the partition orientation. That is, intra-mode candidates can be included by referencing reference positions close to each partition. Alternatively, intra-mode candidates for each partition can be included regardless of the partition orientation. Additionally, the method for including intra-mode candidates can vary depending on whether the reference positions are adjacent or non-adjacent. For example, for adjacent reference positions, intra-modes at different reference positions depending on the partition orientation can be included as intra-mode candidates for the corresponding partition, and for non-adjacent reference positions, intra-modes at the same reference position can be included as intra-mode candidates for the corresponding partition, regardless of the partition orientation.

[0229] Intra-frame mode candidates can be planar modes, DC modes, or directional modes from 2 to 66. Additionally, intra-frame mode candidates can be IBC prediction modes or IntraTMP modes. When the intra-frame mode stored at the reference location is an IBC prediction mode or an IntraTMP mode, prediction blocks can be generated using block vector information stored at the reference location. That is, prediction samples can be generated using already reconstructed samples at the locations indicated by the stored block vectors.

[0230] exist FIG. 21 In the above, left coordinates of the current block are (x, y), and the width and height of the current block are W and H respectively, the adjacent reference positions can be (x-1, y-1), (x+W-1, y-1), (x+W, y-1), (x-1, y+H), and (x-1, y+H-1). However, this is just an example, and other adjacent positions can be defined by conventions between the image encoding device (100) and the image decoding device (200). When the left, left coordinates of the current block are (x, y), and the width and height of the current block are W and H respectively, the non-adjacent reference positions can be represented as NonAdj_pos of the combination of Tables 1 to 3. However, this is just an example, and other non-adjacent positions can be defined by conventions between the image encoding device (100) and the image decoding device (200).

[0231] [Table 1]

[0232] [Table 2]

[0233] [Table 3]

[0234] As another example, when the top-left coordinate of the current block is (x, y), and the width and height of the current block are W and H respectively, the non-adjacent reference position can be represented as NonAdj_pos, a combination of Tables 4 and 5. However, this is just an example, and other non-adjacent positions can be defined by conventions between the image encoding device (100) and the image decoding device (200).

[0235] [Table 4]

[0236] [Table 5]

[0237] According to one implementation, intra-modes stored in a history-based list can be included as intra-mode candidates. That is, among the candidates in the SGPM candidate list, the prediction mode of one candidate (second candidate) can be determined as the intra-prediction mode stored in the history-based list. For example, when a reconstructed block is encoded in SGPM, the intra-mode used in the corresponding partition of that block can be included in the history-based list. Therefore, intra-modes stored in the history-based list can be included as intra-mode candidates for the corresponding partition of the current block. Furthermore, in the process of storing intra-modes in the history-based list, a separate history-based list can be constructed for each partition, or a single history-based list can be constructed regardless of the partition. However, this is merely an example of constructing a history-based list and can be defined differently by conventions between the image encoding device (100) and the image decoding device (200).

[0238] A history-based list can be initialized whenever a CTU row changes. Additionally, the history-based list can be initialized for each CTU. The size of the history-based list can be defined by an agreement between the image encoding device (100) and the image decoding device (200). For example, the size of the history-based list can be 6.

[0239] According to one implementation, intra-frame modes derived from DIMD modes and / or intra-frame modes derived from TIMD modes can be included as intra-frame mode candidates for each partition of the SGPM. That is, among the candidates in the SGPM candidate list, the prediction mode of one candidate (second candidate) can be determined as an intra-frame prediction mode derived based on DIMD or TIMD modes. In this case, multiple intra-frame modes derived from DIMD and / or TIMD modes can be included as intra-frame mode candidates. For example, the best mode and the second-best mode can be included as intra-frame mode candidates. Alternatively, intra-frame modes derived using the upper template region of DIMD and / or TIMD modes, and intra-frame modes derived using the left template region, can be included as intra-frame mode candidates for each partition of the SGPM.

[0240] According to one implementation, intra-modes mapped to (corresponding to) the partitioning direction of the SGPM can be included as intra-mode candidates for each partition of the SGPM. That is, among the candidates in the SGPM candidate list, the prediction mode of a candidate (second candidate) can be determined as the intra-mode prediction mode corresponding to the partitioning direction of the second candidate. For example, when a block is partitioned along a vertical direction, mode 50 corresponding to that partitioning direction can be included as an intra-mode candidate for each partition. Here, mode 50 is the mode that performs intra-mode prediction along the vertical direction, and can be configured such that modes other than mode 50 are indicated by index according to the agreement between the image encoding device (100) and the image decoding device (200).

[0241] According to one implementation, intra-modes perpendicular to the intra-mode mapped to (corresponding to) the partition direction of the SGPM can be included as intra-mode candidates for each partition of the SGPM. That is, among the candidates in the SGPM candidate list, the prediction mode of a candidate (second candidate) can be determined as an intra-prediction mode perpendicular to the partition direction of the second candidate (perpendicular to the directional intra-prediction mode corresponding to the partition direction of the second candidate). For example, when a block is partitioned along a vertical direction, mode 18 perpendicular to that direction can be included as an intra-mode candidate for each partition. Here, mode 18 is a mode that performs intra-prediction along a horizontal direction, and can be configured such that modes other than mode 18 are indicated by index according to an agreement between the image encoding device (100) and the image decoding device (200).

[0242] According to one implementation, when a reference location adjacent to the current block, or a reference location not adjacent to the current block, is encoded using IBC prediction mode, IntraTMP mode, or inter-frame mode (Inter mode), the intra-frame mode stored at that reference location can be used as an intra-frame mode candidate for SGPM. For example, when a reconstructed block is encoded using IBC prediction mode, IntraTMP mode, or inter-frame mode, the intra-frame mode stored at the location indicated by the block vector (or motion vector) stored at the reconstructed block location can be stored at the reconstructed block location using the block vector (or motion vector). Accordingly, when a reference location is encoded using IBC prediction mode, IntraTMP mode, or inter-frame mode, the intra-frame mode stored at that reference location can be used as an intra-frame mode candidate for SGPM.

[0243] According to one implementation, a planar mode or DC mode can be included as an intra-mode candidate for each partition of the SGPM.

[0244] According to one implementation, the number of intra-mode candidates in each partition of the SGPM can be limited. For example, each partition may include a maximum of three intra-mode candidates. When the number of intra-mode candidates is limited in this way, the prediction accuracy can vary depending on the construction order of the intra-mode candidates. To improve prediction accuracy, a specific method for including intra-mode candidates can be defined and used between the image encoding device (100) and the image decoding device (200). For example, in the specific method for including intra-mode candidates, the intra-mode candidates for each partition can be constructed in the following order: TIMD intra-mode, intra-mode mapped to (corresponding to) the block partition direction, intra-mode at the left position of the current block, intra-mode at the top position of the current block, intra-mode at the lower left position of the current block, intra-mode at the upper right position of the current block, intra-mode at the upper left position of the current block, intra-mode perpendicular to the intra-mode corresponding to the partition direction, and planar mode. However, this is merely an example, and the order in which intra-mode candidates are used to construct each partition can vary, and this order can be changed according to an agreement between the image encoding device (100) and the image decoding device (200). Additionally, as in the previous implementation, the intra-mode candidates for each partition can also vary according to the block partitioning direction.

[0245] Embodiment 2

[0246] Implementation method 2 relates to a method utilizing block vector information from SGPM.

[0247] In conventional SGPM, the SGPM candidate list is constructed using only combinations of planar, DC, and directional modes (modes 2 to 67) for each partition, without considering combinations of IntraTMP and IBC prediction modes. However, when IntraTMP or IBC prediction modes are included as intra-mode candidates for each partition of SGPM, the block vector information of SGPM (including the block vector information in the SGPM candidate list) can be utilized in various ways, thereby improving coding performance.

[0248] According to one implementation, when constructing a block vector candidate list for an IBC prediction mode, block vector information from SGPM can be included in the block vector candidate list. The IBC prediction mode can manage multiple block vector candidates in the candidate list (block vector candidate list). One or more block vector information items among the block vector candidates constructed in the block vector candidate list can be identified by signaling. Specifically, the IBC prediction mode can be divided into an IBC merging mode that does not signal MVD (Motion Vector Difference) and an IBC mode that signals MVD; the methods for constructing the block vector candidate list for the two modes can differ. In this case, block vector information from SGPM can be included in the block vector candidate list of the IBC mode. Furthermore, block vector information from SGPM can also be included in the block vector candidate list of the IBC merging mode. Since the methods for constructing the block vector candidate list for the IBC mode and the IBC merging mode can differ, the methods for configuring SGPM block vector information in the block vector candidate list (such as order or conditions) can also differ.

[0249] According to one implementation, IntraTMP mode can manage multiple block vector candidates as a candidate list, and can identify one or more block vector information items included in the candidate list via signaling. In this case, SGPM block vector information can be included in the block vector candidate list of IntraTMP mode.

[0250] According to one implementation, in DBV mode, block vector information stored in one or more luma blocks corresponding to the current chroma block and / or block vector information stored around the corresponding one or more luma blocks can be utilized. For example, when a luma block is encoded in IBC prediction mode or IntraTMP mode, its block vector information can be stored, and the stored block vector information can be used for encoding the chroma block. Therefore, when block vectors are utilized in SGPM, the block vector information of SGPM can be utilized in DBV mode, and by doing so, encoding performance can be improved.

[0251] According to one implementation, when encoding one or more luma blocks corresponding to the current chroma block using SGPM, the block vector information of SGPM can be used for encoding the current chroma block. When using the block vector of the luma block (i.e., the block vector information of the luma block encoded by SGPM) in the chroma block, the block vector information of the luma block can be scaled in different ways according to the color data format (e.g., YUV 4:2:0, YUV 4:2:2, or YUV 4:4:4).

[0252] According to one implementation, during the process of populating the MPM list in MPM mode, when a reconstructed block is encoded using SGPM and block vectors are used, a planar mode can be included in the MPM list. Additionally, during the process of populating the MPM list in MPM mode, when a reconstructed block is encoded using SGPM and block vectors are used, a DC mode can be included in the MPM list. Furthermore, during the process of populating the MPM list in MPM mode, when a reconstructed block is encoded using SGPM and block vectors are used, the intra-frame modes of partitions within the reconstructed block that do not use block vectors can be included in the MPM list.

[0253] According to one implementation, in chroma DM mode, when luma blocks are encoded using SGPM and block vectors are used, a planar mode can be used for chroma block encoding. Additionally, in chroma DM mode, when luma blocks are encoded using SGPM and block vectors are used, a DC mode can be used for chroma block encoding. Furthermore, in chroma DM mode, when luma blocks are encoded using SGPM and block vectors are used, the intra-frame mode of the partitions within the luma block that do not use block vectors can be used for chroma block encoding.

[0254] According to one implementation, in chroma DM mode, when a luma block is encoded using SGPM and block vectors are used, the block vectors used for encoding the luma block can be scaled according to the color data format (e.g., YUV4:2:0, YUV4:2:2, or YUV4:4:4), and the chroma reference block at the location indicated by the scaled block vector is copied and used as the current chroma block. Furthermore, when a luma block is encoded using SGPM and all partitions of the luma block use block vectors, the block vector of one of the partitions can be used to predict the chroma block. Of the two block vectors, the block vector used for chroma block prediction can be determined by an agreement between the image encoding device (100) and the image decoding device (200). For example, it can be agreed to use the block vector of partition 0. As another example, when a luma block is encoded using SGPM and all partitions of the luma block use block vectors, both block vectors can be used for chroma block prediction. Specifically, two chroma prediction blocks can be generated from the two scaled block vectors, and a weighted sum can be performed on them to generate the final prediction block. In this case, the weight values ​​can be determined by agreement between the image encoding device (100) and the image decoding device (200). For example, the weight values ​​can be a weighted average.

[0255] According to one implementation, during the construction of the intra-candidate list for TMRL modes, when a reconstructed block is encoded using SGPM mode and block vectors are used, a planar mode can be included in the intra-candidate list for TMRL modes. Additionally, during the construction of the intra-candidate list for TMRL modes, when a reconstructed block is encoded using SGPM mode and block vectors are used, a DC mode can be included in the intra-candidate list for TMRL modes. Furthermore, during the construction of the intra-candidate list for TMRL modes, when a reconstructed block is encoded using SGPM mode and block vectors are used, the intra-candidate modes of partitions within the reconstructed block that do not use block vectors can be included in the intra-candidate list for TMRL modes.

[0256] According to one implementation, during the filling of intra-mode candidates for each partition of the SGPM, when the reconstructed block is encoded using the SGPM mode and block vectors are used, a planar mode can be included as an intra-mode candidate for each partition. In this case, the intra-mode candidates for each partition can vary depending on the partitioning orientation of the block. For example, an intra-mode obtained by referencing a reconstructed block close to each partition can be included as an intra-mode candidate. That is, the intra-mode of the reconstructed block can be included or not included as an intra-mode candidate for each partition, depending on the location of the reconstructed block.

[0257] According to one implementation, during the filling of intra-mode candidates for each partition of the SGPM, when the reconstructed block is encoded by the SGPM and block vectors are used, the DC mode can be included as an intra-mode candidate for each partition. In this case, the intra-mode candidate for each partition can vary depending on the partitioning orientation of the block. For example, an intra-mode obtained by referencing a reconstructed block close to each partition can be included as an intra-mode candidate. That is, the intra-mode of the reconstructed block can be included or not included as an intra-mode candidate for each partition, depending on the location of the reconstructed block.

[0258] According to one implementation, during the filling of intra-mode candidates for each partition of the SGPM, when the reconstructed block is encoded by the SGPM and block vectors are used, the intra-modes of partitions of the reconstructed block that do not use block vectors can be included as intra-mode candidates for each partition. In this case, the intra-mode candidates for each partition can vary depending on the partition orientation. For example, intra-modes obtained by referencing reconstructed blocks close to each partition can be included as intra-mode candidates. That is, the intra-modes of the reconstructed block may or may not be included as intra-mode candidates for each partition, depending on the location of the reconstructed block.

[0259] According to one implementation, when encoding a reconstructed block using SGPM and employing block vectors, each partition of SGPM can use the block vector of the reconstructed block (or information indicating the block vector) as a candidate. For example, a predicted block for each partition of SGPM for the current block can be generated by copying a reference block at the location indicated by the block vector candidate to the current block.

[0260] FIG. 22 The diagram illustrates, by way of example, an application of a content streaming system according to an embodiment of the present disclosure.

[0261] like FIG. 22 As shown, a content streaming system applying embodiments of this disclosure may include an encoding server, a streaming server, a network server, a media storage device, a user device, and a multimedia input device.

[0262] An encoding server performs the function of compressing content input from multimedia input devices (such as smartphones, cameras, or camcorders) into digital data to generate a bitstream, and then sending that bitstream to a streaming server. As another example, when multimedia input devices such as smartphones, cameras, or camcorders generate bitstreams directly, the encoding server can be omitted.

[0263] The bitstream can be generated by applying the image encoding method and / or image encoding device according to the embodiments of this disclosure, and the streaming server can temporarily store the bitstream during the sending or receiving of the bitstream.

[0264] A streaming server can send multimedia data to a user device via a web server based on a user request, and the web server can act as a medium to notify the user of available services. When a user requests a desired service from the web server, the web server can send the request to the streaming server, and the streaming server can then send the multimedia data to the user. In this scenario, the content streaming system may include a separate control server, which can then perform the role of controlling commands / responses within the appropriate devices of the content streaming system.

[0265] A streaming server can receive content from media storage devices and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a predetermined period of time.

[0266] Examples of user devices may include mobile phones, smartphones, laptops, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation devices, tablet PCs, tablet computers, ultrabooks, wearable devices (e.g., watch-type devices (smartwatches), glasses-type devices (smart glasses), or HMDs (head-mounted displays)), digital TVs, desktop computers, and digital signage.

[0267] In a content streaming system, each server can be operated as a distributed server, and in this case, the data received at each server can be processed in a distributed manner.

[0268] The scope of this disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that enable operation of methods according to various embodiments to be performed on a device or computer, and non-transitory computer-readable media in which such software or instructions are stored and can be performed on a device or computer.

[0269] Industrial applicability

[0270] The embodiments of this disclosure can be used to encode / decode images.

Claims

1. An image decoding method performed by an image decoding device, the image decoding method comprising: Construct candidates for Spatial Geometric Partitioning Mode (SGPM), wherein each of the candidates includes a prediction mode for an intra-prediction mode for each partition divided by the partitioning mode of the SGPM; and Predict the current block based on one of the candidates. Among the candidates, the prediction mode of the first candidate is determined as information indicating the intra-block copy prediction mode of the reference position.

2. The image decoding method according to claim 1, wherein, The prediction pattern of the second candidate among the candidates is determined based on the partitioning direction indicated by the partitioning pattern of the second candidate.

3. The image decoding method according to claim 2, wherein, The prediction pattern of the second candidate is determined based on the reference position in the predetermined reference position that is adjacent to each partition divided in the partitioning direction of the second candidate.

4. The image decoding method according to claim 2, wherein, The prediction mode of the second candidate is determined to be either a directional intra-prediction mode corresponding to the partition direction of the second candidate or a directional intra-prediction mode perpendicular to the directional intra-prediction mode.

5. The image decoding method according to claim 1, wherein, The prediction pattern of the second candidate among the candidates is determined based on a reference position that is not adjacent to the current block.

6. The image decoding method according to claim 1, wherein, The prediction mode of the second candidate among the candidates is determined to be an intra-frame prediction mode stored in a history-based list.

7. The image decoding method according to claim 6, wherein, The history-based list is constructed using the intra-prediction mode of blocks reconstructed via SGPM mode.

8. The image decoding method according to claim 1, wherein, The prediction mode of the second candidate among the candidates was determined to be the DC mode.

9. The image decoding method according to claim 1, wherein, The prediction mode of the second candidate among the candidates is determined to be an intra-prediction mode derived from either DIMD based on the decoder-side intra-mode derivation or TIMD based on the template-side intra-mode derivation.

10. The image decoding method according to claim 1, wherein, The maximum number of candidates included in the candidate list has a predetermined value, and the intra prediction modes based on TIMD derivation, the intra prediction modes at predetermined reference positions adjacent to the current block, the directional intra prediction modes corresponding to each partition direction of the candidates, and the directional intra prediction modes perpendicular to the directional intra prediction modes are determined as the candidate prediction modes in a predetermined order.

11. The image decoding method according to claim 10, wherein, The predetermined sequence is as follows: intra-prediction mode based on TIMD, intra-prediction mode in the direction corresponding to the partition direction, intra-prediction mode in the predetermined reference position adjacent to the current block, and intra-prediction mode in the direction perpendicular to the intra-prediction mode in the direction.

12. The image decoding method according to claim 1, wherein, Based on the prediction mode of the reconstructed block being SGPM, and the block vector being used for the prediction of at least one of the partitions within the reconstructed block, one of the predetermined parameters is determined as the second candidate prediction mode among the candidates, and the predetermined parameters include a plane mode, a DC mode, an intra-prediction mode of a partition not used for prediction by the block vector, and the block vector.

13. An image encoding method performed by an image encoding device, the image encoding method comprising: Construct candidates for Spatial Geometric Partitioning Mode (SGPM), wherein each of the candidates includes a prediction mode for an intra-prediction mode for each partition divided by the partitioning mode of the SGPM; and Predict the current block based on one of the candidates. Among the candidates, the prediction mode of the first candidate is determined as information indicating the intra-block copy prediction mode of the reference position.

14. A computer-readable recording medium storing a bitstream generated by the image encoding method according to claim 13.

15. A method for transmitting a bitstream generated by an image encoding device, wherein, The image encoding method includes: Construct candidates for Spatial Geometric Partitioning Mode (SGPM), wherein each of the candidates includes a prediction mode for an intra-prediction mode for each partition divided by the partitioning mode of the SGPM; and Predict the current block based on one of the candidates. Among the candidates, the prediction mode of the first candidate is determined as information indicating the intra-block copy prediction mode of the reference position.

Images