Image encoding / decoding method and device for performing MIP and lfnst, and method for transmitting bit stream

Abstract

To provide an image decoding method performed by an image decoding device.SOLUTION: A method includes the steps of: generating prediction samples of a current block based on an intra prediction mode; generating residual samples of the current block based on inverse transform; and reconstructing the current block based on the prediction samples and the residual samples. The inverse transform includes inverse primary transform and inverse secondary transform. The intra prediction mode of the current block is derived as a planar mode based on the intra prediction with respect to the current block being MIP (matrix based intra prediction), in order to determine a transform set of the inverse secondary transform. The inverse secondary transform is performed based on information signaled via bit streams, and the information is derived based on a tree type with respect to the current block.SELECTED DRAWING: Figure 22

Description

[Technical field]

[0001] The present disclosure relates to an image encoding / decoding method and device, and more particularly to an image encoding / decoding method and device that applies a low frequency non-seperable transform (LFNST) to a block to which a matrix based intra prediction (MIP) has been applied, and a method for transmitting a bitstream generated by the image encoding method / device of the present disclosure. [Background technology]

[0002] Recently, the demand for high-resolution, high-quality images, for example, HD (High Definition) images and UHD (Ultra High Definition) images, is increasing in various fields. As the resolution and quality of image data increases, the amount of information or bits transmitted increases relatively compared to conventional image data. The increase in the amount of information or bits transmitted leads to an increase in transmission costs and storage costs.

[0003] This requires a highly efficient image compression technique for effectively transmitting, storing, and reproducing high-resolution, high-quality image information. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0005] Another object of the present disclosure is to provide a method and apparatus for encoding / decoding an image by applying LFNST to a block to which MIP has been applied.

[0006] Another object of the present disclosure is to provide a method for transmitting a bitstream generated by the image encoding method or apparatus according to the present disclosure.

[0007] Another object of the present disclosure is to provide a recording medium storing a bitstream generated by the image encoding method or apparatus according to the present disclosure.

[0008] Another object of the present disclosure is to provide a recording medium storing a bitstream that is received by the image decoding device according to the present disclosure, decoded, and used to restore an image.

[0009] The technical problems to be solved by the present disclosure are not limited to the above-mentioned technical problems, and other technical problems not described above will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the following description. [Means for solving the problem]

[0010] An image decoding method according to one aspect of the present disclosure includes a step of performing intra prediction on a current block to generate a predicted block, a step of performing inverse transformation on transformation coefficients of the current block to generate a residual block, and a step of reconstructing the current block based on the predicted block and the residual block, wherein the inverse transformation includes a primary inverse transformation and a secondary inverse transformation, and the secondary inverse transformation may be performed based on whether the intra prediction for the current block is MIP prediction or not.

[0011] In the image decoding method according to the present disclosure, the secondary inverse transform may be performed only when it is determined to perform the secondary inverse transform on the transform coefficients.

[0012] In the image decoding method according to the present disclosure, the decision of whether to perform a secondary inverse transform on the transform coefficients may be made based on information signaled via a bitstream.

[0013] In the image decoding method according to the present disclosure, the secondary inverse transform may include a step of determining a transform set of the secondary inverse transform based on an intra prediction mode of the current block, a step of selecting one of a plurality of transform kernels included in the transform set of the secondary inverse transform, and a step of performing the secondary inverse transform based on the selected transform kernel.

[0014] In the image decoding method according to the present disclosure, when intra prediction for the current block is MIP prediction, the intra prediction mode of the current block used to determine the transform set of the secondary inverse transform can be induced to a predetermined intra prediction mode.

[0015] In the image decoding method according to the present disclosure, when intra prediction for the current block is MIP prediction, the predetermined intra prediction mode can be derived from the MIP mode of the current block based on a predefined mapping table.

[0016] In the image decoding method according to the present disclosure, when the intra prediction for the current block is MIP prediction, the predetermined intra prediction mode may be induced to a planner mode.

[0017] In the image decoding method according to the present disclosure, if the intra prediction for the current block is MIP prediction, the secondary inverse transform for the transform coefficients may be skipped.

[0018] In the image decoding method according to the present disclosure, when intra prediction for the current block is MIP prediction, information indicating whether to perform a secondary inverse transform on the transform coefficients may not be signaled via a bitstream.

[0019] In the image decoding method according to the present disclosure, when intra prediction for the current block is MIP prediction, the transform kernel for the secondary inverse transform of the transform coefficients is not signaled via the bitstream and can be determined to be a predetermined transform kernel.

[0020] In the image decoding method according to the present disclosure, the number of transform kernels available when the current block is MIP predicted may be smaller than the number of transform kernels available when the current block is not MIP predicted.

[0021] In an image decoding method according to the present disclosure, first information indicating whether a secondary inverse transform is applied to the current block and second information indicating a transform kernel to be used for the secondary inverse transform are signaled as separate information, and the second information may be signaled when the first information indicates that a secondary inverse transform is applied to the current block.

[0022] An image decoding device according to another aspect of the present disclosure includes a memory and at least one processor, wherein the at least one processor performs intra prediction on a current block to generate a prediction block, performs an inverse transform on transform coefficients of the current block to generate a residual block, and reconstructs the current block based on the prediction block and the residual block, wherein the inverse transform includes a primary inverse transform and a secondary inverse transform, and the secondary inverse transform may be performed based on whether the intra prediction for the current block is MIP prediction.

[0023] An image encoding method according to another aspect of the present disclosure includes a step of performing intra prediction on a current block to generate a predicted block, a step of generating a residual block of the current block based on the predicted block, and a step of performing a transform on the residual block to generate transform coefficients, where the transform includes a linear transform and a secondary transform, and the secondary transform may be performed based on whether the intra prediction for the current block is MIP prediction or not.

[0024] A transmission method according to another aspect of the present disclosure can transmit a bitstream generated by the image encoding device or image encoding method of the present disclosure.

[0025] A computer-readable recording medium according to another aspect of the present disclosure can store a bitstream generated by the image encoding method or image encoding device of the present disclosure.

[0026] The features described above in the brief summary of the present disclosure are merely exemplary embodiments of the detailed description of the present disclosure that follows and are not intended to limit the scope of the present disclosure. Effect of the Invention

[0027] According to the present disclosure, an image encoding / decoding method and apparatus with improved encoding / decoding efficiency can be provided.

[0028] Furthermore, according to the present disclosure, a method and apparatus for encoding / decoding an image by applying LFNST to a block to which MIP has been applied can be provided.

[0029] According to the present disclosure, there can also be provided a method for transmitting a bitstream generated by the image encoding method or apparatus according to the present disclosure.

[0030] According to the present disclosure, a recording medium storing a bitstream generated by the image encoding method or apparatus according to the present disclosure can be provided.

[0031] Furthermore, according to the present disclosure, it is possible to provide a recording medium storing a bitstream that is received by the image decoding device according to the present disclosure, decoded, and used to restore an image.

[0032] The effects obtained by the present disclosure are not limited to the effects described above, and other effects not described above will be clearly understood by those having ordinary skill in the art to which the present disclosure pertains from the following description. [Brief description of the drawings]

[0033] [Figure 1] FIG. 1 is a schematic diagram illustrating a video coding system to which an embodiment of the present disclosure can be applied. [Diagram 2] 1 is a diagram illustrating an image encoding device to which an embodiment of the present disclosure can be applied; [Diagram 3] 1 is a diagram illustrating an image decoding device to which an embodiment of the present disclosure can be applied; [Figure 4] FIG. 13 is a diagram showing block division types according to a multi-type tree structure. [Diagram 5] FIG. 1 illustrates an exemplary signaling mechanism for partitioning information of a quadtree with nested multi-type tree structure according to the present disclosure. [Figure 6] 1 is a flowchart illustrating an intra-prediction based video / image coding method. [Figure 7] 13 is a diagram illustrating an example configuration of an intra prediction unit 185 according to the present disclosure. [Figure 8] 1 is a flowchart illustrating an intra-prediction based video / image decoding method. [Figure 9] 13 is a diagram illustrating an example configuration of an intra prediction unit 265 according to the present disclosure. [Figure 10] 13 is a flowchart showing an intra-prediction mode signaling procedure in an image encoding device. [Figure 11] 13 is a flowchart showing an intra-prediction mode determination procedure in the image decoding device. [Figure 12] 13 is a flowchart for explaining an intra-prediction mode derivation procedure in more detail. [Figure 13] FIG. 2 is a diagram illustrating intra-prediction directions according to one embodiment of the present disclosure. [Figure 14] FIG. 13 is a diagram illustrating intra-prediction directions according to another embodiment of the present disclosure. [Figure 15]FIG. 13 is a diagram for explaining the ALWIP process for a 4×4 block. [Figure 16] FIG. 13 is a diagram for explaining the ALWIP process for an 8×8 block. [Figure 17] FIG. 13 is a diagram for explaining the ALWIP process for an 8×4 block. [Figure 18] FIG. 13 is a diagram illustrating the ALWIP process for a 16×16 block. [Figure 19] FIG. 2 is a diagram for explaining the averaging step of the ALWIP process according to the present disclosure. [Figure 20] FIG. 2 is a diagram for explaining an interpolation step of the ALWIP process according to the present disclosure. [Figure 21] FIG. 13 is a diagram for explaining a conversion method applied to a residual block. [Figure 22] 1 is a flow chart illustrating a method for performing a secondary transform / inverse transform according to the present disclosure. [Figure 23] FIG. 11 is a diagram for explaining a method performed in an image decoding device depending on whether MIP and LFNST are applied according to another embodiment of the present disclosure. [Figure 24] FIG. 13 is a diagram for explaining a method performed in an image encoding device depending on whether MIP and LFNST are applied according to another embodiment of the present disclosure. [Diagram 25] FIG. 1 illustrates a content streaming system to which an embodiment of the present disclosure can be applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] The present disclosure will be described in detail below with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0035] In describing the embodiments of the present disclosure, if it is determined that a specific description of a known configuration or function may make the gist of the present disclosure unclear, the detailed description thereof will be omitted. In addition, in the drawings, parts that are not related to the description of the present disclosure are omitted, and similar parts are denoted by similar reference numerals.

[0036] In the present disclosure, when a certain component is "connected," "coupled," or "connected" to another component, this includes not only a direct connection relationship, but also an indirect connection relationship in which another component exists between them. Furthermore, when a certain component is described as "including" or "having" another component, this does not exclude another component, but means that the component can further include another component, unless otherwise specified to the contrary.

[0037] In this disclosure, terms such as "first" and "second" are used only for the purpose of distinguishing one component from another component, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of this disclosure, a first component in one embodiment may be called a second component in another embodiment, and similarly, a second component in one embodiment may be called a first component in another embodiment.

[0038] In this disclosure, components that are distinguished from one another are used to clearly describe the characteristics of each component, and do not necessarily mean that the components are separate. In other words, multiple components may be integrated and configured as a single hardware or software unit, or one component may be distributed and configured as multiple hardware or software units. Thus, even if not otherwise stated, such integrated or distributed embodiments are also included in the scope of the present disclosure.

[0039] In the present disclosure, the components described in the various embodiments are not necessarily essential components, and some may be optional components. Therefore, an embodiment consisting of a subset of the components described in one embodiment is also included in the scope of the present disclosure. In addition, an embodiment including other components in addition to the components described in the various embodiments is also included in the scope of the present disclosure.

[0040] The present disclosure relates to image encoding and decoding, and terms used in this disclosure may have ordinary meanings in the technical field to which the present disclosure belongs, unless they are newly defined in this disclosure.

[0041] In this disclosure, a "picture" generally means a unit indicating any one image in a particular time period, a slice / tile is a coding unit constituting a part of a picture, and one picture may be composed of one or more slices / tiles. Also, a slice / tile may include one or more coding tree units (CTUs).

[0042] In this disclosure, a "pixel" or a "pel" may refer to the smallest unit constituting one picture (or image). A term corresponding to a pixel may be a "sample." A sample may generally indicate a pixel or a pixel value, may indicate only a pixel / pixel value of a luma component, or may indicate only a pixel / pixel value of a chroma component.

[0043] In this disclosure, a "unit" may refer to a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. A unit may be used interchangeably with terms such as "sample array," "block," or "area," depending on the case. In a general case, an M×N block may include a set (or array) of samples or transform coefficients consisting of M columns and N rows.

[0044] In the present disclosure, a "current block" may refer to any one of a "current coding block," a "current coding unit," a "block to be coded," a "block to be decoded," or a "block to be processed." If prediction is performed, a "current block" may refer to a "current predicted block" or a "block to be predicted." If transformation (inverse transformation) / quantization (inverse quantization) is performed, a "current block" may refer to a "current transformed block" or a "block to be transformed." If filtering is performed, a "current block" may refer to a "block to be filtered."

[0045] Furthermore, in this disclosure, unless there is an explicit reference to a chroma block, a "current block" may refer to a block including all of the luma component blocks and the chroma component blocks or a "luma block of the current block." A chroma block of the current block may be explicitly expressed by including an explicit reference to a chroma block, such as a "chroma block" or a "current chroma block."

[0046] In the present disclosure, " / " and "," can be interpreted as "and / or." For example, "A / B" and "A, B" can be interpreted as "A and / or B." Also, "A / B / C" and "A, B, C" can mean "at least one of A, B, and / or C."

[0047] In this disclosure, "or" can be interpreted as "and / or." For example, "A or B" can mean 1) only "A," 2) only "B," or 3) "A and B." Or, in this disclosure, "or" can mean "additionally or alternatively."

[0048] Video Coding System Overview

[0049] FIG. 1 is a diagram illustrating a video coding system according to this disclosure.

[0050] A video coding system according to an embodiment may include an encoding device 10 and a decoding device 20. The encoding device 10 may transmit encoded video and / or image information or data to the decoding device 20 in a file or streaming format via a digital storage medium or a network.

[0051] The encoding device 10 according to an embodiment may include a video source generating unit 11, an encoding unit 12, and a transmitting unit 13. The decoding device 20 according to an embodiment may include a receiving unit 21, a decoding unit 22, and a rendering unit 23. The encoding unit 12 may be referred to as a video / image encoding unit, and the decoding unit 22 may be referred to as a video / image decoding unit. The transmitting unit 13 may be included in the encoding unit 12. The receiving unit 21 may be included in the decoding unit 22. The rendering unit 23 may include a display unit, which may be configured as a separate device or an external component.

[0052] The video source generating unit 11 can obtain the video / images through a process of capturing, synthesizing, or generating the video / images. The video source generating unit 11 can include a video / image capture device and / or a video / image generation device. The video / image capture device can include, for example, one or more cameras, a video / image archive containing previously captured videos / images, etc. The video / image generation device can include, for example, a computer, a tablet, a smartphone, etc., and can (electronically) generate the video / images. For example, a virtual video / image can be generated through a computer, etc., in which case the video / image capture process can be replaced by a process in which the associated data is generated.

[0053] The encoder 12 may encode the input video / image. The encoder 12 may perform a series of steps such as prediction, transformation, quantization, etc. for compression and encoding efficiency. The encoder 12 may output the encoded data (encoded video / image information) in a bitstream format.

[0054] The transmitting unit 13 may transmit the encoded video / image information or data output in a bitstream format to the receiving unit 21 of the decoding device 20 via a digital storage medium or a network in a file or streaming format. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit 13 may include an element for generating a media file via a predetermined file format and may include an element for transmitting via a broadcasting / communication network. The receiving unit 21 may extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit 22.

[0055] The decoding unit 22 can decode the video / image by performing a series of steps such as inverse quantization, inverse transformation, and prediction corresponding to the operations of the encoding unit 12.

[0056] The rendering unit 23 can render the decoded video / images. The rendered video / images can be displayed via a display unit.

[0057] Overview of the image encoding device

[0058] FIG. 2 is a diagram illustrating an image encoding device to which an embodiment of the present disclosure can be applied.

[0059] As shown in Fig. 2, the image coding device 100 may include an image division unit 110, a subtraction unit 115, a transformation unit 120, a quantization unit 130, an inverse quantization unit 140, an inverse transformation unit 150, an addition unit 155, a filtering unit 160, a memory 170, an inter prediction unit 180, an intra prediction unit 185, and an entropy coding unit 190. The inter prediction unit 180 and the intra prediction unit 185 may be collectively referred to as a "prediction unit." The transformation unit 120, the quantization unit 130, the inverse quantization unit 140, and the inverse transformation unit 150 may be included in a residual processing unit. The residual processing unit may further include a subtraction unit 115.

[0060] All or at least some of the components constituting the image encoding device 100 may be realized by a single hardware component (e.g., an encoder or a processor) depending on the embodiment. Also, the memory 170 may include a decoded picture buffer (DPB) and may be realized by a digital storage medium.

[0061] The image division unit 110 may divide an input image (or picture, frame) input to the image encoding device 100 into one or more processing units. As an example, the processing units may be called coding units (CUs). The coding units may be obtained by recursively dividing a coding tree unit (CTU) or a largest coding unit (LCU) according to a QT / BT / TT (Quad-tree / Binary-tree / Ternal-tree) structure. For example, one coding unit may be divided into a plurality of coding units of a deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary-tree structure. For dividing the coding units, a quad-tree structure may be applied first, and a binary-tree structure and / or a ternary-tree structure may be applied later. The encoding procedure according to the present disclosure may be performed based on a final coding unit that is not further divided. The maximum coding unit may be used as the final coding unit, and a lower depth coding unit obtained by dividing the maximum coding unit may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and / or restoration, which will be described later. As another example, a processing unit of the coding procedure may be a prediction unit (PU) or a transform unit (TU). The prediction unit and the transform unit may be divided or partitioned from the final coding unit, respectively. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0062] The prediction unit (inter prediction unit 180 or intra prediction unit 185) may perform prediction on a block to be processed (current block) and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied in units of a current block or a CU. The prediction unit may generate various information related to prediction of the current block and transmit it to the entropy encoding unit 190. The information related to prediction may be encoded by the entropy encoding unit 190 and output in a bitstream format.

[0063] The intra prediction unit 185 may predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block or may be located away from the current block according to an intra prediction mode and / or an intra prediction technique. The intra prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the degree of fineness of the prediction direction. However, this is merely an example, and more or less directional prediction modes may be used depending on the setting. The intra prediction unit 185 may also determine a prediction mode to be applied to the current block using prediction modes applied to neighboring blocks.

[0064] The inter prediction unit 180 may derive a predicted block for a current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of a block, a sub-block, or a sample based on the correlation of motion information between a neighboring block and a current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring block may include a spatial neighboring block present in the current picture and a temporal neighboring block present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different from each other. The temporal neighboring block may be called a collocated reference block, a collocated CU (colCU), etc. The reference picture including the temporal neighboring block may be called a collocated picture (colPic). For example, the inter prediction unit 180 may generate information indicating which candidate is used to derive a motion vector and / or a reference picture index of the current block by forming a motion information candidate list based on neighboring blocks. Inter prediction may be performed based on various prediction modes, and for example, in the case of a skip mode and a merge mode, the inter prediction unit 180 may use motion information of neighboring blocks as motion information of the current block. In the case of the skip mode, unlike the merge mode, a residual signal may not be transmitted.In the case of a motion vector prediction (MVP) mode, the motion vector of the current block can be signaled by using the motion vector of a neighboring block as a motion vector predictor and encoding a motion vector difference and an indicator for the motion vector predictor. The motion vector difference can mean the difference between the motion vector of the current block and the motion vector predictor.

[0065] The prediction unit may generate a prediction signal based on various prediction methods and / or prediction techniques, which will be described later. For example, the prediction unit may apply intra prediction or inter prediction for prediction of the current block, and may simultaneously apply intra prediction and inter prediction. A prediction method that simultaneously applies intra prediction and inter prediction for prediction of the current block may be called combined inter and intra prediction (CIIP). The prediction unit may also perform intra block copy (IBC) for prediction of the current block. Intra block copy can be used for content image / video coding such as games, for example, as in screen content coding (SCC). IBC is a method of predicting a current block using an already restored reference block in a current picture that is located a predetermined distance away from the current block. When IBC is applied, the position of the reference block in the current picture may be coded as a vector (block vector) corresponding to the predetermined distance. IBC is basically performed in the current picture, but may be performed similarly to inter prediction in that a reference block is derived in the current picture. That is, the IBC may use at least one of the inter prediction techniques described in this disclosure.

[0066] The prediction signal generated by the prediction unit may be used to generate a restored signal or a residual signal. The subtraction unit 115 may subtract the prediction signal (predicted block, prediction sample array) output from the prediction unit from the input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array). The generated residual signal may be transmitted to the conversion unit 120.

[0067] The transform unit 120 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). Here, the GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. The CNT refers to a transform obtained based on a predicted signal generated using all previously reconstructed pixels. The transform process may be applied to pixel blocks having the same square size, or may be applied to non-square variable-sized blocks.

[0068] The quantization unit 130 may quantize the transform coefficients and transmit the quantized transform coefficients to the entropy coding unit 190. The entropy coding unit 190 may code the quantized signal (information on the quantized transform coefficients) and output the coded signal in a bitstream format. The information on the quantized transform coefficients may be referred to as residual information. The quantization unit 130 may rearrange the quantized transform coefficients in a block format into a one-dimensional vector format based on a coefficient scan order, and may generate information on the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector format.

[0069] The entropy coding unit 190 may perform various coding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy coding unit 190 may also code information required for video / image restoration (e.g., values ​​of syntax elements, etc.) together or separately in addition to the quantized transform coefficients. The coded information (e.g., coded video / image information) may be transmitted or stored in a network abstraction layer (NAL) unit unit in a bitstream format. The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The signaling information, transmitted information and / or syntax elements referred to in this disclosure may be encoded through the above-mentioned encoding procedures and included in the bitstream.

[0070] The bitstream may be transmitted via a network or may be stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as a USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitting unit (not shown) for transmitting the signal output from the entropy encoding unit 190 and / or a storing unit (not shown) for storing the signal may be provided as an internal / external element of the image encoding device 100, or the transmitting unit may be provided as a component of the entropy encoding unit 190.

[0071] The quantized transform coefficients output from the quantization unit 130 can be used to generate a residual signal. For example, the quantized transform coefficients are subjected to inverse quantization and inverse transformation via the inverse quantization unit 140 and the inverse transformation unit 150, so that a residual signal (residual block or residual sample) can be restored.

[0072] The adder 155 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to a prediction signal output from the inter prediction unit 180 or the intra prediction unit 185. When there is no residual for the current block to be processed, such as when a skip mode is applied, a predicted block may be used as a reconstructed block. The adder 155 may be referred to as a reconstruction unit or a reconstructed block generation unit. The generated reconstructed signal may be used for intra prediction of the next current block to be processed in the current picture, and may also be used for inter prediction of the next picture after filtering as described below.

[0073] The filtering unit 160 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 160 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may store the modified reconstructed picture in the memory 170, specifically, in the DPB of the memory 170. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, and the like. The filtering unit 160 may generate various information related to filtering, as will be described later in the description of each filtering method, and transmit the information to the entropy coding unit 190. The information related to filtering may be coded by the entropy coding unit 190 and output in a bitstream format.

[0074] The modified reconstructed picture transmitted to the memory 170 may be used as a reference picture in the inter prediction unit 180. When inter prediction is applied through this, the image encoding device 100 may avoid a prediction mismatch between the image encoding device 100 and the image decoding device, and may also improve encoding efficiency.

[0075] The DPB in the memory 170 may store modified reconstructed pictures to be used as reference pictures in the inter prediction unit 180. The memory 170 may store motion information of blocks from which motion information in the current picture is derived (or coded) and / or motion information of already reconstructed intra-picture blocks. The stored motion information may be transmitted to the inter prediction unit 180 to be used as motion information of spatial surrounding blocks or motion information of temporal surrounding blocks. The memory 170 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 185.

[0076] Overview of the image decoding device

[0077] FIG. 3 is a diagram illustrating an image decoding device to which an embodiment of the present disclosure can be applied.

[0078] 3, the image decoding device 200 may include an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an adder 235, a filtering unit 240, a memory 250, an inter prediction unit 260, and an intra prediction unit 265. The inter prediction unit 260 and the intra prediction unit 265 may be collectively referred to as a "prediction unit." The inverse quantization unit 220 and the inverse transform unit 230 may be included in a residual processing unit.

[0079] All or at least some of the components constituting the image decoding device 200 may be realized by one hardware component (e.g., a decoder or a processor) depending on the embodiment. Also, the memory 170 may include a DPB and may be realized by a digital storage medium.

[0080] The image decoding device 200, which receives a bitstream including video / image information, can reconstruct an image by executing a process corresponding to the process performed by the image encoding device 100 of Fig. 2. For example, the image decoding device 200 can perform decoding using a processing unit applied in the image encoding device. Thus, the processing unit for decoding can be, for example, a coding unit. The coding unit can be obtained by dividing a coding tree unit or a maximum coding unit. Then, the reconstructed image signal decoded and output by the image decoding device 200 can be reproduced by a reproduction device (not shown).

[0081] The image decoding apparatus 200 may receive a signal output from the image encoding apparatus of FIG. 2 in the form of a bitstream. The received signal may be decoded via the entropy decoding unit 210. For example, the entropy decoding unit 210 may derive information (e.g., video / image information) required for image restoration (or picture restoration) by parsing the bitstream. The video / image information may further include information on various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The image decoding apparatus may further use information on the parameter set and / or the general constraint information to decode an image. The signaling information, received information, and / or syntax elements referred to in the present disclosure may be obtained from the bitstream by being decoded via the decoding procedure. For example, the entropy decoding unit 210 may decode information in a bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients related to the residual. More specifically, the CABAC entropy decoding method may receive bins corresponding to each syntax element from the bitstream, determine a context model using syntax element information to be decoded and decoded information of neighboring blocks and a block to be decoded, or information of a symbol / bin decoded in a previous step, predict the occurrence probability of the bin based on the determined context model, and perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method may update the context model using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model.Among the information decoded by the entropy decoding unit 210, information related to prediction is provided to a prediction unit (inter prediction unit 260 and intra prediction unit 265), and residual values ​​entropy-decoded by the entropy decoding unit 210, i.e., quantized transform coefficients and related parameter information, may be input to the inverse quantization unit 220. Also, among the information decoded by the entropy decoding unit 210, information related to filtering may be provided to the filtering unit 240. Meanwhile, a receiving unit (not shown) for receiving a signal output from the image encoding device may be further provided as an internal / external element of the image decoding device 200, or the receiving unit may be provided as a component of the entropy decoding unit 210.

[0082] Meanwhile, the image decoding device according to the present disclosure may be called a video / image / picture decoding device. The image decoding device may include an information decoder (video / image / picture information decoder) and / or a sample decoder (video / image / picture sample decoder). The information decoder may include an entropy decoding unit 210, and the sample decoder may include at least one of an inverse quantization unit 220, an inverse transform unit 230, an adder 235, a filtering unit 240, a memory 250, an inter prediction unit 260, and an intra prediction unit 265.

[0083] The inverse quantization unit 220 may inverse quantize the quantized transform coefficients to output transform coefficients. The inverse quantization unit 220 may rearrange the quantized transform coefficients in a two-dimensional block format. In this case, the rearrangement may be performed based on a coefficient scan order performed in the image encoding device. The inverse quantization unit 220 may perform inverse quantization on the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0084] The inverse transform unit 230 can inversely transform the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0085] The prediction unit may perform prediction on a current block and generate a predicted block including a prediction sample for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on the prediction information output from the entropy decoding unit 210, and may determine a specific intra / inter prediction mode (prediction technique).

[0086] The prediction unit can generate a prediction signal based on various prediction methods (techniques) described below, as has been described in the explanation of the prediction unit of the image encoding device 100.

[0087] The intra prediction unit 265 may predict the current block by referring to samples in the current picture. The description of the intra prediction unit 185 may be similarly applied to the intra prediction unit 265.

[0088] The inter prediction unit 260 may derive a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter prediction unit 260 may configure a motion information candidate list based on the neighboring blocks, and derive a motion vector and / or a reference picture index for the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes (techniques), and the prediction information may include information indicating a mode (technique) of inter prediction for the current block.

[0089] The adder 235 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to a prediction signal (predicted block, predicted sample array) output from a prediction unit (including the inter prediction unit 260 and / or the intra prediction unit 265). When there is no residual for a current block to be processed, such as when a skip mode is applied, the predicted block may be used as a reconstructed block. The description of the adder 155 may be similarly applied to the adder 235. The adder 235 may be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of a next current block to be processed in a current picture, and may also be used for inter prediction of a next picture after filtering as described below.

[0090] The filtering unit 240 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 240 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may store the modified reconstructed picture in the memory 250, specifically, in the DPB of the memory 250. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc.

[0091] The (modified) reconstructed picture stored in the DPB of the memory 250 may be used as a reference picture in the inter prediction unit 260. The memory 250 may store motion information of a block from which motion information in the current picture is derived (or decoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter prediction unit 260 to be used as motion information of a spatial surrounding block or motion information of a temporal surrounding block. The memory 250 may store a reconstructed sample of a reconstructed block in the current picture and transmit it to the intra prediction unit 265.

[0092] In this specification, the embodiments described for the filtering unit 160, inter prediction unit 180 and intra prediction unit 185 of the image encoding device 100 can also be applied in a similar or corresponding manner to the filtering unit 240, inter prediction unit 260 and intra prediction unit 265 of the image decoding device 200, respectively.

[0093] Overview of CTU division

[0094] As mentioned above, a coding unit can be obtained by recursively splitting a coding tree unit (CTU) or a largest coding unit (LCU) according to a QT / BT / TT (Quad-tree / Binary-tree / Ternal-tree) structure. For example, a CTU can be first split into a quad-tree structure. Then, the leaf nodes of the quad-tree structure can be further split according to a multi-type tree structure.

[0095] Partitioning by a quadtree means dividing a current CU (or CTU) into four equal parts. By partitioning by a quadtree, the current CU can be divided into four CUs having the same width and height. If the current CU is not further divided into a quadtree structure, the current CU corresponds to a leaf node of the quadtree structure. The CU corresponding to the leaf node of the quadtree structure is not further divided and can be used as the above-mentioned final coding unit. Alternatively, the CU corresponding to the leaf node of the quadtree structure can be further divided by a multi-type tree structure.

[0096] 4 is a diagram showing the types of division of a block by a multi-type tree structure. Division by a multi-type tree structure can include two divisions by a binary tree structure and two divisions by a ternary tree structure.

[0097] The two divisions according to the binary tree structure may include vertical binary splitting (SPLIT_BT_VER) and horizontal binary splitting (SPLIT_BT_HOR). Vertical binary splitting (SPLIT_BT_VER) refers to a division in which the current CU is divided into two equal parts vertically. As shown in FIG. 4, the vertical binary splitting may generate two CUs having the same height as the height of the current CU and half the width of the current CU. Horizontal binary splitting (SPLIT_BT_HOR) refers to a division in which the current CU is divided into two equal parts horizontally. As shown in FIG. 4, the horizontal binary splitting may generate two CUs having the same height as the height of the current CU and half the width of the current CU.

[0098] The two divisions according to the ternary tree structure may include vertical ternary splitting (SPLIT_TT_VER) and horizontal ternary splitting (SPLIT_TT_HOR). The vertical ternary splitting (SPLIT_TT_VER) divides the current CU vertically at a ratio of 1:2:1. As shown in FIG. 4, the vertical ternary splitting may generate two CUs having the same height as the current CU and a width 1 / 4 of the current CU, and a CU having the same height as the current CU and a width half the current CU. The horizontal ternary splitting (SPLIT_TT_HOR) divides the current CU horizontally at a ratio of 1:2:1. As shown in FIG. 4, the horizontal ternary splitting may generate two CUs having a height 1 / 4 of the current CU and a width equal to the current CU, and a CU having a height half the current CU and a width equal to the current CU.

[0099] FIG. 5 is a diagram illustrating a signaling mechanism of partitioning information for a quadtree with nested multi-type tree structure according to the present disclosure.

[0100] Here, the CTU is treated as the root node of the quadtree and is first partitioned into a quadtree structure. Information (e.g., qt_split_flag) indicating whether or not to perform quadtree splitting for the current CU (CTU or quadtree node (QT_node)) can be signaled. For example, if qt_split_flag is a first value (e.g., "1"), the current CU can be split into a quadtree. Also, if qt_split_flag is a second value (e.g., "0"), the current CU is not split into a quadtree and becomes a leaf node (QT_leaf_node) of the quadtree. The leaf nodes of each quadtree can then be further partitioned into a multitype tree structure. That is, the leaf node of the quadtree can become a node (MTT_node) of the multitype tree. In a multi-type tree structure, a first flag (e.g., mtt_split_cu_flag) may be signaled to indicate whether the current node is further partitioned. If the node is further partitioned (e.g., the first flag is 1), a second flag (e.g., mtt_split_cu_vertical_flag) may be signaled to indicate the splitting direction. For example, if the second flag is 1, the splitting direction may be vertical, and if the second flag is 0, the splitting direction may be horizontal. Then, a third flag (e.g., mtt_split_cu_binary_flag) may be signaled to indicate whether the splitting type is a binary splitting type or a ternary splitting type. For example, if the third flag is 1, the splitting type may be a binary splitting type, and if the third flag is 0, the splitting type may be a ternary splitting type. The nodes of the multitype tree obtained by binary or ternary partitioning can be further partitioned into a multitype tree structure, but the nodes of the multitype tree cannot be partitioned into a quadtree structure.If the first flag is 0, the corresponding node of the multitype tree is not further divided and becomes a leaf node (MTT_leaf_node) of the multitype tree. The CU corresponding to the leaf node of the multitype tree can be used as the final coding unit described above.

[0101] Based on the above-mentioned mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the multi-type tree splitting mode (MttSplitMode) of the CU can be derived as shown in Table 1.

[0102] [Table 1]

[0103] One CTU may include a coding block of luma samples (hereinafter referred to as a "luma block") and two coding blocks of corresponding chroma samples (hereinafter referred to as "chroma blocks"). The above coding tree scheme may be applied to the luma blocks and chroma blocks of a current CU in the same manner, or may be applied separately. Specifically, the luma blocks and chroma blocks in one CTU may be divided into the same block tree structure, which may be represented as a single tree (SINGLE_TREE). Alternatively, the luma blocks and chroma blocks in one CTU may be divided into separate block tree structures, which may be represented as a dual tree (DUAL_TREE). In other words, when a CTU is divided into a dual tree, a block tree structure for the luma blocks and a block tree structure for the chroma blocks may exist separately. In this case, the block tree structure for the luma block may be called a dual tree luma (DUAL_TREE_LUMA), and the block tree structure for the chroma block may be called a dual tree chroma (DUAL_TREE_CHROMA). For P and B slice / tile groups, the luma block and the chroma block in one CTU may be restricted to have the same coding tree structure. However, for I slice / tile groups, the luma block and the chroma block may have separate block tree structures. If a separate block tree structure is applied, a luma coding tree block (CTB) may be divided into CUs based on a specific coding tree structure, and the chroma CTB may be divided into chroma CUs based on another coding tree structure. That is, a CU in an I slice / tile group to which a separate block tree structure is applied may be composed of a coding block of a luma component or a coding block of two chroma components.In addition, a CU in an I slice / tile group and a CU in a P or B slice / tile group to which the same block tree structure is applied can be composed of blocks of three color components (a luma component and two chroma components).

[0104] Although the quadtree coding tree structure with the multi-type tree has been described above, the structure in which the CU is divided is not limited thereto. For example, the BT structure and the TT structure can be interpreted as concepts included in a multiple partitioning tree (MPT) structure, and the CU can be interpreted as being divided by the QT structure and the MPT structure. In an example in which the CU is divided by the QT structure and the MPT structure, a syntax element (e.g., MPT_split_type) including information on whether the leaf node of the QT structure is divided into several blocks and a syntax element (e.g., MPT_split_mode) including information on which direction the leaf node of the QT structure is divided in, vertically or horizontally, are signaled, so that the division structure can be determined.

[0105] In another example, the CUs may be divided in a manner different from that of the QT structure, the BT structure, or the TT structure, that is, unlike the QT structure in which the CUs of the lower depth are divided into 1 / 4 size of the CUs of the higher depth, or the BT structure in which the CUs of the lower depth are divided into 1 / 2 size of the CUs of the higher depth, or the TT structure in which the CUs of the lower depth are divided into 1 / 4 or 1 / 2 size of the CUs of the higher depth, the CUs of the lower depth may be divided into 1 / 5, 1 / 3, 3 / 8, 3 / 5, 2 / 3, or 5 / 8 size of the CUs of the higher depth, as the case may be, and the manner in which the CUs are divided is not limited thereto.

[0106] Overview of Intra Prediction

[0107] Intra prediction according to the present disclosure will be described below.

[0108] Intra prediction may refer to a prediction that generates a prediction sample for a current block based on a reference sample in a picture to which the current block belongs (hereinafter, the current picture). When intra prediction is applied to the current block, neighboring reference samples used for intra prediction of the current block may be derived. The neighboring reference samples of the current block may include a total of 2×nH samples adjacent to the left boundary and bottom-left of the current block of size nW×nH, a total of 2×nW samples adjacent to the top boundary and top-right of the current block, and one sample adjacent to the top-left of the current block. Alternatively, the neighboring reference samples of the current block may include upper neighboring samples of multiple columns and left neighboring samples of multiple rows. In addition, the surrounding reference samples of the current block may include a total of nH samples adjacent to the right boundary of the current block of size nW×nH, a total of nW samples adjacent to the bottom boundary of the current block, and one sample adjacent to the bottom-right of the current block.

[0109] However, some of the neighboring reference samples of the current block may not be decoded or available. In this case, the decoder may substitute the unavailable samples as available samples to construct the neighboring reference samples used for prediction. Alternatively, the decoder may construct the neighboring reference samples used for prediction through interpolation of available samples.

[0110] When neighboring reference samples are derived, (i) a predicted sample may be derived based on an average or an interpolation of neighboring reference samples of the current block, or (ii) a predicted sample may be derived based on a reference sample that exists in a specific (prediction) direction for the predicted sample among the neighboring reference samples of the current block. The case of (i) may be called a non-directional mode or a non-angular mode, and the case of (ii) may be called a directional mode or an angular mode.

[0111] In addition, the prediction sample may be generated by interpolating a first neighboring sample located in a prediction direction of an intra prediction mode of the current block and a second neighboring sample located in the opposite direction based on a prediction target sample of the current block among the neighboring reference samples. The above case may be called linear interpolation intra prediction (LIP).

[0112] Alternatively, a chroma prediction sample may be generated based on a luma sample using a linear model, in which case it may be referred to as a Linear Model (LM) mode.

[0113] In addition, a temporal prediction sample of the current block may be derived based on the filtered neighboring reference samples, and a prediction sample of the current block may be derived by weighting the temporal prediction sample and at least one reference sample derived according to the intra prediction mode among the conventional neighboring reference samples, i.e., non-filtered neighboring reference samples. In this case, it may be called Position Dependent Intra Prediction (PDPC).

[0114] Also, a reference sample line with the highest prediction accuracy may be selected from among multiple reference sample lines surrounding the current block, and a prediction sample may be derived using a reference sample located in a prediction direction from the selected line. At this time, information on the used reference sample line (e.g., intra_luma_ref_idx) may be coded and signaled in a bitstream. In this case, it may be called MRL (multi-reference line intra prediction) or MRL-based intra prediction. If MRL is not applied, a reference sample may be derived from a reference sample line directly adjacent to the current block, and in this case, information on the reference sample line may not be signaled.

[0115] Also, the current block may be divided into vertical or horizontal sub-partitions, and intra prediction may be performed for each sub-partition based on the same intra prediction mode. In this case, the neighboring reference samples for intra prediction may be derived for each sub-partition. That is, in the encoding / decoding order, the restored samples of the previous sub-partition may be used as the neighboring reference samples for the current sub-partition. In this case, the intra prediction mode for the current block is applied to the sub-partitions in the same manner, but the neighboring reference samples may be derived and used for each sub-partition, thereby improving the intra prediction performance in some cases. Such a prediction method may be called ISP (intra sub-partitions) or ISP-based intra prediction.

[0116] The above-mentioned intra prediction techniques may be referred to by various terms such as intra prediction types or additional intra prediction modes, distinguished from directional or non-directional intra prediction modes. For example, the intra prediction techniques (intra prediction types or additional intra prediction modes, etc.) may include at least one of the above-mentioned LIP, LM, PDPC, MRL, and ISP. A general intra prediction method other than the specific intra prediction types such as LIP, LM, PDPC, MRL, and ISP may be referred to as a normal intra prediction type. The normal intra prediction type may be generally applied when the above-mentioned specific intra prediction types are not applied, and prediction may be performed based on the above-mentioned intra prediction modes. Meanwhile, post-processing filtering may be performed on the derived prediction samples as necessary.

[0117] Specifically, the intra prediction procedure may include an intra prediction mode / type determination step, a neighboring reference sample derivation step, and an intra prediction mode / type-based prediction sample derivation step. In addition, a post-processing filtering step may be performed on the derived prediction sample, if necessary.

[0118] Meanwhile, in addition to the above-mentioned intra prediction types, affine linear weighted intra prediction (ALWIP) may be used. The ALWIP may also be called linear weighted intra prediction (LWIP), matrix weighted intra prediction (MWIP), or matrix based intra prediction (MIP). When the ALWIP is applied to a current block, a prediction sample for the current block may be derived by i) using neighboring reference samples subjected to an averaging procedure, ii) performing a matrix-vector-multiplication procedure, and iii) further performing a horizontal / vertical interpolation procedure as necessary. The intra prediction mode used for the ALWIP may be configured to be different from the intra prediction mode (the intra prediction mode described with reference to FIG. 13 and / or FIG. 14) used in the above-mentioned LIP, PDPC, MRL, ISP intra prediction, or normal intra prediction. The intra prediction mode for the ALWIP may be called an ALWIP mode, an LWIP mode, an MWIP mode, or an MIP mode. For example, a matrix and an offset used in the matrix-vector multiplication may be set differently depending on an intra prediction mode for the ALWIP. Here, the matrix may be called an (affine) weighting matrix, and the offset may be called an (affine) offset vector or an (affine) bias vector. A specific ALWIP method will be described later.

[0119] FIG. 6 is a flow chart illustrating an intra-prediction based video / image coding method.

[0120] The encoding method of FIG. 6 may be performed by the image encoding device of FIG. 2. Specifically, step S610 may be performed by the intra prediction unit 185, and step S620 may be performed by the residual processing unit. Specifically, step S620 may be performed by the subtraction unit 115. Step S630 may be performed by the entropy encoding unit 190. The prediction information of step S630 may be derived by the intra prediction unit 185, and the residual information of step S630 may be derived by the residual processing unit. The residual information is information on the residual sample. The residual information may include information on quantized transform coefficients for the residual sample. As described above, the residual sample may be derived as a transform coefficient via the transform unit 120 of the image encoding device, and the transform coefficient may be derived as a quantized transform coefficient via the quantization unit 130. The information on the quantized transform coefficient may be coded in the entropy encoding unit 190 through a residual coding procedure.

[0121] The image encoding apparatus may perform intra prediction on a current block (S610). The image encoding apparatus may determine an intra prediction mode / type for the current block, derive surrounding reference samples for the current block, and then generate predicted samples in the current block based on the intra prediction mode / type and the surrounding reference samples. Here, the steps of determining the intra prediction mode / type, deriving the surrounding reference samples, and generating predicted samples may be performed simultaneously, or one of the steps may be performed prior to the other steps.

[0122] FIG. 7 is a diagram illustrating an example configuration of the intra prediction unit 185 according to the present disclosure.

[0123] As shown in FIG 7, the intra prediction unit 185 of the image encoding device may include an intra prediction mode / type determination unit 186, a reference sample derivation unit 187, and / or a prediction sample derivation unit 188. The intra prediction mode / type determination unit 186 may determine an intra prediction mode / type for the current block. The reference sample derivation unit 187 may derive neighboring reference samples for the current block. The prediction sample derivation unit 188 may derive prediction samples for the current block. Meanwhile, although not shown, if a prediction sample filtering procedure (not shown) is performed, the intra prediction unit 185 may further include a prediction sample filter unit (not shown).

[0124] The image encoding apparatus may determine a mode / type to be applied to the current block from among a plurality of intra prediction modes / types, and may compare rate-distortion costs (RD costs) for the intra prediction modes / types to determine an optimal intra prediction mode / type for the current block.

[0125] Meanwhile, the image coding apparatus may also perform a prediction sample filtering procedure. The prediction sample filtering may be called post-filtering. The prediction sample filtering procedure may filter some or all of the prediction samples. In some cases, the prediction sample filtering procedure may be omitted.

[0126] 6, the image encoding apparatus may generate a residual sample for the current block based on a predicted sample or a filtered predicted sample (S620). The image encoding apparatus may derive the residual sample by subtracting the predicted sample from an original sample of the current block. That is, the image encoding apparatus may derive a residual sample value by subtracting a corresponding predicted sample value from an original sample value.

[0127] The image encoding apparatus may encode image information including information on the intra prediction (prediction information) and residual information on the residual sample (S630). The prediction information may include the intra prediction mode information and / or the intra prediction technique information. The image encoding apparatus may output the encoded image information in a bitstream format. The output bitstream may be transmitted to an image decoding apparatus via a storage medium or a network.

[0128] The residual information may include a residual coding syntax, which will be described later. The image encoding apparatus may transform / quantize the residual samples to derive quantized transform coefficients. The residual information may include information on the quantized transform coefficients.

[0129] Meanwhile, as described above, the image coding apparatus may generate a reconstructed picture (including reconstructed samples and reconstructed blocks). To this end, the image coding apparatus may further inverse quantize / inverse transform the quantized transform coefficients to derive (modified) residual samples. The reason for performing inverse quantization / inverse transform again after transforming / quantizing the residual samples is to derive residual samples identical to the residual samples derived from the image decoding apparatus. The image coding apparatus may generate a reconstructed block including reconstructed samples for the current block based on the predicted samples and the (modified) residual samples. A reconstructed picture for the current picture may be generated based on the reconstructed block. As described above, an in-loop filtering procedure or the like may further be applied to the reconstructed picture.

[0130] FIG. 8 is a flow chart illustrating an intra-prediction based video / image decoding method.

[0131] The image decoding apparatus can perform operations corresponding to those performed by the image encoding apparatus.

[0132] The decoding method of FIG. 8 may be performed by the image decoding apparatus of FIG. 3. Steps S810 to S830 may be performed by the intra prediction unit 265, and the prediction information of step S810 and the residual information of step S840 may be obtained from a bitstream by the entropy decoding unit 210. The residual processing unit of the image decoding apparatus may derive a residual sample for a current block based on the residual information (S840). Specifically, the inverse quantization unit 220 of the residual processing unit may derive a transform coefficient by performing inverse quantization based on the quantized transform coefficient derived based on the residual information, and the inverse transform unit 230 of the residual processing unit may derive a residual sample for the current block by performing inverse transform on the transform coefficient. Step S850 may be performed by the adder 235 or a reconstruction unit.

[0133] Specifically, the image decoding apparatus may derive an intra prediction mode / type for a current block based on received prediction information (intra prediction mode / type information) (S810). The image decoding apparatus may also derive neighboring reference samples for the current block (S820). The image decoding apparatus may generate prediction samples in the current block based on the intra prediction mode / type and the neighboring reference samples (S830). In this case, the image decoding apparatus may perform a prediction sample filtering procedure. The prediction sample filtering may be referred to as post filtering. Some or all of the prediction samples may be filtered by the prediction sample filtering procedure. In some cases, the prediction sample filtering procedure may be omitted.

[0134] The image decoding apparatus may generate a residual sample for the current block based on the received residual information (S840). The image decoding apparatus may generate a reconstructed sample for the current block based on the predicted sample and the residual sample, and derive a reconstructed block including the reconstructed sample (S850). A reconstructed picture for the current picture may be generated based on the reconstructed block. As described above, an in-loop filtering procedure may be further applied based on the reconstructed picture.

[0135] FIG. 9 is a diagram illustrating an example configuration of the intra prediction unit 265 according to the present disclosure.

[0136] As shown in FIG. 9, the intra prediction unit 265 of the image decoding apparatus may include an intra prediction mode / type determination unit 266, a reference sample derivation unit 267, and a prediction sample derivation unit 268. The intra prediction mode / type determination unit 266 determines an intra prediction mode / type for the current block based on intra prediction mode / type information generated and signaled by the intra prediction mode / type determination unit 186 of the image encoding apparatus, and the reference sample derivation unit 266 may derive a neighboring reference sample of the current block from a restored reference area in the current picture. The prediction sample derivation unit 268 may derive a prediction sample of the current block. Meanwhile, although not shown, if the above-mentioned prediction sample filtering procedure is performed, the intra prediction unit 265 may further include a prediction sample filter unit (not shown).

[0137] The intra prediction mode information may include flag information (e.g., intra_luma_mpm_flag) indicating whether a most probable mode (MPM) is applied to the current block or a remaining mode is applied. If the MPM is applied to the current block, the intra prediction mode information may further include index information (e.g., intra_luma_mpm_idx) indicating one of the intra prediction mode candidates (MPM candidates). The intra prediction mode candidates (MPM candidates) may be configured as an MPM candidate list or an MPM list. In addition, if the MPM is not applied to the current block, the intra prediction mode information may further include remaining mode information (e.g., intra_luma_mpm_remainder) indicating one of the remaining intra prediction modes excluding the intra prediction mode candidates (MPM candidates). The image decoding apparatus may determine the intra prediction mode of the current block based on the intra prediction mode information. In addition, a separate MPM list may be configured for the above-mentioned ALWIP. The MPM candidate modes may include intra prediction modes of neighboring blocks (eg, left and upper neighboring blocks) of the current block and additional candidate modes.

[0138] Also, the intra prediction technique may be realized in various forms. As an example, the intra prediction technique information may include index information of intra prediction techniques indicating one of the intra prediction techniques. As another example, the intra prediction technique information may include at least one of reference sample line information (e.g., intra_luma_ref_idx) indicating whether the MRL is applied to the current block and, if so, which reference sample line is used, ISP flag information (e.g., intra_subpartitions_mode_flag) indicating whether the ISP is applied to the current block, ISP type information (e.g., intra_subpartitions_split_flag) indicating a subpartition split type when the ISP is applied, flag information indicating whether PDPC is applied, or flag information indicating whether LIP is applied. In the present disclosure, the ISP flag information may be referred to as an ISP application indicator. Also, the intra prediction type information may include an ALWIP flag indicating whether ALWIP is applied to the current block.

[0139] The intra prediction mode information and / or the intra prediction technique information may be encoded / decoded through a coding method described in the present disclosure. For example, the intra prediction mode information and / or the intra prediction technique information may be encoded / decoded through entropy coding (e.g., CABAC, CAVLC) based on a truncated (rice) binary code.

[0140] The intra-prediction mode / type decision method according to the present disclosure will now be described in more detail.

[0141] When intra prediction is applied to the current block, the intra prediction mode applied to the current block may be determined using the intra prediction mode of the neighboring block. For example, the image decoding apparatus may construct an MPM (most probable mode) list derived based on the intra prediction modes of neighboring blocks (e.g., left and / or upper neighboring blocks) of the current block and additional candidate modes, and may select one of the MPM candidates in the MPM list based on the received MPM index. Alternatively, the image decoding apparatus may select one of the remaining intra prediction modes not included in the MPM list based on the remaining intra prediction mode information. For example, whether the intra prediction mode applied to the current block is among the MPM candidates (i.e., included in the MPM list) or among the remaining modes may be indicated based on an MPM flag (e.g., intra_luma_mpm_flag). A value of 1 of the mpm flag may indicate that the intra prediction mode for the current block is in the mpm candidate (mpm list), and a value of 0 of the mpm flag may indicate that the intra prediction mode for the current block is not in the mpm candidate (mpm list). The mpm index may be signaled in the form of an mpm_idx or intra_luma_mpm_idx syntax element, and the remaining intra prediction mode information may be signaled in the form of a rem_intra_luma_pred_mode or intra_luma_mpm_remainder syntax element. For example, the remaining intra prediction mode information may index the remaining intra prediction modes not included in the mpm candidate (mpm list) among all intra prediction modes in the order of prediction mode numbers, and may indicate one of them. The intra prediction mode may be an intra prediction mode for a luma component (sample).Hereinafter, the intra prediction mode information may include at least one of the mpm flag (e.g., intra_luma_mpm_flag), the mpm index (e.g., mpm_idx or intra_luma_mpm_idx), and the remaining intra prediction mode information (rem_intra_luma_pred_mode or intra_luma_mpm_remainder). In the present disclosure, the MPM list may be referred to by various terms such as an MPM candidate list or a candModeList.

[0142] FIG. 10 is a flowchart showing an intra-prediction mode signaling procedure in an image encoding device.

[0143] 10, the image encoding apparatus may construct an MPM list for a current block (S1010). The MPM list may include candidate intra prediction modes (MPM candidates) that are likely to be applied to the current block. The MPM list may include intra prediction modes of neighboring blocks, and may further include a specific intra prediction mode based on a predetermined method.

[0144] The image encoding apparatus may determine an intra prediction mode of the current block (S1020). The image encoding apparatus may perform prediction based on various intra prediction modes and may perform rate-distortion optimization (RDO) based on the prediction to determine an optimal intra prediction mode. In this case, the image encoding apparatus may determine the optimal intra prediction mode using only MPM candidates included in the MPM list, or may determine the optimal intra prediction mode using not only the MPM candidates included in the MPM list but also the remaining intra prediction modes. Specifically, for example, if the intra prediction type of the current block is a specific type (e.g., LIP, MRL, or ISP) other than a normal intra prediction type, the image encoding apparatus may determine the optimal intra prediction mode using only the MPM candidates. That is, in this case, the intra prediction mode for the current block may be determined only from among the MPM candidates, and in this case, the mpm flag may not be coded / signaled. In the case of the specific type, the image decoding device can infer that the mpm flag is 1 without receiving separate signaling of the mpm flag.

[0145] Meanwhile, in general, when the intra prediction mode of the current block is one of the MPM candidates in the MPM list, the image encoding device may generate an mpm index (mpm idx) that indicates one of the MPM candidates. If the intra prediction mode of the current block is not in the MPM list, the image encoding device may generate remaining intra prediction mode information that indicates the same mode as the intra prediction mode of the current block among the remaining intra prediction modes not included in the MPM list.

[0146] The image encoding apparatus may encode the intra prediction mode information and output it in a bitstream format (S1030). The intra prediction mode information may include the above-mentioned mpm flag, mpm index, and / or remaining intra prediction mode information. In general, the mpm index and the remaining intra prediction mode information are not signaled simultaneously when indicating an intra prediction mode for one block in an alternative relationship. That is, when the mpm flag value is 1, the mpm index may be signaled, and when the mpm flag value is 0, the remaining intra prediction mode information may be signaled. However, as described above, when a specific intra prediction type is applied to the current block, the mpm flag may not be signaled, but its value may be inferred to 1, and only the mpm index may be signaled. That is, in this case, the intra prediction mode information may include only the mpm index.

[0147] In the example shown in FIG. 10, S1020 is illustrated as being performed after S1010, but this is just one example, and S1020 may be performed before S1010 or simultaneously with S1010.

[0148] FIG. 11 is a flowchart showing a procedure for determining an intra-prediction mode in the image decoding apparatus.

[0149] The image decoding apparatus may determine the intra prediction mode of the current block based on the intra prediction mode information determined and signaled by the image encoding apparatus.

[0150] 11, the image decoding apparatus may obtain intra-prediction mode information from a bitstream (S1110). The intra-prediction mode information may include at least one of an mpm flag, an mpm index, and a remaining intra-prediction mode, as described above.

[0151] The image decoding apparatus may configure an MPM list (S1120). The MPM list may be configured in the same manner as the MPM list configured in the image encoding apparatus. That is, the MPM list may include intra-prediction modes of neighboring blocks, and may further include a specific intra-prediction mode according to a predetermined method.

[0152] In the example shown in FIG. 11, S1120 is illustrated as being performed after S1110, but this is just one example, and S1120 may be performed before S1110 or simultaneously with S1110.

[0153] The image decoding apparatus determines an intra prediction mode of the current block based on the MPM list and the intra prediction mode information (S1130). Step S1130 will be described in more detail with reference to FIG.

[0154] FIG. 12 is a flowchart for explaining the intra-prediction mode derivation procedure in more detail.

[0155] Steps S1210 and S1220 in Fig. 12 can respectively correspond to steps S1110 and S1120 in Fig. 11. Therefore, a detailed description of steps S1210 and S1220 will be omitted.

[0156] The image decoding apparatus obtains intra prediction mode information from a bitstream, constructs an MPM list (S1210, S1220), and then determines a predetermined condition (S1230). Specifically, as shown in FIG. 12, when the value of the mpm flag is 1 (Yes in S1230), the image decoding apparatus may derive a candidate indicated by the mpm index among MPM candidates in the MPM list as the intra prediction mode of the current block (S1240). As another example, when the value of the mpm flag is 0 (No in S1230), the image decoding apparatus may derive an intra prediction mode indicated by the remaining intra prediction mode information among the remaining intra prediction modes not included in the MPM list as the intra prediction mode of the current block (S1250). On the other hand, as another example, if the intra prediction type of the current block is a specific type (e.g., LIP, MRL, or ISP, etc.) (Yes in S1230), the image decoding device may derive the candidate indicated by the mpm index in the MPM list as the intra prediction mode of the current block even without checking the mpm flag (S1240).

[0157] FIG. 13 is a diagram illustrating an intra-prediction direction according to an embodiment of the present disclosure.

[0158] For example, the intra prediction modes may include two non-directional intra prediction modes and 33 directional intra prediction modes. The non-directional intra prediction modes may include a planar intra prediction mode and a DC intra prediction mode, and the directional intra prediction modes may include intra prediction modes No. 2 to No. 34. The planar intra prediction mode may be referred to as a planar mode, and the DC intra prediction mode may be referred to as a DC mode.

[0159] Alternatively, in order to capture any edge direction presented in a natural video, the intra prediction modes may include two non-directional intra prediction modes and 65 extended directional intra prediction modes, as shown in Figure 13. The non-directional intra prediction modes may include a planar mode and a DC mode, and the directional intra prediction modes may include intra prediction modes 2 to 66. The extended intra prediction modes may be applied to blocks of all sizes and may be applied to both luma components (luma blocks) and chroma components (chroma blocks).

[0160] Alternatively, the intra prediction modes may include two non-directional intra prediction modes and 129 directional intra prediction modes. The non-directional intra prediction modes may include a planar mode and a DC mode, and the directional intra prediction modes may include intra prediction modes 2 to 130.

[0161] Meanwhile, the intra prediction modes may further include a cross-component linear model (CCLM) mode for chroma samples in addition to the above-mentioned intra prediction modes. The CCLM modes may be divided into L_CCLM, T_CCLM, and LT_CCLM depending on whether the left sample, the top sample, or both are considered for deriving the LM parameters, and may be applied only to the chroma components.

[0162] The intra prediction modes can be indexed, for example, as shown in Table 2 below.

[0163] [Table 2]

[0164] FIG. 14 is a diagram showing intra prediction directions according to another embodiment of the present disclosure. In FIG. 14, the dashed line direction indicates a wide angle mode that is applied only to blocks that are not square. As shown in FIG. 14, in order to capture any edge direction presented in a natural video, the intra prediction modes according to one embodiment may include 93 directional intra prediction modes together with two non-directional intra prediction modes. The non-directional intra prediction modes may include a planner mode and a DC mode. The directional intra prediction modes may include intra prediction modes consisting of No. 2 to No. 80 and No. -1 to No. -14 as indicated by the arrows in FIG. 14. The planner mode may be represented as INTRA_PLANAR, and the DC mode may be represented as INTRA_DC. And, the directional intra prediction modes may be represented as INTRA_ANGULAR-14 to INTRA_ANGULAR-1 and INTRA_ANGULAR2 to INTRA_ANGULAR80.

[0165] On the other hand, as described above, when ALWIP is applied to the current block (e.g., when the value of the LWIP flag or intra_lwip_flag is 1), an MPM list for the ALWIP can be constructed separately, and the MPM flag that can be included in the intra prediction mode information for the ALWIP can be called intra_lwip_mpm_flag, the MPM index can be called intra_lwip_mpm_idx, and the remaining intra prediction mode information can be called intra_lwip_mpm_remainder, respectively.

[0166] Also, various prediction modes may be used for ALWIP, and a matrix and offset for ALWIP may be derived according to an intra prediction mode for ALWIP. As described above, the matrix may be called an (affine) weight matrix, and the offset may be called an (affine) offset vector or an (affine) bias vector. The number of intra prediction modes for ALWIP may be set differently based on the size of the current block. For example, i) if the height and width of the current block (e.g., CB or TB) are each 4, 35 intra prediction modes (i.e., intra prediction modes 0 to 34) may be available, ii) if both the height and width of the current block are 8 or less, 19 intra prediction modes (i.e., intra prediction modes 0 to 18) may be available, and iii) in other cases, 11 intra prediction modes (i.e., intra prediction modes 0 to 10) may be available. For example, when the height and width of the current block are each 4, the block size type is 0, when both the height and width of the current block are 8 or less, the block size type is 1, and otherwise, the block size type is 2. The number of intra prediction modes for ALWIP can be summarized as shown in Table 3. However, this is merely an example, and the block size type and the number of available intra prediction modes can be changed.

[0167] [Table 3]

[0168] Alternatively, the MPM list may be configured to include N MPMs, where N may be 5 or 6.

[0169] To construct the MPM list, three different modes can be considered:

[0170] -Default intra modes

[0171] -Neighbour intra modes

[0172] -Derived intra modes

[0173] For the peripheral intra mode, two peripheral blocks can be considered: a left peripheral block (A) and a top peripheral block (B).

[0174] Also, to construct the MPM list, the following initialized default MPMs may be considered:

[0175] Default 6 MPM modes={A, Planar(0) or DC(1), Vertical(50), HOR(18), VER-4(46), VER+4(54)}

[0176] A pruning process for the two surrounding intra modes may be performed to construct an MPM list. If the two surrounding intra modes are identical to each other and the surrounding intra modes are greater than the DC(1) mode, the MPM list may include the {A, Planar, DC} mode and three derived intra modes. The three derived intra modes may be obtained by adding a predetermined offset value to the surrounding intra modes and / or performing a modulo operation. If the two surrounding intra modes are different from each other, the two surrounding intra modes may be assigned to a first MPM mode and a second MPM mode, and the remaining four MPM modes may be derived from a default mode and / or a surrounding intra mode. In the process of generating the MPM list, a pruning process may be performed to prevent the same modes from being duplicated in the MPM list. A truncated binary code (TBC) may be used for entropy coding of modes other than the MPM modes.

[0177] The above-mentioned MPM list construction method can be used when ALWIP is not applied to the current block. For example, the above-mentioned MPM list construction method can be used to derive an intra prediction mode used in LIP, PDPC, MRL, ISP intra prediction, or normal intra prediction. Meanwhile, the left peripheral block or the upper peripheral block can be coded based on the above-mentioned ALWIP. That is, ALWIP can be applied when coding the left peripheral block or the upper peripheral block. In this case, it is not appropriate to use the ALWIP intra prediction mode number of the peripheral block (left peripheral block / upper peripheral block) to which ALWIP is applied in the MPM list for the current block to which ALWIP is not applied. Therefore, in this case, as an example, the intra prediction mode of the peripheral block (left peripheral block / upper peripheral block) to which ALWIP is applied can be regarded as DC or planner mode. That is, when constructing the MPM list of the current block, the intra prediction mode of the peripheral block coded with ALWIP can be replaced with DC or planner mode. Alternatively, as another example, the intra prediction mode of the neighboring block (left neighboring block / upper neighboring block) to which ALWIP is applied may be mapped to a general intra prediction mode based on a mapping table and used to configure the MPM list of the current block. In this case, the mapping may be performed based on the block size type of the current block. For example, the mapping table may be shown as shown in Table 4.

[0178] [Table 4]

[0179] In Table 4, ALWIP IntraPredMode indicates the ALWIP intra prediction mode of a neighboring block (left neighboring block / upper neighboring block), and block size type (sizeId) indicates the block size type of a neighboring block or a current block. The numbers below the block size type values ​​0, 1, and 2 indicate the general intra prediction mode to which the ALWIP intra prediction mode is mapped for each block size type. For example, if the block size type of the current block is 0 and the ALWIP intra prediction mode number of the neighboring block is 10, the general intra prediction mode number to which the ALWIP intra prediction mode is mapped may be 18. However, the above mapping relationship is merely an example and may be changed.

[0180] Meanwhile, when ALWIP is applied to the current block, an MPM list for the current block to which the ALWIP is applied can be configured separately. The MPM list can be called by various names such as an ALWIP MPM list (or an LWIP MPM list, candLwipModeList) in order to distinguish it from an MPM list when ALWIP is not applied to the current block. Hereinafter, it will be referred to as an ALWIP MPM list for distinction, but it may also be simply called an MPM list.

[0181] The ALWIP MPM list may include n candidates, where n may be 3, for example. The ALWIP MPM list may be constructed based on a left neighboring block and an upper neighboring block of the current block. Here, the left neighboring block may indicate the uppermost block among the neighboring blocks adjacent to the left boundary of the current block. Also, the upper neighboring block may indicate the leftmost block among the neighboring blocks adjacent to the upper boundary of the current block.

[0182] For example, if ALWIP is applied to the left peripheral block, the first candidate intra prediction mode (or candLwipModeA) may be set to be the same as the ALWIP intra prediction mode of the left peripheral block. Also, for example, if ALWIP is applied to the upper peripheral block, the second candidate intra prediction mode (or candLwipModeB) may be set to be the same as the ALWIP intra prediction mode of the upper peripheral block. Meanwhile, the left peripheral block or the upper peripheral block may be coded based on intra prediction other than ALWIP. That is, when coding the left peripheral block or the upper peripheral block, another intra prediction type other than ALWIP may be applied. In this case, it is not appropriate to use the general intra prediction mode number of the peripheral block (left peripheral block / upper peripheral block) to which ALWIP is not applied as a candidate intra mode for the current block to which ALWIP is applied. Therefore, in this case, as an example, the ALWIP intra prediction mode of the peripheral block (left peripheral block / upper peripheral block) to which ALWIP is not applied may be regarded as an ALWIP intra prediction mode of a specific value (e.g., 0, 1, or 2, etc.). Or, as another example, the general intra prediction mode of the peripheral block (left peripheral block / upper peripheral block) to which ALWIP is not applied may be mapped to the ALWIP intra prediction mode based on a mapping table and used to configure the ALWIP MPM list. In this case, the mapping may be performed based on the block size type of the current block. For example, the mapping table may be shown as shown in Table 5.

[0183] [Table 5]

[0184] In Table 5, IntraPredModeY indicates the intra prediction mode of a neighboring block (left neighboring block / upper neighboring block). Here, the intra prediction mode of the neighboring block may be an intra prediction mode for a luma component (sample), i.e., a luma intra prediction mode. Block size type (sizeId) indicates the block size type of a neighboring block or a current block. The numbers below the block size type values ​​0, 1, and 2 indicate the ALWIP intra prediction mode to which the general intra prediction mode is mapped for each block size type. For example, if the block size type of the current block is 0 and the general intra prediction mode of the neighboring block is 10, the ALWIP intra prediction mode number to which the general intra prediction mode is mapped may be 9. However, the above mapping relationship is merely an example and may be changed.

[0185] In addition, the neighboring block (e.g., the left neighboring block / upper neighboring block) may not be available (e.g., located outside the current picture or outside the current tile / tile group / slice), or even if ALWIP is applied to the neighboring block, the ALWIP intra prediction mode of the neighboring block may not be available for the current block depending on the block size type. In this case, a specific ALWIP intra prediction mode predefined for the first candidate and / or the second candidate may be used as the first candidate intra prediction mode or the second candidate intra prediction mode. In addition, a specific ALWIP intra prediction mode predefined for the third candidate may be used as the third candidate intra prediction mode.

[0186] For example, the predefined specific ALWIP intra prediction modes may be shown as in Table 6.

[0187] [Table 6]

[0188] The ALWIP MPM list may be configured based on the first candidate intra prediction mode and the second candidate intra prediction mode. For example, if the first candidate intra prediction mode and the second candidate intra prediction mode are different from each other, the first candidate intra prediction mode may be placed in the 0th candidate (e.g., lwipMpmcand[0]) of the ALWIP MPM list, and the second candidate intra prediction mode may be placed in the 1st candidate (e.g., lwipMpmcand[1]) of the ALWIP MPM list. The second candidate (e.g., lwipMpmcand[2]) of the ALWIP MPM list may use the above-mentioned predefined specific ALWIP intra prediction mode.

[0189] Alternatively, if the first candidate intra prediction mode and the second candidate intra prediction mode are identical to each other, one of the first candidate intra prediction mode and the second candidate intra prediction mode can be entered as the 0th candidate (e.g., lwipMpmcand[0]) of the ALWIP MPM list, and the first candidate (e.g., lwipMpmcand[1]) of the ALWIP MPM list and the second candidate (e.g., lwipMpmcand[2]) of the ALWIP MPM list can use the above-mentioned predefined specific ALWIP intra prediction mode.

[0190] As described above, the ALWIP intra prediction mode of the current block may be derived based on the ALWIP MPM list. In this case, as described above, the MPM flag that may be included in the intra prediction mode information for the ALWIP may be called intra_lwip_mpm_flag, the MPM index may be called intra_lwip_mpm_idx, and the remaining intra prediction mode information may be called intra_lwip_mpm_remainder. The procedure of deriving the ALWIP intra prediction mode from the ALWIP MPM list may be performed as described above with reference to Figures 10 and 11. Alternatively, the ALWIP intra prediction mode of the current block may be directly signaled.

[0191] ALWIP(Affine linear weighted intra prediction)

[0192] The ALWIP according to the present disclosure will be specifically described below.

[0193] ALWIP may also be called MWIP (Matrix weighted intra prediction) or MIP (Matrix based intra prediction). To predict a current block of size W×H by applying ALWIP, one line including H reconstructed neighbouring boundary samples adjacent to the left side of the current block and one line including W reconstructed neighbouring boundary samples adjacent to the top of the current block can be used as input. Unavailable reconstructed neighbouring boundary samples can be replaced with available samples in a manner performed in normal intra prediction. A process of generating a prediction signal by applying ALWIP can include the following three steps.

[0194] First step. Averaging process: By averaging using the surrounding boundary samples, four sample values ​​(when W=H=4) or eight sample values ​​(otherwise) can be derived.

[0195] Second step: Matrix vector multiplication process: By performing a matrix vector multiplication using the averaged sample values ​​as input and adding an offset, a reduced prediction signal for a subsampled set of samples in the original block can be generated.

[0196] Third step: A linear interpolation process can be used to generate prediction signals at the remaining positions by linearly interpolating the prediction signals for the sub-sample sets. The linear interpolation can be a single step linear interpolation in each direction.

[0197] The matrices and offsets required to generate the predicted signal (for a predicted block or a predicted sample) can be obtained from three matrix sets S0, S1, and S2. Set S0 can consist of 18 matrices and 18 offset vectors. In this case, each matrix can consist of 16 rows and 4 columns, and each offset vector can have a size of 16. The matrices and offset vectors of set S0 can be used for blocks of size 4×4.

[0198] Set S1 can consist of 10 matrices and 10 offset vectors, where each matrix consists of 16 rows and 8 columns, and each offset vector can have a size of 16. The matrices and offset vectors in set S1 can be used for blocks of size 4x8, 8x4, and 8x8.

[0199] The set S2 can consist of 6 matrices and 6 offset vectors, where each matrix can consist of 64 rows and 8 columns, and the size of each offset vector can be 64. The matrices and offset vectors in the set S2 can be used for all other types of blocks.

[0200] The total number of multiplications required to compute the matrix-vector product is always less than or equal to 4 × W × H, i.e., a maximum of 4 multiplications per sample are required for the ALWIP mode.

[0201] The ALWIP process for various block types will now be described with reference to Figures 15 to 18. Blocks other than those shown in Figures 15 to 18 can be processed in any one of the methods described with reference to Figures 15 to 18.

[0202] FIG. 15 is a diagram for explaining the ALWIP process for a 4×4 block.

[0203] First, in the averaging step, two average values ​​are obtained along each boundary. That is, two samples of the upper boundary of the current block are selected and averaged to obtain two average values ​​(bdry top ) can be obtained by selecting two samples from the left side of the current block and averaging them to obtain the average value (bdry left ) can be obtained. After that, the four sample values ​​(bdry red ) can be used as input to perform matrix-vector multiplication. k ) can be obtained from the set S0 using the ALWIP mode (mode k). k ) can be added together to produce 16 final predicted samples. In this case, no linear interpolation is required. Thus, a total of (4×16) / (4×4)=4 multiplications can be performed per sample.

[0204] FIG. 16 is a diagram for explaining the ALWIP process for an 8×8 block.

[0205] First, in the averaging step, four average values ​​can be obtained along each boundary. That is, four average values ​​(bdry top ) can be obtained. In addition, four average values ​​(bdry left) can be obtained. After that, the eight sample values ​​(bdry red ) can be used as input to perform matrix-vector multiplication. k ) can be obtained from the set S1 using the ALWIP mode (mode k). k ) are added to the 16 odd-numbered samples in the prediction block (pred red ) can be generated. Thus, a total of (8×16) / (8×8)=2 multiplications can be performed per sample. Finally, pred red samples and the reduced top edge peripheral border samples (bdry red top ) can be used for vertical interpolation. Then, the left side boundary sample (bdry left ) can be used for horizontal interpolation. In this case, since no multiplication operation is required for the interpolation, a total of two multiplications can be performed per sample for ALWIP prediction for an 8x8 block.

[0206] FIG. 17 is a diagram for explaining the ALWIP process for an 8×4 block.

[0207] First, in the averaging step, four average values ​​are obtained along the horizontal boundary. That is, four average values ​​(bdry top ) and the four surrounding boundary samples (bdry left ) can be obtained. After that, the eight sample values ​​(bdry red ) can be used as input to perform matrix-vector multiplication. k ) can be obtained from the set S1 using the ALWIP mode (mode k). k) are added together to produce the samples (pred red ) can be generated. The 16 positions can be odd coordinates horizontally and all coordinates vertically. Therefore, a total of (8 x 16) / (8 x 4) = 4 multiplications can be performed per sample. Finally, pred red The sample and the surrounding boundary sample on the left (bdry left ) for horizontal interpolation. In this case, since no multiplication operations are required for the interpolation, four multiplications can be performed per sample for ALWIP prediction for an 8×4 block.

[0208] The ALWIP process for 4x8 blocks can be the transposed process for 8x4 blocks.

[0209] FIG. 18 is a diagram for explaining the ALWIP process for a 16×16 block.

[0210] First, in the averaging step, four average values ​​can be obtained along the boundary. For example, eight average values ​​can be obtained by selecting and averaging two samples from the peripheral boundary of the current block, or four average values ​​can be obtained by selecting and averaging two of the eight sample values. Alternatively, four samples from the peripheral boundary of the current block can be selected and averaged. Then, the eight sample values ​​(bdry red ) can be used as input to perform matrix-vector multiplication. k ) can be obtained from the set S2 using the ALWIP mode (mode k). k ) are added to the 16 odd-numbered samples in the prediction block (pred red ) can be generated. Thus, a total of (8×64) / (16×16)=2 multiplications can be performed per sample. Finally, predred samples and the reduced top edge peripheral border samples (bdry redII top ) can be used for vertical interpolation. Then, the left side boundary sample (bdry left ) for horizontal interpolation. In this case, since no multiplication operations are required for the interpolation, a total of two multiplications can be performed per sample for ALWIP prediction for a 16×16 block.

[0211] For W×8 blocks (W>8), pred red Since the samples of are at odd coordinates horizontally and at all coordinates vertically, only horizontal interpolation can be performed. red A total of (8×64) / (W×8)=64 / W multiplications per sample may be performed to calculate W. For example, when W is 16, no additional multiplications for linear interpolation may be performed. Also, when W is greater than 16, the number of additional multiplications required per sample for linear interpolation may be less than 2. Thus, the total number of multiplications per sample may be less than or equal to 4.

[0212] For W×4 blocks (W>8), matrix A k can be generated by omitting all rows corresponding to odd entries along the horizontal direction of the reduced block. red contains 32 samples and only horizontal interpolation can be performed. In this case, pred red A total of (8×32) / (W×4)=64 / W multiplications may be performed per sample to compute W. For example, when W is 16, no additional multiplications for linear interpolation may be performed. Also, when W is greater than 16, the number of additional multiplications required per sample for linear interpolation may be less than 2. Thus, the total number of multiplications per sample may be less than or equal to 4.

[0213] The ALWIP process for an 8×H block or a 4×H block can be a transposed process of the process for a W×8 block or a W×4 block.

[0214] The averaging step will now be described in detail.

[0215] FIG. 19 is a diagram for explaining the averaging procedure of the ALWIP process according to the present disclosure.

[0216] The averaging can be applied to the left and / or top boundary of the current block, respectively, where the boundary indicates the surrounding reference samples adjacent to the boundary of the current block, such as the gray samples shown in FIG. left indicates the left neighboring reference sample adjacent to the left boundary of the current block. top indicates the top-nearby reference sample adjacent to the top boundary of the current block.

[0217] If the current block is a 4x4 block, each boundary size may be reduced to two samples based on the averaging process. If the current block is a block other than a 4x4 block, each boundary size may be reduced to four samples based on the averaging process.

[0218] First, the input boundary bdry top and bdry left is the smaller boundary TIFF2024177542000008.tif9120 and It can be reduced to TIFF2024177542000009.tif9133. TIFF2024177542000010.tif8128 and TIFF2024177542000011.tif8118 can consist of two samples in the case of a 4x4 block, and four samples in other cases.

[0219] Specifically, in the case of a 4x4 block, using Equation 1, TIFF2024177542000012.tif8121 can be generated.

[0220]

number

[0221] In the formula 1, i can have a value of 0 or more and less than 2. Also, similarly to the formula 1, TIFF2024177542000014.tif9123 can be generated.

[0222] Instead, the block width W is 4×2. k When this is the case, using Equation 2, TIFF2024177542000015.tif8125 can be generated.

[0223]

number

[0224] In the formula 2, i can have a value of 0 or more and less than 4. Also, similarly to the formula 2, TIFF2024177542000017.tif8125 can be generated.

[0225] The two reduced boundaries thus generated TIFF2024177542000018.tif7126 and TIFF2024177542000019.tif8129 is concatenated and the reduced boundary vector bdry red The reduced boundary vector can have a size of 4 for 4×4 blocks and a size of 8 for other blocks. Equation 3 is expressed as follows based on the mode (ALWIP mode) and the size of the block (W, H). TIFF2024177542000020.tif8121 and Concatenate TIFF2024177542000021.tif8135 and save as bdry red Here's how to generate it:

[0226]

number

[0227] As shown in Equation 3, depending on the size (W, H) of the current block and the ALWIP mode, TIFF2024177542000023.tif8135 and The order of concatenating TIFF2024177542000024.tif8126 can be changed. For example, when the current block is a 4x4 block and the mode is less than 18, bdry red teeth, TIFF2024177542000025.tif8130 or later TIFF2024177542000026.tif8130 or, for example, when the current block is a 4x4 block and mode is 18 or greater, bdry red teeth, TIFF2024177542000027.tif9132 and later TIFF2024177542000028.tif8129, or TIFF2024177542000029.tif8139 and The order of concatenating TIFF2024177542000030.tif8132 may also be determined based on information (e.g., flag information) signaled via the bitstream.

[0228] Finally, in order to interpolate the predicted signal subsampled for large-sized blocks, a second version of the averaged boundary is required. That is, when min(W, H) > 8 and W >= H, W = 8 × 2 l can be represented by, and at this time, the second version of the averaged boundary TIFF2024177542000031.tif8128 can be generated using Equation 4.

[0229]

Number

[0230] In Equation 4, i can have a value of 0 or more and less than 8. Also, when min(W, H) > 8 and W < H, similar to Equation 4, TIFF2024177542000033.tif8148 can be generated.

[0231] Hereinafter, the step of performing a matrix-vector product to generate a reduced predicted signal will be described in detail.

[0232] Using bdry generated in the averaging step red a reduced predicted signal pred red can be generated. The reduced predicted signal pred red is a signal of a downsampled block with a size of W red × H red At this time, W red and H red can be defined as in Equation 5.

[0233]

Number

[0234] The reduced predicted signal pred redcan be generated by matrix vector product and offset addition as shown in Equation 6.

[0235]

number

[0236] In formula 6, A is W red ×H red It can be a matrix with rows and 4 columns (when the current block is a 4x4 block) or 8 columns (otherwise). The offset vector b is of size W red ×H red can be a vector of

[0237] The matrix A and the offset vector b can be obtained from the matrix sets S0, S1, S2 as follows:

[0238] First, the index (idx) may be set to idx(W, H) based on Equation 7. That is, idx may be set based on the width (W) and height (H) of the current block.

[0239]

number

[0240] Also, based on Equation 8, the variable m can be set based on the ALWIP mode and the width (W) and height (H) of the current block.

[0241]

number

[0242] If the index idx is less than or equal to 1, or if the index idx is 2 and min(W,H) is greater than 4, then the matrix A is TIFF2024177542000038.tif7127 and the offset vector b is If the index idx is 2 and min(W,H) is 4, then the matrix A can be determined as follows, if W is 4: TIFF2024177542000040.tif7133 is generated by omitting all rows that correspond to odd x coordinates in the downsampled block, or if H is 4, It can be generated by omitting all rows that correspond to odd y coordinates in the downsampled block in TIFF2024177542000041.tif7132.

[0243] Finally, in the case of Equation 9 below, the downsized prediction signal may have its rows and columns swapped.

[0244]

number

[0245] For W=H=4, the matrix A has 4 columns and 16 rows, so the reduced prediction signal pred red The number of multiplications required to compute is 4. In all other cases, the matrix A has 8 columns and W red ×H red Since it has rows, 8×W red ×H red <= 4 x W x H multiplications are required, i.e. in this case a maximum of 4 multiplications can be performed per sample.

[0246] The linear interpolation step will now be described in detail.

[0247] The interpolation process can be called a linear or bilinear interpolation process. The interpolation process can include two steps of vertical interpolation and horizontal interpolation.

[0248] When W >= H, vertical interpolation can be performed first, and then horizontal interpolation can be performed. When W < H, horizontal interpolation can be performed first, and then vertical interpolation can be performed. In the case of a 4×4 block, the interpolation process can be skipped.

[0249] FIG. 20 is a diagram for explaining the interpolation step of the ALWIP process according to the present disclosure.

[0250] For a W×H block where Max(W, H) >= 8, the prediction signal can be generated by linearly interpolating the downscaled prediction signal pred red (W red ×H red ). Depending on the block form, the linear interpolation can be performed vertically, horizontally, or bidirectionally. When the linear interpolation is performed bidirectionally, the horizontal direction can be performed first when W < H, and the vertical direction can be performed first otherwise. As shown in FIG. 20, for example, in the case of an 8×8 block, vertical interpolation is performed first, and then horizontal interpolation is performed to generate the final prediction signal (pred).

[0251] Hereinafter, for a W×H block where Max(W, H) >= 8 and W >= H, vertical linear interpolation will be described as an example of one-dimensional linear interpolation. However, the following description can also be adaptively applied to horizontal linear interpolation. That is, to avoid duplication of description, only vertical linear interpolation will be described, but the following description can also be applied to horizontal linear interpolation. First, the downscaled prediction signal can be extended to the upper boundary based on the boundary signal. The vertical upsampling factor U ver =H / H red is defined as, then U ver is 2 to the power of u verThe extended reduced prediction signal can be generated by Equation 10.

[0252]

number

[0253] A vertically interpolated prediction signal can be generated from the expanded downscaled prediction signal by performing vertical linear interpolation using Equation 11.

[0254]

number

[0255] The horizontal linear interpolation may be performed in the same manner as the vertical linear interpolation, where the rows and columns and the x and y coordinates may be interchanged, and the extended scaled predicted signal may be the scaled predicted signal extended to the left boundary.

[0256] As described above, the prediction signal of the current block can be finally generated by performing vertical linear interpolation and / or horizontal linear interpolation.

[0257] The conversion / inverse conversion of the residual signal according to the present disclosure will be described in detail below.

[0258] As described above, the image coding apparatus may derive a residual block (residual sample) based on a block (prediction sample) predicted through intra / inter / IBC prediction, etc., and may derive a quantized transform coefficient by applying transform and quantization to the derived residual sample. Information on the quantized transform coefficient (residual information) may be included in a residual coding syntax and output in a bitstream format after encoding. The image decoding apparatus may obtain information on the quantized transform coefficient (residual information) from the bitstream and decode the information to derive a quantized transform coefficient. The image decoding apparatus may derive a residual sample through inverse quantization / inverse transform based on the quantized transform coefficient. As described above, at least one of the quantization / inverse quantization and / or transform / inverse transform may be omitted. When the transform / inverse transform is omitted, the transform coefficient may be referred to as a coefficient or a residual coefficient, or may still be referred to as a transform coefficient for uniformity of expression. Whether or not the transform / inverse transform is omitted may be signaled based on transform_skip_flag.

[0259] The transform / inverse transform may be performed based on a transform kernel. For example, according to the present disclosure, a multiple transform selection (MTS) scheme may be applied. In this case, a part of a plurality of transform kernel sets may be selected and applied to the current block. The transform kernel may be referred to by various terms such as a transform matrix or a transform type. For example, a transform kernel set may indicate a combination of a vertical transform kernel (vertical transform kernel) and a horizontal transform kernel (horizontal transform kernel). For example, MTS index information (e.g., tu_mts_idx syntax element) may be generated / coded in the image coding apparatus and signaled to the image decoding apparatus to indicate any one of the transform kernel sets. For example, the transform kernel set according to the value of the MTS index information may be as shown in Table 7.

[0260] [Table 7]

[0261] In Table 7, tu_mts_idx indicates MTS index information, and trTypeHor and trTypeVer indicate the horizontal and vertical transform kernels, respectively.

[0262] The transform kernel set may also be determined based on, for example, cu_sbt_horizontal_flag and cu__sbt_pos_flag. When cu_sbt_horizontal_flag has a value of 1, it may indicate that the current block is divided horizontally into two transform blocks, and when cu_sbt_horizontal_flag has a value of 0, it may indicate that the current block is divided vertically into two transform blocks. When cu_sbt_pos_flag has a value of 1, it may indicate that tu_cbf_luma, tu_cbf_cb, and tu_cbf_cr for the first transform block of the current block are not present in the bitstream, and when cu_sbt_pos_flag has a value of 0, it may indicate that tu_cbf_luma, tu_cbf_cb, and tu_cbf_cr for the second transform block of the current block are not present in the bitstream. tu_cbf_luma, tu_cbf_cb, and tu_cbf_cr may be syntax elements indicating whether the transform block of the corresponding color component (luma, cb, cr) includes at least one non-zero transform coefficient. For example, tu_cbf_luma may indicate that the corresponding luma transform block includes at least one non-zero transform coefficient when it has a value of 1. As described above, trTypeHor and trTypeVer may be determined according to Table 8 below based on cu_sbt_horizontal_flag and cu__sbt_pos_flag.

[0263] [Table 8]

[0264] In Table 8, for example, when cu_sbt_horizontal_flag is 0 and cu_sbt_pos_flag is 1, trTpeHor and trTypeVer may each be determined to be 1. The transform kernel set may also be determined based on, for example, an intra prediction mode for the current block.

[0265] In the present disclosure, the MTS-based transform may be applied to a primary transform, and a secondary transform may be applied. The secondary transform may be applied only to the coefficients of the upper left w×h region of the coefficient block to which the primary transform is applied, and may be referred to as a reduced secondary transform (RST). For example, the w and / or h may be 4 or 8. In the transform, the primary transform and the secondary transform may be sequentially applied to the residual block, and in the inverse transform, the secondary inverse transform and the primary inverse transform may be sequentially applied to the transform coefficients. The secondary transform (RST transform) may be referred to as a low frequency coefficients transform (LFC transform or LFCT). The secondary inverse transform may be referred to as an inverse LFC transform or inverse LFCT.

[0266] FIG. 21 is a diagram for explaining a conversion method applied to a residual block.

[0267] As shown in FIG. 21, the transform unit 120 of the image encoding device may receive the residual sample, perform a primary transform, generate a transform coefficient (A), and perform a secondary transform to generate a transform coefficient (B). The inverse transform unit 150 of the image encoding device and the inverse transform unit 230 of the image decoding device may receive the transform coefficient (B), perform an inverse secondary transform, generate a transform coefficient (A), and perform an inverse primary transform to generate a residual sample. As described above, the primary transform and the primary inverse transform may be performed based on the MTS. In addition, the secondary transform and the secondary inverse transform may be performed only on the low frequency region (the upper left w×h region of the block).

[0268] The transform / inverse transform may be performed in units of a CU (coding unit) or a TU (transform unit). That is, the transform / inverse transform may be applied to a residual sample in a CU or a residual sample in a TU. The CU size and the TU size may be the same, or there may be a plurality of TUs in a CU domain. Meanwhile, the CU size may generally indicate a luma component (sample) CB (coding block) size. The TU size may generally indicate a luma component (sample) TB (transform block) size. The chroma component (sample) CB or TB size may be derived based on the luma component (sample) CB or TB size according to a component ratio according to a color format (chroma format, for example, 4:4:4, 4:2:2, 4:2:0, etc.). The TU size may be derived based on maxTbSize. In this case, maxTbSize may mean a maximum size that can be transformed. For example, if the CU size is larger than the maxTbSize, a plurality of TUs (TBs) of the maxTbSize may be derived from the CU, and transform / inverse transform may be performed in units of the TUs (TBs). The maxTbSize may be taken into consideration in determining whether various intra prediction types such as ISP are applied. Information on the maxTbSize may be determined in advance, or may be generated and coded in an image coding device and signaled to an image decoding device.

[0269] The secondary / inverse transform will now be described in more detail.

[0270] The secondary transform of the present disclosure may be a mode-dependent non-separable secondary transform (MDNSST). To reduce complexity, MDNSST may be applied only to coefficients in the low frequency region after the primary transform is performed. If both the width (W) and height (H) of the current transform coefficient block are equal to or greater than 8, an 8x8 non-separable secondary transform may be applied to the top left 8x8 region of the current transform coefficient block. Otherwise, if W or H is less than 8, a 4x4 non-separable secondary transform may be applied to the top left min(8,W)×min(8,H) region of the current transform coefficient block. A total of 35x3 non-separable secondary transforms are available for 4x4 and 8x8 blocks. Here, 35 is the number of transform sets specified by the intra prediction mode, and 3 means the number of NSST candidates (candidate kernels) for each intra prediction mode. The mapping relationship between the intra prediction modes and the corresponding transform sets may be as shown in Table 9, for example.

[0271] [Table 9]

[0272] In Table 9, for example, if the intra prediction mode is 0, the transform set for the secondary transform (inverse transform) may be set 0. When the transform set is determined based on the intra prediction mode, it is necessary to identify one of the multiple transform kernels included in the transform set. For this purpose, an index (NSST Idx) may be coded and signaled. If no secondary transform / inverse transform is performed on the current transform block, NSST Idx having a value of 0 may be signaled. Also, MDNSST may not be applied to blocks whose transform is skipped. If NSST Idx having a non-zero value is signaled for the current CU, MDNSST may not be applied to blocks of components whose transform is skipped in the current CU. If transform is skipped for all components in the current CU, or if the number of non-zero coefficients of the transformed block is less than 2, NSST Idx may not be signaled for the current CU. If NSST Idx is not signaled, its value may be inferred to be 0.

[0273] NSST is not applied to the entire block (TU in the case of HEVC) to which the primary transform is applied, but may be applied only to the top-left 8×8 region or 4×4 region. For example, if the size of the block is 8×8 or more, 8×8 NSST may be applied, and if the size is less than 8×8, 4×4 NSST may be applied. In addition, when 8×8 NSST is applied, the block may be divided into 4×4 blocks and 4×4 NSST may be applied to each of them. Both 8×8 NSST and 4×4 NSST follow the configuration of the transform set described above, and since they are non-separable transforms, 8×8 NSST receives 64 data inputs and outputs 64 data, and 4×4 NSST has 16 inputs and 16 outputs.

[0274] In this disclosure, NSST / RT / RST may be referred to as LFNST (low frequency non-seperable transform). LFNST may be applied in a non-separable transform format based on a transform kernel (transform matrix or transform matrix kernel) to low frequency transform coefficients located in the upper left corner region of a transform coefficient block. The NSST index or (R)ST index may be referred to as an LFNST index.

[0275] According to an embodiment of the present disclosure, for a block to which the MIP technique is applied, an index (NSST idx or st_idx syntax) for LFNST can be transmitted in the same manner as in the past. That is, for a current block to which MIP is applied, an index for identifying one of the transform kernels constituting the LFNST transform set can be transmitted.

[0276] According to the present disclosure, since the optimal LFNST kernel can be selected even for an intra-prediction block to which MIP is applied, it is possible to maximize the coding efficiency when both techniques are applied simultaneously. Table 10 shows the syntax of a CU according to an embodiment of the present disclosure.

[0277] [Table 10]

[0278] In Table 10, intra_mip_flag[x0][y0] may indicate that MIP is applied to the luma samples of the current CU when it has a value of 1, and may indicate that MIP is not applied when it has a value of 0. If intra_mip_flag[x0][y0] is not present in the bitstream, its value may be inferred to be 0.

[0279] The syntax elements intra_mip_mpm_flag[x0][y0], intra_mip_mpm_idx[x0][y0], and intra_mip_mpm_remainder[x][y0] in Table 10 can be used to specify the MIP mode for the luma sample. Also, when the top left corner position of the current picture is (0, 0), the coordinate (x0, y0) can be the top left corner position of the luma sample of the current coding block. When intra_mip_mpm_flag[x0][y0] has a value of 1, it can indicate that the MIP mode is derived from intra predicted CUs surrounding the current CU. If intra_mip_mpm_flag[x0][y0] is not present in the bitstream, its value can be inferred to be 1.

[0280] In Table 10, st_idx[x0][y0] may specify a transform kernel (LFNST kernels) applied to the LFNST for the current block. That is, st_idx may indicate any one of the transform kernels included in the LFNST transform set. As described above, the LFNST transform set may be determined based on the intra prediction mode and block size of the current block. In this disclosure, st_idx may also be referred to as lfnst_idx.

[0281] The MIP technique uses a number of MIP modes depending on the block size. For example, when cbWidth and cbHeight indicate the width and height of the current block, a variable (sizeId) that identifies the block size can be derived as follows:

[0282] When cbWidth and cbHeight are both 4, sizeId can be set to 0. Otherwise, when cbWidth and cbHeight are both less than or equal to 8, sizeId can be set to 1. In all other cases, sizeId can be set to 2. For example, when the current block is 16x16, sizeId can be set to 2.

[0283] The number of MIP modes available depending on the sizeId is as shown in Table 11.

[0284] [Table 11]

[0285] That is, the MIP technique can use a minimum of 11 MIP modes and a maximum of 35 MIP modes, whereas the conventional intra-frame prediction can use 67 modes, as shown in FIG.

[0286] In addition, the LFNST technique can determine a transform set (lfnstSetIdx) by referring to Table 12 based on 67 intra prediction modes (lfnstPredModeIntra).

[0287] [Table 12]

[0288] lfnstPredModeIntra in Table 12 is a mode derived based on the intra prediction mode of the current block, and includes the wide-angle mode and CCLM mode described with reference to FIG.

[0289] According to the present disclosure, when the MIP technique is used for the current block, the MIP mode may be converted to a conventional intra prediction mode (the mode described with reference to FIG. 13 and FIG. 14) to determine an index of the transform set of the LFNST. Specifically, the intra prediction mode for determining the index of the transform set may be determined with reference to Table 13 based on the MIP mode and block size (sizeId) of the current block.

[0290] [Table 13]

[0291] In Table 13, MIP mode indicates the MIP mode of the current block, and sizeId indicates the size type of the current block. Also, the numbers under sizeId 0, 1, and 2 indicate the general intra prediction mode (e.g., one of 67 general intra prediction modes) that is mapped to the MIP mode for each block size type. However, the above mapping relationship is merely an example and may be changed.

[0292] For example, if sizeId is 0 and the MIP mode of the current block is 10, the general intra prediction mode number to be mapped may be 18. In this case, for example, lfnstSetIdx has a value of 2 according to Table 12, and an LFNST transform set may be determined based on this. That is, an LFNST transform set having a value of 2 is selected, and among the transform kernels included in the selected transform set, a transform kernel indicated by st_idx (or lfnst_idx) may be used for the secondary transform / inverse transform of the current block.

[0293] FIG. 22 is a flow chart illustrating a method for performing a secondary transform / inverse transform according to the present disclosure.

[0294] The image coding apparatus may perform a secondary transform on the transform coefficients generated by the primary transform in accordance with the order shown in Fig. 22. The image decoding apparatus may perform a secondary inverse transform on the transform coefficients restored from the bitstream in the order shown in Fig. 22.

[0295] First, it may be determined whether LFNST is applied to a current transform block (S2210). The determination of whether LFNST is applied may be performed based on, for example, st_idx or lfnst_idx (NSST idx) restored from a bitstream. If LFNST is not applied, secondary transform / inverse transform may not be performed on the current transform block. If LFNST is applied, it may be determined whether MIP is applied to a current block (S2220). Whether MIP is applied to a current block may be determined using the above-mentioned flag information (e.g., intra_mip_flag). If MIP is applied to a current block, an intra prediction mode for determining an LFNST transform set may be derived (S2230). For example, an intra prediction mode for determining an LFNST transform set may be derived based on the MIP mode. The MIP mode may be restored based on information signaled via a bitstream as described above. The intra prediction mode based on the MIP mode may be derived by a method preset in the image encoding device and the image decoding device. For example, as described with reference to Table 13, step S2230 may be performed using a mapping table between MIP mode and intra prediction mode. However, the above method is not limited to the above method. For example, when MIP is applied, the intra prediction mode (e.g., planner mode) may be derived from a predefined intra prediction mode to determine the LFNST transform set. After step S2230 is performed, the LFNST transform set may be determined based on the derived intra prediction mode (S2240). If the MIP mode is not applied in step S2220, the intra prediction mode of the current block may be used to determine the LFNST transform set (S2240). Step S2240 may correspond to the process of determining lfnstSetIdx described with reference to Table 12. Thereafter, a transform kernel to be used for the secondary transform / inverse transform of the current transform block may be selected from among a plurality of transform kernels included in the LFNST transform set (S2250).The selection of the transform kernel may be performed based on, for example, st_idx or lfnst_idx restored from the bitstream. Finally, the secondary transform / inverse transform may be performed on the current transform block using the selected transform kernel (S2260). The image coding apparatus may determine the optimal mode by comparing rate-distortion costs. Therefore, the image coding apparatus may use the above-mentioned flag information for the determination of step S2210 or step S2220, but is not limited thereto. The image decoding apparatus may make the determination of step S2210 or step S2220 based on information signaled from the image coding apparatus via the bitstream.

[0296] According to the embodiment of the present disclosure described with reference to FIG. 22 , when LFNST is applied to a block to which MIP has been applied, it is possible to derive an intra prediction mode for determining an LFNST transform set, thereby achieving an advantageous effect of performing more efficient LFNST.

[0297] FIG. 23 is a diagram for explaining a method performed in an image decoding device based on whether MIP and LFNST are applied according to another embodiment of the present disclosure.

[0298] According to the embodiment shown in Figure 23, for a block to which the MIP technique is applied, the index (st_idx or lfnst_idx) for LFNST may not be transmitted. That is, if MIP is applied to the current block, the LFNST index is inferred to a value of 0, which may mean that the LFNST technique is not applied to the current block.

[0299] First, it may be determined whether MIP is applied to the current block (S2310). Whether MIP is applied to the current block may be determined using the above-mentioned flag information (e.g., intra_mip_flag). If MIP is applied to the current block, MIP prediction may be performed (S2320), and it may be determined that LFNST is not applied. Therefore, a second inverse transform may not be performed, and a first inverse transform may be performed on the transform coefficients (S2360). Thereafter, the current block may be restored based on a predicted block generated by applying MIP and a residual block generated by inverse transform (S2370). If MIP is not applied to the current block, normal intra prediction may be performed on the current block (S2330). In addition, it may be determined whether LFNST is applied to the current block (S2340). The determination in step S2340 may be made based on st_idx or lfnst_idx (NSST idx) restored from the bitstream. For example, if st_idx is 0, LFNST is not applied, and if st_idx is greater than 0, it may be determined that LFNST is applied. If LFNST is not applied, a secondary inverse transform is not performed on the current transform block, and a primary inverse transform may be performed on the transform coefficients (S2360). Thereafter, the current block may be restored based on a prediction block generated by normal intra prediction and a residual block generated by inverse transform (S2370). If LFNST is applied to the current block, a secondary inverse transform may be performed on the transform coefficients (S2350), and then a primary inverse transform may be performed (S2360). Thereafter, the current block may be restored based on a prediction block generated by normal intra prediction and a residual block generated by inverse transform (S2370). At this time, the secondary inverse transform in step S2350 may be performed based on a LFNST transform set determined based on an intra prediction mode, a transform kernel used for the secondary inverse transform selected based on st_idx, and then the selected transform kernel.

[0300] Table 14 shows the syntax of a CU according to the embodiment shown in FIG.

[0301] [Table 14]

[0302] As shown in Table 14, st_idx can be included in the bitstream only when intra_mip_flag is 0. Therefore, st_idx is not included in the bitstream when intra_mip_flag is 1, i.e., when MIP is applied to the current block. If st_idx does not exist in the bitstream, its value is inferred to 0, and therefore, it can be determined that LFNST is not applied to the current block.

[0303] 23, the LFNST index is not transmitted for a block to which MIP is applied, thereby reducing the amount of bits required to code the index. Also, the image coding device and image decoding device prevent MIP and LFNST from being applied simultaneously, thereby reducing complexity and thus reducing latency.

[0304] FIG. 24 is a diagram for explaining a method performed in an image encoding device based on whether MIP and LFNST are applied according to another embodiment of the present disclosure.

[0305] The encoding method shown in FIG. 24 may correspond to the decoding method shown in FIG.

[0306] First, it may be determined whether MIP is applied to the current block (S2410). Whether MIP is applied to the current block may be determined using the above-mentioned flag information (e.g., intra_mip_flag). However, the image encoding apparatus may perform step S2410 in various ways without being limited thereto. If MIP is applied to the current block, MIP prediction may be performed (S2420), and it may be determined that LFNST is not applied. Therefore, a residual block of the current block may be generated based on a prediction block generated by performing MIP without performing a secondary transformation, and a primary transformation may be performed on the residual block of the current block (S2430). Thereafter, the transform coefficients generated by the transformation may be coded into a bitstream (S2480). If MIP is not applied to the current block, normal intra prediction may be performed on the current block (S2440). A residual block of the current block may be generated based on a prediction block generated by performing normal intra prediction, and a primary transformation may be performed on the generated residual block (S2450). In addition, it may be determined whether LFNST is applied to the current block (S2460). The determination in step S2460 may be made based on st_idx or lfnst_idx (NSST idx). For example, if st_idx is 0, LFNST is not applied, and if st_idx is greater than 0, it may be determined that LFNST is applied. However, without being limited thereto, the image coding apparatus may perform step S2460 in various manners. If LFNST is not applied, transform coefficients generated by the primary transform may be coded into a bitstream without being secondary transformed (S2480). If LFNST is applied to the current block, a secondary transform may be performed on the transform coefficients generated by the primary transform (S2470). The transform coefficients generated by the secondary transform may be coded into a bitstream (S2480).In this case, the secondary transform in step S2470 may be performed based on the selected transform kernel after determining the LFNST transform set based on the intra prediction mode and selecting a transform kernel to be used for the secondary inverse transform. As information on the selected transform kernel, st_idx may be coded and signaled.

[0307] According to another embodiment of the present disclosure, the LFNST index may be derived and used based on a predetermined method without signaling the LFNST index for the block to which MIP is applied. In this case, the secondary transform / inverse transform process may be performed according to the method described with reference to FIG. 22, and the selection of the transform kernel in step S2250 may be performed based on the LFNST index derived by the predetermined method. Alternatively, a separate optimized transform kernel for the block to which MIP is applied may be predefined and used. According to this embodiment, it is possible to have an effect of reducing the amount of bits for encoding the LFNST kernel while selecting the optimal LFNST kernel for the block to which MIP is applied. The LFNST index may be derived based on at least one of a reference line index for intra prediction, an intra prediction mode, a block size, whether MIP is applied, and the like. In addition, in order to select the LFNST transform set, the MIP mode may be converted or mapped to a general intra prediction mode as in the embodiment described with reference to FIG. 22. In this embodiment, since the LFNST index is derived and used without being directly encoded, the syntax of the CU may be the same as that of Table 14.

[0308] According to another embodiment of the present disclosure, the binarization method of the LFNST index can be adaptively performed for blocks to which the MIP technique is applied. More specifically, the number of applicable LFNST transformation kernels can be differently used depending on whether MIP is applied to the current block, thereby selectively changing the binarization method for the LFNST index. For example, one LFNST kernel can be used for blocks to which MIP is applied, and this kernel can be any one of the LFNST kernels applied to blocks to which MIP is not applied. Alternatively, a separate kernel optimized for blocks to which MIP is applied can be defined and used for blocks to which MIP is applied, and this kernel can be different from the LFNST kernel applied to blocks to which MIP is not applied. According to this embodiment, a reduced number of LFNST kernels can be used for blocks to which MIP is applied compared to blocks to which MIP is not applied, thereby reducing the overhead caused by transmitting the LFNST index and achieving a reduction in complexity. For example, as shown in Table 15, the binarization process and cMax value for st_idx can be determined differently depending on the intra_mip_flag value.

[0309] [Table 15]

[0310] According to another embodiment of the present disclosure, another method of transmitting information for LFNST for blocks to which the MIP technique is applied may be provided. In the above example, a single syntax such as st_idx is transmitted as information for LFNST, and when st_idx has a value of 0, it indicates that LFNST is not applied, and when st_idx has a value greater than 0, st_idx indicates a transform kernel to be used for LFNST. That is, it is possible to indicate whether LFNST is applied and the type of transform kernel to be used for LFNST using a single syntax. According to an embodiment of the present disclosure, the information for LFNST may include st_flag, which is a syntax indicating whether LFNST is applied, and st_idx_flag, which is a syntax indicating the type of transform kernel to be used for LFNST when LFNST is applied.

[0311] Table 16 shows the syntax of a CU for another method of transmitting information for LFNST.

[0312] [Table 16]

[0313] As shown in Table 16, information (st_flag) indicating whether LFNST is applied to the current block is signaled, and information (st_idx_flag) indicating the LFNST transformation kernel can be signaled if LFNST is applied to the current block (st_flag is 1).

[0314] Also, similar to the embodiment described with reference to Table 15, a different number of LFNST transform kernels may be used for MIP-applied blocks and non-MIP-applied blocks. For example, only one LFNST transform kernel may be used for MIP-applied blocks. In this case, the transform kernel may be any one of the LFNST transform kernels applied to non-MIP-applied blocks, or may be a separate transform kernel optimized for MIP-applied blocks. In this case, the transmission method of Table 16 may be modified as shown in Table 17.

[0315] [Table 17]

[0316] As shown in Table 17, st_idx_flag can be transmitted only when intra_mip_flag is 0. That is, st_idx_flag can not be transmitted if MIP is applied to the current block.

[0317] st_flag in Tables 16 and 17 is information indicating whether LFNST is applied to the current block, and when not present in the bitstream, it may be inferred to 0. In this disclosure, st_flag may be referred to as lfnst_flag. Also, st_idx_flag may indicate one of two candidate kernels included in the selected LFNST transform set. When st_idx_flag is not present in the bitstream, its value may be inferred to 0. In this disclosure, st_idx_flag may be referred to as lfnst_idx_flag or lfnst_kernel_flag.

[0318] In the examples of Tables 16 and 17, the binarization process of st_flag and st_idx_flag may be as shown in Table 18.

[0319] [Table 18]

[0320] In addition, ctxInc according to the context coded bin of st_flag and st_idx_flag may be as shown in Table 19.

[0321] [Table 19]

[0322] As shown in Table 19, ctxIdx of st_flag can have a value of 0 or 1 when binIdx is 0. For example, ctxInc of st_flag can be derived by Equation 12.

[0323]

number

[0324] As shown in Equation 12, the value of ctxInc used for coding st_flag may be determined differently depending on the value of treetype and / or tu_mts_idx for the current block. Based on the ctxInc, a context model used for coding (CABAC-based) of the st_flag may be derived. Specifically, the context model may be derived by determining a context index (ctxIdx), which may be derived by the sum of variables ctxIdxOffset and ctxInc. In addition, st_idx__flag may be bypass coded / decoded. The bypass coding / decoding may mean coding / decoding an input bin by applying a uniform probability distribution instead of allocating a context.

[0325] According to the example described with reference to Tables 16 and 17, for a block to which MIP is applied, a reduced number of LFNST kernels are used to transmit indexes compared to a block to which MIP is not applied, thereby reducing the overhead and reducing the complexity. Also, as described above, the value of ctxInc used for coding st_flag can be determined differently depending on the treetype and / or tu_mts_idx value for the current block.

[0326] According to another embodiment of the present disclosure, when the binarization method and / or syntax transmission method described with reference to Tables 15 to 19 are used, the LFNST transform kernel can be derived and used. When LFNST is applied to a current block to which MIP is applied, information for selecting an LFNST transform kernel is not signaled, and one of the transform kernels constituting the LFNST transform set can be selected through a derived process, or a separate optimized transform kernel for the block to which MIP is applied can be selected. In this case, an effect of reducing the amount of bits for signaling an optimal LFNST transform kernel for a block to which MIP is applied can be obtained. The selection of the LFNST transform kernel can be performed based on at least one of a reference line index for intra prediction, an intra prediction mode, a block size, and whether MIP is applied. In addition, in order to select an LFNST transform set, the MIP mode can be converted or mapped to a general intra prediction mode as in the embodiment described with reference to FIG. 22.

[0327] Various embodiments according to the present disclosure may be used alone or in combination with other embodiments.

[0328] Although the exemplary method of the present disclosure is expressed as a series of operations for clarity of explanation, this is not intended to limit the order in which the steps are performed, and each step may be performed simultaneously or in a different order, if necessary. To realize the method according to the present disclosure, the steps illustrated may include other steps, some steps may be omitted and the remaining steps may be omitted, or some steps may be omitted and additional steps may be included.

[0329] In the present disclosure, an image encoding device or an image decoding device that performs a predetermined operation (step) can perform the operation (step) to check the execution conditions or circumstances of the operation (step). For example, if it is described that a predetermined operation is performed when a predetermined condition is satisfied, the image encoding device or the image decoding device can perform the predetermined operation after performing an operation to check whether the predetermined condition is satisfied or not.

[0330] The various embodiments of the present disclosure are not intended to enumerate all possible combinations, but are intended to describe representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.

[0331] Additionally, various embodiments of the present disclosure may be implemented using hardware, firmware, software, or a combination thereof, etc. In the case of a hardware implementation, the implementation may be implemented using one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general processors, controllers, microcontrollers, microprocessors, etc.

[0332] In addition, the image decoding device and the image encoding device to which the embodiments of the present disclosure are applied may be included in a multimedia broadcast transmitting / receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as a video communication, a mobile streaming device, a storage medium, a camcorder, a custom video (VoD) service providing device, an over-the-top video (OTT) device, an internet streaming service providing device, a three-dimensional (3D) video device, an image telephone video device, and a medical video device, and may be used to process a video signal or a data signal. For example, the over-the-top video (OTT) device may include a game console, a Blu-ray player, an internet-connected TV, a home theater system, a smartphone, a tablet PC, a digital video recorder (DVR), and the like.

[0333] FIG. 25 is a diagram illustrating a content streaming system to which an embodiment of the present disclosure can be applied.

[0334] As shown in FIG. 25, a content streaming system to which an embodiment of the present disclosure is applied may broadly include an encoding server, a streaming server, a Web server, a media storage, a user device, and a multimedia input device.

[0335] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, camcorder, etc. directly generates a bitstream, the encoding server can be omitted.

[0336] The bitstream may be generated by an image encoding method and / or image encoding device to which an embodiment of the present disclosure is applied, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0337] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server can act as a medium to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server can transmit the multimedia data to the user. In this case, the content streaming system can include a separate control server, and in this case, the control server can control commands / responses between devices in the content streaming system.

[0338] The streaming server may receive the content from a media storage and / or an encoding server. For example, when receiving the content from the encoding server, the content may be received in real time. In this case, the streaming server may store the bitstream for a certain period of time in order to provide a smooth streaming service.

[0339] Examples of the user devices include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head mounted displays (HMDs)), digital TVs, desktop computers, digital signage, and the like.

[0340] Each server in the content streaming system can be operated as a distributed server, in which case data received from each server can be processed in a distributed manner.

[0341] The scope of the present disclosure includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of the various embodiments to be performed on a device or computer, and non-transitory computer-readable medium on which such software or commands can be stored and executed on a device or computer. [Industrial Applicability]

[0342] The embodiments according to the present disclosure can be used to encode / decode images.

Claims

1. An image decoding method performed by an image decoding device, comprising: generating a prediction sample of the current block based on an intra-prediction mode; generating a residual sample of the current block based on an inverse transform; reconstructing the current block based on the predicted sample and the residual sample; The inverse transformation includes a primary inverse transformation and a secondary inverse transformation, In order to determine a transform set of the secondary inverse transform, the intra prediction mode of the current block is derived as a planar mode on the basis that intra prediction for the current block is a matrix based intra prediction (MIP); The method for decoding an image, wherein the secondary inverse transform is performed based on color components of the current block.

2. The image decoding method of claim 1 , wherein the secondary inverse transform is performed only when it is determined that a secondary inverse transform is to be performed on a transform coefficient.

3. The secondary inverse transformation is determining the transform set of a secondary inverse transform based on the intra-prediction mode of the current block; selecting one of a plurality of transform kernels included in the transform set of the inverse secondary transform; and performing the secondary inverse transform based on the selected transform kernel.

4. 2. The image decoding method of claim 1, wherein a transform kernel for a secondary inverse transform of transform coefficients is determined to be a predetermined transform kernel without being signaled via a bitstream based on the intra prediction for the current block being MIP.

5. The image decoding method of claim 1 , wherein a number of transform kernels available when the current block is MIP-ed is less than a number of transform kernels available when the current block is not MIP-ed.

6. First information indicating whether a secondary inverse transform is applied to the current block and second information indicating a transform kernel to be used for the secondary inverse transform are signaled as separate information; The method of claim 1 , wherein the second information is signaled based on the first information indicating that a secondary inverse transform is to be applied to the current block.

7. An image coding method performed by an image coding device, comprising: generating a prediction sample of the current block based on an intra-prediction mode; generating a residual sample of the current block; and generating transform coefficients by performing a transform on the residual samples; The transformation includes a linear transformation and a quadratic transformation, In order to determine the transform set of the secondary transform, the intra prediction mode of the current block is derived as a planar mode on the basis that intra prediction for the current block is a matrix based intra prediction (MIP); The method of claim 1, wherein the secondary transformation is performed based on color components of the current block.

8. A method for transmitting a bitstream generated by an image coding method, comprising the steps of: The image encoding method includes: generating a prediction sample of the current block based on an intra-prediction mode; generating a residual sample of the current block; and generating transform coefficients by performing a transform on the residual samples; The transformation includes a linear transformation and a quadratic transformation, In order to determine the transform set of the secondary transform, the intra prediction mode of the current block is derived as a planar mode on the basis that intra prediction for the current block is a matrix based intra prediction (MIP); The method of claim 1, wherein the secondary transformation is performed based on color components of the current block.

Images