Matrix-based intra-frame prediction device and method

Abstract

According to one embodiment of this document, a method for performing efficient intra prediction can be proposed. In one embodiment, the MIP process can be performed regardless of requirements related to the form of the current block to be coded, and thus signaling is simplified and hardware implementation can have advantages.

Description

Technical Field

[0001] The present document relates to a matrix-based intra-frame prediction device and method. Background Art

[0002] Recently, there is an increasing demand for high-resolution, high-quality images / videos such as 4K or 8K or higher ultra-high definition (UHD) images / videos in various fields. As image / video data has high resolution and high quality, the amount of information or bits to be transmitted increases relative to existing image / video data, and therefore, transmitting image data using a medium such as an existing wired / wireless broadband line or an existing storage medium or storing image / video data using an existing storage medium increases transmission costs and storage costs.

[0003] In addition, interest in and demand for immersive media such as virtual reality (VR) and artificial reality (AR) content and holograms are growing, and broadcasting of images / videos having characteristics different from real images (such as game images) is also growing.

[0004] Therefore, highly efficient image / video compression technology is required to effectively compress, transmit, store or reproduce information of high-resolution, high-quality images / videos having various characteristics as described above.

[0005] Furthermore, in order to improve compression efficiency and enhance subjective / objective visual quality, a matrix-based intra prediction (MIP) process is performed, and a scheme for enhancing data transmission efficiency for the MIP process is discussed. Summary of the invention

[0006] Technical Solution

[0007] According to the embodiments of this document, a method and apparatus for improving image / video coding efficiency are provided.

[0008] According to the embodiments of this document, an efficient intra-frame prediction method and device are provided.

[0009] According to the embodiments of this document, an efficient MIP application method and device are provided.

[0010] According to an embodiment of this document, whether to apply MIP may be adaptively selected according to the type of the current block.

[0011] According to an embodiment of this document, MIP processing may be performed without limiting conditions related to the type of a current block.

[0012] According to an embodiment of this document, the number of MIP modes may be adaptively selected according to the type of the current block.

[0013] According to an embodiment of this document, MIP processing may not be performed on a block having relatively low MIP prediction efficiency.

[0014] According to an embodiment of this document, a smaller number of MIP modes may be used for blocks where the MIP prediction efficiency is relatively low.

[0015] According to an embodiment of this document, a video / image decoding method performed by a decoding device is provided.

[0016] According to an embodiment of this document, a decoding device for performing video / image decoding is provided.

[0017] According to an embodiment of this document, there is provided a video / image encoding method performed by an encoding device.

[0018] According to an embodiment of this document, there is provided an encoding apparatus for performing video / image encoding.

[0019] According to one embodiment of this document, a computer-readable digital storage medium is provided, in which encoded video / image information generated according to the video / image encoding method disclosed in at least one embodiment of the embodiments of this document is stored.

[0020] According to an embodiment of this document, a computer-readable digital storage medium is provided, in which encoded information or encoded video / image information is stored for enabling a decoding device to execute a video / image decoding method disclosed in at least one embodiment of the embodiments of this document.

[0021] Beneficial Effects

[0022] According to the embodiments of this document, the overall image / video compression efficiency can be improved.

[0023] According to the embodiments of this document, subjective / objective visual quality can be enhanced through efficient intra prediction.

[0024] According to the embodiments of this document, MIP processing for image / video coding may be efficiently performed.

[0025] According to an embodiment of this document, MIP processing may not be performed on a block having relatively low MIP prediction efficiency, thereby improving the MIP prediction efficiency.

[0026] According to the embodiments of this document, since a smaller number of MIP modes are used for blocks with relatively low MIP prediction efficiency, the amount of transmission data for the MIP modes can be reduced and the MIP prediction efficiency can be improved.

[0027] According to the embodiments of this document, since MIP processing is performed without limiting conditions related to the type of the current block, signaling can be simplified and there is an advantage in hardware implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 An example of a video / image coding system that can be applied to an embodiment of the present disclosure is schematically illustrated.

[0029] Figure 2 is a diagram schematically explaining a configuration of a video / image encoding device that can be applied to an embodiment of the present disclosure.

[0030] Figure 3 Schematically explains the configuration of a video / image decoding device applicable to an embodiment of the present disclosure.

[0031] Figure 4 An example of an intra-prediction-based image encoding method to which the embodiments of the present disclosure are applicable is schematically illustrated.

[0032] Figure 5 An intra predictor in an encoding device is schematically illustrated.

[0033] Figure 6 An example of an image decoding method based on exemplary intra-frame prediction to which the embodiments of this document are applicable is illustrated.

[0034] Figure 7 An intra predictor in a decoding device is schematically illustrated.

[0035] Figure 8 An example of an intra prediction mode to which embodiments of this document are applicable is illustrated.

[0036] Fig. 9 FIG. 4 illustrates a prediction sample generation process based on MIP according to an embodiment.

[0037] Fig.10 , 11 12 are flowcharts illustrating MIP processing according to an embodiment of this document.

[0038] Fig.13 and Fig.14 An example of a video / image encoding method and related components according to an embodiment of this document is schematically illustrated.

[0039] Fig.15 and Fig.16 An example of a video / image decoding method and related components according to an embodiment of this document is schematically illustrated.

[0040] Fig.17 An example of a content streaming system to which the embodiments disclosed in this document are applicable is illustrated. DETAILED DESCRIPTION

[0041] This document can be modified in various ways and may have various embodiments, and specific embodiments will be illustrated and described in detail in the accompanying drawings. However, this is not intended to limit this document to specific embodiments. The commonly used terms in this specification are used to describe specific embodiments and are not used to limit the technical spirit of this document. Singular expressions include plural expressions unless otherwise clearly expressed in the context. Terms such as "including" or "having" in this specification should be understood to indicate the existence of characteristics, quantities, steps, operations, elements, parts, or combinations thereof described in the specification, rather than excluding the existence or possibility of adding one or more other characteristics, quantities, steps, operations, elements, parts, or combinations thereof.

[0042] At the same time, in order to facilitate the description of different feature functions, the elements in the drawings described in this document are illustrated independently. This does not mean that each element is implemented as separate hardware or separate software. For example, at least two elements can be combined to form a single element, or a single element can be divided into multiple elements. Embodiments in which elements are combined and / or separated are also included in the scope of the rights of this document unless it deviates from the essence of this document.

[0043] Hereinafter, preferred embodiments of the present document will be described in more detail with reference to the accompanying drawings. Hereinafter, in the accompanying drawings, the same reference numerals are used for the same elements, and repeated description of the same elements is omitted.

[0044] This document relates to video / image coding. For example, the methods / embodiments disclosed in this document may be related to the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next generation video / image coding standard after VVC, or other video coding related standards (e.g., the High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), the Basic Video Coding (EVC) standard, and the AVS2 standard).

[0045] This document presents various embodiments of video / image coding, and the embodiments may be performed in conjunction with each other unless otherwise stated.

[0046] In this document, video may mean a collection of a series of images over time. A picture generally means a unit representing an image in a specific time region, and a slice / tile is a unit that constitutes a part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). A picture may consist of one or more slices / tiles. A picture may consist of one or more tile groups. A tile group may include one or more tiles.

[0047] A pixel or picture element (pel) may refer to the smallest unit constituting a picture (or image). In addition, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a value of a pixel, and may represent only a pixel / pixel value of a luminance component, or only a pixel / pixel value of a chrominance component. In addition, a sample may refer to a pixel value in a spatial domain, or, in the case of transforming such a pixel value to a frequency domain, may refer to a transform coefficient in a frequency domain.

[0048] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to the area. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. In some cases, terms such as units, blocks, areas, etc. may be used interchangeably. In general, an M×N block may include a set (or array) of samples (or sample arrays) or transform coefficients consisting of M columns and N rows.

[0049] In this document, the terms " / " and "," are interpreted as indicating "and / or". For example, the expression "A / B" is interpreted as indicating "A and / or B", and "A, B" is interpreted as indicating "A and / or B". In addition, "A / B / C" may mean "at least one of A, B, and / or C". In addition, "A, B, C" may mean "at least one of A, B, and / or C".

[0050] Furthermore, in this document, the term "or" should be interpreted as indicating "and / or". For example, the expression "A or B" may mean 1) only A, 2) only B, and / or 3) both A and B. In other words, the term "or" in this document may mean "additionally or alternatively".

[0051] In the present specification, "at least one of A and B" may mean "only A", "only B", or "both A and B". In addition, in the present specification, the expression "at least one of A or B" or "at least one of A and / or B" may be interpreted as being the same as "at least one of A and B".

[0052] In addition, in the present specification, "at least one of A, B, and C" may mean "only A", "only B", "only C", or "any combination of A, B, and C". In addition, "at least one of A, B, or C" or "at least one of A, B and / or C" may mean "at least one of A, B, and C".

[0053] In addition, brackets used in the present specification may mean "for example". Specifically, in the case of expressing "prediction (intra-frame prediction)", it may be indicated that "intra-frame prediction" is proposed as an example of "prediction". In other words, the term "prediction" in the present specification is not limited to "intra-frame prediction", and it may be indicated that "intra-frame prediction" is proposed as an example of "prediction". In addition, even in the case of expressing "prediction (ie, intra-frame prediction)", it may be indicated that "intra-frame prediction" is proposed as an example of "prediction".

[0054] In this specification, technical features explained separately in one drawing may be implemented separately or may be implemented simultaneously.

[0055] Figure 1 An example of a video / image coding system to which embodiments of this document may be applied is schematically illustrated.

[0056] refer to Figure 1 The video / image coding system may include a source device and a receiving device. The source device may deliver the encoded video / image information or data to the receiving device in the form of a file or stream via a digital storage medium or a network.

[0057] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0058] The video source can obtain the video / image by capturing, synthesizing or generating the process of the video / image. The video source may include a video / image capturing device and / or a video / image generating device. For example, the video / image capturing device may include one or more cameras, a video / image archive including previously captured videos / images, etc. For example, the video / image generating device may include a computer, a tablet computer and a smart phone, and may generate the video / image (electronically). For example, a virtual video / image may be generated by a computer, etc. In this case, the video / image capturing process may be replaced by a process that generates relevant data.

[0059] The encoding device can encode the input video / image. For compression and coding efficiency, the encoding device can perform a series of processes such as prediction, transformation and quantization. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0060] The transmitter may send the encoded image / image information or data output in the form of a bit stream to a receiver of a receiving device in the form of a file or stream through a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for transmission through a broadcast / communication network. The receiver may receive / extract a bit stream and send the received bit stream to a decoding device.

[0061] The decoding device may decode a video / image by performing a series of processes such as dequantization, inverse transformation, and prediction corresponding to the operations of the encoding device.

[0062] The renderer can render the decoded video / image. The rendered video / image can be displayed by a display.

[0063] Figure 2 is a diagram schematically explaining a configuration of a video / image encoding device to which this document is applicable. Hereinafter, a video encoding device may include an image encoding device.

[0064] refer to Figure 2 , the encoding device 200 includes an image segmenter 210, a predictor 220, a residual processor 230 and an entropy encoder 240, an adder 250, a filter 260 and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234 and an inverse transformer 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. According to an embodiment, the image segmenter 210, the predictor 220, the residual processor 230, the entropy encoder 240, the adder 250 and the filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). In addition, the memory 270 may include a decoded picture buffer (DPB), or may be configured by a digital storage medium. The hardware component may also include the memory 270 as an internal / external component.

[0065] The image divider 210 may divide the input image (or picture or frame) input to the encoding device 200 into one or more processors. For example, the processor may be referred to as a coding unit (CU). In this case, the coding unit may be recursively divided from a coding tree unit (CTU) or a maximum coding unit (LCU) according to a quadtree binary tree ternary tree (QTBTTT) structure. For example, a coding unit may be divided into a plurality of coding units of a deeper depth based on a quadtree structure, a binary tree structure, and / or a ternary tree structure. In this case, for example, a quadtree structure may be applied first, and / or a binary tree structure and / or a ternary tree structure may be applied later. Alternatively, a binary tree structure may also be applied first. The coding process according to the present disclosure may be performed based on a final coding unit that is no longer divided. In this case, based on the coding efficiency according to the image feature, the maximum coding unit may be used as the final coding unit, or, if necessary, the coding unit may be recursively divided into coding units of a deeper depth, and the coding unit with the best size may be used as the final coding unit. Here, the coding process may include the prediction, transformation, and reconstruction processes described later. As another example, the processor may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be partitioned or split from the aforementioned final coding unit. The prediction unit may be a unit for sample prediction, and the transform unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from the transform coefficient.

[0066] In some cases, a unit may be used interchangeably with terms such as a block or region. In general, an M×N block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luminance component or only a pixel / pixel value of a chrominance component. A sample may be used as a term corresponding to a pixel or a pixel element of a picture (or image).

[0067] The subtractor 231 can generate a residual signal (residual block, residual sample or residual sample array) by subtracting the prediction signal (prediction block, prediction sample or prediction sample array) output from the predictor 220 from the input image signal (original block, original sample or original sample array), and can send the generated residual signal to the transformer 232. The predictor 220 can perform prediction on the processing target block (hereinafter referred to as the "current block"), and can generate a prediction block including prediction samples for the current block. The predictor 220 can determine whether to apply intra-frame prediction or inter-frame prediction in units of the current block or CU. The predictor can generate and pass various information about the prediction to the entropy encoder 240, such as prediction mode information that will be described later in the explanation of each prediction mode. The information about the prediction can be encoded by the entropy encoder 240 and can be output in the form of a bitstream.

[0068] The intra-frame predictor 222 can predict the current block by referring to samples in the current picture. Depending on the prediction mode, the referenced samples can be located near the current block or can be spaced apart. In intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. For example, the non-directional mode may include a DC mode and a plane mode. For example, depending on the level of detail of the prediction direction, the directional mode may include 33 directional prediction modes or 65 directional prediction modes. However, this is only an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-frame predictor 222 may determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0069] The inter-frame predictor 221 can derive the prediction block of the current block based on the reference block (reference sample array) specified by the motion vector on the reference picture. Here, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information can be predicted in units of blocks, sub-blocks or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be referred to as a collocated reference block, a collocated CU (colCU), etc., and the reference picture including the temporal neighboring block may be referred to as a collocated picture (colPic). For example, the inter-frame predictor 221 may configure a motion information candidate list based on the neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter-frame prediction may be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the inter-frame predictor 221 may use the motion information of the neighboring block as the motion information of the current block. In skip mode, unlike merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the neighboring block may be used as a motion vector predictor, and the motion vector of the current block may be indicated by signaling a motion vector difference.

[0070] The predictor 220 can generate a prediction signal based on various prediction methods described below. For example, the predictor can not only apply intra prediction or inter prediction to predict a block, but also apply intra prediction and inter prediction at the same time. This can be called combined inter and intra prediction (CIIP). In addition, the predictor can perform intra block copy (IBC) for prediction of blocks. Intra block copy can be used for content image / video coding of games, etc., for example, screen content coding (SCC). IBC basically performs prediction in the current picture, but can be performed similarly to inter prediction at the point where the reference block is derived in the current picture. That is, IBC can use at least one of the inter prediction techniques described in this document.

[0071] The prediction signal generated by the inter-frame predictor 221 and / or the intra-frame predictor 222 can be used to generate a reconstruction signal or to generate a residual signal. The transformer 232 can generate a transform coefficient by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, GBT means a transform obtained from a graph when the relationship information between pixels is represented by a graph. CNT means a transform generated based on a prediction signal generated using all previously reconstructed pixels. In addition, the transform process can be applied to square pixel blocks of the same size, or can be applied to blocks of variable size instead of squares.

[0072] The quantizer 233 may quantize the transform coefficients and send them to the entropy encoder 240 and the entropy encoder 240 may encode the quantized signal (information about the quantized transform coefficients) and output a bitstream. Information about the quantized transform coefficients may be referred to as residual information. The quantizer 233 may rearrange the block-type quantized transform coefficients into a one-dimensional vector form based on the coefficient scanning order, and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoder 240 may perform various encoding methods, such as exponential Golomb, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), etc. The entropy encoder 240 may encode information required for video / image reconstruction in addition to the quantized transform coefficients (e.g., the values ​​of syntax elements, etc.) together or separately. The encoding information (e.g., encoded video / image information) may be transmitted or stored in units of NAL (network abstraction layer) in the form of a bitstream. The video / image information may further include information about various parameter sets, such as an adaptive parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. In this document, the information and / or syntax elements that will be described later in this document and transmitted / transmitted with a signal may be encoded by the above-mentioned encoding process and may be included in the bitstream. The bitstream may be transmitted through a network or may be stored in a digital storage medium. The network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) that transmits a signal output from the entropy encoder 240 and / or a storage unit (not shown) that stores the signal may be included as an internal / external element of the encoding device 200, and alternatively, the transmitter may be included in the entropy encoder 240.

[0073] The quantized transform coefficients output from the quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transformation to the quantized transform coefficients by the dequantizer 234 and the inverse transformer 235. The adder 250 adds the reconstructed residual signal to the prediction signal output from the predictor 220 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample or reconstructed sample array). If the block to be processed has no residual, such as the case where the skip mode is applied, the prediction block can be used as a reconstructed block. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current picture, and can be used for inter-frame prediction of the next picture by filtering as described below.

[0074] Meanwhile, luminance mapping and chrominance scaling (LMCS) may be applied during picture encoding and / or reconstruction.

[0075] The filter 260 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 260 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 270 (specifically, the DPB of the memory 270). Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filter 260 can generate various information related to filtering and send the generated information to the entropy encoder 240, as described later in the description of each filtering method. The information related to filtering can be encoded by the entropy encoder 240 and output in the form of a bit stream.

[0076] The modified reconstructed picture transmitted to the memory 270 may be used as a reference picture in the inter predictor 221. When inter prediction is applied by the encoding apparatus, prediction mismatch between the encoding apparatus 200 and the decoding apparatus 300 may be avoided, and encoding efficiency may be improved.

[0077] The DPB of the memory 270 may store the modified reconstructed picture for use as a reference picture in the inter-frame predictor 221. The memory 270 may store the motion information of the block from which the motion information in the current picture is derived (or encoded) and / or the motion information of the block in the reconstructed picture. The stored motion information may be sent to the inter-frame predictor 221 and used as the motion information of the spatial neighboring block or the motion information of the temporal neighboring block. The memory 270 may store the reconstructed samples of the reconstructed blocks in the current picture and may pass the reconstructed samples to the intra-frame predictor 222.

[0078] Figure 3 is a diagram schematically explaining the configuration of a video / image decoding device to which this document is applied.

[0079] refer to Figure 3 , the decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter-frame predictor 331 and an intra-frame predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. According to an embodiment, the entropy decoder 310, the residual processor 320, the predictor 330, the adder 340, and the filter 350 may be configured by a hardware component (e.g., a decoder chipset or a processor). In addition, the memory 360 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium. The hardware component may also include a memory 360 as an internal / external component.

[0080] When a bit stream including video / image information is input, the decoding apparatus 300 can be used with Figure 2 The image is reconstructed correspondingly to the processing of the video / image information in the encoding device. For example, the decoding device 300 can derive the unit / block based on the block partition related information obtained from the bit stream. The decoding device 300 can use the processor applied in the encoding device to perform decoding. Therefore, the decoding processor can be, for example, a coding unit, and the coding unit can be partitioned from the coding tree unit or the largest coding unit according to a quadtree structure, a binary tree structure and / or a ternary tree structure. One or more transform units can be derived from the coding unit. The reconstructed image signal decoded and output by the decoding device 300 can be reproduced by a reproduction device.

[0081] The decoding device 300 may receive the bit stream from Figure 2The received signal may be decoded by the entropy decoder 310. For example, the entropy decoder 310 may parse the bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). The video / image information may further include information about various parameter sets, such as an adaptive parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The decoding device may further decode the picture based on the information about the parameter set and / or the general constraint information. The information and / or syntax elements sent / received by the signal described later in this document may be decoded and obtained from the bitstream by a decoding process. For example, the entropy decoder 310 may decode the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image reconstruction and quantized values ​​of transform coefficients for residuals. More specifically, the CABAC entropy decoding method may receive a bin corresponding to each syntax element in the bitstream, determine a context model using information of the decoding target syntax element, information of the decoding of the neighboring and decoding target blocks, or information of the symbol / bin decoded in the previous stage, and perform arithmetic decoding on the bin by predicting the probability of the bin appearing according to the determined context model, and generate a symbol corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method may update the context model by using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information about prediction among the information decoded by the entropy decoder 310 may be provided to the predictor 330, and information about the residual on which entropy decoding has been performed in the entropy decoder 310, that is, quantized transform coefficients and related parameter information may be input to the dequantizer 321. In addition, information about filtering among the information decoded by the entropy decoder 310 may be provided to the filter 350. Meanwhile, a receiver (not shown) for receiving a signal output from the encoding device may be further configured as an internal / external element of the decoding device 300, or the receiver may be a component of the entropy decoder 310. Meanwhile, the decoding device according to the present invention may be referred to as a video / image / picture decoding device, and the decoding device may be classified into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include an entropy decoder 310, and the sample decoder may include at least one of a dequantizer 321, an inverse transformer 322, a predictor 330, an adder 340, a filter 350, and a memory 360.

[0082] The dequantizer 321 may dequantize the quantized transform coefficient and output the transform coefficient. The dequantizer 321 may rearrange the quantized transform coefficient in the form of a two-dimensional block. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding device. The dequantizer 321 may perform dequantization on the quantized transform coefficient by using a quantization parameter (e.g., quantization step size information) and obtain the transform coefficient.

[0083] The inverse transformer 322 performs an inverse transformation on the transformation coefficients to obtain a residual signal (residual block, residual sample array).

[0084] The predictor may perform prediction on the current block and generate a prediction block including prediction samples for the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on information on prediction output from the entropy decoder 310 and may determine a specific intra / inter prediction mode.

[0085] The predictor can generate a prediction signal based on the various prediction methods described below. For example, the predictor can not only apply intra prediction or inter prediction to predict a block, but also apply intra prediction and inter prediction at the same time. This can be called combined inter and intra prediction (CIIP). In addition, the predictor can perform intra block copying (IBC) for prediction of blocks. Intra block copying can be used for content image / video coding of games, etc., for example, screen content coding (SCC). IBC basically performs prediction in the current picture, but can be performed similar to inter prediction because the reference block is derived in the current picture. That is, IBC can use at least one of the inter prediction techniques described in this document.

[0086] The intra-frame predictor 331 can predict the current block by referring to samples in the current picture. The referenced samples can be located near the current block or can be located at a separate position according to the prediction mode. In intra-frame prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. The intra-frame predictor 331 can determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring block.

[0087] The inter-frame predictor 332 may derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter-frame predictor 332 may configure a motion information candidate list based on neighboring blocks and derive a motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter-frame prediction may be performed based on various prediction modes, and information about the prediction may include information indicating a mode of inter-frame prediction for the current block.

[0088] The adder 340 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictor (330). If the block to be processed has no residual, such as when the skip mode is applied, the prediction block can be used as the reconstructed block.

[0089] The adder 340 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next block to be processed in the current picture, may be output through filtering as described below, or may be used for inter prediction of the next picture.

[0090] At the same time, luminance mapping and chroma scaling (LMCS) can be applied during the picture decoding process.

[0091] The filter 350 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 350 can generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in the memory 360 (specifically, the DPB of the memory 360). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0092] The (modified) reconstructed picture stored in the DPB of the memory 360 can be used as a reference picture in the inter-frame predictor 332. The memory 360 can store the motion information of the block from which the motion information in the current picture is derived (or decoded) and / or the motion information of the block in the reconstructed picture. The stored motion information can be sent to the inter-frame predictor 260 so as to be used as the motion information of the spatial neighboring block or the motion information of the temporal neighboring block. The memory 360 can store the reconstructed samples of the reconstructed blocks in the current picture and pass the reconstructed samples to the intra-frame predictor 331.

[0093] In the present specification, the embodiments explained in the predictor 330, the dequantizer 321, the inverse transformer 322, and the filter 350 of the decoding device 300 can be respectively applied to or correspond to the predictor 220, the dequantizer 234, the inverse transformer 235, and the filter 260 of the encoding device 200 in the same manner.

[0094] As described above, in video coding, prediction is performed to improve compression efficiency. Through this operation, a prediction block including prediction samples for a current block (i.e., a block to be coded) can be generated. Here, the prediction block includes prediction samples in the spatial domain (or pixel domain). The prediction block is derived identically from the encoding device and the decoding device, and the encoding device decodes information (residual information) about the residual between the original block and the prediction block (rather than the original sample value of the original block itself). Image coding efficiency can be improved by notifying the device with a signal. The decoding device can derive a residual block including residual samples based on the residual information, generate a reconstructed block including reconstructed samples by adding the residual block and the prediction block, and generate a reconstructed picture including the reconstructed block.

[0095] The residual information can be generated by transform processing and quantization processing. For example, the encoding device can derive the residual block between the original block and the prediction block, perform transform processing on the residual samples (residual sample array) included in the residual block to derive the transform coefficient, and then derive the quantized transform coefficient by performing quantization processing on the transform coefficient to signal the residual related information (via the bit stream) to the decoding device. Here, the residual information may include value information, position information, transform technology, transform core and quantization parameters of the quantized transform coefficient. The decoding device can perform dequantization / inverse transform processing based on the residual information and derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the prediction block and the residual block. The encoding device can also perform dequantization / inverse transform on the quantized transform coefficients for inter-frame prediction reference of subsequent pictures to derive the residual block, and generate a reconstructed picture based on this.

[0096] In this document, at least one of quantization / dequantization and / or transform / inverse transform may be omitted. When quantization / dequantization is omitted, the quantized transform coefficient may be referred to as a transform coefficient. When transform / inverse transform is omitted, the transform coefficient may be referred to as a coefficient or a residual coefficient, or may still be referred to as a transform coefficient for the sake of uniformity of expression.

[0097] In this document, the quantized transform coefficient and the transform coefficient may be referred to as a transform coefficient and a scaled transform coefficient, respectively. In this case, the residual information may include information about the transform coefficient, and the information about the transform coefficient may be signaled through the residual coding syntax. The transform coefficient may be derived based on the residual information (or information about the transform coefficient), and the scaled transform coefficient may be derived by inversely transforming (scaling) the transform coefficient. The residual sample may be derived based on the inverse transform (transform) of the scaled transform coefficient. This may also be applied / expressed in other parts of this document.

[0098] The predictor of the encoding device / decoding device can derive prediction samples by performing inter-frame prediction in units of blocks. Inter-frame prediction can be a prediction derived in a manner that depends on data elements (e.g., sample values ​​or motion information) of pictures other than the current picture. When inter-frame prediction is applied to the current block, a prediction block (prediction sample array) for the current block can be derived based on a reference block (reference sample array) specified by a motion vector on a reference picture indicated by a reference picture index. Here, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information of the current block can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include inter-frame prediction type (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. A temporally neighboring block may be referred to as a collocated reference block, a collocated CU (colCU), etc., and a reference picture including a temporally neighboring block may be referred to as a collocated picture (colPic). For example, a motion information candidate list may be configured based on a neighboring block of the current block, and a flag or index information indicating which candidate is selected (used) may be signaled to derive a motion vector and / or a reference picture index of the current block. Inter-frame prediction may be performed based on various prediction modes. For example, in the case of a skip mode and a merge mode, the motion information of the current block may be the same as the motion information of the neighboring block. In the skip mode, unlike the merge mode, a residual signal may not be sent. In the case of a motion vector prediction (MVP) mode, the motion vector of the selected neighboring block may be used as a motion vector predictor, and the motion vector of the current block may be signaled. In this case, the motion vector of the current block may be derived using the sum of the motion vector predictor and the motion vector difference.

[0099] According to the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.), the motion information may include L0 motion information and / or L1 motion information. The motion vector in the L0 direction may be referred to as the L0 motion vector or MVL0, and the motion vector in the L1 direction may be referred to as the L1 motion vector or MVL1. Prediction based on the L0 motion vector may be referred to as the L0 prediction, prediction based on the L1 motion vector may be referred to as the L1 prediction, and prediction based on both the L0 motion vector and the L1 motion vector may be referred to as bidirectional prediction. Here, the L0 motion vector may indicate a motion vector associated with the reference picture list L0 (L0), and the L1 motion vector may indicate a motion vector associated with the reference picture list L1 (L1). The reference picture list L0 may include a picture earlier than the current picture in output order as a reference picture, and the reference picture list L1 may include a picture later than the current picture in output order. The preceding picture may be referred to as a forward (reference) picture, and the subsequent picture may be referred to as a backward (reference) picture. The reference picture list L0 may also include a picture later than the current picture in output order as a reference picture. In this case, the previous picture may be indexed first in the reference picture list L0, and the subsequent picture may be indexed later. The reference picture list L1 may also include pictures earlier than the current picture in the output order as reference pictures. In this case, the subsequent picture may be indexed first in the reference picture list 1, and the previous picture may be indexed later. The output order may correspond to a picture order count (POC) order.

[0100] In addition, in the case of performing intra-frame prediction, the correlation between samples can be used, and the difference (ie, residual) between the original block and the predicted block can be obtained. The above-mentioned transformation and quantization can be applied to the residual, and spatial redundancy can be removed thereby. Hereinafter, an encoding method and a decoding method using intra-frame prediction will be described in detail.

[0101] Intra prediction means generating a prediction of a prediction sample for a current block based on a reference sample outside the current block in a picture including the current block (hereinafter referred to as the current picture). Here, the reference sample outside the current block may be referred to as a sample located around the current block. In the case where intra prediction is applied to the current block, neighboring reference samples to be intra-predicted for the current block may be derived.

[0102] For example, in the case where the size (width×height) of the current block is nW×nH, the neighboring reference samples of the current block may include samples adjacent to the left boundary of the current block and a total of 2×nH lower left neighboring samples, samples adjacent to the upper boundary of the current block and a total of 2×nW upper right neighboring samples, and one upper left neighboring sample of the current block. In addition, the neighboring reference samples of the current block may include a plurality of column top neighboring samples and a plurality of row left neighboring samples. In addition, the neighboring reference samples of the current block may include a total of nH samples having a size of nW×nH adjacent to the right boundary of the current block, a total of nW samples adjacent to the bottom boundary of the current block, and one lower right neighboring sample of the current block.

[0103] However, some neighboring reference samples of the current block may not have been decoded yet, or may not be available yet. In this case, the decoding device can configure the neighboring reference samples for prediction by replacing the unavailable samples with available samples. In addition, the decoding device can configure the neighboring reference samples for prediction by interpolating the available samples.

[0104] In the case of deriving neighboring reference samples, (i) the prediction sample may be derived based on an average or interpolation of neighboring reference samples of the current block, or (ii) the prediction sample may be derived based on a reference sample existing in a specific (prediction) direction relative to the prediction sample among neighboring reference samples of the current block. The case of (i) may be applied to the case where the intra prediction mode is a non-directional mode or a non-angular mode, and the case of (ii) may be applied to the case where the intra prediction mode is a directional mode or an angular mode.

[0105] In addition, the prediction samples of the current block among the neighboring reference samples can be generated by interpolating between the first neighboring samples located in the prediction direction of the intra prediction mode of the current block and the second neighboring samples located in the direction opposite to the prediction direction. The above situation can be called linear interpolation intra prediction (LIP). In addition, the chrominance prediction samples can be generated based on the luma samples using a linear model. This situation can be called LM mode.

[0106] In addition, the prediction sample of the current block can be derived by deriving the temporary prediction sample of the current block based on the filtered neighboring reference sample and performing a weighted sum of the temporary prediction sample and at least one reference sample derived according to the intra prediction mode among the existing neighboring reference samples (i.e., the unfiltered neighboring reference sample). The above situation can be called position-dependent intra prediction (PDPC).

[0107] In addition, intra prediction coding may be performed in a method that derives prediction samples using reference samples located in a prediction direction in a reference sample line with the highest prediction accuracy by selecting a corresponding line among neighboring multiple reference sample lines of the current block, and indicates (signals) the reference sample line used at that time to a decoding device. The above situation may be referred to as multi-reference line (MRL) intra prediction or MRL-based intra prediction.

[0108] In addition, when intra prediction is performed based on the same intra prediction mode by dividing the current block into vertical or horizontal sub-partitions, neighboring reference samples can be derived and used in units of sub-partitions. That is, the intra prediction mode for the current block can be equally applied to the sub-partitions, and in this case, since the neighboring reference samples are derived and used in units of sub-partitions, the intra prediction performance can be enhanced in some cases. This prediction method can be referred to as intra sub-partitioning (ISP) or ISP-based intra prediction.

[0109] Unlike the intra prediction mode, the above intra prediction method may be referred to as an intra prediction type. The intra prediction type may be referred to as various terms such as an intra prediction technique or an additional intra prediction mode. For example, the intra prediction type (or additional intra prediction mode) may include at least one of LIP, PDPC, MRL, and ISP as described above. General intra prediction methods other than specific intra prediction types such as LIP, PDPC, MRL, or ISP may be referred to as normal intra prediction types. In the absence of applying the above specific intra prediction type, a normal intra prediction type may generally be applied, and prediction may be performed based on the above intra prediction mode. In addition, the derived prediction samples may be post-filtered as needed.

[0110] In addition to the above intra prediction types, matrix-based intra prediction (hereinafter referred to as MIP) may be used as a method for intra prediction. MIP may be referred to as affine linear weighted intra prediction (ALWIP) or matrix weighted intra prediction (NWIP).

[0111] When MIP is applied to the current block, i) neighboring reference samples on which averaging processing has been performed are used, ii) matrix-vector multiplication processing can be performed, and iii) prediction samples for the current block can be derived by further performing horizontal / vertical interpolation processing as needed. The intra-frame prediction mode used for MIP can be the above-mentioned LIP, PDPC, MRL or ISP intra-frame prediction, but can also be configured differently from the intra-frame prediction mode used in normal intra-frame prediction.

[0112] The intra prediction mode for MIP may be referred to as "affine linear weighted intra prediction mode" or matrix-based intra prediction mode. For example, depending on the intra prediction mode for MIP, the matrix and offset used in the matrix-vector multiplication may be configured differently. Here, the matrix may be referred to as an (affine) weight matrix, and the offset may be referred to as an (affine) offset vector or an (affine) bias vector. In this document, the intra prediction mode for MIP may be referred to as MIP intra prediction mode, linear weighted intra prediction mode, matrix weighted intra prediction mode, or matrix-based intra prediction mode. The detailed MIP method will be described later.

[0113] The following drawings have been prepared to explain the detailed examples of this document. Since the names of the detailed devices described in the drawings, the detailed terms or names (e.g., the names of grammars) are presented as examples, the technical features of this document are not limited to the detailed names used in the following drawings.

[0114] Figure 4 schematically illustrates an example of an intra-prediction-based image encoding method to which the exemplary embodiments of this document are applicable, and Figure 5 An intra predictor in an encoding device is schematically illustrated. Figure 5 The intra-frame predictor in the encoding device shown in FIG. 1 may also be applied to Figure 2 The intra predictor 222 of the encoding device 200 shown in FIG.

[0115] refer to Figure 4 and Figure 5 , S400 may be performed by the intra predictor 222 of the encoding device, and S410 may be performed by the residual processor 230 of the encoding device. Specifically, S410 may be performed by the subtractor 231 of the encoding device. In S420, prediction information may be derived by the intra predictor 222 and encoded by the entropy encoder 240. In S420, residual information may be derived by the residual processor 230 and encoded by the entropy encoder 240. The residual information indicates information about the residual sample. The residual information may include information about the quantized transform coefficient of the residual sample. As described above, the residual sample may be derived by the transform coefficient by the transformer of the encoding device, and the transform coefficient may be derived by the quantized transform coefficient by the quantizer. The information about the quantized transform coefficient may be encoded by the entropy encoder 240 through the residual coding process.

[0116] The encoding device performs intra prediction on the current block (S400). The encoding device may derive an intra prediction mode / type for the current block, derive neighboring reference samples of the current block, and generate prediction samples in the current block based on the intra prediction mode / type and the neighboring reference samples. Here, the processes of determining the intra prediction mode / type, deriving neighboring reference samples, and generating prediction samples may also be performed simultaneously, and any one process may also be performed earlier than the other processes.

[0117] For example, the intra-frame predictor 222 of the encoding device may include an intra-frame prediction mode / type determiner 222-1, a reference sample deriver 222-2, and a prediction sample deriver 222-3, wherein the intra-frame prediction mode / type determiner 222-1 can determine the intra-frame prediction mode / type for the current block, the reference sample deriver 222-2 can derive the neighboring reference samples of the current block, and the prediction sample deriver 222-3 can derive the prediction sample of the current block. At the same time, although not shown, if a prediction sample filtering process is performed, the intra-frame predictor 222 may further include a prediction sample filter (not shown). The encoding device may determine the mode / type applied to the current block among multiple intra-frame prediction modes / types. The encoding device may compare the RD costs for the intra-frame prediction modes / types and determine the best intra-frame prediction mode / type for the current block.

[0118] As described above, the encoding device may also perform a prediction sample filtering process. Prediction sample filtering may be referred to as post-filtering. Some or all prediction samples may be filtered by the prediction sample filtering process. The prediction sample filtering process may be omitted in some cases.

[0119] The encoding apparatus generates residual samples for the current block based on the (filtered) prediction samples (S410). The encoding apparatus may derive the residual samples by comparing the prediction samples with the original samples of the current block based on phases.

[0120] The encoding device may encode image information including information about intra prediction (prediction information) and residual information about residual samples (S420). The prediction information may include intra prediction mode information and intra prediction type information. The residual information may include residual coding syntax. The encoding device may derive quantized transform coefficients by performing transform / quantization of the residual samples. The residual information may include information about the quantized transform coefficients.

[0121] The encoding device can output the encoded image information in the form of a bit stream. The output bit stream can be transmitted to the decoding device via a storage medium or a network.

[0122] As described above, the encoding device can generate a reconstructed picture (including reconstructed samples and reconstructed blocks). To this end, the encoding device can derive (modified) residual samples by performing dequantization / inverse transformation of the quantized transform coefficients again. The reason why the residual samples are dequantized / inverse transformed again after being transformed / quantized is to derive the same residual samples as the residual samples derived by the decoding device as described above. The encoding device can generate a reconstructed block including reconstructed samples for the current block based on the predicted samples and the (modified) residual samples. Based on the reconstructed block, a reconstructed picture for the current picture can be generated. As described above, the loop filtering process can be further applied to the reconstructed picture.

[0123] Figure 6 schematically illustrates an example of an intra-prediction-based image decoding method to which the exemplary embodiments of this document are applicable, and Figure 7 An intra predictor in a decoding device is schematically illustrated. Figure 7 The intra-frame predictor in the illustrated decoding device can also be applied to the same or corresponding one. Figure 2 The intra predictor 331 of the decoding device 300 shown in FIG.

[0124] refer to Figure 6 and Figure 7 , the decoding device may perform operations corresponding to the aforementioned operations performed by the encoding device. S600 to S620 may be performed by the intra-frame predictor 331 of the decoding device, and the prediction information in S600 and the residual information in S630 may be obtained from the bit stream by the entropy decoder 310 of the decoding device. The residual processor 320 of the decoding device may derive residual samples for the current block based on the residual information. Specifically, the dequantizer 321 of the residual processor 320 may derive transform coefficients by performing dequantization based on the quantized transform coefficients derived based on the residual information, and the inverter 322 of the residual processor may derive residual samples for the current block by inversely transforming the transform coefficients. S640 may be performed by the adder 340 or the reconstructor of the decoding device.

[0125] The decoding device may derive an intra prediction mode / type for the current block based on the received prediction information (intra prediction mode / type information) (S600). The decoding device may derive neighboring reference samples of the current block (S610). The decoding device generates prediction samples in the current block based on the intra prediction mode / type and the neighboring reference samples (S620). In this case, the decoding device may perform a prediction sample filtering process. Prediction sample filtering may be referred to as post filtering. Some or all prediction samples may be filtered through the prediction sample filtering process. The prediction sample filtering process may be omitted in some cases.

[0126] The decoding device generates a residual sample for the current block based on the received residual information (S630). The decoding device may generate a reconstructed sample for the current block based on the predicted sample and the residual sample, and derive a reconstructed block including the reconstructed sample (S640). A reconstructed picture for the current picture may be generated based on the reconstructed block. As described above, a loop filtering process, etc. may be further applied to the reconstructed picture.

[0127] Here, the intra-frame predictor 331 of the decoding device may include an intra-frame prediction mode / type determiner 331-1, a reference sample deriver 331-2, and a prediction sample deriver 331-3, wherein the intra-frame prediction mode / type determiner 331-1 may determine the intra-frame prediction mode / type for the current block based on the intra-frame prediction mode / type information acquired by the entropy decoder 310, the reference sample deriver 331-2 may derive the neighboring reference samples of the current block, and the prediction sample deriver 331-3 may derive the prediction sample of the current block. Meanwhile, although not illustrated, if the aforementioned prediction sample filtering process is performed, the intra-frame predictor 331 may further include a prediction sample filter (not illustrated).

[0128] For example, the intra prediction mode information may include, for example, flag information (e.g., intra_luma_mpm_flag), which indicates whether the most probable mode (MPM) is applied to the current block or whether the residual mode is applied to the current block. At this time, if the MPM is applied to the current block, the prediction mode information may also include index information (e.g., intra_luma_mpm_idx) indicating one of the intra prediction mode candidates (MPM candidates). The intra prediction mode candidate (MPM candidate) may be composed of an MPM candidate list or an MPM list. In addition, if the MPM is not applied to the current block, the intra prediction mode information may also include residual mode information (e.g., intra_luma_mpm_remainder) indicating one of the remaining intra prediction modes other than the intra prediction mode candidate (MPM candidate). The decoding device may determine the intra prediction mode of the current block based on the intra prediction mode information.

[0129] In addition, the intra prediction type information may be implemented in various forms. As an example, the intra prediction type information may include intra prediction type index information indicating one of the intra prediction types. As another example, the intra prediction type information may include at least one of the following items: reference sample line information indicating whether MRL is applied to the current block and which reference sample line is used if MRL is applied (e.g., intra_luma_ref_idx), ISP flag information indicating whether ISP is applied to the current block (e.g., intra_subpartitions_mode_flag), ISP type information indicating the segmentation type of the sub-partition when ISP is applied (e.g., intra_subpartitions_split_flag), flag information indicating whether PDCP is applied, or flag information indicating whether LIP is applied. In addition, the intra prediction type information may include a MIP flag indicating whether MIP is applied to the current block.

[0130] The above intra-frame prediction mode information and / or intra-frame prediction type information can be encoded / decoded by the coding method described in this document. For example, the above intra-frame prediction mode information and / or intra-frame prediction type information can be encoded / decoded by entropy coding based on truncated (Rice) binary code (e.g., CABAC, CAVLC).

[0131] In addition, in the case of applying intra prediction, the intra prediction mode of the neighboring block can be used to determine the intra prediction mode applied to the current block. For example, the decoding device can select one of the mpm candidates in the mpm list derived from the intra prediction mode of the neighboring blocks (e.g., the left and / or top neighboring blocks) of the current block and the additional candidate mode based on the received most likely mode (mpm) index, or can select one of the remaining intra prediction modes not included in the mpm candidates (and the plane mode) based on the remaining intra prediction mode information. The mpm list can be constructed to include or exclude the plane mode as a candidate. For example, if the mpm list includes the plane mode as a candidate, the mpm list can have 6 candidates, and if the mpm list does not include the plane mode as a candidate, the mpm list can have 5 candidates. If the mpm list does not include the plane mode as a candidate, a non-planar flag (e.g., intra_luma_not_planar_flag) indicating whether the intra prediction mode of the current block is not a plane mode can be signaled. For example, the mpm flag can be first signaled, and the mpm index and the non-planar flag can be signaled when the value of the mpm flag is 1. In addition, the mpm index can be signaled when the value of the non-planar flag is 1. Here, constructing the mpm list that does not include the planar mode as a candidate first identifies whether the intra-frame prediction mode is the planar mode by first signaling a flag (non-planar flag), because the planar mode is always considered to be an mpm rather than a planar mode that is not an mpm.

[0132] For example, it can be indicated based on an mpm flag (e.g., intra_luma_mpm_flag) whether the intra prediction mode applied to the current block is in an mpm candidate (and planar mode) or in a residual mode. An mpm flag value of 1 can indicate that the intra prediction mode for the current block is in the mpm candidate (and planar mode), and an mpm flag value of 0 can indicate that the intra prediction mode for the current block is not in the mpm candidate (and planar mode). A non-planar flag (e.g., intra_luma_not_planar_flag) value of 0 can indicate that the intra prediction mode for the current block is a planar mode, and a non-planar flag value of 1 can indicate that the intra prediction mode for the current block is not a planar mode. The mpm index can be signaled in the form of an mpm_idx or intra_luma_mpm_idx syntax element, and the remaining intra prediction mode information can be signaled in the form of a rem_intra_luma_pred_mode or intra_luma_mpm_remainder syntax element. For example, the remaining intra prediction mode information may index the remaining intra prediction modes not included in the mpm candidates (and planar modes) among the entire intra prediction modes in the order of their prediction mode numbers, and may indicate one of them. The intra prediction mode may be an intra prediction mode for a luminance component (sample). In the following, the intra prediction mode information may include at least one of an mpm flag (e.g., intra_luma_mpm_flag), a non-planar flag (e.g., intra_luma_not_planar_flag), an mpm index (e.g., mpm_idx or intra_luma_mpm_idx), and remaining intra prediction mode information (rem_intra_luma_pred_mode or intra_luma_mpm_remainder). In this document, the mpm list may be referred to as various terms such as an mpm candidate list, a candidate mode list (candModelList), and a candidate intra prediction mode list.

[0133] Typically, when blocks of an image are segmented, the current block and the neighboring blocks to be coded have similar image properties. Therefore, the current block and the neighboring blocks are likely to have the same or similar intra-frame prediction modes. Therefore, the encoder can encode the intra-frame prediction mode of the current block using the intra-frame prediction mode of the neighboring blocks. For example, the encoder / decoder can form a most probable mode (MPM) list for the current block. The MPM list may also be referred to as an MPM candidate list. Here, MPM may mean a mode used to improve coding efficiency by taking into account the similarity between the current block and the neighboring blocks when coding the intra-frame prediction mode.

[0134] Figure 8An example of an intra prediction mode to which embodiments of this document are applicable is illustrated.

[0135] refer to Figure 8 , the mode can be divided into an intra prediction mode with horizontal directivity and an intra prediction mode with vertical directivity around the 34th intra prediction mode of the upper left diagonal prediction direction. Figure 8 , H and V mean horizontal directivity and vertical directivity, respectively. Each of the numbers -32 to 32 indicates a displacement of 1 / 32 units on a sample grid position. Intra-prediction modes No. 2 to 33 have horizontal directivity, and intra-prediction modes No. 34 to 66 have vertical directivity. Intra-prediction mode No. 18 and intra-prediction mode No. 50 indicate a horizontal intra-prediction mode and a vertical intra-prediction mode, respectively. Intra-prediction mode No. 2 may be referred to as a lower left diagonal intra-prediction mode, intra-prediction mode No. 34 may be referred to as an upper left diagonal intra-prediction mode, and intra-prediction mode No. 66 may be referred to as an upper right diagonal intra-prediction mode.

[0136] In addition, the intra prediction mode used in the above-mentioned MIP is not an existing directional mode, but can indicate a matrix and offset for intra prediction. That is, the matrix and offset for intra prediction can be derived by the intra mode for MIP. In this case, when deriving the intra mode for generating the above-mentioned typical intra prediction or MPM list, the intra prediction mode of the block predicted by the MIP can be configured as a pre-configured mode (e.g., plane mode or DC mode). In addition, according to another example, the intra mode of the MIP can be mapped to a plane mode, a DC mode, or a directional intra mode based on the block size.

[0137] Hereinafter, as one method for intra prediction, matrix-based intra prediction (hereinafter, MIP) will be described.

[0138] As described above, matrix-based intra prediction may be referred to as affine linear weighted intra prediction (ALWIP) or matrix weighted intra prediction (MWIP). In order to predict samples of a rectangular block having a width W and a height H, MIP uses one H line among the reconstructed adjacent left boundary samples of the block and one W line among the reconstructed adjacent top boundary samples of the block as input values. If the reconstructed samples are not available, reference samples may be generated by an interpolation method that has been applied in typical intra prediction.

[0139] Fig. 9 2 is a diagram for explaining a process of generating a MIP-based prediction sample according to an embodiment. Fig. 9 , the MIP processing will be described as follows.

[0140] 1. Averaging

[0141] Through the averaging process, 4 boundary samples can be extracted in the case of W=H=4, and 8 boundary samples can be extracted in other cases.

[0142] 2. Matrix-vector multiplication processing

[0143] A matrix-vector multiplication is performed with the input of the average sample, followed by the addition of an offset. By doing so, a reduced prediction sample for the subsampled set of samples in the original block can be derived.

[0144] 3. (Linear) interpolation processing

[0145] The prediction samples for the remaining positions are generated from the prediction samples of the subsampled sample set by linear interpolation with a single step size in each direction.

[0146] The matrices and offset vectors required to generate a prediction block or prediction sample may be selected from three sets S0, S1 and S2 for matrices.

[0147] The set S0 can be composed of 18 matrices A0 i ,i∈{0,…,17} and 18 offset vectors b0 i ,i∈{0,…,17}. For example, each matrix A0 i ,i∈{0,…,17} can have 16 rows and 4 columns. In the example, each offset vector b0 i , i∈{0,…,17} can be of size 16. The matrices and offset vectors of set S0 can be used for blocks of size 4×4.

[0148] The set S1 can be composed of 10 matrices A1 i ,i∈{0,…,9} and 10 offset vectors b1 i ,i∈{0,…,9}. For example, each matrix A1 i , i∈{0,…,9} can have 16 rows and 8 columns. In the example, each offset vector b1 i , i∈{0,…,9} can be of size 16. The matrices and offset vectors of set S1 can be used for blocks of size 4×8, 8×4, and 8×8.

[0149] Finally, the set S2 can be composed of 6 matrices A2 i ,i∈{0,…,5} and 6 offset vectors b2 i ,i∈{0,…,5}. For example, each matrix A2 i , i∈{0,…,5} can have 64 rows and 8 columns. In the example, each offset vector b2 i, the size of i∈{0,…,5} may be 64. The matrices and offset vectors of set S2 or some of them may be used for all other block types of sizes to which sets S0 and S1 are not applicable.

[0150] The total number of multiplications required to compute the matrix-vector product may always be equal to or less than 4 × W × H. For example, in MIP mode, up to four multiplications may be required per sample.

[0151] Regarding the embodiment of averaging neighboring samples, outside the boundary samples, in the case of W=H=4, four samples can be extracted by averaging, and 8 samples can be extracted by averaging. left and bdry top It can be reduced to a smaller boundary by averaging neighboring boundary samples according to a predefined rule that depends on the block size. and

[0152] Since the two reduction boundaries and Connect to the reduced boundary vector bdry red , so in the case of 4×4 type blocks, bdry red The size of is 4, and in the case of all other blocks its size is 8.

[0153] In the case where the "mode" is referred to as MIP mode, the reduction boundary vector bdry red The range of MIP mode values ​​(mode) can be defined in the following formula.

[0154] [Formula 1]

[0155]

[0156] Regarding the embodiment of matrix-vector multiplication, the matrix-vector multiplication can be performed using the average sample as input. The reduced input vector bdry red One of them generates a reduced prediction signal pred red The prediction samples are for a width of W red And the height is H red The signal of the downsampling block. For example, W red and H red It can be defined in the following formula.

[0157] [Formula 2]

[0158]

[0159]

[0160] Reduce the prediction sample predred It can be calculated by adding the offset after performing the matrix-vector multiplication and can be derived as follows.

[0161] [Formula 3]

[0162] pred red =A.bdry red +b

[0163] Here, A represents the red ×H red A matrix of rows and 4 columns in the case where W and H are 4 (W=H=4), or 8 columns in all other cases, and b represents a matrix of size W red ×H red Vector.

[0164] The matrix A and the offset vector b may be selected from the sets S0, S1, and S2. For example, the index idx=idx(W,H) may be defined as follows.

[0165] [Formula 4]

[0166]

[0167] If idx is equal to or less than 1 (idx≤1) or idx is 2, and the smaller value between W and H is greater than 4 (min(W,H)>4), then set A to And set b to If idx is 2, the smaller value between W and H is 4 (min(W,H) = 4), and W is 4, then A becomes the value corresponding to the odd x coordinate in the downsampled block. The rows of A are removed. In addition, if H is 4, A becomes the matrix corresponding to the odd y coordinates in the downsampled block. The matrix with the columns removed.

[0168] Since A consists of 4 columns and 16 rows when W = H = 4, we calculate pred red The number of multiplications required is 4. In all other cases, since A consists of 8 columns and W red ×H red So we can confirm that each sample requires at most four multiplications to calculate pred red .

[0169] The interpolation process may be referred to as linear interpolation or bilinear interpolation. As shown in the figure, the interpolation process may include two steps: 1) vertical interpolation and 2) horizontal interpolation.

[0170] In the case where W >= H, vertical linear interpolation can be applied first, and then horizontal interpolation can be applied. In the case where W < H, horizontal linear interpolation can be applied first, and then vertical linear interpolation can be applied. In the case of a 4×4 block, the interpolation process can be omitted.

[0171] In the case of a W×H block where max(W,H) ≥ 8, from the reduced prediction samples pred red ×H red on, the prediction samples are derived. Depending on the block type, linear interpolation is performed vertically, horizontally, or in both directions. In the case of applying linear interpolation in both directions, in the case where W < H, linear interpolation is applied first in the horizontal direction, otherwise linear interpolation is applied first in the vertical direction.

[0172] In the case of a W×H block where max(W,H) ≥ 8 and W >= H, it can be considered that there is no general loss. In this case, one-dimensional linear interpolation is performed as follows. If there is no general loss, the linear interpolation in the vertical direction can be fully explained.

[0173] First, the reduced prediction samples are extended to the top by the boundary signal. The vertical upsampling coefficient U ver = H / H red can be defined, and if configured as then the extended reduced prediction samples can be configured as in the following equation.

[0174] For each coding unit (CU) in the intra mode, a flag indicates whether the MIP mode is applied to the corresponding current block. If the MIP mode is applied, the MPM flag can be signaled, and it can be indicated whether the prediction mode is one of the MPM modes. For example, three modes can be considered for MPM. In the example, the MPM modes can be context-coded by truncated binarization. Non-MPM modes can be coded as fixed-length codes (FLC). By performing mode mapping between the normal intra prediction mode and the MIP intra prediction mode based on a predefined mapping table depending on the block size (i.e., idx(W,H) ∈ {0,1,2}), the derivation of this MPM can be combined with the normal intra prediction mode. The following equation can represent the forward (from the normal mode to the MIP mode) and / or reverse (from the MIP mode to the normal mode) mode mapping table.

[0175] [Equation 5]

[0176] predmode ALWIP = map_angular_to_alwip idx [predmode Angular

[0177] [Equation 6]​​​

[0178]

[0179] In an example, 35 modes are available for a block (e.g., max(W, H) <= 8 && W*H < 32). In another example, 19 modes are available for a block (e.g., max(W, H) = 8). 11 modes are available for a block (e.g., max(W, H) > 8). In addition, to reduce memory consumption, two modes can share the same matrix and / or offset vector, as shown in the following equation.

[0180] [Formula 7]

[0181]

[0182] Hereinafter, a method for maximizing performance while reducing the complexity of the MIP technique will be described. The embodiments to be described later may be performed individually or in combination.

[0183] According to an embodiment of the present document, it is possible to adaptively select (determine) whether to apply MIP according to the type of the current (luminance) block in the MIP process. In an existing embodiment, if the difference between the width and height of the luminance block exceeds four times (for example, 32×4, 64×8, 64×4, 4×32, 8×64, 4×64, 128×4, 128×8, 128×16, 4×128, 8×128, and 16×128), MIP is not performed, and / or a flag indicating whether MIP is applied is not sent. In this embodiment, a method for improving encoding and decoding efficiency by extending or removing MIP when MIP is applied is provided.

[0184] In one of the examples of this embodiment, MIP may not be applied when the width of the current (luminance) block is 4 and its height is equal to or greater than 16, or MIP may not be applied when the height of the current (luminance) block is 4 and the width is equal to or greater than 16. Some coding unit syntaxes according to this example may be the same as the following table.

[0185] [Table 1]

[0186]

[0187] In the above example, MIP is not applied to 4×16 and 16×4 blocks where MIP efficiency is low, so the MIP efficiency can be improved.

[0188] In another example among the examples of this embodiment, MIP may not be applied when the width of the current (luminance) block is 4 and its height is equal to or greater than 32, and MIP may not be applied when the height of the current (luminance) block is 4 and its width is equal to or greater than 32. Some coding unit syntaxes according to this example may be as shown in the following table.

[0189] [Table 2]

[0190]

[0191] Through the above-mentioned examples, for 8×64 and 64×8 blocks to which MIP can be effectively and easily applied, MIP can be applied, and thus MIP efficiency can be improved.

[0192] According to another example of this embodiment, the block type restriction for MIP can be removed. For example, in this example, MIP can always be applied to all (compiled) blocks. In the case where the MIP flag is signaled (parsed), the conditional determination of the width and / or height of the current (luminance) block can be removed. Therefore, the MIP parsing conditions can be simplified, and in addition, the complexity of software and hardware implementations can be reduced. Some compilation unit syntaxes according to this example can be shown in the following table.

[0193] [Table 3]

[0194]

[0195] The test results of applying this example can be shown in the following table.

[0196] [Table 4]

[0197]

[0198] Referring to the above table, the results of all intra frames (AI, allintra) and random access (RA, random access) are given in the above test. As shown in the above table, it is confirmed that there is no performance degradation in this embodiment. That is, in the case of the proposed method 3, since the signaling (parsing) burden can be simplified by relaxing the MIP flag signaling (parsing) conditions, it has advantages in hardware implementation and does not cause encoding and decoding performance degradation.

[0199] According to this embodiment, Figure 4 In S400 and / or S410, it may be determined whether to apply MIP to the current block. In this case, as described above, it may be adaptively determined whether to apply MIP according to the type of the current (luminance) block. In this case, in S420, the encoding device may adaptively encode intra_mip_flag.

[0200] In the embodiments of this document, the number of MIP modes can be adaptively determined (selected) according to the type of the current (luminance) block in the MIP process. In the existing embodiment, 4×4 luminance blocks (MipSizeId 0) can use 35 MIP modes, 4×4, 8×4 or 8×8 luminance blocks (MipSizeId 1) can use 19 MIP modes, and other luminance blocks (MipSizeId 2) can use 11 MIP modes.

[0201] In this embodiment, a smaller number of MIP modes can be used for block types with low MIP efficiency. Through this embodiment, the amount of transmission data of information on MIP modes can be reduced.

[0202] Fig.10 , 11 12 are flowcharts illustrating MIP processing according to an embodiment of this document.

[0203] refer to Fig.10 , in the Fig.10 In a related embodiment, if the difference between the width and height of the current (luminance) block exceeds four times the width or height of the current (luminance) block (if the height of the current block is greater than four times the width of the current block, or if the width of the current block is greater than four times the height of the current block), a MIP mode having fewer than 11 basic modes (MipSizeId 2) may be used, and information related to the MIP mode may be sent.

[0204] refer to Fig.11 , in the Fig.11 In a related embodiment, if the width of the current (luminance) block is 4 and its height is equal to or greater than 16, a MIP mode whose number is less than 11 basic modes (MipSizeId 2) may be used, and information related to the MIP mode may be sent. Further, if the height of the current (luminance) block is 4 and its width is equal to or greater than 16, a MIP mode whose number is less than 11 basic modes (MipSizeId 2) may be used, and information related to the MIP mode may be sent.

[0205] refer to Fig.12 , in the Fig.12In a related embodiment, if the width of the current (luminance) block is 4 and its height is equal to or greater than 32, a MIP mode whose number is less than 11 basic modes (MipSizeId 2) may be used, and information related to the MIP mode may be sent. Further, if the height of the current (luminance) block is 4 and its width is equal to or greater than 32, a MIP mode whose number is less than 11 basic modes (MipSizeId 2) may be used, and information related to the MIP mode may be sent.

[0206] By following Figures 10 to 12 In an embodiment, by using a small number of MIP modes for blocks with low MIP prediction efficiency (block types with a large difference between width and height), the amount of transmission data of the MIP mode can be reduced, and thereby, the MIP prediction efficiency can be enhanced.

[0207] In addition, in reference Figures 10 to 12 In the explained embodiment, only a part of the 11 MIP modes may be used, and this is to increase MIP efficiency through data reduction. In an example, only one of the 11 MIP modes may be used, and in this case, MIP mode data may not be transmitted. In another example, only two of the 11 MIP modes may be used, and in this case, 1-bit MIP mode data may be transmitted. In yet another example, only four of the 11 MIP modes may be used, and in this case, 2-bit MIP mode data may be transmitted. One, two, or four of the 11 modes may be selected according to the order of the MIP cores in the existing embodiment, or may be selected according to the order of the MIP cores. Figures 10 to 12 The selection is made in the order of the MIP selection mode probability in blocks as in the method explained above.

[0208] Fig.13 and Fig.14 An example of a video / image encoding method and related components according to an embodiment of this document is schematically illustrated.

[0209] Fig.13 The method disclosed in can be Figure 2 or Fig.14 Specifically, for example, Fig.13 The S1300 and S1310 can be made of Fig.14 The predictor 220 of the encoding device is performed, Fig.13 The S1320 can be Fig.14 The residual processor 230 of the encoding device is used to perform, and Fig.13 The S1330 can be Fig.14 The entropy encoder 240 of the encoding device performs. In addition, although Fig.13Not shown in the figure, the prediction sample or prediction related information can be obtained by Fig.14 The predictor 220 of the encoding device may derive the residual information from the original sample or the predicted sample by the residual processor 230 of the encoding device, and the entropy encoder 240 of the encoding device may generate a bitstream from the residual information or the prediction related information. Fig.13 The methods disclosed in may include the embodiments described above in this document.

[0210] refer to Fig.13 , the encoding device may derive whether matrix-based intra prediction (MIP) is applied to the current block (S1300). If the MIP is applied to the current block, the encoding device may generate intra-MIP flag information indicating that the MIP is applied to the current block. If the MIP is not applied to the current block, the encoding device may generate intra-MIP flag information (e.g., intra_mip_flag) indicating that the MIP is not applied to the current block. The encoding device may output image information including the intra-MIP flag information as a bitstream.

[0211] The encoding apparatus may generate intra prediction samples for the current block (S1310). The encoding apparatus may generate the intra prediction samples based on determining that the MIP is applied to the current block.

[0212] The encoding apparatus may generate residual samples for the current block (S1320). The encoding apparatus may generate residual samples based on intra-prediction samples. The encoding apparatus may generate residual samples based on the difference between the original samples and the intra-prediction samples for the current block.

[0213] The encoding device may encode the image information including the information about the MIP and the information about the residual sample (S1330). The information about the MIP may include intra-frame MIP flag information related to whether the MIP is applied to the current block. The intra-frame MIP flag information may be related to whether the intra-frame prediction mode type for the current block is MIP. In addition, the information about the MIP may include intra-frame MIP mode information related to the MIP applied to the current block.

[0214] Information about residual samples may be referred to as residual information. The encoding device may derive residual information based on the residual samples. The encoding device may derive transform coefficients based on a transform process for the residual samples. For example, the transform process may include at least one of DCT, DST, GBT, or CNT. The encoding device may derive quantized transform coefficients based on a quantization process for the transform coefficients. The quantized transform coefficients may have a one-dimensional vector form based on a coefficient scanning order. The encoding device may generate residual information representing the quantized transform coefficients. The residual information may be generated by various encoding methods, such as exponential Golomb, CAVLC, and CABAC.

[0215] The encoded video / image information can be output in the form of a bit stream. The bit stream can be sent to a decoding device via a network or a storage medium.

[0216] According to an embodiment of this document, the image / video information may include various information. For example, the image / video information may include information disclosed in at least one of Tables 1 to 3 as described above.

[0217] In an embodiment, a matrix for MIP (MIP weight matrix) may be derived based on the width and / or height of the current block.

[0218] In an embodiment, intra prediction samples for the current block may be generated based on a matrix.

[0219] In an embodiment, the image information may include a sequence parameter set (SPS). The SPS may include MIP available flag information (eg, sps_mip_enabled_flag) related to whether the MIP is available.

[0220] In an embodiment, in order to generate an intra-frame prediction sample, the encoding device may derive a reduced boundary sample by downsampling the reference sample (boundary sample) adjacent to the current block, generate a MIP sample based on the product between the reduced boundary sample and the matrix, and perform upsampling of the MIP sample. The intra-frame prediction sample may be generated based on the upsampled MIP sample.

[0221] In an embodiment, the height of the current block may be greater than four times the width of the current block.

[0222] In an embodiment, the width of the current block may be greater than four times the height of the current block.

[0223] In an embodiment, the size of the current block is 32×4, 4×32, 64×8, 8×64, 64×4, 4×64, 128×4, 128×8, 128×16, 4×128, 8×128, or 16×128.

[0224] In an embodiment, the image information may include MIP mode information. A matrix may be further derived based on the MIP mode information, and / or a syntax element bin string for the MIP mode information may be binarized by a truncated binarization method.

[0225] In an embodiment, a matrix may be derived based on three matrix sets classified according to the size of the current block, and / or each of the three matrix sets may include a plurality of matrices.

[0226] Fig.15 and Fig.16An example of a video / image decoding method and related components according to an embodiment of this document is schematically illustrated.

[0227] Fig.15 The method disclosed in can be Figure 3 or Fig.16 Specifically, for example, Fig.15 The S1500 can be made of Fig.16 The entropy decoder 310 of the decoding device is used to perform, Fig.15 The S1510 and S1520 can be Fig.16 The predictor 330 of the decoding device is executed, and Fig.15 The S1530 can be Fig.16 The decoding device is executed by the adder 340. In addition, although Fig.15 Not shown in the figure, the prediction related information or residual information can be obtained by Fig.16 The entropy decoder 310 of the decoding device is derived from the bit stream, the residual samples can be derived from the residual information by the residual processor 320 of the decoding device, the predicted samples can be derived from the prediction related information by the predictor 330 of the decoding device, and the reconstructed block or the reconstructed picture can be derived from the residual samples or the predicted samples by the adder 340 of the decoding device. Fig.15 The methods disclosed in may include the above-mentioned embodiments in this document.

[0228] refer to Fig.15 , the decoding device may receive image information including intra-frame MIP flag information (e.g., intra_mip_flag) related to whether IMP is applied to the current block through a bitstream (S1500). For example, the decoding device may obtain the intra-frame MIP flag information by parsing and decoding the bitstream. Here, the bitstream may be referred to as coded (image) information.

[0229] The image / video information may include various information according to the embodiments of this document. For example, the image / video information may include information disclosed in at least one of Tables 1 to 3 as described above.

[0230] The decoding apparatus may derive the intra prediction type of the current block through the MIP (S1510). The decoding apparatus may derive the intra prediction type through the MIP based on the intra MIP flag information.

[0231] The decoding apparatus may generate intra prediction samples for the current block (S1520). The decoding apparatus may generate the intra prediction samples based on the MIP. The decoding apparatus may generate the intra prediction samples based on neighboring reference samples in a current picture including the current block.

[0232] The decoding device may generate a reconstructed sample for the current block (S1530). The reconstructed sample may be generated based on the intra-prediction sample. The decoding device may directly use the prediction sample as the reconstructed sample according to the prediction mode, or may generate the reconstructed sample by adding the residual sample to the prediction sample.

[0233] In an embodiment, a matrix for MIP (MIP weight matrix) may be derived based on the width and / or height of the current block.

[0234] In an embodiment, intra prediction samples for the current block may be generated based on a matrix.

[0235] In an embodiment, the image information may include a sequence parameter set (SPS). The SPS may include MIP available flag information (eg, sps_mip_enabled_flag) related to whether the MIP is available.

[0236] In an embodiment, in order to generate an intra-frame prediction sample, the decoding device may derive a reduced boundary sample by downsampling the reference sample (boundary sample) adjacent to the current block, generate a MIP sample based on the product between the reduced boundary sample and the matrix, and / or perform upsampling of the MIP sample. The intra-frame prediction sample may be generated based on the upsampled MIP sample.

[0237] In an embodiment, the height of the current block may be greater than four times the width of the current block.

[0238] In an embodiment, the width of the current block may be greater than four times the height of the current block.

[0239] In an embodiment, the size of the current block is 32×4, 4×32, 64×8, 8×64, 64×4, 4×64, 128×4, 128×8, 128×16, 4×128, 8×128, or 16×128.

[0240] In an embodiment, the image information may include MIP mode information. A matrix may be further derived based on the MIP mode information, and / or a syntax element bin string for the MIP mode information may be binarized by a truncated binarization method.

[0241] In an embodiment, a matrix may be derived based on three matrix sets classified according to the size of the current block, and / or each of the three matrix sets may include a plurality of matrices.

[0242] In the presence of residual samples for the current block, the decoding device may receive information about the residual for the current block. The information about the residual may include a transform coefficient for the residual sample. The decoding device may derive the residual sample (or residual sample array) for the current block based on the residual information. Specifically, the decoding device may derive the quantized transform coefficient based on the residual information. The quantized transform coefficient may have a one-dimensional vector form based on the coefficient scanning order. The decoding device may derive the transform coefficient based on the dequantization process for the quantized transform coefficient. The decoding device may derive the residual sample based on the transform coefficient.

[0243] The decoding device may generate a reconstructed sample based on the (intra-frame) prediction sample and the residual sample, and may derive a reconstructed block or a reconstructed picture based on the reconstructed sample. Specifically, the decoding device may generate a reconstructed sample based on the sum of the (intra-frame) prediction sample and the residual sample. Thereafter, as described above, if necessary, the decoding device may apply a loop filtering process (e.g., deblocking filtering and / or SAO process) to the reconstructed picture to improve the subjective / objective picture quality.

[0244] For example, the decoding device can obtain image information including all or some of the above information (or syntax elements) by decoding the bit stream or the encoded information. In addition, the bit stream or the encoded information can be stored in a computer-readable storage medium, or the above decoding method can be executed.

[0245] In the above-described embodiments, methods are described based on a flow chart with a series of steps or frames. The present disclosure is not limited to the order of the above-described steps or frames. Some steps or frames may occur simultaneously with other steps or frames as described above or in an order different from other steps or frames as described above. In addition, it will be appreciated by those skilled in the art that the steps shown in the above-described flow chart are not exclusive and may include additional steps, or one or more steps in the flow chart may be deleted without affecting the scope of this document.

[0246] The method according to the above-mentioned embodiment of this document can be implemented in the form of software, and the encoding device and / or decoding device according to this document can be included in a device that performs image processing such as a TV, a computer, a smart phone, a set-top box, and a display device.

[0247] When the embodiments in this document are implemented in software, the above methods can be implemented as modules (processing, functions, etc.) that perform the above functions. The module can be stored in a memory and executed by a processor. The memory can be inside or outside the processor and can be connected to the processor by various known means. The processor may include an application specific integrated circuit (ASIC), other chipsets, logic circuits, and / or data processing devices. The memory may include a read-only memory (ROM), a random access memory (RAM), a flash memory, a memory card, a storage medium, and / or other storage devices. In other words, the embodiments described in this document can be implemented and executed on a processor, a microprocessor, a controller, or a chip. For example, the functional units shown in each of the accompanying drawings can be implemented and executed on a computer, a processor, a microprocessor, a controller, or a chip. In this case, information about instructions or algorithms for implementation can be stored in a digital storage medium.

[0248] In addition, the decoding device and the encoding device to which this document is applied may be included in a multimedia broadcast transmission / reception device, a mobile communication terminal, a home theater video device, a digital theater video device, a surveillance camera, a video chat device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camera, a VoD service provider, an over-the-top (OTT) video device, an Internet streaming service provider, a three-dimensional (3D) video device, a teleconferencing video device, a traffic user device (i.e., a vehicle-mounted user device, an aircraft user device, a ship user device, etc.), and a medical video device, and may be used to process video signals and data signals. For example, an over-the-top (OTT) video device may include a game console, a Blu-ray player, an Internet access television, a home theater system, a smart phone, a tablet computer, a digital video recorder (DVR), etc.

[0249] In addition, the processing method of applying this document can be generated in the form of a program executed by a computer and can be stored in a computer-readable recording medium. Multimedia data with a data structure according to the present disclosure can also be stored in a computer-readable recording medium. Computer-readable recording media include all kinds of storage devices that store data readable by a computer system. For example, computer-readable recording media may include BD, universal serial bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, computer-readable recording media include media implemented in the form of carrier waves (i.e., transmission via the Internet). In addition, the bit stream generated by the encoding method can be stored in a computer-readable recording medium or can be sent via a wired or wireless communication network.

[0250] In addition, the embodiments of this document can be implemented using a computer program product according to the program code, and the program code can be executed in a computer according to the embodiments of this document. The program code can be stored on a computer-readable carrier.

[0251] Fig.17 An example of a content streaming system to which the embodiments disclosed in this document can be applied is shown.

[0252] refer to Fig.17 The content streaming system to which the embodiments of this document are applied may mainly include an encoding server, a streaming server, a network (web) server, a media storage, a user device, and a multimedia input device.

[0253] The encoding server compresses the content input from the multimedia input device (e.g., smart phone, camera, camcorder, etc.) into digital data to generate a bit stream, and sends the bit stream to the streaming server. As another example, when the multimedia input device such as a smart phone, camera, camcorder, etc. directly generates the bit stream, the encoding server can be omitted.

[0254] A bitstream may be generated by applying the encoding method or the bitstream generating method of the embodiment of the present disclosure, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0255] The streaming server sends multimedia data to the user device through the web server based on the user's request, and the web server serves as a medium for notifying the user of the service. When the user requests the desired service from the web server, the web server passes the request to the streaming server, and the streaming server sends the multimedia data to the user. In this case, the content streaming system may include a separate control server. In this case, the control server is used to control the command / response between the devices in the content streaming system.

[0256] The streaming server can receive content from a media storage and / or encoding server. For example, when receiving content from an encoding server, the content can be received in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a predetermined time.

[0257] Examples of user devices may include mobile phones, smart phones, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, tablet PCs, tablet PCs, ultrabooks, wearable devices (e.g., smart watches, smart glasses, head-mounted displays), digital TVs, desktop computers, digital signage, etc. Each server in the content streaming system may be operated as a distributed server, and in this case, data received from each server may be distributed.

[0258] Each server in the content streaming system may be operated as a distributed server, and in this case, data received from each server may be processed in a distributed manner.

[0259] The claims described herein may be combined in various ways. For example, the technical features of the method claims of this document may be combined and implemented as a device, and the technical features of the device claims of this document may be combined and implemented as a method. In addition, the technical features of the method claims of this document and the technical features of the device claims of this document may be combined to be implemented as a device, and the technical features of the method claims of this document and the technical features of the device claims of this document may be combined and implemented as a method.

Claims

1. A method for decoding an image performed by a decoding device, the method comprising: receiving, via a bitstream, image information including intra-frame matrix-based intra prediction (MIP) flag information related to whether intra-frame MIP is applied to a current block; Based on the intra MIP flag information, deriving an intra prediction type for the current block through the MIP; Based on the MIP, generating intra prediction samples for the current block; as well as generating a reconstructed sample for the current block based on the intra-frame prediction sample, wherein, for the current block whose size is equal to 32×4, the MIP flag information for the current block is obtained from the bitstream, wherein, based on the width and height of the current block, a matrix for the MIP is derived, and Wherein, the intra-frame prediction samples are generated based on the matrix.

2. The image decoding method according to claim 1, wherein: The image information includes a sequence parameter set (SPS), and The SPS includes MIP availability flag information related to whether the MIP is available.

3. The image decoding method according to claim 1, wherein: Generating the intra prediction sample comprises: deriving reduced boundary samples by downsampling reference samples adjacent to the current block; generating MIP samples based on the product between the reduced boundary samples and the matrix; and Upsampling the MIP samples, Wherein, the intra-frame prediction samples are generated based on the up-sampled MIP samples.

4. The image decoding method according to claim 1, wherein: The image information includes MIP mode information, wherein the matrix is ​​further derived based on the MIP mode information, and The syntax element bin string used for the MIP mode information is binarized by a truncated binarization method.

5. A method for encoding an image performed by an encoding device, the method comprising: Derivation whether matrix-based intra prediction (MIP) is applied to the current block; generating the MIP flag information based on whether the MIP is applied to the current block; generating intra prediction samples for the current block based on whether the MIP is applied to the current block; Based on the intra prediction samples, generating residual samples for the current block; as well as encoding image information including information about the MIP, information about the residual samples and the MIP flag information, wherein, for the current block whose size is equal to 32×4, the MIP flag information for the current block is generated and included in the image information, wherein, based on the width and height of the current block, a matrix for the MIP is derived, and Wherein, the intra-frame prediction samples are derived based on the matrix.

6. The image encoding method according to claim 5, wherein: The image information includes a sequence parameter set (SPS), and The SPS includes MIP available flag information related to whether the MIP is available.

7. The image encoding method according to claim 5, wherein: Generating the intra prediction sample comprises: deriving reduced boundary samples by downsampling reference samples adjacent to the current block; generating MIP samples based on the product between the reduced boundary samples and the MIP matrix; and Upsampling the MIP samples, Wherein, the intra-frame prediction samples are generated based on the up-sampled MIP samples.

8. The image encoding method according to claim 5, wherein: The image information includes MIP mode information, wherein the matrix is ​​further derived based on the MIP mode information, and The syntax element bin string used for the MIP mode information is binarized by a truncated binarization method.

9. A non-transitory computer-readable digital storage medium for storing encoded image information generated by a method comprising: Derivation whether matrix-based intra prediction (MIP) is applied to the current block; generating the MIP flag information based on whether the MIP is applied to the current block; Based on determining that the MIP is applied to the current block, generating intra prediction samples for the current block; Based on the intra prediction samples, generating residual samples for the current block; as well as encoding image information including information about the MIP, information about the residual samples and the MIP flag information, wherein, for the current block whose size is equal to 32×4, the MIP flag information for the current block is generated and included in the image information, wherein, based on the width and height of the current block, a matrix for the MIP is derived, and Wherein, the intra-frame prediction samples are generated based on the matrix.

10. A method for transmitting image data, the method comprising: Obtaining a bitstream for the image, wherein the bitstream is generated based on: deriving whether matrix-based intra prediction (MIP) is applied to a current block, generating the MIP flag information based on whether the MIP is applied to the current block, generating intra prediction samples for the current block based on whether the MIP is applied to the current block, generating residual samples for the current block based on the intra prediction samples, and encoding image information including information about the MIP, information about the residual samples, and the MIP flag information; as well as sending said data comprising said bit stream, wherein, for the current block whose size is equal to 32×4, the MIP flag information for the current block is generated and included in the image information, wherein, based on the width and height of the current block, a matrix for the MIP is derived, and Wherein, the intra-frame prediction samples are generated based on the matrix.

Images