MIP-based video encoding / decoding method, bitstream transmission method, and recording medium storing bitstream.

Abstract

The present disclosure provides a MIP-based video encoding / decoding method, a method for transmitting a bitstream, and a recording medium for storing a bitstream. The video decoding method according to the present disclosure includes the steps of determining a prediction mode for a current block and deriving information related to MIP (Matrix-based Intra Prediction) based on the prediction mode being an intra-prediction mode, wherein the MIP may be applied based on the size of the current block or the number of pixels contained in the current block.

Description

[Technical Field]

[0001] This disclosure relates to a MIP-based video encoding / decoding method, a method for transmitting a bitstream, and a recording medium for storing a bitstream, and more particularly to a MIP (matrix-based intraprediction)-based video encoding / decoding method, a method for transmitting a bitstream, and a recording medium for storing a bitstream using only a portion of a reference sample. [Background technology]

[0002] In recent years, demand for high-resolution, high-quality video, such as HD (High Definition) and UHD (Ultra High Definition) video, has been increasing in various fields. The higher the resolution and quality of video data, the greater the amount of information or bits transmitted compared to existing video data. This increase in the amount of information or bits transmitted leads to increased transmission and storage costs.

[0003] Therefore, highly efficient video compression technology is desired to effectively transmit, store, and play back high-resolution, high-quality video information. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The purpose of this disclosure is to provide a video encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0005] Furthermore, this disclosure aims to provide a video encoding / decoding method and apparatus based on in-screen prediction and selected reference samples.

[0006] Furthermore, this disclosure aims to provide a video encoding / decoding method and apparatus that improves intra-screen prediction efficiency.

[0007] Furthermore, this disclosure aims to provide a video encoding / decoding method and apparatus that can perform in-screen prediction adaptively according to the current block size or reference sample when performing in-screen prediction.

[0008] Furthermore, this disclosure aims to provide a non-temporary computer-readable recording medium for storing a bitstream generated by a video encoding method or apparatus relating to this disclosure.

[0009] Furthermore, this disclosure aims to provide a non-temporary computer-readable recording medium that stores a bitstream that is received and decoded by the video decoding device relating to this disclosure and used for restoring video.

[0010] Furthermore, this disclosure aims to provide a method for transmitting a bitstream generated by a video encoding method or apparatus relating to this disclosure.

[0011] The technical challenges addressed in this disclosure are not limited to those mentioned above, and other technical challenges not mentioned above will be clearly understood by those with ordinary skill in the art to which this disclosure pertains from the following description. [Means for solving the problem]

[0012] A video decoding method according to one aspect of the present disclosure includes the steps of determining a prediction mode for a current block and deriving information related to a Matrix-based Intra Prediction (MIP) based on the prediction mode being an intra-prediction mode, wherein the MIP may be applied based on the size of the current block or the number of pixels contained in the current block.

[0013] On the other hand, as an example, the matrix for the MIP may be determined based on the size of the current block or the number of pixels contained in the current block.

[0014] Meanwhile, as an example, when it is determined that the MIP is applied to the current block based on the information related to the MIP, the value of the prediction block of the current block may be limited to a specific value based on a specific condition.

[0015] Meanwhile, as an example, the specific condition may be based on the reference sample value of the current block for the MIP.

[0016] Meanwhile, as an example, when the specific condition is satisfied, the MIP application to the current block may be omitted regardless of the information related to the MIP.

[0017] Meanwhile, as an example, when it is determined that the MIP is applied to the current block based on the information related to the MIP, the prediction block of the current block may be generated using only the top reference samples of the current block.

[0018] Meanwhile, as an example, the top reference samples may include the top-right reference sample of the current block.

[0019] Meanwhile, as an example, when it is determined that the MIP is applied to the current block based on the information related to the MIP, the prediction block of the current block may be generated using only the left reference samples of the current block.

[0020] Meanwhile, as an example, the left reference samples may include the bottom-left reference sample of the current block.

[0021] Meanwhile, as an example, the prediction block of the current block is generated using a part of the left reference samples and the top reference samples of the current block, and at least one of the left reference samples or the top reference samples may be partially adjacent to the current block.

[0022] On the other hand, as an example, when it is determined that MIP is applied to the current block based on the information related to the MIP, the position of the reference sample of the current block is determined, and the position of the reference sample of the current block may be derived based on the information signaled from the bitstream.

[0023] A video encoding method according to an aspect of the present disclosure includes the steps of determining whether to apply MIP (matrix-based intra prediction) to a current block, and determining information related to the MIP, wherein the MIP may be applied based on the size of the current block or the number of pixels included in the current block.

[0024] A computer-readable recording medium according to still another aspect of the present disclosure may store a bitstream generated by the video encoding method or the video encoding apparatus of the present disclosure.

[0025] A transmission method according to still another aspect of the present disclosure may transmit a bitstream generated by the video encoding apparatus or the video encoding method of the present disclosure.

[0026] The features briefly summarized above with respect to the present disclosure are merely exemplary aspects of the detailed description of the present disclosure described later, and do not limit the scope of the present disclosure. Advantageous Effects of Invention

[0027] According to the present disclosure, it is possible to provide a video encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0028] Further, according to the present disclosure, it is possible to provide a video encoding / decoding method and apparatus based on intra prediction and selected partial reference samples.

[0029] Further, according to the present disclosure, when performing intra prediction, intra prediction is performed adaptively according to the size of the current block or reference samples, etc., so that coding efficiency and quality can be improved.

[0030] Furthermore, this disclosure provides a video encoding / decoding method and apparatus that can generate in-screen prediction blocks that are adaptive to the size / number of pixels of the reference sample area and / or the current block during in-screen prediction.

[0031] Furthermore, this disclosure provides a video encoding / decoding method and apparatus that improves intra-screen prediction efficiency.

[0032] Furthermore, this disclosure makes it possible to provide a non-temporary computer-readable recording medium for storing a bitstream generated by a video encoding method or apparatus relating to this disclosure.

[0033] Furthermore, this disclosure makes it possible to provide a non-temporary computer-readable recording medium that stores a bitstream that is received and decoded by the video decoding device relating to this disclosure and used for restoring video.

[0034] Furthermore, this disclosure provides a method for transmitting a bitstream generated by a video encoding method or apparatus relating to this disclosure.

[0035] The effects obtained from this disclosure are not limited to those mentioned above, and any other effects not mentioned above will be clearly understood by a person with ordinary skill in the art to which this disclosure pertains from the following description. [Brief explanation of the drawing]

[0036] [Figure 1] This is a schematic diagram showing a video coding system to which the embodiments of this disclosure can be applied. [Figure 2] This is a schematic diagram showing a video encoding device to which the embodiments of this disclosure can be applied. [Figure 3] This is a schematic diagram showing an image decoding device to which the embodiments of this disclosure can be applied. [Figure 4]This flowchart shows an example of an intra-predictive mode signaling method in an encoding device. [Figure 5] This flowchart shows an example of a method for determining the intra-prediction mode in a decoding device. [Figure 6] This figure shows an example of a surrounding block used for MPM list induction. [Figure 7-11] This figure shows an example of the overall processes of averaging, matrix-vector multiplication, and linear interpolation applicable to this disclosure. [Figure 12] This figure shows an example of a boundary averaging process applicable to this disclosure. [Figure 13] This figure shows an example of a linear interpolation process applicable to this disclosure. [Figure 14] This figure illustrates a MIP prediction method applicable to this disclosure. [Figure 15] This figure illustrates a method for performing MIP prediction according to one embodiment of the present disclosure. [Figure 16] This figure illustrates an MIP-based video encoding or decoding method according to one embodiment of the present disclosure. [Figure 17] This figure illustrates a MIP prediction method applicable to this disclosure. [Figure 18] This figure illustrates a method for performing MIP prediction according to one embodiment of the present disclosure. [Figures 19A-19B] This figure illustrates an MIP-based video encoding or decoding method according to one embodiment of the present disclosure. [Figure 20] This figure illustrates a MIP prediction method applicable to this disclosure. [Figure 21] This figure illustrates a method for performing MIP prediction according to one embodiment of the present disclosure. [Figure 22] This figure illustrates an MIP-based video encoding or decoding method according to one embodiment of the present disclosure. [Figure 23] This figure illustrates an MIP-based video encoding or decoding method according to one embodiment of the present disclosure. [Figure 24]This figure illustrates an MIP-based video encoding or decoding method according to one embodiment of the present disclosure. [Figure 25] This figure shows a video decoding method according to one embodiment of the present disclosure. [Figure 26] This figure shows a video encoding method according to one embodiment of the present disclosure. [Figure 27] This figure illustrates a content streaming system to which the embodiments of this disclosure can be applied. [Modes for carrying out the invention]

[0037] The embodiments of this disclosure are described below in detail with reference to the accompanying drawings, so as to be easily implemented by a person with ordinary skill in the art to which this disclosure pertains. However, this disclosure may be embodied in various other forms and is not limited to the embodiments described herein.

[0038] In describing embodiments of this disclosure, if a specific description of a known configuration or function is deemed to obscure the gist of this disclosure, such detailed description will be omitted. In the figures, parts unrelated to the description of this disclosure will be omitted, and similar parts will be denoted by similar reference numerals.

[0039] In this disclosure, when one component is described as being “linked,” “joined,” or “connected” to another component, this may include not only direct linkages but also indirect linkages where other components exist in between. Furthermore, when one component is described as “containing” or “having” another component, this means, unless otherwise specified, that it may contain further other components rather than excluding them.

[0040] In this disclosure, terms such as "first," "second," etc., are used solely to distinguish one component from another, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0041] In this disclosure, components are distinguished from each other solely to clearly describe their respective characteristics, and this does not necessarily mean that these components are separate. That is, multiple components may be integrated to constitute a single hardware or software unit, or a single component may be distributed to constitute multiple hardware or software units. Therefore, such integrated or distributed embodiments are also included in the scope of this disclosure, even without specific mention.

[0042] In this disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments consisting of a subset of the components described in one embodiment are also included in the scope of this disclosure. Furthermore, embodiments that further include other components in addition to the components described in various embodiments are also included in the scope of this disclosure.

[0043] This disclosure relates to the encoding and decoding of video, and unless otherwise defined herein, the terms used herein may have their ordinary meanings in the art to which this disclosure pertains.

[0044] In this disclosure, "picture" generally refers to a unit representing a single video image for a specific time period, and "slice / tile" is an encoding unit that constitutes a part of a picture. A single picture may consist of one or more slices / tiles. A slice / tile may also contain one or more CTUs (coding tree units).

[0045] In this disclosure, “pixel” or “pel” can mean the smallest unit that constitutes a picture (or video). The term “sample” may also be used as a counterpart to pixel. A sample may generally represent a pixel or a pixel value, or it may represent only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component.

[0046] In this disclosure, “unit” may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information associated with that region. A unit may, as it may be, be replaced by terms such as “sample array,” “block,” or “area.” In general, an MxN block may include a sample (or sample array) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0047] In this disclosure, “current block” can mean one of the following: “current coding block,” “current coding unit,” “block to encode,” “block to decode,” or “block to process.” When prediction is performed, “current block” can mean “current prediction block” or “block to predict.” When transformation (inverse transformation) / quantization (inverse quantization) is performed, “current block” can mean “current transformation block” or “block to transform.” When filtering is performed, “current block” can mean “block to filter.”

[0048] In this disclosure, "current block" may mean a block containing both a luma component block and a chroma component block, or "the luma block of the current block," unless otherwise explicitly stated as a chroma block. The luma component block of the current block may be expressed with an explicit mention of a luma component block, such as "luma block" or "current luma block." Similarly, the chroma component block of the current block may be expressed with an explicit mention of a chroma component block, such as "chroma block" or "current chroma block."

[0049] In this disclosure, " / " and "," may be interpreted as "and / or." For example, "A / B" and "A, B" may be interpreted as "A and / or B." Also, "A / B / C" and "A, B, C" may mean "at least one of A, B and / or C."

[0050] In this disclosure, “or” may be interpreted as “and / or.” For example, “A or B” may mean 1) “A” only, 2) “B” only, or 3) “A and B.” Alternatively, in this disclosure, “or” may mean “additionally or alternatively.”

[0051] Overview of the video coding system

[0052] Figure 1 is a schematic diagram showing a video coding system to which the embodiments of this disclosure can be applied.

[0053] A video coding system according to one embodiment may include an encoding device 10 and a decoding device 20. The encoding device 10 can transmit encoded video and / or image information or data to the decoding device 20 in file or streaming form via a digital storage medium or network.

[0054] An encoding device 10 according to one embodiment may include a video source generation unit 11, an encoding unit 12, and a transmission unit 13. A decoding device 20 according to one embodiment may include a receiving unit 21, a decoding unit 22, and a rendering unit 23. The encoding unit 12 may be called a video / image encoding unit, and the decoding unit 22 may be called a video / image decoding unit. The transmission unit 13 may be included in the encoding unit 12. The receiving unit 21 may be included in the decoding unit 22. The rendering unit 23 may include a display unit, and the display unit may be composed of a separate device or external component.

[0055] The video source generation unit 11 can acquire video / images through video / image capture, synthesis, or generation processes. The video source generation unit 11 may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, or a video / image archive containing previously captured video / images. The video / image generation device may include, for example, a computer, a tablet, and a smartphone, and can generate video / images (electronically). For example, virtual video / images may be generated by a computer, in which case the video / image capture process may be replaced by a process in which related data is generated.

[0056] The encoding unit 12 can encode the input video / image data. The encoding unit 12 can perform a series of procedures such as prediction, transformation, and quantization for compression and encoding efficiency. The encoding unit 12 can output the encoded data (encoded video / image information) in the form of a bitstream.

[0057] The transmitting unit 13 can acquire encoded video / image information or data output in bitstream form and transmit it in file or streaming form to the receiving unit 21 of the decoding device 20 or other external object via a digital storage medium or network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray (registered trademark: same hereinafter), HDD, SSD, etc. The transmitting unit 13 may include elements for generating media files in a predetermined file format and may include elements for transmission via a broadcast / communication network. The transmitting unit 13 may be provided as a transmission device separate from the encoding unit 12, in which case the transmission device may include at least one processor that acquires encoded video / image information or data output in bitstream form and a transmitting unit that transmits it in file or streaming form. The receiving unit 21 can extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit 22.

[0058] The decoding unit 22 can decode the video / image by performing a series of procedures such as inverse quantization, inverse transform, and prediction, which correspond to the operation of the encoding unit 12.

[0059] The rendering unit 23 can render the decoded video / image. The rendered video / image may be displayed through the display unit.

[0060] Overview of video encoding equipment

[0061] Figure 2 is a schematic diagram showing a video encoding device to which the embodiments of this disclosure can be applied.

[0062] As shown in Figure 2, the video encoding device 100 may include a video splitting unit 110, a subtraction unit 115, a conversion unit 120, a quantization unit 130, an inverse quantization unit 140, an inverse conversion unit 150, an addition unit 155, a filtering unit 160, a memory 170, an inter-prediction unit 180, an intra-prediction unit 185, and an entropy encoding unit 190. The inter-prediction unit 180 and the intra-prediction unit 185 may be collectively called the "prediction unit". The conversion unit 120, the quantization unit 130, the inverse quantization unit 140, and the inverse conversion unit 150 may be included in the residual processing unit. The residual processing unit may further include a subtraction unit 115.

[0063] Depending on the embodiment, all or at least some of the multiple components constituting the video encoding device 100 may be embodied as a single hardware component (e.g., an encoder or a processor). Furthermore, the memory 170 may include a DPB (decoded picture buffer) and may be embodied by a digital storage medium.

[0064] The video splitting unit 110 can split the input video (or picture, frame) input to the video encoding device 100 into one or more processing units. For example, the processing units may be called coding units (CUs). Coding units can be obtained by recursively splitting a coding tree unit (CTU) or the largest coding unit (LCU) using a QT / BT / TT (Quad-tree / binary-tree / ternary-tree) structure. For example, one coding unit may be split into multiple coding units of deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary-tree structure. For the splitting of coding units, a quad-tree structure may be applied first, followed by a binary-tree structure and / or a ternary-tree structure. The coding procedure according to this disclosure may be performed based on the final coding unit that is not further split. The maximum coding unit may be used directly as the final coding unit, or a lower-depth coding unit obtained by dividing the maximum coding unit may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and / or restoration, which will be described later. As another example, the processing unit of the coding procedure may be a prediction unit (PU) or a transformation unit (TU). The prediction unit and the transformation unit may be divided or partitioned from the final coding unit, respectively. The prediction unit may be a unit of sample prediction, and the transformation unit may be a unit that derives transformation coefficients and / or a unit that derives a residual signal from transformation coefficients.

[0065] The prediction unit (inter-prediction unit 180 or intra-prediction unit 185) can make predictions for the block to be processed (current block) and generate a predicted block that includes prediction samples for the current block. The prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block or on a CU basis. The prediction unit can generate various information regarding the prediction of the current block and transmit it to the entropy encoding unit 190. The prediction information may be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

[0066] The intra-prediction unit 185 can predict the current block by referring to a sample in the current picture. The referenced sample may be located in the vicinity of the current block or at a distance from it, depending on the intra-prediction mode and / or intra-prediction method. The intra-prediction mode may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the accuracy of the prediction direction. However, this is an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-prediction unit 185 can also determine the prediction mode to be applied to the current block using the prediction modes applied to the surrounding blocks.

[0067] The interprediction unit 180 can derive a predicted block relative to the current block based on a reference block (reference sample array) identified by motion vectors on the reference picture. In this case, in order to reduce the amount of motion information transmitted in interprediction mode, motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of motion information between the surrounding blocks and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include interprediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, the surrounding blocks may include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. The reference picture containing the reference block and the reference picture containing the temporal neighboring block may be the same or different from each other. The temporal neighboring block may be called a collocated reference block, colCU, etc. The reference picture containing the temporal neighboring block may be called a collocated picture (colPic). For example, the interpretation unit 180 can construct a motion information candidate list based on surrounding blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Interpretation may be performed based on various prediction modes; for example, in skip mode and merge mode, the interpretation unit 180 can use the motion information of surrounding blocks as the motion information of the current block. In skip mode, unlike merge mode, the residual signal does not need to be transmitted.In motion vector prediction (MVP) mode, the motion vectors of surrounding blocks are used as motion vector predictors, and the motion vector of the current block can be signaled by encoding the motion vector difference and an indicator for the motion vector predictor. The motion vector difference represents the difference between the motion vector of the current block and the motion vector predictor.

[0068] The prediction unit can generate a prediction signal based on various prediction methods and / or prediction techniques described later. For example, the prediction unit may apply intra-prediction or inter-prediction to predict the current block, or it may apply intra-prediction and inter-prediction simultaneously. A prediction method that applies intra-prediction and inter-prediction simultaneously to predict the current block may be called CIIP (combined inter and intra prediction). The prediction unit can also perform intra-block copy (IBC) to predict the current block. Intra-block copy may be used, for example, for coding content images / videos such as games, as in SCC (screen content coding). IBC is a method of predicting the current block using a reference block that has already been restored in the current picture at a predetermined distance from the current block. When IBC is applied, the position of the reference block in the current picture may be encoded as a vector (block vector) corresponding to the predetermined distance. IBC basically performs prediction within the current picture, but it may be performed similarly to inter-prediction in that it derives the reference block within the current picture. In other words, IBC can use at least one of the interpretation methods described in this disclosure.

[0069] The predicted signal generated by the prediction unit may be used to generate a restored signal or a residual signal. The subtraction unit 115 can generate a residual signal (residual block, residual sample array) by subtracting the predicted signal output from the prediction unit (predicted block, predicted sample array) from the input video signal (original block, original sample array). The generated residual signal may be transmitted to the conversion unit 120.

[0070] The transformation unit 120 can generate transformation coefficients by applying a transformation method to the residual signal. For example, the transformation method may include at least one of the following: DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT refers to the transformation obtained from a graph when the relationship information between pixels is represented by this graph. CNT refers to the transformation obtained by generating a prediction signal using all previously reconstructed pixels and obtaining a transformation based on it. The transformation process may be applied to pixel blocks of the same size and square shape, or to blocks of a variable size instead of square shape.

[0071] The quantization unit 130 can quantize the conversion coefficients and transmit them to the entropy encoding unit 190. The entropy encoding unit 190 can encode the quantized signal (information about the quantized conversion coefficients) and output it as a bitstream. The information about the quantized conversion coefficients may be called residual information. The quantization unit 130 can rearrange the block-shaped quantized conversion coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate information about the quantized conversion coefficients based on the one-dimensional vector form of the quantized conversion coefficients.

[0072] The entropy encoding unit 190 can perform various encoding methods, such as exponential Golomb, CAVLC (context-adaptive variable length coding), and CABAC (context-adaptive binary arithmetic coding). In addition to the quantized conversion coefficients, the entropy encoding unit 190 can also encode information necessary for video / image restoration (e.g., the values ​​of syntax elements) together or separately. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information about various parameter sets, such as an adaptation parameter set (APS), picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS). The video / image information may also further include general constraint information. The signaling information, transmitted information, and / or syntax elements referred to in this disclosure may be encoded by the encoding procedure described above and included in the bitstream.

[0073] The bitstream may be transmitted over a network or stored on a digital storage medium. Here, the network may include broadcasting networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitting unit (not shown) for transmitting the signal output from the entropy encoding unit 190 and / or a storage unit (not shown) for storing it may be provided as an internal / external element of the video encoding device 100, or the transmitting unit may be provided as a component of the entropy encoding unit 190.

[0074] The quantized conversion coefficients output from the quantization unit 130 may be used to generate a resistive signal. For example, by applying inverse quantization and inverse transformation to the quantized conversion coefficients in the inverse quantization unit 140 and the inverse transformation unit 150, a resistive signal (residual block or resistive sample) can be reconstructed.

[0075] The adder 155 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the predicted signal output from the inter-prediction unit 180 or the intra-prediction unit 185. When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block may be used as the reconstructed block. The adder 155 may be called the reconstruction unit or the reconstructed block generation unit. The generated reconstructed signal may be used for intra-prediction of the next block to be processed in the current picture, or, as described later, may be used for inter-prediction of the next picture after filtering.

[0076] The filtering unit 160 can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit 160 can apply various filtering methods to the restored picture to generate a modified restored picture, and store the modified restored picture in the memory 170, specifically in the DPB of the memory 170. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, and bilateral filter. The filtering unit 160 can generate various filtering information and transmit it to the entropy encoding unit 190, as will be described later in the description of each filtering method. The filtering information may be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

[0077] The corrected restored picture transmitted to memory 170 may be used as a reference picture in the interpretation unit 180. This allows the video encoding device 100 to avoid prediction mismatches between the video encoding device 100 and the video decoding device when interpretation is applied, and also improves encoding efficiency.

[0078] The DPB in memory 170 can store the corrected restored picture for use as a reference picture in the inter-prediction unit 180. Memory 170 can store motion information of blocks from which motion information in the current picture has been derived (or encoded) and / or motion information of blocks in the picture that have already been restored. The stored motion information may be transmitted to the inter-prediction unit 180 for use as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory 170 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 185.

[0079] Overview of the video decoding device

[0080] Figure 3 is a schematic diagram showing an image decoding device to which the embodiments of this disclosure can be applied.

[0081] As shown in Figure 3, the video decoding device 200 may include an entropy decoding unit 210, an inverse quantization unit 220, an inverse transformation unit 230, an addition unit 235, a filtering unit 240, a memory 250, an inter-prediction unit 260, and an intra-prediction unit 265. The inter-prediction unit 260 and the intra-prediction unit 265 can be collectively referred to as the "prediction unit". The inverse quantization unit 220 and the inverse transformation unit 230 may be included in the residual processing unit.

[0082] All or at least some of the multiple components constituting the video decoding device 200 may be embodied as a single hardware component (e.g., a decoder or processor) depending on the embodiment. Furthermore, the memory 170 may include a DPB and may be embodied by a digital storage medium.

[0083] A video decoding device 200 that receives a bitstream containing video / image information can restore the image by performing a process corresponding to the process performed by the video encoding device 100 in Figure 2. For example, the video decoding device 200 can perform decoding using the processing unit applied in the video encoding device. Therefore, the decoding processing unit may be, for example, a coding unit. The coding unit may be a coding tree unit, or it may be obtained by dividing the largest coding unit. The restored video signal decoded and output by the video decoding device 200 may then be played back by a playback device (not shown).

[0084] The video decoding device 200 can receive the signal output from the video encoding device shown in Figure 2 in the form of a bitstream. The received signal may be decoded by the entropy decoding unit 210. For example, the entropy decoding unit 210 can parse the bitstream to derive information necessary for video restoration (or picture restoration) (e.g., video / image information). The video / image information may further include information about various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). The video / image information may also further include general constraint information. The video decoding device may further utilize the parameter set information and / or the general constraint information to decode the video. The signaling information, received information, and / or syntax elements referred to in this disclosure may be obtained from the bitstream by decoding through the decoding procedure. For example, the entropy decoding unit 210 can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements necessary for image restoration and the quantized values ​​of conversion coefficients related to the residual. More specifically, the CABAC entropy decoding method receives bins corresponding to each syntax element in the bitstream, determines a context model using the syntax element information to be decoded and the decoding information of the surrounding blocks and the blocks to be decoded, or symbol / bin information decoded in a previous stage, predicts the probability of bin occurrence based on the determined context model, performs arithmetic decoding of the bins, and generates symbols corresponding to the values ​​of each syntax element.In this case, the CABAC entropy decoding method can update the context model using the decoded symbol / bin information for the context model of the next symbol / bin after determining the context model. Information related to prediction from the information decoded by the entropy decoding unit 210 is provided to the prediction unit (inter-prediction unit 260 and intra-prediction unit 265), and residual values ​​that have been entropy decoded by the entropy decoding unit 210, i.e., quantized conversion coefficients and related parameter information, may be input to the inverse quantization unit 220. In addition, information related to filtering from the information decoded by the entropy decoding unit 210 may be provided to the filtering unit 240. On the other hand, a receiving unit (not shown) that receives signals output from the video encoding device may be further provided as an internal / external element of the video decoding device 200, or the receiving unit may be provided as a component of the entropy decoding unit 210.

[0085] On the other hand, the video decoding device according to this disclosure may be called a video / image / picture decoding device. The video decoding device may include an information decoder (video / image / picture information decoder) and / or a sample decoder (video / image / picture sample decoder). The information decoder may include an entropy decoding unit 210, and the sample decoder may include at least one of an inverse quantization unit 220, an inverse transformation unit 230, an addition unit 235, a filtering unit 240, a memory 250, an inter-prediction unit 260, and an intra-prediction unit 265.

[0086] The inverse quantization unit 220 can inverse quantize the quantized transformation coefficients and output the transformation coefficients. The inverse quantization unit 220 can rearrange the quantized transformation coefficients in a two-dimensional block form. In this case, the rearrangement may be performed based on the coefficient scan order performed by the video encoding device. The inverse quantization unit 220 can perform inverse quantization on the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) and obtain the transformation coefficients.

[0087] The inverse conversion unit 230 can inversely convert the conversion coefficients to obtain residual signals (residual blocks, residual sample arrays).

[0088] The prediction unit can make predictions for the current block and generate a predicted block containing prediction samples for the current block. Based on the prediction information output from the entropy decoding unit 210, the prediction unit can determine whether intra-prediction or inter-prediction is applied to the current block and can determine a specific intra / inter-prediction mode (prediction method).

[0089] As mentioned in the description of the prediction unit of the video coding device 100, the prediction unit can generate prediction signals based on various prediction methods (techniques) described later.

[0090] The intra-prediction unit 265 can predict the current block by referring to the samples in the current picture. The description of the intra-prediction unit 185 may also apply to the intra-prediction unit 265.

[0091] The interprediction unit 260 can derive a predicted block relative to the current block based on a reference block (reference sample array) identified by motion vectors on the reference picture. In this case, in order to reduce the amount of motion information transmitted in interprediction mode, motion information can be predicted in block, subblock, or sample units based on the correlation of motion information between the surrounding block and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include interprediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, the surrounding block may include spatially neighboring blocks present in the current picture and temporally neighboring blocks present in the reference picture. For example, the interprediction unit 260 can construct a motion information candidate list based on the surrounding blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Interprediction may be performed based on various prediction modes (methods), and the prediction information may include information indicating the mode (method) of interprediction for the current block.

[0092] The adder 235 can generate a restored signal (restored picture, restored block, restored sample array) by adding the acquired residual signal to the predicted signal (predicted block, predicted sample array) output from the prediction unit (including the inter-prediction unit 260 and / or intra-prediction unit 265). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block may be used as the restored block. The description of the adder 155 may also apply to the adder 235. The adder 235 may be called the restore unit or the restored block generation unit. The generated restored signal may be used for intra-prediction of the next block to be processed in the current picture, or, as described later, may be used for inter-prediction of the next picture after filtering.

[0093] The filtering unit 240 can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit 240 can apply various filtering methods to the restored picture to generate a modified restored picture, and the modified restored picture can be stored in the memory 250, specifically in the DPB of the memory 250. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, and bilateral filter.

[0094] The restored picture stored (modified) in the DPB of memory 250 may be used as a reference picture in the inter-prediction unit 260. Memory 250 can store motion information of blocks from which motion information in the current picture has been derived (or decoded) and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit 260 for use as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory 250 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 265.

[0095] In this specification, the embodiments described for the filtering unit 160, inter-prediction unit 180, and intra-prediction unit 185 of the video encoding device 100 may be applied identically or in a corresponding manner to the filtering unit 240, inter-prediction unit 260, and intra-prediction unit 265 of the video decoding device 200, respectively.

[0096] Intra Prediction Mode / Type Determination

[0097] When intraprediction is applied, the intraprediction mode applied to the current block may be determined using the intraprediction modes of the surrounding blocks. For example, the decoding device may select one of the mpm candidates in the mpm (most probable mode) list derived based on the intraprediction modes of the surrounding blocks of the current block (e.g., the surrounding blocks to the left and / or above) and further candidate modes, based on the received mpm index, or it may select one of the remaining intraprediction modes not included in the mpm candidates (and planar modes), based on the remaining intraprediction mode information. The mpm list may or may not include planar modes as candidates. For example, if the mpm list includes planar modes as candidates, the mpm list may have six candidates, and if the mpm list does not include planar modes as candidates, the mpm list may have three candidates. If the aforementioned mpm list does not include planar mode as a candidate, a not-planar flag (e.g., intra_luma_not_planar_flag) may be signaled to indicate that the current intra-prediction mode of the block is not planar mode. For example, the mpm flag may be signaled first, and the mpm index and not-planar flag may be signaled if the value of the mpm flag is 1. Also, the mpm index may be signaled if the value of the not-planar flag is 1. Here, the reason why the aforementioned mpm list is configured not to include planar mode as a candidate is not because planar mode is not an mpm, but because planar mode is always considered as an mpm, so the flag (not-planar flag) is signaled first to check whether it is a planar mode or not.

[0098] For example, whether the intra-prediction mode applied to the current block is within the MPM candidate (and planar mode) or within the remaining mode may be indicated based on the MPM flag (e.g., intra_luma_mpm_flag). A value of 1 for the MPM flag indicates that the intra-prediction mode for the current block is within the MPM candidate (and planar mode), and a value of 0 for the MPM flag indicates that the intra-prediction mode for the current block is not within the MPM candidate (and planar mode). A value of 0 for the not planar flag (e.g., intra_luma_not_planar_flag) indicates that the intra-prediction mode for the current block is planar mode, and a value of 1 for the not planar flag indicates that the intra-prediction mode for the current block is not planar mode. The mpm index may be signaled in the form of an mpm_idx or intra_luma_mpm_idx syntax element, and the remaining intra prediction mode information may be signaled in the form of a rem_intra_luma_pred_mode or intra_luma_mpm_remainder syntax element. For example, the remaining intra prediction mode information can indicate one of the remaining intra prediction modes from the overall intra prediction modes that are not included in the mpm candidates (and planar modes), indexed in order of prediction mode number. The intra prediction mode may be an intra prediction mode for a luma component (sample). Hereinafter, the intra prediction mode information may include at least one of the following: the mpm flag (e.g., intra_luma_mpm_flag), the not planar flag (e.g., intra_luma_not_planar_flag), the mpm index (e.g., mpm_idx or intra_luma_mpm_idx), or the remaining intra prediction mode information (rem_intra_luma_pred_mode or intra_luma_mpm_remainder). In this document, the MPM list may be referred to by various terms such as MPM candidate list, candModeList, etc.When MIP is applied to the current block, a separate mpm flag for MIP (e.g., intra_mip_mpm_flag), an mpm index (e.g., intra_mip_mpm_idx), and remaining intra prediction mode information (e.g., intra_mip_mpm_remainder) may be signaled, while the not planar flag is not signaled.

[0099] The intra-predictive mode signaling procedure in the encoding device and the intra-predictive mode determination procedure in the decoding device may be carried out, for example, as follows.

[0100] Figure 4 is a flowchart showing an example of an intra-predictive mode signaling method in an encoding device.

[0101] Referring to Figure 4, the encoding device configures an MPM list for the current block (S400). The MPM list may include candidate intra-prediction modes (MPM candidates) that are likely to be applied to the current block. The MPM list may also include intra-prediction modes for surrounding blocks, and may further include specific intra-prediction modes by a predetermined method. A specific method for configuring the MPM list will be described later.

[0102] The encoding device determines the intra-prediction mode for the current block (S410). The encoding device can perform predictions based on various intra-prediction modes and determine the optimal intra-prediction mode based on RDO (rate-distortion optimization) thereon. In this case, the encoding device may determine the optimal intra-prediction mode using only the MPM candidates and planar modes configured in the MPM list, or it may determine the optimal intra-prediction mode using the remaining intra-prediction modes in addition to the MPM candidates and planar modes configured in the MPM list. To give a specific example, if the intra-prediction type of the current block is not a normal intra-prediction type but a specific type (e.g., LIP, MRL, or ISP), the encoding device can determine the optimal intra-prediction mode by considering only the MPM candidates and planar modes as intra-prediction mode candidates for the current block. That is, in this case, the intra-prediction mode for the current block may be determined from among the MPM candidates and planar modes, and in this case, the mpm flag does not need to be encoded / signaled. In this case, the decoding device can infer that the mpm flag is 1 even if the mpm flag is not separately signaled.

[0103] On the other hand, generally, if the intra-prediction mode of the current block is not the planar mode but one of the MPM candidates in the MPM list, the encoding device generates an mpm index (mpm idx) indicating one of the MPM candidates. If the intra-prediction mode of the current block is not found in the MPM list, the device generates remaining intra-prediction mode information that indicates the same mode as the intra-prediction mode of the current block from among the remaining intra-prediction modes not included in the MPM list (and planar mode).

[0104] The encoding device can encode intra-prediction mode information and output it in the form of a bitstream. The intra-prediction mode information may include the mpm flag, not planar flag, mpm index, and / or remaining intra-prediction mode information described above. In general, the mpm index and the remaining intra-prediction mode information have an alternative relationship and are not signaled simultaneously when indicating an intra-prediction mode for a block. That is, either an mpm flag value of 1 and the not planar flag or mpm index are signaled together, or an mpm flag value of 0 and the remaining intra-prediction mode information are signaled together. However, as described above, if a specific intra-prediction type is applied to the current block, the mpm flag may not be signaled, and only the not planar flag and / or mpm index may be signaled. That is, in this case, the intra-prediction mode information may include only the not planar flag and / or mpm index.

[0105] The decoding device can determine the intra-prediction mode in response to the intra-prediction mode information determined and signaled by the encoding device.

[0106] Figure 5 is a flowchart showing an example of an intra-prediction mode determination method in a decoding device.

[0107] Referring to Figure 5, the decoding device obtains intra-prediction mode information from the bitstream (S500). As described above, the intra-prediction mode information may include at least one of the mpm flag, not planar flag, mpm index, and remaining intra-prediction mode.

[0108] The decoding device configures an MPM list (S510). The MPM list is configured identically to the MPM list configured by the encoding device. That is, the MPM list may include intra-prediction modes for peripheral blocks, and may further include specific intra-prediction modes by a predetermined method. A specific method for configuring the MPM list will be described later.

[0109] Although it is stated that S510 is performed after S500, this is merely an example, and S510 may be performed before S500, or at the same time.

[0110] The decoding device determines the intra-prediction mode of the current block based on the MPM list and the intra-prediction mode information (S520). For example, if the value of the mpm flag is 1, the decoding device may derive the planar mode as the intra-prediction mode of the current block, or derive the candidate indicated by the mpm index from among the MPM candidates in the MPM list (not planar flag-based) as the intra-prediction mode of the current block. As another example, if the value of the mpm flag is 0, the decoding device may derive the intra-prediction mode indicated by the remaining intra-prediction mode information from among the remaining intra-prediction modes not included in the MPM list and planar mode as the intra-prediction mode of the current block. On the other hand, as yet another example, if the intra-prediction type of the current block is a specific type (e.g., LIP, MRL, or ISP), the decoding device may derive the planar mode or the candidate indicated by the mpm index in the MPM list as the intra-prediction mode of the current block without checking the mpm flag.

[0111] For example, the not planar flag may be signaled when the MRL is not currently applied to a block (i.e., intra_luma_ref_idx == 0), and may be omitted when the MRL is currently applied to a block (i.e., intra_luma_ref_idx != 0). If the not planar flag is omitted, the decoding device may estimate its value to be 1.

[0112] On the other hand, the intra-prediction mode may include two directional intra-prediction modes and 65 directional intra-prediction modes. The non-directional intra-prediction mode may include a planar intra-prediction mode and a DC intra-prediction mode, and the directional intra-prediction mode may include intra-prediction modes 2 through 66. The extended directional intra-prediction mode may be applied to blocks of all sizes and may be applied to either the luma component or the chroma component.

[0113] On the other hand, the intra-prediction mode may further include a CCLM (cross-component linear model) mode for chroma samples, in addition to the intra-prediction mode described above. The CCLM mode may be divided into LT_CCLM, L_CCLM, and T_CCLM depending on whether the left sample, the upper sample, or both are considered for the derivation of the LM parameters, and may be applied only to chroma components.

[0114] The intra-prediction mode may be indexed as shown in Table 1 below, for example.

[0115] [Table 1]

[0116] On the other hand, the intra prediction type (or additional intra prediction mode, etc.) may include at least one of the above-mentioned LIP, PDPC, MRL, ISP, and MIP. The intra prediction type may be indicated based on intra prediction type information, and the intra prediction type information may be embodied in various forms. For example, the intra prediction type information may include intra prediction type index information that indicates one of the intra prediction types. As another example, the intra prediction type information may include at least one of the following: reference sample line information (e.g., intra_luma_ref_idx) indicating whether the MRL is applied to the current block and, if so, which reference sample line is used; ISP flag information (e.g., intra_subpartitions_mode_flag) indicating whether the ISP is applied to the current block; ISP type information (e.g., intra_subpartitions_split_flag) indicating the split type of the subpartition when the ISP is applied; flag information indicating whether the PDCP is applied; or flag information indicating whether the LIP is applied. Furthermore, the intra prediction type information may include an MIP flag (or may be called intra_mip_flag) indicating whether or not MIP is applied to the current block.

[0117] On the other hand, as described above, when an MIP is applied to the current block (for example, when the value of intra_mip_flag is 1), a separate MPM list may be configured for the MIP, and the MPM flag that may be included in the intra prediction mode information for the MIP may be called intra_mip_mpm_flag, the MPM index may be called intra_mip_mpm_idx, and the remaining intra prediction mode information may be called intra_mip_mpm_remainder.

[0118] Furthermore, various prediction modes may be used for MIP, and the intra-prediction modes for MIP can be used to derive the matrix and offset for MIP. As described above, the matrix may be called the (MIP)weighted matrix, and the offset may be called the (MIP)offset vector or (MIP)bias vector. The number of intra-prediction modes for MIP may be set differently based on the size of the current block. For example, i) if the height and width of the current block (e.g., CB or TB) are both 4, then 35 intra-prediction modes (i.e., intra-prediction modes 0-34) may be available; ii) if both the height and width of the current block are 8 or less, then 19 intra-prediction modes (i.e., intra-prediction modes 0-18) may be available; and iii) in other cases, then 11 intra-prediction modes (i.e., intra-prediction modes 0-10) may be available. For example, if the current block height and width are both 4, then block size type 0 is defined as block size type 1 when both the current block height and width are 8 or less, and block size type 2 for all other cases, then the number of intra-prediction modes for MIP may be organized as shown in the following table. However, this is an example, and the number of block size types and available intra-prediction modes may be changed. In this document, intra-prediction modes for MIP may be referred to as MIP intra-prediction modes, MIP prediction modes, or MIP modes.

[0119] [Table 2]

[0120] On the other hand, the ECM (enhanced compression model) introduced a secondary MPM list. The existing primary MPM (PMPM) list consists of 6 entries, and the secondary MPM (SMPM) list contains 16 entries. First, a general MPM list with 22 entries is constructed, and then the first 6 entries from the general MPM list are included in the PMPM list, and the remaining entries are included in the SMPM list. In the general MPM list, the first entry is the planar mode, and the remaining entries consist of intra-modes for the left (L), upper (A), lower left (BL), upper right (AR), and upper left (AL) peripheral blocks as shown in Figure 6, directional modes with an offset added to the first two available directional modes of the peripheral block, and the default mode.

[0121] When the CU block is vertical, the order of the surrounding blocks may be top (A), left (L), lower left (BL), upper right (AR), and upper left (AL). Alternatively, the order may be left (L), top (A), lower left (BL), upper right (AR), and upper left (AL).

[0122] The PMPM flag is parsed, and if its value is 1, the PMPM index is parsed to determine which entry in the PMPM list is selected. Alternatively, the SPMPM flag is parsed to determine whether or not to parse the SPMPM index or the remaining mode.

[0123] Derivation of peripheral reference samples

[0124] When intraprediction is applied to the current block, peripheral reference samples to be used for intraprediction of the current block may be derived. The peripheral reference samples of the current block may include a total of 2 x nH samples adjacent to the left boundary and bottom-left of the current block of nW x nH size, a total of 2 x nW samples adjacent to the top boundary and top-right of the current block, and one sample adjacent to the top-left of the current block. Alternatively, the peripheral reference samples of the current block may include upper peripheral samples for multiple columns and left peripheral samples for multiple rows. Furthermore, the peripheral reference samples of the current block may include a total of nH samples adjacent to the right boundary of the current block of nW x nH size, a total of nW samples adjacent to the bottom boundary of the current block, and one sample adjacent to the bottom-right of the current block.

[0125] On the other hand, when MRL, which will be described later, is applied, the reference sample may be located on lines 1 to 3 on the left / above side, rather than on line 0 which is adjacent to the current block. In this case, the number of surrounding reference samples may increase further. The specific regions and numbers of surrounding reference samples will be described later.

[0126] On the other hand, when the ISP described later is applied, the surrounding reference samples can be derived on a subpartition basis.

[0127] Currently, some of the surrounding reference samples in a block may not yet be decoded or available. In this case, the decoder can construct the surrounding reference samples to be used for prediction by interpolating the available samples.

[0128] Currently, some of the surrounding reference samples in a block may not yet be decoded or available. In this case, the decoder can construct the surrounding reference samples to be used for prediction by extrapolating the available samples. Starting from the bottom left corner and reaching the top right corner reference sample, the decoder can construct the samples by updating the available samples with the latest available sample, while substituting or padding undecoded or unavailable pixels with the latest available sample.

[0129] Derivation of intra-prediction mode / type-based prediction samples

[0130] The prediction unit of the encoding / decoding device can derive a reference sample from the surrounding reference samples of the current block using the intra-prediction mode, and can generate a predicted sample of the current block based on the reference sample.

[0131] For example, (i) a predicted sample can be derived based on the average or interpolation of neighboring reference samples of the current block, and (ii) a predicted sample can be derived based on a reference sample among the neighboring reference samples of the current block that is located in a specific (predicted) direction relative to the predicted sample. Case (i) may be called a non-directional mode or non-angular mode, and case (ii) may be called a directional mode or angular mode. Alternatively, the predicted sample may be generated by interpolation between the first and second neighboring samples, which are located in the opposite direction to the prediction direction of the intra-prediction mode of the current block, with respect to the predicted sample of the current block. In this case, it may be called linear interpolation intra-prediction (LIP). Furthermore, temporary predicted samples for the current block can be derived based on filtered peripheral reference samples, and predicted samples for the current block can be derived by performing a weighted sum on the temporary predicted samples and at least one reference sample derived by the intra-prediction mode from the existing peripheral reference samples, i.e., unfiltered peripheral reference samples. In this case, it may be called PDPC (Position dependent intra prediction). Alternatively, intra-prediction coding can be performed by selecting the reference sample line with the highest prediction accuracy from the peripheral multi-reference sample lines of the current block, deriving predicted samples using the reference samples located in the prediction direction on that line, and then instructing (signaling) the decoding device to use the reference sample line. In this case, it may be called MRL (multi-reference line intra prediction) or MRL-based intra-prediction. Additionally, the current block can be divided into vertical or horizontal subpartitions, and intra-prediction can be performed based on the same intra-prediction mode, but peripheral reference samples can be derived and used on a subpartition-by-subpartition basis.In other words, in this case, the intra-prediction mode for the current block is applied identically to the subpartition, but by deriving and utilizing surrounding reference samples on a subpartition-by-subpartition basis, the intra-prediction performance can be improved in some cases. Such a prediction method may be called IPS (intra sub-partitions) or IPS-based intra-prediction. The specific details will be described later. Furthermore, when the prediction direction based on the prediction sample indicates the area between surrounding reference samples, that is, when the prediction direction indicates a fractional sample position, the value of the prediction sample can be derived by interpolating multiple reference samples located around the prediction direction (around the fractional sample position).

[0132] The intra-prediction method described above may be called an intra-prediction type, distinct from the intra-prediction mode. The intra-prediction type may be referred to by various terms, such as an intra-prediction method or an additional intra-prediction mode. For example, the intra-prediction type (or additional intra-prediction mode, etc.) may include at least one of the LIP, PDPC, MRL, and ISP described above. Information regarding the intra-prediction type may be encoded by an encoding device and included in a bitstream, and signaled to a decoding device. Information regarding the intra-prediction type may be embodied in various forms, such as flag information indicating whether or not to apply each intra-prediction type, or index information indicating one of a plurality of intra-prediction types.

[0133] The MPM list for deriving the intra-prediction mode described above may be configured differently depending on the intra-prediction type. Alternatively, the MPM list may be configured commonly regardless of the intra-prediction type.

[0134] Derivation of peripheral reference samples

[0135] When intraprediction is applied to the current block, peripheral reference samples to be used for intraprediction of the current block may be derived. The peripheral reference samples of the current block may include a total of 2 x nH samples adjacent to the left boundary and bottom-left of the current block of nW x nH size, a total of 2 x nW samples adjacent to the top boundary and top-right of the current block, and one sample adjacent to the top-left of the current block. Alternatively, the peripheral reference samples of the current block may include upper peripheral samples for multiple columns and left peripheral samples for multiple rows. Furthermore, the peripheral reference samples of the current block may include a total of nH samples adjacent to the right boundary of the current block of nW x nH size, a total of nW samples adjacent to the bottom boundary of the current block, and one sample adjacent to the bottom-right of the current block.

[0136] On the other hand, when MRL (Multiple Reference Line) is applied, the reference sample may be located on lines 1 to 3 on the left / above side, rather than line 0 which is adjacent to the current block. In this case, the number of peripheral reference samples may increase further. The specific regions and numbers of peripheral reference samples will be described later.

[0137] On the other hand, when ISP (Intra Sub-Partitions) is applied, the aforementioned peripheral reference samples can be derived on a sub-partition basis.

[0138] DIMD(Decoder side intra mode derivation)

[0139] In DIMD, the intra-prediction may be derived as a weighted average of the planar and two induced directions. For this purpose, two angular modes are selected from the HoG (Histogram of Gradient) calculated from the surrounding pixels of the current block. Once the two modes are selected, their predictors (prediction blocks) and planar predictors are normally calculated, and then the weighted average may be used as the final predictor (final prediction block) of the current block. At this time, the corresponding amplitudes in the HoG are used for each of the two modes to determine the weights.

[0140] Since the induced intra-mode is included in the primary list of the intra-MPM, the DIMD process may be performed before the MPM list is constructed. The primary induced intra-mode of a DIMD block is saved with the block and may be used to construct the MPM list of surrounding blocks.

[0141] The DIMD chroma mode can use the DIMD induction method to derive the chroma intra-prediction mode of the current block based on the previously reconstructed peripheral Y, Cb, and Cr samples in the second peripheral row and column shown in Figure 12. Specifically, in order to build the HoG, horizontal and vertical gradients may be calculated not only for the previously reconstructed Cb and Cr samples, but also for each collocated previously reconstructed chroma sample of the current chroma block. Subsequently, chroma intra-prediction of the current chroma block may be performed using the intra-prediction mode with the largest histogram amplitude value.

[0142] If the intra-prediction mode derived from the DIMD chroma mode is the same as the intra-prediction mode derived from the DM mode, the intra-prediction mode with the second largest histogram amplitude value may be used as the DIMD chroma mode. A predetermined CU level flag may be signaled to indicate whether or not the above-mentioned DIMD chroma mode is applied.

[0143] Fusion for TIMD (Fusion for template-based intra-mode derivation)

[0144] For each intra-prediction mode within the MPM, the SATD between the template's predicted sample and the recovered sample may be calculated. The first two intra-prediction modes with the smallest SATD may then be selected as the TIMD modes. These two TIMD modes may be fused using weighted values, and such weighted intra-predictions may now be used to code the CU. The derivation of the TIMD modes may include the position-dependent intra-prediction combination (PDPC) described above.

[0145] While comparing the costs of the two selected modes described above with a predetermined threshold, a cost factor 2 may be applied as shown in Equation 1 below.

[0146]

number

[0147] If the condition in equation 1 is true, the fusion described above may be applied. Conversely, if the condition in equation 1 is false, only mode 1 may be used.

[0148] On the other hand, the weighted values ​​for the above modes may be calculated from the respective SATD costs as shown in the following equation 2.

[0149]

number

[0150] MIP(Matrix-based intra prediction)

[0151] Figures 7 to 11 illustrate the MIP (Matrix-based intracellular prediction) process applicable to this disclosure.

[0152] MIP (Matrix-based intra prediction) may also be called ALWIP (Affine linear weighted intra prediction) or MIP (or MWIP) (Matrix weighted intra prediction). To predict samples for a rectangular block with width W and height H, MIP takes as input one line consisting of H reconstructed samples adjacent to the left boundary of the block, and one line consisting of W reconstructed samples adjacent to the top boundary. If reconstructed samples are unavailable, they may be generated in the same way as they would be generated by existing intra-prediction methods.

[0153] The prediction signal can be generated in basically the following steps.

[0154] 1. Of the boundary samples, if W=H=4, 4 samples can be extracted by averaging; otherwise, 8 samples can be extracted (averaging process).

[0155] 2. The matrix-vector multiplication performed after the offset addition may be done using the averaged samples as input. As a result of this process, a reduced prediction signal for the subsampled sample set of the original block may be obtained (matrix-vector multiplication process).

[0156] 3. Predicted signals at the remaining positions may be generated from the predicted signals of the subsampling set by linear interpolation, which is single-step linear interpolation in each direction ((linear) interpolation process).

[0157] [Table A]

[0158] In matrix-vector product operations, the number of multiplication operations required may always be 4·W·H or less. In other words, in MIP mode, a maximum of four multiplication operations may be required for each sample.

[0159] Overview of the entire MIP process

[0160] The entire process of averaging, matrix-vector multiplication, and linear interpolation will be explained with reference to Figures 7 to 11. On the other hand, block forms not shown in Figures 7 to 11 may also be processed as shown in Figures 7 to 11.

[0161] 1. In a 4x4 block, the MIP may be calculated by taking two average values ​​along the axis of each boundary. The resulting four input samples may be input to a matrix vector multiplication. The matrix may be obtained from set S0. After offset addition, 16 final predicted samples may be obtained. Linear interpolation is not required to generate the predicted signal; that is, (4·16) / (4·4)=4 multiplications may be performed for each sample.

[0162] 2. In a 4x4 block, the MIP can take four average values ​​along the axis of each boundary. The resulting eight input samples may be input to a matrix vector multiplication. The matrix may be obtained from set S1. Thus, 16 samples may be obtained at odd positions in the prediction block. That is, (8·16) / (8·8)=2 multiplications may be performed for each sample. After offset addition, these samples may be vertically interpolated using the reduced upper boundary. Horizontal interpolation may be performed using the existing (original) left boundary. In this case, multiplication is not required in the interpolation process. That is, only a total of two multiplications per sample may be required to derive the MIP prediction.

[0163] In a 3.8x4 block, the MIP can take the four existing boundary values ​​of the left boundary and the four average values ​​along the horizontal axis of the boundary. The resulting eight input samples may be input to a matrix vector multiplication. The matrix may be obtained from set S1. Thus, 16 samples may be obtained at each vertical position and odd-numbered horizontal positions of the prediction block. That is, (8·16) / (8·4)=4 multiplications may be performed for each sample. After offset addition, these samples may be horizontally interpolated using the existing (original) left boundary. In this case, no additional multiplication process is required in the interpolation process. That is, only a total of four multiplications per sample may be necessary to derive the MIP prediction. Transposed cases may also be handled in this way.

[0164] 4. In a 16x16 block, the MIP can take four mean values ​​along each axis of the boundary. The resulting eight input samples may be input to a matrix vector multiplication. The matrix may be obtained from set S2. Thus, 64 samples may be obtained at odd positions in the prediction block. That is, each sample may undergo (8·64) / (16·16)=2 multiplications. After offset addition, these samples may be vertically interpolated using the eight mean values ​​at the upper boundary. Horizontal interpolation may be performed using the existing (original) left boundary. In this case, no additional multiplication process is required in the interpolation process. That is, only a total of two multiplications per sample may be necessary to derive the MIP prediction.

[0165] The process can be applied essentially the same way to larger blocks. In this case, it can be seen that the number of multiplication operations for each sample is less than 4.

[0166] In a Wx8 block where W>8, since samples are provided at odd horizontal and vertical positions, only horizontal interpolation may be required. In this case, to calculate the reduced prediction, a multiplication of (8·64) / (W·8)=64 / W may be performed for each sample. When W=16, no additional multiplication is required for linear interpolation, but when W>16, the number of additional multiplications per sample required for linear interpolation may be less than two. Therefore, the total number of multiplications per sample may be 4 or less.

[0167] Finally, in the case of a W x 4 block where W > 8, A k Assume that this is the matrix obtained by excluding all rows corresponding to odd items along the horizontal axis of the downsampled block. In this case, the output size is 32, and only horizontal interpolation can be performed. For the reduced prediction calculation, a multiplication of (8·32) / (W·4)=64 / W may be performed for each sample. If W=16, no additional multiplication is required, but if W>16, fewer than two multiplications may be required for each sample for linear interpolation. Therefore, the total number of multiplications may be 4 or less. The transposed case may also be handled in this way.

[0168] Boundary averaging

[0169] Figure 12 shows an example of a boundary averaging process applicable to this disclosure. For example, boundary averaging may be included in the MIP process.

[0170] [Table B]

[0171] For a 4x4 block, the definition may be as follows for 0 ≤ i < 2.

[0172]

number

[0173] In other cases, the block width W is W = 4.2 k If so, then for 0 ≤ i < 4, it may be defined as follows:

[0174]

number

[0175] [Table C]

[0176] On the other hand, if the variable mode indicates MIP mode, the following concatenation may be defined.

[0177]

number

[0178] Ultimately, in interpolating the subsampled prediction signal, for larger blocks, a corrected version of the averaged boundary, i.e., a second version, may be needed. That is, min(W,H)>8 and W≧H, W=8*2 l And for 0 ≤ i < 8, the following may be defined.

[0179]

number

[0180] Reduced prediction signal generation by matrix-vector multiplication

[0181] [Table D]

[0182]

number

[0183] Reduced prediction signal pred red may be derived by calculating a matrix-vector product and adding an offset.

[0184] [Formula]

[0185] Here, A may be a matrix having W red ·H red rows and 4 columns when W=H=4, and 8 columns in other cases. Here, b may be a vector of W red ·H red size.

[0186] The matrix A and the vector b may be obtained from any one of sets S0, S1, S2 as follows. The index idx=idx(W,H) may be defined as follows.

[0187] [Formula]

[0188] [Table E]

[0189] Finally, in the following cases, the reduced prediction signal may be replaced with a transposed signal.

[0190] [Table F]

[0191] pred red The total number of multiplications required for the calculation may be 4 when W=H=4, which is because A has 16 rows and 4 columns in this case. In other cases, A has 8 columns and W red ·H red rows, in which case 8·W red ·Hred It can be seen that up to 4·W·H multiplications are required. That is, in this case, pred red may require a maximum of 4 multiplications per sample to be calculated.

[0192] Linear interpolation

[0193] FIG. 13 is a diagram illustrating an example of a linear interpolation process applicable to the present disclosure. As an example, the linear interpolation process may be included in a MIP process. As an example, the interpolation process may be a linear interpolation or bilinear interpolation process. The interpolation process may include two stages consisting of 1) vertical interpolation and 2) horizontal interpolation as shown in FIG. 13. If W>=H, vertical linear interpolation may be applied first, and then horizontal linear interpolation may be applied subsequently. If W<H, horizontal linear interpolation may be applied first, and then vertical linear interpolation may be applied subsequently. In a 4x4 block, the interpolation process may be omitted.

[0194] In a W×H block where max(W,H)≧8, the prediction signal is obtained by linear interpolation from W red ×H red reduced prediction signal pred red may be generated based on. Based on the block form, linear interpolation may be applied in the vertical direction, the horizontal direction, or both directions. When linear interpolation is applied to both directions, it may be applied in the horizontal direction first, and when W<H, it may be applied in the vertical direction first.

[0195]

Table G

[0196]

Equation

[0197] In this case, the vertical linear interpolation prediction signal may be generated from the extended reduced prediction signal as follows.

[0198]

number

[0199] Examples

[0200] This disclosure relates to intra (in-screen) prediction, and more specifically, to a technique for performing adaptive Matrix-based Intra Prediction (MIP) under various conditions. According to one embodiment of this disclosure, Matrix-based Intra Prediction (MIP) may be performed using only a portion of the reference sample, or it may be performed adaptively based on the reference sample, conditions related to the current block, etc.

[0201] Figure 14 illustrates a MIP prediction method applicable to this disclosure. According to an example in Figure 14, for the prediction of the current block, a surrounding reference sample can be used as an input, and the predicted value of the current block can be obtained as an output through a neural network.

[0202] On the other hand, current MIP prediction can generate predicted blocks using all left- and top-edge reference samples around the block. However, the samples of the current block may have a more biased correlation with some reference samples, such as the top or left-edge reference samples. Therefore, predicting the current block using partial reference samples can yield higher prediction accuracy.

[0203] Accordingly, this disclosure proposes a method for performing MIP prediction adaptively to the reference sample region as various embodiments. Specifically, it proposes a method for performing MIP prediction using only a portion of the surrounding reference samples of the current prediction block.

[0204] Furthermore, this disclosure proposes, as various embodiments, a method for adaptively performing MIP prediction based on conditions related to a reference sample and / or the current block. More specifically, matters related to MIP application may be determined, such as the adaptive determination of MIP prediction result values ​​based on the size of the current block, the number of pixels in the current block and / or the values ​​of the reference sample, or the determination of a matrix for MIP prediction.

[0205] Furthermore, when describing various embodiments in this disclosure, even when singular expressions such as sample, pixel, picture, and block are used, they can refer to multiple samples, pixels, picture, and blocks, and of course, they can also refer to a single sample, pixel, picture, or block.

[0206] The embodiments of this disclosure will be described in detail below with reference to the drawings.

[0207] Figure 15 is a diagram illustrating a method for performing MIP prediction according to one embodiment of the present disclosure, and Figure 16 is a diagram illustrating an MIP-based video coding or decoding method according to one embodiment of the present disclosure. In this embodiment, a method for performing MIP prediction using only a portion of the reference samples from the current block will also be described with reference to various figures, including Figures 15 and 16.

[0208] On the other hand, the video encoding method or video decoding method shown in Figure 16 may be performed in a video encoder or video decoder, respectively, and the video encoder and video decoder may include the devices described above with reference to other figures.

[0209] First, an MIP-based video encoding / decoding method (S1610) according to one embodiment of the present disclosure may be performed. For example, the S1610 method may be performed based on a selected set of reference samples.

[0210] For example, to perform MIP-based video coding or decoding, some of the left-hand reference samples and top-edge reference samples may be selected. For example, the top-edge reference sample of the current block may be selected to be used for MIP (S1620). As mentioned above, pixels in the current block may have a more biased correlation with some reference samples, such as the top-edge or left-hand reference samples, and therefore, when predicting the current block using partial reference samples, a higher prediction accuracy can be obtained. Here, according to one embodiment of the present disclosure shown in Figure 15, only the top-edge reference sample can be selected for MIP prediction and used as the MIP prediction input value. In this case, only the top-edge reference sample may be selected as the neural network input value (input) for MIP prediction. On the other hand, as shown in Figure 15(a), the top-edge reference sample may correspond to a sample immediately adjacent to the current block. That is, according to the embodiment in Figure 15(a), only the reference sample directly above the current block can also be used. However, as another example, as shown in Figure 15(b), the top reference sample may include a number of adjacent samples adjacent to the top edge of the current block that are greater than the width of the current block (e.g., a form further extended to the right), or it may include reference samples of multiple reference sample lines. According to the embodiment in Figure 15(b), multiple lines of reference samples at the top edge and top-right of the current block can also be used. As yet another example, as shown in Figure 15(c), the top reference sample may include a number of adjacent samples adjacent to the top edge of the current block that are greater than the width of the current block (e.g., a form further extended to the left and / or right), or it may include reference samples of multiple reference sample lines. According to the embodiment in Figure 15(c), the top edge, top-right, or top-left reference sample of the current block can also be used. On the other hand, the number of reference samples further extended to the left or right, or the number of reference sample lines, is not limited to those disclosed in Figure 15.Furthermore, for the method proposed in this embodiment, some samples within one or more reference sample lines may be used for MIP prediction, and only a portion of all upper reference samples in the current block may be used for MIP prediction as proposed in this embodiment. On the other hand, it should be said that the method of using only a portion of the reference samples obtained by the examples listed above, as well as the case of using the upper and upper-right reference samples of the current block for MIP prediction of the current block, is also included as an embodiment of this disclosure. For example, only reference samples induced by subsampling (where the subsampling ratio may be any number) may be used, and a method of using only reference samples segmented based on a specific threshold may be used. For example, reference samples may be used for MIP prediction only if their value is above or below a specific threshold, or within a specific range, and the specific threshold may be signaled from the bitstream or induced based on other parameters (e.g., the size of the current block).

[0211] Subsequently, an MIP mode may be selected based on the MIP mode information (S1630). The MIP mode may be the one currently applied to the block, or it may be explicitly signaled in the bitstream. Alternatively, at this stage, the video encoder may, after selecting the MIP mode, proceed to the step of determining the MIP mode information.

[0212] Subsequently, a neural network-based MIP may be applied (S1640) based on the selected MIP mode. In other words, an MIP-based prediction may be performed. On the other hand, as an example, based on the selected reference sample input values, predicted block sample values, which are the output values, may be obtained through a neural network as shown in Figures 14 and / or 15. Here, the neural network may be adaptively selected based on the reference sample input values. In other words, step S1640 may include a process in which an MIP-based prediction is applied based on the selected input sample values ​​and the selected neural network. As an example, a neural network consisting of a single matrix may be used to perform the MIP. On the other hand, the predicted block sample values, which are the output values, may be derived as follows.

[0213]

number

[0214] In the above formula, r represents the reference sample input value selected by the method proposed in this embodiment, and pred represents the predicted block sample value, which is the output value. In other words, the predicted block sample value may be derived based on the selected reference sample value. For example, A may represent a trained neural network matrix, and the Clip function is a clipping function that adjusts (clips) the output predicted sample (pred) value within a certain sample value range, and may be a function for limiting the lower and / or upper limits. In this case, the clipping range may be predefined. That is, the output value pred may be obtained by performing an operation on a matrix A with respect to the input value r. The size of matrix A may be adaptively changed according to the sizes of the input and output values.

[0215] On the other hand, the neural network configuration for obtaining the output value pred for the input value r proposed in this embodiment is not limited to the above example. That is, in addition to a neural network using a single simple matrix, multiple matrix operations may be used by configuring multiple hidden layers, and bias terms and activation functions may be used for each neural network layer.

[0216] Furthermore, when outputting predicted samples, values ​​may be output in the frequency domain in addition to the spatial domain. In this case, an inverse transform may be performed on the predicted (pred) value to obtain the final predicted sample value.

[0217] On the other hand, as illustrated above, the size of the neural network matrix that may be included in the matrix for MIP may be adaptively selected depending on the input and output values. When MIP is applied to blocks of a certain size or larger, the size of the matrix and the input vector (e.g., r) become very large, which imposes a high computational complexity on the video decoding and / or encoding device. Therefore, in terms of coding efficiency and computational complexity, it may be required to reduce the size of the input vector and the MIP matrix (or kernel). Thus, in one embodiment of this disclosure, the selected input values, i.e., the selected reference samples, can be sampled by a method such as appropriate downsampling and then used as input to the neural network. That is, the selected reference samples can be sampled by downsampling at a ratio of 1:2, 1:4, or 1:8 and used as input values ​​to the neural network. In this case, since the size of the input values ​​is reduced, the size of the neural network matrix can be reduced, resulting in a significant reduction in the overall MIP computation.

[0218] On the other hand, the sampling method and ratio of the reference samples selected in this example are not limited to the above example, and the input value can be increased by upsampling in addition to downsampling. The sampling method and ratio of the reference samples can also be adaptively selected according to the shape and size of the block and the number of selected reference samples. Furthermore, it is possible to use the selected reference samples as neural network input values ​​after preprocessing them through various filtering methods such as low-pass filtering and high-pass filtering. It is also possible to apply a transform to the selected reference samples to change them to the frequency domain and then input them into the neural network, or to calculate the mean of the selected reference samples and use the residual sample value obtained by subtracting that mean from all reference samples as the neural network input value.

[0219] On the other hand, the filtering and sampling methods for input values ​​described above can be similarly applied to output values. That is, if the number of output values ​​in the neural network, i.e., the number of output samples, is less than or greater than the predicted number of samples required for the current block size, the output values ​​can be upsampled or downsampled to match the predicted number of samples required for the current block size. Furthermore, if a transformation is applied to the input values, and frequency domain input values ​​are used, or if residual samples other than the mean are used as input values, this can also be corrected in the output values.

[0220] As illustrated above, the number of input and output samples for the neural network matrix operation proposed in this disclosure is not limited to specific numbers.

[0221] Furthermore, post-processing filtering can be applied to the prediction blocks predicted by the method proposed in this disclosure. A Position-Dependent Intra Prediction Combination (PDPC) method can also be applied to the prediction blocks predicted by the method proposed in this disclosure. As an example, a smoothing filter can also be applied to the prediction blocks predicted by the method proposed in this invention.

[0222] On the other hand, as an example, when constructing a neural network for the method proposed in this embodiment, the coefficients of the matrices used in operations within the neural network may have various bit precisions. For example, when performing 10-bit precision matrix operations, the matrix coefficients may have a range of coefficient values ​​within a specific range (e.g., 0 to 1023 or -512 to 511), and the input and output values ​​may be adaptively adjusted accordingly. Such bit precision values ​​are not limited to the above examples and may be adaptively changed based on the number of layers in the neural network, etc.

[0223] On the other hand, in the method proposed in this embodiment, the number of neural matrix modes (the number of modes in the proposed MIP method) may be configured in various ways based on the mode (e.g., intra-mode or inter-mode), the width of the input block, the height of the input block, the number of pixels in the input block, the position of subblocks within the block, the explicitly signaled syntax elements, the statistical characteristics of surrounding pixels, and whether or not quadratic transformations are used. For example, the number of neural matrix modes can be adaptively selected according to the current block size. Alternatively, the same number of neural matrix modes can be selected for all block sizes.

[0224] For example, the coefficients of the neural network matrix may be defined adaptively for all block configurations. For example, neural network coefficients corresponding to each block shape from 4x4 to 256x256 blocks may be defined. The number of input and output values ​​of the neural network matrix may be predetermined depending on the block configuration. For example, as illustrated in Figure 15(a), MIP prediction is performed using only one upper reference sample line, and 49 types of neural network matrices may be required that match the input values ​​of 4, 8, 16, 32, 64, 128, 256 and the output values ​​of 4, 8, 16, 32, 64, 128, 256. Furthermore, each of the 49 neural network matrices may have matrix coefficients corresponding to the number of MIP modes proposed in this embodiment. For example, if the number of MIP modes in a neural network matrix with an input value of 8 and an output value of 16 (which may correspond to an 8x16 block configuration) is 35, then this neural network matrix may have a total of 8x16x35 matrix coefficients. Alternatively, as another example, only neural matrix matrices that fit a specific block shape may be defined, and may have limited input and output values. In this case, the neural matrix for the specific block shape may be utilized by appropriate pre-processing or post-processing of the input and output values. For example, if only neural matrix matrices that fit a square block shape are defined, the input values ​​of the square block neural matrix can be mapped by pre-processing a reference sample for a non-square block (e.g., downsampling or upsampling). The output values ​​can also be mapped to predicted values ​​for a non-square block by post-processing (e.g., downsampling or upsampling).

[0225] On the other hand, according to the method proposed in this embodiment, the neural matrix mode (the mode of the proposed MIP method) may be selected based on at least one of the following: mode (e.g., intermode or intramode), input block width, input block height, number of pixels in the input block, subblock position within the block, explicitly signaled syntax elements, statistical characteristics of surrounding pixels, and whether or not a quadratic transformation is used. In this case, the encoder can transmit the selected mode information to the decoder by selecting and applying an appropriate binary method. For example, the encoder can set a specific mode to MPM mode, such as the transmission of intramalma mode, and then perform binary conversion on the remaining modes after setting them separately. In this case, binary bits can be saved by appropriate context modeling during each binary conversion. Alternatively, as another example, mode information can be transmitted to the decoder using binary conversion such as truncated binary / truncated unary / fixed length, taking into account the total number of MIP modes.

[0226] On the other hand, in order to select the type of primary and secondary transformation in a predicted block predicted by the method proposed in this embodiment, the transformation mode may be selected based on at least one of the following: neural matrix mode, input block width, input block height, number of pixels in the input block, subblock position within the block, explicitly signaled syntax elements, and statistical characteristics of surrounding pixels. For example, when selecting the type of primary transformation for a block to which the prediction method proposed in this disclosure is applied, the MTS (multiple transform set) transformation method may be applied. For example, when selecting the type of secondary transformation for a block to which the prediction method proposed in this disclosure is applied, a secondary transformation kernel that matches the planar intra mode may be selected.

[0227] On the other hand, for the MIP mode selection proposed in this embodiment, candidate MIP prediction modes for the current block may be determined as in DIMD or TIMD, or the MIP prediction mode for the current block may be determined. That is, candidate MIP prediction modes or the MIP prediction mode for the block may be inferred by utilizing surrounding reference samples as in DIMD or TIMD.

[0228] As another example, the mode candidate analogy method used in DIMD for MIP mode prediction in the method proposed in this embodiment may be applied. That is, a pixel gradient can be calculated using only the region of the upper reference sample, and the first or second MIP prediction mode obtained by this gradient may be set as the MIP prediction mode candidate for the current block.

[0229] As another example, the mode inference method used in TIMD for MIP mode prediction in the method proposed in this embodiment may be applied. That is, TIMD template matching may be performed using only the region of the upper reference sample. The MIP prediction mode candidates to which TIMD template matching is applied may be selected as MPM, or all of the multiple MIP prediction modes may be selected, or only a portion of any number of the multiple MIP prediction modes may be selected.

[0230] On the other hand, whether or not the method proposed in this embodiment is applied may be signaled by information in HLS (high-level syntax) such as VPS, SPS, PPS, Picture Header, Slice Header, DCI, etc. As an example, in order to determine whether or not the method proposed in this embodiment is applied on a PPS basis, information indicating whether or not the method proposed in this embodiment is applied may be included and signaled within the PPS.

[0231] Additional information regarding whether the method proposed in this embodiment is applied may be explicitly signaled, but as another example, the method proposed in this embodiment may be inferred without signaling of additional information, or it may be adaptively selected. For example, whether or not the method proposed in this embodiment is applied may be signaled on a CTU or CU basis using a 1-bit flag.

[0232] For example, the method proposed in this embodiment may only be used if the MIP mode is defined to be used in HLS (high level syntax). Furthermore, the application of the method proposed in this embodiment may be determined by a method in which additional information is signaled within the MIP mode. For example, if the value of information regarding the application of MIP (e.g., MIP flag) is a specific value (e.g., 1, true), the information regarding the application of the method proposed in this embodiment may be signaled by a 1-bit flag.

[0233] Alternatively, if the method proposed in this embodiment can be applied depending on the presence or absence of a specific block size or shape or specific conditions, then the applicability of the method proposed in this embodiment may be signaled by a 1-bit flag only in that case. For example, if the height of the block is four times or more the width of the block, the method proposed in this embodiment does not need to be applied, and therefore the signaling of additional information regarding its applicability may be omitted.

[0234] On the other hand, as another example, under certain conditions, it is possible to implicitly infer whether or not the method proposed in this embodiment is applicable. For example, if there is no left-side reference sample for the current block, such as a block at the left-side boundary of the image, or if a left-side reference sample cannot be used, signaling information regarding whether or not the method proposed in this embodiment is applicable may be omitted. Signaling information regarding whether or not MIP mode is applied may also be omitted. In this case, it may be inferred that the MIP method proposed in this embodiment is always applied. If the left-side reference sample for the current block is a CTU boundary / tile boundary / slice boundary / sub-picture boundary, signaling information regarding whether or not the method proposed in this embodiment is applicable may be omitted. Signaling information regarding whether or not MIP mode is applied may also be omitted. In this case, it may be inferred that the MIP method proposed in this embodiment is always applied.

[0235] On the other hand, as an example, the decision of whether or not to apply the proposed method to this embodiment at a higher level defined in the HLS (high-level syntax) may adaptively determine whether or not to transmit information indicating whether or not the proposed method is applied to this embodiment at the coding unit level. For example, if the value of the information indicating whether or not the proposed method is applied to this embodiment, signaled in the SPS, is a specific value (e.g., 0, false) (i.e., indicating that the proposed method is not used at the SPS level), then the proposed method is not applied to this embodiment at the coding unit level either, and signaling of information indicating whether or not it is used is not required.

[0236] Figures 17 and 18 illustrate a method for performing MIP prediction according to one embodiment of the present disclosure, and Figure 19 illustrates a MIP-based video coding or decoding method according to one embodiment of the present disclosure. In this embodiment, a method for performing MIP prediction using only a portion of the reference samples from the current block will also be described with reference to all of Figures 17 to 19.

[0237] On the other hand, the video encoding method or video decoding method shown in Figures 19A and 19B may be performed in a video encoder or video decoder, respectively, and the video encoder and video decoder may include the devices described above with reference to other drawings. First, the video encoding method and / or video decoding method shown in Figure 19A will be described.

[0238] First, an MIP-based video encoding / decoding method (S1910) according to one embodiment of the present disclosure may be performed. For example, the S1910 method may be performed based on a selected set of reference samples.

[0239] For example, some of the left-hand reference samples and top-edge reference samples may be selected to perform MIP-based video encoding or decoding. For example, the left-hand reference samples of the current block may be selected to be used for MIP (S1920).

[0240] As mentioned above, the pixels of the current block may have a more biased correlation with respect to some reference samples, such as the top or left-hand reference sample. Therefore, when predicting the current block using partial reference samples, higher prediction accuracy can be obtained. Here, according to one embodiment of the present disclosure shown in Figure 18, only the left-hand reference sample can be selected for MIP prediction and used as the MIP prediction input value. In this case, only the left-hand reference sample may be selected as the neural network input value (input) for MIP prediction. On the other hand, as shown in Figure 18(a), the left-hand reference sample may correspond to a sample immediately adjacent to the current block. That is, according to the embodiment in Figure 18(a), only the reference sample directly to the left of the current block can be used. However, as another example, as shown in Figure 18(b), the left-hand reference sample may include a number of adjacent samples adjacent to the left of the current block that are greater than the height of the current block (for example, a form that extends further to the bottom), or the left-hand reference sample may include reference samples from multiple reference sample lines. According to the embodiment in Figure 18(b), multiple lines of reference samples at the top and left-bottom of the current block can also be utilized. As yet another example, as shown in Figure 18(c), the left-side reference samples may include a number of adjacent samples adjacent to the left side of the current block but exceeding the height of the current block (e.g., in a form further extended to the top and / or bottom), or may include reference samples from multiple reference sample lines. According to the embodiment in Figure 18(c), the left side, left-top, or left-bottom reference samples of the current block can also be utilized. On the other hand, the number of reference samples further extended to the top or bottom, or the number of reference sample lines, is not limited to those disclosed in Figure 18. Furthermore, for the method proposed in this embodiment, some samples within one or more reference sample lines may be utilized for MIP prediction, and only a portion of the entire left-side reference samples of the current block may be utilized for the MIP prediction proposed in this embodiment. On the other hand, the reference samples for MIP prediction may be reference samples on the left side and / or left-top of the current block, or they may be reference samples obtained by the examples described above.For example, only reference samples obtained by subsampling (where the subsampling ratio may be any arbitrary number) may be used, and a method of using only reference samples classified based on a specific threshold may be employed. The description of the specific threshold is the same as the content described above with reference to Figure 14, Figure 15, Figure 16, etc., and duplicate description thereof will be omitted.

[0241] Thereafter, the MIP mode may be selected based on the MIP mode information (S1930). The MIP mode may be applied to the current block, and may be explicitly signaled in the bitstream. Meanwhile, in the case of the current stage, the video encoder may replace the step with a step of determining the MIP mode information after selecting the MIP mode.

[0242] Thereafter, neural network-based MIP may be applied based on the selected MIP mode (S1940). In other words, MIP-based prediction may be performed. Meanwhile, as an example, based on the selected reference sample input value, as shown in Figure 14 and / or Figure 17, the prediction block sample value which is the output value may be obtained through a neural network. Here, the neural network may be one adaptively selected based on the reference sample input value. In other words, the step S1940 may include a process of applying MIP-based prediction based on the selected input sample value and the selected neural network.

[0243] As an example, the embodiments related to the configuration of the neural network, the neural network matrix, the processing of the input values and output values of the neural network, the signaling of information related to the embodiments, and / or the MIP prediction mode, etc. are as described above with reference to Figure 14, Figure 15, and Figure 16, and duplicate description thereof will be omitted.

[0244] One embodiment of the video encoding and / or video decoding method shown in Figure 19B will be described based on the embodiment described above with reference to Figures 14 to 19A. As an example, according to Figure 19B, it may be determined whether to use only the reference sample located at the top of the current block based on the embodiment in Figures 14 to 16, or to use only the reference sample located on the left side of the current block based on the embodiment in Figures 17 to 19A.

[0245] First, an MIP-based video encoding / decoding method (S1950) according to one embodiment of the present disclosure may be performed. For example, the S1950 method may be performed based on a selected set of reference samples.

[0246] For example, to perform MIP-based video coding or decoding (S1950), some of the left-hand reference samples and top-edge reference samples may be selected. For example, only one of the left-hand reference samples or top-edge reference samples in the current block may be selected to be used for MIP (S1960). For example, information regarding whether to use the top-edge reference sample or the left-hand reference sample as a reference sample may be explicitly signaled, but it may also be guided by an agreement between the encoder / decoder without other information signaling. For example, if there is no other information signaling, a TIMD template matching method is used, but TIMD template matching may be performed using only the top-edge reference sample region, or TIMD template matching may be performed using only the left-hand reference sample region. In this case, the template matching error values ​​of each reference sample may be compared, and the reference sample region with the smaller error value may be used as the input sample for MIP. At this time, the number of samples in each sample region may differ, and a process such as dividing the error value of each reference sample region by the number of samples in each sample region and comparing them may be added for accurate comparison.

[0247] Subsequently, an MIP mode may be selected based on the MIP mode information (S1970). The MIP mode may be the one currently applied to the block, or it may be explicitly signaled in the bitstream. Alternatively, at this stage, the video encoder may, after selecting the MIP mode, proceed to the step of determining the MIP mode information.

[0248] Subsequently, a neural network-based MIP may be applied based on the selected MIP mode (S1980).

[0249] The embodiment shown in Figure 19B may be applied to the same concepts described in Figures 14 to 19A, so any redundant explanations regarding this will be omitted.

[0250] Figures 20 and 21 are diagrams illustrating a method for performing MIP prediction according to one embodiment of the present disclosure, and Figure 22 is a diagram illustrating an MIP-based video coding or decoding method according to one embodiment of the present disclosure. In this embodiment, a method for performing MIP prediction using only a portion of the reference samples from the current block will also be described with reference to all of Figures 20, 21, and 22.

[0251] On the other hand, the video encoding method or video decoding method shown in Figure 22 may be performed in a video encoder or video decoder, respectively, and the video encoder and video decoder may include the devices described above with reference to other drawings.

[0252] First, an MIP-based video encoding / decoding method (S2210) according to one embodiment of the present disclosure may be performed. For example, the S2210 method may be performed based on a selected set of reference samples.

[0253] As an example, some of the left-side reference samples and top-edge reference samples may be selected for MIP-based video coding or decoding. On the other hand, only some of the left-side and top-edge reference samples of the current block may be selected to be used for MIP (S2220). That is, in the embodiments of Figures 20 and 21, some or extended left-side reference samples and some or extended top-edge reference samples may be selected and used. As mentioned above, pixels in the current block may have a more biased correlation with respect to some reference samples, such as top-edge or left-side reference samples, and therefore, when predicting the current block using partial or extended reference samples, a higher prediction accuracy can be obtained. Here, according to one embodiment of the present disclosure relating to Figure 21, some or extended reference samples from adjacent reference samples located on the left side or top edge can be selected and used as MIP prediction input values ​​for MIP prediction. In this case, some left-side reference samples and some top-edge reference samples may be selected as neural network input values ​​(input) for MIP prediction. On the other hand, as shown in Figure 21(a), the left-side reference sample and the top-end reference sample may correspond to samples immediately adjacent to the current block. That is, according to the embodiment in Figure 21(a), only the reference sample directly to the left of the current block can be used, or only the reference sample directly above the current block can be used. In other words, only reference samples adjacent to the current block may be used. However, as shown in Figure 21(a), although reference blocks immediately adjacent to the current block are used, the block located at the top of the left-side reference sample corresponding to the height of the current block is excluded, and the left-side reference sample further extended to the bottom may be included in the reference sample. Also, as shown in Figure 21(a), although reference samples immediately adjacent to the current block are used, the block located at the left of the top-end reference sample corresponding to the width of the current block is excluded, and the top-end reference sample further extended to the right may be included in the reference sample.However, as another example, as shown in Figure 21(b), the left-side reference sample may include a number of adjacent reference samples adjacent to the left side of the current block, but fewer than the number of samples corresponding to the height of the current block, or the left-side reference sample may include reference samples from multiple reference sample lines. Also, as shown in Figure 21(b), the top-end reference sample may include a number of adjacent reference samples adjacent to the top edge of the current block, but fewer than the number of samples corresponding to the width of the current block, or the top-end reference sample may include reference samples from multiple reference sample lines. Furthermore, the reference samples may include samples located at the top left edge of the current block. That is, a further extension of the top-end reference sample to the left, or a further extension of the left-side reference sample to the top, may be selected. According to the embodiment in Figure 21(b), multiple lines of reference samples on the left side, top edge, and / or left-top edge of the current block can also be used. On the other hand, as yet another example, as shown in Figure 21(c), the left-side reference samples of the current block include reference samples corresponding to the height of the current block, but the top-end reference samples may include only a smaller number of adjacent samples than the number of reference samples corresponding to the width of the current block. In other words, only a portion of the top-end adjacent reference samples may be included in the top-end reference samples.

[0254] On the other hand, the number of reference samples or reference sample lines that extend further to the upper or lower end is not limited to those disclosed in Figure 21. Also, for the method proposed in this embodiment, some samples within one or more reference sample lines may be utilized for MIP prediction, or only a portion of all left-hand reference samples or upper-end reference samples in the current block may be utilized for MIP prediction as proposed in this embodiment.

[0255] Subsequently, an MIP mode may be selected based on the MIP mode information (S2230). The MIP mode may be the one currently applied to the block, or it may be explicitly signaled in the bitstream. Alternatively, at this stage, the video encoder may, after selecting the MIP mode, proceed to the step of determining the MIP mode information.

[0256] Subsequently, a neural network-based MIP may be applied (S2240) based on the selected MIP mode. In other words, MIP-based prediction may be performed. On the other hand, as an example, based on the selected reference sample input values, predicted block sample values, which are output values, may be obtained through a neural network, as shown in Figure 14 and / or Figure 20. Here, the neural network may be adaptively selected based on the reference sample input values. In other words, step S2240 may include a process in which MIP-based prediction is applied based on the selected input sample values ​​and the selected neural network. As an example, examples relating to the neural network configuration, neural network matrix, processing of neural network input values ​​and output values, signaling of information related to the embodiment and / or MIP prediction mode have been described above with reference to Figures 14, 15, 16, 17, 18, 19A, 19B, 20, and 21, and a redundant explanation will be omitted.

[0257] On the other hand, additional information (e.g., 2 bits) may be signaled, and this information may determine which of the embodiments in Figures 14 to 19B is used. Alternatively, the method proposed in this embodiment may be determined by an agreement between the encoder / decoder without signaling. For example, the mode analogy method used in TIMD can be applied. That is, TIMD template matching may be performed using only the upper reference sample region, or using only the left reference sample. TIMD template matching may be performed using only some of the upper reference samples and / or some of the lower reference sample regions. Furthermore, the template matching error values ​​of each reference sample may be compared, and the reference sample region with the smaller error value may become the MIP input value. In this case, the number of samples in each sample region may be different, and an additional process may be added to divide the error value of each reference sample region by the number of samples in each sample region for accurate comparison.

[0258] On the other hand, information related to the MIP according to one embodiment of the present disclosure may be signaled. The following describes information that may be signaled with reference to an exemplary table, but the names and order of the information described may be changed, and it may be signaled at various levels (for example, SPS (Sequence parameter set), PPS (Picture Parameter set), Slice Header, Picture Header, CU (Coding Unit), PU (Prediction Unit), etc.).

[0259] Figure 23 is a diagram illustrating a video encoding method or a video decoding method that can be performed using a video encoding device or video decoding device according to one embodiment of the present disclosure.

[0260] As an example, according to one embodiment shown in Figure 23, MIP may be performed based on adaptive matrix selection (S2310). Below, we propose a method for adaptively selecting and applying an applicable neural network matrix, i.e., a neural network matrix, when predicting MIP.

[0261] To perform S2310, first, the block size of the current block to which MIP prediction can be applied may be compared with a specific threshold (S2320). For example, MIP prediction may be performed using a neural network composed of a single matrix, as shown in equation 12 above. In equation 12, r is a reference sample, meaning the input value is a reference sample, and pred can mean the output value is the predicted block sample value. A can mean the trained neural network matrix, and the Clip function can mean a function that adjusts the output predicted value pred to a certain range, i.e., clips it. That is, the output value pred may be obtained by a single matrix operation on the input value r. In this case, the size of the matrix, i.e., A, may be changed according to the sizes of the input and output values. In one embodiment of this disclosure, a method is proposed for adaptively selecting and applying the neural network matrix applied when predicting MIP, i.e., A in the example of equation 12 above. For example, the neural network matrix may be adaptively selected when the following specific conditions are met.

[0262] - Example 1: When the block size or the number of pixels in a block for which MIP prediction is performed is less than or greater than a specific size, a neural network matrix may be applied that does not apply sampling (e.g., downsampling and / or upsampling) to the input value (r) and output value (pred). Otherwise, a neural network matrix may be applied that does apply sampling to the input value and output value.

[0263] - Example 2: When the size or the number of pixels of a block performing MIP prediction is equal to or less than or equal to or greater than a specific size, a more elaborate and complex neural network matrix may be applied. The elaborate neural network matrix may include a matrix that is capable of processing a greater number of input values and generating output values, has a larger number of matrix factors, or requires an additional process to derive the matrix even when a single matrix A is used. On the other hand, in other cases, that is, when the block size is equal to or greater than or equal to or less than the specific size, a relatively simple neural network matrix may be used.

[0264] In the embodiment of Figure 23, when the size of the block is equal to or less than a specific threshold, an improved matrix may be used for MIP (S2330). On the other hand, when the size of the block is equal to or greater than the specific threshold, a relatively simple or general matrix may be used for MIP (S2340). However, the present disclosure is not limited to Figure 23, and a case opposite to that of Figure 23 or other criteria described above may also be used for selecting the MIP matrix, which should also be said to be included in the present disclosure.

[0265] On the other hand, in the above examples, the size of the current block may mean the width of the current block (number of horizontal pixels (block width)) or the height of the current block (number of vertical pixels (block height)), and may be defined as the number of pixels in the current block. As another example, the presence or absence of application of sampling (for example, downsampling or upsampling) or a factor value may be determined in consideration of the case where the block size is larger or smaller than a threshold value.

[0266] For example, even if the current block size is less than or equal to a specific size (e.g., 8x8), the block size may be relatively larger compared to adjacent blocks with large changes in pixel value distribution. Therefore, a neural network matrix that does not apply sampling to input and output values ​​may be used for MIP. In other words, for blocks of a specific size (e.g., 4x4, 4x8, 8x4, 8x8, etc.), sampling does not need to be applied to input and output values. In this case, a neural network matrix corresponding to the block size (e.g., 4x4, 4x8, 8x4, 8x8 blocks, etc.) may be required.

[0267] As an example, assuming that MIP prediction is performed using only the upper reference sample, as described above with reference to Figures 14 to 16 and Figure 19B, and assuming that the current block size is 8x8, then 8 input values ​​may be required, and 64 output values ​​may be required. In addition, a corresponding 8x64 size neural network matrix may be required.

[0268] For example, if the number of pixels in a block is less than or equal to a specific value (e.g., 128), a neural network matrix that does not apply sampling to input and output values ​​may be used for MIP. In other words, for blocks of a specific size (e.g., 4x4, 4x8, 4x16, 4x32, 8x4, 8x8, 8x16, 16x4, 16x8, 32x4, etc.), sampling does not need to be applied to input and output values, and a matrix corresponding to the specific size may be required.

[0269] As an example, assuming that MIP prediction is performed using only the left-hand reference sample, as described above with reference to Figures 17 to 19B, and assuming that the current block size is 8x16, then 16 input values ​​may be required, and 128 output values ​​may be required. In addition, a corresponding 16x128 size neural network matrix may be required.

[0270] On the other hand, applying the method proposed in this embodiment can be expected to improve MIP prediction performance in small block sizes. Furthermore, as the block size increases, the required neural network matrix factors, i.e., neural network matrix coefficients, increase geometrically. Therefore, applying the method proposed in this embodiment can efficiently suppress the increase in neural network matrix coefficient data, reduce computational complexity, and improve coding efficiency and quality.

[0271] On the other hand, this disclosure is not limited to the examples above for matrix configurations for obtaining the output value pred for an input value r, as proposed in some embodiments. That is, in addition to a neural network using one simple matrix, multiple matrix operations can be used with multiple hidden layer configurations, and bias terms and / or activation functions may be used for each neural network layer. Furthermore, in addition to spatial domain values, frequency domain values ​​can be selected as output values, in which case the final predicted sample value can be obtained by performing an inverse-transform of the predicted value, i.e., the pred value, which is the output value.

[0272] As illustrated above, the size of the neural network matrix may be adaptively selected according to the input and output values. In this embodiment, the input values ​​selected by the method proposed in the above example, i.e., the selected reference samples, can be sampled and used as neural network inputs. That is, as described above, the selected reference samples can be sampled at a specific ratio (e.g., a 1:2, 1:4, or 1:8 ratio). The sampling method and ratio of the selected reference samples in this embodiment are not limited to the above example; the input values ​​can also be increased / decreased by sampling, and the sampling method and ratio of the reference samples can be adaptively selected according to the block shape and size, and the number of selected reference samples. Furthermore, as described above, the selected reference samples can be preprocessed by various filtering methods before being used as neural network input values. Additionally, values ​​obtained by applying a transformation to the selected reference samples to change them to the frequency domain can be used as matrix input values, or residual sample values ​​can be used as input values, but this has been explained above, and a redundant explanation will be omitted.

[0273] The above-described filtering and sampling methods for input values ​​can be similarly applied to output values. Furthermore, if frequency domain input values ​​are used, or if residual samples are used as input values, these can be corrected in the output values, as described above, and a redundant explanation will be omitted.

[0274] As illustrated above, the number of input and output samples for matrix operations is not limited to specific numbers.

[0275] Furthermore, as mentioned above, post-processing filtering can also be applied to the predicted blocks. This is explained above, and a redundant explanation will be omitted.

[0276] Furthermore, as mentioned above, when constructing a neural network, the matrix factors, or coefficients, may have various bit precisions. This has been explained above, and a redundant explanation will be omitted.

[0277] On the other hand, as mentioned above, the number of MIP prediction modes can be configured in various ways based on the prediction mode (e.g., intra mode, inter mode), the width of the input block, the height of the input block, the number of pixels in the input block, the position of subblocks within the block, explicitly signaled syntactic elements, the statistical properties of surrounding pixels, and whether or not quadratic transformations are used, and a specific prediction mode may be selected based on these factors. As an example, the number of MIP prediction modes can be adaptively selected according to the current block size. Alternatively, the neural matrix modes, i.e., the number of MIP prediction modes, may be the same for all block sizes. This has been explained above, and a redundant explanation will be omitted.

[0278] On the other hand, in an encoder, in order to transmit the selected mode information to the decoder, the mode information can be binary-coded using an appropriate binary-coded method before being transmitted to the decoder. This has been explained above, and a redundant explanation will be omitted.

[0279] Furthermore, the selection of primary and secondary transformation types, MIP mode selection (e.g., TIMD or DIMD-based), signaling of related information (e.g., HLS), and / or induction of related information in the prediction block predicted by the method proposed in this embodiment are as described above, and a redundant explanation will be omitted.

[0280] On the other hand, while the embodiment in Figure 23 states that the improved matrix is ​​used for MIP when the block size is below a certain threshold, this disclosure is not limited to this. The opposite case is also possible, where the improved matrix is ​​used for MIP when the block size is above a certain threshold, and this should also be considered an embodiment of this disclosure.

[0281] Figure 24 is a diagram illustrating a video encoding method or video decoding method that can be performed by a video encoding device or video decoding device according to one embodiment of the present disclosure. In this embodiment, an adaptive MIP method for specific input values ​​is proposed when predicting MIP.

[0282] As an example, according to one embodiment shown in Figure 24, regardless of whether MIP is permitted and / or applied, it may be further determined (S2410) whether MIP is actually performed depending on a specific input value. In other words, even if the permission and / or application of MIP is determined by other information or other parameters, we propose a method in which MIP is not actually performed and the output value is limited to a specific value if the MIP input value satisfies certain conditions.

[0283] To perform S2410, a comparison may first be made between the input values ​​for MIP prediction (e.g., input values) and a reference sample for MIP prediction (S2420). For example, it may be determined whether all input values ​​are 0 or not. Alternatively, prior to the S2420 process, a process may be carried out in which a reference sample is selected and the input values ​​for MIP (e.g., input vectors) are determined.

[0284] As an example, as mentioned above, MIP prediction can be performed using a neural network composed of a single matrix, as shown in Equation 12. However, when the input value r is used in the prediction, we propose a method for adaptively performing MIP prediction.

[0285] - Example 1: If all the reference sample values ​​selected for MIP prediction are the same, or if all the input values ​​for MIP prediction are a specific value (e.g., 0) (e.g., all input vectors are 0), the result of multiplying the input value by the matrix (e.g., matrix A in equation 12) will always be a specific value (e.g., 0), so MIP prediction can be omitted without performing extra complex matrix multiplication operations. In this case, all output values ​​can be set to a specific value k. Here, the value of k may be adaptively changed depending on the current MIP mode and other parameters. As an example, if all the input values ​​for MIP prediction are 0, the output value can be obtained by the following equation 13.

[0286]

number

[0287] In equation 13, output is the output value, bitdepth is information related to the bit depth, mode_num is the current MIP mode number, and max_mode_num can be said to be the total number of MIP modes.

[0288] For example, if all MIP input values ​​in a 16x16 block are 0, the bit depth is 10, there are MIP modes from 0 to 6 (max_mode_num=6), and the current MIP mode is 3 (mode_num=3), then applying the above example formula will result in an output value of 512.

[0289] - Example 2: If all the reference sample values ​​selected for MIP prediction are the same, or if all the input values ​​for MIP prediction are a specific value (for example, if all input vectors are 0), MIP prediction can be performed after setting the first input value as shown in the following formula.

[0290]

number

[0291] In equation 14 above, input[x] represents the x-th value of the input value (e.g., the input vector), bitdepth represents the bit depth of the input pixel value, and ref[0] represents the first value of the reference sample selected for MIP. The A value can represent a specific value that has been set in advance, but according to equation 14, the MIP prediction may be performed after replacing the initial input value with a value other than 0. The A value may be a predefined value, or it may be fixed to a value of (1 << (bitdepth-1)) as an example, or it may be adaptively changed based on the current MIP mode or other parameters.

[0292] - Example 3: When all the reference sample values ​​selected for MIP prediction are the same, or when all the input values ​​for MIP prediction are a specific value (e.g., if 0, all input vectors are 0), the output values ​​can all be set to a specific value k. In this case, the MIP mode is limited to one, and additional MIP mode signaling may be omitted. For example, the output value k may be selected as the average of the selected reference sample values, or it may be set to the same value as in other examples (e.g., Example 1 or Example 2). Also, in this example, since the MIP mode is unified to one, signaling of additional information for the mode may be omitted.

[0293] In the embodiment shown in Figure 24, if all MIP prediction input values ​​are 0, the MIP application rule based on the example described above may be applied (S2430). In other words, if the MIP prediction input values ​​are 0, MIP may not be performed, and the output value may be determined to a specific value. Otherwise, MIP based on other embodiments described above may be applied (S2440), which may be one of the MIP embodiments described above with reference to Figures 14 to 23, etc.

[0294] However, this disclosure is not limited to Figure 24, and other criteria for input values ​​may also be used to determine whether or not MIP is performed, and these should also be considered included in this disclosure.

[0295] The following describes a video decoding method performed by a video decoding device (decoder) and a video encoding method performed by a video encoding device (encoder) according to one embodiment of the present disclosure, with reference to Figures 25 and 26. As an example, Figures 25 and 26 may be based on the embodiment described above with reference to Figures 14 to 24, and explanations that overlap with the above description of the embodiment will be omitted below.

[0296] Figure 25 shows a video decoding method performed by a video decoding device (decoder) according to one embodiment of the present disclosure.

[0297] As an example, in a video decoding method performed by a video decoding device, the prediction mode of the current block may be determined (S2510), and this step may include whether an intra-prediction mode or an inter-prediction mode is applied to the current block.

[0298] Based on the fact that the prediction mode currently applied to the block is an intra-prediction mode, information related to MIP (Matrix-based Intra Prediction) may be derived (S2520). This step may include a decision on whether or not MIP is permitted, whether or not MIP is applied, and if it is determined that MIP is applied, it may include a process of deriving information related to the applied MIP.

[0299] On the other hand, as an example, MIP may be applied based on the size of the current block or the number of pixels contained in the current block. More specifically, the matrix for MIP may be determined based on the size of the current block or the number of pixels contained in the current block. On the other hand, if it is determined that MIP should be applied to the current block based on information related to MIP, the predicted block values ​​of the current block may be restricted to specific values ​​based on certain conditions. The specific conditions may be based on the reference sample values ​​of the current block for MIP. As an example, if certain conditions are met, MIP application to the current block may be omitted regardless of the information related to MIP. On the other hand, if it is determined that MIP should be applied to the current block based on information related to MIP, the predicted block of the current block may be generated using only the top reference sample or the left reference sample of the current block. On the other hand, the top reference sample may include the top right reference sample of the current block, and the left reference sample may include the bottom left reference sample of the current block. As another example, the predicted block of the current block is generated using a portion of the left-hand reference samples and / or a portion of the top-edge reference samples of the current block, and at least one of the left-hand or top-edge reference samples may be partially adjacent to the current block. On the other hand, if it is decided that MIP should be applied to the current block based on information related to MIP, the position of the reference samples of the current block is determined, and the position of the reference samples of the current block may be guided based on information signaled from the bitstream.

[0300] On the other hand, since Figure 25 corresponds to one embodiment of the present disclosure, it is obvious that some steps may be changed, some steps may be in a different order, some steps may be omitted, or some steps may be performed simultaneously, and this is also included in one embodiment of the present disclosure.

[0301] Figure 26 shows a video encoding method performed by a video encoding device according to one embodiment of the present disclosure.

[0302] According to a video encoding method performed by a video encoding device according to one embodiment of the present disclosure, it may be determined (S2610) whether or not to apply MIP (matrix-based intra prediction) to the current block. Prior to step S2610, a process may be performed to determine what prediction mode (e.g., intra prediction mode or inter prediction mode) is to be applied to the current block. Subsequently, a step may be performed to determine information related to MIP (S2620).

[0303] On the other hand, as described above, MIP may be applied based on the size of the current block or the number of pixels contained in the current block, and more specifically, the matrix for MIP may be determined based on the size of the current block or the number of pixels contained in the current block. On the other hand, if it is decided that MIP should be applied to the current block, the predicted block values ​​of the current block may be restricted to specific values ​​based on certain conditions. These specific conditions may be based on the reference sample values ​​of the current block for MIP. As an example, if certain conditions are met, MIP application to the current block may be omitted regardless of whether MIP is applied or not. On the other hand, if it is decided that MIP should be applied to the current block, the predicted block of the current block may be generated using only the top reference sample or left reference sample of the current block. On the other hand, the top reference sample may include the right top reference sample of the current block, and the left reference sample may include the left bottom reference sample of the current block. As another example, the predicted block of the current block may be generated using a portion of the left reference sample and / or a portion of the top reference sample of the current block, and at least one of the left reference sample or top reference sample may be partially adjacent to the current block. On the other hand, if it is determined that MIP is applied to the current block, the position of the reference sample in the current block is determined, but information regarding the position of the reference sample in the current block may be encoded and signaled in a bitstream.

[0304] On the other hand, as an example, the bitstream generated by the video encoding method may be stored on a non-temporary computer-readable medium.

[0305] As an example, a method for transmitting a bitstream generated by a video encoding method may also be proposed.

[0306] On the other hand, since Figure 26 corresponds to one embodiment of the present disclosure, it is obvious that some steps may be changed, some steps may be in a different order, some steps may be omitted, or some steps may be performed simultaneously, and this is also included in one embodiment of the present disclosure.

[0307] The exemplary methods in this disclosure are expressed as a series of actions for clarity of explanation, but this is not intended to limit the order in which the steps are performed, and each step may be performed simultaneously or in a different order, if necessary. To embody the methods relating to this disclosure, the exemplary steps may further include other steps, or the remaining steps may be included with some steps excluded, or yet another set of steps may be included with some steps excluded.

[0308] In this disclosure, a video encoding device or video decoding device that performs a predetermined operation (stage) may perform an operation (stage) to confirm the conditions or status of the execution of said operation (stage). For example, if it is stated that a predetermined operation is performed when a predetermined condition is met, the video encoding device or video decoding device may perform an operation to confirm whether or not the predetermined condition is met, and then perform the predetermined operation.

[0309] The various embodiments of this disclosure are not intended to enumerate all possible combinations, but rather to illustrate representative aspects of this disclosure. The matters described in the various embodiments may be applied independently or in combination of two or more.

[0310] Furthermore, various embodiments of this disclosure may be embodied in hardware, firmware, software, or a combination thereof. In the case of hardware embodiment, they may be embodied in one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.

[0311] Furthermore, the video decoding and video encoding devices to which the embodiments of this disclosure are applied may be included in multimedia broadcasting transceivers, mobile communication terminals, home cinema video equipment, digital cinema video equipment, surveillance cameras, video conferencing equipment, real-time communication equipment such as video communication, mobile streaming equipment, storage media, camcorders, video-on-demand (VoD) service providers, over-the-top (OTT) video equipment, internet streaming service providers, 3D video equipment, video-phone video equipment, and medical video equipment, and may be used to process video signals or data signals. For example, over-the-top (OTT) video equipment may include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, and digital video recorders (DVRs).

[0312] Figure 27 illustrates a content streaming system to which the embodiments of this disclosure can be applied.

[0313] As shown in Figure 27, a content streaming system to which an embodiment of the present disclosure is applied may broadly include an encoding server, a streaming server, a web server, media storage, user equipment, and multimedia input devices.

[0314] The encoding server is responsible for compressing content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream, and transmitting this bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, or camcorder directly generates the bitstream, the encoding server may be omitted.

[0315] The bitstream may be generated by a video encoding method and / or video encoding apparatus to which an embodiment of the present disclosure is applied, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0316] The streaming server transmits multimedia data to user devices based on user requests via a web server, and the web server can act as an intermediary to inform users of available services. When a user requests a desired service from the web server, the web server transmits it to the streaming server, and the streaming server can transmit multimedia data to the user. In this case, the content streaming system may include a separate control server, in which case the control server can play a role in controlling commands and responses between the devices within the content streaming system.

[0317] The streaming server can receive content from media storage and / or encoding servers. For example, when receiving content from the encoding server, the content can be received in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0318] Examples of user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (such as smartwatches, smart glasses, and HMDs), digital TVs, desktop computers, and digital signage.

[0319] Each server within the aforementioned content streaming system may be operated as a distributed server, in which case the data received by each server may be processed in a distributed manner.

[0320] The scope of this disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that enable the operation of various embodiments to be performed on a device or computer, and non-transitory computer-readable medium on which such software or instructions are stored and executable on a device or computer.

[0321] [Industrial applicability] The embodiments described herein can be used for encoding / decoding video.

[0322] [Claims when filing an international application] [Claim 1] A video decoding method performed by a video decoding device, Currently, we are in the stage of determining the prediction mode for the block; The process includes: a step of deriving information related to MIP (Matrix-based Intra Prediction) based on the prediction mode being an intra-prediction mode; The MIP is an image decoding method applied based on the size of the current block or the number of pixels contained in the current block. [Claim 2] The video decoding method according to claim 1, wherein the matrix for the MIP is determined based on the size of the current block or the number of pixels contained in the current block. [Claim 3] The video decoding method according to claim 1, wherein, based on information related to the MIP, it is determined that the MIP is applied to the current block, and based on certain conditions, the value of the predicted block of the current block is restricted to a specific value. [Claim 4] The video decoding method according to claim 3, wherein the aforementioned specific conditions are based on sample values ​​of the current block for the MIP. [Claim 5] The video decoding method according to claim 4, wherein if the aforementioned specific conditions are met, the application of the MIP to the current block is omitted regardless of the information related to the MIP. [Claim 6] The video decoding method according to claim 1, wherein, if it is determined that the MIP is applied to the current block based on the information related to the MIP, the predicted block of the current block is generated using only the upper end reference sample of the current block. [Claim 7] The video decoding method according to claim 6, wherein the upper end reference sample includes the upper right end reference sample of the current block. [Claim 8] The video decoding method according to claim 1, wherein, if it is determined that the MIP is applied to the current block based on the information related to the MIP, the predicted block of the current block is generated using only the left-hand reference sample of the current block. [Claim 9] The video decoding method according to claim 8, wherein the left-side reference sample includes the lower-left reference sample of the current block. [Claim 10] The predicted block of the current block is generated using a portion of the left-side reference sample and the top-end reference sample of the current block. The video decoding method according to claim 1, wherein at least one of the left-side reference sample or the upper-end reference sample is partially adjacent to the current block. [Claim 11] If it is determined that the MIP is applied to the current block based on the information related to the MIP, the position of the reference sample in the current block is determined. The video decoding method according to claim 1, wherein the position of the reference sample of the current block is derived based on information signaled from the bitstream. [Claim 12] A video encoding method, Currently, we are at the stage of deciding whether or not to apply MIP (matrix-based intracellular prediction) to the block; The step includes determining information related to the aforementioned MIP; The MIP is a video encoding method applied based on the size of the current block or the number of pixels contained in the current block. [Claim 13] A non-temporary computer-readable medium on which a bitstream generated by a video encoding method is stored, The aforementioned video encoding method is Currently, we are at the stage of deciding whether or not to apply MIP (matrix-based intracellular prediction) to the block; The step includes determining information related to the aforementioned MIP; The aforementioned MIP is applied based on the size of the current block or the number of pixels contained in the current block. [Claim 14] A method for transmitting a bitstream, The process includes the step of transmitting a bitstream generated by a video encoding method; The aforementioned video encoding method is Currently, we are at the stage of deciding whether or not to apply MIP (matrix-based intracellular prediction) to the block; The step of determining information related to the aforementioned MIP; A method by which the MIP is applied based on the size of the current block or the number of pixels contained in the current block.

Claims

1. A video decoding method performed by a video decoding device, The current stage is to determine the prediction mode for the block; The process includes: a step of deriving information related to MIP (Matrix-based Intra Prediction) based on the fact that the prediction mode is an intra prediction mode; The MIP is an image decoding method applied based on the size of the current block or the number of pixels contained in the current block.

2. The video decoding method according to claim 1, wherein the matrix for the MIP is determined based on the size of the current block or the number of pixels contained in the current block.

3. The video decoding method according to claim 1, wherein, based on information related to the MIP, it is determined that the MIP is applied to the current block, and based on certain conditions, the value of the predicted block of the current block is restricted to a specific value.

4. The video decoding method according to claim 3, wherein the aforementioned specific conditions are based on sample values ​​of the current block for the MIP.

5. The video decoding method according to claim 4, wherein if the above-mentioned specific conditions are met, the application of the MIP to the current block is omitted regardless of the information related to the MIP.

6. The video decoding method according to claim 1, wherein, if it is determined that the MIP is applied to the current block based on the information related to the MIP, the predicted block of the current block is generated using only the upper end reference sample of the current block.

7. The video decoding method according to claim 6, wherein the upper end reference sample includes the upper right end reference sample of the current block.

8. The video decoding method according to claim 1, wherein, if it is determined that the MIP is applied to the current block based on the information related to the MIP, the predicted block of the current block is generated using only the left-hand reference sample of the current block.

9. The video decoding method according to claim 8, wherein the left-side reference sample includes the lower-left reference sample of the current block.

10. The predicted block of the current block is generated using a portion of the left-side reference sample and the top-end reference sample of the current block. The video decoding method according to claim 1, wherein at least one of the left-side reference sample or the upper-end reference sample is partially adjacent to the current block.

11. If it is determined that the MIP is applied to the current block based on the information related to the MIP, the position of the reference sample in the current block is determined. The video decoding method according to claim 1, wherein the position of the reference sample of the current block is guided based on information signaled from the bitstream.

12. A video encoding method, The current stage is to decide whether or not to apply MIP (matrix-based intra prediction) to the block; The step includes determining information related to the aforementioned MIP; The MIP is a video encoding method applied based on the size of the current block or the number of pixels contained in the current block.

13. A non-temporary computer-readable medium on which a bitstream generated by a video encoding method is stored, The aforementioned video encoding method is The current stage is to decide whether or not to apply MIP (matrix-based intra prediction) to the block; The step includes determining information related to the aforementioned MIP; The MIP is applied based on the size of the current block or the number of pixels contained in the current block.

14. A method for transmitting a bitstream, The process includes the step of transmitting a bitstream generated by a video encoding method; The aforementioned video encoding method is The current stage is to decide whether or not to apply MIP (matrix-based intra prediction) to the block; The step of determining information related to the aforementioned MIP; A method in which the MIP is applied based on the size of the current block or the number of pixels contained in the current block.

Images