Image encoding / decoding method and device, and recording medium for storing bitstream

Abstract

Provided are an image encoding / decoding method and device, a recording medium for storing a bitstream, and a transmission method. The image decoding method may comprise the steps of: deriving a first intra prediction mode, a second intra prediction mode, and a third intra prediction mode of a current block from among a plurality of candidate intra prediction modes on the basis of a current template including adjacent samples of the current block; deriving a first prediction block, a second prediction block, and a third prediction block of the current block on the basis of the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode, respectively; and deriving a final prediction block of the current block by performing a weighted sum of the first prediction block, the second prediction block, and the third prediction block, wherein the third intra prediction mode is derived as one of a DC mode, a planar mode, and a neural network-based intra prediction (NNIP) mode, and the neural network-based intra prediction mode is performed on the basis of a neural network model that takes the adjacent samples of the current block as inputs.

Description

Video encoding / decoding method, device, and recording medium storing bitstream

[0001] The present disclosure relates to a video encoding / decoding method, a device, and a recording medium storing a bitstream. Specifically, the present disclosure relates to a video encoding / decoding method, a device, and a recording medium storing a bitstream based on a template-based intra mode derivation (TIMD) method considering neural network-based intra prediction (NNIP).

[0002] Recently, the demand for high-resolution, high-quality images, such as Ultra High Definition (UHD) images, is increasing across various application fields. Furthermore, interest in and demand for immersive media, such as Virtual Reality (VR) content, Artificial Reality (AR) content, and holograms, is also increasing. Furthermore, broadcasting of images with characteristics different from reality, such as game images, is also on the rise. As image data increases in resolution and quality, the relative amount of data increases compared to conventional image data. Therefore, transmitting image data using existing media such as wired and wireless broadband lines or storing it using existing storage media increases transmission and storage costs. To address these issues arising from the increasing resolution and quality of data, highly efficient image encoding / decoding technologies for higher-resolution and higher-quality images are required.

[0003] Template-based intra mode derivation (TIMD), used in image encoding / decoding technology, is a method of predicting a template based on arbitrary angular or non-angular intra prediction candidate modes and generating a prediction signal through the template.

[0004] According to the TIMD method, there is a limitation that template analysis is possible only by directional or non-directional intra prediction mode that generates prediction values ​​based on existing fixed rules.

[0005] The present disclosure aims to provide a video encoding / decoding method and device with improved encoding / decoding efficiency.

[0006] In addition, the present disclosure aims to provide a recording medium storing a bitstream generated by an image decoding method or device according to the present disclosure.

[0007] In addition, the present disclosure aims to provide a method for considering neural network-based intra prediction (NNIP) in a template-based intra prediction mode derivation method to solve the above problems.

[0008] A video decoding method according to one embodiment of the present disclosure includes the steps of: deriving a first intra prediction mode, a second intra prediction mode, and a third intra prediction mode of a current block based on a current template including adjacent samples of the current block from among a plurality of candidate intra prediction modes; deriving a first prediction block, a second prediction block, and a third prediction block of the current block based on each of the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode; and deriving a final prediction block of the current block by weighting the first prediction block, the second prediction block, and the third prediction block, wherein the third intra prediction mode is derived as any one of a DC mode, a Planar mode, and a Neural-network based intra prediction (NNIP) mode, and the Neural-network based intra prediction mode may be performed based on a neural network model that inputs adjacent samples of the current block.

[0009] In the above image decoding method, a step of obtaining information indicating whether the neural network-based intra prediction mode is used is further included, and when the information indicates that the neural network-based intra prediction mode is used, the third intra prediction mode can be derived as the neural network-based intra prediction mode.

[0010] In the above image decoding method, the step of deriving the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode may be performed based on a similarity between the predicted value of the current template based on each of the plurality of candidate intra prediction modes and the current template.

[0011] In the above image decoding method, the third intra prediction mode can be derived based on the similarity between some pixels included in the current template and the predicted values ​​of the some pixels based on the neural network-based intra prediction mode.

[0012] In the above image decoding method, the current template may include at least one of a left template and an upper template.

[0013] In the above image decoding method, the similarity may be determined based on any one of the SATD (sum of absolute transformed differences) method, the SAD (sum of absolute differences) method, the SSD (sum of squared differences) method, and the MR-SAD (mean removed sum of absolute differences) method.

[0014] In the above image decoding method, the plurality of candidate intra prediction modes may be classified into a first candidate intra prediction mode group and a second candidate intra prediction mode group, the first intra prediction mode and the second intra prediction mode may be derived based on the first candidate intra prediction mode group, and the third intra prediction mode may be derived based on the second candidate intra prediction mode group.

[0015] In the above image decoding method, the first candidate intra prediction mode group may include a directional intra prediction mode, and the second candidate intra prediction mode group may include a DC mode, a Planar mode, and a neural network-based intra prediction mode.

[0016] In the image decoding method, the weight applied to the first prediction block may be determined based on the similarity between the predicted value of the current template predicted based on the first intra prediction mode and the current template, the weight applied to the second prediction block may be determined based on the similarity between the predicted value of the current template predicted based on the second intra prediction mode and the current template, and the weight applied to the third prediction block may be determined based on the similarity between the predicted value of the current template predicted based on the third intra prediction mode and the current template.

[0017] In the above image decoding method, the current block may be a luminance block.

[0018] A video encoding method according to one embodiment of the present disclosure includes the steps of: deriving a first intra prediction mode, a second intra prediction mode, and a third intra prediction mode of a current block based on a current template including adjacent samples of the current block from among a plurality of candidate intra prediction modes; deriving a first prediction block, a second prediction block, and a third prediction block of the current block based on each of the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode; and deriving a final prediction block of the current block by weighting the first prediction block, the second prediction block, and the third prediction block, wherein the third intra prediction mode is derived as one of a DC mode, a Planar mode, and a Neural-network based intra prediction (NNIP) mode, and the Neural-network based intra prediction mode may be performed based on a neural network model that inputs adjacent samples of the current block.

[0019] A non-transitory computer-readable recording medium storing a bitstream generated by an image encoding method according to one embodiment of the present disclosure can store the bitstream generated by the image encoding method.

[0020] A bitstream transmission method according to one embodiment of the present disclosure can transmit a bitstream generated by the image encoding method.

[0021] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0022] According to the present disclosure, a video encoding / decoding method and device with improved encoding / decoding efficiency can be provided.

[0023] Additionally, according to the present disclosure, a method for considering neural network-based intra prediction in deriving a template-based intra prediction mode can be provided.

[0024] Additionally, according to the present disclosure, prediction accuracy can be improved.

[0025] Additionally, according to the present disclosure, the amount of bits transmitted can be saved and coding efficiency can be improved.

[0026] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0027] FIG. 1 is a block diagram showing a configuration according to one embodiment of an encoding device to which the present disclosure is applied.

[0028] FIG. 2 is a block diagram showing a configuration according to one embodiment of a decryption device to which the present disclosure is applied.

[0029] FIG. 3 is a diagram schematically illustrating a video coding system to which the present disclosure can be applied.

[0030] FIG. 4 is a diagram for explaining a template-based intra prediction mode derivation method according to one embodiment of the present disclosure.

[0031] FIG. 5 is a diagram for explaining a neural network-based intra prediction method according to one embodiment of the present disclosure.

[0032] FIG. 6 is a diagram for explaining a cost value calculation method of NNIP mode according to one embodiment of the present disclosure.

[0033] FIG. 7 is a flowchart illustrating a decryption method according to one embodiment of the present disclosure.

[0034] FIG. 8 is a diagram exemplifying a content streaming system to which an embodiment according to the present disclosure can be applied.

[0035] A video decoding method according to one embodiment of the present disclosure includes the steps of: deriving a first intra prediction mode, a second intra prediction mode, and a third intra prediction mode of a current block based on a current template including adjacent samples of the current block from among a plurality of candidate intra prediction modes; deriving a first prediction block, a second prediction block, and a third prediction block of the current block based on each of the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode; and deriving a final prediction block of the current block by weighting the first prediction block, the second prediction block, and the third prediction block, wherein the third intra prediction mode is derived as any one of a DC mode, a Planar mode, and a Neural-network based intra prediction (NNIP) mode, and the Neural-network based intra prediction mode may be performed based on a neural network model that inputs adjacent samples of the current block.

[0036] The present disclosure is susceptible to various modifications and embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure. In the drawings, similar reference numerals designate the same or similar functions throughout. The shapes and sizes of elements in the drawings may be provided for illustrative purposes only for clarity. The detailed description of the exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments, while different from one another, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure. Furthermore, it should be understood that the positions or arrangements of individual components within each disclosed embodiment may be modified without departing from the spirit and scope of the embodiment. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the illustrative embodiments, if properly described, is defined only by the appended claims, along with the full scope equivalents to which such claims are entitled.

[0037] While terms such as "first" and "second" may be used herein to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0038] The components shown in the embodiments of the present disclosure are independently depicted to represent different characteristic functions, and do not imply that each component is composed of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two components among each component may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separate embodiments of each component are also included in the scope of the present disclosure as long as they do not deviate from the essence of the present disclosure.

[0039] The terminology used in this disclosure is merely used to describe specific embodiments and is not intended to limit the present disclosure. The singular expression includes the plural expression unless the context clearly dictates otherwise. In addition, some components of the present disclosure may not be essential components that perform essential functions in the present disclosure and may be optional components merely for performance enhancement. The present disclosure may be implemented by including only components essential to implementing the essence of the present disclosure, excluding components used solely for performance enhancement, and a structure including only essential components, excluding optional components used solely for performance enhancement, is also within the scope of the present disclosure.

[0040] In an embodiment, the term "at least one" may mean one of a number greater than or equal to 1, such as 1, 2, 3, and 4. In an embodiment, the term "a plurality of" may mean one of a number greater than or equal to 2, such as 2, 3, and 4.

[0041] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if a detailed description of a related known configuration or function is judged to obscure the gist of this specification, the detailed description will be omitted. The same reference numerals will be used for identical components in the drawings, and duplicate descriptions of the same components will be omitted.

[0042] Glossary of Terms

[0043] Hereinafter, “video” may mean a single picture constituting a video, or may refer to the video itself. For example, “encoding and / or decoding of a video” may mean “encoding and / or decoding of a video,” or may mean “encoding and / or decoding of one of the videos constituting the video.”

[0044] Hereinafter, the terms "video" and "movie" may be used interchangeably and have the same meaning. Furthermore, the target image may be an encoding target image, which is the target of encoding, and / or a decoding target image, which is the target of decoding. Furthermore, the target image may be an input image input to an encoding device, or an input image input to a decoding device. Here, the target image may have the same meaning as the current image.

[0045] Hereinafter, the terms encoder and image encoding device may be used interchangeably and have the same meaning.

[0046] Hereinafter, the terms decoder and image decoding device may be used interchangeably and have the same meaning.

[0047] Hereinafter, “image”, “picture”, “frame” and “screen” may be used with the same meaning and may be used interchangeably.

[0048] Hereinafter, the term "target block" may refer to an encoding target block, which is the target of encoding, and / or a decoding target block, which is the target of decoding. Furthermore, the target block may refer to a current block, which is the target of current encoding and / or decoding. For example, the terms "target block" and "current block" may be used interchangeably and have the same meaning.

[0049] Hereinafter, "block" and "unit" may be used with the same meaning and may be used interchangeably. In addition, "unit" may mean including a luminance component block and a corresponding chroma component block to distinguish it from a block. For example, a coding tree unit (CTU) may be composed of one luma component (Y) coding tree block (CTB) and two chroma component (Cb, Cr) coding tree blocks associated with it.

[0050] Hereinafter, the terms “sample,” “pixel,” and “pixel” may be used interchangeably and have the same meaning. Here, a sample may represent a basic unit that constitutes a block.

[0051] Hereinafter, “inter” and “between screens” may be used interchangeably and have the same meaning.

[0052] Hereinafter, “intra” and “within screen” may be used interchangeably and have the same meaning.

[0053]

[0054] FIG. 1 is a block diagram showing a configuration according to one embodiment of an encoding device to which the present disclosure is applied.

[0055] The encoding device (100) may be an encoder, a video encoding device, or an image encoding device. A video may include one or more images. The encoding device (100) may sequentially encode one or more images.

[0056] Referring to FIG. 1, the encoding device (100) may include an image segmentation unit (110), an intra prediction unit (120), a motion prediction unit (121), a motion compensation unit (122), a switch (115), a subtractor (113), a transformation unit (130), a quantization unit (140), an entropy encoding unit (150), an inverse quantization unit (160), an inverse transformation unit (170), an adder (117), a filter unit (180), and a reference picture buffer (190).

[0057] Additionally, the encoding device (100) can generate a bitstream including encoded information through encoding an input image and output the generated bitstream. The generated bitstream can be stored in a computer-readable recording medium or can be streamed via a wired / wireless transmission medium.

[0058] The video segmentation unit (110) can segment the input video into various forms to increase the efficiency of video encoding / decoding. That is, the input video is composed of multiple pictures, and one picture can be hierarchically segmented and processed for compression efficiency, parallel processing, etc. For example, one picture can be segmented into one or more tiles or slices, which can then be segmented into multiple Coding Tree Units (CTUs). Alternatively, one picture can first be segmented into multiple sub-pictures defined as groups of rectangular slices, and each sub-picture can then be segmented into the tiles / slices. Here, the sub-pictures can be utilized to support the function of partially independently encoding / decoding and transmitting the picture. Since multiple sub-pictures can each be individually restored, there is an advantage of easy editing in applications that configure multi-channel input into a single picture. In addition, tiles can be segmented horizontally to generate bricks. Here, a brick can be utilized as the basic unit of intra-picture parallel processing. In addition, one CTU can be recursively split into a quadtree (QT), and the terminal node of the split can be defined as a coding unit (CU). The CU can be split into a prediction unit (PU) and a transformation unit (TU), and prediction and splitting can be performed. Meanwhile, the CU can be utilized as a prediction unit and / or a transformation unit itself. Here, for flexible splitting, each CTU can be recursively split into a multi-type tree (MTT) as well as a quadtree (QT). Splitting of a CTU into a multi-type tree can start from the terminal node of a QT, and the MTT can be composed of a binary tree (BT) and a triple tree (TT).For example, the MTT structure can be divided into vertical binary split mode (SPLIT_BT_VER), horizontal binary split mode (SPLIT_BT_HOR), vertical ternary split mode (SPLIT_TT_VER), and horizontal ternary split mode (SPLIT_TT_HOR). In addition, the minimum block size (MinQTSize) of the quad tree of the luminance block during splitting can be set to 16x16, the maximum block size (MaxBtSize) of the binary tree can be set to 128x128, and the maximum block size (MaxTtSize) of the triple tree can be set to 64x64. In addition, the minimum block size (MinBtSize) of the binary tree and the minimum block size (MinTtSize) of the triple tree can be set to 4x4, and the maximum depth (MaxMttDepth) of the multi-type tree can be set to 4. Additionally, to improve the encoding efficiency of the I slice, a dual tree can be applied that uses different CTU partition structures for luminance and chrominance components. On the other hand, in the P and B slices, the luminance and chrominance CTBs (Coding Tree Blocks) within the CTU can be partitioned into a single tree that shares the coding tree structure.

[0059] The encoding device (100) may perform encoding on the input image in intra mode and / or inter mode. Alternatively, the encoding device (100) may perform encoding on the input image in a third mode (e.g., IBC mode, Palette mode, etc.) other than the intra mode and inter mode. However, if the third mode has functional characteristics similar to the intra mode or inter mode, it may be classified as intra mode or inter mode for convenience of explanation. In the present disclosure, the third mode will be classified and described separately only when a specific description is required.

[0060] When the intra mode is used as the prediction mode, the switch (115) can be switched to intra, and when the inter mode is used as the prediction mode, the switch (115) can be switched to inter. Here, the intra mode can mean an intra-screen prediction mode, and the inter mode can mean an inter-screen prediction mode. The encoding device (100) can generate a prediction block for an input block of an input image. In addition, after the prediction block is generated, the encoding device (100) can encode a residual block using a residual of the input block and the prediction block. The input image can be referred to as a current image that is a current encoding target. The input block can be referred to as a current block that is a current encoding target or an encoding target block.

[0061] When the prediction mode is intra mode, the intra prediction unit (120) can use samples of blocks already encoded / decoded around the current block as reference samples. The intra prediction unit (120) can perform spatial prediction on the current block using the reference samples, and can generate prediction samples for the input block through spatial prediction. Here, intra prediction can mean prediction within the screen.

[0062] As an intra prediction method, non-directional prediction modes such as DC mode and Planar mode, as well as directional prediction modes (e.g., 65 directions) can be applied. Here, the intra prediction method can be expressed as an intra prediction mode or an intra-screen prediction mode.

[0063] When the prediction mode is inter mode, the motion prediction unit (121) can search for an area that best matches the input block from the reference image during the motion prediction process and derive a motion vector using the searched area. At this time, the area can be used as a search area. The reference image can be stored in the reference picture buffer (190). Here, when encoding / decoding for the reference image is processed, it can be stored in the reference picture buffer (190).

[0064] The motion compensation unit (122) can generate a prediction block for the current block by performing motion compensation using a motion vector. Here, inter prediction may mean inter-screen prediction or motion compensation.

[0065] The above motion prediction unit (121) and motion compensation unit (122) can generate a prediction block by applying an interpolation filter to a portion of an area within a reference image when the value of the motion vector does not have an integer value. In order to perform inter-screen prediction or motion compensation, it is possible to determine whether the motion prediction and motion compensation method of the prediction unit included in the corresponding encoding unit is one of Skip Mode, Merge Mode, Advanced Motion Vector Prediction (AMVP) mode, and Intra Block Copy (IBC) mode based on the encoding unit, and perform inter-screen prediction or motion compensation according to each mode.

[0066] In addition, based on the above inter-screen prediction method, the AFFINE mode of sub-PU based prediction, the SbTMVP (Subblock-based Temporal Motion Vector Prediction) mode, and the MMVD (Merge with MVD) mode and the GPM (Geometric Partitioning Mode) mode of PU based prediction can be applied. In addition, in order to improve the performance of each mode, the HMVP (History based MVP), the PAMVP (Pairwise Average MVP), the CIIP (Combined Intra / Inter Prediction), the AMVR (Adaptive Motion Vector Resolution), the BDOF (Bi-Directional Optical-Flow), the BCW (Bi-predictive with CU Weights), the LIC (Local Illumination Compensation), the TM (Template Matching), and the OBMC (Overlapped Block Motion Compensation) can be applied.

[0067] Among these, AFFINE mode is a technology that is used in both AMVP and MERGE modes and also has high encoding efficiency. In the existing video coding standard, since MC (Motion Compensation) is performed by considering only the parallel translation of the block, there was a disadvantage in that it could not properly compensate for motions that occur in reality, such as zoom in / out and rotation. To supplement this, a 4-parameter affine motion model using two control point motion vectors (CPMV) and a 6-parameter affine motion model using three control point motion vectors can be applied to inter prediction. Here, CPMV is a vector representing the affine motion model of one of the upper left, upper right, and lower left of the current block.

[0068] The subtractor (113) can generate a residual block using the difference between the input block and the predicted block. The residual block may also be referred to as a residual signal. The residual signal may refer to the difference between the original signal and the predicted signal. Alternatively, the residual signal may be a signal generated by transforming, quantizing, or transforming and quantizing the difference between the original signal and the predicted signal. The residual block may be a residual signal in block units.

[0069] The transform unit (130) can perform a transform on the residual block to generate a transform coefficient and output the generated transform coefficient. Here, the transform coefficient may be a coefficient value generated by performing a transform on the residual block. When the transform skip mode is applied, the transform unit (130) may also skip the transform on the residual block.

[0070] Quantized levels can be generated by applying quantization to transform coefficients or residual signals. In the following embodiments, quantized levels may also be referred to as transform coefficients.

[0071] For example, a 4x4 luminance residual block generated through intra prediction can be transformed using a basis vector based on DST (Discrete Sine Transform), and the remaining residual blocks can be transformed using a basis vector based on DCT (Discrete Cosine Transform). In addition, through RQT (Residual Quad Tree) technology, the transform block is divided into a quad tree shape for one block, and after performing transformation and quantization on each transform block divided through RQT, a coded block flag (cbf) can be transmitted to increase encoding efficiency when all coefficients become 0.

[0072] Another alternative is to apply Multiple Transform Selection (MTS) technology, which selectively performs transformation using multiple transformation bases. That is, instead of dividing CUs into TUs via RQT, a Sub-block Transform (SBT) technology can perform a function similar to TU division. Specifically, SBT is applied only to inter-screen prediction blocks, and unlike RQT, it can divide the current block into ½ or ¼ blocks vertically or horizontally, and then perform transformation on only one of the blocks. For example, in a vertically divided block, the transformation can be performed on the leftmost or rightmost block, and in a horizontally divided block, the transformation can be performed on the topmost or bottommost block.

[0073] Additionally, LFNST (Low Frequency Non-Separable Transform), a secondary transform technique that further transforms the residual signal converted to the frequency domain through DCT or DST, can be applied. LFNST additionally performs a transform on the low-frequency region of 4x4 or 8x8 in the upper left, which allows the residual coefficients to be concentrated in the upper left.

[0074] The quantization unit (140) can generate a quantized level by quantizing a transform coefficient or residual signal according to a quantization parameter (QP), and can output the generated quantized level. At this time, the quantization unit (140) can quantize the transform coefficient using a quantization matrix.

[0075] For example, a quantizer with QP values ​​of 0 to 51 can be used. Alternatively, if the image size is larger and high encoding efficiency is required, a QP of 0 to 63 can be used. In addition, a Dependent Quantization (DQ) method that uses two quantizers instead of a single quantizer can be applied. DQ performs quantization using two quantizers (e.g., Q0 and Q1), but even without signaling information about the use of a specific quantizer, the quantizer to be used for the next transform coefficient can be selected based on the current state through a state transition model.

[0076] The entropy encoding unit (150) can generate a bitstream by performing entropy encoding according to a probability distribution on values ​​produced by the quantization unit (140) or coding parameter values ​​produced during the encoding process, and can output the bitstream. The entropy encoding unit (150) can perform entropy encoding on information about image samples and information for decoding the image. For example, the information for decoding the image can include syntax elements, etc.

[0077] When entropy encoding is applied, a small number of bits are allocated to symbols with a high occurrence probability, and a large number of bits are allocated to symbols with a low occurrence probability, thereby representing the symbols, whereby the size of the bit string for the symbols to be encoded can be reduced. The entropy encoding unit (150) can use an encoding method such as exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), or Context-Adaptive Binary Arithmetic Coding (CABAC) for entropy encoding. For example, the entropy encoding unit (150) can perform entropy encoding using a Variable Length Coding / Code (VLC) table. In addition, the entropy encoding unit (150) may perform arithmetic encoding using the binarization method, probability model, and context model derived from the binarization method of the target symbol and the probability model of the target symbol / bin.

[0078] In this regard, when applying CABAC, the table probability update method can be changed to a simple formula-based table update method to reduce the size of the probability table stored in the decryption device. Furthermore, two different probability models can be used to obtain more accurate symbol probability values.

[0079] The entropy encoding unit (150) can change a two-dimensional block form coefficient into a one-dimensional vector form through a transform coefficient scanning method to encode a transform coefficient level (quantized level).

[0080] Coding parameters may include not only information (flags, indexes, etc.) encoded in an encoding device (100) and signaled to a decoding device (200), such as syntax elements, but also information derived during an encoding process or a decoding process, and may mean information necessary when encoding or decoding an image.

[0081] Here, signaling a flag or index may mean that the encoder entropy encodes the flag or index and includes it in the bitstream, and that the decoder entropy decodes the flag or index from the bitstream.

[0082] The encoded current image can be used as a reference image for other images to be processed later. Accordingly, the encoding device (100) can reconstruct or decode the encoded current image again and store the reconstructed or decoded image as a reference image in the reference picture buffer (190).

[0083] The quantized level can be dequantized in the dequantization unit (160) and inversely transformed in the inverse transformation unit (170). The dequantized and / or inversely transformed coefficients can be combined with a prediction block through an adder (117), and a reconstructed block can be generated by combining the dequantized and / or inversely transformed coefficients and the prediction block. Here, the dequantized and / or inversely transformed coefficients refer to coefficients on which at least one of dequantization and inverse transformation has been performed, and may refer to a reconstructed residual block. The dequantization unit (160) and the inverse transformation unit (170) can be performed in the reverse process of the quantization unit (140) and the transformation unit (130).

[0084] The restoration block may pass through a filter unit (180). The filter unit (180) may apply a deblocking filter, a sample adaptive offset (SAO), an adaptive loop filter (ALF), a bilateral filter (BIF), a Luma Mapping with Chroma Scaling (LMCS), etc. as a filtering technique, in whole or in part, to the restoration sample, restoration block, or restoration image. The filter unit (180) may also be referred to as an in-loop filter. In this case, the in-loop filter is also used as a name excluding LMCS.

[0085] A deblocking filter can remove block distortion that occurs at the boundaries between blocks. Whether to apply a deblocking filter to the current block can be determined based on the samples contained in several columns or rows within the block. When applying a deblocking filter to a block, different filters can be applied depending on the required deblocking filtering strength.

[0086] Sample adaptive offset can be used to compensate for encoding errors by adding an appropriate offset value to sample values. Sample adaptive offset can compensate for the offset from the original image on a sample-by-sample basis for deblocked images. This can be done by dividing the samples contained in the image into a fixed number of regions, determining the regions to be offset, and applying the offset to those regions. Alternatively, the offset can be applied by considering the edge information of each sample.

[0087] Bilateral filter (BIF) can also compensate for the offset from the original image on a sample-by-sample basis for the deblocked image.

[0088] An adaptive loop filter can perform filtering based on a comparison between a reconstructed image and the original image. By dividing the samples contained in the image into predetermined groups and determining the filter to be applied to each group, filtering can be performed differentially for each group. Information regarding whether to apply an adaptive loop filter can be signaled for each coding unit (CU), and the shape and filter coefficients of the adaptive loop filter applied to each block can vary.

[0089] In LMCS (Luma Mapping with Chroma Scaling), luma mapping (LM) refers to remapping luminance values ​​through a piece-wise linear model, and chroma scaling (CS) refers to a technique that scales the residual values ​​of chrominance components according to the average luminance value of the prediction signal. In particular, LMCS can be utilized as an HDR correction technique that reflects the characteristics of HDR (High Dynamic Range) images.

[0090] The restored block or restored image that has passed through the filter unit (180) may be stored in the reference picture buffer (190). The restored block that has passed through the filter unit (180) may be a part of the reference image. In other words, the reference image may be a restored image composed of restored blocks that have passed through the filter unit (180). The stored reference image may be used for inter-screen prediction or motion compensation thereafter.

[0091] FIG. 2 is a block diagram showing a configuration according to one embodiment of a decryption device to which the present disclosure is applied.

[0092] The decoding device (200) may be a decoder, a video decoding device, or an image decoding device.

[0093] Referring to FIG. 2, the decoding device (200) may include an entropy decoding unit (210), an inverse quantization unit (220), an inverse transformation unit (230), an intra prediction unit (240), a motion compensation unit (250), an adder (201), a switch (203), a filter unit (260), and a reference picture buffer (270).

[0094] The decoding device (200) can receive a bitstream output from the encoding device (100). The decoding device (200) can receive a bitstream stored in a computer-readable recording medium, or a bitstream streamed through a wired / wireless transmission medium. The decoding device (200) can perform decoding on the bitstream in intra mode or inter mode. In addition, the decoding device (200) can generate a restored image or a decoded image through decoding, and can output the restored image or the decoded image.

[0095] If the prediction mode used for decryption is intra mode, the switch (203) can be switched to intra. If the prediction mode used for decryption is inter mode, the switch (203) can be switched to inter.

[0096] The decoding device (200) can decode the input bitstream to obtain a reconstructed residual block and generate a prediction block. Once the reconstructed residual block and the prediction block are obtained, the decoding device (200) can generate a reconstructed block to be decoded by adding the reconstructed residual block and the prediction block. The block to be decoded may be referred to as a current block.

[0097] The entropy decoding unit (210) can generate symbols by performing entropy decoding according to a probability distribution for the bitstream. The generated symbols may include symbols in the form of quantized levels. Here, the entropy decoding method may be the reverse process of the entropy encoding method described above.

[0098] The entropy decoding unit (210) can change a one-dimensional vector-shaped coefficient into a two-dimensional block-shaped coefficient through a transform coefficient scanning method to decode a transform coefficient level (quantized level).

[0099] The quantized level can be inversely quantized in the inverse quantization unit (220) and inversely transformed in the inverse transformation unit (230). The quantized level can be generated as a restored residual block as a result of performing inverse quantization and / or inverse transformation. At this time, the inverse quantization unit (220) can apply a quantization matrix to the quantized level. The inverse quantization unit (220) and inverse transformation unit (230) applied to the decoding device can apply the same technology as the inverse quantization unit (160) and inverse transformation unit (170) applied to the encoding device described above.

[0100] When intra mode is used, the intra prediction unit (240) can generate a predicted block by performing spatial prediction on the current block using sample values ​​of already decoded blocks surrounding the block to be decoded. The intra prediction unit (240) applied to the decoding device can apply the same technology as the intra prediction unit (120) applied to the encoding device described above.

[0101] When the inter mode is used, the motion compensation unit (250) can generate a prediction block by performing motion compensation using a motion vector and a reference image stored in the reference picture buffer (270) on the current block. The motion compensation unit (250) can generate a prediction block by applying an interpolation filter to a portion of the reference image when the value of the motion vector does not have an integer value. In order to perform motion compensation, it is possible to determine whether the motion compensation method of the prediction unit included in the corresponding encoding unit is skip mode, merge mode, AMVP mode, or current picture reference mode based on the encoding unit, and motion compensation can be performed according to each mode. The motion compensation unit (250) applied to the decoding device can apply the same technology as the motion compensation unit (122) applied to the encoding device described above.

[0102] The adder (201) can add the restored residual block and the predicted block to generate a restored block. The filter unit (260) can apply at least one of an Inverse-LMCS, a deblocking filter, a sample adaptive offset, and an adaptive loop filter to the restored block or restored image. The filter unit (260) applied to the decoding device can apply the same filtering technology as that applied to the filter unit (180) applied to the encoding device described above.

[0103] The filter unit (260) can output a restored image. The restored block or restored image can be stored in the reference picture buffer (270) and used for inter prediction. The restored block that has passed through the filter unit (260) can be a part of the reference image. In other words, the reference image can be a restored image composed of restored blocks that have passed through the filter unit (260). The stored reference image can be used for inter-screen prediction or motion compensation thereafter.

[0104] FIG. 3 is a diagram schematically illustrating a video coding system to which the present disclosure can be applied.

[0105] A video coding system according to one embodiment may include an encoding device (10) and a decoding device (20). The encoding device (10) may transmit encoded video and / or image information or data to the decoding device (20) in the form of a file or streaming through a digital storage medium or a network.

[0106] An encoding device (10) according to one embodiment may include a video source generation unit (11), an encoding unit (12), and a transmission unit (13). A decoding device (20) according to one embodiment may include a reception unit (21), a decoding unit (22), and a rendering unit (23). The encoding unit (12) may be referred to as a video / image encoding unit, and the decoding unit (22) may be referred to as a video / image decoding unit. The transmission unit (13) may be included in the encoding unit (12). The reception unit (21) may be included in the decoding unit (22). The rendering unit (23) may include a display unit, and the display unit may be configured as a separate device or an external component.

[0107] The video source generation unit (11) can obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source generation unit (11) can include a video / image capture device and / or a video / image generation device. The video / image capture device can include, for example, one or more cameras, a video / image archive including previously captured video / images, etc. The video / image generation device can include, for example, a computer, a tablet, a smartphone, etc., and can (electronically) generate video / images. For example, a virtual video / image can be generated through a computer, etc., in which case the video / image capture process can be replaced with a process of generating related data.

[0108] The encoding unit (12) can encode the input video / image. The encoding unit (12) can perform a series of procedures such as prediction, transformation, and quantization for compression and encoding efficiency. The encoding unit (12) can output encoded data (encoded video / image information) in the form of a bitstream. The detailed configuration of the encoding unit (12) can also be configured in the same manner as the encoding device (100) of FIG. 1 described above.

[0109] The transmission unit (13) can transmit encoded video / image information or data output in the form of a bitstream to the reception unit (21) of the decoding device (20) via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (13) can include an element for generating a media file through a predetermined file format and can include an element for transmission via a broadcasting / communication network. The reception unit (21) can extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit (22).

[0110] The decoding unit (22) can decode video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding unit (12). The detailed configuration of the decoding unit (22) can also be configured in the same manner as the decoding device (200) of FIG. 2 described above.

[0111] The rendering unit (23) can render the decrypted video / image. The rendered video / image can be displayed through the display unit.

[0112]

[0113] Recently, active research has been conducted on applying deep learning technology to image compression. Unlike conventional intra prediction techniques based on fixed rules, neural network-based intra prediction (NNIP) methods utilize reconstructed reference samples and / or generated prediction blocks surrounding the current block as input to a neural network model, generating prediction signals with higher accuracy. Furthermore, this approach can contribute to improved compression efficiency by reducing residual signals.

[0114] In this specification, we propose a method for considering neural network-based intra prediction in template-based intra mode derivation (TIMD). Prior to this, the template-based intra prediction mode derivation method is described with reference to FIG. 4.

[0115]

[0116] FIG. 4 is a diagram illustrating a template-based intra prediction mode derivation method (hereinafter, referred to as the TIMD method) according to one embodiment of the present disclosure. The TIMD method is a method of deriving an intra prediction mode by analyzing a template of a current block to generate a prediction signal.

[0117] Meanwhile, the intra prediction mode induced by the TIMD method can be called the TIMD mode.

[0118] In Fig. 4, a predetermined area adjacent to the current block (400) can be defined as the current template (401).

[0119] Referring to FIG. 4, any candidate intra prediction modes may be applied to reference pixels (Reference of the template, 402) of the template, which are pixels adjacent to the current template (Current template, 401), to generate a prediction value of the current template (401). The size of the current template (401) may be L1xN and / or L2xM, and L1, L2, N, and M may all be positive integers.

[0120] The current template is an already restored region, and the similarity between the restored sample value of the current template (401) and the predicted value of the current template can be derived. At this time, the similarity can be derived based on the cost value of the predicted sample of the current template (401) calculated using an arbitrary cost function.

[0121] Meanwhile, the aforementioned cost function may be any one of the cost functions such as SAD (sum of absolute differences), SATD (sum of absolute transformed differences), SSD (sum of squared differences), and MR-SAD (mean removed sum of absolute differences).

[0122] And, among any candidate intra prediction modes, N intra prediction modes can be derived as TIMD modes in the order of low cost values, where N is any positive integer.

[0123] And, based on the derived TIMD modes, prediction values ​​of the current block (400) can be generated.

[0124] And, the prediction values ​​of the current block (400) generated based on the derived TIMD modes can be weighted and combined to generate the final prediction value of the current block (400).

[0125] Meanwhile, any candidate intra prediction mode may be an angular mode and / or a non-angular mode. Here, the non-angular mode may include a DC mode and a Planar mode, etc.

[0126] For example, among any candidate intra prediction modes, two directional intra prediction modes and one non-directional intra prediction mode can be derived as TIMD modes. Then, prediction values ​​of the current block can be generated based on each of the derived TIMD modes. Then, the generated prediction values ​​can be weighted and combined to generate a final prediction value of the current block.

[0127] Meanwhile, according to one embodiment of the present disclosure, any candidate intra prediction mode may include not only directional modes and / or non-directional modes, but also non-traditional intra modes. Here, non-traditional intra modes may mean prediction modes other than the conventional directional modes and non-directional modes, such as intra template matching prediction (intraTMP) mode and intra block copy (IBC) mode.

[0128]

[0129] Meanwhile, according to the above-described embodiment, one of the cost functions such as SAD, SATD, SSD, and MR-SAD is used to derive the similarity, but a predetermined number of multiple cost functions may be used together.

[0130] Specifically, N intra prediction modes can be derived as TIMD modes for each cost function. Then, the prediction samples generated for each cost function can be weighted and combined to generate a final prediction value for each cost function.

[0131] Additionally, among the final predicted values ​​generated for each cost function, the predicted value with superior predictive performance can be selected. Here, superior predictive performance can mean that the predicted value has minimal distortion.

[0132] Meanwhile, information about the selected prediction value can be explicitly signaled in the encoder / decoder. Here, information about the selected prediction value can be information about which cost function was used to generate the selected prediction value.

[0133]

[0134] Below, we will explain how to generate the final predicted value in the TIMD method.

[0135] In the TIMD method, prediction values ​​for the current block can be generated based on the derived TIMD modes. Then, the generated prediction values ​​can be weighted and combined to generate a final prediction value for the current block.

[0136] Meanwhile, the final predicted value of the current block can be generated as in mathematical formula 1.

[0137]

[0138] In Equation 1, Pred represents the final predicted value of the current block. And Pred i (i=1, 2, …, k) refers to the predicted value generated based on the ith TIMD mode among the derived TIMD modes, and w i (i=1, 2…, k) represents the weight value of each predicted value. Here, w i are real numbers greater than or equal to 0, and w i The sum of the fields is 1. Meanwhile, k is an arbitrary positive real number, which represents the total number of induced TIMD modes.

[0139] According to one embodiment, w i can be determined by an arbitrary value promised in the encoder / decoder. In this case, w imay not be signaled separately.

[0140] According to another embodiment, w i can be determined by the encoder and transmitted to the decoder.

[0141] According to another embodiment, w i can be calculated as in mathematical formula 2.

[0142]

[0143] In mathematical expression 2, w i refers to the weighted value of the prediction value generated based on the ith TIMD mode among the induced TIMD modes.

[0144] Also, in Equation 2, Cost j represents the cost value of the jth TIMD mode. Therefore, sumCost represents the total sum of the cost values ​​of the derived TIMD modes.

[0145] Additionally, in mathematical expression 2, k represents the total number of induced TIMD modes.

[0146]

[0147] FIG. 5 is a diagram illustrating a neural network-based intra prediction method (hereinafter, referred to as the NNIP method) according to one embodiment of the present disclosure. Here, the NNIP method may refer to a method of performing prediction using a neural network model.

[0148] Referring to Fig. 5, the L-shaped region (left, top left, and top regions) containing the neighboring reference samples of the current block (Current block, 500) can be referred to as the current template (Current template, 501). At this time, the top region included in the current template (501) can be referred to as the top template (Top template, 502), and the left region can be referred to as the left template (Left template, 503). In addition, the top left region included in the current template (501) can be referred to as the left-top template (Left-top template, 504).

[0149] In Fig. 5, the size of the current template is (2W x t) + (txt) + (tx 2H). Here, W and H represent the width and height of the current block (500), and t may be a predetermined positive integer. For example, t may be 4 or 8.

[0150] In the NNIP method, reference samples included in the current template (501) can be used as inputs to a neural network model. The neural network model can receive these reference samples as inputs and output a predicted value for the current block.

[0151] Meanwhile, one of multiple predefined neural networks can be selected depending on the size of the current block. For example, there can be a separate neural network for each block of size 4X4, 8X4, 16X4, 8X8, 16X8, and 16X16.

[0152] Meanwhile, information regarding reference samples that serve as input to the neural network model can be determined by the encoder and transmitted to the decoder. Specifically, the size and shape of the current template (501) can be determined by the encoder and transmitted to the decoder.

[0153] Meanwhile, the reference sample that is the input of the neural network model of Fig. 5 is an example, and the location, number, shape and size of the template of the reference samples used as the input of the neural network model can be defined in various ways.

[0154] For example, only the reference samples included in any one of the top template (502), left template (503), and upper left template (504) may be used, or the reference samples may be changed by changing the size or shape of the template.

[0155] As another example, the number of reference samples that serve as input to the neural network model may be determined based on the size of the current block (500). For example, the size of the current block (500) and the number of reference samples may be proportional.

[0156] Meanwhile, the current block may be a luma block.

[0157] Meanwhile, the NNIP method according to Fig. 5 uses the surrounding reference samples of the current block as inputs to the neural network model, but as an example, the generated prediction block can also be used as inputs to the neural network model.

[0158]

[0159] Below, we describe how to incorporate NNIP mode into the TIMD method. This allows for the generation of prediction signals with higher accuracy and contributes to improved compression efficiency by reducing residual signals.

[0160] According to one embodiment of the present disclosure, in a TIMD method, an NNIP mode may be included in any candidate intra prediction mode. Accordingly, among the candidate intra prediction modes including the NNIP mode, N intra prediction modes in descending order of cost values ​​may be derived as TIMD modes. Then, prediction values ​​of the current block may be generated based on the derived TIMD modes. Then, the prediction values ​​of the current block generated using the derived TIMD modes may be weighted and combined to generate a final prediction value of the current block.

[0161] According to one embodiment of the present disclosure, the NNIP mode can be considered as an additional TIMD mode in the TIMD method. Accordingly, N TIMD modes can be derived in order of their lowest cost values ​​among any candidate intra prediction modes, and prediction values ​​for the current block can be generated based on each of the derived TIMD modes and the NNIP mode. Furthermore, the generated prediction values ​​can be weighted and combined to generate a final prediction value for the current block.

[0162] According to one embodiment of the present disclosure, in a TIMD method, a non-directional mode can be replaced with an NNIP mode. Specifically, N TIMD modes can be derived in descending order of cost values ​​from among candidate intra prediction modes including directional and non-directional modes. At this time, if a non-directional mode is derived as any one of the TIMD modes, the non-directional mode is replaced with an NNIP mode, and a prediction value of the current block based on the NNIP mode can be generated. Thereafter, the prediction value generated based on the NNIP mode and the prediction values ​​based on the remaining TIMD modes can be weighted and combined to generate a final prediction value of the current block.

[0163] Meanwhile, when the non-directional mode in the TIMD method is replaced with the NNIP mode, the same weight value can be applied to the prediction value generated based on the NNIP mode as when it was generated based on the non-directional mode.

[0164]

[0165] According to one embodiment of the present disclosure, in a TIMD method, candidate intra prediction modes can be classified into M groups. Specifically, after candidate intra prediction modes including all / some of directional modes, non-directional modes, non-traditional intra modes, and NNIP modes are classified into M groups, a predetermined number of intra prediction modes with low cost values ​​can be derived as TIMD modes for each group. Then, prediction values ​​of a current block can be generated based on the derived TIMD modes. Then, the generated prediction values ​​of the current block can be weighted and combined to generate a final prediction value of the current block. Here, M is an arbitrary positive integer.

[0166] For example, candidate intra prediction modes can be classified into a first group including directional modes and a second group including non-directional modes and NNIP modes. In the first group, K intra prediction modes can be derived as TIMD modes, and in the second group, L intra prediction modes can be derived as TIMD modes. Here, K and L are arbitrary positive integers. For example, K can be 2 and L can be 1.

[0167] As another example, the candidate intra prediction modes can be classified into a first group including directional modes, a second group including non-directional modes, and a third group including non-traditional intra modes and NNIP modes. A predetermined number of intra prediction modes in the first group can be derived as TIMD modes, a predetermined number of intra prediction modes in the second group can be derived as TIMD modes, and a predetermined number of intra prediction modes in the third group can be derived as TIMD modes. For example, two intra prediction modes in the first group, one intra prediction mode in the second group, and one intra prediction mode in the third group can be derived as TIMD modes.

[0168] As another example, candidate intra prediction modes can be classified into a first group including directional modes, non-traditional intra modes, and NNIP modes, and a second group including non-directional modes. A predetermined number of intra prediction modes in the first group can be derived as TIMD modes, and a predetermined number of intra prediction modes in the second group can be derived as TIMD modes. For example, two intra prediction modes can be derived as TIMD modes in the first group, and one intra prediction mode can be derived as TIMD modes in the second group.

[0169] The classification of candidate intra prediction modes described above is merely an example, and all or some of the modes that may be included in the candidate intra prediction modes may be classified into M groups in various ways.

[0170]

[0171] According to one embodiment of the present disclosure, when NNIP is used, the NNIP mode can be considered in the TIMD method without separate additional signaling.

[0172] For example, if the NNIP activation information indicates that the NNIP mode is used, the TIMD method may consider the NNIP mode without any additional signaling. Here, the NNIP activation information may be information indicating whether the NNIP mode is used.

[0173] Meanwhile, NNIP activation information can be expressed as an activation flag or a deactivation flag.

[0174] According to one embodiment of the present disclosure, whether the NNIP mode is considered in the TIMD method can be implicitly determined according to a predetermined condition.

[0175] For example, a given condition may mean that the prediction mode of the current block is a given intra prediction mode. Specifically, if the prediction mode of the current block is a given intra prediction mode, the NNIP mode may be considered in the TIMD method.

[0176] As another example, a given condition may mean that the current block size is a given size. Specifically, if the current block size is a given size, the NNIP mode may be considered in the TIMD method.

[0177] As another example, a given condition may be that the value of the quantization parameter (QP) corresponding to the current block is a given value. Specifically, if the value of the QP corresponding to the current block is a given value, the NNIP mode may be considered in the TIMD method.

[0178] As another example, a given condition may mean that the NNIP application information indicates that the NNIP mode is considered. Here, the NNIP application information may be information indicating whether the NNIP mode is considered in the TIMD method.

[0179] Specifically, NNIP mode can be considered in the TIMD method if information on whether NNIP is applied is explicitly signaled and the decoder indicates that the information indicates that NNIP mode is considered.

[0180] Meanwhile, information on whether NNIP is applied can be signaled at a higher level such as a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a picture header (PH), and a slice header (SH), or can be signaled at a lower level such as a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), and a transform unit (TU).

[0181] Meanwhile, information on whether NNIP is applied can be signaled independently from other syntax elements.

[0182] Alternatively, information on whether NNIP is applied can be signaled dependently on other syntax elements. For example, it can be signaled dependently on NNIP activation information. Here, NNIP activation information can be information indicating whether NNIP mode is used.

[0183] Alternatively, the information may be signaled dependently on TIMD activation information, wherein the TIMD activation information may be information indicating whether the TIMD method is used.

[0184]

[0185] According to one embodiment of the present disclosure, in a TIMD method, the NNIP mode can always be induced into the TIMD mode under certain conditions. Specifically, in a TIMD method, if a certain condition is satisfied, the NNIP mode can be induced into any of the TIMD modes regardless of the cost value.

[0186] For example, a given condition may mean that the prediction mode of the current block is a given intra prediction mode. Specifically, if the prediction mode of the current block is a given intra prediction mode, the NNIP mode may be induced into the TIMD mode.

[0187] As another example, a given condition may mean that the current block size is a given size. Specifically, if the current block size is a given size, NNIP mode can be induced into TIMD mode.

[0188] As another example, a given condition may be that the value of the quantization parameter corresponding to the current block is a given value. Specifically, if the value of the QP corresponding to the current block is a given value, the NNIP mode can be induced into the TIMD mode.

[0189] As another example, a given condition may mean that the NNIP activation information indicates that NNIP mode is used. Here, the NNIP activation information may be information indicating whether NNIP mode is used.

[0190] Specifically, if NNIP activation information is explicitly signaled and the decoder indicates that NNIP mode is used, NNIP mode can be induced into TIMD mode.

[0191] Meanwhile, NNIP activation information can be expressed as an activation flag or a deactivation flag.

[0192] Meanwhile, NNIP activation information can be signaled at higher levels such as video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture header (PH), and slice header (SH), or at lower levels such as coding tree unit (CTU), coding unit (CU), prediction unit (PU), and transform unit (TU).

[0193]

[0194] According to the above-described embodiment, in the TIMD method, a prediction value of a template is generated based on candidate intra prediction modes, and a similarity between a reconstructed sample value of a current template and a prediction value of the current template can be derived. Here, the similarity can be derived based on a cost value of a prediction sample of the current template calculated using an arbitrary cost function. In addition, among any candidate intra prediction modes, N intra prediction modes can be derived as TIMD modes in the order of having a low cost value.

[0195] According to this specification, the NNIP mode can be considered as a candidate intra prediction mode in the TIMD method. Therefore, a cost value must also be derived for the NNIP mode to derive the TIMD mode.

[0196] Hereinafter, a method for deriving a cost value of NNIP mode will be described with reference to Fig. 6.

[0197] FIG. 6 is a diagram for explaining a cost value calculation method of NNIP mode according to one embodiment of the present disclosure.

[0198] In Fig. 6, the upper area of ​​the current block (600) can be referred to as the top template (610), and the left area can be referred to as the left template (620).

[0199] In Fig. 6, the NNIP method can be applied to the top template (610) to generate a predicted value of the top template. That is, the predicted value of the top template (610) can be generated by using the neighboring pixels of the top template (611) as input to the neural network model.

[0200] In addition, the similarity between the restored sample value of the upper template (610) and the predicted value of the upper template can be derived. Here, the similarity can be derived based on the cost value of the predicted sample of the upper template calculated using an arbitrary cost function.

[0201] In addition, the NNIP method can be applied to the left template (620) to generate a prediction value of the left template. That is, the prediction value of the left template can be generated by using the neighboring pixels of the left template (Neighboring pixels of Left template 621) as input to the neural network model.

[0202] In addition, the similarity between the restored sample value of the left template (620) and the predicted value of the left template can be derived. Here, the similarity can be derived based on the cost value of the predicted sample of the left template calculated using an arbitrary cost function.

[0203] Meanwhile, the cost value of the NNIP mode can be calculated by considering only the cost value based on the prediction sample of either the upper template (610) or the left template (620). In this case, the NNIP method can be applied to only either the upper template (610) or the left template (620).

[0204] Alternatively, the cost value of the NNIP mode can be calculated by considering the cost values ​​of the prediction samples for each of the upper template (610) and the left template (620). In this case, the NNIP method can be applied to the upper template (610) and the left template (620).

[0205]

[0206] According to one embodiment of the present disclosure, the NNIP method may be applied to some pixels of the upper template (610) and / or the left template (620) to calculate the cost value of the NNIP mode.

[0207] For example, by using the surrounding pixels (611) of the upper template as input to the neural network model, predicted values ​​of some pixels included in the upper template (610) can be generated.

[0208] And, the similarity between the restored sample values ​​and the predicted values ​​of some pixels can be derived. Here, the similarity can be derived based on the cost values ​​of the predicted samples calculated using an arbitrary cost function.

[0209] As another example, by using the surrounding pixels (621) of the left template as input to the neural network model, predicted values ​​of some pixels included in the left template (620) can be generated.

[0210] And, the similarity between the restored sample values ​​and the predicted values ​​of some pixels can be derived. Here, the similarity can be derived based on the cost values ​​of the predicted samples calculated using an arbitrary cost function.

[0211] Meanwhile, the cost value of the NNIP mode can be calculated by considering only the cost value based on the prediction sample of either some pixels of the upper template (610) or some pixels of the left template (620). In this case, the NNIP method can be applied to only some pixels of the upper template (610) or some pixels of the left template (620).

[0212] Alternatively, the cost value of the NNIP mode can be calculated by considering the cost values ​​of the prediction samples of some pixels of the upper template (610) and the prediction samples of some pixels of the left template (620) together. In this case, the NNIP method can be applied to some pixels of the upper template (610) and some pixels of the left template (620).

[0213] Meanwhile, the positions and number of pixels in the upper template and / or left template to which the NNIP method is applied can be defined in various ways. For example, some pixels to which the NNIP method is applied may be pixels at sub-sampling positions.

[0214] Meanwhile, some of the surrounding pixels (611, 621) of the upper template and / or the left template may be used as inputs to a neural network model, and prediction values ​​of some of the pixels included in the upper template and / or the left template may be generated. At this time, the positions and numbers of some of the surrounding pixels of the upper template and / or the left template may be defined in various ways. For example, some of the surrounding pixels of the upper template and / or the left template, which serve as inputs to the neural network model, may be pixels at sub-sampling positions.

[0215]

[0216] According to one embodiment of the present disclosure, the NNIP method may be applied to the predicted values ​​of the upper template (610) and / or the left template (620) to calculate the cost value in the NNIP mode. As a result, the cost value may be calculated based on the improved predicted values ​​of the upper template (610) and / or the left template (620).

[0217] For example, based on the surrounding pixels (611) of the upper template, a prediction value of the upper template (610) can be generated according to an arbitrary intra prediction mode. Here, the arbitrary intra prediction mode can be any one of prediction modes such as a directional mode, a non-directional mode, a non-traditional intra mode, and an NNIP mode.

[0218] In addition, the NNIP method can be applied to the generated prediction value. That is, the surrounding pixels of the generated prediction value can be used as input to the neural network model to generate the final prediction value.

[0219] And, the similarity between the restored sample value of the upper template (610) and the final predicted value can be derived. Here, the similarity can be derived based on the cost value of the final predicted sample calculated using an arbitrary cost function.

[0220] As another example, based on the surrounding pixels (621) of the left template, a prediction value of the left template (620) can be generated according to an arbitrary intra prediction mode. Here, the arbitrary intra prediction mode can be any one of prediction modes such as a directional mode, a non-directional mode, a non-traditional intra mode, and an NNIP mode.

[0221] In addition, the NNIP method can be applied to the generated prediction value. That is, the surrounding pixels of the generated prediction value can be used as input to the neural network model to generate the final prediction value.

[0222] And, the similarity between the restored sample value of the left template (620) and the final predicted value can be derived. Here, the similarity can be derived based on the cost value of the final predicted sample calculated using an arbitrary cost function.

[0223] Meanwhile, the cost value of the NNIP mode can be calculated by considering only the cost value based on the final prediction sample of either the upper template (610) or the left template (620). In this case, the NNIP method can be applied to only either the prediction value of the upper template (610) or the prediction value of the left template (620).

[0224] Alternatively, the cost value in NNIP mode can be calculated by considering the cost values ​​of the final prediction samples for each of the upper template (610) and the left template (620). In this case, the NNIP method can be applied to the prediction value of the upper template (610) and the prediction value of the left template (620).

[0225] Meanwhile, according to the above-described embodiment, the prediction value of the upper template and / or the left template is generated based on the surrounding pixels (611, 621) of the upper template and / or the left template, but the prediction value may be generated based on some pixels as well as all pixels included in the surrounding pixels of the upper template and / or the left template. In addition, the NNIP method may be applied to the generated prediction value. At this time, the positions and numbers of some pixels among the surrounding pixels of the upper template and / or the left template may be defined in various ways.

[0226] Additionally, according to the above-described embodiment, prediction values ​​for the upper template and / or the left template are generated, but prediction values ​​for some pixels included in the upper template and / or the left template may also be generated. Furthermore, the NNIP method may be applied to the generated prediction values. At this time, the positions and number of some pixels included in the upper template and / or the left template may be defined in various ways.

[0227]

[0228] Meanwhile, the shape and size of the upper template (610) and / or the left template (620) of FIG. 6 are examples, and the shape and size of the templates may be defined in various ways. In addition, the surrounding pixels (611) of the upper template and the surrounding pixels (621) of the left template of FIG. 6 are examples, and the range and number of the surrounding pixels may be defined in various ways.

[0229] Meanwhile, any cost function that serves as the basis for deriving the cost value in NNIP mode may be any of the cost functions such as SAD, SATD, SSD, and MR-SAD. Alternatively, as in the aforementioned embodiment, a predetermined number of multiple cost functions may be used together.

[0230]

[0231] FIG. 7 is a flowchart illustrating a decryption method according to one embodiment of the present disclosure. The decryption method of FIG. 7 can be performed by an image decryption device.

[0232] The video decoding device can derive a first intra prediction mode, a second intra prediction mode, and a third intra prediction mode of the current block based on a current template including adjacent samples of the current block among a plurality of candidate intra prediction modes (S700).

[0233] Meanwhile, the third intra prediction mode can be derived from any one of the DC mode, the Planar mode, and the neural-network based intra prediction (NNIP) mode.

[0234] Meanwhile, the neural network-based intra prediction mode can be performed based on a neural network model that takes adjacent samples of the current block as input.

[0235] Meanwhile, the step of deriving the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode may be performed based on the similarity between the predicted value of the current template based on each of the plurality of candidate intra prediction modes and the current template.

[0236] Meanwhile, the third intra prediction mode can be derived based on the similarity between some pixels included in the current template and the predicted values ​​of some pixels based on the neural network-based intra prediction mode.

[0237] Meanwhile, the similarity may be determined based on any one of the SATD (sum of absolute transformed differences) method, the SAD (sum of absolute differences) method, the SSD (sum of squared differences) method, and the MR-SAD (mean removed sum of absolute differences) method.

[0238] Meanwhile, the current template may include at least one of a left template and a top template.

[0239] Meanwhile, the plurality of candidate intra prediction modes may be classified into a first candidate intra prediction mode group and a second candidate intra prediction mode group, and the first intra prediction mode and the second intra prediction mode may be derived based on the first candidate intra prediction mode group, and the third intra prediction mode may be derived based on the second candidate intra prediction mode group.

[0240] Meanwhile, the first candidate intra prediction mode group may include a directional intra prediction mode, and the second candidate intra prediction mode group may include a DC mode, a Planar mode, and a neural network-based intra prediction mode.

[0241] Meanwhile, the current block may be a luminance block.

[0242] And, the image decoding device can derive the first prediction block, the second prediction block, and the third prediction block of the current block based on each of the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode (S710).

[0243] And, the image decoding device can derive the final prediction block of the current block by weighting the first prediction block, the second prediction block, and the third prediction block (S720).

[0244] Meanwhile, a step of obtaining information indicating whether the neural network-based intra prediction mode is used may be further included, and if the information indicates that the neural network-based intra prediction mode is used, the third intra prediction mode may be derived as the neural network-based intra prediction mode.

[0245] Meanwhile, the weight applied to the first prediction block may be determined based on the similarity between the predicted value of the current template predicted based on the first intra prediction mode and the current template, the weight applied to the second prediction block may be determined based on the similarity between the predicted value of the current template predicted based on the second intra prediction mode and the current template, and the weight applied to the third prediction block may be determined based on the similarity between the predicted value of the current template predicted based on the third intra prediction mode and the current template.

[0246] Meanwhile, the steps described in FIG. 7 can be performed in the same manner in an image encoding method. Furthermore, a bitstream can be generated by an image encoding method including the steps described in FIG. 7. The bitstream can be stored on a non-transitory computer-readable recording medium and can also be transmitted (or streamed).

[0247]

[0248] FIG. 8 is a diagram exemplifying a content streaming system to which an embodiment according to the present disclosure can be applied.

[0249] As illustrated in FIG. 8, a content streaming system to which an embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0250] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, and CCTVs into digital data, generates a bitstream, and transmits it to the streaming server. Alternatively, if multimedia input devices such as smartphones, cameras, and CCTVs directly generate bitstreams, the encoding server may be omitted.

[0251] The above bitstream can be generated by an image encoding method and / or an image encoding device to which an embodiment of the present disclosure is applied, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0252] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server can act as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server may control commands / responses between each device within the content streaming system.

[0253] The streaming server can receive content from a media repository and / or encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0254] Examples of the user devices may include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), digital TVs, desktop computers, digital signage, etc.

[0255] Each server within the above content streaming system can be operated as a distributed server, in which case data received from each server can be processed in a distributed manner.

[0256]

[0257] The above embodiments can be performed in the same or corresponding manner in an encoding device and a decoding device. In addition, an image can be encoded / decoded using at least one or a combination of at least one of the above embodiments.

[0258] The order in which the above embodiments are applied may be different in the encoding device and the decoding device. Alternatively, the order in which the above embodiments are applied may be the same in the encoding device and the decoding device.

[0259] The above embodiments can be performed for each of the luminance and chrominance signals. Alternatively, the above embodiments can be performed identically for the luminance and chrominance signals.

[0260] In the above embodiments, the methods are described based on a flowchart as a series of steps or units. However, the present disclosure is not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps described above. Furthermore, those skilled in the art will understand that the steps depicted in the flowchart are not exclusive, and that other steps may be included, or one or more steps in the flowchart may be deleted without affecting the scope of the present disclosure.

[0261] The above embodiments may be implemented in the form of program commands that can be executed by various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure, or may be known and usable by those skilled in the art of computer software.

[0262] The bitstream generated by the encoding method according to the above embodiment can be stored in a non-transitory computer-readable recording medium. In addition, the bitstream stored in the non-transitory computer-readable recording medium can be decoded by the decoding method according to the above embodiment.

[0263] Here, examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa.

[0264] Although the present disclosure has been described above with specific details such as specific components and limited examples and drawings, these are provided only to help a more general understanding of the present disclosure, and the present disclosure is not limited to the above examples, and a person having ordinary knowledge in the technical field to which the present disclosure belongs can make various modifications and variations from this description.

[0265] Therefore, the spirit of the present disclosure should not be limited to the embodiments described above, and all modifications that are equivalent or equivalent to the following claims as well as the claims are considered to fall within the scope of the spirit of the present disclosure.

[0266] The present invention can be used in a device for encoding / decoding an image and a recording medium storing a bitstream.

Claims

A step of deriving a first intra prediction mode, a second intra prediction mode, and a third intra prediction mode of the current block based on a current template including adjacent samples of the current block among a plurality of candidate intra prediction modes; A step of deriving a first prediction block, a second prediction block, and a third prediction block of the current block based on each of the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode; and A step of deriving a final prediction block of the current block by weighting the first prediction block, the second prediction block, and the third prediction block, The third intra prediction mode is derived from one of the DC mode, the Planar mode, and the Neural-network based intra prediction (NNIP) mode, An image decoding method, characterized in that the neural network-based intra prediction mode is performed based on a neural network model that uses adjacent samples of the current block as input. In the first paragraph, Further comprising a step of obtaining information indicating whether the neural network-based intra prediction mode is used, An image decoding method, characterized in that when the information indicates that the neural network-based intra prediction mode is used, the third intra prediction mode is derived from the neural network-based intra prediction mode. In the first paragraph, The step of deriving the first intra prediction mode, the second intra prediction mode and the third intra prediction mode comprises: An image decoding method characterized in that it is performed based on the similarity between the predicted value of the current template and the current template based on each of the plurality of candidate intra prediction modes. In the third paragraph, An image decoding method characterized in that the above similarity is determined based on any one of the SATD (sum of absolute transformed differences) method, the SAD (sum of absolute differences) method, the SSD (sum of squared differences) method, and the MR-SAD (mean removed sum of absolute differences) method. In the first paragraph, An image decoding method, characterized in that the third intra prediction mode is derived based on the similarity between some pixels included in the current template and the predicted values ​​of some pixels based on the neural network-based intra prediction mode. In the first paragraph, An image decoding method, characterized in that the current template includes at least one of a left template and an upper template. In the first paragraph, The above multiple candidate intra prediction modes are classified into a first candidate intra prediction mode group and a second candidate intra prediction mode group, The first intra prediction mode and the second intra prediction mode are derived based on the first candidate intra prediction mode group, A video decoding method, characterized in that the third intra prediction mode is derived based on the second candidate intra prediction mode group. In paragraph 7, The above first candidate intra prediction mode group includes a directional intra prediction mode, An image decoding method, characterized in that the second candidate intra prediction mode group includes a DC mode, a Planar mode, and a neural network-based intra prediction mode. In the first paragraph, The weight applied to the first prediction block is determined based on the similarity between the predicted value of the current template predicted based on the first intra prediction mode and the current template, The weight applied to the second prediction block is determined based on the similarity between the predicted value of the current template predicted based on the second intra prediction mode and the current template, A video decoding method, characterized in that the weight applied to the third prediction block is determined based on the similarity between the predicted value of the current template predicted based on the third intra prediction mode and the current template. In the first paragraph, An image decoding method, characterized in that the current block is a luminance block. A step of deriving a first intra prediction mode, a second intra prediction mode, and a third intra prediction mode of the current block based on a current template including adjacent samples of the current block among a plurality of candidate intra prediction modes; A step of deriving a first prediction block, a second prediction block, and a third prediction block of the current block based on each of the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode; and A step of deriving a final prediction block of the current block by weighting the first prediction block, the second prediction block, and the third prediction block, The third intra prediction mode is derived from one of the DC mode, the Planar mode, and the Neural-network based intra prediction (NNIP) mode, An image encoding method, characterized in that the neural network-based intra prediction mode is performed based on a neural network model that uses adjacent samples of the current block as input. In a non-transitory computer-readable recording medium storing a bitstream generated by a video encoding method, The above image encoding method is, A step of deriving a first intra prediction mode, a second intra prediction mode, and a third intra prediction mode of the current block based on a current template including adjacent samples of the current block among a plurality of candidate intra prediction modes; A step of deriving a first prediction block, a second prediction block, and a third prediction block of the current block based on each of the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode; and A step of deriving a final prediction block of the current block by weighting the first prediction block, the second prediction block, and the third prediction block, The third intra prediction mode is derived from one of the DC mode, the Planar mode, and the Neural-network based intra prediction (NNIP) mode, A recording medium, characterized in that the neural network-based intra prediction mode is performed based on a neural network model that takes adjacent samples of the current block as input. In a method for transmitting a bitstream generated by a video encoding method, The above transmission method includes a step of transmitting the bitstream, The above image encoding method is, A step of deriving a first intra prediction mode, a second intra prediction mode, and a third intra prediction mode of the current block based on a current template including adjacent samples of the current block among a plurality of candidate intra prediction modes; A step of deriving a first prediction block, a second prediction block, and a third prediction block of the current block based on each of the first intra prediction mode, the second intra prediction mode, and the third intra prediction mode; and A step of deriving a final prediction block of the current block by weighting the first prediction block, the second prediction block, and the third prediction block, The third intra prediction mode is derived from one of the DC mode, the Planar mode, and the Neural-network based intra prediction (NNIP) mode, A transmission method, characterized in that the neural network-based intra prediction mode is performed based on a neural network model that takes adjacent samples of the current block as input.

Images