Image encoding / decoding method and device, and recording medium storing bit stream

Abstract

Provided is an image decoding method comprising the steps of: determining an intra prediction mode of a current block; determining a first reference sample of the current block of the reconstruction region; deriving a second reference sample of the current block of the unreconstructed region; and performing intra prediction according to an intra prediction mode based on at least one of the first reference sample and the second reference sample.

Description

Technical Field

[0001] The present invention relates to an image encoding / decoding method and apparatus, and a recording medium storing a bitstream. More particularly, the present invention relates to an image encoding / decoding method and apparatus using an intra-frame prediction method, and a recording medium storing a bitstream. Background Art

[0002] Recently, in various application fields, the demand for high-resolution, high-quality images such as ultra-high definition (UHD) images has increased. As the resolution and quality of image data become higher, the amount of data increases relatively compared to existing image data. Therefore, when image data is transmitted using media such as existing wired and wireless broadband lines or stored using existing storage media, the transmission and storage costs increase. In order to solve these problems that arise as the resolution and quality of image data become higher, efficient image encoding / decoding technology is required for images with higher resolution and quality.

[0003] Intra-frame prediction is a method for predicting block samples by referencing samples from the same picture. In the absence of an I-picture reference, blocks can be encoded / decoded using intra-frame prediction alone, so improving the accuracy of intra-frame prediction is highly desirable. Therefore, various intra-frame prediction methods are being explored to improve intra-frame prediction accuracy. Summary of the Invention

[0004] Technical issues

[0005] An object of the present invention is to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0006] In addition, an object of the present invention is to provide a recording medium for storing a bit stream generated by the image decoding method or apparatus provided by the present invention.

[0007] Technical Solution

[0008] An image decoding method according to an embodiment of the present invention may include: determining an intra-frame prediction mode of a current block; determining a first reference sample of the current block in a reconstructed area; deriving a second reference sample of the current block in an unreconstructed area; and performing intra-frame prediction according to the intra-frame prediction mode based on at least one of the determined first reference sample or the derived second reference sample.

[0009] According to one embodiment, the second reference sample may be derived based on a neural network model, and the sample of the reconstructed region is input to the neural network model.

[0010] According to one embodiment, the input samples of the reconstructed area may include at least one of a left reconstructed sample, an upper reconstructed sample, an upper left reconstructed sample, an upper right reconstructed sample, or a lower left reconstructed sample of the current block.

[0011] According to one embodiment, the input samples of the reconstructed area may consist of one or more samples in a left column and one or more samples in an upper row adjacent to the current block.

[0012] According to one embodiment, the second reference sample may be derived based on adjacent samples of a matching block searched through template matching based on a current template of the current block.

[0013] According to one embodiment, two or more matching blocks may be searched for by template matching based on a template of a current block, a second reference sample may be determined based on a weighted sum of adjacent samples at two or more corresponding positions of the two or more matching blocks, and a weight used in the weighted sum of the two or more adjacent samples may be determined by a similarity between the reference templates of the two or more matching blocks and the current template.

[0014] According to one embodiment, the second reference samples may be derived based on neighboring samples of a matching block of the reconstructed region indicated by the block vector of the current block.

[0015] According to one embodiment, the second reference samples may consist of one or more samples in a right column and one or more samples in a lower row adjacent to the current block.

[0016] According to one embodiment, determining the intra prediction mode of the current block may include determining the intra prediction mode from a predetermined intra prediction mode group, and the predetermined intra prediction mode group may include at least one right direction intra prediction mode and at least one downward direction intra prediction mode.

[0017] According to one embodiment, the predetermined intra-frame prediction mode group may include at least one left-direction intra-frame prediction mode and at least one upward-direction intra-frame prediction mode, the at least one left-direction intra-frame prediction mode may correspond one-to-one with at least one right-direction intra-frame prediction mode, and the at least one upward-direction intra-frame prediction mode may correspond one-to-one with at least one downward-direction intra-frame prediction mode.

[0018] According to one embodiment, when determining the intra-frame prediction mode of the current block, when bidirectional intra-frame prediction is applied to the current block, a first intra-frame prediction mode can be selected from at least one left-direction intra-frame prediction mode and at least one upward-direction intra-frame prediction mode of a predetermined intra-frame prediction mode group, and the first intra-frame prediction mode and the second intra-frame prediction mode can be determined as the intra-frame prediction mode of the current block, and the second intra-frame prediction mode has a prediction direction opposite to the prediction direction of the first intra-frame prediction mode.

[0019] According to one embodiment, a prediction value of a current sample may be determined based on a weighted sum of a reference sample of the current sample according to a first intra prediction mode and a reference sample of the current sample according to a second intra prediction mode, and a weight applied to the weighted sum of the reference sample may be determined by at least one of a position or a prediction direction of the current sample.

[0020] According to one embodiment, the image decoding method may include: generating a residual block of the current block by inverse transforming the transformation coefficient data of the current block, and in the inverse transform, a first inverse transform method or a second inverse transform method may be applied, the first inverse transform method applies a separable main inverse transform and a secondary inverse transform, and the second inverse transform method applies an inseparable main inverse transform.

[0021] According to one embodiment, in generating a residual block of a current block, when applying a first inverse transform method, for a separable main inverse transform, a multi-transform set can be selected from a plurality of multi-transform sets according to at least one of a transform size or an intra-frame prediction mode, and a transform core applied to the separable main inverse transform can be selected from a plurality of transform cores of the multi-transform set; for a secondary inverse transform, an inseparable transform set can be selected from a plurality of inseparable transform sets according to the intra-frame prediction mode, and a transform core applied to the secondary inverse transform can be selected from a plurality of transform cores of the inseparable transform set; and the plurality of multi-transform sets may include a multi-transform set corresponding to a right-direction intra-frame prediction mode and a downward-direction intra-frame prediction mode, and the plurality of inseparable transform sets may include an inseparable transform set corresponding to a right-direction intra-frame prediction mode and a downward-direction intra-frame prediction mode.

[0022] According to one embodiment, when the prediction directions of the first intra prediction mode and the second intra prediction mode are opposite to each other, the multi-transform set and the inseparable transform set selected when the first intra prediction mode is applied to the current block may be the same as the multi-transform set and the inseparable transform set selected when the second intra prediction mode is applied to the current block.

[0023] According to one embodiment, in generating a residual block of a current block, when applying a second inverse transform method, for an inseparable main inverse transform, an inseparable main transform set can be selected from a plurality of inseparable main transform sets according to at least one of a transform size or an intra-frame prediction mode, and a transform core applied to the inseparable main inverse transform can be selected from a plurality of transform cores of the inseparable main transform set, and the plurality of inseparable main transform sets can include inseparable main transform sets corresponding to a right-direction intra-frame prediction mode and a downward-direction intra-frame prediction mode.

[0024] According to one embodiment, when the prediction directions of the first intra prediction mode and the second intra prediction mode are symmetrical to each other, the inseparable primary transform set selected when the first intra prediction mode is applied to the current block may be the same as the inseparable primary transform set selected when the second intra prediction mode is applied to the current block.

[0025] An image encoding method according to an embodiment of the present invention may include: determining an intra-frame prediction mode of a current block; determining a first reference sample of the current block in a reconstructed area; deriving a second reference sample of the current block in an unreconstructed area; and performing intra-frame prediction according to the intra-frame prediction mode based on at least one of the determined first reference sample or the derived second reference sample.

[0026] A non-volatile computer-readable recording medium according to an embodiment of the present invention stores a bit stream generated by an image encoding method.

[0027] A transmission method according to an embodiment of the present invention transmits a bit stream generated by an image encoding method.

[0028] The features briefly summarized above are merely exemplary aspects of the detailed description of the invention described below and do not limit the scope of the invention.

[0029] Beneficial effects

[0030] The present invention proposes various embodiments of a method for generating a second reference sample of an unreconstructed area.

[0031] In addition, the present invention proposes various implementations of a method for generating an intra prediction block based on omnidirectional reference samples.

[0032] In addition, the present invention proposes various embodiments of a method of determining a transform set and a transform kernel corresponding to an intra prediction mode when omnidirectional reference samples are supported.

[0033] According to the above various embodiments, as the intra-frame prediction accuracy is improved, the overall encoding efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1is a block diagram showing the configuration of an encoding device according to an embodiment of the present invention

[0035] Figure 2 is a block diagram showing the configuration of a decoding device according to an embodiment of the present invention.

[0036] Figure 3 FIG. 1 is a diagram schematically illustrating a video coding and decoding system to which the present invention is applicable.

[0037] Figure 4 Examples of positions of first and second reference samples around the current block are shown.

[0038] Figure 5 and Figure 6 is a schematic diagram for explaining a reference sample derivation method based on a neural network according to an embodiment of the present invention.

[0039] Figure 7 An embodiment of a method of searching for a matching block referenced by a current block based on a current template of the current block is shown.

[0040] Figure 8 An embodiment of a method for determining a second reference sample referred to by a current block based on template matching is shown.

[0041] Figure 9 A method for determining a second reference sample based on two reference templates is shown.

[0042] Figure 10 A method for deriving reference samples based on block vectors is shown.

[0043] Figure 11 and Figure 12 A unidirectional intra prediction mode based on the first reference sample is shown.

[0044] Figure 13 An intra prediction mode is shown that extends the directionality of the intra prediction mode by considering omnidirectional reference samples.

[0045] Figure 14 An implementation of an intra prediction method according to the omnidirectional intra prediction mode is shown.

[0046] Figure 15 A content streaming system to which an embodiment according to the present invention is applicable is exemplarily shown. DETAILED DESCRIPTION

[0047] Best Mode for Carrying Out the Invention

[0048] An image decoding method according to an embodiment of the present invention may include: determining an intra-frame prediction mode of a current block; determining a first reference sample of the current block in a reconstructed area; deriving a second reference sample of the current block in an unreconstructed area; and performing intra-frame prediction according to the intra-frame prediction mode based on at least one of the determined first reference sample or the derived second reference sample.

[0049] Embodiments for carrying out the invention

[0050] The present invention may have various modifications and embodiments, and specific embodiments are shown in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents or alternatives contained within the spirit and technical scope of the present invention. The same figure marks in the drawings indicate the same or similar functions in various aspects. For a clearer description, the shapes and sizes of the elements in the drawings can be provided by way of example. The detailed description of the exemplary embodiments described below refers to the drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different from each other, but not necessarily mutually exclusive. For example, without departing from the spirit and scope of the present invention with reference to one embodiment, the specific shapes, structures and features described herein can be implemented in other embodiments. It should also be understood that without departing from the spirit and scope of the embodiment, the position or arrangement of each component within each disclosed embodiment can be changed. Accordingly, the detailed description set forth below is not intended to be restrictive, and the scope of the exemplary embodiments is limited only by the full range of equivalent forms given by the appended claims and these claims (if appropriately described).

[0051] In the present invention, the terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component without departing from the scope of the present invention. The terms are and / or include a combination of multiple related descriptive items or any item in multiple related descriptive items.

[0052] The components shown in the embodiments of the present invention are depicted independently to indicate different characteristic functions, and do not represent that each component is formed as a separate hardware or software configuration unit. That is, for ease of explanation, each component is listed and included as a separate component, and at least two of the components can be combined to form a single component, or a component can be divided into multiple components to perform functions. As long as it does not depart from the essence of the present invention, embodiments in which the components are integrated and embodiments in which each component is divided are also included in the scope of the present invention.

[0053] The terms used in the present invention are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In addition, some components of the present invention are not essential components for performing necessary functions in the present invention and can be optional components that are only used to improve performance. The present invention can be realized by only including essential components for realizing the gist of the present invention and not including components that are only used to improve performance, and the structure that only includes essential components and does not include optional components that are only used to improve performance is also included in the scope of the present invention.

[0054] In an embodiment, the term "at least one" may refer to one of a number greater than or equal to 1, such as 1, 2, 3, and 4. In an embodiment, the term "plurality" may refer to one of a number greater than or equal to 2, such as 2, 3, and 4.

[0055] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. When describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions will obscure the subject matter of this specification, the detailed description will be omitted, the same reference numerals will be used for the same components in the drawings, and repeated description of the same components will be omitted.

[0056] Description of terms

[0057] Hereinafter, "image" may refer to a picture constituting a video, or may refer to the video itself. For example, "encoding and / or decoding of an image" may refer to "encoding and / or decoding of a video," or may refer to "encoding and / or decoding of one of the pictures constituting the video."

[0058] Hereinafter, "moving image" and "video" may be used interchangeably with each other in the same meaning. Furthermore, a target image may be an encoding target image that is the target of encoding and / or a decoding target image that is the target of decoding. Furthermore, a target image may be an input image input to an encoding device or an input image input to a decoding device. Here, the target image may have the same meaning as the current image.

[0059] Hereinafter, “image,” “picture,” “frame,” and “picture” may be used with the same meaning and may be used interchangeably.

[0060] Hereinafter, a "target block" may be an encoding target block that is a target of encoding and / or a decoding target block that is a target of decoding. In addition, a target block may be a current block that is a target of current encoding and / or decoding. For example, "target block" and "current block" may be used with the same meaning and may be used interchangeably.

[0061] In the following, "block" and "unit" can be used with the same meaning and can be used interchangeably. In addition, "unit" can mean a luma component block and its corresponding chroma component block to distinguish it from a block. For example, a coding tree unit (CTU) can consist of a luma component (Y) coding tree block (CTB) and its associated two chroma component (Cb, Cr) coding tree blocks.

[0062] Hereinafter, “sample”, “picture element”, and “pixel” may be used with the same meaning and may be used interchangeably. Herein, a sample may represent a basic unit constituting a block.

[0063] Hereinafter, “inter-frame” and “inter-screen” may be used with the same meaning and may be used interchangeably.

[0064] Hereinafter, “intra-frame” and “intra-screen” may be used with the same meaning and may be used interchangeably.

[0065] Figure 1 is a block diagram showing the configuration of an encoding device according to an embodiment of the present invention.

[0066] The encoding device 100 may be an encoder, a video encoding device, or an image encoding device. A video may include one or more images. The encoding device 100 may encode one or more images sequentially.

[0067] refer to Figure 1 The encoding device 100 may include: an image partitioning unit 110, an intra-frame prediction unit 120, a motion prediction unit 121, a motion compensation unit 122, a switch 115, a subtractor 113, a transform unit 130, a quantization unit 140, an entropy encoding unit 150, an inverse quantization unit 160, an inverse transform unit 170, an adder 117, a filtering unit 180 and a reference picture buffer 190.

[0068] In addition, the encoding device 100 can generate a bit stream including information encoded by encoding the input image and output the generated bit stream. The generated bit stream can be stored in a computer-readable recording medium or can be streamed through a wired / wireless transmission medium.

[0069] The image partitioning unit 110 can partition the input image into various forms to improve the efficiency of video encoding / decoding. That is, the input video consists of multiple pictures, and for compression efficiency, parallel processing, etc., a picture can be partitioned and processed hierarchically. For example, a picture can be partitioned into one or more tiles or slices, and then partitioned again into multiple codec tree units (CTUs). Alternatively, a picture can first be partitioned into multiple sub-pictures defined as rectangular slice groups, and each sub-picture can be partitioned into tiles / slices. Here, sub-pictures can be used to support the function of partially independently encoding / decoding and transmitting pictures. Since multiple sub-pictures can be reconstructed separately, this has the advantage of easy editing in applications where multi-channel input is configured as one picture. In addition, tiles can be divided horizontally to generate bricks. Here, bricks can be used as basic units for parallel processing within a picture. In addition, a CTU can be recursively partitioned into a quad tree (QT), and the terminal node of the partition can be defined as a coding unit (CU). The CU can be partitioned into a PU (Prediction Unit) as a prediction unit and a TU (Transform Unit) as a transform unit to perform prediction and partitioning. On the other hand, the CU can be used as a prediction unit and / or a transform unit itself. Here, for flexible partitioning, each CTU can be recursively partitioned into multi-type trees (MTT) and quadtrees (QT). Partitioning the CTU into multi-type trees can start from the terminal node of the QT, and the MTT can be composed of a binary tree (BT) and a triple tree (TT). For example, the MTT structure can be classified into a vertical binary split mode (SPLIT_BT_VER), a horizontal binary split mode (SPLIT_BT_HOR), a vertical ternary split mode (SPLIT_TT_VER), and a horizontal ternary split mode (SPLIT_TT_HOR). In addition, during partitioning, the minimum block size (MinQTSize) of the quadtree of the luma block can be set to 16 × 16, the maximum block size (MaxBtSize) of the binary tree can be set to 128 × 128, and the maximum block size (MaxTtSize) of the ternary tree can be set to 64 × 64. In addition, the minimum block size (MinBtSize) of the binary tree and the minimum block size (MinTtSize) of the ternary tree can be specified as 4 × 4, and the maximum depth (MaxMttDepth) of the multi-type tree can be specified as 4. In addition, in order to improve the coding efficiency of the I slice, a dual tree that uses the CTU partition structure of the luma component and the chroma component differently can be applied.On the other hand, in P slices and B slices, the luma and chroma coding tree blocks (CTBs) within a CTU can be partitioned into a single tree that shares the coding tree structure.

[0070] The encoding device 100 can perform encoding on the input image in intra mode and / or inter mode. Alternatively, the encoding device 100 can perform encoding on the input image in a third mode other than intra mode and inter mode (for example, IBC mode, palette mode, etc.). However, if the third mode has functional characteristics similar to those of intra mode or inter mode, it can be classified as intra mode or inter mode for ease of explanation. In the present invention, the third mode is classified and described separately only when a specific description of the third mode is required.

[0071] When the intra mode is used as the prediction mode, the switch 115 can switch to intra, and when the inter mode is used as the prediction mode, the switch 115 can switch to inter. Here, the intra mode can represent the intra prediction mode, and the inter mode can represent the inter prediction mode. The encoding device 100 can generate a prediction block for the input block of the input image. In addition, the encoding device 100 can encode the residual block using the residual of the input block and the prediction block after generating the prediction block. The input image can be referred to as the current image that is the current encoding target. The input block can be referred to as the current block that is the current encoding target or the encoding target block.

[0072] When the prediction mode is intra mode, the intra prediction unit 120 can use samples of blocks that have been encoded / decoded around the current block as reference samples. The intra prediction unit 120 can perform spatial prediction of the current block by using the reference samples, or generate prediction samples of the input block through spatial prediction. In this article, intra prediction can refer to intra-frame prediction.

[0073] As the intra prediction method, non-directional prediction modes such as DC mode and planar mode and directional prediction modes (for example, 65 directions) can be applied. Here, the intra prediction method can be expressed as an intra prediction mode or an intra-screen prediction mode.

[0074] When the prediction mode is inter mode, the motion prediction unit 121 can retrieve the area that best matches the input block from the reference image during the motion prediction process and derive the motion vector by using the retrieved area. In this case, a search area can be used as the area. The reference image can be stored in the reference picture buffer 190. Here, when encoding / decoding the reference image, it can be stored in the reference picture buffer 190.

[0075] The motion compensation unit 122 may generate a prediction block of the current block by performing motion compensation using a motion vector. Herein, inter prediction may refer to inter-picture prediction or motion compensation.

[0076] When the value of the motion vector is not an integer, the motion prediction unit 121 and the motion compensation unit 122 may generate a prediction block by applying an interpolation filter to a partial area of ​​the reference picture. In order to perform inter-frame prediction or motion compensation, it may be determined based on the codec unit whether the motion prediction and motion compensation mode of the prediction unit included in the codec unit is one of the skip mode, merge mode, advanced motion vector prediction (AMVP) mode, and intra block copy (IBC) mode, and inter-frame prediction or motion compensation may be performed according to each mode.

[0077] In addition, based on the above-mentioned inter-frame prediction method, the affine mode based on sub-PU prediction, the subblock-based temporal motion vector prediction (SbTMVP) mode, the PU-based prediction and MVD merge (Merge with MVD, MMVD) mode and the geometric partitioning mode (Geometric Partitioning Mode, GPM) mode can be applied. In addition, in order to improve the performance of each mode, history based MVP (HMVP), pairwise average MVP (PAMVP), combined intra / inter prediction (CIIP), adaptive motion vector resolution (AMVR), bidirectional optical flow (BDOF), bidirectional prediction with CU weights (BCW), local illumination compensation (LIC), template matching (TM), overlapped block motion compensation (OBMC), etc. can be applied.

[0078] The subtractor 113 may generate a residual block by utilizing the difference between the input block and the prediction block. The residual block may be referred to as a residual signal. The residual signal may represent the difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming or quantizing, or transforming and quantizing, the difference between the original signal and the prediction signal. The residual block may be a residual signal of a block unit.

[0079] The transform unit 130 may generate a transform coefficient by performing a transform on the residual block and output the generated transform coefficient. In this article, the transform coefficient may be a coefficient value generated by performing a transform on the residual block. When the transform skip mode is applied, the transform unit 130 may skip the transform of the residual block.

[0080] The quantization level may be generated by applying quantization to the transform coefficients or the residual signal. Hereinafter, the quantization level may also be referred to as the transform coefficient in the embodiments.

[0081] For example, the 4×4 luma residual block generated by intra prediction is transformed using a discrete sine transform (DST)-based basis vector, and the remaining residual block can be transformed using a discrete cosine transform (DCT)-based basis vector. In addition, the transform block is partitioned into a quadtree shape of one block using the residual quadtree (RQT) technology, and after transforming and quantizing each transform block partitioned by the RQT, a coded block flag (cbf) can be transmitted when all coefficients become 0 to improve coding efficiency.

[0082] As another alternative, a Multiple Transform Selection (MTS) technique that selectively uses multiple transform bases to perform transforms can be applied. That is, instead of partitioning the CU into TUs through RQT, a function similar to TU partitioning can be performed through sub-block transform (SBT) technology. Specifically, SBT is only applied to inter-frame prediction blocks, and unlike RQT, the current block can be partitioned into 1 / 2 or 1 / 4 sizes in the vertical or horizontal direction, and then the transform can be performed on only one of the blocks. For example, if the current block is partitioned vertically, the transform can be performed on the leftmost or rightmost block, and if the current block is partitioned horizontally, the transform can be performed on the topmost or bottommost block.

[0083] In addition, a low-frequency non-separable transform (LFNST) can be applied. This is a secondary transform technique that additionally transforms the transformed residual signal into the frequency domain through DCT or DST. LFNST additionally transforms the 4×4 or 8×8 low-frequency region on the upper left side so that the residual coefficients can be concentrated on the upper left side.

[0084] The quantization unit 140 may generate a quantization level by quantizing the transform coefficient or the residual signal according to a quantization parameter (QP) and output the generated quantization level. Herein, the quantization unit 140 may quantize the transform coefficient by using a quantization matrix.

[0085] For example, a quantizer with a QP value of 0 to 51 may be used. Alternatively, if the image size is large and high coding efficiency is required, a QP of 0 to 63 may be used. In addition, a dependent quantization (DQ) method using two quantizers instead of one quantizer may be applied. DQ performs quantization using two quantizers (e.g., Q0 and Q1), but even without signaling information about the use of a specific quantizer, a quantizer for the next transform coefficient may be selected based on the current state through a state transition model.

[0086] The entropy coding unit 150 may generate a bitstream by performing entropy coding on the values ​​calculated by the quantization unit 140 or the codec parameter values ​​calculated when performing coding according to the probability distribution, and output the bitstream. The entropy coding unit 150 may perform entropy coding on information about samples of an image and information for decoding the image. For example, the information for decoding the image may include syntax elements.

[0087] When entropy coding is applied, symbols are represented so that a smaller number of bits are assigned to symbols with a high probability of occurrence and a larger number of bits are assigned to symbols with a low probability of occurrence, thereby reducing the size of the bitstream for the symbol to be encoded. The entropy coding unit 150 can perform entropy coding using coding methods such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. For example, the entropy coding unit 150 can perform entropy coding by utilizing a variable length coding / code (VLC) table. In addition, the entropy coding unit 150 can derive a binarization method of the target symbol and a probability model of the target symbol / binary, and perform arithmetic coding and decoding by utilizing the derived binarization method and context model.

[0088] In this regard, when CABAC is applied, in order to reduce the size of the probability table stored in the decoding device, the table probability update method can be changed to a table update method using a simple equation and applied. In addition, two different probability models can be used to obtain more accurate symbol probability values.

[0089] In order to encode the transformation coefficient level (quantization level), the entropy encoding unit 150 may change the coefficients in a two-dimensional block form into a one-dimensional vector form through a transformation coefficient scanning method.

[0090] The codec parameters may include information (flags, indexes, etc.) encoded in the encoding device 100 and signaled to the decoding device 200, such as syntax elements, and information derived during the encoding or decoding process, and may represent information required when encoding or decoding an image.

[0091] Herein, signaling a flag or an index may mean that the corresponding flag or index is entropy-encoded in an encoder and included in a bitstream, and may mean that the corresponding flag or index is entropy-decoded from a bitstream in a decoder.

[0092] The encoded current image can be used as a reference image for another image to be processed later. Therefore, the encoding device 100 can reconstruct or decode the encoded current image again and store the reconstructed or decoded image as a reference image in the reference picture buffer 190.

[0093] The quantization level may be dequantized in the dequantization unit 160 or inversely transformed in the inverse transform unit 170. The dequantized and / or inversely transformed coefficients may be added to the prediction block by the adder 117. Herein, the dequantized and / or inversely transformed coefficients may represent coefficients on which at least one of dequantization and inverse transformation has been performed, and may represent a reconstructed residual block. The dequantization unit 160 and the inverse transform unit 170 may perform the inverse process of the quantization unit 140 and the transform unit 130.

[0094] The reconstructed block may pass through the filtering unit 180. The filtering unit 180 may utilize all or some filtering techniques to apply a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter (BIF), a luma mapping with chroma scaling (LMCS) filter, etc. to reconstruct the sample, reconstruct the block, or reconstruct the image. The filtering unit 180 may be referred to as an in-loop filter. In this case, the name in-loop filter may also be used without LMCS.

[0095] A deblocking filter can remove block distortion generated at the boundaries between blocks. To determine whether to apply a deblocking filter, a determination can be made based on samples included in a number of rows or columns included in the block whether to apply the deblocking filter to the current block. When applying a deblocking filter to a block, different filters can be applied depending on the desired deblocking filter strength.

[0096] To compensate for coding errors using sample adaptive offset, an appropriate offset value can be added to the sample value. Sample adaptive offset can correct the offset from the original image in units of samples for the deblocked image. A method can be used in which the samples included in the image are partitioned into a predetermined number of regions, the regions to which the offset is applied are determined, and the offset is applied to the determined regions, or a method can be used in which the offset is applied taking into account edge information about each sample.

[0097] The bilateral filter (BIF) can also correct the deviation from the original image on a sample-by-sample basis for the image on which deblocking has been performed.

[0098] The adaptive loop filter can perform filtering based on the comparison result of the reconstructed image and the original image. The samples included in the image can be partitioned into predetermined groups, the filter to be applied to each group can be determined, and differential filtering can be performed on each group. Information on whether to apply the ALF can be signaled by the codec unit (CU), and the form and coefficients of the adaptive loop filter to be applied to each block can be varied.

[0099] In luma mapping with chroma scaling (LMCS), luma mapping (LM) refers to remapping luma values ​​using a piecewise linear model, and chroma scaling (CS) refers to a technique for scaling the residual values ​​of chroma components according to the average luma value of the prediction signal. In particular, LMCS can be used as an HDR correction technique to reflect the characteristics of High Dynamic Range (HDR) images.

[0100] The reconstructed block or reconstructed image that has passed through the filtering unit 180 may be stored in the reference picture buffer 190. The reconstructed block that has passed through the filtering unit 180 may be part of a reference image. That is, the reference image is a reconstructed image composed of the reconstructed block that has passed through the filtering unit 180. The stored reference image may be used later for inter-frame prediction or motion compensation.

[0101] Figure 2 is a block diagram showing the configuration of a decoding device according to an embodiment of the present invention.

[0102] The decoding device 200 may be a decoder, a video decoding device, or an image decoding device.

[0103] refer to Figure 2 The decoding device 200 may include: an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra-frame prediction unit 240, a motion compensation unit 250, an adder 201, a switch 203, a filtering unit 260 and a reference picture buffer 270.

[0104] The decoding device 200 can receive the bitstream output from the encoding device 100. The decoding device 200 can receive the bitstream stored in a computer-readable recording medium, or can receive the bitstream streamed via a wired / wireless transmission medium. The decoding device 200 can decode the bitstream in intra-frame mode or inter-frame mode. In addition, the decoding device 200 can generate a reconstructed image or a decoded image generated by decoding, and output the reconstructed image or the decoded image.

[0105] When the prediction mode for decoding is intra mode, the switch 203 may switch to intra. Alternatively, when the prediction mode for decoding is inter mode, the switch 203 may switch to inter.

[0106] The decoding device 200 can obtain a reconstructed residual block by decoding the input bitstream and generate a prediction block. When the reconstructed residual block and the prediction block are obtained, the decoding device 200 can generate a reconstructed block that becomes the decoding target by adding the reconstructed residual block and the prediction block. The decoding target block can be referred to as the current block.

[0107] The entropy decoding unit 210 may generate symbols by entropy decoding the bitstream according to the probability distribution. The generated symbols may include symbols in the form of quantization levels. In this article, the entropy decoding method may be the inverse process of the above-mentioned entropy encoding method.

[0108] The entropy decoding unit 210 may change the coefficients of the one-dimensional vector shape into coefficients of the two-dimensional block shape through a transform coefficient scanning method to decode the transform coefficient level (quantization level).

[0109] The quantization level may be dequantized in the dequantization unit 220 or inversely transformed in the inverse transform unit 230. The quantization level may be the result of dequantization and / or inverse transformation and may be generated as a reconstructed residual block. In this article, the dequantization unit 220 may apply a quantization matrix to the quantization level. The dequantization unit 220 and the inverse transform unit 230 applied to the decoding device may apply the same technology as the dequantization unit 160 and the inverse transform unit 170 applied to the aforementioned encoding device.

[0110] When intra mode is used, the intra prediction unit 240 can generate a prediction block by performing spatial prediction on the current block using sample values ​​of blocks already decoded around the target block. The intra prediction unit 240 applied to the decoding device can apply the same technology as the intra prediction unit 120 applied to the aforementioned encoding device.

[0111] When using inter-frame mode, the motion compensation unit 250 can generate a prediction block by performing motion compensation on the current block using a motion vector and a reference image stored in the reference picture buffer 270. When the value of the motion vector is not an integer value, the motion compensation unit 250 can generate a prediction block by applying an interpolation filter to a partial area within the reference image. In order to perform motion compensation, it can be determined based on the codec unit whether the motion compensation mode of the prediction unit included in the corresponding codec unit is skip mode, merge mode, AMVP mode, or current picture reference mode, and motion compensation can be performed according to each mode. The motion compensation unit 250 applied to the decoding device can apply the same technology as the motion compensation unit 122 applied to the above-mentioned encoding device.

[0112] The adder 201 can generate a reconstructed block by adding the reconstructed residual block and the prediction block. The filtering unit 260 can apply at least one of an inverse LMCS, a deblocking filter, a sample adaptive offset, and an adaptive loop filter to the reconstructed block or the reconstructed image. The filtering unit 260 applied to the decoding device can apply the same filtering technology as the filtering unit 180 applied to the aforementioned encoding device.

[0113] The filtering unit 260 may output a reconstructed image. The reconstructed block or image may be stored in the reference picture buffer 270 and used for inter-frame prediction. The reconstructed block that has passed through the filtering unit 260 may be part of a reference image. In other words, the reference image may be a reconstructed image composed of the reconstructed blocks that have passed through the filtering unit 260. The stored reference image may be later used for inter-frame prediction or motion compensation.

[0114] Figure 3 FIG. 1 is a diagram schematically illustrating a video coding and decoding system to which the present invention is applicable.

[0115] The video coding system according to the embodiment may include an encoding device 10 and a decoding device 20. The encoding device 10 may transmit encoded video and / or image information or data to the decoding device 20 in the form of a file or streaming via a digital storage medium or a network.

[0116] The encoding device 10 according to the embodiment may include a video source generating unit 11, an encoding unit 12, and a transmitting unit 13. The decoding device 20 according to the embodiment may include a receiving unit 21, a decoding unit 22, and a rendering unit 23. The encoding unit 12 may be referred to as a video / image encoding unit, and the decoding unit 22 may be referred to as a video / image decoding unit. The transmitting unit 13 may be included in the encoding unit 12. The receiving unit 21 may be included in the decoding unit 22. The rendering unit 23 may include a display unit, and the display unit may be configured as a separate device or an external component.

[0117] The video source generation unit 11 can obtain a video / image by capturing, synthesizing, or generating a video / image. The video source generation unit 11 may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive including previously captured videos / images, etc. The video / image generation device may include, for example, a computer, a tablet computer, a smartphone, etc., and may (electronically) generate the video / image. For example, a virtual video / image may be generated by a computer, etc., in which case the video / image capture process may be replaced by a process for generating relevant data.

[0118] The encoding unit 12 can encode the input video / image. For compression and encoding efficiency, the encoding unit 12 can perform a series of processes such as prediction, transformation and quantization. The encoding unit 12 can output the encoded data (encoded video / image information) in the form of a bit stream. The detailed configuration of the encoding unit 12 can also be the same as above. Figure 1 The encoding device 100 is configured in the same manner.

[0119] The transmitting unit 13 can transmit the encoded video / image information or data output in the form of a bitstream to the receiving unit 21 of the decoding device 20 via a digital storage medium or a network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit 13 may include components for generating a media file in a predetermined file format and may also include components for transmission via a broadcast / communication network. The receiving unit 21 may extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit 22.

[0120] The decoding unit 22 can decode the video / image by performing a series of processes such as inverse quantization, inverse transformation, and prediction corresponding to the operations of the encoding unit 12. The detailed configuration of the decoding unit 22 can also be compared with Figure 2 The above-mentioned decoding device 200 is configured in the same manner.

[0121] The rendering unit 23 may render the decoded video / image. The rendered video / image may be displayed by the display unit.

[0122] In the present invention, various implementations of omnidirectional intra prediction for intra prediction will be described. Omnidirectional intra prediction refers to a technique for performing intra prediction on the current block using reference samples in all directions surrounding the current block. Considering that the encoding and decoding order is generally based on raster scanning, the left reference samples and the upper reference samples of the current block are generally used for intra prediction of the current block. However, by deriving the right reference samples and the lower reference samples of the current block in various ways, intra prediction modes that only have left or upper directivity can be extended omnidirectionally. Therefore, in the present invention, methods for generating intra prediction blocks using omnidirectional reference samples and methods for deriving right and lower reference samples of the current block are discussed. In addition, when introducing omnidirectional intra prediction, it is necessary to discuss methods for selecting transform kernels for intra prediction modes with right or lower directivity. Therefore, in the present invention, methods for selecting transform kernels for blocks to which omnidirectional intra prediction is applied are discussed.

[0123] In the following, Figures 4 to 10Various methods for generating second reference samples using reconstructed samples around the current block are proposed in . Specifically, a method for generating second reference samples based on a neural network, a method for generating second reference samples based on template matching (TM), and a method for generating second reference samples based on motion information will be described.

[0124] Figure 4 Examples of positions of first and second reference samples around the current block are shown.

[0125] exist Figure 4 , the first reference samples may include samples located at the left side L1, L2, L3, L4, BL, L6, L7, and L8 402, the upper side A1, A2, A3, A4, AR, A6, A7, and A8 404, and the upper left side AL 406 of the current block 400. In addition, the second reference samples may include samples located at the right side R1, R2, R3, and R4 408, the lower side B1, B2, B3, and B4 410, and the lower right side BR 412 of the current block.

[0126] Figure 4 An example of the positions of the first reference sample and the second reference sample is shown. Accordingly, the samples included in the first reference sample and the second reference sample can be determined based on the position of the current block. The number of samples included in the first reference sample and the second reference sample is not limited.

[0127] Figure 5 and Figure 6 is a schematic diagram for explaining a reference sample derivation method based on a neural network according to an embodiment of the present invention.

[0128] refer to Figure 5 , the encoder / decoder may input samples of the reconstructed area 520 surrounding the current block 510 to the neural network processing unit 500 to derive second reference samples of the unreconstructed area. The second reference samples of the unreconstructed area may derive lower reference samples B1 to B8 530, right reference samples R1 to R4 540, and lower right reference sample BR 550.

[0129] On the other hand, the reconstructed area 520 used as the input of the neural network processing unit may include the left sample area (L1×2h), the upper sample area (2w×L2), and the upper left sample area (L1×L2) of the current block. Specifically, the number of reconstructed samples used as the input of the neural network processing unit may be (L1×2h)+(2w×L2)+(L1×L2). As another example, the number of reconstructed samples used as the input of the neural network processing unit may be (L1×h)+(w×L2)+(L1×L2). L1 represents the left width of the reconstructed area 520, and L2 represents the upper width of the reconstructed area 520.

[0130] On the other hand, the size of the reconstruction region 520 can be determined based on signaling information. For example, the sizes of L1 and L2 can be sent from the encoder to the decoder as predetermined syntax elements. Alternatively, the sizes of L1 and L2 can be determined by considering the size, color components, shape, etc. of the current block 510. Alternatively, the sizes of L1 and L2 can have fixed values.

[0131] exist Figure 5 In the case of , only one row of samples on the lower side and one column of samples on the right side are generated by the neural network processing unit. Figure 6 In the embodiment, two or more rows or columns of right reference samples, lower reference samples and lower-right reference samples may be generated.

[0132] refer to Figure 6 , the encoder / decoder can derive samples of the unreconstructed region 630 by inputting samples of the reconstructed region 620 surrounding the current block 610 into the neural network processing unit 600. The number of samples in the unreconstructed region 630 may be (w×N)+(M×h)+(M×N). Here, L1 represents the left width of the reconstructed region 620, and L2 represents the top width of the reconstructed region 620. In addition, M represents the right width of the unreconstructed region 630, and N represents the bottom width of the unreconstructed region 630.

[0133] On the other hand, the size of the unreconstructed region 630, which is the output of the neural network processing unit, can be determined based on signaling information. For example, the sizes of M and N can be sent from the encoder to the decoder as predetermined syntax elements. Alternatively, the sizes of M and N can be determined by considering the size, color components, shape, etc. of the current block 610. Alternatively, the sizes of M and N can have fixed values.

[0134] The neural network processing unit can be implemented as a neural network model. Here, the artificial neural network model can represent a deep neural network including one or more neural layers. In addition, the neural network model can include all or some of the convolutional layers, fully connected layers, and pooling layers. The neural network model can be implemented in a form including a single type of neural network layer, or in a form combining different types of layers.

[0135] On the other hand, the initial internal parameters of the neural network model used in the neural network processing unit are pre-learned, but can be additionally learned during the encoding / decoding process.

[0136] Figures 7 to 9 This figure is a schematic diagram for explaining a method for deriving reference samples based on template matching according to an embodiment of the present invention. The intra template matching prediction (Intra TMP) method refers to a method of using template matching to search for the best prediction block in the reconstructed area of ​​the current picture and copy it to generate a prediction block for the current block.

[0137] Figure 7 An embodiment of a method of searching for a matching block referenced by a current block based on a current template of the current block is shown.

[0138] refer to Figure 7 , the adjacent ┌-shaped area (i.e., the left area, the upper area, and the upper-left area) of the current block 710 can be defined as the current template 720. Then, a reference template 730 that is most similar to the current template 720 can be searched within the predefined search ranges R1, R2, R3, and R4 of the reconstructed area of ​​the current picture 700. In addition, based on the reference template 730, a matching block 740 to be referenced by the current block 710 is determined.

[0139] The predefined search ranges R1, R2, R3, and R4 may be defined as the current coding tree unit (CTU), the upper left CTU, the upper CTU, and the left CTU of the current block, respectively. Furthermore, within the predefined search ranges, the reference template may be searched based on a predefined search order. For example, the reference template may be searched in a zigzag order of R1, R4, R3, and R2. Furthermore, information about the search range may be determined by the encoder and sent to the decoder.

[0140] The size and shape of the current template 720 of the current block 710 can be arbitrarily determined. For example, the current template 720 can include both the left reconstruction area and the upper reconstruction area of ​​the current block 710. In addition, the current template 720 can include only the upper reconstruction area or the left reconstruction area of ​​the current block 710. Information about the size and shape of the current template can be determined by the encoder and sent to the decoder.

[0141] In addition, the upper width T1 and the left width T2 of the current template 720 can be determined by the encoder and sent to the decoder. Alternatively, T1 and T2 can be determined by the size, color components, and shape of the current block. Alternatively, T1 and T2 can have fixed values ​​regardless of the current block.

[0142] Figure 8 An embodiment of a method for determining a second reference sample referred to by a current block based on template matching is shown.

[0143] First, something like Figure 7 , searching for reference template 830 based on current template 820. Figure 7 In , the corresponding matching block 740 is determined from the reference template 730, and in Figure 8 , a corresponding matching reference sample 850 is determined from the reference template 830. Then, a second reference sample of a second reference sample region 860 of the current block 810 is determined from the matching reference sample 850.

[0144] The matching reference sample 850 includes the right reference sample, the lower reference sample, and the lower right reference sample of the matching block 840 .

[0145] In addition, the size of the second reference sample area is (w×L2)+(L1×h)+(L1×L2). Here, w and h represent the width and height of the current block, and L1 and L2 represent the upper width and left width of the second reference sample area. w, h, L1, and L2 are arbitrary positive numbers. L1 and L2 can be determined based on the size, color components, and shape of the current block. Alternatively, L1 and L2 can be determined by the encoder and sent to the decoder. Alternatively, L1 and L2 can be fixed values.

[0146] exist Figure 8 In the embodiment, one reference template 830 is used to determine the second reference sample 860. However, according to an embodiment, the second reference sample 860 may be determined by synthesizing matching reference samples from N templates, where N is any positive number greater than or equal to 2.

[0147] Figure 9 A method for determining a second reference sample based on two reference templates is shown.

[0148] according to Figure 9 , searching for a first reference template 902 and a second reference template 912 corresponding to the current template of the current block. Then, a first matching reference sample 904 is derived from the first reference template 902, and a second matching reference sample 914 is derived from the second reference template 912. Then, a second reference sample of the current block is determined by a weighted sum of the first matching reference sample 904 and the second matching reference sample 914. By determining the second reference sample of the current block based on two or more reference templates, the accuracy of the second reference sample of the current block can be improved.

[0149] At this time, weights w1 and w2 in the weighted sum are applied to the first matching reference sample 904 and the second matching reference sample 914, respectively. Here, 0≤w1≤1, 0≤w2≤1, and w-1+w-2=1 are satisfied. w1 and w2 can be sent from the encoder or can be fixed values. Alternatively, w1 and w2 can be determined based on the similarity between the first reference template 902 and the current template, and the similarity between the second reference template 912 and the current template. For example, a high weight can be set to be applied to the matching reference template of the matching block with high similarity.

[0150] According to the above method for generating second reference samples based on template matching, the second reference samples can be generated by performing the same template matching process at the decoder as that at the encoder without transmitting / parsing syntax related to the generation of the second reference samples.

[0151] Figure 10 A method for deriving reference samples based on block vectors is shown.

[0152] Different from the above method of generating reference samples based on template matching, Figure 10 The method for deriving reference samples based on block vectors described in transmits / parses syntax associated with block vectors to derive second reference samples.

[0153] exist Figure 10 In the embodiment, the block vector 1050 of the current block 1020 is used to search for a matching block 1030 that is most similar to the current block within the predefined search ranges R1, R2, R3, and R4 in the reconstructed area. Alternatively, the block vector 1050 of the current block 1020 can be used to search for a matching block 1030 that is most similar to the current block in an area outside the predefined search range. Then, a second reference sample 1010 of the current block 1020 can be determined from the matching reference sample 1040 of the determined matching block 1030.

[0154] At this time, the size of the second reference sample area is (w×L2)+(L1×h)+(L1×L2). Here, w and h represent the width and height of the current block, and L1 and L2 represent the upper and left widths of the second reference sample area. w, h, L1, and L2 are arbitrary positive numbers. L1 and L2 can be determined based on the size, color components, and shape of the current block. Alternatively, L1 and L2 can be determined in the encoder and sent to the decoder. Alternatively, L1 and L2 can be fixed values.

[0155] In the following, based on Figures 11 to 13 , a method of generating an intra-frame prediction block based on a first reference sample and a generated second reference sample will be described.

[0156] Figure 11 and Figure 12 The unidirectional intra prediction mode based on the first reference sample (left reference sample, upper reference sample and upper left reference sample) is shown. Figure 11 and Figure 12 The unidirectional intra prediction mode predicts the current block based on the first reference samples in the left, top, and upper-left directions. Therefore, the unidirectional intra prediction mode consists of prediction modes in the left, top, upper-left, upper-right, and lower-left directions. Figure 11 The intra prediction mode group includes a total of 33 directional intra prediction modes in the left or upper direction. Figure 12 The intra prediction mode group includes a total of 65 directional intra prediction modes in the left or upper direction.

[0157] As described above, the current block can be predicted by additionally referring to the second reference samples (right reference samples, lower reference samples, and lower-right reference samples). Therefore, the directionality of the intra prediction mode can be extended to the right direction, lower direction, and lower-right direction.

[0158] Figure 13 An intra prediction mode is shown in which the directionality of the intra prediction mode is extended by considering omnidirectional reference samples (first reference samples and second reference samples).

[0159] exist Figure 13 In the omnidirectional intra prediction mode, the omnidirectional intra prediction mode may include intra prediction modes in the left, upper and upper left directions and the right, lower and lower right directions. Figure 11 and Figure 12 The intra prediction mode is different, and the intra prediction mode can be as follows Figure 13 Shown expanded in omnidirectional directions.

[0160] Figure 13 An example of omnidirectionally extending 2M-2 (planar mode, DC mode, mode 2, ..., mode 2M-3) intra prediction modes is shown. Here, M is an arbitrary positive integer.

[0161] If Figure 13 The omnidirectional intra prediction mode proposed in Figure 11 If the 35 intra prediction modes are used, the number of intra prediction modes can be 65. Figure 13 The omnidirectional intra prediction mode proposed in Figure 12 If there are 67 intra prediction modes, the number of intra prediction modes can be 129. This method can be used as an example by omnidirectionally extending N intra prediction modes. Here, N is an arbitrary positive integer.

[0162] Therefore, the intra-frame prediction mode of the current block can be determined from the intra-frame prediction mode group, and the intra-frame prediction mode group includes at least one right-direction intra-frame prediction mode, at least one downward-direction intra-frame prediction mode, at least one left-direction intra-frame prediction mode, and at least one upward-direction intra-frame prediction mode. In addition, in the intra-frame prediction mode group, at least one left-direction intra-frame prediction mode can be set to correspond one-to-one with at least one right-direction intra-frame prediction mode, and at least one upward-direction intra-frame prediction mode can be set to correspond one-to-one with at least one downward-direction intra-frame prediction mode. Here, the one-to-one correspondence can be established between two directional intra-frame prediction modes with opposite prediction directions. In addition, the left-direction intra-frame prediction mode refers to a mode with a prediction direction close to the left direction, the upward-direction intra-frame prediction mode refers to a mode with a prediction direction close to the upward direction, the right-direction intra-frame prediction mode refers to a mode with a prediction direction close to the right direction, and the downward-direction intra-frame prediction mode refers to a mode with a prediction direction close to the downward direction.

[0163] Figure 13 The angular differences in the prediction directions of adjacent intra prediction modes may be non-uniform. However, according to an embodiment, the angular differences in the prediction directions of adjacent intra prediction modes may be set to be uniform.

[0164] according to Figure 11 and Figure 12 The unidirectional intra prediction mode of the intra prediction is based on Figure 11 and Figure 12 The prediction block is generated by copying the values ​​of the first reference samples (left reference sample, upper reference sample and upper left reference sample) according to the directionality of the intra prediction mode. Figure 13 The unidirectional intra prediction can refer to the second reference sample (right reference sample, lower reference sample and lower right reference sample) and the first reference sample according to the prediction direction. Figure 13 The proposed unidirectional intra prediction method using omnidirectional reference samples is the same as the existing unidirectional intra prediction method that generates intra prediction blocks according to the directionality of the intra prediction mode.

[0165] A problem with unidirectional intra prediction using only the first reference sample is that the prediction accuracy for current samples that are far from the reconstructed reference sample is low. However, by using unidirectional intra prediction using both the first and second reference samples, the prediction accuracy for current samples that are far from the reconstructed reference sample can be improved.

[0166] In addition to unidirectional intra prediction, the current block can also be predicted using bidirectional intra prediction by referencing both a first reference sample on the left, upper, and upper left sides and a second reference sample on the right, lower, and lower right sides. Bidirectional intra prediction, by referencing both the first and second reference samples, can further improve the prediction accuracy of current samples that are far from the reconstructed reference samples. Bidirectional intra prediction allows reference to two reference samples located in opposite directions, and the current sample can be predicted using a weighted sum of the two reference samples. In this case, the weight used in the weighted sum of the two reference samples is determined by the position of the current sample and the prediction direction.

[0167] The following describes a method for determining a transform kernel for a current block based on the aforementioned omnidirectional intra prediction mode. First, the method for determining a transform kernel for a current block when a separable primary transform and secondary transform are applied will be described. Next, the method for determining a transform kernel for a current block when a non-separable primary transform is applied will be described.

[0168] Hereinafter, a method of determining a transform kernel of a primary transform and a transform kernel of a secondary transform for an omni-directional intra prediction mode will be described.

[0169] exist Figure 13 In , an intra prediction mode for a prediction direction referring to a second reference sample (a right reference sample, a lower reference sample, and a lower-right reference sample) is introduced. Figure 13 Modes M+1 to 2M-3 are new intra prediction modes based on the second reference sample. Therefore, a method of determining a transform kernel of a separable primary transform and a transform kernel of a secondary transform for Modes M+1 to 2M-3 will be described below.

[0170] First, the method of determining the transform kernel of the separable main transform and the transform kernel of the secondary transform for Mode M+1 to Mode 2M-3 will be explained. In the separable main transform, separable transform kernels in the horizontal and vertical directions are used. The separable main transform can use various transform kernels, such as Discrete Cosine Transform (DCT) Type 2, DCT Type 7, DCT Type 8, DCT Type 5, DCT Type 4, DST Type 1, and Identity Transform (IDT).

[0171] In a separable main transform, a multi-transform set and / or a transform kernel may be determined based on the transform unit size (TU size) and the intra prediction mode information. In this case, the transform set and / or the transform kernel may be determined based on at least one or more of various information (e.g., the width, height, area, aspect ratio, intra prediction mode information, whether a secondary transform is used, a quantization parameter (QP), statistical characteristics of neighboring pixels, and syntax elements explicitly signaled). With this in mind, Figure 13 The omnidirectional prediction modes are grouped into N transform sets, and K transform kernels can be configured to be included in each transform set. Here, N and K are positive integers satisfying 1≤N≤2M-2 and 1≤K, respectively.

[0172] At this point, offline training can be used to generate Figure 13 The newly generated separable main transform kernels of modes M+1 to 2M-3 in the right, bottom, and bottom-right directions are applied to the encoder and decoder. Alternatively, appropriate transform kernels can be derived in the encoder and decoder respectively to reflect the diversity of content in the image.

[0173] Alternatively, predefined transform kernels can be divided into multiple categories. Furthermore, syntax elements indicating which transform kernels corresponding to which lists are to be used can be transmitted at higher or lower layers. Alternatively, a transform kernel of a predetermined separable primary transform can be used based on at least one of transform size or intra prediction mode. Here, the higher layers include sequence parameter sets, picture parameter sets, video parameter sets, adaptation parameter sets, picture headers, slice headers, etc. Furthermore, the lower layers include codec tree units, codec units, transform units, etc.

[0174] According to one embodiment, the core for Mode 2 to Mode M can be applied to Figure 13 For example, the same transform kernel can be set to be used between intra prediction modes whose prediction directions are opposite to each other. Figure 13 Used between Mode 3 and Mode M+1.

[0175] Multiple transform sets and / or transform kernels for a separable primary transform can be determined using either an explicit or implicit method. According to the explicit method, omnidirectional intra prediction modes are grouped into N transform sets. Furthermore, K transform kernels can be used for each transform set. Therefore, a transform set and / or transform kernel can be selected by transmitting / parsing the index of the transform set and / or transform kernel for the current block.

[0176] The implicit method can implicitly derive information about the transform set and transform kernel of the current block in the encoder or decoder. That is, the transform kernel for the separable main transform can be implicitly derived by using at least one of the width, height, area, aspect ratio, intra prediction mode information, whether to use secondary transform, quantization parameter, neighboring pixels of the current block, or other information of the current block.

[0177] A secondary transform is performed on the coefficients in the low-frequency region of the primary transform coefficients generated by the separable primary transform. Unlike the separable primary transform, which uses horizontally and vertically separable kernels, the secondary transform uses a non-separable kernel. Therefore, the secondary transform is called a low-frequency non-separable transform (LFNST).

[0178] The transform set and / or transform kernel of the low-frequency inseparable transform may be determined based on information about the intra prediction mode. In this case, the transform set and / or transform kernel may be determined based on at least one of the width, height, area, aspect ratio, intra prediction mode information, quantization parameter, statistical characteristics of neighboring pixels, and explicitly signaled syntax elements of the current block. With this in mind, Figure 13 The omnidirectional intra prediction modes are grouped into S transform sets, and T transform kernels can be configured to be included in each transform set. Here, S and T are positive integers satisfying 1≤S≤2M-2 and 1≤T, respectively.

[0179] At this point, offline training can be used to generate Figure 13 The newly generated transform kernels for the secondary transform of mode M+1 to mode 2M-3 in the right, bottom, and right-bottom directions are applied to the encoder and decoder. Alternatively, appropriate kernels can be derived in the encoder and decoder respectively to reflect the diversity of content in the image. Alternatively, the predefined transform kernels can be divided into multiple categories. In addition, syntax elements indicating which kernels corresponding to which list to use can be transmitted at a high or low level.

[0180] According to one embodiment, the core for Mode 2 to Mode M can be applied to Figure 13 For example, the same transform kernel can be set to be used between intra prediction modes whose prediction directions are opposite to each other. Figure 13 Used between Mode 3 and Mode M+1.

[0181] Either an explicit method or an implicit method can be used to determine the inseparable transform set and / or transform kernel for the secondary transform. According to the explicit method, the omnidirectional intra prediction modes are grouped into S transform sets. In addition, T transform kernels can be used for each transform set. Therefore, the transform set and / or transform kernel can be selected by transmitting / parsing the index of the transform set and / or transform kernel of the current block.

[0182] The implicit method can implicitly derive information about the transform set and transform kernel of the current block in the encoder or decoder. That is, the transform kernel used for the secondary transform can be implicitly derived by using at least one of the width, height, area, aspect ratio, intra prediction mode information, whether the secondary transform is used, quantization parameter, neighboring pixels of the current block, or other information about the current block.

[0183] Hereinafter, a method of determining a transform kernel of a primary transform and a transform kernel of a secondary transform by considering directionality of an intra prediction mode will be described.

[0184] according to Figure 13 , the intra prediction mode can include two non-directional intra prediction modes (planar mode and DC mode) and 2M-4 directional intra prediction modes. Here, M is an arbitrary positive integer. In addition to the two non-directional intra prediction modes, the remaining 2M-4 directional intra prediction modes can have the same directionality. That is, Figure 13 As shown, mode 2 and mode M, mode 3 and mode M+1, and mode M-1 and mode 2M-3 have the same directionality. Considering the characteristics of the intra prediction mode, separate transform sets and / or transform kernels for primary and secondary transforms may be determined.

[0185] Any one of an explicit method and an implicit method can be used to determine the multiple transform sets and / or transform kernels of the main transform. According to the explicit method, the omnidirectional intra-frame prediction modes are grouped into N transform sets. In addition, K transform kernels can be used for each transform set. Therefore, the transform set and / or transform kernel can be selected by transmitting / parsing the index of the transform set and / or transform kernel of the current block. In addition, taking into account the directionality of the intra-frame prediction mode, the same transform set and / or transform kernel can be applied to a pair of intra-frame prediction modes (mode X, mode X+(M-2)) with the same directionality. In this case, N and K are positive integers satisfying 1≤N≤(2M-4) / 2 and 1≤K, respectively.

[0186] The implicit method can implicitly derive information about the transform set and transform kernel of the current block in the encoder or decoder. That is, the transform kernel for the separable main transform can be implicitly derived by using at least one of the width, height, area, aspect ratio, intra prediction mode information, whether to use secondary transform, quantization parameter, neighboring pixels of the current block, and other information of the current block.

[0187] Any of the explicit method and the implicit method can be used to determine the inseparable transform set and / or transform kernel of the secondary transform. According to the explicit method, the omnidirectional intra-frame prediction modes are grouped into S transform sets. In addition, T transform kernels can be used for each transform set. Therefore, the transform set and / or transform kernel can be selected by transmitting / parsing the index of the transform set and / or transform kernel of the current block. In addition, taking into account the directionality of the intra-frame prediction mode, the same transform set and / or transform kernel can be applied to a pair of intra-frame prediction modes (mode X and mode X+(M-2)) with the same directionality. In this case, S and T are positive integers satisfying 1≤S≤(2M-4) / 2 and 1≤T, respectively.

[0188] The implicit method can implicitly derive information about the transform set and transform kernel of the current block in the encoder or decoder. That is, the transform kernel for the secondary transform can be implicitly derived by using at least one of the current block width, height, area, aspect ratio, intra prediction mode information, whether to use secondary transform, quantization parameter, neighboring pixels of the current block, and other information about the current block.

[0189] Hereinafter, a method of determining a transform kernel of a non-separable main transform for an omni-directional intra prediction mode will be described.

[0190] In the non-separable primary transform, similar to the low frequency non-separable transform (LFNST), non-separable kernels are used in the horizontal and vertical directions instead of separable kernels. In the non-separable primary transform, the transform set and / or transform kernel can be determined based on the intra prediction mode information.

[0191] exist Figure 13 In , an intra prediction mode for a prediction direction referring to a second reference sample (a right reference sample, a lower reference sample, and a lower-right reference sample) is introduced. Figure 13 Modes M+1 to 2M-3 in [ 0 ] are new intra prediction modes based on the second reference sample. Therefore, a method of determining a transform kernel for the inseparable main transform of modes M+1 to 2M-3 will be described below.

[0192] In a non-separable primary transform (NSPT), a transform set and / or a transform kernel may be determined based on the transform size and intra prediction mode information. In this case, the transform set and / or the transform kernel may be determined based on at least one of various information such as the width, height, area, aspect ratio, intra prediction mode information, whether a secondary transform is used, quantization parameters, statistics of neighboring pixels, and syntax elements explicitly signaled of the current block. With this in mind, Figure 13 The omnidirectional prediction modes are grouped into N transform sets, and K transform kernels can be configured to be included in each transform set. Here, N and K are positive integers satisfying 1≤N≤2M-2 and 1≤K, respectively.

[0193] At this point, offline training can be used to generate Figure 13 The newly generated transform kernels of the separable main transform for modes M+1 to 2M-3 in the right, bottom, and bottom-right directions are applied to the encoder and decoder. Alternatively, appropriate kernels can be derived in the encoder and decoder respectively to reflect the diversity of content in the image.

[0194] Alternatively, the predefined transform kernels may be divided into multiple categories. In addition, a syntax element indicating which transform kernels corresponding to which list are to be used may be transmitted at a high or low layer. Alternatively, a transform kernel of an inseparable main transform predetermined according to the intra prediction mode may be used.

[0195] Either an explicit method or an implicit method can be used to determine the multiple transform sets and / or transform kernels of the inseparable primary transform. According to the explicit method, the omnidirectional intra prediction modes are grouped into N transform sets. In addition, K transform kernels can be used for each transform set. Therefore, the transform set and / or transform kernel can be selected by transmitting / parsing the index of the transform set and / or transform kernel of the current block.

[0196] The implicit method can implicitly derive information about the transform set and transform kernel of the current block in the encoder or decoder. That is, the transform kernel for the inseparable main transform can be implicitly derived using at least one of the width, height, area, aspect ratio, intra prediction mode information, whether a secondary transform is used, quantization parameter, neighboring pixels of the current block, and other information of the current block.

[0197] Hereinafter, a method of determining a transform kernel of a non-separable main transform by considering the symmetry of an intra prediction mode will be described.

[0198] according to Figure 13, the intra prediction mode can include two non-directional intra prediction modes (planar mode and DC mode) and 2M-4 directional intra prediction modes. Here, M is an arbitrary positive integer. In addition to the two non-directional intra prediction modes, the remaining 2M-4 directional intra prediction modes can be symmetrical with each other around mode M / 2+1 (prediction mode in the upper left prediction direction). That is, Figure 13 As shown, mode 3 and mode M-1, as well as mode 2M-3 and mode M+1, are symmetrical to each other. For example, when M is 66, mode 2 and mode 66, mode 18 and mode 50, etc., may be symmetrical to each other. Considering the characteristics of such intra prediction modes, a transform set and / or transform kernel of an inseparable main transform may be determined.

[0199] In order to determine the transform set and / or transform kernel of the inseparable main transform, the omnidirectional intra prediction mode is grouped into N transform sets. In addition, K transform kernels can be used in each transform set. Therefore, the transform set and / or transform kernel can be selected by transmitting / parsing the index of the transform set and / or transform kernel of the current block. In addition, taking into account the symmetry of the intra prediction mode, the same transform set and / or transform kernel can be applied to a pair of intra prediction modes that are symmetrical around mode M / 2+1. In this case, N and K are positive integers satisfying 1≤N≤(2M-4) / 2 and 1≤K, respectively.

[0200] When a prediction mode symmetric to prediction mode M / 2+1 is applied to the current block, the transform kernel is applied to the transposed input data. Transposing the input data represented as a two-dimensional matrix refers to configuring the N×M two-dimensional block data using the transposed rows and columns in the two-dimensional block data of size M×N (M and N are positive integers greater than or equal to 2). For example, if the data arrangement order (the order when arranging two-dimensional data in one dimension for the main transform) for configuring the one-dimensional input vector of mode 2M-3 is in the row-first direction, then the data arrangement order of mode M+1, which is symmetric to mode 2M-3, is in the column-first direction. Conversely, if the data arrangement order for the one-dimensional input vector used to configure mode 2M-3 is in the column-first direction, then the data arrangement order of mode M+1, which is symmetric to mode 2M-3, is in the row-first direction. This data arrangement order is also performed in the inverse transform of the inseparable main transform. The prediction directions of modes M / 2+1 and 3M / 2-1 are centered between the horizontal and vertical directions. Therefore, when mode M / 2+1 or mode 3M / 2-1 is applied to the current block, the two-dimensional block data can be arranged into a one-dimensional input vector by selecting row-first direction or column-first direction arrangement.

[0201] As a specific example, when M is 66, mode 82 and mode 114 may be symmetrical to each other. Mode 82 and mode 114 may correspond to the same transform set and / or transform kernel. When the current block to which mode 82 is applied is transformed, the two-dimensional input data may be sorted into a one-dimensional input vector in a row-major direction, and the transform kernel may be applied to the sorted one-dimensional input vector. In addition, when the current block to which mode 114 is applied is transformed, the two-dimensional input data may be sorted into a one-dimensional input vector in a column-major direction, and the transform kernel may be applied to the sorted one-dimensional input vector.

[0202] If the current block is a non-square block, the symmetry of the square block cannot be applied. Therefore, not only the intra prediction mode can be applied, but also the symmetry of the block form in a transposed relationship can be applied, so that the two-dimensional block data can be sorted into a one-dimensional input vector. For example, Figure 13 The W×H block with mode 2 and the H×W block with mode M in can be applied to the same transform set and / or transform kernel due to their symmetry with each other. Here, W and H are different positive integers (W≠H).

[0203] For example, for a W×H block with mode 2, a one-dimensional input vector can be configured by sorting the two-dimensional input data according to a predetermined sorting order (e.g., row-first order or column-first order). Additionally, for an H×W block with mode M, a one-dimensional input vector can be configured by sorting the two-dimensional input data according to a sorting order opposite to the sorting order applied to the W×H block. Furthermore, a non-separable main transform can be applied to the one-dimensional input vector of the W×H block with mode 2 and the one-dimensional input vector of the H×W block with mode M according to the same transform set and / or transform kernel. This sorting order is also performed in the inverse transform of the non-separable main transform.

[0204] Figure 14 An implementation of an intra prediction method according to the omnidirectional intra prediction mode is shown.

[0205] At step 1402, the intra prediction mode of the current block is determined.

[0206] According to one embodiment, the intra prediction mode of the current block may be determined from a predetermined intra prediction mode group. In addition, the predetermined intra prediction mode group may include at least one left intra prediction mode, at least one upward intra prediction mode, at least one right intra prediction mode, and at least one downward intra prediction mode.

[0207] According to one embodiment, at least one left-direction intra-frame prediction mode may correspond one-to-one with at least one right-direction intra-frame prediction mode. In addition, at least one upward-direction intra-frame prediction mode may correspond one-to-one with at least one downward-direction intra-frame prediction mode.

[0208] According to one embodiment, bidirectional intra-frame prediction can be applied to the current block. In this case, a first intra-frame prediction mode for bidirectional intra-frame prediction can be selected from at least one left-direction intra-frame prediction mode and at least one upward-direction intra-frame prediction mode in a predetermined intra-frame prediction mode group. Then, a second intra-frame prediction mode having a prediction direction opposite to that of the first intra-frame prediction mode can be determined as the intra-frame prediction mode for bidirectional intra-frame prediction. In other words, two intra-frame prediction modes having opposite prediction directions can be applied to bidirectional intra-frame prediction.

[0209] According to one embodiment, when bidirectional intra prediction is applied, a prediction value of the current sample may be determined based on a weighted sum of a reference sample of the current sample according to a first intra prediction mode and a reference sample of the current sample according to a second intra prediction mode. In this case, a weight applied to the weighted sum of the reference samples may be determined by at least one of a position of the current sample or a prediction direction.

[0210] At step 1404 , a first reference sample of a current block of the reconstructed area is determined.

[0211] In step 1406, second reference samples of the current block of the unreconstructed area are derived. The second reference samples may consist of one or more samples in a right column and one or more samples in a lower row adjacent to the current block.

[0212] According to one embodiment, a second reference sample can be derived based on a neural network model, and the samples of the reconstructed region are input to the neural network model. The input samples of the reconstructed region may include at least one of a left reconstructed sample, an upper reconstructed sample, an upper left reconstructed sample, an upper right reconstructed sample, or a lower left reconstructed sample of the current block. In addition, the input samples of the reconstructed region may be composed of one or more samples in a left column adjacent to the current block and one or more samples in an upper row.

[0213] According to one embodiment, the second reference sample may be derived based on adjacent samples of a matching block searched through template matching based on a current template of the current block.

[0214] According to one embodiment, two or more matching blocks may be searched for by template matching based on the template of the current block. Alternatively, a second reference sample may be determined based on a weighted sum of adjacent samples at two or more corresponding positions of the two or more matching blocks. In this case, the weight used in the weighted sum of the two or more adjacent samples may be determined by the similarity between the reference templates of the two or more matching blocks and the current template.

[0215] According to one embodiment, the second reference samples may be derived based on neighboring samples of a matching block of the reconstructed region indicated by the block vector of the current block.

[0216] At step 1408 , intra prediction is performed according to an intra prediction mode based on at least one of the determined first reference sample or the derived second reference sample.

[0217] The current block may be encoded or decoded according to the prediction method performed in steps 1402 to 1408. In addition, the bitstream generated by the encoder according to the prediction method performed in steps 1402 to 1408 may be stored in a recording medium or transmitted to an external encoder.

[0218] In addition, when Figure 14 When the intra-frame prediction method is applied to image decoding, the image decoding method may further include the step of generating a residual block of the current block by inverse transforming the transform coefficient data of the current block. In this case, in the inverse transform, a first inverse transform method of applying a separable main inverse transform and a secondary inverse transform or a second inverse transform method of applying a non-separable main inverse transform may be selected for the current block.

[0219] According to one embodiment, when the first inverse transform method is applied, for a separable main inverse transform, a multi-transform set can be selected from a plurality of multi-transform sets according to at least one of a transform size or an intra-frame prediction mode, and a transform kernel applied to the separable main inverse transform can be selected from a plurality of transform kernels in the multi-transform set.

[0220] In addition, when the first transformation method is applied, for the secondary inverse transformation, an inseparable transform set can be selected from multiple inseparable transform sets according to the intra-frame prediction mode, and a transform core applied to the secondary inverse transformation can be selected from multiple transform cores of the inseparable transform set.

[0221] According to one embodiment, the plurality of multi-transform sets may include multi-transform sets corresponding to the right direction intra-frame prediction mode and the downward direction intra-frame prediction mode, and the plurality of inseparable transform sets may include inseparable transform sets corresponding to the right direction intra-frame prediction mode and the downward direction intra-frame prediction mode.

[0222] According to one embodiment, when the prediction directions of the first intra-frame prediction mode and the second intra-frame prediction mode are opposite to each other, the multi-transform set and the inseparable transform set selected when the first intra-frame prediction mode is applied and the multi-transform set and the inseparable transform set selected when the second intra-frame prediction mode is applied can be set to the same.

[0223] According to one embodiment, when the second inverse transform method is applied, for the non-separable main inverse transform, an non-separable main transform set can be selected from a plurality of non-separable main transform sets according to at least one of a transform size or an intra-frame prediction mode, and a transform kernel applied to the non-separable main inverse transform can be selected from a plurality of transform kernels of the non-separable main transform set.

[0224] According to one embodiment, the plurality of inseparable main transform sets may include inseparable main transform sets corresponding to a right-direction intra prediction mode and a downward-direction intra prediction mode.

[0225] According to one embodiment, when the prediction directions of the first intra prediction mode and the second intra prediction mode are symmetrical to each other, the inseparable primary transform set selected when the first intra prediction mode is applied to the current block can be set to be the same as the inseparable primary transform set selected when the second intra prediction mode is applied to the current block.

[0226] In addition, when Figure 14 When the intra-frame prediction method is applied to image coding, the image coding method may further include generating transform coefficient data for the current block by transforming the residual block of the current block. In this case, during the transform, a first transform method that applies a separable main transform and a secondary transform, or a second transform method that applies a non-separable main transform, may be selected for the current block.

[0227] According to one embodiment, when the first transform method is applied, for a separable main transform, a multi-transform set can be selected from a plurality of multi-transform sets according to at least one of a transform size or an intra-frame prediction mode, and a transform kernel applied to the separable main transform can be selected from a plurality of transform kernels in the multi-transform set.

[0228] In addition, when the first transform method is applied, for the secondary transform, an inseparable transform set can be selected from multiple inseparable transform sets according to the intra prediction mode, and a transform core applied to the secondary transform can be selected from multiple transform cores of the inseparable transform set.

[0229] According to one embodiment, the plurality of multi-transform sets may include multi-transform sets corresponding to the right direction intra-frame prediction mode and the downward direction intra-frame prediction mode, and the plurality of inseparable transform sets may include inseparable transform sets corresponding to the right direction intra-frame prediction mode and the downward direction intra-frame prediction mode.

[0230] According to one embodiment, when the prediction directions of the first intra-frame prediction mode and the second intra-frame prediction mode are opposite to each other, the multi-transform set and the inseparable transform set selected when the first intra-frame prediction mode is applied and the multi-transform set and the inseparable transform set selected when the second intra-frame prediction mode is applied can be set to the same.

[0231] According to one embodiment, when the second transform method is applied, for the inseparable main transform, an inseparable main transform set can be selected from a plurality of inseparable main transform sets according to at least one of a transform size or an intra prediction mode, and a transform kernel applied to the inseparable main transform can be selected from a plurality of transform kernels in the inseparable main transform set.

[0232] According to one embodiment, the plurality of inseparable main transform sets may include inseparable main transform sets corresponding to a right-direction intra prediction mode and a downward-direction intra prediction mode.

[0233] According to one embodiment, when the prediction directions of the first intra prediction mode and the second intra prediction mode are symmetrical to each other, the inseparable primary transform set selected when the first intra prediction mode is applied to the current block can be set to be the same as the inseparable primary transform set selected when the second intra prediction mode is applied to the current block.

[0234] Figure 15 A content streaming system to which an embodiment according to the present invention is applicable is exemplarily shown.

[0235] like Figure 15 As shown, the content streaming system to which the embodiment of the present invention is applied may mainly include an encoding server, a streaming server, a network server, a media storage device, a user device, and a multimedia input device.

[0236] The encoding server compresses the content received from the multimedia input device such as a smartphone, a camera, a CCTV, etc. into digital data to generate a bitstream and transmits it to the streaming server. As another example, if the multimedia input device such as a smartphone, a camera, a CCTV, etc. directly generates the bitstream, the encoding server can be omitted.

[0237] A bitstream may be generated by the image encoding method and / or the image encoding apparatus to which the embodiments of the present invention are applied, and a streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0238] The streaming server transmits multimedia data to a user device based on a user request via a network server. The network server can also function as an intermediary, notifying the user of any available services. When a user requests a desired service from the network server, the network server transmits the request to the streaming server, which then transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, which can control commands and responses between devices within the content streaming system.

[0239] The streaming server can receive content from a media storage device and / or an encoding server. For example, when receiving content from an encoding server, the content can be received in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a period of time.

[0240] Examples of user devices may include mobile phones, smartphones, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, tablet PCs, tablet PCs, ultrabooks, wearable devices (e.g., smart watches, smart glasses, HMDs), digital televisions, desktop computers, digital signage, etc.

[0241] Each server in the above-described content streaming system may operate as a distributed server, in which case data received from each server may be distributed and processed.

[0242] The above embodiments may be performed in the same or corresponding manner in the encoding device and the decoding device. In addition, an image may be encoded / decoded using at least one or a combination of at least one of the above embodiments.

[0243] The order of applying the above-described embodiments may be different in the encoding device and the decoding device. Alternatively, the order of applying the above-described embodiments may be the same in the encoding device and the decoding device.

[0244] The above-described embodiments may be performed on each of the luminance signal and the chrominance signal. Alternatively, the above-described embodiments may be performed identically for the luminance signal and the chrominance signal.

[0245] In the above embodiments, the method is described based on a flow chart having a series of steps or units, but the present invention is not limited to the order of the steps. On the contrary, some steps can be performed simultaneously with other steps or in a different order. In addition, it should be understood by those skilled in the art that the steps in the flow chart are not mutually exclusive, and other steps can be added to the flow chart or some steps can be deleted from the flow chart without affecting the scope of the present invention.

[0246] The embodiments may be implemented in the form of program instructions that are executable by various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include independent program instructions, data files, data structures, etc., or a combination thereof. The program instructions recorded in the computer-readable recording medium may be specially designed and constructed for the present invention, or may be well known to those skilled in the art in the field of computer software technology.

[0247] The bit stream generated by the encoding method according to the above embodiment can be stored in a non-volatile computer-readable recording medium. In addition, the bit stream stored in the non-volatile computer-readable recording medium can be decoded by the decoding method according to the above embodiment.

[0248] Examples of computer-readable recording media include: magnetic recording media such as hard disks, floppy disks, and magnetic tapes; optical data storage media such as CD-ROMs or DVD-ROMs; magneto-optical media such as floppy disks; and hardware devices such as read-only memory (ROM), random access memory (RAM), flash memory, etc., which are specifically configured to store and execute program instructions. Examples of program instructions include not only machine language codes formatted by a compiler, but also high-level language codes that can be implemented by a computer using an interpreter. A hardware device can be configured to be operated by one or more software modules or vice versa to perform the process according to the present invention.

[0249] Although the present invention has been described in terms of specific items such as detailed elements and limited embodiments and drawings, they are provided only to help a more comprehensive understanding of the present invention, and the present invention is not limited to the above embodiments. It will be understood by those skilled in the art that various modifications and changes can be made based on the above description.

[0250] Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the full scope of the appended claims and their equivalents should fall within the scope and spirit of the present invention.

[0251] Industrial Applicability

[0252] The present invention can be used in an apparatus for encoding / decoding an image and a recording medium for storing a bit stream.

Claims

1. An image decoding method, comprising: determining an intra prediction mode for a current block; determining a first reference sample of a current block of the reconstructed region; deriving a second reference sample of the current block in the unreconstructed area; as well as Intra prediction is performed according to an intra prediction mode based on at least one of the determined first reference sample or the derived second reference sample.

2. The image decoding method according to claim 1, wherein: A second reference sample is derived based on the neural network model, and the sample of the reconstructed region is input into the neural network model.

3. The image decoding method according to claim 2, wherein: The input samples of the reconstructed area include at least one of a left reconstructed sample, an upper reconstructed sample, an upper-left reconstructed sample, an upper-right reconstructed sample, or a lower-left reconstructed sample of the current block.

4. The image decoding method according to claim 2, wherein: The input samples of the reconstructed area consist of one or more samples in a left column adjacent to the current block and one or more samples in an upper row.

5. The image decoding method according to claim 1, wherein: The second reference sample is derived based on adjacent samples of a matching block searched through template matching according to the current template of the current block.

6. The image decoding method according to claim 5, in, Search for two or more matching blocks by template matching based on the template of the current block, wherein the second reference sample is determined based on a weighted sum of adjacent samples at two or more corresponding positions of the two or more matching blocks, and The weights used in the weighted sum of two or more adjacent samples are determined by the similarity between the reference templates and the current template of the two or more matching blocks.

7. The image decoding method according to claim 1, wherein: The second reference samples are derived based on neighboring samples of a matching block of the reconstructed region indicated by the block vector of the current block.

8. The image decoding method according to claim 1, wherein: The second reference samples consist of one or more samples in a right column and one or more samples in a lower row adjacent to the current block.

9. The image decoding method according to claim 1, in, Determining an intra prediction mode for the current block includes determining an intra prediction mode from a predetermined intra prediction mode group, and The predetermined intra-frame prediction mode group includes at least one right-direction intra-frame prediction mode and at least one downward-direction intra-frame prediction mode.

10. The image decoding method according to claim 9, in, The predetermined intra-frame prediction mode group includes at least one left-direction intra-frame prediction mode and at least one upward-direction intra-frame prediction mode, The at least one left-direction intra-frame prediction mode corresponds to the at least one right-direction intra-frame prediction mode in a one-to-one manner, and The at least one upward intra-frame prediction mode corresponds one-to-one to the at least one downward intra-frame prediction mode.

11. The image decoding method according to claim 9, wherein: When determining the intra prediction mode of the current block, When bidirectional intra prediction is applied to the current block, a first intra prediction mode is selected from at least one left-direction intra prediction mode and at least one upward-direction intra prediction mode of a predetermined intra prediction mode group, and The first intra prediction mode and a second intra prediction mode are determined as intra prediction modes of a current block, the second intra prediction mode having a prediction direction opposite to that of the first intra prediction mode.

12. The image decoding method according to claim 11, in, determining a prediction value of the current sample based on a weighted sum of a reference sample of the current sample according to the first intra prediction mode and a reference sample of the current sample according to the second intra prediction mode, and The weight applied to the weighted sum of the reference samples is determined by at least one of the position of the current sample or the prediction direction.

13. The image decoding method according to claim 9, comprising: Generate the residual block of the current block by inverse transforming the transform coefficient data of the current block, In the inverse transformation, the first inverse transformation method or the second inverse transformation method is applied, the first inverse transformation method applies a separable main inverse transformation and a secondary inverse transformation, and the second inverse transformation method applies an inseparable main inverse transformation.

14. The image decoding method according to claim 13, wherein: In the residual block that generates the current block, When the first inverse transform method is applied, For a separable main inverse transform, a multi-transform set is selected from a plurality of multi-transform sets according to at least one of a transform size or an intra prediction mode, and a transform kernel applied to the separable main inverse transform is selected from a plurality of transform kernels in the multi-transform set. For the secondary inverse transform, an inseparable transform set is selected from a plurality of inseparable transform sets according to the intra prediction mode, and a transform kernel applied to the secondary inverse transform is selected from a plurality of transform kernels in the inseparable transform set, and The multiple transform sets include multiple transform sets corresponding to a right intra prediction mode and a downward intra prediction mode, and the multiple inseparable transform sets include inseparable transform sets corresponding to a right intra prediction mode and a downward intra prediction mode.

15. The image decoding method according to claim 14, wherein: When prediction directions of the first intra prediction mode and the second intra prediction mode are opposite to each other, the multiple transform set and the inseparable transform set selected when the first intra prediction mode is applied to the current block are the same as the multiple transform set and the inseparable transform set selected when the second intra prediction mode is applied to the current block.

16. The image decoding method according to claim 13, wherein: In the residual block that generates the current block, When the second inverse transform method is applied, for the non-separable main inverse transform, an non-separable main transform set is selected from a plurality of non-separable main transform sets according to at least one of a transform size or an intra prediction mode, and a transform kernel applied to the non-separable main inverse transform is selected from a plurality of transform kernels in the non-separable main transform set, and The plurality of inseparable main transform sets include inseparable main transform sets corresponding to a right-direction intra prediction mode and a downward-direction intra prediction mode.

17. The image decoding method according to claim 16, wherein: When prediction directions of the first intra prediction mode and the second intra prediction mode are symmetrical to each other, the inseparable primary transform set selected when the first intra prediction mode is applied to the current block is the same as the inseparable primary transform set selected when the second intra prediction mode is applied to the current block.

18. An image encoding method, comprising: determining an intra prediction mode for a current block; determining a first reference sample of a current block of the reconstructed region; deriving a second reference sample of the current block in the unreconstructed area; as well as Intra prediction is performed according to an intra prediction mode based on at least one of the determined first reference sample or the derived second reference sample.

19. A computer-readable recording medium for storing a bit stream generated by an image encoding method, the image encoding method comprising: determining an intra prediction mode for a current block; determining a first reference sample of a current block of the reconstructed region; deriving a second reference sample of the current block in the unreconstructed area; as well as Intra prediction is performed according to an intra prediction mode based on at least one of the determined first reference sample or the derived second reference sample.

20. A method for transmitting a bit stream generated by an image encoding method, the method comprising: Encoding the image based on an image coding method; as well as Transmits a bitstream consisting of an encoded image, The image encoding method includes: determining an intra prediction mode for a current block; determining a first reference sample of a current block of the reconstructed region; deriving a second reference sample of the current block of the unreconstructed area; and Intra prediction is performed according to an intra prediction mode based on at least one of the determined first reference sample or the derived second reference sample.

Images