Image encoding / decoding method and apparatus, and recording medium having stored bit stream

Abstract

The invention provides an image encoding / decoding method, an image encoding / decoding device, a recording medium having a stored bit stream, and a transmission method. The image decoding method comprises the steps of: determining an intra prediction mode of a current block; deriving a reference sample of the unreconstructed region; and performing intra prediction according to an intra prediction mode based on the derived reference samples in the unreconstructed region and / or the reference samples in the reconstructed region, in which the reference samples of the unreconstructed region are in a region located at least one of a lower left side, a lower side, a lower right side, a right side, or an upper right side of the current block.

Description

Technical Field

[0001] The present invention relates to an image encoding / decoding method and apparatus and a recording medium for storing a bit stream. More specifically, the present invention relates to an image encoding / decoding method and apparatus based on reference sample derivation for intra-frame prediction, an intra-frame prediction method using the image encoding / decoding method and apparatus, and a recording medium for storing a bit stream. Background Art

[0002] Recently, in various application fields, the demand for high-resolution, high-quality images such as ultra-high definition (UHD) images is increasing. As image data becomes higher in resolution and quality, the amount of data increases relatively compared to existing image data. Therefore, when image data is transmitted using a medium such as existing wired and wireless broadband lines or stored using existing storage media, transmission and storage costs increase. In order to solve these problems that arise as image data becomes higher in resolution and quality, efficient image encoding / decoding technology for images with higher resolution and quality is required.

[0003] In the existing intra prediction method, when the neighboring reference samples of the current block cannot be used, the reference samples generated by a simple filling method are used for intra prediction. In this way, when the reference samples are derived by using a simple filling method, there is a problem of low prediction accuracy. Summary of the invention

[0004] Technical issues

[0005] An object of the present invention is to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0006] Another object of the present invention is to provide a recording medium for storing a bit stream generated by the image decoding method or apparatus according to the present invention.

[0007] Another object of the present invention is to provide an improved reference sample derivation for solving the problems of the reference sample derivation method and the intra prediction method using the same.

[0008] Technical Solution

[0009] The image decoding method according to the embodiment of the present invention may include: determining an intra prediction mode of a current block; deriving a reference sample of an unreconstructed area; and performing intra prediction according to the intra prediction mode based on at least one of the derived reference sample of the unreconstructed area or the reference sample of the reconstructed area. The reference sample of the unreconstructed area may be a reference sample of an area located at least one of the lower left side, the lower side, the lower right side, the right side, or the upper right side of the current block.

[0010] In the image decoding method, reference samples of the unreconstructed area may be derived based on neighboring samples of a matching block by performing a template matching search.

[0011] In the image decoding method, reference samples of the unreconstructed area may be derived based on neighboring samples of a matching block in a current picture indicated by motion information of the current block.

[0012] In the image decoding method, reference samples of the unreconstructed area may be derived based on a neural network model having reference samples of the reconstructed area as input.

[0013] In the image decoding method, the intra prediction mode may be determined as any one of a planar mode, a vertical planar mode, a horizontal planar mode, a DC mode, or an angular mode.

[0014] In the image decoding method, when the intra-frame prediction mode is determined to be a planar mode, intra-frame prediction can be performed based on reference samples of a reconstructed area corresponding to the upper side of a current pixel, reference samples of a reconstructed area corresponding to the left side, reference samples of an unreconstructed area corresponding to the lower side, and reference samples of an unreconstructed area corresponding to the right side.

[0015] In the image decoding method, when the intra prediction mode is determined to be the vertical plane mode, intra prediction may be performed based on reference samples of a reconstructed area corresponding to an upper side of a current pixel and reference samples of an unreconstructed area corresponding to a lower side.

[0016] In the image decoding method, when the intra prediction mode is determined to be the horizontal plane mode, intra prediction may be performed based on reference samples of a reconstructed area corresponding to a left side of a current pixel and reference samples of an unreconstructed area corresponding to a right side.

[0017] In the image decoding method, when the intra-frame prediction mode is determined to be the DC mode, intra-frame prediction can be performed based on reference samples of a reconstructed area located on the upper side of the current block, reference samples of a reconstructed area located on the left side, reference samples of an unreconstructed area located on the lower side, and reference samples of an unreconstructed area located on the right side.

[0018] In the image decoding method, determining the intra prediction mode may include parsing a plane flag, and determining the intra prediction mode by parsing any one of plane mode information and intra prediction mode information according to the plane flag.

[0019] The image encoding method according to the embodiment of the present invention may include: determining an intra prediction mode of a current block; deriving a reference sample of an unreconstructed area; and performing intra prediction according to the intra prediction mode based on at least one of the derived reference sample of the unreconstructed area or the reference sample of the reconstructed area. The reference sample of the unreconstructed area may be a reference sample of an area located at least one of the lower left side, the lower side, the lower right side, the right side, or the upper right side of the current block.

[0020] A non-volatile computer-readable recording medium according to an embodiment of the present invention may store a bitstream generated by an image encoding method, the image encoding method comprising: determining an intra prediction mode of a current block; deriving a reference sample of an unreconstructed area; and performing intra prediction according to the intra prediction mode based on at least one of the derived reference sample of the unreconstructed area or the reference sample of the reconstructed area. The reference sample of the unreconstructed area may be a reference sample of an area located at least one of the lower left side, the lower side, the lower right side, the right side, or the upper right side of the current block.

[0021] The transmission method according to the embodiment of the present invention can transmit a bit stream generated by an image encoding method, the image encoding method comprising: determining an intra prediction mode of a current block; deriving a reference sample of an unreconstructed area; and performing intra prediction according to the intra prediction mode based on at least one of the derived reference sample of the unreconstructed area or the reference sample of the reconstructed area. The reference sample of the unreconstructed area can be a reference sample of an area located at least one of the lower left side, lower side, lower right side, right side, or upper right side of the current block.

[0022] The features briefly summarized above with respect to the present invention are merely exemplary aspects of the following detailed description of the invention and do not limit the scope of the invention.

[0023] Beneficial Effects

[0024] According to the present invention, an image encoding / decoding method and apparatus with improved encoding / decoding efficiency can be provided.

[0025] In addition, according to the present invention, an improved reference sample derivation and an intra prediction method using the reference sample derivation can be provided.

[0026] In addition, according to the present invention, intra-frame prediction accuracy can be improved.

[0027] Those skilled in the art will understand that the effects that can be achieved by the present invention are not limited to the contents that have been specifically described above, and other advantages of the present invention will be more clearly understood from the detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a block diagram showing the configuration of an encoding device according to an embodiment of the present invention.

[0029] Figure 2 is a block diagram showing the configuration of a decoding device according to an embodiment of the present invention.

[0030] Figure 3 FIG. 1 is a schematic diagram schematically showing a video coding system to which the present invention can be applied.

[0031] Figure 4 is a schematic diagram for explaining a padding-based reference sample derivation method according to an embodiment of the present invention.

[0032] Figure 5 is a diagram for explaining reference samples for intra prediction according to an embodiment of the present invention.

[0033] Figures 6 to 8 is a schematic diagram for explaining a reference sample derivation method based on a neural network according to an embodiment of the present invention.

[0034] Figures 9 to 11 is a schematic diagram for explaining a reference sample derivation method based on template matching according to an embodiment of the present invention.

[0035] Figure 12 to Figure 14 is a schematic diagram for explaining a reference sample derivation method based on motion information according to an embodiment of the present invention.

[0036] Fig.15 is a schematic diagram for explaining an embodiment of a plurality of reference sample lines according to an embodiment of the present invention.

[0037] Fig.16 is a schematic diagram for explaining a plane mode according to an embodiment of the present invention.

[0038] Fig.17 is a schematic diagram for explaining a horizontal plane mode according to an embodiment of the present invention.

[0039] Fig.18 is a schematic diagram for explaining a vertical plane mode according to an embodiment of the present invention.

[0040] Fig.19 is a schematic diagram for explaining a DC mode according to an embodiment of the present invention.

[0041] Fig. 20 and Fig.21 is a schematic diagram for explaining an angle pattern according to an embodiment of the present invention.

[0042] Fig. 22 is a flowchart illustrating an intra prediction mode parsing method according to an embodiment of the present invention.

[0043] Fig.23 is a flowchart illustrating an image decoding method according to an embodiment of the present invention.

[0044] Fig.24 A content streaming system to which an embodiment according to the present invention is applicable is exemplarily shown. DETAILED DESCRIPTION

[0045] The present invention may have various modifications and embodiments, and specific embodiments are shown in the drawings and described in detail in the detailed specification. However, this is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents or alternatives contained in the spirit and technical scope of the present invention. The same figure marks in the drawings indicate the same or similar functions in various aspects. For a clearer description, the shapes and sizes of the elements in the drawings can be provided by way of example. The detailed description of the exemplary embodiments described below refers to the drawings, which show specific embodiments by way of example. These embodiments are described in full detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different from each other, but not necessarily mutually exclusive. For example, without departing from the spirit and scope of the present invention with reference to an embodiment, the specific shapes, structures and features described herein can be implemented in other embodiments. It should also be understood that without departing from the spirit and scope of the embodiment, the position or arrangement of each component in each disclosed embodiment can be changed. Accordingly, the detailed description set forth below is not intended to be restrictive, and the scope of the exemplary embodiments is limited only by the full range of equivalent forms assigned by the attached claims and these claims (if appropriately described).

[0046] In the present invention, the terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are only used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The terms and / or include a combination of multiple related descriptive items or any item in multiple related descriptive items.

[0047] The components shown in the embodiments of the present invention are depicted independently to indicate different characteristic functions, and do not represent that each component is formed as a separate hardware or software configuration unit. That is, for ease of explanation, each component is listed and included as a separate component, and at least two of the components can be combined to form a single component, or one component can be divided into multiple components to perform functions, as long as it does not depart from the essence of the present invention, the embodiment in which the components are integrated and the embodiment in which each component is divided are also included in the scope of the present invention.

[0048] The terms used in the present invention are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In addition, some components of the present invention are not necessary components for performing necessary functions in the present invention, and can be optional components that are only used to improve performance. The present invention can be realized by only including necessary components for realizing the gist of the present invention and not including components that are only used to improve performance, and the structure that only includes necessary components and does not include optional components that are only used to improve performance is also included in the scope of the present invention.

[0049] In an embodiment, the term "at least one" may refer to one of a number greater than or equal to 1, such as 1, 2, 3, and 4. In an embodiment, the term "plurality" may refer to one of a number greater than or equal to 2, such as 2, 3, and 4.

[0050] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. When describing the embodiments of the present specification, if it is determined that the detailed description of the related known configuration or function will obscure the subject of the present specification, the detailed description will be omitted, the same figure numerals will be used for the same components in the drawings, and the repeated description of the same components will be omitted.

[0051] Description of terms

[0052] Hereinafter, "image" may refer to a picture constituting a video, and may also refer to the video itself. For example, "encoding and / or decoding of an image" may refer to "encoding and / or decoding of a video", and may also refer to "encoding and / or decoding of one of the images constituting the video".

[0053] Hereinafter, "moving image" and "video" may be used with the same meaning and may be used interchangeably. In addition, the target image may be an encoding target image as a target of encoding and / or a decoding target image as a target of decoding. In addition, the target image may be an input image input to an encoding device, and may be an input image input to a decoding device. Here, the target image may have the same meaning as the current image.

[0054] Hereinafter, an encoder and an image encoding device may be used with the same meaning and may be used interchangeably.

[0055] Hereinafter, a decoder and an image decoding device may be used with the same meaning and may be used interchangeably.

[0056] Hereinafter, “image”, “picture”, “frame” and “screen” may be used with the same meaning and may be used interchangeably.

[0057] Hereinafter, a "target block" may be an encoding target block as a target of encoding and / or a decoding target block as a target of decoding. In addition, a target block may be a current block as a target of current encoding and / or decoding. For example, "target block" and "current block" may be used with the same meaning and may be used interchangeably.

[0058] In the following, "block" and "unit" may be used with the same meaning and may be used interchangeably. In addition, a "unit" may be represented as including a luma component block and a chroma component block corresponding thereto, so as to distinguish it from a block. For example, a coding tree unit (CTU) may consist of a luma component (Y) coding tree block (CTB) and two chroma component (Cb, Cr) coding tree blocks associated therewith.

[0059] Hereinafter, "sample", "picture element", and "pixel" may be used with the same meaning and may be used interchangeably. Herein, a sample may represent a basic unit constituting a block.

[0060] Hereinafter, “inter-frame” and “inter-screen” may be used with the same meaning and may be used interchangeably.

[0061] Hereinafter, “intra-frame” and “intra-screen” may be used with the same meaning and may be used interchangeably.

[0062] Figure 1 is a block diagram showing the configuration of an encoding device according to an embodiment of the present invention.

[0063] The encoding device 100 may be an encoder, a video encoding device, or an image encoding device. A video may include one or more images. The encoding device 100 may sequentially encode one or more images.

[0064] refer to Figure 1 The encoding device 100 may include: an image partitioning unit 110, an intra-frame prediction unit 120, a motion prediction unit 121, a motion compensation unit 122, a switch 115, a subtractor 113, a transform unit 130, a quantization unit 140, an entropy encoding unit 150, an inverse quantization unit 160, an inverse transform unit 170, an adder 117, a filtering unit 180 and a reference picture buffer 190.

[0065] In addition, the encoding device 100 may generate a bitstream including information encoded by encoding the input image, and output the generated bitstream. The generated bitstream may be stored in a computer-readable recording medium, or may be streamed through a wired / wireless transmission medium.

[0066] The image partitioning unit 110 can partition the input image into various forms to improve the efficiency of video encoding / decoding. That is, the input video consists of multiple pictures, and for compression efficiency, parallel processing, etc., a picture can be hierarchically partitioned and processed. For example, a picture can be partitioned into one or more tiles or slices, and then partitioned into multiple codec tree units (CTUs) again. Alternatively, a picture can first be partitioned into multiple sub-pictures defined as rectangular slice groups, and each sub-picture can be partitioned into tiles / slices. Here, sub-pictures can be used to support the function of partially independently encoding / decoding and transmitting pictures. Since multiple sub-pictures can be reconstructed separately, this has the advantage of easy editing in applications where multi-channel inputs are configured as one picture. In addition, tiles can be divided horizontally to generate bricks. Here, bricks can be used as basic units for parallel processing within a picture. In addition, a CTU can be recursively partitioned into a quad tree (quad tree, QT), and the terminal node of the partition can be defined as a coding unit (Coding Unit, CU). The CU can be partitioned into a PU (Prediction Unit) as a prediction unit and a TU (Transform Unit) as a transform unit to perform prediction and partitioning. On the other hand, the CU can be used as a prediction unit and / or a transform unit itself. Here, for flexible partitioning, each CTU can be recursively partitioned into multi-type trees (MTT) and quadtrees (QT). Partitioning the CTU into multi-type trees can start from the terminal node of the QT, and the MTT can be composed of a binary tree (BT) and a triple tree (TT). For example, the MTT structure can be classified into a vertical binary split mode (SPLIT_BT_VER), a horizontal binary split mode (SPLIT_BT_HOR), a vertical ternary split mode (SPLIT_TT_VER), and a horizontal ternary split mode (SPLIT_TT_HOR). In addition, during partitioning, the minimum block size (MinQTSize) of the quadtree of the luma block can be set to 16×16, the maximum block size (MaxBtSize) of the binary tree can be set to 128×128, and the maximum block size (MaxTtSize) of the ternary tree can be set to 64×64. In addition, the minimum block size (MinBtSize) of the binary tree and the minimum block size (MinTtSize) of the ternary tree can be specified as 4×4, and the maximum depth (MaxMttDepth) of the multi-type tree can be specified as 4. In addition, in order to improve the coding efficiency of the I slice, a dual tree that uses the CTU partition structure of the luma component and the chroma component differently can be applied.On the other hand, in P slices and B slices, the luma and chroma coding tree blocks (CTBs) within a CTU can be partitioned into a single tree that shares the coding tree structure.

[0067] The encoding device 100 may perform encoding on the input image in intra mode and / or inter mode. Alternatively, the encoding device 100 may perform encoding on the input image in a third mode other than intra mode and inter mode (e.g., IBC mode, palette mode, etc.). However, if the third mode has functional characteristics similar to those of the intra mode or inter mode, it may be classified as the intra mode or inter mode for ease of explanation. In the present invention, the third mode is classified and described separately only when a specific description of the third mode is required.

[0068] When the intra mode is used as the prediction mode, the switch 115 can switch to the intra mode, and when the inter mode is used as the prediction mode, the switch 115 can switch to the inter mode. Here, the intra mode can represent the intra prediction mode, and the inter mode can represent the inter prediction mode. The encoding device 100 can generate a prediction block for an input block of an input image. In addition, the encoding device 100 can encode a residual block using the residual of the input block and the prediction block after generating the prediction block. The input image can be referred to as the current image as the current encoding target. The input block can be referred to as the current block or the encoding target block as the current encoding target.

[0069] When the prediction mode is intra mode, the intra prediction unit 120 may use samples of the already encoded / decoded blocks around the current block as reference samples. The intra prediction unit 120 may perform spatial prediction of the current block by utilizing the reference samples, or generate prediction samples of the input block by spatial prediction. In this article, intra prediction may refer to intra-picture prediction.

[0070] As the intra prediction method, non-directional prediction modes such as DC mode and planar mode and directional prediction modes (for example, 65 directions) may be applied. Here, the intra prediction method may be expressed as an intra prediction mode or an intra-screen prediction mode.

[0071] When the prediction mode is the inter mode, the motion prediction unit 121 can retrieve the area that best matches the input block from the reference image in the motion prediction process, and derive the motion vector by using the retrieved area. In this case, the search area can be used as the area. The reference image can be stored in the reference picture buffer 190. Here, when encoding / decoding of the reference image is performed, it can be stored in the reference picture buffer 190.

[0072] The motion compensation unit 122 may generate a prediction block of the current block by performing motion compensation using a motion vector. Herein, inter prediction may mean inter-picture prediction or motion compensation.

[0073] When the value of the motion vector is not an integer, the motion prediction unit 121 and the motion compensation unit 122 may generate a prediction block by applying an interpolation filter to a partial area of ​​the reference picture. In order to perform inter-frame prediction or motion compensation, it may be determined based on the codec unit whether the motion prediction and motion compensation mode of the prediction unit included in the codec unit is one of the skip mode, merge mode, advanced motion vector prediction (AMVP) mode, and intra block copy (IBC) mode, and inter-frame prediction or motion compensation may be performed according to each mode.

[0074] In addition, based on the above-mentioned inter-frame prediction method, the affine (AFFINE) mode based on sub-PU prediction, the sub-block-based temporal motion vector prediction (SbTMVP) mode, the PU-based prediction and MVD merge (Merge with MVD, MMVD) mode and the geometric partitioning mode (Geometric Partitioning Mode, GPM) mode can be applied. In addition, in order to improve the performance of each mode, history based MVP (HMVP), pairwise average MVP (PAMVP), combined intra / inter prediction (CIIP), adaptive motion vector resolution (AMVR), bi-directional optical flow (BDOF), bi-predictive with CU Weights (BCW), local illumination compensation (LIC), template matching (TM), overlapped block motion compensation (OBMC), etc. can be applied.

[0075] Among them, the affine mode is a technology used in both AMVP and MERGE modes, and also has high coding efficiency. In existing video codec standards, since motion compensation (MC) is performed by considering only the parallel movement of blocks, it has the disadvantage of not being able to properly compensate for the motion that occurs in reality (e.g., zooming in / out and rotation). To supplement this, a four-parameter affine motion model using two control point motion vectors (CPMV) and a six-parameter affine motion model using three control point motion vectors can be used and applied to inter-frame prediction. Here, CPMV is a vector representing an affine motion model of one of the upper left, upper right, and lower left sides of the current block.

[0076] The subtractor 113 may generate a residual block by utilizing the difference between the input block and the prediction block. The residual block may be referred to as a residual signal. The residual signal may represent the difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming or quantizing, or transforming and quantizing the difference between the original signal and the prediction signal. The residual block may be a residual signal of a block unit.

[0077] The transform unit 130 may generate a transform coefficient by performing a transform on the residual block, and output the generated transform coefficient. Herein, the transform coefficient may be a coefficient value generated by performing a transform on the residual block. When the transform skip mode is applied, the transform unit 130 may skip the transform of the residual block.

[0078] The quantization level may be generated by applying quantization to a transform coefficient or a residual signal. Hereinafter, the quantization level may also be referred to as a transform coefficient in an embodiment.

[0079] For example, a 4×4 luminance residual block generated by intra prediction is transformed using a base vector based on a discrete sine transform (DST), and the remaining residual block can be transformed using a base vector based on a discrete cosine transform (DCT). In addition, the transform block is partitioned into a quadtree shape of a block using a residual quad tree (RQT) technique, and after transforming and quantizing each transform block partitioned by the RQT, a coded block flag (cbf) can be transmitted when all coefficients become 0 to improve coding efficiency.

[0080] As another alternative, a Multiple Transform Selection (MTS) technique that selectively uses multiple transform bases to perform transforms can be applied. That is, instead of partitioning the CU into TUs through RQT, a function similar to TU partitioning can be performed through a sub-block transform (SBT) technique. Specifically, SBT is only applied to inter-prediction blocks, and unlike RQT, the current block can be partitioned into a size of 1 / 2 or 1 / 4 in the vertical or horizontal direction, and then the transform can be performed only on one of the blocks. For example, if the current block is partitioned vertically, the transform can be performed on the leftmost or rightmost block, and if the current block is partitioned horizontally, the transform can be performed on the topmost or bottommost block.

[0081] In addition, a low frequency non-separable transform (LFNST) can be applied, which is a secondary transform technique that additionally transforms the transformed residual signal into the frequency domain by DCT or DST. LFNST additionally performs a transform on a 4×4 or 8×8 low frequency region on the upper left side so that the residual coefficients can be concentrated on the upper left side.

[0082] The quantization unit 140 may generate a quantization level by quantizing the transform coefficient or the residual signal according to a quantization parameter (QP), and output the generated quantization level. Herein, the quantization unit 140 may quantize the transform coefficient by using a quantization matrix.

[0083] For example, a quantizer using a QP value of 0 to 51 may be used. Alternatively, if the image size is large and high coding efficiency is required, a QP of 0 to 63 may be used. In addition, a dependent quantization (DQ) method using two quantizers instead of one quantizer may be applied. DQ performs quantization using two quantizers (e.g., Q0 and Q1), but even if there is no signaling information about the use of a specific quantizer, the quantizer used for the next transform coefficient may be selected based on the current state through a state transition model.

[0084] The entropy encoding unit 150 may generate a bit stream by performing entropy encoding on the value calculated by the quantization unit 140 or the encoding parameter value calculated when encoding is performed according to the probability distribution, and output the bit stream. The entropy encoding unit 150 may perform entropy encoding on information about samples of an image and information for decoding an image. For example, the information for decoding an image may include a syntax element.

[0085] When entropy coding is applied, the symbol is represented so that a smaller number of bits are assigned to symbols with a high probability of occurrence and a larger number of bits are assigned to symbols with a low probability of occurrence, so the size of the bit stream for the symbol to be encoded can be reduced. The entropy coding unit 150 can use coding methods such as exponential Golomb, context-adaptive variable length coding (context-adaptive variable length coding, CAVLC), context-adaptive binary arithmetic coding (context-adaptive binary arithmetic coding, CABAC) for entropy coding. For example, the entropy coding unit 150 can perform entropy coding by using a variable length coding / code (variable length coding / code, VLC) table. In addition, the entropy coding unit 150 can derive a binarization method of the target symbol and a probability model of the target symbol / binary, and perform arithmetic coding and decoding by using the derived binarization method and context model.

[0086] In this regard, when CABAC is applied, in order to reduce the size of the probability table stored in the decoding device, the table probability update method can be changed to a table update method using a simple equation and applied. In addition, two different probability models can be used to obtain more accurate symbol probability values.

[0087] In order to encode the transformation coefficient level (quantization level), the entropy encoding unit 150 may change the coefficients in a two-dimensional block form into a one-dimensional vector form through a transformation coefficient scanning method.

[0088] The coding and decoding parameters may include information (flags, indexes, etc.) encoded in the encoding device 100 and signaled to the decoding device 200, such as syntax elements, and information derived during the encoding or decoding process, and may represent information required when encoding or decoding an image.

[0089] Herein, signaling a flag or an index may mean that in an encoder, the corresponding flag or index is entropy encoded and included in a bitstream, and may mean that in a decoder, the corresponding flag or index is entropy decoded from the bitstream.

[0090] The encoded current image can be used as a reference image for another image to be processed later. Therefore, the encoding device 100 can reconstruct or decode the encoded current image again and store the reconstructed or decoded image in the reference picture buffer 190 as a reference image.

[0091] The quantization level may be dequantized in the dequantization unit 160 or may be inversely transformed in the inverse transform unit 170. The dequantized and / or inversely transformed coefficients may be added to the prediction block by the adder 117. Herein, the dequantized and / or inversely transformed coefficients may represent coefficients on which at least one of dequantization and inverse transformation is performed, and may represent a reconstructed residual block. The dequantization unit 160 and the inverse transform unit 170 may be performed as inverse processes of the quantization unit 140 and the transform unit 130.

[0092] The reconstructed block may pass through the filtering unit 180. The filtering unit 180 may apply a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter (BIF), a luma mapping with chroma scaling (LMCS) filter, etc. to the reconstructed sample, the reconstructed block, or the reconstructed image using all or some filtering techniques. The filtering unit 180 may be referred to as an in-loop filter. In this case, the in-loop filter may also be used as a name that does not include LMCS.

[0093] The deblocking filter can remove block distortion generated in the boundary between blocks. In order to determine whether to apply the deblocking filter, it can be determined whether to apply the deblocking filter to the current block based on samples included in several rows or columns included in the block. When the deblocking filter is applied to the block, different filters can be applied according to the required deblocking filter strength.

[0094] In order to compensate for the coding error using sample adaptive offset, an appropriate offset value may be added to the sample value. Sample adaptive offset can correct the offset from the original image in sample units for the deblocked image. A method of partitioning the samples included in the image into a predetermined number of regions, determining the region to which the offset is applied, and applying the offset to the determined region, or a method of applying the offset in consideration of edge information about each sample may be used.

[0095] The bilateral filter (BIF) can also correct the deviation from the original image on a sample-by-sample basis for the image on which deblocking has been performed.

[0096] The adaptive loop filter may perform filtering based on the comparison result of the reconstructed image with the original image. The samples included in the image may be partitioned into predetermined groups, the filter to be applied to each group may be determined, and differential filtering may be performed on each group. Information on whether the ALF is applied may be signaled by a codec unit (CU), and the form and coefficients of the adaptive loop filter to be applied to each block may vary.

[0097] In luma mapping with chroma scaling (LMCS), luma mapping (LM) means remapping luma values ​​by a piecewise linear model, and chroma scaling (CS) means a technique for scaling the residual values ​​of chroma components according to the average luma value of a prediction signal. In particular, LMCS can be used as an HDR correction technique that reflects the characteristics of a High Dynamic Range (HDR) image.

[0098] The reconstructed block or the reconstructed image that has passed through the filtering unit 180 may be stored in the reference picture buffer 190. The reconstructed block that has passed through the filtering unit 180 may be part of the reference image. That is, the reference image is a reconstructed image composed of the reconstructed block that has passed through the filtering unit 180. The stored reference image may be used for inter-frame prediction or motion compensation later.

[0099] Figure 2 is a block diagram showing the configuration of a decoding device according to an embodiment of the present invention.

[0100] The decoding device 200 may be a decoder, a video decoding device, or an image decoding device.

[0101] refer to Figure 2 The decoding device 200 may include: an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra-frame prediction unit 240, a motion compensation unit 250, an adder 201, a switch 203, a filtering unit 260 and a reference picture buffer 270.

[0102] The decoding device 200 may receive a bit stream output from the encoding device 100. The decoding device 200 may receive a bit stream stored in a computer-readable recording medium, or may receive a bit stream streamed through a wired / wireless transmission medium. The decoding device 200 may decode the bit stream in an intra-frame mode or an inter-frame mode. In addition, the decoding device 200 may generate a reconstructed image or a decoded image generated by decoding, and output the reconstructed image or the decoded image.

[0103] When the prediction mode for decoding is the intra mode, the switch 203 may switch to the intra mode. Alternatively, when the prediction mode for decoding is the inter mode, the switch 203 may switch to the inter mode.

[0104] The decoding device 200 can obtain a reconstructed residual block by decoding the input bit stream and generate a prediction block. When the reconstructed residual block and the prediction block are obtained, the decoding device 200 can generate a reconstructed block that becomes a decoding target by adding the reconstructed residual block and the prediction block. The decoding target block can be called a current block.

[0105] The entropy decoding unit 210 may generate symbols by entropy decoding the bit stream according to the probability distribution. The generated symbols may include symbols in the form of quantization levels. In this article, the entropy decoding method may be an inverse process of the above entropy encoding method.

[0106] The entropy decoding unit 210 may change the coefficients of the one-dimensional vector shape into coefficients of the two-dimensional block shape through a transform coefficient scanning method to decode the transform coefficient level (quantization level).

[0107] The quantization level may be inversely quantized in the inverse quantization unit 220 or inversely transformed in the inverse transform unit 230. The quantization level may be a result of inverse quantization and / or inverse transformation and may be generated as a reconstructed residual block. Herein, the inverse quantization unit 220 may apply a quantization matrix to the quantization level. The inverse quantization unit 220 and the inverse transform unit 230 applied to the decoding device may apply the same technology as the inverse quantization unit 160 and the inverse transform unit 170 applied to the aforementioned encoding device.

[0108] When the intra mode is used, the intra prediction unit 240 can generate a prediction block by performing spatial prediction on the current block, which uses sample values ​​of blocks that have been decoded around the decoding target block. The intra prediction unit 240 applied to the decoding device can apply the same technology as the intra prediction unit 120 applied to the aforementioned encoding device.

[0109] When the inter-frame mode is used, the motion compensation unit 250 can generate a prediction block by performing motion compensation using a motion vector and a reference image stored in the reference picture buffer 270 on the current block. When the value of the motion vector is not an integer value, the motion compensation unit 250 can generate a prediction block by applying an interpolation filter to a partial area within the reference image. In order to perform motion compensation, it can be determined based on the codec unit whether the motion compensation mode of the prediction unit included in the corresponding codec unit is a skip mode, a merge mode, an AMVP mode, or a current picture reference mode, and motion compensation can be performed according to each mode. The motion compensation unit 250 applied to the decoding device can apply the same technology as the motion compensation unit 122 applied to the above-mentioned encoding device.

[0110] The adder 201 may generate a reconstructed block by adding the reconstructed residual block and the prediction block. The filtering unit 260 may apply at least one of an inverse LMCS, a deblocking filter, a sample adaptive offset, and an adaptive loop filter to the reconstructed block or the reconstructed image. The filtering unit 260 applied to the decoding device may apply the same filtering technique as the filtering unit 180 applied to the aforementioned encoding device.

[0111] The filtering unit 260 may output a reconstructed image. The reconstructed block or the reconstructed image may be stored in the reference picture buffer 270 and used for inter-frame prediction. The reconstructed block that has passed through the filtering unit 260 may be part of the reference image. That is, the reference image may be a reconstructed image composed of the reconstructed blocks that have passed through the filtering unit 260. The stored reference image may be used for inter-frame prediction or motion compensation later.

[0112] Figure 3 FIG. 1 is a schematic diagram schematically showing a video coding system to which the present invention can be applied.

[0113] The video coding system according to the embodiment may include an encoding device 10 and a decoding device 20. The encoding device 10 may transmit the encoded video and / or image information or data to the decoding device 20 in the form of a file or streaming through a digital storage medium or a network.

[0114] The encoding device 10 according to the embodiment may include a video source generating unit 11, an encoding unit 12, and a transmitting unit 13. The decoding device 20 according to the embodiment may include a receiving unit 21, a decoding unit 22, and a rendering unit 23. The encoding unit 12 may be referred to as a video / image encoding unit, and the decoding unit 22 may be referred to as a video / image decoding unit. The transmitting unit 13 may be included in the encoding unit 12. The receiving unit 21 may be included in the decoding unit 22. The rendering unit 23 may include a display unit, and the display unit may be configured as a separate device or an external component.

[0115] The video source generation unit 11 can obtain the video / image by capturing, synthesizing or generating the video / image process. The video source generation unit 11 may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive including previously captured videos / images, etc. The video / image generation device may include, for example, a computer, a tablet computer, a smart phone, etc., and may (electronically) generate the video / image. For example, a virtual video / image may be generated by a computer, etc., in which case the video / image capture process may be replaced by a process for generating relevant data.

[0116] The encoding unit 12 may encode the input video / image. For compression and encoding efficiency, the encoding unit 12 may perform a series of processes such as prediction, transformation, and quantization. The encoding unit 12 may output the encoded data (encoded video / image information) in the form of a bit stream. The detailed configuration of the encoding unit 12 may also be the same as described above. Figure 1 The encoding device 100 is configured in the same manner.

[0117] The sending unit 13 may send the encoded video / image information or data output in the form of a bit stream to the receiving unit 21 of the decoding device 20 through a digital storage medium or a network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The sending unit 13 may include an element for generating a media file in a predetermined file format, and may include an element for transmission through a broadcast / communication network. The receiving unit 21 may extract / receive a bit stream from a storage medium or a network and send it to the decoding unit 22.

[0118] The decoding unit 22 can decode the video / image by performing a series of processes such as inverse quantization, inverse transformation, and prediction corresponding to the operations of the encoding unit 12. The detailed configuration of the decoding unit 22 can also be similar to Figure 2 The above-mentioned decoding device 200 is configured in the same manner.

[0119] The rendering unit 23 may render the decoded video / image. The rendered video / image may be displayed by the display unit.

[0120] In the following, reference Figures 4 to 23 , a reference sample derivation method according to an embodiment of the present invention and an intra prediction method using the reference sample derivation method will be described.

[0121] In the present invention, the reference sample may mean a reference pixel. In addition, when the sample of the reconstructed area is used as the reference sample of the intra prediction, this may be referred to as a reconstructed reference sample, and when the sample of the unreconstructed area is derived and used as the reference sample of the intra prediction, this may be referred to as a derived reference sample.

[0122] Figure 4 is a schematic diagram for explaining a padding-based reference sample derivation method according to an embodiment of the present invention.

[0123] exist Figure 4, C1, C2, C3, and C4 are sub-blocks of the current block 410. Since encoding / decoding is performed in a raster scan order, when the C4 sub-block is encoded / decoded, the C1, C2, and C3 sub-blocks are reconstructed blocks. Therefore, the reconstructed samples of the C1, C2, and C3 sub-blocks can be used together with the previously reconstructed samples for intra prediction of the C4 sub-block as the reconstructed reference sample 420. Figure 4 The gray area in may represent a reconstructed area where encoding / decoding is performed, and the white area may represent an area where encoding / decoding is not performed.

[0124] On the other hand, if the intra prediction mode of the C4 subblock is a mode smaller than the horizontal mode or a mode larger than the vertical mode, intra prediction must be performed using the lower left reference sample 430 or the upper right reference sample 440, respectively. However, since the reference samples of the two regions cannot be used in the encoding / decoding order, the reference samples of the two regions can be derived by filling the available samples in the reconstructed region.

[0125] However, in this case, it is difficult to reflect the characteristics of each reference sample in the corresponding area. As a result, the accuracy of the reference sample is low, and thus the accuracy of the intra prediction is reduced.

[0126] Figure 5 is a diagram for explaining reference samples for intra prediction according to an embodiment of the present invention.

[0127] refer to Figure 5 , the reference samples used for intra prediction of the current block 500 may be composed of at least one of left reference samples L1 to L8, upper left reference samples UL, upper reference samples U1 to U8, upper right reference samples U9 to U16, right reference samples R1 to R8, lower right reference samples BR, lower reference samples B1 to B8, or lower left reference samples L9 to L16. The reference samples according to the above positions may be a sample region including a plurality of samples, or may be a reference sample composed of a single sample.

[0128] For example, the reference sample used for intra prediction may be determined from a left reference sample, an upper left reference sample, an upper reference sample, an upper right reference sample, and a lower left reference sample. Here, the reconstructed reference sample may be a left reference sample, an upper left reference sample, and an upper reference sample, and the derived reference sample may be an upper right reference sample and a lower left reference sample. In this specification, the type of reference sample region may be referred to as a first type reference sample region.

[0129] As another example, the reference sample for intra prediction may be determined from a left reference sample, an upper left reference sample, an upper reference sample, a right reference sample, a lower right reference sample, and a lower reference sample. Here, the reconstructed reference sample may be a left reference sample, an upper left reference sample, and an upper reference sample, and the derived reference sample may be a right reference sample, a lower right reference sample, and a lower reference sample. In the present specification, the type of reference sample region may be referred to as a second type reference sample region.

[0130] As another example, the reference sample for intra prediction may be determined from a left reference sample, an upper left reference sample, an upper right reference sample, a right reference sample, a lower right reference sample, a lower reference sample, and a lower left reference sample. Here, the reconstructed reference sample may be a left reference sample, an upper left reference sample, and an upper reference sample, and the derived reference sample may be an upper right reference sample, a lower left reference sample, a right reference sample, a lower right reference sample, and a lower reference sample. In the present specification, the type of the reference sample region may be referred to as a third type reference sample region.

[0131] Figures 6 to 8 is a schematic diagram for explaining a reference sample derivation method based on a neural network according to an embodiment of the present invention. Specifically, Figure 6 , Figure 7 and Figure 8 They are schematic diagrams for explaining the neural network-based reference sample derivation methods for constructing the first type of reference sample region, the second type of reference sample region, and the third type of reference sample region, respectively.

[0132] refer to Figure 6 , the encoder / decoder can input samples of the reconstructed area 620 around the current block 610 to the neural network processing unit 600 to derive lower left reference samples L1 to L4 630 and upper right reference samples U1 to U4 640 of the unreconstructed area.

[0133] refer to Figure 7 , the encoder / decoder can input samples of the reconstructed area 720 around the current block 710 to the neural network processing unit 700 to derive lower reference samples B1 to B4 730, right reference samples R1 to R4 740 and lower right reference sample BR 750 of the unreconstructed area.

[0134] refer to Figure 8, the encoder / decoder can input samples of the reconstructed area 820 around the current block 810 to the neural network processing unit 800 to derive lower left reference samples L1 to L4 830, upper right reference samples U1 to U4 840, lower reference samples B1 to B4 850, right reference samples R1 to R4 860 and lower right reference sample BR870 of the unreconstructed area.

[0135] On the other hand, the size of the reconstructed samples around the current block used as the input of the neural network processing unit can be determined based on the signaling information. For example, the size of the reconstructed samples used as the input of the neural network processing unit may be (R1×2H)+(2W×R2)+(R1×R2). At this time, the sizes of R1 and R2 may be determined in the encoder and sent to the decoder. As another example, the size of the reconstructed samples used as the input of the neural network processing unit may be (R1×H)+(W×R2)+(R1×R2).

[0136] On the other hand, the size of the reconstructed samples around the current block as input to the neural network processing unit may be a predetermined fixed size.

[0137] The neural network processing unit may be implemented as a neural network model. Here, the artificial neural network model may represent a deep neural network including one or more neural network layers. In addition, the neural network model may include all or part of a convolutional layer, a fully connected layer, and a pooling layer. The neural network model may be implemented in a form including one type of neural network layer, or may be implemented in a form of additionally combining different types of layers.

[0138] On the other hand, the initial internal parameters of the neural network model used in the neural network processing unit are pre-learned, but may be additionally learned during the encoding / decoding process.

[0139] Figures 9 to 11 is a schematic diagram for explaining a reference sample derivation method based on template matching according to an embodiment of the present invention. Specifically, Fig. 9 , Fig.10 and Fig.11 They are schematic diagrams for explaining reference sample derivation methods based on template matching for constructing first-type reference sample regions, second-type reference sample regions, and third-type reference sample regions, respectively.

[0140] The intra-frame template matching prediction (Intra TMP) method refers to a method of searching for the best prediction block in the reconstructed area of ​​the current picture using template matching and copying it to generate a prediction block of the current block. Figures 9 to 11In the reference sample derivation method based on template matching described in , the reference samples around the current block can be derived from the reconstructed area similar to the above-mentioned intra-frame prediction method based on template matching.

[0141] refer to Fig. 9 , the adjacent ┌ area (i.e., the left side, the upper side, and the upper left area) of the current block 910 may be defined as the current template 920. Then, the reference template 930 that is most similar to the current template 920 may be searched within the predefined search ranges R1, R2, R3, and R4 of the reconstructed area of ​​the current picture 900. Then, the lower left reference sample and the upper right reference sample 960 of the unreconstructed area of ​​the current block 910 may be derived based on the lower left sample and the upper right sample 950 of the corresponding matching block 940 of the determined reference template 930.

[0142] refer to Fig.10 , the adjacent ┌ area (i.e., the left side, the upper side, and the upper left side area) of the current block 1010 may be defined as the current template 1020. Then, the reference template 1030 that is most similar to the current template 1020 may be searched within the predefined search ranges R1, R2, R3, and R4 of the reconstructed area of ​​the current picture 1000. Then, the lower side reference sample, the right side reference sample, and the lower right side reference sample 1060 of the unreconstructed area of ​​the current block 1010 may be derived based on the lower side sample, the right side sample, and the lower right side sample 1050 of the corresponding matching block 1040 of the determined reference template 1030.

[0143] refer to Fig.11 , the adjacent ┌ area (i.e., the left side, upper side, and upper left area) of the current block 1110 may be defined as the current template 1120. Then, a reference template 1130 that is most similar to the current template 1120 may be searched within predefined search ranges R1, R2, R3, and R4 of the reconstructed area of ​​the current picture 1100. Then, the lower left reference sample, the upper right reference sample, the lower side reference sample, the right reference sample, and the lower right reference sample 1160 of the unreconstructed area of ​​the current block 1110 may be derived based on the lower left sample, the upper right sample, the lower side sample, the right sample, and the lower right sample 1150 of the corresponding matching block 1040 of the determined reference template 1130.

[0144] Figures 9 to 11 The predefined search ranges R1, R2, R3 and R4 in may be defined as a current CTU (codec tree unit), an upper left CTU, an upper CTU and a left CTU including the current block, respectively.

[0145] In addition, within the predefined search range, the reference template may be searched based on a predefined search order. For example, the reference template may be searched in a zigzag order of R1, R4, R3, and R2.

[0146] On the other hand, information about the search range and the size and shape of the current template may be determined by the encoder and sent to the decoder.

[0147] exist Figures 9 to 11 In the reference sample derivation method based on template matching proposed in , the reference sample can be derived by performing the same template matching process as the encoder in the decoder without signaling (transmitting / parsing) the syntax related to the reference sample derivation.

[0148] Figure 12 to Figure 14 is a schematic diagram for explaining a reference sample derivation method based on motion information according to an embodiment of the present invention. Specifically, Fig.12 , Fig.13 and Fig.14 1 and 2 are schematic diagrams for explaining the reference sample derivation method based on motion information for constructing the first type of reference sample region, the second type of reference sample region and the third type of reference sample region. Different from the reference sample derivation method based on template matching, Figure 12 to Figure 14 In the reference sample derivation method based on motion information described in , reference samples are derived by transmitting / parsing syntax related to motion information.

[0149] refer to Fig.12 , the matching block 1220 may be derived within a predefined search range R1, R2, R3, and R4 of the reconstructed region of the current picture 1200 based on the motion information (motion vector) of the current block 1210. In addition, the lower left side reference sample and the upper right side reference sample 1240 of the unreconstructed region of the current block 1210 may be derived based on the lower left side sample and the upper right side sample 1230 of the matching block 1220.

[0150] refer to Fig.13 , the matching block 1320 may be derived within the predefined search ranges R1, R2, R3, and R4 of the reconstructed region of the current picture 1300 based on the motion information (motion vector) of the current block 1310. In addition, the lower side reference sample, the right side reference sample, and the lower right side reference sample 1340 of the unreconstructed region of the current block 1310 may be derived based on the lower side sample, the right side sample, and the lower right side sample 1330 of the matching block 1320.

[0151] refer to Fig.14, a matching block 1420 may be derived within a predefined search range R1, R2, R3, and R4 of a reconstructed region of the current picture 1400 based on motion information (motion vector) of the current block 1410. In addition, a lower left reference sample, an upper right reference sample, a lower reference sample, a right reference sample, and a lower right reference sample 1440 of an unreconstructed region of the current block 1410 may be derived based on a lower left sample, an upper right sample, a lower sample, a right sample, and a lower right sample 1430 of the matching block 1420.

[0152] Although it has been described in Figure 12 to Figure 14 The matching block can be derived within the search range defined in the image, and the matching block can be derived based on the motion information in the reconstructed area in the current picture.

[0153] On the other hand, the motion information used to derive the reference samples may be determined by the encoder and sent to the decoder.

[0154] Fig.15 FIG. 1 is a schematic diagram for explaining an embodiment of a plurality of reference sample lines according to an embodiment of the present invention. Figures 4 to 14 In the example, the reference sample for intra prediction is described as a sample line, but is not limited to this, and can be as follows Fig.15 As shown in FIG. 1 , a reference sample is determined from at least one sample line among the plurality of sample lines. That is, the reference sample can be determined according to Figures 4 to 14 The reference sample derivation method is used to derive reference samples of multiple reference sample lines.

[0155] On the other hand, the plurality of reference sample lines may be identified by indexes. For example, the indexes (0 to n, n being a positive integer) may be set in order adjacent to the current block 1500.

[0156] Hereinafter, an intra prediction method using reference samples derived by the above-described reference sample derivation method will be described for each intra prediction mode.

[0157] Fig.16 is a schematic diagram for explaining a plane mode according to an embodiment of the present invention.

[0158] In the planar mode according to the embodiment of the present invention, intra prediction can be performed using the left and upper reconstructed reference samples and the right and lower derived reference samples. Here, the right and lower derived reference samples can be derived by the above reference sample derivation method.

[0159] exist Fig.16, Pred(x, y) represents the current pixel 1610 to be predicted, Rec(-1, y) and Rec(x, -1) represent the reconstructed reference samples corresponding to the left and upper sides of the current pixel, respectively, and Ref(W, y) and Ref(x, H) represent the derived reference samples corresponding to the right and lower sides of the current pixel, respectively. In addition, W and H represent the width (horizontal) and height (vertical) of the current block 1600.

[0160] The current pixel Pred(x, y) for the planar mode according to an embodiment of the present invention may be calculated using Equations 1 to 3.

[0161] [Equation 1]

[0162] PredV(x,y)=((H-1-y)×Rec(x,-1)+(y+1)×Ref(x,H))>>log 2 H

[0163] [Equation 2]

[0164] PredH(x,y)=((W-1-x)×Rc(-1,y)+(x+1)×Ref(W,y))>>log 2 W

[0165] [Equation 3]

[0166] Pred(x,y)=(PredV(x,y)×predH(x,y)+W×H)>>(log 2 W+log 2 H+1)

[0167] When the intra prediction mode is the planar mode, a prediction block of the current block may be generated by calculating prediction values ​​for all pixels (0≤x≤W-1, 0≤y≤H-1) in the current block using Equations 1 to 3.

[0168] Fig.17 is a schematic diagram for explaining a horizontal plane mode according to an embodiment of the present invention.

[0169] In the horizontal plane mode according to an embodiment of the present invention, intra prediction can be performed using the left reconstructed reference samples and the right derived reference samples. Here, the right derived reference samples can be derived by the above reference sample derivation method.

[0170] exist Fig.17, Pred(x, y) represents the current pixel 1710 to be predicted, Rec(-1, y) represents the reconstructed reference sample corresponding to the left side of the current pixel, and Ref(W, y) represents the derived reference sample corresponding to the right side of the current pixel. In addition, W and H represent the width (horizontal) and height (vertical) of the current block 1700.

[0171] The current pixel Pred(x, y) for the horizontal plane mode according to an embodiment of the present invention can be calculated using Equation 4.

[0172] [Equation 4]

[0173] Pred(x,y)=((W-1-x)×Rec(-1,y)+(x+1)×Ref(W,y)+(W>>1))>>log 2 W

[0174] When the intra prediction mode is the horizontal plane mode, a prediction block of the current block may be generated by calculating prediction values ​​for all pixels (0≤x≤W-1, 0≤y≤H-1) in the current block using Equation 4.

[0175] Fig.18 is a schematic diagram for explaining a vertical plane mode according to an embodiment of the present invention.

[0176] In the vertical plane mode according to the embodiment of the present invention, intra prediction can be performed using the upper reconstructed reference samples and the lower derived reference samples. Here, the lower derived reference samples can be derived by the above reference sample derivation method.

[0177] exist Fig.18 , Pred(x, y) represents the current pixel 1810 to be predicted, Rec(x, -1) represents the reconstructed reference sample corresponding to the upper side of the current pixel, and Ref(x, H) represents the derived reference sample corresponding to the lower side of the current pixel. In addition, W and H represent the width (horizontal) and height (vertical) of the current block 1800.

[0178] The current pixel Pred(x, y) for the vertical plane mode according to an embodiment of the present invention can be calculated using Equation 5.

[0179] [Equation 5]

[0180] Pred(x,y)=((H-1-y)×Rec(x,-1)+(y+1)×Ref(x,H)+(H>>1))>>log 2 H

[0181] When the intra prediction mode is the vertical plane mode, a prediction block of the current block may be generated by calculating prediction values ​​for all pixels (0≤x≤W-1, 0≤y≤H-1) in the current block using Equation 5.

[0182] Fig.19 is a schematic diagram for explaining a DC mode according to an embodiment of the present invention.

[0183] In the DC mode according to the embodiment of the present invention, intra prediction can be performed using left and upper reconstructed reference samples and right and lower derived reference samples. Here, the right and lower derived reference samples can be derived by the above reference sample derivation method.

[0184] exist Fig.19 Middle,U 1 ~U W and L 1 ~L H represents the reconstructed samples on the upper and left sides, and R 1 ~R H and B 1 ~B W In addition, W and H represent the width (horizontal) and height (vertical) of the current block 1900 .

[0185] In the DC mode according to an embodiment of the present invention, the same prediction value may be assigned to all pixels in the current block, and the prediction value may be calculated using Equation 6.

[0186] [Equation 6]

[0187]

[0188] Also, a prediction block for the DC mode may be generated by generating a prediction value for the DC mode using Equation 6 and allocating it to all pixels in the current block (0≤x≤W-1, 0≤y≤H-1).

[0189] If the intra prediction mode is the DC mode, a prediction block of the current block may be generated by assigning a prediction value calculated using Equation 6 to all pixels in the current block (0≤x≤W-1, 0≤y≤H-1).

[0190] exist Fig.19 In , all neighboring reference samples of the current block are used to calculate the prediction value for DC mode.

[0191] However, this is an example, and considering the complexity, the pixel values ​​at specific positions may be sampled and selected, and then the prediction value for the DC mode may be calculated using the selected pixel values.

[0192] As an example, one pixel may be sampled from each of the left reference sample, the upper reference sample, the right reference sample, and the lower reference sample, and a prediction value for the DC mode may be calculated based on the sampled pixels. In this case, the prediction value of the DC mode may be calculated using Equation 7 or Equation 8.

[0193] [Equation 7]

[0194]

[0195] [Equation 8]

[0196]

[0197] In Equation 7 or Equation 8, a prediction value for the DC mode is calculated by sampling one pixel from each of a left reference sample, an upper reference sample, a right reference sample, and a lower reference sample.

[0198] On the other hand, according to an embodiment of the present invention, the prediction value for the DC mode can be calculated by sampling N pixels from each of the left reference sample, the upper reference sample, the right reference sample, and the lower reference sample. Here, N can be any positive integer.

[0199] Fig. 20 and Fig.21 is a schematic diagram for explaining an angle pattern according to an embodiment of the present invention.

[0200] Fig. 20 is a diagram for explaining intra prediction using upper right reference samples U9 to U16 when the intra prediction mode is an angle mode greater than the vertical mode (ie, when the intra prediction mode is an angle mode indicating a clockwise direction with respect to the vertical direction).

[0201] refer to Fig. 20 , the dotted line indicates the directionality of the intra prediction mode. Figure 4 If the C4 sub-block is the same as that of the pixel 2010, there is a problem that the upper right reference samples U9 to U16 are not reconstructed and therefore cannot be used as reference samples for intra prediction of the current pixel 2010.

[0202] like Figure 4 As described above, it can be obtained by using the adjacent samples U reconstructed based on the previous 1 The upper right reference samples U9 to U16 are derived by padding U8 to U9. However, when the upper right reference samples are derived by padding, it is difficult to reflect the characteristics of each reference sample of the corresponding area. As a result, the accuracy of the reference samples is low, and the accuracy of intra prediction is reduced.

[0203] On the other hand, the upper right reference samples U9 to U16 can be derived based on any one of the neural network-based reference sample prediction method, the template matching-based reference sample prediction method, and the motion information-based reference sample prediction method according to an embodiment of the present invention.

[0204] Fig.21 It is a diagram for explaining intra prediction using lower left reference samples L9 to L16 when the intra prediction mode is an angle mode smaller than the horizontal mode (ie, when the intra prediction mode is an angle mode indicating a counterclockwise direction based on the horizontal direction).

[0205] refer to Fig.21 , the dotted line indicates the directionality of the intra prediction mode. Figure 4 If the C4 sub-block is the same as that of the pixel 2110, there is a problem that the lower left reference samples L9 to L16 are not reconstructed and therefore cannot be used as reference samples for intra-frame prediction of the current pixel 2110.

[0206] like Figure 4 As described above, the lower left reference samples L9 to L16 can be derived by padding based on the previously reconstructed adjacent samples L1 to L8. However, when the lower left reference samples are derived by padding, it is difficult to reflect the various features of the reference samples in the corresponding area. As a result, the accuracy of the reference samples will be low, and the accuracy of the intra-frame prediction will be reduced.

[0207] On the other hand, the lower left reference samples L9 to L16 can be derived based on any one of the neural network-based reference sample prediction method, the template matching-based reference sample prediction method, and the motion information-based reference sample prediction method according to an embodiment of the present invention.

[0208] On the other hand, according to the embodiment of the present invention, the lower reference sample and the right reference sample can be derived by any one of a neural network-based reference sample prediction method, a template matching-based reference sample prediction method, and a motion information-based reference sample prediction method. Accordingly, the angle intra-frame prediction mode may include all modes representing 360 degrees of omnidirectionality. By implementing 360-degree omnidirectional intra-frame prediction via reference samples derived in this way, the problem of unidirectional intra-frame prediction using only previously reconstructed reference samples, that is, the problem of low prediction accuracy of current samples far away from the reconstructed reference samples, can be solved.

[0209] In addition, by deriving the lower reference sample and the right reference sample, bidirectional intra prediction can be implemented. Here, bidirectional intra prediction is an intra prediction method that generates a prediction value by weighted sum of reference samples determined based on the direction of the intra prediction mode and reference samples determined based on the opposite direction of the intra prediction mode.

[0210] Fig. 22 is a flowchart illustrating an intra prediction mode parsing method according to an embodiment of the present invention.

[0211] The plane mode and DC mode according to the embodiment of the present invention are modes that replace the existing plane mode and DC mode, and the horizontal plane mode and the vertical plane mode are new prediction modes that do not exist in the existing intra-frame prediction mode. If the horizontal plane mode and the vertical plane mode are used, the number of intra-frame prediction modes can be increased by 2.

[0212] Hereinafter, a method of transmitting / parsing horizontal plane mode and vertical plane mode information will be described.

[0213] The horizontal plane mode and the vertical plane mode are considered as angle modes, and information about the horizontal plane mode and the vertical plane mode can be transmitted / parsed. In this case, the number of angle modes increases by 2, but the same syntax transmission / parse method as the existing method can be used.

[0214] On the other hand, the plane flag and the plane mode information may be transmitted / parsed. Here, the plane flag may indicate whether one of the plane mode, horizontal plane mode and vertical plane mode is executed, and the plane mode information may indicate which of the plane mode, horizontal plane mode and vertical plane mode is executed.

[0215] Fig. 22 is a flow chart illustrating a method of parsing intra prediction mode information.

[0216] refer to Fig. 22 , the image decoding device may parse the plane flag (S2210). Then, if the parsed plane flag indicates that one of the plane mode, the horizontal plane mode, and the vertical plane mode is used (S2220-Yes), the image decoding device may parse the plane mode information to determine the intra-frame prediction mode as one of the plane mode, the horizontal plane mode, and the vertical plane mode (S2230).

[0217] On the other hand, if the parsed plane flag indicates that none of the planar mode, horizontal plane mode, and vertical plane mode is used (S2220-No), the image decoding device may determine the intra prediction mode by parsing the intra prediction mode information not including the planar mode (S2240).

[0218] [Table 1]

[0219] model Codeword Plane Mode 0 (or 1) Horizontal plane mode 10(or 01) Vertical plane mode 11 (or 00)

[0220] Table 1 is a table showing code words of a plane mode, a horizontal plane mode, and a vertical plane mode. In Table 1, the code words of the horizontal plane mode and the vertical plane mode can be arbitrarily determined. For example, the code word of the horizontal plane mode can be assigned to 11 (or 00), and the code word of the vertical plane mode can be assigned to 10 (or 01).

[0221] On the other hand, when the reference sample derivation method of the present invention is selectively applied only to the planar mode and the DC mode, the lower reference sample and the right reference sample can be derived based on any one of the reference sample prediction methods based on a neural network, the reference sample prediction method based on template matching, and the reference sample prediction method based on motion information.

[0222] Alternatively, when the reference sample derivation method of the present invention is only selectively applied to the angle mode, the upper right reference sample and the lower left reference sample can be derived based on any one of the reference sample prediction methods based on a neural network, the reference sample prediction method based on template matching, and the reference sample prediction method based on motion information.

[0223] Alternatively, when the reference sample derivation method of the present invention is applied to plane mode, horizontal plane mode, vertical plane mode, DC mode and angle mode, the upper right reference sample, lower left reference sample, lower reference sample, lower right reference sample and right side reference sample can be derived based on any one of the reference sample prediction methods based on neural network, the reference sample prediction method based on template matching and the reference sample prediction method based on motion information.

[0224] Fig.23 is a flowchart illustrating an image decoding method according to an embodiment of the present invention. Fig.23 The image decoding method can be performed by an image decoding device.

[0225] The image decoding apparatus may determine an intra prediction mode of a current block (S2310). Here, the intra prediction mode may be determined as one of a plane mode, a vertical plane mode, a horizontal plane mode, a DC mode, and an angle mode.

[0226] Specifically, the step of determining the intra prediction mode may include the step of parsing a plane flag, and the step of parsing one of the plane mode information and the intra prediction mode information according to the plane flag to determine the intra prediction mode.

[0227] The plane flag may be information indicating whether to execute one of a plane mode, a horizontal plane mode, and a vertical plane mode.

[0228] The planar mode information may be information indicating which of the planar mode, the horizontal planar mode, and the vertical planar mode is performed.

[0229] The intra prediction mode information may be information indicating an intra prediction mode other than the planar mode, the horizontal plane mode, and the vertical plane mode.

[0230] On the other hand, Fig. 22 A method for determining an intra-frame prediction mode by parsing a plane flag, plane mode information, and intra-frame prediction mode information is described in detail.

[0231] Then, the image decoding device can derive a reference sample of the unreconstructed area (S2320). Here, the reference sample of the unreconstructed area can be a reference sample of an area located at least one of the lower left side, lower side, lower right side, right side and upper right side of the current block.

[0232] On the other hand, reference samples of the unreconstructed area may be derived based on neighboring samples of a matching block by performing a template matching search.

[0233] Alternatively, reference samples of the unreconstructed area may be derived based on neighboring samples of a matching block in the current picture indicated by motion information of the current block.

[0234] Alternatively, reference samples of the unreconstructed region may be derived based on a neural network model having reference samples of the reconstructed region as input.

[0235] On the other hand, Figures 4 to 15 The method for deriving reference samples for unreconstructed regions is described in detail in .

[0236] Then, the image decoding apparatus may perform intra prediction according to the intra prediction mode based on at least one of the derived reference sample of the unreconstructed region or the reference sample of the reconstructed region ( S2330 ).

[0237] If the intra-frame prediction mode of the current block is determined to be the planar mode, intra-frame prediction can be performed based on reference samples of the reconstructed area corresponding to the upper side of the current pixel, reference samples of the reconstructed area corresponding to the left side, reference samples of the unreconstructed area corresponding to the lower side, and reference samples of the unreconstructed area corresponding to the right side.

[0238] If the intra prediction mode of the current block is determined to be the vertical plane mode, the image decoding method may perform intra prediction based on reference samples of a reconstructed area corresponding to an upper side of the current pixel and reference samples of an unreconstructed area corresponding to a lower side.

[0239] If the intra prediction mode of the current block is determined to be the horizontal plane mode, the image decoding method may perform intra prediction based on reference samples of the reconstructed area corresponding to the left side of the current pixel and reference samples of the unreconstructed area corresponding to the right side.

[0240] If the intra-frame prediction mode of the current block is determined to be the DC mode, the image decoding method can perform intra-frame prediction based on reference samples of the reconstructed area located on the upper side of the current block, reference samples of the reconstructed area located on the left side, reference samples of the unreconstructed area located on the lower side, and reference samples of the unreconstructed area located on the right side.

[0241] On the other hand, Figures 16 to 21 The intra prediction method according to the intra prediction mode is described in detail in.

[0242] On the other hand, it can be performed in the same manner as in the image encoding method Fig.23 Alternatively, you can include Fig.23 The image encoding method of the steps described in the above is used to generate a bit stream. The bit stream can be stored in a non-volatile computer-readable recording medium and can also be transmitted (or streamed).

[0243] Fig.24 A content streaming system to which an embodiment according to the present invention can be applied is exemplarily shown.

[0244] like Fig.24 As shown, the content streaming system to which the embodiment of the present invention is applied may mainly include an encoding server, a streaming server, a network server, a media storage device, a user device and a multimedia input device.

[0245] The encoding server compresses the content received from the multimedia input device such as a smartphone, a camera, a CCTV, etc. into digital data to generate a bit stream and transmits it to the streaming server. As another example, if the multimedia input device such as a smartphone, a camera, a CCTV, etc. directly generates a bit stream, the encoding server can be omitted.

[0246] A bitstream may be generated by the image encoding method and / or the image encoding device to which the embodiments of the present invention are applied, and a streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0247] The streaming server sends multimedia data to the user device based on a user request via the network server, and the network server can act as an intermediary to notify the user of any available services. When the user requests a desired service from the network server, the network server sends it to the streaming server, and the streaming server can send the multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server can control the command / response between the devices within the content streaming system.

[0248] The streaming server can receive content from a media storage device and / or an encoding server. For example, when receiving content from an encoding server, the content can be received in real time. In this case, in order to provide a stable streaming service, the streaming server can store the bitstream for a period of time.

[0249] Examples of user devices may include mobile phones, smart phones, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, tablet PCs, tablet PCs, ultrabooks, wearable devices (e.g., smart watches, smart glasses, HMDs), digital televisions, desktop computers, digital signage, etc.

[0250] Each server in the above-described content streaming system may operate as a distributed server, in which case data received from each server may be distributed and processed.

[0251] The above-mentioned embodiments may be performed in the same or corresponding manner in the encoding device and the decoding device. In addition, an image may be encoded / decoded using at least one or a combination of at least one of the above-mentioned embodiments.

[0252] The order of applying the above-described embodiments may be different in the encoding device and the decoding device. Alternatively, the order of applying the above-described embodiments may be the same in the encoding device and the decoding device.

[0253] The above-described embodiments may be performed on each of the luminance signal and the chrominance signal. Alternatively, the above-described embodiments of the luminance signal and the chrominance signal may be performed identically.

[0254] In the above-mentioned embodiment, the method is described based on a flow chart with a series of steps or units, but the present invention is not limited to the order of the steps, on the contrary, some steps can be performed simultaneously with other steps or in different orders. In addition, it should be understood by those of ordinary skill in the art that the steps in the flow chart are not mutually exclusive, and other steps can be added to the flow chart or some steps can be deleted from the flow chart without affecting the scope of the present invention.

[0255] The embodiments can be implemented in the form of program instructions, which can be executed by various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include independent program instructions, data files, data structures, etc., or a combination thereof. The program instructions recorded in the computer-readable recording medium may be specially designed and constructed for the present invention, or are well known to those of ordinary skill in the field of computer software technology.

[0256] The bit stream generated by the encoding method according to the above-mentioned embodiment can be stored in a non-volatile computer-readable recording medium. In addition, the bit stream stored in the non-volatile computer-readable recording medium can be decoded by the decoding method according to the above-mentioned embodiment.

[0257] Examples of computer-readable recording media include: magnetic recording media such as hard disks, floppy disks, and magnetic tapes; optical data storage media such as CD-ROMs or DVD-ROMs; magneto-optical media such as floppy disks; and hardware devices such as read-only memory (ROM), random access memory (RAM), flash memory, etc., which are specially configured to store and execute program instructions. Examples of program instructions include not only machine language codes formatted by a compiler, but also high-level language codes that can be implemented by a computer using an interpreter. The hardware device can be configured to be operated by one or more software modules or vice versa to perform the process according to the present invention.

[0258] Although the present invention has been described in terms of specific items such as detailed elements and limited embodiments and drawings, they are provided only to help a more comprehensive understanding of the present invention, and the present invention is not limited to the above embodiments. It will be appreciated by those skilled in the art that various modifications and changes can be made based on the above description.

[0259] Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the full scope of the appended claims and their equivalents will fall within the scope and spirit of the present invention.

[0260] Industrial Applicability

[0261] The present invention can be used in an apparatus for encoding / decoding an image and a recording medium for storing a bit stream.

Claims

1. An image decoding method, comprising: Determining an intra prediction mode for a current block; deriving reference samples for the unreconstructed regions; performing intra prediction according to an intra prediction mode based on at least one of the derived reference samples of the unreconstructed region or the reference samples of the reconstructed region, The reference sample of the unreconstructed area is a reference sample of an area located at least one of the lower left side, the lower side, the lower right side, the right side or the upper right side of the current block.

2. The image decoding method according to claim 1, wherein: Reference samples of the unreconstructed region are derived based on neighboring samples of a matching block by performing a template matching search.

3. The image decoding method according to claim 1, wherein: Reference samples of the unreconstructed area are derived based on neighboring samples of a matching block in a current picture indicated by motion information of the current block.

4. The image decoding method according to claim 1, wherein: Reference samples of the unreconstructed region are derived based on a neural network model having the reference samples of the reconstructed region as input.

5. The image decoding method according to claim 1, wherein: The intra prediction mode is determined to be any one of a planar mode, a vertical planar mode, a horizontal planar mode, a DC mode, or an angular mode.

6. The image decoding method according to claim 5, wherein: When the intra-frame prediction mode is determined to be the planar mode, intra-frame prediction is performed based on reference samples of the reconstructed area corresponding to the upper side of the current pixel, reference samples of the reconstructed area corresponding to the left side, reference samples of the unreconstructed area corresponding to the lower side, and reference samples of the unreconstructed area corresponding to the right side.

7. The image decoding method according to claim 5, wherein: When the intra prediction mode is determined to be the vertical plane mode, intra prediction is performed based on reference samples of a reconstructed area corresponding to an upper side of the current pixel and reference samples of a non-reconstructed area corresponding to a lower side.

8. The image decoding method according to claim 5, wherein: When the intra prediction mode is determined to be the horizontal plane mode, intra prediction is performed based on reference samples of a reconstructed area corresponding to the left side of the current pixel and reference samples of a non-reconstructed area corresponding to the right side.

9. The image decoding method according to claim 5, wherein: When the intra-frame prediction mode is determined to be the DC mode, intra-frame prediction is performed based on reference samples of the reconstructed area located on the upper side of the current block, reference samples of the reconstructed area located on the left side, reference samples of the unreconstructed area located on the lower side, and reference samples of the unreconstructed area located on the right side.

10. The image decoding method according to claim 5, wherein: Determining the intra prediction mode includes: Parsing flat logos; and The intra prediction mode is determined by parsing any one of the plane mode information and the intra prediction mode information according to the plane flag.

11. An image encoding method, comprising: Determining an intra prediction mode for a current block; deriving reference samples for the unreconstructed regions; as well as performing intra prediction according to an intra prediction mode based on at least one of the derived reference samples of the unreconstructed region or the reference samples of the reconstructed region, The reference sample of the unreconstructed area is a reference sample of an area located at least one of the lower left side, the lower side, the lower right side, the right side or the upper right side of the current block.

12. A non-volatile computer-readable recording medium storing a bit stream generated by an image encoding method, the image encoding method comprising: Determining an intra prediction mode for a current block; deriving reference samples for the unreconstructed regions; as well as performing intra prediction according to an intra prediction mode based on at least one of the derived reference samples of the unreconstructed region or the reference samples of the reconstructed region, The reference sample of the unreconstructed area is a reference sample of an area located at least one of the lower left side, the lower side, the lower right side, the right side or the upper right side of the current block.

13. A method for transmitting a bit stream generated by an image encoding method, the method comprising: Transmit bit stream, The encoding methods include: Determining an intra prediction mode for a current block; deriving reference samples for the unreconstructed regions; and performing intra prediction according to an intra prediction mode based on at least one of the derived reference samples of the unreconstructed region or the reference samples of the reconstructed region, The reference sample of the unreconstructed area is a reference sample of an area located at least one of the lower left side, the lower side, the lower right side, the right side or the upper right side of the current block.

Images