Image encoding / decoding method and device

Abstract

The present invention relates to an image encoding / decoding method and device. The image decoding method according to an embodiment of the present invention comprises the steps of: deriving an input value of a neural network model on the basis of reference samples adjacent to the current block; deriving intra prediction-related information about the current block by inputting the input value to the neural network model; and generating a prediction block of the current block on the basis of the intra prediction-related information about the current block, wherein the intra prediction-related information about the current block includes at least one of information about an intra prediction mode of the current block, information about an intra prediction method of the current block, an intra prediction value of the current block, or information about a transform kernel applied to the current block, and when the current block is adjacent to a boundary of a prescribed region including the current block, available reference samples may be reference samples located inside the prescribed region.

Description

Video encoding / decoding method and device

[0001] The present invention relates to a method and device for encoding / decoding an image, and more particularly, to a method and device for encoding / decoding an image using a neural network-based intra prediction method.

[0002] Recently, the demand for multimedia data, such as video, has been rapidly increasing. In particular, the demand for high-resolution, high-quality video, such as HD (High Definition) and UHD (Ultra High Definition) video, is growing across a wide range of applications. High-resolution, high-quality video data typically requires significantly more data volume than conventional video data. Consequently, the transmission and storage costs for storing and / or transmitting high-resolution, high-quality video data increase compared to conventional video data.

[0003] To solve these problems, high-efficiency image encoding / decoding technology for images with higher resolution and quality is required.

[0004] In order to encode an image, various techniques are used, such as an intra prediction technique that predicts the pixel values ​​included in the current picture using pixel information in the current picture, an intra prediction technique that predicts the pixel values ​​included in the current picture from the pictures before or after the current picture, a transform and quantization technique for compressing the energy of the residual signal, which is the difference between the predicted signal and the original signal, and an entropy coding technique that assigns short codes to values ​​with high appearance frequencies and long codes to values ​​with low appearance frequencies. In addition, various tools are being developed to implement each technique in order to improve the efficiency of image coding. In addition, in order to decode an encoded image, the image can be restored and reproduced through an image decoding technique that uses a technique and tools corresponding to the image coding technique.

[0005] Using these video encoding and decoding technologies, video data can be effectively compressed, transmitted, stored, and played back.

[0006] The present disclosure aims to provide an image encoding / decoding method and device that improves a neural network-based intra prediction method so that it can be applied to various situations and enhances the compression efficiency of image data.

[0007] The technical challenges to be achieved through this disclosure are not limited to the technical challenges mentioned above. Furthermore, other technical challenges not mentioned in this disclosure will be readily apparent to those skilled in the art from this disclosure.

[0008] According to one embodiment of the present invention, a video decoding method includes a step of deriving an input value of a neural network model based on available reference samples among reference samples adjacent to a current block, a step of inputting the input value into the neural network model to derive intra-prediction-related information of the current block, and a step of generating a prediction block of the current block based on the intra-prediction-related information of the current block, wherein the intra-prediction-related information of the current block includes at least one of information regarding an intra-prediction mode of the current block, information regarding an intra-prediction method of the current block, an intra-prediction value of the current block, and information regarding a transform kernel applied to the current block, and when the current block is adjacent to a boundary of a predetermined area including the current block, the available reference sample may be a reference sample located inside the predetermined area.

[0009] In the above image decoding method, the reference samples adjacent to the current block may include a block adjacent to the left of the current block, a block adjacent to the top of the current block, and a block adjacent to the top left of the current block.

[0010] In the above image decoding method, the reference samples adjacent to the current block may further include a block adjacent to the lower left of the current block and a block adjacent to the upper right of the current block.

[0011] In the above image decoding method, the input values ​​of the neural network model may be characterized by including the available reference samples and additional sample values ​​derived based on the available reference samples.

[0012] In the above image decoding method, the prediction block of the current block may be generated based on one of the intra prediction value of the current block output by the neural network and the intra prediction value of the current block derived based on the intra prediction mode of the current block output by the neural network.

[0013] In the above image decoding method, the prediction block of the current block may be generated by weighting and adding the intra prediction value of the current block derived based on the intra prediction mode of the current block output by the neural network and the intra prediction value of the current block output by the neural network.

[0014] In the above image decoding method, the neural network model may output an intra prediction value of the current block based on an input value whose range has been changed, and the range of the input value may be changed based on an internal operation bit depth.

[0015] In the above image decoding method, the neural network model may be characterized in that the range of the intra prediction value of the current block output based on the input value whose range has been changed is changed based on the bit depth on the domain of the current block.

[0016] In the above image decoding method, the prediction block of the current block may be generated based on one of an intra prediction value derived based on an intra prediction mode of the current block and an intra prediction value derived based on an intra prediction mode derived based on signal processing.

[0017] In the above image decoding method, the prediction block of the current block may be generated by weighting an intra prediction value derived based on an intra prediction mode of the current block and an intra prediction value derived based on an intra prediction mode derived based on signal processing.

[0018] In the above image decoding method, information on a transform kernel applied to the current block may be characterized by indicating one MTS set among MTS (multiple transform selection) sets including a plurality of transform kernels.

[0019] In the above image decoding method, information on a transform kernel applied to the current block may be characterized by indicating one NSPT (non-separable primary transform) set among NSPT sets including a plurality of transform kernels.

[0020] In the above image decoding method, information on a transform kernel applied to the current block may be characterized by indicating one LFNST (low-frequency non-separable transform) set among LFNST sets including a plurality of transform kernels.

[0021] According to one embodiment of the present invention, a video encoding method includes a step of deriving an input value of a neural network model based on available reference samples among reference samples adjacent to a current block, a step of deriving intra-prediction-related information of the current block by inputting the input value into the neural network model, and a step of generating a prediction block of the current block based on the intra-prediction-related information of the current block, wherein the intra-prediction-related information of the current block includes at least one of information regarding an intra-prediction mode of the current block, information regarding an intra-prediction method of the current block, an intra-prediction value of the current block, and information regarding a transform kernel applied to the current block, and when the current block is adjacent to a boundary of a predetermined area including the current block, the available reference sample may be a reference sample located inside the predetermined area.

[0022] A non-transitory computer-readable recording medium storing a bitstream generated by an image encoding method according to one embodiment of the present invention comprises the steps of: deriving an input value of a neural network model based on available reference samples among reference samples adjacent to a current block; inputting the input value into the neural network model to derive intra-prediction-related information of the current block; and generating a prediction block of the current block based on the intra-prediction-related information of the current block, wherein the intra-prediction-related information of the current block includes at least one of information regarding an intra-prediction mode of the current block, information regarding an intra-prediction method of the current block, an intra-prediction value of the current block, and information regarding a transform kernel applied to the current block, wherein when the current block is adjacent to a boundary of a predetermined area including the current block, the available reference sample is a reference sample located inside the predetermined area.

[0023] A method for transmitting a bitstream generated by a video encoding method according to an embodiment of the present invention includes the steps of transmitting a bitstream, deriving an input value of a neural network model based on available reference samples among reference samples adjacent to a current block, inputting the input value into the neural network model to derive intra-prediction-related information of the current block, and generating a prediction block of the current block based on the intra-prediction-related information of the current block, wherein the intra-prediction-related information of the current block includes at least one of information regarding an intra-prediction mode of the current block, information regarding an intra-prediction method of the current block, an intra-prediction value of the current block, and information regarding a transform kernel applied to the current block, wherein when the current block is adjacent to a boundary of a predetermined region including the current block, the available reference sample is a reference sample located inside the predetermined region.

[0024] The present disclosure can provide an image encoding / decoding method and device that improves a neural network-based intra prediction method applicable to various situations and enhances the compression efficiency of image data.

[0025] In addition, according to the present disclosure, a recording medium storing a bitstream generated by the image encoding method or device of the present invention can be provided.

[0026] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below.

[0027] FIG. 1 is a block diagram showing an image encoding device according to one embodiment of the present invention.

[0028] FIG. 2 is a block diagram showing an image decoding device according to one embodiment of the present invention.

[0029] FIG. 3 is a diagram schematically showing a video coding system to which the present invention can be applied.

[0030] FIG. 4 is a drawing exemplarily showing a content streaming system to which an embodiment according to the present invention can be applied.

[0031] FIG. 5 is a diagram for explaining a neural network-based intra prediction method according to one embodiment of the present disclosure.

[0032] FIG. 6 is a diagram for explaining restored pixels used to derive context in a neural network-based intra prediction method according to an embodiment of the present disclosure.

[0033] FIG. 7 is a diagram for explaining a current block corresponding to a boundary condition in a neural network-based intra prediction method according to an embodiment of the present disclosure.

[0034] FIG. 8 is a diagram illustrating an example of a structure of partitioned blocks based on a dual tree according to an embodiment of the present invention.

[0035] FIG. 9 is a flowchart illustrating an image decoding method for predicting a chrominance block based on information of a luminance block according to an embodiment of the present disclosure.

[0036] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention. Throughout the description of each drawing, similar reference numerals have been used to designate similar components.

[0037] While terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0038] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0039] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0040] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. Hereinafter, identical components in the drawings will be designated by the same reference numerals, and redundant descriptions of identical components will be omitted.

[0041] FIG. 1 is a block diagram showing an image encoding device according to one embodiment of the present invention.

[0042] Referring to FIG. 1, an image encoding device (100) may include an image segmentation unit (101), an intra prediction unit (102), an inter prediction unit (103), a subtraction unit (104), a transformation unit (105), a quantization unit (106), an entropy encoding unit (107), an inverse quantization unit (108), an inverse transformation unit (109), an addition unit (110), a filter unit (111), and a memory (112).

[0043] Each component shown in Fig. 1 is independently depicted to indicate different characteristic functions in the video encoding device, and does not mean that each component is composed of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two components among each component may be combined to form a single component, or one component may be divided into multiple components to perform a function, and such integrated and separate embodiments of each component are also included in the scope of the present invention as long as they do not deviate from the essence of the present invention.

[0044] Additionally, some components may not be essential components that perform essential functions of the present invention, but may be optional components merely used to enhance performance. The present invention may be implemented by including only components essential to implementing the essence of the present invention, excluding components used solely for performance enhancement. A structure that includes only essential components, excluding optional components used solely for performance enhancement, is also within the scope of the present invention.

[0045] The image segmentation unit (101) can segment an input image into at least one block. At this time, the input image can have various shapes and sizes such as a sequence, a picture, a slice, a tile, a segment, a tile group, a coding tree unit, etc. According to another embodiment, the image segmentation unit (101) can segment one input picture into a plurality of sub-pictures defined as a group of rectangular slices, segment each sub-picture into the tiles / slices, and segment the tiles / slices into coding tree units.

[0046] In addition, the image segmentation unit (101) can recursively segment the segmented coding tree unit. The terminal node segmented from the coding tree unit may be referred to as a coding unit (CU). The block may mean a coding unit (CU), or a prediction unit (PU) or a transformation unit (TU) segmented from the coding unit (CU). The segmentation may be performed based on at least one of a quadtree, a binary tree, and a ternary tree. The quadtree is a method of segmenting an upper block into lower blocks whose width and height are half of those of the upper block. The binary tree is a method of segmenting an upper block into lower blocks whose width or height is half of that of the upper block. The ternary tree is a method of segmenting an upper block into three lower blocks. For example, the three lower blocks may be obtained by segmenting the width or height of the upper block at a ratio of 1:2:1. Through the binary tree-based partitioning described above, blocks can have not only square but also non-square shapes. Blocks can first be partitioned into a quad tree. Blocks corresponding to leaf nodes of the quad tree can be left unpartitioned, or can be partitioned into a binary tree or a ternary tree. Leaf nodes of the binary tree or ternary tree can be units of encoding, prediction, and / or transformation.

[0047] The image segmentation unit (101) can recursively segment the CTU into not only a quad tree (QT) but also a multi-type tree (MTT). Here, the MTT can be composed of a binary tree (BT) and a triple tree (TT). For example, the MTT structure can be divided into a vertical binary tree segmentation mode (SPLIT_BT_VER), a horizontal binary tree segmentation mode (SPLIT_BT_HOR), a vertical ternary tree segmentation mode (SPLIT_TT_VER), and a horizontal ternary tree segmentation mode (SPLIT_TT_HOR).

[0048] In addition, the image segmentation unit (101) can segment a CTU by applying a dual tree that uses different CTU segmentation structures for luminance and chrominance components, or by applying a single tree in which luminance and chrominance CTBs (Coding Tree Blocks) within a CTU share a coding tree structure.

[0049] The prediction unit (102, 103) may include an intra-prediction unit (102) that performs intra-prediction and an inter-prediction unit (103) that performs inter-prediction. The prediction unit (102, 103) may determine whether to use intra-prediction or inter-prediction for a prediction unit. In addition, the prediction unit (102, 103) may determine specific information (e.g., intra-prediction mode, inter-prediction mode, motion vector, reference picture, etc.) according to the determined prediction method. At this time, the processing unit where the prediction is performed and the processing unit where the prediction method and specific contents are determined may be different. For example, the prediction unit (102, 103) may determine the prediction method and prediction mode for each prediction unit, and perform prediction according to the transformation unit.

[0050] In another embodiment, the prediction unit may encode the input image using a third mode (e.g., IBC mode, Palette mode, etc.) other than the intra mode and the inter mode. However, if the third mode has functional characteristics similar to the intra mode or the inter mode, the third mode may be classified as the intra mode or the inter mode. In this disclosure, the third mode will be described only when a specific description thereof is required.

[0051] The intra prediction unit (102) can generate a prediction block of the current block based on the intra prediction mode of the current block and reference pixel information around the current block, which is pixel information within the current picture. If a neighboring block of the current block is predicted by inter prediction, the reference pixels included in the inter-predicted neighboring block can be replaced with reference pixels within another neighboring block that has been intra-predicted. That is, if a reference pixel is not available, the intra prediction unit (102) can perform intra prediction of the current block by replacing the unavailable reference pixel with at least one reference pixel among the available reference pixels.

[0052] The intra prediction unit (102) can use multiple reference pixel lines for intra prediction of the current block. If multiple reference pixel lines are available, information indicating a reference pixel line used for intra prediction among the multiple reference pixel lines can be signaled.

[0053] Intra prediction modes used for intra prediction may include a directional prediction mode that uses reference pixel information according to the prediction direction, and a non-directional mode that does not use directional information. Additionally, the mode for predicting luminance information and the mode for predicting chrominance information may be different, and the intra prediction mode information of the luminance component block or the predicted luminance signal information may be utilized to predict chrominance information.

[0054] Alternatively, the intra prediction unit (102) may perform intra prediction on the current block by applying at least one mode among decoder-side intra mode derivation (DIMD), Occurrence-based intra coding (OBIC), extrapolation filter based intra prediction mode (EIP), template based intra mode derivation (TIMD), spatial geometric partitioning mode (SGPM) mode, intra template matching prediction (IntraTMP), and intra block copy. When the intra prediction mode of the block is a predetermined mode, the intra prediction unit (102) may perform intra prediction on the current block by using a block vector indicating a block other than the current block.

[0055] The intra prediction unit (102) may include a reference sample filter, an interpolation filter, and a DC filter. The reference sample filter is a filter that performs filtering on the reference pixels of the current block and may be adaptively applied depending on the prediction mode, size, shape, and / or whether the reference pixel of the current prediction unit is included in a reference pixel line immediately adjacent to the current block. If the prediction mode of the current block is a mode that does not perform reference pixel filtering, the reference pixel filter may not be applied.

[0056] An interpolation filter is a filter that interpolates and filters prediction samples of the current block, and can be adaptively applied depending on the prediction mode, size, shape, and / or whether the reference pixel of the current prediction unit is included in the reference pixel line immediately adjacent to the current block.

[0057] If the prediction mode of the current block is DC mode, a prediction block can be generated by applying a DC filter.

[0058] In one embodiment, the intra prediction unit (102) may perform intra prediction using a pre-trained neural network (NN) model. For example, the intra prediction unit (102) may derive an intra prediction mode of a block to which the DIMD mode is applied and use a pre-trained neural network model to perform intra prediction.

[0059] The inter prediction unit (103) generates a prediction block using the previously restored reference image stored in the memory (112) and the inter prediction mode and motion information. Here, inter prediction may mean motion prediction or motion compensation.

[0060] The inter prediction unit (103) can set the inter mode of a prediction unit included in an encoding unit to one of the Skip Mode, Merge Mode, and Advanced Motion Vector Prediction (AMVP) mode in order to perform motion prediction and / or motion compensation. Then, the inter prediction unit (103) can perform motion prediction and / or motion compensation for the prediction unit according to the set mode.

[0061] In addition, the inter prediction unit (103) can perform motion prediction and / or motion compensation for a prediction unit by applying the AFFINE mode of sub-PU based prediction, the SbTMVP (Subblock-based Temporal Motion Vector Prediction) mode, and the MMVD (Merge with MVD) mode and the GPM (Geometric Partitioning Mode) mode of PU based prediction based on the inter prediction mode. In addition, the inter prediction unit (103) can perform motion prediction and / or motion compensation for the prediction unit by applying HMVP (History based MVP), PAMVP (Pairwise Average MVP), CIIP (Combined Intra / Inter Prediction), AMVR (Adaptive Motion Vector Resolution), DMVR (Decoder side Motion Vector Refinement), BDOF (Bi-Directional Optical-Flow), PROF (Prediction Refinement With Optical Flow), BCW (Bi-predictive with CU Weights), LIC (Local Illumination Compensation), TM (Template Matching), OBMC (Overlapped Block Motion Compensation), etc. to improve the performance of each mode.

[0062] Here, AFFINE mode can be used in both AMVP and MERGE modes. It is a technique with high encoding efficiency. AFFINE mode can be a prediction mode that uses a 4-parameter affine motion model using two control point motion vectors (CPMV) or a 6-parameter affine motion model using three control point motion vectors. Here, CPMV can be a vector representing an affine motion model of any one of the top left, top right, and bottom left of the current block.

[0063] The motion information may include, for example, a motion vector, a reference picture index, a list 1 prediction flag, a list 0 prediction flag, a half-sample interpolation filter index, a bidirectional prediction weight index, etc.

[0064] According to one embodiment, the inter prediction unit (103) may perform inter prediction using a pre-trained neural network model. For example, the inter prediction unit (103) may synthesize a reference frame using a pre-trained neural network model and perform inter prediction based on the synthesized reference frame.

[0065] A residual block containing residual value information, which is the difference value between the prediction unit generated in the prediction unit (102, 103) and the original block of the prediction unit, can be generated. The generated residual block can be input to the transformation unit (130) and transformed.

[0066] The subtraction unit (104) subtracts the block to be encoded from the prediction block generated by the intra prediction unit (102) or inter prediction unit (103) to generate a residual block of the current block. The residual value (residual block) between the generated prediction block and the original block can be input to the transformation unit (105).

[0067] Additionally, the prediction mode information, motion vector information, etc. used for prediction can be encoded together with the residual value in the entropy encoding unit (107) and transmitted to the decoder. When using a specific encoding mode, it is also possible to encode the original block as is and transmit it to the decoding unit without generating a prediction block through the prediction unit (102, 103).

[0068] The transformation unit (105) can perform a transformation on a residual block including residual data to generate and output a transform coefficient. Here, the transform coefficient may be a coefficient value generated by performing a transformation on the residual block. When the transform skip mode is applied, the transformation unit (105) may also skip the transformation on the residual block.

[0069] The conversion unit (105) can determine a conversion type and a conversion kernel based on at least one of encoding parameters such as the size, color component, and prediction mode of the conversion block, and perform conversion on the conversion block using the determined conversion type and conversion kernel.

[0070] According to one embodiment, the transformation unit (105) may perform transformation on a 4x4 luminance residual block generated as an intra prediction result using a transformation type and transformation kernel according to DST (Discrete Sine Transform), and may perform transformation on the remaining residual blocks using a transformation type and transformation kernel according to DCT (Discrete Cosine Transform).

[0071] According to another embodiment, the transform unit (105) may apply the Multiple Transform Selection (MTS) technology that performs the transform by selectively using several transform types and transform kernels. That is, the transform unit (105) may perform the transform in units of sub-blocks using the Sub-block Transform (SBT) technology. Specifically, the SBT may be applied only to inter-prediction blocks, and the current block may be divided into ½ or ¼ sizes in the vertical or horizontal direction, and the transform may be performed on only one of the blocks. For example, the transform unit (105) may perform the transform on the leftmost or rightmost block among the vertically divided current blocks, and may perform the transform on the topmost or bottommost block among the horizontally divided current blocks.

[0072] According to another embodiment, the transform unit (105) may apply a non-separable primary transform (NSPT) technique that performs the transform by selectively using multiple transform kernels based on the intra prediction mode or the size and / or shape of the block.

[0073] According to another embodiment, the transform unit (105) may apply LFNST (Low Frequency Non-Separable Transform), which is a technology that applies a secondary transform to a residual signal that has been transformed into a frequency domain through DCT or DST. LFNST additionally performs a transform on a 4x4 or 8x8 low-frequency region in the upper left, thereby concentrating the residual coefficients in the upper left.

[0074] The quantization unit (106) can quantize the transform coefficients or residual signals converted to the frequency domain by the transform unit (105) according to a quantization parameter (QP). The quantization parameter can vary depending on the block or the importance of the image. The value produced by the quantization unit (106) can be provided to the dequantization unit (108) and the entropy encoding unit (107).

[0075] The above transformation unit (105) and / or quantization unit (106) may be optionally included in the image encoding device (100). That is, the image encoding device (100) may encode the residual block by performing at least one of transformation or quantization on the residual data of the residual block, or by skipping both transformation and quantization. Even if neither transformation nor quantization is performed in the image encoding device (100), or neither transformation nor quantization is performed, a block that is input to the entropy encoding unit (107) is typically referred to as a transformation block.

[0076] The entropy encoding unit (107) can generate and output a bitstream by performing entropy encoding according to a probability distribution on values ​​output by the quantization unit (106), coding parameter values ​​output during the encoding process, information for decoding an image, etc. Here, the information for decoding an image may include syntax elements, etc.

[0077] Coding parameters may include not only information (flags, indexes, etc.) encoded in an encoding device (100) and signaled to a decoding device (200), such as syntax elements, but also information derived during an encoding process or a decoding process, and may mean information necessary when encoding or decoding an image.

[0078] The entropy encoding unit (107) can encode various information such as coefficient information of a transform block, block type information, prediction mode information, division unit information, prediction unit information, transmission unit information, motion vector information, reference frame information, block interpolation information, and filtering information. The coefficients of a transform block can be encoded in units of sub-blocks within the transform block.

[0079] For encoding the coefficients of a transform block, various syntax elements can be encoded, such as Last_sig, a syntax element indicating the position of the first non-zero coefficient in reverse scan order, Coded_sub_blk_flag, a flag indicating whether there is at least one non-zero coefficient in the subblock, Sig_coeff_flag, a flag indicating whether the coefficient is non-zero, Abs_greater1_flag, a flag indicating whether the absolute value of the coefficient is greater than 1, Abs_greater2_flag, a flag indicating whether the absolute value of the coefficient is greater than 2, and Sign_flag, a flag indicating the sign of the coefficient. The residual value of the coefficient that is not encoded by the above syntax elements alone can be encoded through the syntax element remaining_coeff.

[0080] When entropy coding is applied, a small number of bits are allocated to symbols with a high occurrence probability, and a large number of bits are allocated to symbols with a low occurrence probability, thereby representing the symbols, thereby reducing the size of the bit string for the symbols to be encoded. The input data is entropy encoded. For example, entropy coding can use various coding methods such as Exponential Golomb and CABAC (Context-Adaptive Binary Arithmetic Coding).

[0081] The inverse quantization unit (108) and the inverse transformation unit (109) can inverse quantize the values ​​quantized in the quantization unit (106) and inversely transform the values ​​transformed in the transformation unit (105). The residual values ​​generated in the inverse quantization unit (108) and the inverse transformation unit (109) can be combined with the prediction units predicted through the motion estimation unit, motion compensation unit, and intra prediction unit (102) included in the prediction unit (102, 103) to generate a reconstructed block. The addition unit (110) adds the prediction blocks generated in the prediction units (102, 103) and the residual blocks generated through the inverse transformation unit (109) to generate a reconstructed block.

[0082] The filter unit (111) can apply a deblocking filter, a sample adaptive offset (SAO), an adaptive loop filter (ALF), a bilateral filter (BIF), and a LMCS (Luma Mapping with Chroma Scaling) as a filtering technique, in whole or in part, to a restored sample, restored block, or restored image.

[0083] A deblocking filter can remove block distortion caused by boundaries between blocks in a reconstructed picture. To determine whether to perform deblocking, a deblocking filter can be applied to the current block based on the pixels contained in several columns or rows within the block. When applying a deblocking filter to a block, a strong filter or a weak filter can be applied depending on the required deblocking filtering strength. Furthermore, when applying a deblocking filter, horizontal and vertical filtering can be processed in parallel when performing vertical and horizontal filtering.

[0084] Sample adaptive offset may be a method of correcting the offset from the original image on a sample basis for an image on which deblocking has been performed. The filter unit (111) may use a method of dividing the samples included in the image into a certain number of regions, determining the regions on which the offset is to be performed, and applying the offset to the regions, or a method of applying the offset by considering edge information of each sample. Here, the sample adaptive offset may be at least one of a general sample adaptive offset, a bilateral filter, and a cross-component sample adaptive offset (CCSAO).

[0085] Adaptive Loop Filtering (ALF) can be performed based on the comparison of the filtered restored image with the original image. After dividing the pixels included in the image into predetermined groups, a filter to be applied to each group can be determined, and filtering can be performed differentially for each group. Information regarding whether to apply ALF can be transmitted by luminance signal for each coding unit (CU), and the shape and filter coefficients of the ALF filter to be applied can vary depending on each block. Furthermore, an ALF filter of the same form (fixed form) can be applied regardless of the characteristics of the target block.

[0086] An adaptive loop filter can perform filtering based on a comparison between a reconstructed image and the original image. By dividing the samples contained in the image into predetermined groups and determining the filter to be applied to each group, filtering can be performed differentially for each group. Information regarding whether to apply an adaptive loop filter can be signaled for each coding unit (CU), and the shape and filter coefficients of the adaptive loop filter applied to each block can vary.

[0087] According to one embodiment, the filter unit (111) may filter all or part of a restored sample, restored block, or restored image using a pre-trained neural network model. Specifically, the filter unit (111) may apply an adaptive loop filter to all or part of a restored sample, restored block, or restored image using a pre-trained neural network model.

[0088] The memory (112) can store a restored block or picture produced through the filter unit (111). The memory (112) can include a reference picture buffer. In addition, the restored block or picture stored in the memory (112) can be provided to the prediction unit (102, 103) when performing inter prediction.

[0089] Next, an image decoding device according to one embodiment of the present invention will be described with reference to the drawings.

[0090] FIG. 2 is a block diagram showing an image decoding device (200) according to one embodiment of the present invention.

[0091] Referring to FIG. 2, the image decoding device (200) may include an entropy decoding unit (201), an inverse quantization unit (202), an inverse transformation unit (203), a prediction unit (204, 205), an addition unit (206), a filter unit (207), and a memory (208).

[0092] The image decoding device (200) can receive a bitstream output by the image encoding device (100). The image decoding device (200) can receive a bitstream stored in a computer-readable recording medium, or can receive a bitstream streamed through a wired / wireless transmission medium. The image decoding device (200) can decode the bitstream to generate a restored image or a decoded image, and can output the restored image or the decoded image.

[0093] The entropy decoding unit (201) can generate symbols by performing entropy decoding according to a probability distribution for the bitstream. The generated symbols may include symbols in the form of quantized levels. Here, the entropy decoding method may be the reverse process of the entropy encoding method described above.

[0094] The entropy decoding unit (201) can change a one-dimensional vector-shaped coefficient into a two-dimensional block-shaped coefficient through a transform coefficient scanning method to decode a transform coefficient level (quantized level).

[0095] The entropy decoding unit (201) can perform entropy decoding in a procedure opposite to that of the entropy encoding unit (107) of the video encoding device (100). For example, various methods such as Exponential Golomb and CABAC (Context-Adaptive Binary Arithmetic Coding) can be applied in response to the method performed in the video encoder.

[0096] The entropy decoding unit (201) can decode and obtain various information such as coefficient information of the transform block as described above, block type information, prediction mode information, division unit information, prediction unit information, transmission unit information, motion vector information, reference frame information, block interpolation information, and filtering information.

[0097] The inverse quantization unit (202) performs inverse quantization on a quantized transform block to generate a transform block. It operates substantially the same as the inverse quantization unit (108) of Fig. 1.

[0098] The inverse transform unit (203) performs an inverse transform on the transform block to generate a residual block. At this time, the transform method can be determined based on information regarding the prediction method (inter or intra prediction), the size and / or shape of the block, the intra prediction mode, etc. It operates substantially the same as the inverse transform unit (109) of FIG. 1.

[0099] According to one embodiment, the inverse transform unit (203) may perform inverse transform using a transform type and transform kernel according to DST (Discrete Sine Transform) on the transform coefficient levels of the 4x4 luminance component generated as an intra prediction result, and may perform inverse transform using a transform type and transform kernel according to DCT (Discrete Cosine Transform) on the remaining transform coefficient levels.

[0100] According to another embodiment, the inverse transform unit (203) may apply MTS (Multiple Transform Selection) technology to perform transformation by selectively using multiple transformation kernels.

[0101] According to another embodiment, the inverse transform unit (203) may apply LFNST (Low Frequency Non-Separable Transform), which is a technology that applies a secondary inverse transform to a transform coefficient level inversely transformed through a DCT or DST-based transform type and transform kernel.

[0102] According to another embodiment, the inverse transform unit (203) may apply a non separable primary transform (NSPT) technique that performs inverse transform by selectively using a transform kernel based on the intra prediction mode or the size and / or shape of the block.

[0103] The prediction unit (204, 205) can generate a prediction block based on the prediction block generation related information provided by the entropy decoding unit (201) and the previously decoded block or picture information provided by the memory (208).

[0104] The prediction unit (204, 205) may include an intra prediction unit (204) and an inter prediction unit (205). The prediction unit (204, 205) may receive various information such as prediction unit information input from the entropy decoding unit (201), prediction mode information of the intra prediction method, and motion prediction-related information of the inter prediction method, and may distinguish a prediction unit from a current encoding unit and determine a prediction mode of the prediction unit.

[0105] The intra prediction unit (204) can generate a prediction block of the current block based on the intra prediction mode of the current block and reference pixel information around the current block, which is pixel information within the current picture.

[0106] The intra prediction unit (204) can generate a prediction block based on reference pixel information surrounding the current block, which is pixel information within the current picture. The intra prediction unit (204) can use multiple reference pixel lines for intra prediction. When multiple reference pixel lines are available, the intra prediction unit (204) can obtain information indicating a reference pixel line used for intra prediction from among the multiple reference pixel lines.

[0107] The intra prediction mode used for intra prediction may be a directional prediction mode or a non-directional mode. Furthermore, the mode for predicting luminance information may be different from the mode for predicting chrominance information, and the intra prediction mode information of the luminance component block or the predicted luminance signal information may be utilized to predict chrominance information.

[0108] According to one embodiment, the intra prediction unit (204) can perform intra prediction using a pre-trained neural network model. The intra prediction unit (204) can perform intra prediction using the same neural network model as the intra prediction unit (102) of FIG. 1.

[0109] The intra prediction unit (204) operates substantially the same as the intra prediction unit (102) of FIG. 1.

[0110] The inter prediction unit (205) may perform inter prediction on the current prediction unit based on information included in at least one picture among the previous picture or the subsequent picture of the current picture including the current prediction unit, using information necessary for inter prediction of the current prediction unit provided by the image encoding device (100). Alternatively, inter prediction may be performed based on information of some pre-restored area within the current picture including the current prediction unit.

[0111] The inter prediction unit (205) can set the inter mode of a prediction unit included in an encoding unit to one of the skip mode, merge mode, and advanced motion vector prediction (AMVP) mode in order to perform motion prediction and / or motion compensation. In addition, the inter prediction unit (205) can perform motion compensation on the prediction unit according to the set mode.

[0112] In addition, the inter prediction unit (205) can perform motion compensation for the prediction unit by applying the AFFINE mode of sub-PU based prediction, the SbTMVP (Subblock-based Temporal Motion Vector Prediction) mode, and the MMVD (Merge with MVD) mode and the GPM (Geometric Partitioning Mode) mode of PU based prediction based on the inter prediction mode. In addition, the inter prediction unit (205) can perform motion compensation for the prediction unit by applying the HMVP (History based MVP), the PAMVP (Pairwise Average MVP), the CIIP (Combined Intra / Inter Prediction), the AMVR (Adaptive Motion Vector Resolution), the BDOF (Bi-Directional Optical-Flow), the BCW (Bi-predictive with CU Weights), the LIC (Local Illumination Compensation), the TM (Template Matching), the OBMC (Overlapped Block Motion Compensation), etc. to improve the performance of each mode.

[0113] The motion information may include, for example, a motion vector, a reference picture index, a list 1 prediction flag, a list 0 prediction flag, a half-sample interpolation filter index, a bidirectional prediction weight index, etc.

[0114] According to one embodiment, the inter prediction unit (205) can perform inter prediction using a pre-trained neural network model. The inter prediction unit (205) can perform inter prediction using the same neural network model as the inter prediction unit (103) of FIG. 1.

[0115] The inter prediction unit (205) can operate substantially the same as the inter prediction unit (103) of FIG. 1.

[0116] The addition unit (206) adds the prediction block generated by the intra prediction unit (204) or inter prediction unit (205) and the residual block generated by the inverse transformation unit (203) to generate a restored block. It operates substantially the same as the addition unit (110) of Fig. 1.

[0117] The filter unit (207) can reduce various types of noise occurring in restored blocks. The filter unit (207) can include a deblocking filter, a sample adaptive offset, an adaptive loop filter, a bilateral filter, and an LMCS.

[0118] The filter unit (207) can receive information on whether each filter is applied, information on the strength of the filter, etc. from the image encoding device (100). The filter unit (207) of the image decoding device (200) can receive filter-related information provided from the image encoding device (100) and perform filtering on the corresponding block in the image decoding device (200).

[0119] According to one embodiment, the filter unit (207) may filter all or part of a restored sample, restored block, or restored image using a pre-trained neural network model. Specifically, the filter unit (207) may apply an adaptive loop filter to all or part of a restored sample, restored block, or restored image using a pre-trained neural network model.

[0120] The filter unit (207) can operate substantially the same as the filter unit (111) of FIG. 1.

[0121] The memory (208) can store the restoration block generated by the addition unit (206). For example, the memory (208) can include a reference picture buffer. The memory (208) can operate substantially the same as the memory (112) of FIG. 1.

[0122]

[0123] FIG. 3 is a diagram schematically showing a video coding system to which the present invention can be applied.

[0124] A video coding system according to one embodiment may include an encoding device (10) and a decoding device (20). The encoding device (10) may transmit encoded video and / or image information or data to the decoding device (20) in the form of a file or streaming through a digital storage medium or a network.

[0125] An encoding device (10) according to one embodiment may include an image generating unit (11), an encoding unit (12), and a transmission unit (13). A decoding device (20) according to one embodiment may include a receiving unit (21), a decoding unit (22), and an image reproducing unit (23). The encoding unit (12) may be referred to as a video / image encoding unit, and the decoding unit (22) may be referred to as a video / image decoding unit. The transmission unit (13) may be included in the encoding unit (12). The receiving unit (21) may be included in the decoding unit (22). The image reproducing unit (23) may include a display unit, and the display unit may be configured as a separate device or an external component.

[0126] The image generation unit (11) can obtain a video / image through a process of capturing, synthesizing, or generating a video / image. The image generation unit (11) can include a video / image capture device and / or a video / image generation device. The video / image capture device can include, for example, one or more cameras, a video / image archive including previously captured video / images, etc. The video / image generation device can include, for example, a computer, a tablet, a smartphone, etc., and can (electronically) generate a video / image. For example, a virtual video / image can be generated through a computer, etc., in which case the video / image capture process can be replaced with a process of generating related data.

[0127] The encoding unit (12) can encode the input video / image. The encoding unit (12) can perform a series of procedures such as prediction, transformation, and quantization for compression and encoding efficiency. The encoding unit (12) can output encoded data (encoded video / image information) in the form of a bitstream. The detailed configuration of the encoding unit (12) can be configured in the same manner as the encoding device (100) of FIG. 1 described above.

[0128] The transmission unit (13) can transmit encoded video / image information or data output in the form of a bitstream to the reception unit (21) of the decoding device (20) via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (13) can include an element for generating a media file through a predetermined file format and can include an element for transmission via a broadcasting / communication network. The reception unit (21) can extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit (22).

[0129] The decoding unit (22) can decode video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding unit (12). The detailed configuration of the decoding unit (22) can be configured identically to the decoding device (200) of FIG. 2 described above.

[0130] The image playback unit (23) can render decrypted video / images. The rendered video / images can be displayed through the display unit.

[0131]

[0132] FIG. 4 is a drawing exemplarily showing a content streaming system to which an embodiment according to the present invention can be applied.

[0133] As illustrated in FIG. 4, a content streaming system to which an embodiment of the present invention is applied may largely include a multimedia input device, a media storage, an encoding server, a streaming server, a web server, and a user device.

[0134] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, CCTVs, etc. into digital data, creates a bitstream, and transmits it to the streaming server. Alternatively, the encoding server compresses content already stored in a media storage into digital data, creates a bitstream, and transmits it to the streaming server.

[0135] As another example, if multimedia input devices such as smartphones, cameras, CCTVs, etc. directly generate bitstreams, the encoding server may be omitted.

[0136] The above bitstream can be generated by a video encoding method and / or a video encoding device to which an embodiment of the present invention is applied, and the streaming server can temporarily or non-temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0137] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server can act as an intermediary to inform the user of available services. When the user device requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server can transmit multimedia data to the user device. At this time, the content streaming system may include a separate control server, and in this case, the control server may play a role in controlling commands / responses between each device within the content streaming system.

[0138] The streaming server can receive content from a media repository and / or encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0139] Examples of the user devices may include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), digital TVs, desktop computers, digital signage, etc.

[0140] Each server within the above content streaming system can be operated as a distributed server, in which case data received from each server can be processed in a distributed manner.

[0141]

[0142] FIG. 5 is a diagram illustrating a neural network-based intra prediction method according to an embodiment of the present disclosure. Here, a neural network-based intra prediction block may refer to a prediction block generated by neural network-based intra prediction.

[0143] Referring to Fig. 5, the neural network-based intra prediction method may be a method of performing intra prediction for the current block using a neural network model. The neural network-based intra prediction method may use a neural network model corresponding to the size of the current block. Here, the neural network model may be one of models of sizes 4×4, 8×4, 16×4, 32×4, 8×8, 16×8, and 16×16. The neural network model corresponding to a block of size w×h can be referred to as a neural network model It may be a model that takes as input a context derived from restored pixels adjacent to the current block (501) and outputs intra prediction related information of the current block.

[0144] The context (X) can be generated using the restored pixels adjacent to the left and / or upper side of the current block (501). Then, the generated context can be preprocessed. Then, the preprocessed context ( ) can be input to the neural network for block prediction. The neural network model for block prediction is the initial prediction value for the block ( ), kernel index (grpIdx) of LFNST (Low-Frequency Non-Separable Transform) i ) and intra prediction mode index (repIdx).

[0145] Here, the kernel index (grpIdx i ) can indicate whether to transpose the primary transform coefficient when lfnstIdx of the primary transform coefficient generated by applying DCT in the vertical and horizontal directions to the residual derived from the neural network-based intra prediction result is i.

[0146] Here, the intra prediction mode index (repIdx) can have a value of 0 to 66, and can indicate one of the non-directional intra prediction modes and / or the directional intra prediction modes supported by the video codec standard.

[0147] And, the prediction block (503) for the current block has an initial prediction value ( ) can be generated by applying postprocessing.

[0148] Neural network-based intra prediction methods can be prediction methods that apply inference based on a model generated through training. The model generated through training can be applied to the encoder and decoder. In neural network-based intra prediction methods, both the encoder and decoder generate identical output values ​​using the model generated through training.

[0149] A model generated through training must produce consistent output values ​​regardless of the operating platform. Therefore, the model generated through training can perform operations based on integer values. Furthermore, both the weights in the inference model and the values ​​generated during the propagation process are integer values.

[0150] The input values ​​of the model generated through training used in the encoder and decoder may include the reconstructed image around the current block, the predicted image of the current block, the prediction mode of the current block, the QP value, the slice type, the IPB block type, the size of the current block, the total number of pixels of the current block, the width of the current block, the height of the current block, the tree type of the current block (e.g., whether a single tree or a dual tree is used), etc.

[0151] The output values ​​derived from the model generated through training may include a prediction value for coding the current block, a transformation block kernel information value for transformation applied to the current block, an intra mode information value of the current block, an information value indicating an intra prediction method, etc.

[0152] The models generated through training can be defined separately for the luma component and the chroma component, or the models generated through training can be defined identically for the luma component and the chroma component.

[0153] Here, the neural network-based intra prediction method can be determined based on the size of the current block and the signaled information.

[0154] If the current block size corresponds to one of the predetermined size candidates, an nnFlag may be encoded, indicating whether a neural network-based intra prediction method is to be applied to the current block. Here, if the value of nnFlag is 1, a neural network-based intra prediction method may be applied to the current block. Conversely, if the value of nnFlag is 0, a signal processing-based intra prediction method may be applied to the current block.

[0155] Neural network-based intra prediction methods can utilize models generated through training. The surrounding restored image values, which can be used as input values ​​for the model, are organized in an L-shape centered around the current coding block. The referenced restored pixels can be determined based on the size of the current coding block. For example, if the size of the current coding block is w * h, the referenced pixel area may include the restored pixels at the bottom left, left edge, top left, top right, and bottom right of the current coding block.

[0156] The restored pixels used to derive context in a neural network-based intra prediction method can be as follows.

[0157]

[0158] FIG. 6 is a diagram for explaining restored pixels used to derive context in a neural network-based intra prediction method according to an embodiment of the present disclosure.

[0159] Referring to FIG. 6, the restored pixels used to derive the context may include samples of h_r1 x (w + w_r2 + w_r3) at the top of the current block, samples of (h + h_r2 + h_r3) x w_r1 at the left of the current block, and samples of h_r1 x w_r1 at the top left of the current block.

[0160] The value of w_r1 can be equal to w. Alternatively, if the value of w is less than the threshold, the value of w_r1 can be set to w / 2. For example, if the threshold is 8, the value of w_r1 can be set to 4. Alternatively, the value of w_r1 can always be set to a default value regardless of the size of w. For example, the value of w_r1 can always be fixedly set to a referenceable 4-pixel line. Alternatively, the value of w_r1 can be set to a smaller value among the values ​​of w and h (w_r1=min(w, h)) considering the value of w, the total number of pixels in the block, and the pixel lines that can be referenced from the surrounding blocks. Alternatively, the value of w_r1 can be set to 0.

[0161] The value of w_r2 can be equal to w. Alternatively, if the value of w is less than the threshold, the value of w_r2 can be set to w / 2. For example, if the threshold is 8, the value of w_r2 can be set to 4. Alternatively, the value of w_r2 can always be set to a default value regardless of the size of w. For example, the value of w_r2 can always be fixed to a referenceable 4-pixel line. Alternatively, the value of w_r2 can be set to a smaller value among the values ​​of w and h (w_r2=min(w, h)) considering the value of w, the total number of pixels in the block, and the pixel lines that can be referenced from the surrounding blocks. Alternatively, the value of w_r2 can be set to 0.

[0162] The value of w_r3 can be equal to w. Alternatively, if the value of w is less than the threshold, the value of w_r3 can be set to w / 2. For example, if the threshold is 8, the value of w_r3 can be set to 4. Alternatively, the value of w_r3 can always be set to a default value regardless of the size of w. For example, the value of w_r3 can always be fixed to a referenceable 4-pixel line. Alternatively, the value of w_r3 can be set to a smaller value among the values ​​of w and h (w_r3=min(w, h)) considering the value of w, the total number of pixels in the block, and the pixel lines that can be referenced from the surrounding blocks. Alternatively, the value of w_r3 can be set to 0.

[0163] The value of h_r1 can be equal to h. Alternatively, if the value of h is less than the threshold, the value of h_r1 can be set to h / 2. For example, if the threshold is 8, the value of h_r1 can be set to 4. Alternatively, the value of h_r1 can always be set to a default value regardless of the size of h. For example, the value of h_r1 can always be fixedly set to a referenceable 4-pixel line. Alternatively, the value of h_r1 can be set to a smaller value among the values ​​of w and h (h_r1=min(w, h)) considering the value of h, the total number of pixels in the block, and the pixel lines that can be referenced from the surrounding blocks. Alternatively, the value of h_r1 can be set to 0.

[0164] The value of h_r2 can be equal to h. Alternatively, if the value of h is less than the threshold, the value of h_r2 can be set to h / 2. For example, if the threshold is 8, the value of h_r2 can be set to 4. Alternatively, the value of h_r2 can always be set to a default value regardless of the size of h. For example, the value of h_r2 can always be fixed to a referenceable 4-pixel line. Alternatively, the value of h_r2 can be set to a smaller value among the values ​​of w and h (h_r2=min(w, h)) considering the value of h, the total number of pixels in the block, and the pixel lines that can be referenced from the surrounding blocks. Alternatively, the value of h_r2 can be set to 0.

[0165] The value of h_r3 can be equal to h. Alternatively, if the value of h is less than the threshold, the value of h_r3 can be set to h / 2. For example, if the threshold is 8, the value of h_r3 can be set to 4. Alternatively, the value of h_r3 can always be set to a default value regardless of the size of h. For example, the value of h_r3 can always be fixed to a referenceable 4-pixel line. Alternatively, the value of h_r3 can be set to a smaller value among the values ​​of w and h (h_r3=min(w, h)) considering the value of h, the total number of pixels in the block, and the pixel lines that can be referenced from the surrounding blocks. Alternatively, the value of h_r3 can be set to 0.

[0166]

[0167] A neural network-based intra prediction method can utilize a model generated through training. The input value of the model may include a prediction value for the current block. In one embodiment, the prediction value for the current block may be an intra prediction value based on one of the intra prediction modes. For example, an encoder and decoder performing a neural network-based intra prediction method may input a prediction value generated using a planar mode into the model.

[0168] In another embodiment, the value of the current block prediction may be an intra prediction value based on an intra prediction mode derived through Decoder-side intra mode derivation (DIMD). For example, the value of the current block prediction may be an intra prediction value based on one intra prediction mode having the highest probability among the intra prediction modes derived through DIMD. Alternatively, the value of the current block prediction may be a prediction value obtained by weighting the intra prediction values ​​based on two intra prediction modes having the highest probabilities among the intra prediction modes derived through DIMD. An encoder and decoder that perform a neural network-based intra prediction method may input a prediction value generated using the intra prediction mode derived through DIMD into the model.

[0169] In another embodiment, the value of the current block prediction may be an intra prediction value derived through an intra template matching (IntraTMP) method. For example, the value of the current block prediction may be a single intra prediction value derived through an intra template matching method. Alternatively, the value of the current block prediction may be a prediction value obtained by weighting two intra prediction values ​​derived through an intra template matching method. An encoder and decoder that perform a neural network-based intra prediction method may input the intra prediction value derived through the intra template matching method into the model.

[0170] In another embodiment, the value of the current block prediction may be an intra prediction value derived through an intra block copy (IBC) method. For example, the value of the current block prediction may be a single intra prediction value derived through the intra block copy method. Alternatively, the value of the current block prediction may be a prediction value obtained by weighting two intra prediction values ​​derived through the intra block copy method. An encoder and decoder that perform a neural network-based intra prediction method may input the intra prediction value derived through the intra block copy method into the model.

[0171] In another embodiment, the value of the current block prediction may be an intra prediction value based on an intra prediction mode derived through Occurrence-based Intra Coding (OBIC). For example, the value of the current block prediction may be an intra prediction value based on one intra prediction mode having the highest frequency among the intra prediction modes derived through OBIC. Alternatively, the value of the current block prediction may be a prediction value obtained by weighting the intra prediction values ​​based on two intra prediction modes having the highest frequencies among the intra prediction modes derived through OBIC. An encoder and decoder that perform a neural network-based intra prediction method may input a prediction value generated using an intra prediction mode derived through OBIC into a model.

[0172]

[0173] In neural network-based intra prediction methods, the model used for inference can be generated by considering the width, height, size, shape, aspect ratio, etc. of the current coding block. For example, a model for a neural network-based intra prediction method can be generated and used for each block size supported by a video encoding / decoding method.

[0174] Alternatively, models of neural network-based intra prediction methods can be generated and used for coding blocks of a certain size. That is, coding blocks of a certain size can share models of neural network-based intra prediction methods with coding blocks of a different size.

[0175] In one embodiment, when the shape of the coding block is rectangular, the input value can be transposed and a model corresponding to the transposed input value can be used. For example, when a model for a 16x8 block and a model for an 8x16 block are shared and used, the 16x8 block can use the input as is to generate an output, and the 8x16 block can transpose the input value to change it to a 16x8 input value and then use the 16x8 model.

[0176] In another embodiment, if a model corresponding to a coding block of a certain size does not exist, scaling up or down can be applied to the model's input values. Furthermore, scaling up or down can also be applied to the model's output values.

[0177] For example, if the maximum model size is 16x16 and the current block size is 64x64, downsampling by 4 times can be applied to the input value. In this case, the size of the downsampled input value is 16x16, so the downsampled input value can be input to the inference model. Then, the 16x16 output value of the inference model can be upsampled to generate a prediction value of size 64x64.

[0178] For another example, if the minimum model size is 32x32 and the current block size is 16x16, upsampling by a factor of 2 can be applied to the input value. In this case, the size of the upsampled input value is 32x32, so the upsampled input value can be input to the inference model. Then, the 32x32 output value of the inference model can be downsampled to generate a prediction value of size 16x16.

[0179] Signal processing-based filters can be used for upsampling and downsampling. These signal processing-based filters can include at least one of a DCT-based filter, a Sinc function-based filter, a bilinear-based filter, and a Lantzos filter. Alternatively, techniques such as neural network-based filters and pooling layers can be used independently or in combination with an inference model for upsampling and downsampling.

[0180] Alternatively, if there is no model corresponding to a coding block of a certain size, the coding block of a certain size may share a model of a neural network-based intra prediction method with coding blocks of other sizes.

[0181] According to one embodiment, if a model corresponding to a coding block of a specific size does not exist, multiple models can be merged to create a single model, and a neural network-based intra prediction method can be performed using the created model.

[0182] For example, if the maximum size of the model is 32x32 and the size of the current block is 64x64, four 32x32 models can be merged to create an inference model for a 64x64-sized block. Then, a prediction value of the current block can be generated using the inference model for the generated 64x64-sized block. As another example, if the maximum size of the model is 32x32 and the size of the current block is 64x32, two 32x32 models can be merged to create an inference model for a 64x32-sized block. Then, a prediction value of the current block can be generated using the inference model for the generated 64x32-sized block.

[0183] In another embodiment, if a model corresponding to a coding block of a specific size does not exist, a model of a size smaller than the specific size can be used to generate a prediction value for the coding block.

[0184] For example, if the maximum size of the model is 16x16 and the size of the current block is 32x32, an input value of size 16x16 can be extracted from the current block and input to the model of size 16x16. Then, the output value of size 16x16 output by the model can be placed in the area of ​​size 16x16 of the prediction block of the current block. In this case, the last value of the area of ​​size 16x16 of the current prediction block can be expanded and padded to the remaining area. Alternatively, the prediction value of an area other than the area of ​​size 16x16 of the current prediction block can be determined as a DC prediction value or a prediction value generated through a predetermined intra prediction mode.

[0185] Alternatively, the 16x16 sized output value output by the model may be blended with a prediction value generated by another intra prediction method, and the blended prediction value may be placed in a 16x16 sized area of ​​the prediction block of the current block. Then, the last value of the 16x16 sized area of ​​the current prediction block may be expanded and padded into the remaining area. Alternatively, the prediction value of an area other than the 16x16 sized area of ​​the current prediction block may be determined as a DC prediction value or a prediction value generated through a predetermined intra prediction mode.

[0186]

[0187] If the current block location, to which a neural network-based intra prediction method is applied, falls within certain boundary conditions, special techniques for neural network-based intra prediction may be required. This is because, if the current block location falls within certain boundary conditions, referenced pixels adjacent to the current block may not be available. In other words, if the current block location falls within certain boundary conditions, available input values ​​for the model used in the neural network-based intra prediction method may not exist.

[0188] In such cases, the application of a neural network-based intra prediction method may be determined explicitly or implicitly. Alternatively, the application of a modified neural network-based intra prediction method may be determined explicitly or implicitly.

[0189] Here, the boundary conditions may include cases where the location of the current block is as follows or the input values ​​for the neural network-based intra prediction method of the current block are as follows.

[0190] For example, if the current block's location is at the left, top-left, bottom-left, top, right, top-right, or bottom-right of a frame, slice, picture, tile, etc., the current block's location may correspond to a boundary condition. Alternatively, if the current block's location is adjacent to a CTU or CU that is not managed by a buffer, the current block's location may correspond to a boundary condition. Additionally, if no reference pixels adjacent to the current block are available, the current block may correspond to a boundary condition.

[0191]

[0192] The location of the current block corresponding to the boundary condition and the method of applying the neural network-based intra prediction method to the current block may be as described below.

[0193]

[0194] FIG. 7 is a diagram for explaining a current block corresponding to a boundary condition in a neural network-based intra prediction method according to an embodiment of the present disclosure.

[0195] Referring to Fig. 7, the current block is located at the left end of the slice, and the location of the current block may correspond to a boundary condition. In this case, the samples of (h + h_r2 + h_r3) x w_r1 on the left side of the current block and h_r1 x w_r1 on the upper left side of the current block are samples outside the slice containing the current block, and may be unavailable samples. Therefore, the reconstructed pixels used to derive the context may only include the samples of h_r1 x (w + w_r2 + w_r3) on the upper side of the current block.

[0196] On the other hand, if the current block is located at the top of the slice, the samples of h_r1 x (w + w_r2 + w_r3) at the top of the current block and the samples of h_r1 x w_r1 at the top-left of the current block are samples outside the slice containing the current block, and may be unavailable samples. Therefore, the reconstructed pixels used to derive the context may only include the samples of (h + h_r2 + h_r3) x w_r1 at the left of the current block.

[0197] If the location of the current block corresponds to a boundary condition as above, at least some of the reference samples adjacent to the current block may not be used as input values.

[0198] According to one embodiment, if the location of the current block corresponds to a boundary condition such that at least one reference sample adjacent to the current block is unavailable, the encoder may implicitly determine not to use a neural network-based intra prediction method. Furthermore, encoding information regarding the neural network-based intra prediction method may be omitted. Furthermore, if at least one reference sample adjacent to the current block is unavailable, the decoder may implicitly determine not to use a neural network-based intra prediction method.

[0199] In another embodiment, if the location of the current block corresponds to a boundary condition and at least one reference sample adjacent to the current block is unavailable, a neural network-based intra prediction method that utilizes only referential pixels after excluding non-referential restoration pixels may be used. Here, the neural network-based intra prediction method that utilizes only referential pixels may be used only when the number of referential pixels is greater than a predetermined number.

[0200] For example, in a model requiring a total of 40 reference pixel values ​​as input values, if the number of referenceable pixels is less than 20, a neural network-based intra prediction method that uses only the referenceable pixels may be implicitly determined not to be used. On the other hand, if the number of referenceable pixels is 20 or more, a neural network-based intra prediction method that uses the referenceable pixels may be used.

[0201] Here, if the model can derive an output value using 20 input values, the model can derive an output value using 20 referenceable pixel values.

[0202] On the other hand, if the model can derive output values ​​using 40 input values, 40 input values ​​are generated using 20 referenceable pixel values, and the model can derive output values ​​using the generated 40 input values.

[0203] Here, additionally generated input values ​​can be generated by copying adjacent pixels of unavailable reference pixels, upsampling available reference pixels, adding DC values ​​of available pixels, copying values ​​of available reference pixels, applying transposition to the current block to expand pixels adjacent to the applied current block, performing template matching based on available reference pixels and utilizing samples of the corresponding template, etc.

[0204] If the current block's location falls within a boundary condition but a wrap-around pixel is available, the wrap-around pixel can be used as an input value in place of the unavailable reference pixel. For example, if the current block is located at the left end of a given region and a wrap-around pixel is available, the wrap-around pixel located on the right side of the same height can be used as an input value for a neural network-based intra prediction method instead of the left-side reconstructed image value required by the model.

[0205] If multiple variants of neural network-based intra prediction methods are available, the encoder can select one of the variants based on rate-distortion optimization (RDO). Furthermore, the encoder can signal information indicating the neural network-based intra prediction method to the decoder via the bitstream.

[0206]

[0207] Using a neural network-based intra prediction method, a predicted value for the current block can be derived. The predicted value obtained using the neural network-based intra prediction method can itself be used as the intra prediction value for the current block.

[0208] Alternatively, according to a neural network-based intra prediction method, the accuracy of the intra prediction value derived through the neural network and the intra prediction value using the intra prediction mode value derived through the neural network can be evaluated, and the intra prediction value with higher accuracy can be used. Here, information about the intra prediction value with higher accuracy among the intra prediction value derived through the neural network and the intra prediction value using the intra prediction mode value derived through the neural network can be determined explicitly or implicitly. When information about the intra prediction value with higher accuracy is explicitly determined, the encoder can signal information about the intra prediction value with higher accuracy to the decoder through the bitstream.

[0209] For example, if the accuracy of the intra prediction value derived through the neural network is low, the intra prediction value derived through the neural network is not used, and the intra prediction value using the intra prediction mode value derived through the neural network can be used as the intra prediction value according to the neural network-based intra prediction method.

[0210] Alternatively, an intra prediction value with higher accuracy may be used among the intra prediction values ​​derived through a neural network and the intra prediction mode values ​​derived through a neural network. Information regarding the intra prediction value with higher accuracy may be signaled to the decoder.

[0211] Alternatively, the intra prediction value derived through the neural network and the intra prediction mode value derived through the neural network can be blended to be used as the intra prediction value according to the neural network-based intra prediction method. Here, the weight of the intra prediction value derived through the neural network and the intra prediction mode value derived through the neural network can be 1:1. Alternatively, the weight of the intra prediction value derived through the neural network and the intra prediction mode value derived through the neural network can be set explicitly or implicitly. If the weight is set explicitly, information about the weight can be signaled to the decoder.

[0212]

[0213] According to a neural network-based intra prediction method, a prediction value for the current block can be derived. In this case, the prediction value for the current block can be arranged according to the shape of the predicted block. Furthermore, the range of the prediction value for the current block can be changed based on the bit-depth range. The average value of the reference can be added to the prediction value of the current block whose range has been changed, and clipping can be performed according to the bit depth of the current pixel to derive the prediction value for the current block.

[0214] According to the neural network-based intra prediction method, the input value can be boosted using the internal operation bit depth to secure the precision of the prediction value. Here, the internal operation bit depth can be set to be the same as the bit depth used in the transform and motion compensation process of the codec, such as 16 bits, 32 bits, 64 bits, 128 bits, or 256 bits. Alternatively, in order to improve the prediction precision of the neural network-based intra prediction method, the internal operation bit depth can be set to 16 bits, 32 bits, 64 bits, 128 bits, or 256 bits.

[0215]

[0216] When an output value is derived based on a boosted input value using an internal operation bit depth, the domain of the output value can be reduced to within the bit depth range. Here, a shift operation can be utilized to reduce the domain of the output value to within the bit depth range. For example, if the domain of an input value with a 10-bit depth is boosted using an internal operation bit depth of 16 bits, a 6-bit left shift operation can be applied to the output predicted value. Therefore, the bit depth of the final predicted value can be changed to 10 bits. In this case, to improve the precision of the final predicted value, a 3-bit value, which is half of the 6 bits, can be added to the output predicted value before applying the 6-bit left shift operation.

[0217] When output values ​​are derived based on boosted input values ​​using internal computation bit depth, clipping can be applied to the output values ​​even if the domain of the output values ​​is changed to fall within the bit depth range. As a result, the final predicted value can be set to have a value within a given bit range. The output value is 0 to 2. b It can be clipped to have a value of -1. 2 b The value of -1 can be changed to any value in the domain range depending on the intended use.

[0218] For example, considering the video range of the format specified by ITU-R, etc., the final clipping is 2 b - It can be performed to output a value less than 1. In this case, the range of clipping can be set according to the format video range used. According to another example, the range of the video can be set to 2 depending on the characteristics of the video, such as SDR, HDR video, etc. b It can be performed to output a value less than -1. In this case, clipping can be performed according to the image characteristics.

[0219]

[0220] According to a neural network-based intra prediction method, an intra prediction mode value for the current block can be derived. The intra prediction mode value derived by the neural network-based intra prediction method can be utilized to derive an intra prediction value for the current block.

[0221] Information related to the intra prediction mode derived by the neural network-based intra prediction method can be utilized as information related to the intra prediction mode of the current block when using the most probable mode (MPM) in the intra prediction process of surrounding blocks in the future.

[0222] When applying a neural network-based intra prediction method to the current block, the intra prediction mode value for the current block may not exist. Therefore, when using MPM in the intra prediction process for subsequent blocks adjacent to the current block, intra prediction mode information to reference from the current block may not exist.

[0223] In this case, according to one embodiment, the MPM of the subsequent block may include a neural network-based intra prediction method. Alternatively, according to another embodiment, the candidate mode of the MPM of the subsequent block may include an intra prediction mode derived by the neural network-based intra prediction method. For example, if the intra prediction mode derived by the neural network-based intra prediction method is a DC mode, the MPM of the subsequent block may include the DC mode. According to another embodiment, the MPM may not include the neural network-based intra prediction method and the intra prediction mode derived by the neural network-based intra prediction method, but may include a predefined default mode. For example, the default mode may be a predetermined non-directional mode and / or a predetermined directional mode. Alternatively, the MPM may not include the neural network-based intra prediction method and the intra prediction mode derived by the neural network-based intra prediction method, but may include an intra prediction mode derived by a method such as DIMD, intra-template matching, or OBIC.

[0224]

[0225] Information related to intra prediction modes derived by a neural network-based intra prediction method can be utilized for DM-based intra prediction of a chroma block corresponding to the current luma block.

[0226] If the intra prediction mode of the current chroma block is DM and the current luma block is predicted through a neural network-based intra prediction method, the intra prediction mode of the chroma block corresponding to the current luma block can be derived based on information about the neural network-based intra prediction method.

[0227] For example, if the intra prediction mode of the current chroma block is DM and the luma block corresponding to the current chroma block is predicted using a neural network-based intra prediction method, the current chroma block can be predicted using the neural network-based intra prediction method. In this case, the neural network-based intra prediction method for the chroma block can use a model separate from the model used in the neural network-based intra prediction method for the luma block.

[0228] Alternatively, if the intra prediction mode of the current chroma block is DM and the luma block corresponding to the current chroma block is predicted through a neural network-based intra prediction method, the intra prediction mode of the current chroma block may be set to the intra prediction mode derived through the neural network-based intra prediction method.

[0229] Alternatively, if the intra prediction mode of the current chroma block is DM and the luma block corresponding to the current chroma block is predicted using a neural network-based intra prediction method, the intra prediction mode of the current chroma block may be set to a predefined default mode. For example, the default mode may be a predetermined non-directional mode and / or a predetermined directional mode. Alternatively, the intra prediction mode of the current chroma block may be set to an intra prediction mode derived using a method such as DIMD, intra template matching, or OBIC.

[0230]

[0231] The intra-prediction mode-related information derived by a neural network-based intra-prediction method can be utilized in other intra-prediction mode derivation methods. For example, the intra-prediction mode-related information derived by a neural network-based intra-prediction method can be used as a replacement for intra-prediction modes derived through methods such as DIMD, intra-template matching, and OBIC.

[0232] Here, the intra prediction mode-related information derived by the neural network-based intra prediction method can always replace the derived intra prediction mode. Alternatively, based on the comparison result between each intra prediction mode and a predetermined threshold, the intra prediction mode-related information derived by the neural network-based intra prediction method can replace the derived intra prediction mode. Alternatively, based on the RDO cost value for each intra prediction mode, either the intra prediction mode-related information derived by the neural network-based intra prediction method or one of the derived intra prediction modes can be selectively utilized.

[0233] Alternatively, an intra prediction value for the current block can be derived by blending an intra prediction value based on an intra prediction mode derived by a neural network-based intra prediction method and an intra prediction value based on an induced intra prediction mode.

[0234] Information about intra-prediction modes derived by a neural network-based intra-prediction method and a method for selectively and / or fully utilizing the derived intra-prediction modes can be determined explicitly or implicitly. Furthermore, the determined method can be performed identically in an encoder and a decoder. When the information about intra-prediction modes derived by a neural network-based intra-prediction method and a method for selectively and / or fully utilizing the derived intra-prediction modes are explicitly determined, the encoder can signal information about the determined method to the decoder via a bitstream.

[0235]

[0236] If the current block is predicted, a residual block of the current block can be generated. Here, a core transform and a non-separable secondary transform can be applied to the residual block of the current block. If the current block is an intra-prediction block, the residual block generated through the prediction block can also have a certain directionality. Therefore, applying a transform to the residual block of the current block based on the intra-prediction mode of the current block can maximize the compression efficiency of the residual block.

[0237] To this end, according to a neural network-based intra prediction method, the model can output information about the transformation kernel applied to the differential block of the current block. Specifically, the model can derive information about whether DCT2 is applied in the core transform, whether MTS (Multiple Transform Selection) is applied, and / or whether a non-separable primary transform (NSPT) is applied. Furthermore, the model can derive information indicating whether a low-frequency non-separable transform (LFNST) is applied after the core transform.

[0238] Furthermore, according to the neural network-based intra prediction method, the model can directly derive kernel information that is highly correlated with the predicted values ​​derived from the neural network model. For example, according to the neural network-based intra prediction method, the model can derive information for determining the kernel of the MTS. For the MTS, multiple MTS sets are defined, and each MTS set can include multiple candidate kernels.

[0239] Here, according to the neural network-based intra prediction method, the model can derive information indicating one MTS set among multiple MTS sets. Furthermore, one kernel can be selected from among candidate kernels included in the determined MTS set. Here, one kernel can be set based on the RDO cost value of each kernel, and information about the set kernel can be signaled via the bitstream.

[0240] Alternatively, according to a neural network-based intra prediction method, the model may output information indicating a combination of one MTS set from among multiple MTS sets and one candidate kernel included in the MTS set. In this case, according to the neural network-based intra prediction method, one candidate kernel may be set through the output information.

[0241] Additionally, according to the neural network-based intra prediction method, the model can derive information for determining the kernel of NSPT. Multiple NSPT sets are defined for NSPT, and each NSPT set can include multiple (e.g., three) candidate kernels.

[0242] Here, according to the neural network-based intra prediction method, the model can derive information indicating one NSPT set among multiple NSPT sets. Furthermore, one kernel can be selected from among candidate kernels included in the determined NSPT set. Here, one kernel can be set based on the RDO cost value of each kernel, and information about the set kernel can be signaled via the bitstream.

[0243] Alternatively, according to a neural network-based intra prediction method, the model may output information indicating a combination of one NSPT set from among multiple NSPT sets and one candidate kernel included in the NSPT set. In this case, according to the neural network-based intra prediction method, one candidate kernel may be set through the output information.

[0244] Additionally, according to the neural network-based intra prediction method, the model can derive information for determining the kernel of LFNST. Multiple LFNST sets are defined for LFNST, and each LFNST set can include multiple (e.g., three) candidate kernels.

[0245] Here, according to the neural network-based intra prediction method, the model can derive information indicating one LFNST set among multiple LFNST sets. Furthermore, one kernel can be selected from among candidate kernels included in the determined LFNST set. Here, one kernel can be set based on the RDO cost value of each kernel, and information about the set kernel can be signaled through the bitstream.

[0246] Alternatively, according to a neural network-based intra prediction method, the model may output information indicating a combination of one LFNST set from among multiple LFNST sets and one candidate kernel included in the LFNST set. In this case, according to the neural network-based intra prediction method, one candidate kernel may be set through the output information.

[0247] Additionally, according to the neural network-based intra prediction method, when a secondary transformation is applied, information indicating whether to apply a transpose operation to the coefficients derived through the core transformation can be output.

[0248]

[0249] According to a neural network-based intra prediction method, information regarding intra-prediction and inter-prediction methods applicable to the current block can be output. For example, if the intra-subpartition (ISP) method is not suitable for predicting the current block, the model can derive information indicating that the ISP method cannot be applied to the current block. Based on this information, the encoder can choose not to apply the ISP method to intra-prediction of the current block and omit encoding information regarding the ISP method. However, ISP is merely an example, and it will be apparent that the method can be applied to other intra-prediction and inter-prediction methods as well.

[0250]

[0251] As another example, information regarding the combination of prediction methods and / or transformation methods applicable to the current block can be output. For example, according to a neural network-based intra prediction method, the model can derive information indicating that a specific combination of the ISP method and LFNST method should be used for the current block. Based on this information, the encoder can apply the ISP method and LFNST to the intra prediction of the current block, and omit encoding information not used in the intra prediction of the current block. However, it should be understood that ISP and LFNST are only examples, and the present invention can be applied to other prediction methods and transformation methods.

[0252]

[0253] Neural network-based intra prediction methods can be applied to chroma blocks as well as luma blocks. Encoders and decoders can apply neural network-based intra prediction methods to chroma blocks using models generated through training.

[0254] The input values ​​of the model generated through training used in the encoder and decoder may include a reconstructed image surrounding the current chroma block, a predicted image of the current chroma block, a prediction mode of the current chroma block, a QP value, a slice type, an IPB block type, a size of the current chroma block, a total number of pixels of the current chroma block, a width of the current chroma block, a height of the current chroma block, a tree type of the current chroma block (e.g., whether a single tree or a dual tree is used), a reconstructed value of a luma block corresponding to the current chroma block, a predicted value of the luma block, and an intra prediction mode of the luma block.

[0255] In addition, the output value derived from the model generated through training may include a prediction value of the current chroma block, a transformation block kernel information value for transformation applied to the current chroma block, an intra mode information value of the current chroma block, an information value indicating an intra prediction method, etc.

[0256]

[0257] Alternatively, when applying a neural network-based chroma intra prediction method, whether or not to apply the neural network-based chroma intra prediction method can be determined explicitly or implicitly.

[0258] Alternatively, a neural network-based chroma intra prediction method and a signal prediction-based chroma intra prediction method may be selectively applied.

[0259] For example, a chroma intra prediction value derived through a neural network-based chroma intra prediction method can replace a chroma intra prediction value predicted through a signal prediction-based chroma intra prediction method.

[0260] Alternatively, a chroma intra prediction value derived through a neural network-based chroma intra prediction method can replace a chroma intra prediction value predicted through a signal prediction-based chroma intra prediction method based on information about the intra prediction mode of a luma block corresponding to the current chroma block.

[0261] Alternatively, a chroma intra prediction value derived through a neural network-based chroma intra prediction method can replace a chroma intra prediction value predicted through a signal prediction-based chroma intra prediction method based on a comparison result between a value of a chroma intra prediction mode and a predetermined threshold value.

[0262] Alternatively, based on the RDO cost value for each intra prediction mode, one of the intra prediction mode-related information derived by the neural network-based chroma intra prediction method or the signal prediction-based chroma intra prediction mode can be selectively utilized.

[0263] Alternatively, the intra prediction value for the current chroma block can be derived by blending the intra prediction value derived by the neural network-based chroma intra prediction method and the intra prediction value with the signal prediction-based chroma intra prediction. The method of blending the intra prediction value derived by the neural network-based chroma intra prediction method and the intra prediction value with the signal prediction-based chroma intra prediction can be performed identically by the encoder and the decoder. If the method of blending the intra prediction value derived by the neural network-based chroma intra prediction method and the intra prediction value with the signal prediction-based chroma intra prediction is explicitly determined, the encoder can signal information about the chroma prediction method to the decoder through the bitstream.

[0264] The method of selectively and / or implicitly utilizing the intra prediction mode-related information derived by the neural network-based chroma intra prediction method and the intra prediction mode-related information derived by the signal prediction-based intra prediction method can be determined explicitly or implicitly. If the chroma intra prediction method is explicitly determined, the encoder can signal the information about the chroma prediction method to the decoder through the bitstream.

[0265] For example, if a luma block corresponding to the current chroma block is predicted using a neural network-based intra prediction method, the current chroma block can also be predicted using a neural network-based intra prediction method.

[0266] The application of a neural network-based chroma intra prediction method can be determined by the encoder. Furthermore, the encoder can signal information about the application of a neural network-based chroma intra prediction method to the decoder via the bitstream.

[0267]

[0268] When a single tree is applied to the current block, luma and chroma blocks can always be split into pairs. Therefore, the split luma and chroma blocks can always overlap each other. Consequently, a luma block in the same location can overlap the split chroma block.

[0269] On the other hand, when a dual tree is applied to an intra-slice or inter-slice, the block partitioning of luma blocks and chroma blocks can be performed independently. Therefore, the partitioned luma blocks and chroma blocks may not overlap identically. When referencing luma information to predict a chroma block, the reference sample area for referencing luma information can be set as follows.

[0270]

[0271] FIG. 8 is a diagram illustrating an example of a structure of partitioned blocks based on a dual tree according to an embodiment of the present invention.

[0272] Referring to FIG. 8, a luma coding tree block and a chroma coding tree block can be defined. As illustrated in FIG. 8, a chroma coding tree block can be vertically partitioned based on a binary tree. As a result, a current chroma unit can be derived from the chroma coding tree block.

[0273] On the other hand, a luma coding tree block corresponding to a chroma coding tree block can be partitioned based on a quad tree. Each of the quad tree-split blocks can then be partitioned vertically based on a binary tree or horizontally based on a binary tree. Alternatively, the luma coding tree block and its sub-blocks can be partitioned based on a ternary tree.

[0274] That is, as illustrated in FIG. 8, when the tree type of the coding tree unit is a dual tree, the block partition structures of the luma coding tree block and the chroma coding tree block may be different from each other. As a result, the luma coding tree block corresponding to the current chroma block may include a plurality of luma blocks. For example, the luma coding tree block corresponding to the current chroma block may include a plurality of blocks in addition to a luma block including a C sample, a luma block including a TL sample, a luma block including a TR sample, a luma block including a BL sample, and a luma block including a BR sample. Accordingly, the chroma blocks and luma blocks partitioned into the coding tree blocks may not overlap each other in the same manner.

[0275] On the other hand, if the luma area corresponding to the chroma block includes multiple blocks, the luma block corresponding to the chroma block can be determined as follows.

[0276] The luma block corresponding to the chroma block can be determined as the luma block with the largest size among the blocks located in the luma region corresponding to the chroma block. That is, information about the largest luma block in the luma region corresponding to the chroma block can be utilized to predict the current chroma block.

[0277] The luma block corresponding to a chroma block can be determined by all blocks located in the luma region corresponding to the chroma block. That is, information on all blocks in the luma region corresponding to the chroma block can be utilized to predict the current chroma block.

[0278] A luma block corresponding to a chroma block can be determined as a luma block including a sample (C in Fig. 8) located at the center of a luma region corresponding to the chroma block. That is, information of a luma block including a center pixel of a luma region corresponding to the chroma block can be utilized to predict the current chroma block.

[0279] A luma block corresponding to a chroma block can be determined as a luma block including a sample (TL in Fig. 8) located at the upper left of the luma region corresponding to the chroma block. That is, information of a luma block including a sample at the upper left of the luma region corresponding to the chroma block can be utilized to predict the current chroma block.

[0280] A luma block corresponding to a chroma block can be determined as a luma block including a sample located at the lower left (BL in Fig. 8) of the luma region corresponding to the chroma block. That is, information of a luma block including a sample located at the lower left of the luma region corresponding to the chroma block can be utilized to predict the current chroma block.

[0281] A luma block corresponding to a chroma block can be determined as a luma block including a sample (TR in Fig. 8) located at the upper right of the luma region corresponding to the chroma block. That is, information of a luma block including a sample at the upper right of the luma region corresponding to the chroma block can be utilized to predict the current chroma block.

[0282] A luma block corresponding to a chroma block can be determined as a luma block including a sample (BR in Fig. 8) located at the lower right of the luma region corresponding to the chroma block. That is, information of a luma block including a sample at the lower right of the luma region corresponding to the chroma block can be utilized to predict the current chroma block.

[0283]

[0284] An image decoding method and an image encoding method for an intra prediction method based on an artificial neural network according to an embodiment of the present disclosure may be as described below.

[0285]

[0286] FIG. 9 is a flowchart illustrating an image decoding method for an intra prediction method according to an embodiment of the present disclosure. The image decoding method of FIG. 9 can be performed by an image decoding device.

[0287] Referring to FIG. 9, the image decoding device can derive input values ​​of a neural network model based on available reference samples among reference samples adjacent to the current block (S910).

[0288] Here, the reference samples adjacent to the current block may include a block adjacent to the left of the current block, a block adjacent to the top of the current block, and a block adjacent to the upper left of the current block. Additionally, the reference samples adjacent to the current block may further include a block adjacent to the lower left of the current block and a block adjacent to the upper right of the current block.

[0289] Here, if the current block is adjacent to the boundary of a given area containing the current block, the available reference sample may be a reference sample located inside the given area.

[0290] The input values ​​of the neural network model may include available reference samples and additional sample values ​​derived based on the available reference samples.

[0291] The image decoding device can input input values ​​into a neural network model to derive intra prediction related information of the current block (S920).

[0292] Here, the intra prediction related information of the current block may include at least one of information about the intra prediction mode of the current block, information about the intra prediction method of the current block, an intra prediction value of the current block, and information about a transformation kernel applied to the current block.

[0293] Here, information on the transformation kernel applied to the current block can indicate one MTS set among MTS (multiple transform selection) sets including multiple transformation kernels.

[0294] Alternatively, the information on the transformation kernel applied to the current block may indicate one NSPT set among NSPT sets including multiple transformation kernels.

[0295] Alternatively, the information on the transformation kernel applied to the current block may indicate one LFNST set among LFNST sets including multiple transformation kernels.

[0296] The video decoding device can generate a prediction block of the current block based on intra prediction related information of the current block (S930).

[0297] Here, the prediction block of the current block can be generated based on one of the intra prediction values ​​of the current block output by the neural network and the intra prediction mode of the current block output by the neural network.

[0298] Alternatively, the prediction block of the current block can be generated by weighting the intra prediction value of the current block derived based on the intra prediction value of the current block output by the neural network and the intra prediction mode of the current block output by the neural network.

[0299] Meanwhile, the neural network model can output the intra prediction value of the current block based on input values ​​whose ranges have been changed. Here, the range of the input values ​​can be changed based on the internal operation bit depth. Furthermore, the range of the intra prediction value of the current block output by the neural network model can be changed based on the bit depth in the domain of the current block.

[0300] Alternatively, the prediction block of the current block may be generated based on one of the intra prediction values ​​derived based on the intra prediction mode of the current block and the intra prediction values ​​derived based on the intra prediction mode derived based on signal processing.

[0301] Alternatively, the prediction block of the current block can be generated by weighting an intra prediction value derived based on an intra prediction mode of the current block and an intra prediction value derived based on an intra prediction mode derived based on signal processing.

[0302]

[0303] Meanwhile, the steps described in FIG. 9 can be performed in the same manner in an image encoding method. Furthermore, a bitstream can be generated by an image encoding method including the steps described in FIG. 9. The bitstream can be stored on a non-transitory computer-readable recording medium and can also be transmitted (or streamed).

[0304]

[0305] While the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and individual steps may be performed simultaneously or in different orders, if desired. To implement a method according to this disclosure, additional steps may be included in addition to the steps illustrated, some steps may be excluded and the remaining steps included, or some steps may be excluded and additional steps included.

[0306] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combinations of two or more.

[0307] Various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, the embodiments may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general processors, controllers, microcontrollers, microprocessors, etc.

[0308] Alternatively, various embodiments of the present disclosure may be implemented in the form of program commands that can be executed by various computer components and recorded on a computer-readable recording medium. Furthermore, a bitstream generated by the encoding method according to the above embodiment may be stored on a non-transitory computer-readable recording medium.

[0309] The computer-readable recording medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure or may be known and available to those skilled in the art of computer software.

[0310] As described above, the present disclosure has been described based on specific details, such as specific components, and limited embodiments and drawings. However, the embodiments of the present disclosure are provided merely to facilitate a general understanding of the present disclosure and are not intended to limit the present disclosure to these embodiments. Accordingly, those skilled in the art will appreciate that various modifications and variations can be made based on the description.

[0311] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and all things that are modified equally or equivalently to the following claims as well as the claims are considered to fall within the scope of the idea of ​​the present invention.

[0312] The present invention can be used in a device for encoding an image, a device for decoding an image, and a recording medium for storing a bitstream.

Claims

1. In the video decryption method, A step of deriving an input value of a neural network model based on available reference samples among reference samples adjacent to the current block; A step of inputting the above input value into the neural network model to derive intra prediction related information of the current block; and A step of generating a prediction block of the current block based on intra prediction related information of the current block, The intra prediction related information of the current block above is: Includes at least one of information about an intra prediction mode of the current block, information about an intra prediction method of the current block, an intra prediction value of the current block, and information about a transformation kernel applied to the current block. An image decoding method, characterized in that when the current block is adjacent to the boundary of a predetermined area including the current block, the available reference sample is a reference sample located inside the predetermined area.

2. In paragraph 1, Reference samples adjacent to the current block above are, An image decoding method, characterized in that it includes a block adjacent to the left of the current block, a block adjacent to the top of the current block, and a block adjacent to the top left of the current block.

3. In paragraph 2, Reference samples adjacent to the current block above are, An image decoding method, characterized in that it further includes a block adjacent to the lower left of the current block and a block adjacent to the upper right of the current block.

4. In paragraph 1, An image decoding method, characterized in that the input values ​​of the neural network model include the available reference samples and additional sample values ​​derived based on the available reference samples.

5. In paragraph 1, The prediction block of the current block above is, An image decoding method, characterized in that it is generated based on one of the intra prediction value of the current block output by the neural network and the intra prediction value of the current block derived based on the intra prediction mode of the current block output by the neural network.

6. In paragraph 1, The prediction block of the current block above is, An image decoding method characterized in that it is generated by weighting and adding the intra prediction value of the current block output by the neural network and the intra prediction value of the current block derived based on the intra prediction mode of the current block output by the neural network.

7. In paragraph 1, The above neural network model outputs the intra prediction value of the current block based on the input value whose range has been changed, An image decoding method, characterized in that the range of the above input values ​​is changed based on an internal operation bit depth.

8. In paragraph 7, An image decoding method, characterized in that the range of the intra prediction value of the current block output by the neural network model is changed based on the bit depth on the domain of the current block.

9. In paragraph 1, The prediction block of the current block above is, A video decoding method, characterized in that it is generated based on one of an intra prediction value derived based on an intra prediction mode of the current block and an intra prediction value derived based on an intra prediction mode derived based on signal processing.

10. In paragraph 1, The prediction block of the current block above is, An image decoding method characterized in that it is generated by weighting and adding an intra prediction value derived based on an intra prediction mode of the current block and an intra prediction value derived based on an intra prediction mode derived based on signal processing.

11. In paragraph 1, Information about the transformation kernel applied to the current block above, An image decoding method, characterized in that it indicates one MTS set among MTS sets including a plurality of transform kernels.

12. In paragraph 1, Information about the transformation kernel applied to the current block above, An image decoding method characterized by indicating one NSPT set among NSPT sets including a plurality of transform kernels.

13. In paragraph 1, Information about the transformation kernel applied to the current block above, An image decoding method, characterized in that it indicates one LFNST set among LFNST sets including a plurality of transform kernels.

14. In the video encoding method, A step of deriving an input value of a neural network model based on available reference samples among reference samples adjacent to the current block; A step of inputting the above input value into the neural network model to derive intra prediction related information of the current block; and A step of generating a prediction block of the current block based on intra prediction related information of the current block, The intra prediction related information of the current block above is: Includes at least one of information about an intra prediction mode of the current block, information about an intra prediction method of the current block, an intra prediction value of the current block, and information about a transformation kernel applied to the current block. A video encoding method, characterized in that when the current block is adjacent to the boundary of a predetermined area including the current block, the available reference sample is a reference sample located inside the predetermined area.

15. In a non-transitory computer-readable recording medium storing a bitstream generated by an image encoding method, The above image encoding method is, A step of deriving an input value of a neural network model based on available reference samples among reference samples adjacent to the current block; A step of inputting the above input value into the neural network model to derive intra prediction related information of the current block; and A step of generating a prediction block of the current block based on intra prediction related information of the current block, The intra prediction related information of the current block above is: Includes at least one of information about an intra prediction mode of the current block, information about an intra prediction method of the current block, an intra prediction value of the current block, and information about a transformation kernel applied to the current block. A non-transitory computer-readable recording medium, characterized in that when the current block is adjacent to the boundary of a predetermined area including the current block, the available reference sample is a reference sample located inside the predetermined area.

16. In a method for transmitting a bitstream generated by a video encoding method, The above transmission method includes a step of transmitting the bitstream, The above image encoding method is, A step of deriving an input value of a neural network model based on available reference samples among reference samples adjacent to the current block; A step of inputting the above input value into the neural network model to derive intra prediction related information of the current block; and A step of generating a prediction block of the current block based on intra prediction related information of the current block, The intra prediction related information of the current block above is: Includes at least one of information about an intra prediction mode of the current block, information about an intra prediction method of the current block, an intra prediction value of the current block, and information about a transformation kernel applied to the current block. A transmission method, characterized in that when the current block is adjacent to the boundary of a predetermined area including the current block, the available reference sample is a reference sample located inside the predetermined area.

Images