Image decoding apparatus and image encoding apparatus using ai, and method using same

Abstract

According to an embodiment, a method for decoding an image includes: obtaining a motion vector of a current block; obtaining a preliminary prediction block based on a reference block indicated by the motion vector in the reference image; obtaining a final prediction block of the current block by applying at least one of a picture order count (POC) map, the preliminary prediction block, and a quantization error map to a neural network, where the POC map includes a POC difference between the reference image and a current image including the current block; and reconstructing the current block on the basis of a final prediction block and a residual block obtained from the bitstream, in which sample values of a quantization error map are calculated on the basis of quantization parameters for the reference block.

Description

Technical Field

[0001] The present disclosure relates to a method and apparatus for processing an image, and more particularly, to a method and apparatus for encoding / decoding an image using artificial intelligence (AI). Background Art

[0002] In a codec such as H.264 Advanced Video Coding (AVC) or High Efficiency Video Coding (HEVC), an image may be divided into blocks, and the blocks may be encoded and decoded through inter-frame prediction or intra-frame prediction.

[0003] Intra prediction may refer to a method of compressing an image by removing spatial redundancy in the image, and inter prediction may be a method of compressing an image by removing temporal redundancy between images.

[0004] In the encoding process, a prediction block of the current block may be generated through intra prediction or inter prediction, a residual block may be generated by subtracting the prediction block from the current block, and residual samples of the residual block may be transformed and quantized.

[0005] In the decoding process, the residual samples of the residual block can be generated by inverse quantizing and inverse transforming the quantized transform coefficients of the residual block, and the current block can be reconstructed by adding the prediction block generated by intra prediction or inter prediction to the residual block. The reconstructed current block can be processed according to one or more filtering algorithms and then can be output.

[0006] In codecs such as H.264 AVC or HEVC, rule-based prediction modes may be used for inter prediction of a current block. Rule-based prediction modes may include, for example, skip mode, merge mode, or advanced motion vector prediction (AMVP) mode.

[0007] However, as the resolution of an image increases and the contents of the image diversify, a prediction mode that can flexibly consider the characteristics of the image is required. Summary of the invention

[0008] Technical Solution A method of decoding an image according to an embodiment of the present disclosure may include obtaining a motion vector of a current block.

[0009] In an embodiment of the present disclosure, the method may include: obtaining a preliminary prediction block by using a reference block indicated by a motion vector in a reference image.

[0010] In an embodiment of the present disclosure, the method may include: obtaining a final prediction block of the current block by applying at least one of a picture order count (POC) map, a preliminary prediction block, or a quantization error map to a neural network, wherein the POC map includes a POC difference between a reference image and a current image including the current block.

[0011] In an embodiment of the present disclosure, the method may include: reconstructing the current block by using a final prediction block and a residual block obtained from a bitstream.

[0012] In an embodiment of the present disclosure, the sample value of the quantization error map may be calculated according to the quantization parameter used for the reference block.

[0013] A method of encoding an image according to an embodiment of the present disclosure includes obtaining a motion vector indicating a reference block corresponding to a current block in a reference image.

[0014] In an embodiment of the present disclosure, the method may include: obtaining a final prediction block of the current block by applying at least one of a picture order count (POC) map including a reference image and a current image including the current block, a preliminary prediction block obtained based on the reference block, or a quantization error map to a neural network.

[0015] In an embodiment of the present disclosure, the method may include obtaining a residual block by using the current block and a final prediction block.

[0016] In an embodiment of the present disclosure, the method may include generating a bitstream including information about the residual block.

[0017] In an embodiment of the present disclosure, the sample value of the quantization error map may be calculated according to the quantization parameter used for the reference block.

[0018] An image decoding apparatus according to an embodiment of the present disclosure may include at least one memory storing at least one instruction and at least one processor configured to operate according to the at least one instruction.

[0019] In an embodiment of the present disclosure, the at least one processor may be configured to obtain a motion vector of the current block.

[0020] In an embodiment of the present disclosure, the at least one processor may be configured to obtain the preliminary prediction block by using a reference block indicated by a motion vector in a reference image.

[0021] In an embodiment of the present disclosure, the at least one processor may be configured to obtain a final prediction block of the current block by applying at least one of a picture order count (POC) map, a preliminary prediction block, or a quantization error map to a neural network, wherein the POC map includes a POC difference between a reference image and a current image including the current block.

[0022] In an embodiment of the present disclosure, the at least one processor may be configured to reconstruct the current block by using the final prediction block and a residual block obtained from a bitstream.

[0023] In an embodiment of the present disclosure, the sample value of the quantization error map may be calculated according to the quantization parameter used for the reference block.

[0024] According to an embodiment of the present disclosure, an image encoding apparatus may include at least one memory storing at least one instruction and at least one processor configured to operate according to the at least one instruction.

[0025] In an embodiment of the present disclosure, the at least one processor may be configured to obtain a motion vector indicating a reference block corresponding to the current block in the reference image.

[0026] In an embodiment of the present disclosure, the at least one processor may be configured to obtain a final prediction block of the current block by applying at least one of a picture order count (POC) map including a reference image and a current image including the current block, a preliminary prediction block obtained based on the reference block, or a quantization error map to a neural network.

[0027] In an embodiment of the present disclosure, the at least one processor may be configured to obtain a residual block by using the current block and a final prediction block.

[0028] In an embodiment of the present disclosure, the at least one processor may be configured to generate a bitstream including information about the residual block.

[0029] In an embodiment of the present disclosure, the sample value of the quantization error map may be calculated according to the quantization parameter used for the reference block. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which: Figure 1 is a diagram showing a configuration of an image decoding device according to an embodiment of the present disclosure; Figure 2 is a diagram showing a configuration of an artificial intelligence (AI)-based prediction decoder according to an embodiment of the present disclosure; Figure 3is a diagram showing spatial neighboring blocks and temporal neighboring blocks of a current block according to an embodiment of the present disclosure; Figure 4 is a diagram showing reference blocks indicated by a motion vector of List 0 and a motion vector of List 1 according to an embodiment of the present disclosure; Figure 5 is a diagram for describing a method of obtaining a quantization error map based on a quantization parameter for a reference block according to an embodiment of the present disclosure; Figure 6 is a diagram for describing a method of obtaining a quantization error map based on a quantization parameter for a reference block according to an embodiment of the present disclosure; Figure 7 is a diagram for describing a method of obtaining a quantization error map based on a quantization parameter for a reference block according to an embodiment of the present disclosure; Figure 8 is a diagram for describing a method of obtaining a quantization error map based on a quantization parameter for a reference block according to an embodiment of the present disclosure; Fig. 9 is a diagram showing the structure of a neural network according to an embodiment of the present disclosure; Fig.10 is a diagram for describing a convolution operation at a convolution layer according to an embodiment of the present disclosure; Fig.11 is a diagram for describing a method for obtaining an enlarged preliminary prediction block according to an embodiment of the present disclosure; Fig.12 is a diagram for describing a method of obtaining an enlarged preliminary prediction block when a boundary of a reference block corresponds to a boundary of a reference image according to an embodiment of the present disclosure; Fig.13 is a diagram for describing a method of obtaining an enlarged quantization error map according to an embodiment of the present disclosure; Fig.14 is a diagram showing reconstructed samples and non-reconstructed samples in a current image according to an embodiment of the present disclosure; Fig.15 is a diagram for describing a method for obtaining an enlarged current reconstructed block according to an embodiment of the present disclosure; Fig.16 is a diagram illustrating a plurality of weight sets according to an embodiment of the present disclosure; Fig.17 is a diagram illustrating an image decoding method performed by an image decoding device according to an embodiment of the present disclosure; Fig.18 is a diagram showing a syntax according to an embodiment of the present disclosure; Fig.19 is a diagram showing a configuration of an image encoding device according to an embodiment of the present disclosure; Fig. 20is a diagram showing a configuration of an AI-based predictive encoder according to an embodiment of the present disclosure; Fig.21 is a diagram for describing a method of changing a motion vector of fractional precision into a motion vector of integer precision according to an embodiment of the present disclosure; Fig. 22 is a diagram illustrating an image encoding method performed by an image encoding device according to an embodiment of the present disclosure; Fig.23 is a diagram for describing a method of training a neural network according to an embodiment of the present disclosure; and Fig.24 is a diagram for describing a general encoding and decoding process of an image according to an embodiment of the present disclosure.

[0031] Best Mode A method of decoding an image according to an embodiment of the present disclosure may include obtaining a motion vector of a current block.

[0032] In an embodiment of the present disclosure, the method may include: obtaining a preliminary prediction block using a reference block indicated by a motion vector in a reference image.

[0033] In an embodiment of the present disclosure, the method may include: obtaining a final prediction block of the current block by applying at least one of a picture order count (POC) map, a preliminary prediction block, or a quantization error map to a neural network, wherein the POC map includes a POC difference between a reference image and a current image including the current block.

[0034] In an embodiment of the present disclosure, the method may include reconstructing the current block using the final prediction block and a residual block obtained from a bitstream.

[0035] In an embodiment of the present disclosure, the sample value of the quantization error map may be calculated according to the quantization parameter used for the reference block. DETAILED DESCRIPTION

[0036] Since the present disclosure allows various changes and many examples, the specific embodiments of the present disclosure will be shown in the drawings and described in detail in the written description. However, this is not intended to limit the present disclosure to a specific implementation mode, and all changes, equivalents and substitutes that do not depart from the spirit and technical scope of the various embodiments of the present disclosure may be included in the present disclosure.

[0037] In the description of the embodiments of the present disclosure, some detailed explanations of the related art may be omitted when it is considered that they may unnecessarily obscure the description. In addition, the numbers used in the description (e.g., first, second, etc.) are merely identifiers for distinguishing one element from another, and are not intended to indicate an order or preference unless otherwise indicated.

[0038] Throughout this disclosure, the expression "at least one of a, b, or c" refers to only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0039] In the present disclosure, when elements are described as being “connected” or “coupled” to each other, the elements may be directly connected or coupled to each other or may be connected or coupled to each other with intervening elements therebetween, unless otherwise specified.

[0040] In the present disclosure, with respect to an element represented as a "unit" or a "module", two or more elements may be combined into one element or one element may be divided into two or more elements according to subdivided functions. In addition to its own main function, each element described below may additionally perform some or all of the functions performed by another element, and some of the main functions of each element may be completely performed by another component.

[0041] In the present disclosure, an “image” or a “picture” may refer to a still image, a moving image including a plurality of consecutive still images (or frames), or a video.

[0042] In addition, in the present disclosure, "neural network" may refer to a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model that uses a specific algorithm. The neural network may be referred to as a deep neural network.

[0043] In the present disclosure, "weight" may refer to a value used in the operation process of each layer constituting a neural network. For example, when an input value is applied to a specific operation expression, a weight may be used. A weight is a value set as a result of training and may be updated using separate training data.

[0044] In the present disclosure, a "current block" may refer to a block currently being processed. The current block may be a slice, a tile, a maximum coding unit, a coding unit, a prediction unit, or a transform unit divided from a current image.

[0045] In the present disclosure, a "sample" may refer to data assigned to a sampling position in data to be processed, such as an image, a block, a filter kernel, or a map. For example, a sample may include a pixel in a two-dimensional (2D) image.

[0046] Before describing the image decoding apparatus 100 and the image encoding apparatus 1900 according to the embodiment of the present disclosure, Fig.24 An example of general image encoding and decoding processing is described.

[0047] Fig.24 is a diagram illustrating a general image encoding and decoding process according to an embodiment of the present disclosure.

[0048] The encoding device 2410 may transmit a bitstream generated by encoding an image to the decoding device 2450, and the decoding device 2450 may reconstruct the image by receiving the bitstream and decoding the bitstream.

[0049] In an embodiment of the present disclosure, the prediction encoder 2415 of the encoding device 2410 can output a prediction block by inter-frame prediction or intra-frame prediction of the current block, and the transform and quantization unit 2420 (shown as "T+Q") can output quantized transform coefficients by transforming and quantizing residual samples of the residual block between the prediction block and the current block.

[0050] The entropy encoder 2425 may encode the quantized transform coefficients and may output a bitstream.

[0051] The quantized transform coefficients may be dequantized and inverse transformed by a dequantization and inverse transform unit 2430 (shown as “Q -1 +T -1 ”) is reconstructed in the spatial domain as a residual block including residual samples. The reconstructed block obtained by adding the prediction block to the residual block can be output as a filtered block using the deblocking filtering unit 2435 and the loop filtering unit 2440. The reconstructed image including the filtered block can be used as a reference image for the next image in the prediction encoder 2415.

[0052] The bitstream received by the decoding apparatus 2450 may be reconstructed into a residual block including residual samples in the spatial domain through the entropy decoder 2455 and the inverse quantization and inverse transform unit 2460 .

[0053] A reconstructed block may be generated by combining the residual block and the prediction block output from the prediction decoder 2475, and the reconstructed block may be output as a filtered block through the deblocking filtering unit 2465 and the loop filtering unit 2470. A reconstructed image including the filtered block may be used as a reference image for a next image in the prediction decoder 2475.

[0054] In an embodiment of the present disclosure, the prediction encoder 2415 and the prediction decoder 2475 may perform prediction encoding and prediction decoding on the current block according to a rule-based prediction mode and / or a neural network-based prediction mode.

[0055] In an embodiment of the present disclosure, the rule-based prediction mode may include merge mode, skip mode, advanced motion vector prediction (AMVP) mode, bidirectional optical flow (BDOF) mode, or bidirectional prediction with coding unit (CU) level weight (BCW).

[0056] In an embodiment of the present disclosure, the prediction encoder 2415 and the prediction decoder 2475 may apply a rule-based prediction mode and a neural network-based prediction mode to the current block.

[0057] Reference Figures 1 to 23 An example of a neural network-based prediction model according to an embodiment of the present disclosure is described in detail.

[0058] Reference Figure 1 , the image decoding apparatus 100 may include a bitstream parser 110 and a decoder 130. The decoder 130 may include an artificial intelligence (AI)-based prediction decoder 132 and a reconstruction unit 134.

[0059] In an embodiment of the present disclosure, the bitstream parser 110 may be Fig.24 The decoder 130 corresponds to the entropy decoder 2455. In an embodiment of the present disclosure, the decoder 130 may correspond to the inverse quantization and inverse transform unit 2460, the prediction decoder 2475, the deblocking filter unit 2465, and the loop filter unit 2470.

[0060] The bitstream parser 110 and the decoder 130 may be implemented as, by, or using at least one processor. The bitstream parser 110 and the decoder 130 may operate according to at least one instruction stored in at least one memory.

[0061] Despite Figure 1 The bitstream parser 110 and the decoder 130 are shown separately in the figure, but the embodiment is not limited to this. For example, in some embodiments, the bitstream parser 110 and the decoder 130 may be implemented by one processor. In this case, the bitstream parser 110 and the decoder 130 may be implemented as a dedicated processor, implemented by a dedicated processor, or implemented using a dedicated processor, or may be implemented by a combination of software and a general-purpose processor (such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU)). In addition, when the bitstream parser 110 and the decoder 130 are implemented as a dedicated processor, implemented by a dedicated processor, or implemented using a dedicated processor, the dedicated processor may include at least one of a memory for implementing an embodiment of the present disclosure or a memory processor for using an external memory.

[0062] In an embodiment of the present disclosure, the bitstream parser 110 and the decoder 130 may include multiple processors. In this case, the bitstream parser 110 and the decoder 130 may be implemented by a combination of dedicated processors, or a combination of software and multiple general-purpose processors (such as AP, CPU and GPU).

[0063] The bitstream parser 110 may obtain a bitstream including a result of encoding an image.

[0064] The bitstream parser 110 may receive a bitstream from the image encoding apparatus 1900 through a network.

[0065] In an embodiment of the present disclosure, the bitstream parser 110 may obtain a bitstream from any data storage medium including a magnetic medium (such as a hard disk, a floppy disk, or a magnetic tape), an optical recording medium (such as a compact disk read-only memory (CD-ROM) or a digital versatile disk (DVD)), and a magneto-optical medium (such as a floppy disk).

[0066] The bitstream parser 110 may obtain a plurality of pieces of information required to reconstruct an image by parsing the bitstream.

[0067] In an embodiment of the present disclosure, the bitstream parser 110 may obtain syntax elements for reconstructing an image from a bitstream. Binary values ​​corresponding to the syntax elements may be included in the bitstream according to the image hierarchical structure. The bitstream parser 110 may obtain the syntax elements by entropy encoding the binary values ​​included in the bitstream.

[0068] In an embodiment of the present disclosure, the bitstream parser 110 may transmit information about a motion vector and information about a residual block obtained from a bitstream to the decoder 130 .

[0069] The decoder 130 may obtain the current reconstructed block based on the information transmitted from the bitstream parser 110. The current reconstructed block may refer to a block obtained by decoding the current block.

[0070] In an embodiment of the present disclosure, the AI-based prediction decoder 132 may obtain a final prediction block of a current block using information about a motion vector.

[0071] The AI-based prediction decoder 132 may obtain a final prediction block of the current block using AI (eg, a neural network).

[0072] A mode in which the AI-based prediction decoder 132 obtains a final prediction block of a current block using a neural network may be defined as a neural network-based prediction mode.

[0073] The reconstruction unit 134 may obtain a residual block of the current block using the information about the residual block provided from the bitstream parser 110. In an embodiment of the present disclosure, the information about the residual block may include information about a quantized transform coefficient.

[0074] In an embodiment of the present disclosure, the reconstruction unit 134 may obtain a residual block in the spatial domain by dequantizing and inverse transforming the quantized transform coefficients.

[0075] The reconstruction unit 134 may obtain a current reconstructed block corresponding to the current block using the final prediction block and the residual block. In an embodiment of the present disclosure, the reconstruction unit 134 may obtain the current reconstructed block by adding the sample values ​​of the final prediction block to the sample values ​​of the residual block.

[0076] Reference Figure 2 An example of the AI-based predictive decoder 132 is described in more detail.

[0077] Figure 2 is a diagram showing a configuration of the AI-based prediction decoder 132 according to an embodiment of the present disclosure.

[0078] Reference Figure 2 , the AI-based prediction decoder 132 may include a motion information obtainer 210 , a prediction block obtainer 220 , a neural network setter 230 , and a neural network 240 .

[0079] The neural network 240 may be stored in a memory. In an embodiment of the present disclosure, the neural network 240 may be implemented as, by, or using an AI processor.

[0080] The motion information obtainer 210 may obtain the motion vector of the current block using the information about the motion vector. As described below, when the precision of the motion vector of the current block is changed from fractional precision to integer precision by the motion information obtainer 2010 of the image encoding apparatus 1900, the motion information obtainer 210 may obtain the motion vector of the current block with integer precision.

[0081] The information about the motion vector may include information indicating at least one motion vector candidate in the motion vector candidate list, for example, a flag or an index. In an embodiment of the present disclosure, the information about the motion vector may also include information about a residual motion vector corresponding to a difference between a predicted motion vector of a current block and a motion vector of the current block.

[0082] The prediction block obtainer 220 may obtain a preliminary prediction block using the reference image and the motion vector of the current block obtained by the motion information obtainer 210 .

[0083] In an embodiment of the present disclosure, the motion information obtainer 210 and the prediction block obtainer 220 may obtain a preliminary prediction block of the current block according to a rule-based prediction mode.

[0084] The rule-based prediction mode may include a merge mode, a skip mode, or an AMVP mode.

[0085] According to the merge mode or the skip mode, the motion information obtainer 210 can construct a motion vector candidate list including motion vectors of neighboring blocks of the current block as motion vector candidates, and can determine a motion vector candidate indicated by information included in the bitstream from the motion vector candidates included in the motion vector candidate list as the motion vector of the current block.

[0086] In addition, according to the AMVP mode, the motion information obtainer 210 may construct a motion vector candidate list including motion vectors of neighboring blocks of the current block as motion vector candidates, and may determine a motion vector candidate indicated by information included in the bitstream as a predicted motion vector of the current block from the motion vector candidates included in the motion vector candidate list. The motion information obtainer 210 may determine the motion vector of the current block using the predicted motion vector and the residual motion vector of the current block.

[0087] The merge mode, the skip mode, or the AMVP mode is an example of the rule-based prediction mode, and in an embodiment of the present disclosure, the rule-based prediction mode may further include a decoder-side motion vector refinement (DMVR) mode.

[0088] In the merge mode, the skip mode, and the AMVP mode, the process of constructing the motion vector candidate list may be performed in common. Figure 3 Examples of neighboring blocks that may be included in the motion vector candidate list are described.

[0089] Reference Figure 3 , the neighboring blocks of the current block 300 may include spatially neighboring blocks (e.g., block A0, block A1, block B0, block B1, and block B2) that are spatially adjacent to the current block 300 and temporal blocks (e.g., block Col and block Br) that are temporally adjacent to the current block 300.

[0090] In an embodiment of the present disclosure, the spatially neighboring blocks may include at least one of a lower left corner block A0, a lower left block A1, an upper right corner block B0, an upper right block B1, or an upper left corner block B2.

[0091] In an embodiment of the present disclosure, the temporally neighboring block may include at least one of: a block Col located at the same point as the current block in a collocated image having a picture order count (POC) different from that of the current image including the current block 300, or a block Br spatially adjacent to the block Col located at the same point.

[0092] The block Br may be located at the lower right of the block Col, where the block Col is located at the same point as the current block 300. The block Col located at the same point as the current block 300 may be a block including a pixel corresponding to a center pixel in the current block 300 among pixels included in the co-located image.

[0093] The motion information obtainer 210 may determine the availability of neighboring blocks according to a specific order, and may sequentially include motion vectors of the neighboring blocks as motion vector candidates in the motion vector candidate list according to the determination result.

[0094] In an embodiment of the present disclosure, when a neighboring block is intra predicted, the motion information obtainer 210 may determine that there is no availability of a block.

[0095] In an embodiment of the present disclosure, the motion vector obtained by the motion information obtainer 210 may include the motion vector of list 0, the motion vector of list 1, or the motion vector of list 0 and the motion vector of list 1.

[0096] The motion vector of list 0 may be a motion vector indicating a reference block in a reference image included in list 0 (or, for example, reference image list 0), and the motion vector of list 1 may be a motion vector indicating a reference block in a reference image included in list 1 (or, for example, reference image list 1).

[0097] The prediction block obtainer 220 may obtain a preliminary prediction block using a reference block indicated by a motion vector in a reference image.

[0098] In an embodiment of the present disclosure, the preliminary prediction block may be obtained by applying interpolation to a reference block indicated by a motion vector in a reference image. Accordingly, the preliminary prediction block may include sub-pixels obtained by applying filtering to integer pixels.

[0099] In an embodiment of the present disclosure, a reference block indicated by a motion vector in a reference image may be determined as a preliminary prediction block. For example, when the precision of the motion vector of the current block is integer precision, the reference block indicated by the motion vector may be determined as a preliminary prediction block.

[0100] When the motion vector of List 0 is obtained by the motion information obtainer 210 , the prediction block obtainer 220 may obtain a reference block indicated by the motion vector of List 0 in a reference image included in List 0 , and may obtain a preliminary prediction block of List 0 using the reference block.

[0101] In an embodiment of the present disclosure, when the motion vector of List 1 is obtained by the motion information obtainer 210, the prediction block obtainer 220 can obtain a reference block indicated by the motion vector of List 1 in a reference image included in List 1, and can use the reference block to obtain a preliminary prediction block of List 1.

[0102] In an embodiment of the present disclosure, when the motion vector of list 0 and the motion vector of list 1 are obtained by the motion information obtainer 210, the prediction block obtainer 220 may use the reference block indicated by the motion vector of list 0 in the reference image included in list 0 to obtain the prediction block of list 0, and may use the reference block indicated by the motion vector of list 1 in the reference image included in list 1 to obtain the prediction block of list 1.

[0103] Figure 4is a diagram illustrating reference blocks indicated by a motion vector of List 0 and a motion vector of List 1 according to an embodiment of the present disclosure.

[0104] like Figure 4 As shown in , when the motion vector mv1 of list 0 and the motion vector mv2 of list 1 are obtained for the current block 300 in the current image 400, the prediction block obtainer 220 may obtain a first reference block 415 indicated by the motion vector mv1 of list 0 in the first reference image 410 included in list 0, and may obtain a second reference block 435 indicated by the motion vector mv2 of list 1 in the second reference image 430 included in list 1. A first preliminary prediction block of list 0 may be obtained from the first reference block 415, and a second preliminary prediction block of list 1 may be obtained from the second reference block 435.

[0105] Return to reference Figure 2 , the neural network setter 230 can obtain data to be input to the neural network 240.

[0106] In an embodiment of the present disclosure, the neural network setter 230 may obtain data to be input to the neural network 240 based on a reference image, a preliminary prediction block, and a quantization parameter for the reference block.

[0107] In an embodiment of the present disclosure, at least one of the preliminary prediction block, the POC map, or the quantization error map may be input to the neural network 240 through the neural network setter 230 .

[0108] Hereinafter, an example of data input to the neural network 240 is described in detail. A preliminary prediction block, which may be a block determined to be similar to the current block in, for example, a rule-based prediction mode, may be used to obtain a final prediction block more similar to the current block.

[0109] The quantization parameter for the reference block may be used to quantize / dequantize residual data of the reference block in a process of encoding / decoding the reference block. The amount of error according to quantization / dequantization may vary according to the quantization parameter. For example, the quantization parameter may refer to the amount of error or distortion included in the reference block reconstructed by encoding / decoding.

[0110] Since the value calculated based on the quantization parameter can be input to the neural network 240, the neural network 240 can consider the reliability of the samples of the preliminary prediction block or the influence of the samples of the preliminary prediction block on the samples of the final prediction block when obtaining the final prediction block from the preliminary prediction block. As described below, the neural network 240 can be trained so that the difference between the current block (or original block) for training and the final prediction block for training is reduced, and the final prediction block for training is output based on the quantization error map for training and the preliminary prediction block for training. Therefore, the neural network 240 can check the influence of the quantization error map for training on the final prediction block for training, and can output the final prediction block for training similar to the current block for training.

[0111] The POC map may include the difference between the POC of the current image and the POC of the reference image as a sample value. In an embodiment, the difference may be referred to as a POC difference. The POC may indicate an image output order. Therefore, the POC difference between the current image and the reference image may refer to an output order difference between the current image and the reference image or a time difference between the current image and the reference image. Because the position or size of an object may be changed in consecutive images due to the movement of the object, the neural network 240 may output a final prediction block that is more similar to the current block by training on the time difference between the current image and the reference image.

[0112] In an embodiment of the present disclosure, the neural network 240 may include one or more convolution layers. The neural network 240 may output a final prediction block by processing at least one of the preliminary prediction block, the POC map, or the quantization error map input from the neural network setter 230.

[0113] As shown below Fig. 9 and Fig.10 As described above, the neural network 240 can determine the sample value of the final prediction block individually by applying a specific operation to the input data. The motion vector can be calculated for each block of the image in the rule-based prediction mode, while the sample of the final prediction block can be determined individually in the neural network-based prediction mode according to the embodiment. Therefore, the neural network 240 can consider the motion vector of the current block for each sample. Therefore, based on the neural network-based prediction mode according to the embodiment of the present disclosure, the final prediction block obtained according to the neural network-based prediction mode of the embodiment of the present disclosure can be more similar to the current block than the prediction block obtained according to the rule-based prediction mode.

[0114] Reference Figures 5 to 8 An example of a method of obtaining a quantization error map input to the neural network 240 is described.

[0115] Figures 5 to 8 is a diagram for describing a method of obtaining a quantization error map based on a quantization parameter for a reference block according to an embodiment of the present disclosure.

[0116] In an embodiment of the present disclosure, the sample values ​​of the quantization error map may be calculated from or based on the quantization parameter for the reference block.

[0117] The quantization parameter for the reference block may be obtained from a bitstream including information for decoding the reference block.

[0118] In an embodiment of the present disclosure, the quantization error map may include quantization error values ​​calculated based on the quantization parameters as sample values.

[0119] The quantization error value may indicate an amount of error that may be caused by quantization and inverse quantization applied to residual samples in the process of encoding and decoding the reference block.

[0120] When the quantization error value is large, this may mean that the difference between the transform coefficient before quantization and the transform coefficient after inverse quantization may be large. As the difference between the transform coefficient before quantization and the transform coefficient after inverse quantization increases, the identity between the original block and the reference block obtained by decoding the encoded data may decrease.

[0121] Since an error caused by quantization and inverse quantization may correspond to an artifact, neural network-based inter-frame prediction may be performed by considering a quantization error value.

[0122] In an embodiment of the present disclosure, the quantization error value may be calculated based on the following equation 1: [Equation 1] Quantization error value = quantization step size ^2 / 12 Referring to Equation 1, the quantization error value may be proportional to the square of the quantization step size.

[0123] The quantization step size may be used to quantize the transform coefficient, and the transform coefficient may be quantized by dividing the transform coefficient by the quantization step size. In addition, the quantized transform coefficient may be dequantized by multiplying the quantized transform coefficient by the quantization step size.

[0124] The quantization step size can be approximated by the following equation 2: [Equation 2] Quantization step size = 2^(quantization parameter / n) / quantization scale [quantization parameter%n] In Equation 2, quantization scaling [quantization parameter % n] may indicate a scaling value indicated by a quantization parameter among predetermined n scaling values. As an example, since six scaling values ​​(26214, 23302, 20560, 18396, 16384, and 14564) may be defined in the HEVC codec, n may be 6 according to the HEVC codec.

[0125] Referring to Equation 1 and Equation 2, as the quantization parameter increases, the quantization step size may increase and the quantization error value may increase.

[0126] In an embodiment of the present disclosure, the quantization error map may include a quantization step size calculated based on a quantization parameter as a sample value.

[0127] Reference Figure 5 , when the reference block 510 includes 16 samples and a quantization parameter for the 16 samples is a1, the sample value of the quantization error map 530 may have a2 calculated based on a1.

[0128] Used for Figure 5 The quantization parameter of the reference block 510 may be set for the reference block 510, or may be set for an upper block (e.g., a slice including the reference block 510) of the reference block 510. In other words, the neural network setter 230 may obtain the sample value of the quantization error map 530 from the quantization parameter set for the reference block 510 or the quantization parameter set for the slice including the reference block 510.

[0129] When the quantization parameter for the reference block 510 is set for an upper layer block of the reference block (e.g., a slice including the reference block 510), the same quantization parameter may be applied to blocks included in the slice. For example, since a quantization error map of a block included in a slice may be obtained based on the quantization parameter set for the slice, the amount of information included in the bitstream or the number of pieces of information may be less than when the quantization parameter is set for each block or each sample.

[0130] Next, refer to Figure 6 , a quantization parameter for the reference block 510 - 1 may be set for each sample of the reference block 510 - 1 .

[0131] The neural network setter 230 may obtain a sample value of the quantization error map 530 - 1 from a quantization parameter for each sample of the reference block 510 - 1 .

[0132] For example, the neural network setter 230 may calculate the value of the upper left sample 631 of the quantization error graph 530-1 as a2 based on the quantization parameter a1 of the upper left sample 611 of the reference block 510-1, and may calculate the value of the sample 632 located to the right of the upper left sample 631 of the quantization error graph 530-1 as b2 based on the quantization parameter b1 of the sample 612 located to the right of the upper left sample 611 of the reference block 510-1.

[0133] When a quantization parameter is set for each sample of the reference block 510-1, the amount of information obtained from the bitstream to check the quantization parameter for the reference block 510-1 can be increased. However, since the final prediction block of the current block can be obtained using the reference block 510-1 with a small error, the number of bits used to express the residual block between the current block and the final prediction block can be reduced.

[0134] Next, refer to Figure 7 , the neural network setter 230 may divide the quantization error map 530-2 into sub-regions corresponding to the lower layer blocks, such as a first sub-region 750, a second sub-region 760, a third sub-region 770, and a fourth sub-region 780, and the lower layer blocks may include, for example, a first lower layer block 710, a second lower layer block 720, a third lower layer block 730, and a fourth lower layer block 740 of the reference block 510-2, and may calculate the sample point values ​​respectively included in the sub-regions 750 to 780 of the quantization error map 530-2 based on the quantization parameters of the samples at specific positions in the lower layer blocks 710, 720, 730, and 740 of the reference block 510-2.

[0135] In an embodiment of the present disclosure, when the reference block 510 - 2 corresponds to a coding unit, the lower layer blocks 710 , 720 , 730 , and 740 may correspond to prediction units.

[0136] In an embodiment of the present disclosure, the specific position may include an upper left position in the lower layer block.

[0137] like Figure 7 As shown, the sample values ​​of the first sub-region 750 of the quantization error map 530-2 may have a2 calculated based on the quantization parameter a1 for the sample 711 at the upper left position among the samples of the first lower layer block 710 in the reference block 510-2.

[0138] Also, the sample values ​​of the second sub-region 760 of the quantization error map 530-2 may have e2 calculated based on the quantization parameter e1 for the sample 721 at the upper left position among the samples of the second lower layer block 720 in the reference block 510-2.

[0139] In addition, the sample values ​​of the third sub-area 770 of the quantization error map 530-2 may have c2 calculated based on the quantization parameter c1 of the sample 731 at the upper left position among the samples of the third lower layer block 730 in the reference block 510-2, and the sample values ​​of the fourth sub-area 780 of the quantization error map 530-2 may have b2 calculated based on the quantization parameter b1 of the sample 741 at the upper left position among the samples of the fourth lower layer block 740 of the reference block 510-2.

[0140] For example, the sample at the specific position may include a sample at the center position in the lower layer block. When the width and height of the specific area are divided into two halves, the sample at the center position may refer to a sample at the lower left position, a sample at the upper left position, a sample at the lower right position, or a sample at the upper right position.

[0141] Figure 8 An example is shown in which a sample point located at a center position is a sample point located at a lower left position when the width and height of a specific area are divided into two halves.

[0142] like Figure 8 As shown, the sample values ​​of the first sub-region 850 of the quantization error map 530-3 may have a2 calculated based on the quantization parameter a1 for the sample 811 at the center position among the samples of the first lower layer block 710 in the reference block 510-2.

[0143] Also, the sample values ​​of the second sub-region 860 of the quantization error map 530-3 may have e2 calculated based on the quantization parameter e1 for the sample 821 at the center position among the samples of the second lower layer block 720 in the reference block 510-2.

[0144] In addition, the sample values ​​of the third sub-region 870 of the quantization error map 530-3 may have a2 calculated based on the quantization parameter a1 of the sample 831 at the center position among the samples of the third lower layer block 730 in the reference block 510-2, and the sample values ​​of the fourth sub-region 880 of the quantization error map 530-3 may have c2 calculated based on the quantization parameter c1 of the sample 841 at the center position among the samples of the fourth lower layer block 740 in the reference block 510-2.

[0145] Reference Figure 7 and Figure 8 The described samples at the upper left position and the samples at the center position are merely examples, and in an embodiment of the present disclosure, specific positions in the lower layer blocks 710, 720, 730, and 740 for obtaining sample values ​​of sub-regions of the quantization error graphs 530-2 and 530-3 may be changed in various ways.

[0146] When setting the quantization parameter for the reference block for each lower layer block or each sample, since the sample values ​​of the sub-regions of the quantization error maps 530-2 and 530-3 are calculated based on the quantization parameters of the samples at specific positions in the lower layer blocks 710, 720, 730, and 740 for the reference block 510-2, as shown in the reference block 510-2, Figure 7 and Figure 8 As in the embodiment of FIG. 5 , the quantization error graphs 530 - 2 and 530 - 3 can be obtained more quickly.

[0147] In an embodiment of the present disclosure, the neural network setter 230 may select different methods (e.g., Figures 5 to 8 Any one of the methods for obtaining a quantization error map) can be selected, and the quantization error map can be obtained according to the selected method.

[0148] As described above, the final prediction block may be obtained based on at least one of the preliminary prediction block, the quantization error map, or the POC map input to the neural network 240. Fig. 9 An example of the structure of the neural network 240 is described.

[0149] Fig. 9 is a diagram showing the structure of the neural network 240 according to an embodiment of the present disclosure.

[0150] like Fig. 9 As shown, the preliminary prediction block 902 , the quantization error map 904 , and the POC map 906 may be input to the first convolutional layer 910 .

[0151] In an embodiment of the present disclosure, the sizes of the preliminary prediction block 902 , the quantization error map 904 , and the POC map 906 may be the same as the size of the current block.

[0152] Because the POC map 906 may include the POC differences between the current image and the reference image as sample values, the sample values ​​in the POC map 906 may all be the same.

[0153] Fig. 9 The notation 6×5×5×32 shown in the first convolution layer 910 of may indicate that a convolution operation is performed on input data having six channels using 32 filter kernels of size 5×5. As a result of the convolution operation, 32 feature maps may be generated by the 32 filter kernels.

[0154] When the first convolution layer 910 processes input data having six channels, the process may be performed by considering that the current block is bidirectionally predicted.

[0155] In an embodiment of the present disclosure, when the current block is bidirectionally predicted, the first motion vector of list 0 and the second motion vector of list 1 can be derived for the current block, and the first reference image included in list 0 and the second reference image included in list 1 can be obtained as reference images of the current block.

[0156] In addition, a first preliminary prediction block may be obtained based on a first reference block in a first reference image indicated by a first motion vector of List 0, and a second preliminary prediction block may be obtained based on a second reference block in a second reference image indicated by a second motion vector of List 1. For example, as data input to the neural network 240, a first preliminary prediction block, a second preliminary prediction block, a first quantization error map including sample values ​​calculated based on a quantization parameter for the first reference block, a second quantization error map including sample values ​​calculated based on a quantization parameter for the second reference block, a first POC map including a POC difference between a current image and the first reference image, and a second POC map including a POC difference between the current image and the second reference image may be obtained.

[0157] The first convolution layer 910 may perform a convolution operation on the first preliminary prediction block, the second preliminary prediction block, the first quantization error map, the second quantization error map, the first POC map, and the second POC map using 32 filter kernels of a size of 5×5.

[0158] When unidirectional prediction (e.g., list 0 prediction or list 1 prediction) is applied to the current block, for example, when only the first motion vector of list 0 or only the second motion vector of list 1 is obtained for the current block, only input data having three channels may be obtained. Fig. 9 The number of channels processed by the first convolution layer 910 is 6, so input data having three channels can be increased to input data having six channels.

[0159] The neural network setter 230 may copy the first preliminary prediction block (or, for example, the second preliminary prediction block), the first quantization error map (or, for example, the second quantization error map), and the first POC map (or, for example, the second POC map) to obtain two first preliminary prediction blocks (or two second preliminary prediction blocks), two first quantization error maps (or two second quantization error maps), and two first POC maps (or two second POC maps), and may input input data having six channels to the neural network 240.

[0160] The feature map generated by the first convolutional layer 910 may represent unique features of the input data. For example, the feature map may represent features in the vertical direction, features in the horizontal direction, or edge features of the input data.

[0161] Reference Fig.10 An example of a convolution operation at the first convolution layer 910 is described in detail.

[0162] A feature map 1030 may be generated based on multiplication and addition between weights of a filter kernel 1010 of size 5×5 used in the first convolutional layer 910 and sample values ​​in the input data 1005 (eg, preliminary prediction block 902 ) corresponding to the weights.

[0163] Since 32 filter kernels may be used in the first convolution layer 910, 32 feature maps may be generated through a convolution operation using the 32 filter kernels.

[0164] Fig.10 Samples I1 to I49 shown in the input data 1005 in FIG. 1004 may represent samples of the input data 1005, and samples F1 to F25 shown in the filter kernel 1010 may represent samples of the filter kernel 1010. In addition, samples M1 to M9 shown in the feature map 1030 may represent samples of the feature map 1030.

[0165] During the convolution operation, multiplication may be performed between the values ​​of sample points I1 to I5, I8 to I12, I15 to I19, I22 to I26, and I29 to I33 of the input data 1005 and sample points F1 to F25 of the filter kernel 1010, and the value obtained by combining (e.g., adding) the results of the multiplication may be assigned as the value of M1 of the feature map 1030.

[0166] When the stride of the convolution operation is 1, multiplication can be performed between the values ​​of sample points I2 to I6, I9 to I13, I16 to I20, I23 to I27 and I30 to I34 of the input data 1005 and the sample points F1 to F25 of the filter kernel 1010, and the value obtained by combining the results of the multiplication can be assigned as the value of M2 of the feature map 1030.

[0167] When a convolution operation is performed between the values ​​of samples in the input data 1005 and the samples of the filter kernel 1010 while the filter kernel 1010 moves to the last sample in the input data 1005 with a stride, a feature map 1030 having a specific size may be obtained.

[0168] although Fig.10 The convolutional layers included in the neural network 240 are shown to operate according to a convolution operation, but with reference to Fig.10 The described convolution operation is merely an example, and embodiments of the present disclosure are not limited thereto.

[0169] Return to reference Fig. 9 , the feature map of the first convolutional layer 910 can be input to the first activation layer 920.

[0170] The first activation layer 920 may apply a nonlinear feature to each feature map. The first activation layer 920 may include, but is not limited to, at least one of a sigmoid function, a Tanh function, or a rectified linear unit (ReLU) function.

[0171] When the first activation layer 920 applies a nonlinear feature, this may mean that some sample point values ​​of the feature map are changed and output. In this case, the change may be performed by applying a nonlinear feature.

[0172] The first activation layer 920 may determine whether to transmit the sample values ​​of the feature map to the second convolution layer 930. For example, some of the sample values ​​of the feature map may be activated by the first activation layer 920 and may be transmitted to the second convolution layer 930, and other sample values ​​may be deactivated by the first activation layer 920 and may not be transmitted to the second convolution layer 930. Unique features of the input data indicated by the feature map may be emphasized by the first activation layer 920.

[0173] The feature map 925 output from the first activation layer 920 may be input to the second convolutional layer 930 . Fig. 9 Any one of the feature maps 925 may be a reference map at the first activation layer 920. Fig.10 The feature map 1030 is described as a result of the processing.

[0174] The notation 32×5×5×32 shown in the second convolution layer 930 may indicate that a convolution operation is performed on the feature map 925 having 32 channels using 32 filter kernels of size 5×5. The output of the second convolution layer 930 may be input to the second activation layer 940. The second activation layer 940 may apply a nonlinear feature to the input feature map.

[0175] The feature map 945 output from the second activation layer 940 may be input to the third convolution layer 950. The symbol 32×5×5×1 shown in the third convolution layer 950 may indicate that a convolution operation is performed on 32 feature maps 945 using a filter kernel of size 5×5 to generate a final prediction block 955.

[0176] although Fig. 9 The neural network 240 is shown to include three convolution layers (a first convolution layer 910, a second convolution layer 930, and a third convolution layer 950) and two activation layers (a first activation layer 920 and a second activation layer 940), but this is merely an example, and the embodiment is not limited thereto. For example, in the embodiment, the number of convolution layers and activation layers included in the neural network 240 may be changed in various ways.

[0177] In addition, according to an embodiment of the present disclosure, the neural network 240 may be implemented by a recurrent neural network (RNN). In this case, Fig. 9 The CNN structure of the neural network 240 can be changed to an RNN structure.

[0178] In an embodiment of the present disclosure, the image decoding apparatus 100 and the image encoding apparatus 1900 may include at least one arithmetic logic unit (ALU) for a convolution operation and an arithmetic operation of each activation operation.

[0179] The ALU may be implemented as, by, or using a processor. For a convolution operation, the ALU may include a multiplier and an adder, wherein the multiplier performs multiplication between the sample values ​​of the input data or the feature map output from the previous layer and the sample values ​​of the filter kernel, and the adder adds the results of the multiplication.

[0180] For arithmetic operations at each activation layer, the ALU may include a multiplier and a comparator, wherein the multiplier multiplies the input sample value by a predetermined weight used in at least one of a sigmoid function, a Tanh function, or a ReLU function, and the comparator compares the result of the multiplication with a specific value to determine whether to send the input sample value to the next layer.

[0181] according to Fig.10 When a filter kernel 1010 of size 5×5 is applied to input data 1005 of size 7×7, a feature map 1030 of size 3×3 is obtained.

[0182] Generally, when a convolution operation is performed on unpadded input data, output data having a size smaller than the input data is output. Therefore, in order to match the size of the final prediction block to the size of the current block, padding may be performed on the input data.

[0183] exist Fig.10 In the embodiment of FIG. 1005 , in order to obtain a feature map 1030 having the same size as the input data 1005 , padding with a distance of 2 may be performed in the left direction, the right direction, the upper direction, and the lower direction of the input data 1005 .

[0184] In an embodiment of the present disclosure, in order to obtain a final prediction block having the same size as that of the current block, the neural network setter 230 may fill at least one of the preliminary prediction block, the quantization error map, or the POC map, and may input at least one of the enlarged preliminary prediction block, the enlarged quantization error map, or the enlarged POC map obtained by filling into the neural network 240.

[0185] In an embodiment of the present disclosure, before performing a convolution operation on input data in each convolution layer of the neural network 240 or data output from a previous layer, the input data or data output from the previous layer may be added, so the size of the data before the convolution operation and the size of the data after the convolution operation may remain the same. Therefore, even when at least one of the enlarged preliminary prediction block, the enlarged quantization error map, or the enlarged POC map is input to the neural network 240, the size of the final prediction block output from the neural network 240 may be the same as the size of at least one of the enlarged preliminary prediction block, the enlarged quantization error map, or the enlarged POC map. In this case, in an embodiment of the present disclosure, the final prediction block may be cropped so that the size of the final prediction block output from the neural network 240 is the same as the size of the current block.

[0186] In an embodiment of the present disclosure, the neural network setter 230 may calculate the enlargement distance for padding based on the number of convolutional layers included in the neural network 240, the size of the filter kernel used in each convolutional layer, and the stride.

[0187] In an embodiment of the present disclosure, when the size of the kernel used in each convolutional layer included in the neural network 240 is k i (where i=0, 1, ..., L-1) and the stride in each convolutional layer is s i (where i=0, 1, ..., L-1), the magnification distance can be calculated according to the following equation 3: [Equation 3]

[0188]

[0189] In Equation 3, h may represent a zoom-in distance in a horizontal direction, v may represent a zoom-in distance in a vertical direction, M may represent a size of input data in a horizontal direction, and N may represent a size of input data in a vertical direction.

[0190] When the sizes of the filter kernels used in the convolutional layers included in the neural network 240 are all the same as k and the strides in the convolutional layers are all s, Equation 3 may be modified to Equation 4: [Equation 4]

[0191]

[0192] The neural network setter 230 may determine the magnification distance in the horizontal direction and the magnification distance in the vertical direction for filling the preliminary prediction block, the quantization error map, and the POC map based on Equation 3 or Equation 4, and may obtain the magnified preliminary prediction block, the magnified quantization error map, and the magnified POC map that also include samples corresponding to the magnification distance.

[0193] In an embodiment of the present disclosure, when the magnification distance in the horizontal direction and the magnification distance in the vertical direction are 1, the neural network setter 230 can obtain an enlarged preliminary prediction block by adding samples enlarged by the magnification distance of 1 in the left and right directions from the preliminary prediction block and adding samples enlarged by the magnification distance of 1 in the upper and lower directions of the preliminary prediction block.

[0194] In an embodiment of the present disclosure, in order to obtain an enlarged preliminary prediction block, an enlarged quantization error map, and an enlarged POC map, the neural network setter 230 may add a sample point of a predetermined value outside the boundaries of the preliminary prediction block, the quantization error map, and the POC map. For example, the predetermined value may be 0.

[0195] In an embodiment of the present disclosure, considering that the preliminary prediction block can be obtained from a reference block that is part of a reference image, the neural network setter 230 can consider neighboring samples of the reference block when filling the preliminary prediction block, the quantization error map, and the POC map. For example, the preliminary prediction block can be filled using samples adjacent to the reference block instead of using predetermined sample values, and the spatial characteristics of the reference image can also be considered when performing inter-frame prediction on the current block.

[0196] Fig.11 is a diagram for describing a method of obtaining an enlarged preliminary prediction block according to an embodiment of the present disclosure.

[0197] like Fig.11 As shown, when the magnification distance in the horizontal direction is h and the magnification distance in the vertical direction is v, the neural network setter 230 may obtain an enlarged preliminary prediction block including samples of the preliminary prediction block and samples 1120 corresponding to the magnification distance h among samples adjacent to the reference block 1110 in the reference image 1100. Therefore, the horizontal distance of the enlarged preliminary prediction block may be 2h greater than that of the preliminary prediction block, and the vertical distance of the enlarged preliminary prediction block may be 2v greater than that of the preliminary prediction block.

[0198] Fig.12 11 is a diagram for describing a method of obtaining an enlarged preliminary prediction block when a boundary of a reference block 1110 corresponds to a boundary of a reference image 1100 according to an embodiment of the present disclosure.

[0199] When the magnification distance in the horizontal direction and the magnification distance in the vertical direction are determined to be 3, the neural network setter 230 can obtain an enlarged preliminary prediction block, which includes samples in the preliminary prediction block and neighboring samples located outside the boundary of the reference block 1110 and located within the magnification distance 3.

[0200] like Fig.12 As shown, when the reference image 1100 is divided into six blocks (e.g., a left block 1210, a lower left block 1220, a lower block 1230, a lower right block 1240, a right block 1250, and a reference block 1110) and the block located at the upper center position is the reference block 1110, the neural network setter 230 may select a neighboring sample point that is located within an enlarged distance of 3 from the left boundary of the reference block 1110 and is located in the left block 1210 of the reference block 1110, a neighboring sample point that is located within an enlarged distance of 3 from the right boundary of the reference block 1110 and is located in the right block 1250 of the reference block 1110, and a neighboring sample point that is located within an enlarged distance of 3 from the lower boundary of the reference block 1110 and is located in the lower block 1230 of the reference block 1110. In this case, in order to determine the enlarged preliminary prediction block having a quadrilateral shape, neighboring samples located in a lower left block 1220 of the reference block 1110 and neighboring samples located in a lower right block 1240 of the reference block 1110 may also be selected.

[0201] In an embodiment of the present disclosure, when the boundary of the reference block 1110 matches the boundary of the reference image 1100, for example, when the upper boundary of the reference block 1110 matches the upper boundary of the reference image 1100, Fig.10 As shown, the neighboring sample points located outside the upper boundary of the reference block 1110 may not be included in the reference image 1100. Therefore, the neural network setter 230 may determine the neighboring sample points 1260 located outside the upper boundary of the reference block 1110 using the sample points in the reference image 1100 that are closest to each neighboring sample point 1260 located outside the upper boundary of the reference block 1110.

[0202] Reference Fig.10 , the value of the neighboring sample point located in the leftmost column among the neighboring sample points 1260 located outside the upper boundary of the reference block 1110 can be determined as the value a of the closest sample point in the neighboring block 1210, and the value of the neighboring sample point located in the rightmost column among the neighboring sample points 1260 can be determined as the value k of the closest sample point in the neighboring block 1250.

[0203] The neural network setter 230 may apply the enlarged preliminary prediction block having a size of 11×11 to the neural network 240, and may obtain a final prediction block having a size of 5×5 which is the same as the size of the current block, wherein the enlarged preliminary prediction block having a size of 11×11 may be larger than the reference block 1110 (and the preliminary prediction block) having a size of 5×5.

[0204] Since the size of the quantization error map input to the neural network 240 may be the same as the size of the enlarged preliminary prediction block, when the enlarged preliminary prediction block is input to the neural network 240, the neural network setter 230 may obtain an enlarged quantization error map having the same size as the size of the enlarged preliminary prediction block, an example of which is referred to as Fig.13 is described.

[0205] Fig.13 13 is a diagram for describing a method of obtaining an enlarged quantization error map 1300 according to an embodiment of the present disclosure.

[0206] Fig.13 A quantization error graph 530 - 4 including four samples and an enlarged quantization error graph 1300 enlarged according to an enlargement distance 1 are shown.

[0207] When quantization parameters for four samples included in the reference block are a1, b1, c1, and a1, respectively, the first to fourth samples 1301 to 1304 in the quantization error map 530-4 may have sample values ​​a2, b2, c2, and a2.

[0208] When samples in the preliminary prediction block and neighboring samples adjacent to the reference block constitute an enlarged preliminary prediction block, values ​​of neighboring samples outside the quantization error map 530-4 may be determined according to quantization parameters for neighboring samples outside the boundary of the reference block.

[0209] As in Fig.13 , a left sample 1310 of the first sample 1301 may have a value of e2, and a left sample 1315 of the third sample 1303 may have a value of a2. Here, e2 and a2 may be calculated based on a quantization parameter for a sample located to the left of a sample corresponding to the first sample 1301 in the reference block and a quantization parameter located to the left of a sample corresponding to the third sample 1303, respectively.

[0210] In addition, the lower left sample point 1320 of the third sample point 1303 may have a value of f2, the lower sample point 1325 of the third sample point 1303 may have a value of c2, and the lower sample point 1330 of the fourth sample point 1304 may have a value of a2. In addition, the lower right sample point 1335 of the fourth sample point 1304 may have a value of e2, the right sample point 1340 of the fourth sample point 1304 may have a value of d2, and the right sample point 1345 of the second sample point 1302 may have a value of e2.

[0211] As reference Fig.12 As described above, when a boundary (e.g., an upper boundary) of the reference block 1110 corresponds to a boundary of the reference image 1100, the neighboring samples 1260 located outside the boundary of the reference block 1110 may be determined from the closest samples available for the neighboring samples 1260. Similarly, the values ​​of the neighboring samples located outside the boundary of the quantization error map 530-4 may be determined as the closest samples available for the neighboring samples.

[0212] When the upper boundary of the reference block corresponds to the boundary of the reference image, such as Fig.13 As shown, the samples 1305, 1360, 1355, and 1350 located outside the upper boundary of the quantization error graph 530-4 may have values ​​of e2, a2, b2, and e2. For example, the sample 1305 located above the left of the first sample 1301 may be determined from the left sample 1310 closest to the first sample 1301, and the sample 1360 located above the first sample 1301 may be determined from the first sample 1301 closest to the first sample 1301. In addition, the sample 1355 located above the second sample 1302 may be determined from the second sample 1302 closest to the second sample 1302, and the sample 1350 located above the right of the second sample 1302 may be determined from the right sample 1345 closest to the second sample 1302.

[0213] In an embodiment of the present disclosure, since the sample values ​​in the POC map correspond to the POC difference between the current image and the reference image, when the POC map is filled according to the zoom distance, the sample values ​​in the zoomed POC map may all have the POC difference between the current image and the reference image.

[0214] In an embodiment of the present disclosure, when there is a reconstructed sample point before the current block in the current image, the block including the sample point may be further input to the neural network 240, referring to Fig.14 and Fig.15 An example of neural network 240 is described.

[0215] Fig.14 14 is a diagram showing reconstructed samples and non-reconstructed samples in the current image 1400 according to an embodiment of the present disclosure.

[0216] The current image 1400 may be reconstructed according to a predetermined scanning order. For example, blocks and samples in the current image 1400 may be reconstructed according to raster scanning.

[0217] like Fig.14 As shown, when the current image 1400 is reconstructed according to raster scanning, samples located on the left side of the current block 1410 and samples located above the current block 1410 may have been reconstructed before the current block 1410 is reconstructed.

[0218] Because the samples that have been reconstructed before the current block 1410 are spatially adjacent to the current block, information useful for generating a final prediction block similar to the current block 1410 can be provided to the neural network 240.

[0219] In an embodiment of the present disclosure, when the neural network setter 230 inputs at least one of the enlarged preliminary prediction block, the enlarged quantization error map, or the enlarged POC map to the neural network 240, the neural network setter 230 may also input an enlarged current reconstruction block having the same size as at least one of the enlarged preliminary prediction block, the enlarged quantization error map, or the enlarged POC map to the neural network 240.

[0220] The enlarged current reconstructed block may include samples corresponding to the enlargement distance among samples reconstructed before the current block 1410 .

[0221] Reference Fig.14 When the zoom-in distance is 1, the neural network setter 230 may use the sample points 1420 located at the zoom-in distance 1 from the boundary of the current block 1410 among the sample points reconstructed before the current block 1410 to obtain the zoomed-in current reconstructed block.

[0222] exist Fig.14 In the example shown, since the sample point 1420 corresponding to the enlarged distance 1 in the left direction and the upper direction of the current block 1410 has been reconstructed, but the sample point in the current block 1410 and the sample point 1430 corresponding to the enlarged distance 1 in the right direction and the lower direction of the current block 1410 have not been reconstructed, the neural network setter 230 can obtain the unreconstructed sample points based on the enlarged preliminary prediction block, and its example refers to Fig.15 is described.

[0223] Fig.15 15 is a diagram for describing a method of obtaining an enlarged current reconstructed block 1500 according to an embodiment of the present disclosure.

[0224] Reference Fig.15 , an enlarged current reconstructed block 1500 can be obtained, which includes the sample points 1420 that have been reconstructed before the current block 1410 and the samples 1125 of the enlarged preliminary prediction block 1150 except the samples corresponding to the sample points 1420 that have been reconstructed before the current block.

[0225] In an embodiment of the present disclosure, when there are two enlarged preliminary prediction blocks because the current block is bidirectionally predicted, Fig.15 The enlarged preliminary prediction block 1150 may include an average value of sample values ​​of the two enlarged preliminary prediction blocks as a sample value.

[0226] Because the enlarged current reconstruction block 1500 can be combined with the enlarged preliminary prediction block, the enlarged quantization error map and the enlarged POC Figure 1 The image is processed by the neural network 240 so that the spatial characteristics in the current image can also be taken into account.

[0227] In an embodiment of the present disclosure, the neural network setter 230 may select any one of a plurality of weight sets according to a specific criterion, and may allow the neural network 240 to operate according to the selected weight set.

[0228] Each of the plurality of weight sets may include weights used in operation processing of a layer included in the neural network 240 .

[0229] In an embodiment of the present disclosure, the neural network setter 230 may select a weight set to be used to obtain a final prediction block from a plurality of weight sets based on at least one of the size of the current block, the prediction direction of the current block, the quantization parameter used for the reference block, the layer to which the current image belongs in the image hierarchy, or information obtained from a bitstream.

[0230] Reference Fig.16 , the neural network setter 230 can obtain a final predicted block that is more similar to the current block by setting a weight set indicated by an index obtained from the bitstream among weight set A, weight set B, and weight set C to the neural network 240.

[0231] although Fig.16 An example of selecting a weight set based on the obtained information is shown, but the embodiment is not limited thereto. For example, in an embodiment of the present disclosure, a weight set to be used to obtain a final prediction block may be selected from a plurality of weight sets based on a result of comparing the size of the current block with a predetermined threshold.

[0232] For example, when the size of the current block is equal to or greater than 64×64, weight set C may be selected; when the size of the current block is equal to or greater than 16×16 and less than 64×64, weight set B may be selected; and when the size of the current block is less than 16×16, weight set C may be selected.

[0233] For example, when the current image corresponds to layer 1 in the image hierarchy, weight set A may be selected; when the current image corresponds to layer 2, weight set B may be selected; and when the current image corresponds to layer 3, weight set C may be selected.

[0234] Each of the plurality of weight sets may be generated as a result of training the neural network 240. For example, the weights may be obtained by training the neural network 240 according to different training purposes or objectives. Fig.16 The weight sets A, B, and C of the neural network 240 may be changed, for example, by using different types of training images to train the neural network 240, or by calculating the loss information in different ways.

[0235] For example, in the following description Fig.23 In the process of training the neural network 240 shown in , loss information 2306 corresponding to the difference between the final prediction block 2305 for training and the current block 2301 for training may be used. Since there are many methods of calculating the difference between two blocks, weight set A may be generated by training the neural network 240 based on the loss information calculated according to the first method, and weight set B may be generated by training the neural network 240 based on the loss information calculated according to the second method. In addition, weight set C may be generated by training the neural network 240 based on the loss information calculated according to the third method.

[0236] In an embodiment of the present disclosure, the neural network setter 230 may select a neural network to be used to obtain a final prediction block from a plurality of neural networks, and may obtain a final prediction block of a current block by applying input data (e.g., a preliminary prediction block) to the selected neural network. A plurality of neural networks may be included in the AI-based prediction decoder 132.

[0237] The multiple neural networks may differ from each other in at least one of the type of layers, the number of layers, the size of the filter kernel, or the stride.

[0238] In an embodiment of the present disclosure, the neural network setter 230 may select a neural network to be used to obtain a final prediction block from a plurality of neural networks based on at least one of the size of the current block, the prediction direction of the current block, a quantization parameter for a reference block, a layer to which the current image belongs in an image hierarchy, or information obtained from a bitstream.

[0239] In an embodiment of the present disclosure, the neural network setter 230 may determine whether to apply a neural network-based prediction mode based on at least one of information obtained from a bitstream, a prediction direction of a current block, or whether an enlarged preliminary prediction block is outside a boundary of a reference image.

[0240] For example, when the information obtained from the bitstream indicates that the neural network-based prediction mode is not applied, the preliminary prediction block obtained by the prediction block obtainer 220 may be transmitted to the reconstruction unit 134 .

[0241] In addition, for example, when the enlarged preliminary prediction block is outside the boundary of the reference image, the neural network setter 230 may determine that the neural network-based prediction mode is not applied to the current block, and when the enlarged preliminary prediction block is located within the reference image, for example, the boundary of the reference block does not correspond to the boundary of the reference image, the neural network setter 230 may determine that the neural network-based prediction mode is applied to the current block.

[0242] In addition, for example, when the prediction direction of the current block is bidirectional, such as when the motion vector of list 0 and the motion vector of list 1 are obtained for the current block, the neural network setter 230 may determine that the neural network-based prediction mode is applied to the current block, and when the prediction direction of the current block is unidirectional, the neural network setter 230 may determine that the neural network-based prediction mode is not applied to the current block.

[0243] Fig.17 is a diagram illustrating an image decoding method performed by the image decoding apparatus 100 according to an embodiment of the present disclosure.

[0244] In operation S1710, the image decoding apparatus 100 may obtain a motion vector of a current block.

[0245] In an embodiment of the present disclosure, the image decoding device 100 may obtain a motion vector of a current block according to a rule-based prediction mode. The rule-based prediction mode may include a merge mode, a skip mode, an AMVP mode, a BDOF mode, a BCW mode, or a DMVR mode. The prediction mode information included in the bitstream may be used to determine which of several rule-based prediction modes should be used to obtain the motion vector of the current block.

[0246] In an embodiment of the present disclosure, the image decoding apparatus 100 may obtain a motion vector of a current block using information (eg, a flag or an index) included in a bitstream.

[0247] In an embodiment of the present disclosure, the image decoding apparatus 100 may construct a motion vector candidate list including spatially neighboring blocks and / or temporally neighboring blocks of a current block as motion vector candidates to obtain a motion vector of the current block.

[0248] In operation S1720, the image decoding apparatus 100 may obtain a preliminary prediction block using a motion vector of the current block and a reference image of the current block.

[0249] In an embodiment of the present disclosure, the image decoding apparatus 100 may obtain a preliminary prediction block using a reference block indicated by a motion vector of a current block in a reference image.

[0250] In an embodiment of the present disclosure, a process of obtaining a preliminary prediction block similar to a current block from a reference image using a motion vector may be referred to as a motion compensation process.

[0251] In an embodiment of the present disclosure, the preliminary prediction block may correspond to a result of applying interpolation to a reference block indicated by a motion vector of the current block in a reference image. Accordingly, the preliminary prediction block may include sub-pixels obtained by applying filtering to integer pixels.

[0252] In operation S1730, the image decoding apparatus 100 may obtain a final prediction block of the current block by applying at least one of a POC map including a POC difference between a current image including the current block and a reference image, a preliminary prediction block, or a quantization error map to the neural network 240.

[0253] In an embodiment of the present disclosure, the sample values ​​of the quantization error map may be calculated based on the quantization parameter for the reference block.

[0254] In an embodiment of the present disclosure, the image decoding apparatus 100 may fill at least one of the POC map, the preliminary prediction block, or the quantization error map, and may obtain a final prediction block of the current block by applying at least one of the enlarged POC map, the enlarged preliminary prediction block, or the enlarged quantization error map obtained by filling to the neural network 240. In an embodiment of the present disclosure, the image decoding apparatus 100 may also input the enlarged current reconstructed block to the neural network 240.

[0255] In operation S1740, the image decoding apparatus 100 may reconstruct the current block using the residual block obtained from the bitstream and the final prediction block.

[0256] In an embodiment of the present disclosure, the image decoding apparatus 100 may obtain the current block by adding the sample value of the final prediction block to the sample value of the residual block.

[0257] In an embodiment of the present disclosure, the image decoding apparatus 100 may obtain information about a quantized transform coefficient from a bitstream, and may obtain a residual block by applying inverse quantization and inverse transformation to the quantized transform coefficient.

[0258] The image including the reconstructed current block may be used to decode the next block.

[0259] Fig.18 is a diagram illustrating syntax according to an embodiment of the present disclosure.

[0260] In an embodiment of the present disclosure, a neural network-based prediction mode may be used together with a skip mode, a merge mode, or an AMVP mode.

[0261] Reference Fig.18 , in S1810 , when the skip mode is applied to the current block (inter_skip=1), merge_data() may be called, and NNinter() may be called in S1840 in merge_data() to apply the neural network-based prediction mode to the current block.

[0262] Furthermore, in S1820 , when the merge mode is applied to the current block (merge_flag=1), merge_data() may be called, and NNinter() may be called in S1840 in merge_data() to apply the neural network-based prediction mode to the current block.

[0263] Furthermore, in S1830, when the AMVP mode is applied to the current block, NNinter() may be called to apply a neural network-based prediction mode to the current block.

[0264] According to an embodiment of the present disclosure, since a motion vector of a current block obtained according to an existing skip mode, merge mode, or AMVP mode is adjusted based on the neural network 240 for each sample, a final prediction block more similar to the current block may be obtained.

[0265] Fig.19 19 is a diagram showing a configuration of an image encoding device 1900 according to an embodiment of the present disclosure.

[0266] Reference Fig.19 , the image encoding apparatus 1900 may include an encoder 1910 and a bitstream generator 1930. The encoder 1910 may include an AI-based prediction encoder 1912 and a residual data obtainer 1914.

[0267] In an embodiment of the present disclosure, the encoder 1910 may be Fig.24 The prediction encoder 2415, the transform and quantization unit 2420, the inverse quantization and inverse transform unit 2430, the deblocking filter unit 2435 and the loop filter unit 2440 correspond to the bitstream generator 1930. In an embodiment of the present disclosure, the bitstream generator 1930 may be connected to Fig.24 The entropy encoder 2425 corresponds to .

[0268] The encoder 1910 and the bitstream generator 1930 may be implemented as, by, or using at least one processor. The encoder 1910 and the bitstream generator 1930 may operate according to at least one instruction stored in at least one memory.

[0269] Despite Fig.19The encoder 1910 and the bitstream generator 1930 are shown separately in the figure, but the embodiment is not limited thereto. For example, the encoder 1910 and the bitstream generator 1930 can be implemented by one processor. In this case, the encoder 1910 and the bitstream generator 1930 can be implemented as a special processor, implemented by a special processor, or implemented using a special processor, or can be implemented by a combination of software and a general-purpose processor (such as an AP, a CPU, or a GPU). In addition, when the encoder 1910 and the bitstream generator 1930 are implemented as a special processor, implemented by a special processor, or implemented using a special processor, the special processor may include a memory for implementing the embodiments of the present disclosure or a memory processor for using an external memory.

[0270] In an embodiment of the present disclosure, the encoder 1910 and the bitstream generator 1930 may include multiple processors. In this case, the encoder 1910 and the bitstream generator 1930 may be implemented by a combination of dedicated processors or a combination of software and multiple general-purpose processors such as AP, CPU, and GPU.

[0271] The encoder 1910 may encode the current block using the reference image of the current block. Information about the residual block and information about the motion vector may be output as a result of encoding the current block.

[0272] In an embodiment of the present disclosure, the encoder 1910 may not output information about the residual block according to a rule-based encoding mode (eg, skip mode) for the current block.

[0273] In an embodiment of the present disclosure, the AI-based prediction encoder 1912 may use the current block and the reference image to obtain a final prediction block of the current block. The final prediction block may be sent to the residual data obtainer 1914.

[0274] The residual data obtainer 1914 may obtain a residual block corresponding to a difference between the current block and the final prediction block.

[0275] In an embodiment of the present disclosure, the residual data obtainer 1914 may obtain a residual block by subtracting the sample value of the final prediction block from the sample value of the current block.

[0276] In an embodiment of the present disclosure, the residual data obtainer 1914 may obtain a quantized transform coefficient by applying transform and quantization to samples of the residual block.

[0277] The information about the residual block and the information about the motion vector obtained by the encoder 1910 may be transmitted to the bitstream generator 1930 .

[0278] In an embodiment of the present disclosure, the information about the residual block may include information about the quantized transform coefficient (eg, a flag indicating whether the quantized transform coefficient is 0).

[0279] In an embodiment of the present disclosure, the information about the motion vector may include information indicating one or more motion vector candidates included in the motion vector candidate list, for example, a flag or an index.

[0280] In an embodiment of the present disclosure, the information about the motion vector may include a differential motion vector between a motion vector of the current block and a predicted motion vector.

[0281] The bitstream generator 1930 may generate a bitstream including a result of encoding the current block.

[0282] The bitstream may be transmitted through the network to the image decoding apparatus 100. In an embodiment of the present disclosure, the bitstream may be stored in any data storage medium including a magnetic medium (such as a hard disk, a floppy disk, or a magnetic tape), an optical recording medium (such as a CD-ROM or a DVD), and a magneto-optical medium (such as a floppy disk).

[0283] In an embodiment of the present disclosure, the bitstream generator 1930 may generate a bitstream by entropy encoding a syntax element corresponding to a result of encoding an image.

[0284] Fig. 20 is a diagram showing a configuration of an AI-based predictive encoder 1912 according to an embodiment of the present disclosure.

[0285] Reference Fig. 20 , the AI-based prediction encoder 1912 may include a motion information obtainer 2010, a prediction block obtainer 2020, a neural network setter 2030 and a neural network 2040.

[0286] The neural network 2040 may be stored in a memory. In an embodiment of the present disclosure, the neural network 2040 may be implemented as, by, or using an AI processor.

[0287] The motion information obtainer 2010 may search for a block similar to the current block in the reference image, and may obtain a motion vector indicating the searched block.

[0288] In an embodiment of the present disclosure, in order to encode the motion vector of the current block, the motion information obtainer 2010 may construct a motion vector candidate list including the motion vector of at least one of the spatial neighboring blocks or the temporal neighboring blocks of the current block as a motion vector candidate.

[0289] In an embodiment of the present disclosure, the motion vector obtained by the motion information obtainer 2010 may include the motion vector of list 0, the motion vector of list 1, or the motion vector of list 0 and the motion vector of list 1.

[0290] The motion vector of list 0 may be a motion vector indicating a reference block in a reference image included in list 0 (or reference image list 0), and the motion vector of list 1 may be a motion vector indicating a reference block in a reference image included in list 1 (or reference image list 1).

[0291] The prediction block obtainer 2020 may obtain a preliminary prediction block using a reference block indicated by a motion vector in a reference image.

[0292] In an embodiment of the present disclosure, the preliminary prediction block may be obtained by applying interpolation to a reference block indicated by a motion vector in a reference image. In this case, the preliminary prediction block may include sub-pixels obtained by filtering integer pixels.

[0293] In an embodiment of the present disclosure, the motion information obtainer 2010 and the prediction block obtainer 2020 may obtain a preliminary prediction block according to a rule-based prediction mode.

[0294] As described above, when the motion vector of List 0 is obtained by the motion information obtainer 2010 , the prediction block obtainer 2020 may obtain a preliminary prediction block using a reference block indicated by the motion vector of List 0 in a reference image included in List 0 .

[0295] In an embodiment of the present disclosure, when the motion vector of List 1 is obtained by the motion information obtainer 2010 , the prediction block obtainer 2020 may obtain a preliminary prediction block using a reference block indicated by the motion vector of List 1 in a reference image included in List 1 .

[0296] In an embodiment of the present disclosure, when the motion vector of list 0 and the motion vector of list 1 are obtained by the motion information obtainer 2010, the prediction block obtainer 2020 may use the reference block indicated by the motion vector of list 0 in the reference image included in list 0 to obtain the preliminary prediction block of list 0, and may use the reference block indicated by the motion vector of list 1 in the reference image included in list 1 to obtain the preliminary prediction block of list 1.

[0297] The neural network setter 2030 may obtain data to be input to the neural network 2040 .

[0298] In an embodiment of the present disclosure, the neural network setter 2030 may obtain data to be input to the neural network 2040 based on a reference image, a preliminary prediction block, and a quantization parameter for the reference block.

[0299] In an embodiment of the present disclosure, when at least one of the preliminary prediction block, the POC map, or the quantization error map is applied to the neural network 2040 by the neural network setter 2030, a final prediction block of the current block may be obtained.

[0300] Neural network setter 2030 and neural network 2040 may be used with Figure 2 The neural network setter 230 and the neural network 240 included in the AI-based prediction decoder 132 are the same or similar, and therefore, redundant or repeated descriptions thereof may be omitted.

[0301] In an embodiment of the present disclosure, the neural network setter 2030 may select different methods for obtaining the quantization error map (e.g., Figures 5 to 8 Any one of the methods for obtaining a quantization error map) can be selected, and the quantization error map can be obtained according to the selected method.

[0302] In an embodiment of the present disclosure, the neural network setter 2030 may select different methods for obtaining the quantization error map based on the cost (e.g., Figures 5 to 8 During cost calculation, a rate-distortion cost may be used.

[0303] In an embodiment of the present disclosure, the motion information obtainer 2010 may change the precision of the motion vector of the current block from fractional precision to integer precision, and the prediction block obtainer 2020 may obtain a preliminary prediction block according to the motion vector with integer precision. In an embodiment of the present disclosure, a reference block indicated by a motion vector with integer precision in a reference image may be determined as a preliminary prediction block.

[0304] When the precision of the motion vector of the current block is changed to integer precision, the number of bits used to express the motion vector of the current block may be reduced, and thus, the bit rate of a bitstream may be reduced.

[0305] Another reason why the precision of the motion vector of the current block may be changed from fractional precision to integer precision may be that when information about a motion vector with low precision is intentionally provided to the neural network 2040, the neural network 2040 may derive an accurate motion vector from an inaccurate motion vector.

[0306] The precision change of the motion vector according to an embodiment of the present disclosure may be related to the AMVR mode included in the Versatile Video Coding (VVC) standard. The AMVR mode may be a mode for adaptively selecting and using a residual motion vector and a resolution of a motion vector.

[0307] In the AMVR mode, the motion vector and the residual motion vector may generally be encoded / decoded using any one of the resolutions of 1 / 4 pixel, 1 / 2 pixel, 1 pixel, and 4 pixels.

[0308] When a current block is encoded / decoded using a 1-pixel or 4-pixel motion vector in AMVR mode, a final prediction block, such as a final prediction block generated by a neural network-based prediction mode according to a higher resolution motion vector, may be anticipated.

[0309] In addition, when the current block is encoded / decoded using a motion vector of 1 / 4 pixel or 1 / 2 pixel in the AMVR mode, since 1 / 4 pixel or 1 / 2 pixel is changed to 1 pixel in the neural network based prediction mode, the number of bits required to signal the motion vector can be reduced, and therefore, a final prediction block with high quality can be obtained by processing the preliminary prediction block through the neural network 2040.

[0310] Reference Fig.21 An example of a method by which the motion information obtainer 2010 changes the precision of the motion vector of the current block from fractional precision to integer precision is described.

[0311] Fig.21 is a diagram for describing a method of changing a motion vector of fractional precision into a motion vector of integer precision according to an embodiment of the present disclosure.

[0312] In an embodiment of the present disclosure, when the precision of the motion vector A of the current block is fractional precision (eg, 1 / 2 pixel, 1 / 4 pixel, or 1 / 8 pixel), the motion information obtainer 2010 may change the motion vector A of the current block to indicate an integer pixel.

[0313] exist Fig.21 In the example shown in , the motion vector A may indicate the coordinates (19 / 4, 27 / 4) 2110 based on the coordinates (0,0). Since the coordinates (19 / 4, 27 / 4) 2110 are not integer pixels, the motion information obtainer 2010 may adjust the motion vector A to indicate an integer pixel.

[0314] The coordinates of the neighboring integer pixels around the coordinate (19 / 4, 27 / 4) 2110 may be (16 / 4, 28 / 4) 2130, (16 / 4, 24 / 4) 2120, (20 / 4, 28 / 4) 2140, and (20 / 4, 24 / 4) 2150. In this case, the motion information obtainer 2010 may change the motion vector A to indicate the coordinate (20 / 4, 28 / 4) 2140 located at the upper right end, instead of the coordinate (19 / 4, 27 / 4) 2110.

[0315] In an embodiment of the present disclosure, the motion information obtainer 2010 may change the motion vector A to indicate the coordinate 2120 located at the lower left end, the coordinate 2130 located at the upper left end, or the coordinate 2150 located at the lower right end.

[0316] The method of changing the fractional precision of the motion vector A to integer precision may be referred to as motion vector rounding.

[0317] Fig. 22 is a diagram illustrating an image encoding method performed by the image encoding apparatus 1900 according to an embodiment of the present disclosure.

[0318] In operation S2210, the image encoding apparatus 1900 may obtain a motion vector of a current block using a reference image.

[0319] In an embodiment of the present disclosure, the image encoding apparatus 1900 may obtain a motion vector indicating a block similar to a current block in a reference image.

[0320] In an embodiment of the present disclosure, a process of obtaining a motion vector of a current block using a reference image may be referred to as a motion prediction process.

[0321] In operation S2220, the image encoding apparatus 1900 may obtain a preliminary prediction block using a motion vector of the current block and a reference image of the current block.

[0322] In an embodiment of the present disclosure, the image encoding apparatus 1900 may obtain a preliminary prediction block using a reference block indicated in a reference image by a motion vector of a current block.

[0323] In an embodiment of the present disclosure, a process of obtaining a preliminary prediction block similar to a current block from a reference image may be referred to as a motion compensation process.

[0324] In an embodiment of the present disclosure, the preliminary prediction block may correspond to a result of applying interpolation to a reference block indicated by a motion vector of the current block in a reference image.

[0325] In operation S2230, the image encoding apparatus 1900 may obtain a final prediction block of the current block by applying at least one of a POC map, a preliminary prediction block, or a quantization error map to the neural network 2040, the POC map including a POC difference between a current image including the current block and a reference image as a sample value.

[0326] In an embodiment of the present disclosure, the sample values ​​of the quantization error map may be calculated based on the quantization parameter for the reference block.

[0327] In an embodiment of the present disclosure, the image encoding apparatus 1900 may fill at least one of the POC map, the preliminary prediction block, or the quantization error map, and may obtain a final prediction block of the current block by applying at least one of the enlarged POC map, the enlarged preliminary prediction block, or the enlarged quantization error map obtained by filling to the neural network 2040. In this case, in an embodiment of the present disclosure, the image encoding apparatus 1900 may also input the enlarged current reconstructed block to the neural network 2040.

[0328] In operation S2240, the image encoding apparatus 1900 may obtain a residual block using the current block and the final prediction block.

[0329] In an embodiment of the present disclosure, the image encoding apparatus 1900 may obtain a residual block by subtracting a sample value of a final prediction block from a sample value of a current block.

[0330] In an embodiment of the present disclosure, the image encoding apparatus 1900 may obtain a quantized transform coefficient by applying transform and quantization to samples of the residual block.

[0331] In operation S2250, the image encoding apparatus 1900 may generate a bitstream including information about the residual block.

[0332] In an embodiment of the present disclosure, the information about the residual block may include information about the quantized transform coefficient (eg, a flag indicating whether the quantized transform coefficient is 0).

[0333] In an embodiment of the present disclosure, the bitstream may further include information about a motion vector of the current block. The information about the motion vector of the current block may include information indicating one or more motion vector candidates included in the motion vector candidate list, for example, a flag or an index.

[0334] In an embodiment of the present disclosure, the information about the motion vector may include a differential motion vector between a motion vector of the current block and a predicted motion vector.

[0335] In an embodiment of the present disclosure, the bitstream may not include information about the residual block. For example, when the rule-based prediction mode applied to the current block is the skip mode, the bitstream may not include information about the residual block for the current block.

[0336] Reference Fig.23 An example of a method of training the neural network 240 used by at least one of the image decoding apparatus 100 or the image encoding apparatus 1900 is described.

[0337] Fig.23 2 is a diagram for describing a method of training a neural network 240 according to an embodiment of the present disclosure.

[0338] Fig.23 The current block 2301 for training may correspond to the above-mentioned current block. In addition, the preliminary prediction block 2302 for training, the quantization error map 2303 for training, and the POC map 2304 for training may correspond to Figure 2 and Fig. 20 The preliminary prediction block 2302 for training may correspond to a block identified in a reference image for training by a motion vector of the current block 2301 for training.

[0339] Based on the method of training the neural network 240 according to the present disclosure, the neural network 240 may be trained so that the final prediction block 2305 for training output from the neural network 240 is the same as or similar to the current block 2301 for training. To this end, the neural network 240 may be trained using loss information 2306 corresponding to the difference between the final prediction block 2305 for training and the current block 2301 for training.

[0340] Reference Fig.23 An example of a process of training the neural network 240 is described in detail. First, a preliminary prediction block 2302 for training, a quantization error map 2303 for training, and a POC map 2304 for training may be input to the neural network 240, and a final prediction block 2305 for training may be output from the neural network 240. The neural network 240 may operate according to a preset weight.

[0341] Loss information 2306 corresponding to the difference between the final prediction block 2305 for training and the current block 2301 for training may be calculated, and a weight set in the neural network 240 may be updated according to the loss information 2306. The neural network 240 may update the weights to reduce or minimize the loss information 2306.

[0342] The loss information 2306 may include at least one of an L1 norm value, an L2 norm value, a structural similarity index metric (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, or a video multi-method assessment fusion (VMAF) value, which indicates the difference between the current block 2301 used for training and the final prediction block 2305 used for training.

[0343] The neural network 240 according to an embodiment of the present disclosure may be trained by a training device. The training device may be the image decoding device 100 or the image encoding device 1900. According to an embodiment, the training device may be an external server. In this case, the trained neural network 240 and weights trained by the external server may be sent to the image decoding device 100 and the image encoding device 1900.

[0344] The purpose of the image decoding device 100 and the image encoding device 1900 and the method thereof using AI according to an embodiment of the present disclosure may be to obtain a final prediction block 955 that is more similar to the current block 300 or 1410 compared to the existing rule-based prediction mode.

[0345] In addition, the image decoding apparatus 100 and the image encoding apparatus 1900 using AI and methods thereof according to an embodiment of the present disclosure may be intended to reduce a bit rate of a bitstream including information about a residual block.

[0346] The method of decoding an image according to an embodiment of the present disclosure may include obtaining a motion vector of a current block (300; 1410) (S1710).

[0347] In an embodiment of the present disclosure, the method of decoding an image may include: obtaining a preliminary prediction block (902) using a reference block (415; 435; 510; 510-1; 510-2; 1110) indicated by a motion vector in a reference image (410; 430; 1100) (S1720).

[0348] In an embodiment of the present disclosure, a method for decoding an image may include: obtaining a final prediction block (955) (S1730) of a current block (300; 1410) by applying a picture order count (POC) map (906), a preliminary prediction block (902), or a quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) to a neural network (240), wherein the POC map (906) includes a POC difference between a reference image (410; 430; 1100) and a current image (400; 1400) including the current block (300; 1410).

[0349] In an embodiment of the present disclosure, the method of decoding an image may include: reconstructing a current block (300; 1410) based on a final prediction block (955) and a residual block obtained from a bitstream (S1740).

[0350] In an embodiment of the present disclosure, sample values ​​of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) may be calculated based on a quantization parameter for a reference block (415; 435; 510; 510-1; 510-2; 1110).

[0351] In an embodiment of the present disclosure, the sample value of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) may correspond to a quantization step size or a quantization error value calculated based on a quantization parameter.

[0352] In an embodiment of the present disclosure, the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) may be divided into sub-regions corresponding to the lower layer blocks of the reference block (415; 435; 510; 510-1; 510-2; 1110), and sample values ​​respectively included in the sub-regions of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) may be calculated based on quantization parameters for samples at positions in the lower layer blocks of the reference block (415; 435; 510; 510-1; 510-2; 1110).

[0353] In an embodiment of the present disclosure, obtaining a final prediction block (955) of a current block (300; 1410) may include obtaining a final prediction block (955) of the current block (300; 1410) by applying at least one of an enlarged POC map, an enlarged preliminary prediction block (1150), or an enlarged quantization error map (1300) to a neural network (240).

[0354] In an embodiment of the present disclosure, at least one of the enlarged POC map, the enlarged preliminary prediction block (1150) or the enlarged quantization error map (1300) can be obtained by filling at least one of the POC map (906), the preliminary prediction block (902) or the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) according to the enlargement distance.

[0355] In an embodiment of the present disclosure, a neural network may include one or more convolutional layers.

[0356] In an embodiment of the present disclosure, the method for decoding an image may further include: determining a zoom-in distance based on the number of one or more convolutional layers included in the neural network (240), the size of a filter kernel used in each convolutional layer, and a stride, and obtaining an enlarged preliminary prediction block (1150), the enlarged preliminary prediction block (1150) including neighboring samples corresponding to the zoom-in distance among neighboring samples outside a boundary of a reference block (415; 435; 510; 510-1; 510-2; 1110) in the reference image (410; 430; 1100) and samples of the preliminary prediction block (902).

[0357] In an embodiment of the present disclosure, the method for decoding an image may further include: obtaining an enlarged quantization error map (1300), the enlarged quantization error map (1300) including sample values ​​calculated based on quantization parameters for neighboring samples corresponding to the enlarged distance in the reference image (410; 430; 1100) and quantization parameters for the reference block (415; 435; 510; 510-1; 510-2; 1110).

[0358] In an embodiment of the present disclosure, based on a boundary of a reference block (415; 435; 510; 510-1; 510-2; 1110) corresponding to a boundary of a reference image (410; 430; 1100), a neighboring sample point corresponding to a zoom-in distance may be determined from the closest sample points available in the reference image (410; 430; 1100).

[0359] In an embodiment of the present disclosure, obtaining a final prediction block (955) of a current block (300; 1410) may include applying an enlarged current reconstructed block (1500) together with at least one of an enlarged POC map, an enlarged preliminary prediction block (1150), or an enlarged quantization error map (1300) to a neural network (240).

[0360] In an embodiment of the present disclosure, the enlarged current reconstructed block (1500) may include neighboring samples (1420) reconstructed before the current block (300; 1410) in the current image (400; 1400) and samples (1125) of the enlarged preliminary prediction block (1150) except for samples corresponding to the neighboring samples (1420) reconstructed before the current block (300; 1410).

[0361] In an embodiment of the present disclosure, the method for decoding an image may further include: selecting a weight set to be used to obtain a final prediction block (955) from a plurality of weight sets based on the size of the current block (300; 1410), the prediction direction of the current block (300; 1410), a quantization parameter for a reference block (415; 435; 510; 510-1; 510-2; 1110), the layer to which the current image (400; 1400) belongs in an image hierarchy, or at least one of information obtained from a bitstream.

[0362] In an embodiment of the present disclosure, a final prediction block (955) may be obtained based on the neural network (240) operating according to a selected set of weights.

[0363] In an embodiment of the present disclosure, a method for encoding an image according to an embodiment of the present disclosure may include obtaining a motion vector (S2210) indicating a reference block (415; 435; 510; 510-1; 510-2; 1110) corresponding to a current block (300; 1410) in a reference image (410; 430; 1100).

[0364] In an embodiment of the present disclosure, a method for encoding an image may include obtaining a final prediction block (955) of a current block (300; 1410) by applying a POC map (906) including a picture order count (POC) difference between a reference image (410; 430; 1100) and a current image (400; 1400) including a current block (300; 1410) to a neural network (2040) (S2230), a preliminary prediction block (902) obtained based on a reference block (415; 435; 510; 510-1; 510-2; 1110), or a quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904).

[0365] In an embodiment of the present disclosure, the method of encoding an image may include obtaining a residual block (S2240) based on a current block (300; 1410) and a final prediction block (955).

[0366] In an embodiment of the present disclosure, the method of encoding an image may include generating a bitstream including information about a residual block ( S2250 ).

[0367] In an embodiment of the present disclosure, sample values ​​of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) may be calculated based on a quantization parameter for a reference block (415; 435; 510; 510-1; 510-2; 1110).

[0368] In an embodiment of the present disclosure, the method of encoding an image may further include: changing the precision of the obtained motion vector from fractional precision to integer precision.

[0369] In an embodiment of the present disclosure, a reference block ( 415 ; 435 ; 510 ; 510 - 1 ; 510 - 2 ; 1110 ) indicated by a motion vector having integer precision may be determined as a preliminary prediction block ( 902 ).

[0370] In an embodiment of the present disclosure, obtaining a final prediction block (955) of a current block (300; 1410) may include obtaining a final prediction block (955) of the current block (300; 1410) by applying at least one of an enlarged POC map, an enlarged preliminary prediction block (1150), or an enlarged quantization error map (1300) to a neural network (2040).

[0371] In an embodiment of the present disclosure, at least one of the enlarged POC map, the enlarged preliminary prediction block (1150) or the enlarged quantization error map (1300) can be obtained by filling at least one of the POC map (906), the preliminary prediction block (902) or the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) according to the enlargement distance.

[0372] An image decoding apparatus according to an embodiment of the present disclosure may include at least one memory configured to store at least one instruction, and at least one processor configured to execute the at least one instruction.

[0373] In an embodiment of the present disclosure, the at least one processor of the image decoding device may be configured to obtain a motion vector of the current block (300; 1410).

[0374] In an embodiment of the present disclosure, the at least one processor of the image decoding device may be configured to obtain a preliminary prediction block (902) based on a reference block (415; 435; 510; 510-1; 510-2; 1110) indicated by a motion vector in a reference image (410; 430; 1100).

[0375] In an embodiment of the present disclosure, the at least one processor of the image decoding device may be configured to obtain a final prediction block (955) of a current block (300; 1410) by applying a picture order count (POC) map (906), a preliminary prediction block (902), or a quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) to a neural network (240), wherein the POC map (906) includes a POC difference between a reference image (410; 430; 1100) and a current image (400, 1400) including the current block (300; 1410).

[0376] In an embodiment of the present disclosure, the at least one processor of the image decoding device may be configured to reconstruct the current block (300; 1410) based on the final prediction block (955) and the residual block obtained from the bitstream.

[0377] In an embodiment of the present disclosure, sample values ​​of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) may be calculated based on a quantization parameter for a reference block (415; 435; 510; 510-1; 510-2; 1110).

[0378] According to an embodiment of the present disclosure, an image encoding apparatus may include at least one memory configured to store at least one instruction and at least one processor configured to execute the at least one instruction.

[0379] In an embodiment of the present disclosure, the at least one processor of the image encoding device may be configured to obtain a motion vector indicating a reference block (415; 435; 510; 510-1; 510-2; 1110) corresponding to the current block (300; 1410) in the reference image (410; 430; 1100).

[0380] In an embodiment of the present disclosure, the at least one processor of the image encoding device may be configured to obtain a final prediction block (955) of the current block (300; 1410) by applying to the neural network (2040) at least one of a POC map (906) including a picture order count (POC) difference between a reference image (410; 430; 1100) and a current image (400; 1400) including the current block (300; 1410), a preliminary prediction block (902) obtained based on the reference block (415; 435; 510; 510-1; 510-2; 1110), or a quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904).

[0381] In an embodiment of the present disclosure, the at least one processor of the image encoding device may be configured to obtain a residual block based on the current block (300, 1410) and the final prediction block (955).

[0382] In an embodiment of the present disclosure, the at least one processor of the image encoding device may be configured to generate a bitstream including information about the residual block.

[0383] In an embodiment of the present disclosure, sample values ​​of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) may be calculated based on a quantization parameter for a reference block (415; 435; 510; 510-1; 510-2; 1110).

[0384] In an embodiment of the present disclosure, the image decoding device 100 and the image encoding device 1900 using AI and the method thereof according to an embodiment of the present disclosure can obtain a final prediction block 955 that is more similar to the current block 300 or 1410 than the existing rule-based prediction mode.

[0385] In addition, the image decoding apparatus 100 and the image encoding apparatus 1900 using AI and methods thereof according to an embodiment of the present disclosure may reduce a bit rate of a bitstream including information about a residual block.

[0386] The embodiments of the present disclosure may be provided as a computer-executable program, and the program may be stored in a machine-readable storage medium.

[0387] The machine-readable storage medium may be provided as a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not include a signal (e.g., an electromagnetic wave) and is tangible, but does not distinguish whether data is stored in the storage medium semi-permanently or temporarily. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data.

[0388] According to an embodiment of the present disclosure, the method according to various embodiments of the present disclosure may be provided in a computer program product. A computer program product is a product that can be purchased between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a CD-ROM), or distributed (e.g., downloaded or uploaded) between two user devices (e.g., smart phones) directly or online through an application store. When distributed online, at least a portion of the computer program product (e.g., a downloadable application) may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or a relay server.

[0389] Although the scope of the present disclosure has been described with reference to the embodiments, the scope of the present disclosure is not limited thereto, and a person of ordinary skill in the art may make various modifications and changes without departing from the scope of the present disclosure.

Claims

1. A method for decoding an image, the method comprising: Obtaining a motion vector of the current block (300; 1410) (S1710); Obtaining a preliminary prediction block (902) based on a reference block (415; 435; 510; 510-1; 510-2; 1110) indicated by a motion vector in a reference image (410; 430; 1100) (S1720); By applying the picture order count POC map (906), the preliminary prediction block (902), or the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) to obtain a final prediction block (955) of the current block (300; 1410) (S1730), wherein the POC map (906) includes a POC difference between a reference image (410; 430; 1100) and a current image (400; 1400) including the current block (300; 1410); and reconstructing the current block (300; 1410) based on the final prediction block (955) and the residual block obtained from the bitstream (S1740), Therein, the sample values ​​of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) are calculated based on the quantization parameters used for the reference block (415; 435; 510; 510-1; 510-2; 1110).

2. The method according to claim 1, wherein: Quantization error map (530; 530-1； 530-2； 530-3； 530-4; 904) corresponds to a quantization step size or a quantization error value calculated based on a quantization parameter for the reference block.

3. The method according to any one of claims 1 and 2, wherein: Quantization error map (530; 530-1; 530-2； 530-3； 530-4; 904) is divided into sub-regions corresponding to the lower layer blocks of the reference block (415; 435; 510; 510-1; 510-2; 1110), and Wherein, the sample values ​​respectively included in the sub-areas of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) are calculated based on the quantization parameters of the samples at the positions in the lower layer block of the reference block (415; 435; 510; 510-1; 510-2; 1110).

4. The method according to any one of claims 1 to 3, wherein: Obtaining a final prediction block (955) of the current block (300; 1410) comprises: applying at least one of the enlarged POC map, the enlarged preliminary prediction block (1150), or the enlarged quantization error map (1300) to the neural network (240); Wherein, at least one of the enlarged POC map, the enlarged preliminary prediction block (1150) or the enlarged quantization error map (1300) is obtained by filling at least one of the POC map (906), the preliminary prediction block (902), or the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) according to the enlarged distance.

5. The method for decoding an image according to any one of claims 1 to 4, wherein: The neural network (240) includes one or more convolutional layers, and Wherein, the method further comprises: determining the zoom-in distance based on the number of the one or more convolutional layers included in the neural network (240), the size of the filter kernel used in the one or more convolutional layers, and the stride; and An enlarged preliminary prediction block (1150) is obtained, the enlarged preliminary prediction block (1150) including neighboring samples corresponding to the enlarged distance among neighboring samples outside the boundary of the reference block (415; 435; 510; 510-1; 510-2; 1110) in the reference image (410; 430; 1100) and samples of the preliminary prediction block (920).

6. The method according to any one of claims 1 to 5, further comprising: An enlarged quantization error map (1300) is obtained, the enlarged quantization error map (1300) comprising sample values ​​calculated based on the quantization parameters for neighboring samples corresponding to the enlarged distance in the reference image (410; 430; 1100) and the quantization parameters for the reference block (415; 435; 510; 510-1; 510-2; 1110).

7. The method according to any one of claims 1 to 6, wherein: Based on a boundary of the reference block (415; 435; 510; 510-1; 510-2; 1110) corresponding to a boundary of a reference image (410; 430; 1100), a neighboring sample point corresponding to the zoom-in distance is determined from the closest sample points available in the reference image (410; 430; 1100).

8. The method according to any one of claims 1 to 7, wherein: Obtaining a final prediction block (955) of the current block (300; 1410) comprises: applying the enlarged current reconstructed block (1500) to the neural network (240) together with at least one of the enlarged POC map, the enlarged preliminary prediction block (1150), or the enlarged quantization error map (1300); The enlarged current reconstructed block (1500) includes neighboring samples (1420) reconstructed before the current block (300; 1410) in the current image (400; 1400) and samples (1125) of the enlarged preliminary prediction block (1150) except for samples corresponding to the neighboring samples (1420).

9. The method according to any one of claims 1 to 8, further comprising: A weight set for obtaining a final prediction block (955) is selected from a plurality of weight sets based on at least one of a size of the current block (300; 1410), a prediction direction of the current block (300; 1410), a quantization parameter for the reference block (415; 435; 510; 510-1; 510-2; 1110), a layer to which the current image (400; 1400) belongs in an image hierarchy, or information obtained from a bitstream, and Therein, a final prediction block (955) is obtained based on the neural network (240) operating according to the selected set of weights. 10 . A computer-readable recording medium having recorded thereon a program for executing the method according to claim 1 on a computer.

11. A method for encoding an image, the method comprising: obtaining a motion vector indicating a reference block (415; 435; 510; 510-1; 510-2; 1110) corresponding to the current block (300; 1410) in the reference image (410; 430; 1100) (S2210); By using the picture order count POC map (906), the preliminary prediction block (902) obtained based on the reference block (415; 435; 510; 510-1; 510-2; 1110), or the quantization error map (530; 530-1; 530-2; 530-3； 530-4; 904) is applied to a neural network (2040) to obtain a final prediction block (955) of the current block (300; 1410), wherein the POC map (906) includes a POC difference (S2230) between a reference image (410; 430; 1100) and a current image (400; 1400) including the current block (300; 1410); Obtaining a residual block (S2240) based on the current block (300, 1410) and the final prediction block (955); and generating a bitstream including information about the residual block (S2250), Therein, the sample values ​​of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) are calculated based on the quantization parameters used for the reference block (415; 435; 510; 510-1; 510-2; 1110).

12. The method according to claim 11, further comprising: Change the precision of the obtained motion vector from fractional precision to integer precision, Therein, the reference block (415; 435; 510; 510-1; 510-2; 1110) indicated by the motion vector with integer precision is determined as a preliminary prediction block (902).

13. The method according to any one of claims 11 and 12, wherein: Obtaining a final prediction block (955) of the current block (300; 1410) comprises: applying at least one of the enlarged POC map, the enlarged preliminary prediction block (1150) or the enlarged quantization error map (1300) to the neural network (2040), Wherein, at least one of the enlarged POC map, the enlarged preliminary prediction block (1150), or the enlarged quantization error map (1300) is obtained by filling at least one of the POC map (906), the preliminary prediction block (902), or the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) according to the enlarged distance.

14. An image decoding device, comprising: at least one memory configured to store at least one instruction; as well as At least one processor is configured to execute the at least one instruction to perform the following operations: Get the motion vector of the current block (300; 1410); Obtaining a preliminary prediction block (902) based on a reference block (415; 435; 510; 510-1; 510-2; 1110) indicated by a motion vector in a reference image (410; 430; 1100); By applying the picture order count POC map (906), the preliminary prediction block (902), or the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) to obtain a final prediction block (955) of the current block (300; 1410), wherein the POC map (906) includes a POC difference between a reference image (410; 430; 1100) and a current image (400; 1400) including the current block (300; 1410); and reconstructing the current block (300; 1410) based on the final prediction block (955) and the residual block obtained from the bitstream, Therein, the sample values ​​of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) are calculated based on the quantization parameters used for the reference block (415; 435; 510; 510-1; 510-2; 1110).

15. An image encoding device, comprising: at least one memory configured to store at least one instruction; as well as At least one processor is configured to execute the at least one instruction to perform the following operations: obtaining a motion vector indicating a reference block (415; 435; 510; 510-1; 510-2; 1110) in a reference image (410; 430; 1100) corresponding to the current block (300; 1410); The picture order count POC map (906), the preliminary prediction block (902) obtained based on the reference block (415; 435; 510; 510-1; 510-2; 1110), or the quantization error map (530; 530-1; 530-2；530-3； 530-4; 904) is applied to a neural network (2040) to obtain a final prediction block (955) for the current block (300; 1410), wherein the POC map (906) includes a POC difference between a reference image (410; 430; 1100) and a current image (400; 1400) including the current block (300; 1410), Obtaining a residual block based on the current block (300; 1410) and the final prediction block (955); and generating a bitstream including information about the residual block, Therein, the sample values ​​of the quantization error map (530; 530-1; 530-2; 530-3; 530-4; 904) are calculated based on the quantization parameters used for the reference block (415; 435; 510; 510-1; 510-2; 1110).

Images