Image decoding apparatus and method, and image encoding apparatus and method

Abstract

Provided are an image decoding method and apparatus that obtain intra prediction feature data of a current block from a bitstream, determine an intra stream of reference pixels indicating a current pixel in the current block by applying the intra prediction feature data, neighboring pixels of the current block, and encoding context information of the current block to a neural network, a prediction pixel of the current pixel is obtained based on the intra-frame stream of the current block, and the current block is reconstructed based on the prediction pixel.

Description

Technical Field

[0001] The present disclosure relates to an image decoding method, an image decoding device, an image encoding method, and an image encoding device, and more particularly, to a method and a device for performing intra prediction. Background Art

[0002] In image processing technology, codecs such as H.264 Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC) can be used to process images. In such codecs, an image can be divided into blocks, and each block can be predictively encoded and predictively decoded by inter-frame prediction or intra-frame prediction.

[0003] Intra-frame prediction is a method of compressing an image by removing spatial redundancy in the image, and inter-frame prediction is a method of compressing an image by removing temporal redundancy between images.

[0004] Intra prediction is performed by predicting all pixels in a specific block in one intra prediction direction based on one intra prediction mode. In addition, in the intra prediction operation, interpolation of reference pixels may be performed by using fixed filter coefficients of a predetermined filter.

[0005] Recently, a technology for encoding / decoding an image by using artificial intelligence (AI) has been proposed, and a method for efficiently encoding / decoding an image by performing intra prediction by using AI (eg, a neural network) has been utilized. Summary of the invention

[0006] Solution to the problem

[0007] Embodiments of the present disclosure provide a method and apparatus for performing intra prediction more efficiently.

[0008] According to an embodiment of the present disclosure, an image decoding method may include: obtaining intra-frame prediction feature data of a current block from a bitstream, obtaining intra-frame flow information indicating a reference pixel of a current pixel in the current block based on the intra-frame prediction feature data, neighboring pixels of the current block and encoding context information of the current block, obtaining predicted pixels of the current pixel based on the intra-frame flow information of the current block, and reconstructing the current block based on the predicted pixels.

[0009] According to an embodiment of the present disclosure, an image decoding device may include a memory storing one or more instructions, and at least one processor configured to operate according to the one or more instructions. The at least one processor may be configured to obtain intra-frame prediction feature data of a current block from a bitstream. The at least one processor may be configured to obtain intra-frame stream information indicating a reference pixel of a current pixel in the current block based on the intra-frame prediction feature data, adjacent pixels of the current block, and encoding context information of the current block. The at least one processor may be configured to obtain a predicted pixel of the current pixel based on the intra-frame stream information of the current block. The at least one processor may be configured to reconstruct the current block based on the predicted pixel.

[0010] According to an embodiment of the present disclosure, an image encoding method may include: obtaining intra-frame prediction feature data of the current block and intra-frame flow information indicating reference pixels corresponding to the original pixels of the current block based on original pixels of the current block, first neighboring pixels of the current block and encoding context information of the current block, obtaining predicted pixels of the original pixels based on the intra-frame flow information of the current block, and encoding the intra-frame prediction feature data by reconstructing the current block based on the predicted pixels.

[0011] According to an embodiment of the present disclosure, an image encoding device may include a memory storing one or more instructions, and at least one processor configured to operate according to the one or more instructions. The at least one processor may be configured to obtain intra-frame prediction feature data of the current block and intra-frame flow information indicating reference pixels corresponding to the original pixels of the current block based on the original pixels of the current block, the first adjacent pixels of the current block, and the encoding context information of the current block. The at least one processor may be configured to obtain predicted pixels of the original pixels based on the intra-frame flow information of the current block. The at least one processor may be configured to encode the intra-frame prediction feature data by reconstructing the current block based on the predicted pixels.

[0012] According to an embodiment of the present disclosure, an image decoding method may include: obtaining, based on a first neighboring pixel of a current block and encoding context information, intra-frame flow information indicating a reference pixel corresponding to a current pixel in the current block, and weight information of the reference pixel and a plurality of second neighboring pixels adjacent to the reference pixel, obtaining a predicted pixel of the current pixel based on the intra-frame flow information and the weight information, and reconstructing the current block based on the predicted pixel.

[0013] According to an embodiment of the present disclosure, an image decoding device may include a memory storing one or more instructions, and at least one processor configured to operate according to the one or more instructions. The at least one processor may be configured to obtain an intra-frame stream indicating a reference pixel corresponding to a current pixel in the current block, and weight information of the reference pixel and a plurality of second adjacent pixels adjacent to the reference pixel based on a first adjacent pixel of the current block and coding context information. The at least one processor may be configured to obtain a predicted pixel of the current pixel based on the intra-frame stream information and the weight information. The at least one processor may be configured to reconstruct the current block based on the predicted pixel.

[0014] According to an embodiment of the present disclosure, an image encoding method may include: obtaining, based on a first neighboring pixel of a current block and encoding context information, intra-frame flow information indicating a reference pixel corresponding to a current pixel in the current block, and weight information of the reference pixel and a plurality of second neighboring pixels adjacent to the reference pixel, obtaining a predicted pixel of the current pixel based on the intra-frame flow information and the weight information, and encoding the current block based on the predicted pixel.

[0015] According to an embodiment of the present disclosure, an image encoding device may include a memory storing one or more instructions, and at least one processor configured to operate according to the one or more instructions. The at least one processor may be configured to obtain, based on adjacent pixels of the current block and coding context information, intra-frame flow information indicating a reference pixel corresponding to a current pixel in the current block, and weight information of the reference pixel and a plurality of second adjacent pixels adjacent to the reference pixel. The at least one processor may be configured to obtain a predicted pixel of the current pixel based on the intra-frame flow information and the weight information. The at least one processor may be configured to encode the current block based on the predicted pixel. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The embodiments will be more clearly understood through the following detailed description in conjunction with the accompanying drawings, in which:

[0017] Figure 1 is a diagram showing image encoding and decoding processing.

[0018] Figure 2 is a diagram showing blocks split from an image according to a tree structure.

[0019] Figure 3 is a diagram illustrating an intra-frame flow according to an embodiment of the present disclosure.

[0020] Figure 4 is a diagram for describing an intra-frame flow according to an embodiment of the present disclosure.

[0021] Figure 5is a diagram illustrating weights of reference pixels indicated by an intra stream and neighboring pixels of the reference pixels according to an embodiment of the present disclosure.

[0022] Figure 6 is a diagram for describing weights of reference pixels and neighboring pixels of the reference pixels indicated by an intra stream according to an embodiment of the present disclosure.

[0023] Figure 7 is a schematic diagram illustrating an image encoding method according to an embodiment of the present disclosure.

[0024] Figure 8 is a diagram for describing a neural network used in an image encoding method according to an embodiment of the present disclosure.

[0025] Fig. 9 Show Figure 8 Examples of the structure of the encoding neural network and the structure of the decoding neural network.

[0026] Fig.10 is a schematic diagram illustrating an image decoding method according to an embodiment of the present disclosure.

[0027] Fig.11 is a diagram for describing a neural network used in an image decoding method according to an embodiment of the present disclosure.

[0028] Fig.12 Show Fig.11 An example of the structure of a decoding neural network.

[0029] Fig.13 is a diagram for describing a method of training a neural network used in an image encoding method and an image decoding method according to an embodiment of the present disclosure.

[0030] Fig.14 Schematic diagrams illustrating an image encoding method and an image decoding method according to an embodiment of the present disclosure.

[0031] Fig.15 is a diagram for describing a method of training a neural network used in an image encoding method and an image decoding method according to an embodiment of the present disclosure.

[0032] Fig.16 is a flowchart illustrating an image decoding method according to an embodiment of the present disclosure.

[0033] Fig.17 is a diagram showing a configuration of an image decoding device according to an embodiment of the present disclosure.

[0034] Fig.18 is a flowchart illustrating an image encoding method according to an embodiment of the present disclosure.

[0035] Fig.19 is a diagram showing the configuration of an image encoding device according to an embodiment of the present disclosure.

[0036] Fig. 20 is a flowchart illustrating an image decoding method according to an embodiment of the present disclosure.

[0037] Fig.21 is a diagram showing a configuration of an image decoding device according to an embodiment of the present disclosure.

[0038] Fig. 22 is a flowchart illustrating an image encoding method according to an embodiment of the present disclosure.

[0039] Fig.23 is a diagram showing the configuration of an image encoding device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] All terms used herein are briefly described, and embodiments of the present disclosure are fully described below. These embodiments are example embodiments, and therefore, the present disclosure is not limited thereto, and may be implemented in various other forms. As is conventional in the art, embodiments may be described and illustrated according to blocks as shown in the accompanying drawings, which perform one or more functions described. These blocks (which may be referred to herein as units or modules, etc., or by names such as devices, logic, circuits, counters, comparators, generators, converters, etc.) may be physically implemented by analog and / or digital circuits including one or more of logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, etc., and may also be implemented or driven by software and / or firmware (configured to perform the functions or operations described herein).

[0041] Throughout this disclosure, the expression “at least one of a, b, or c” refers to only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0042] The present disclosure may include various embodiments and modifications, and specific embodiments thereof are shown in the accompanying drawings and will be described in detail herein. However, it should be understood that the present disclosure is not limited to specific embodiments, and includes all modifications, equivalents and substitutes included in the spirit and scope of the present disclosure.

[0043] In the following description of the embodiments, when it is considered that the detailed description of the prior art may unnecessarily obscure the subject matter of the present disclosure, the detailed description of the prior art will be omitted. In addition, the numbers (e.g., first and second) used in the process of describing the embodiments of the present disclosure are merely identification symbols for distinguishing an element from another element.

[0044] Furthermore, herein, when an element is referred to as being “connected” or “coupled” to another element, the element may be directly connected or coupled to the other element or may be indirectly connected or coupled to the other element via one or more other intermediate elements, unless otherwise specified.

[0045] In addition, in this document, for elements represented as "...part (unit)", "...module", etc., two or more elements may be combined into one element, or one element may be divided into two or more elements for each subdivided function. Also, each element described below may additionally perform some or all functions of other elements in addition to its main function, and some main functions of each element may be exclusively performed by other elements.

[0046] Furthermore, in this document, an “image or picture” may refer to a still image (or frame), a moving image including a plurality of consecutive still images, or a video.

[0047] Here, the "neural network" may be a representative example of an artificial neural network model that simulates cranial nerves, and is not limited to an artificial neural network model that uses a specific algorithm. The neural network may also be referred to as a deep neural network.

[0048] Here, "parameter" may be a value used in the operation process of each layer constituting the neural network, and may be used, for example, when an input value is applied to a specific operation expression. Parameters may be values ​​set as a result of training, and may be updated by separate training data as needed.

[0049] Herein, "feature data" may refer to data obtained by processing input data by a neural network or an encoder based on a neural network. Feature data may be one-dimensional or two-dimensional data. However, the present disclosure is not limited thereto. Feature data may include one-dimensional or two-dimensional data containing various samples. Feature data may also be referred to as a potential tensor or potential representation. Feature data may represent potential features in data output by a decoder based on a neural network.

[0050] In this document, "sample" may refer to data to be processed as data assigned to sampling locations in one-dimensional or two-dimensional data such as images, blocks, or feature data. For example, a sample may include pixels in a two-dimensional image. Two-dimensional data may also be referred to as a "map."

[0051] In addition, in this document, "current block" may refer to a block currently to be processed. The current block may be a slice, a tile, a maximum coding unit, a coding unit, a prediction unit, or a transform unit divided from a current image.

[0052] Furthermore, herein, 'intra stream' may refer to a reference pixel indicated by each pixel in a block.

[0053] Before describing an image decoding method, an image decoding device, an image encoding method, and an image encoding device according to an embodiment of the present disclosure, reference will be made to Figure 1 and Figure 2 Describes the image encoding and decoding process.

[0054] Figure 1 is a diagram showing image encoding and decoding processing.

[0055] In the image encoding and decoding process, the encoding device 110 may transmit a bit stream generated by encoding the image to the decoding device 150 , and the decoding device 150 may reconstruct the image by receiving and decoding the bit stream.

[0056] For example, the encoding device 110 may include a prediction encoder 115, a transformer and quantizer 120, an entropy encoder 125, an inverse quantizer and inverse transformer 130, a deblocking filter 135, and a loop filter 140, and the decoding device 150 may include an entropy decoder 155, an inverse quantizer and inverse transformer 160, a deblocking filter 165, a loop filter 170, and a prediction decoder 175. Specifically, in the encoding device 110, the prediction encoder 115 may output a prediction block through inter-frame prediction and intra-frame prediction, and the transformer and quantizer 120 may output a quantized transformation coefficient by transforming and quantizing a residual sample of a residual block between the prediction block and the current block. The entropy encoder 125 may output a bitstream by encoding the quantized transformation coefficient.

[0057] The quantized transform coefficients may be reconstructed from a residual block including residual samples in the spatial domain by the inverse quantizer and inverse transformer 130. A reconstructed block obtained by adding the prediction block and the residual block may be output as a filtered block by the deblocking filter 135 and the loop filter 140. The reconstructed image including the filtered block may be used as a reference image for the next input image in the prediction encoder 115.

[0058] The bitstream received by the decoding device 150 may be reconstructed from a residual block including residual samples in the spatial domain through the entropy decoder 155 and the inverse quantizer and inverse transformer 160. A reconstructed block may be generated by combining the residual block and the prediction block output from the prediction decoder 175, and the reconstructed block may be output as a filtered block through the deblocking filter 165 and the loop filter 170. The reconstructed image including the filtered block may be used as a reference image of the next image in the prediction decoder 175.

[0059] The loop filter 140 of the encoding device 110 may perform loop filtering by using filter information input according to user input or system setting. The filter information used by the loop filter 140 may be transmitted to the decoding device 150 through the entropy encoder 125. The loop filter 170 of the decoding device 150 may perform loop filtering based on the filter information input from the entropy decoder 155.

[0060] In the image encoding and decoding process, the image can be divided and encoding and decoding can be performed on the blocks divided from the image. For example, the image can be divided hierarchically. Figure 2 Describes the blocks partitioned from the image.

[0061] Figure 2 is a diagram showing blocks segmented (or divided) from the image 200 according to the tree structure.

[0062] According to an embodiment of the present disclosure, the image 200 may be segmented into one or more slices or one or more tiles. A slice may include multiple tiles. For example, the image 200 may be segmented into one or more slices or one or more tiles.

[0063] A slice or tile can be a sequence of one or more largest decoding units (CUs).

[0064] The maximum coding unit may be split into one or more coding units. A coding unit may be a reference block for determining a prediction mode. In other words, it may be determined whether an intra prediction mode or an inter prediction mode is applied to each coding unit. Here, the maximum coding unit may be referred to as a maximum coding block, and a coding unit may be referred to as a coding block.

[0065] The size of a coding unit may be equal to or smaller than the size of a maximum coding unit. Since a maximum coding unit is a coding unit having a maximum size, it may also be referred to as a coding unit. Intra prediction may be referred to as "intra-prediction", and inter prediction may be referred to as "inter-prediction".

[0066] One or more prediction units for intra prediction or inter prediction may be determined from the coding unit. The size of the prediction unit may be equal to or smaller than the size of the coding unit.

[0067] In addition, one or more transform units for transform and quantization may be determined from the coding unit. The size of the transform unit may be equal to or smaller than the size of the coding unit. The transform unit may be a reference block for transform and quantization, and the residual samples of the coding unit may be transformed and quantized for each transform unit in the coding unit.

[0068] Herein, the current block may be a slice, a tile, a maximum coding unit, a coding unit, a prediction unit, or a transform unit partitioned (or divided) from the image 200. In addition, the lower block of the current block may be a block partitioned from the current block, and for example, when the current block is the maximum coding unit, the lower block may be a coding unit, a prediction unit, or a transform unit. In addition, the upper block of the current block may be a block including the current block as a part thereof, and for example, when the current block is the maximum coding unit, the upper block may be a picture sequence, a picture, a slice, or a tile. The lower block of the current block and the upper block of the current block may refer to blocks according to a hierarchy in which the image 200 is divided. For example, the lower layer block of the current block may refer to a lower layer unit, which is a further subdivided unit of the current block. On the other hand, the upper block of the current block may refer to a higher unit of which the current block is a part.

[0069] In the following, reference will be made to Figures 3 to 23 An image decoding method, an image decoding device, an image encoding method, and an image encoding device according to an embodiment of the present disclosure are described.

[0070] Figure 3 is a diagram illustrating an intra-frame flow according to an embodiment of the present disclosure.

[0071] Performing prediction on all pixels in a block in one intra prediction direction according to one intra prediction mode may not cover various types of video content.All pixels in a block may have different prediction directions indicating the best reference pixels for the corresponding pixels.

[0072] refer to Figure 3 , pixels in a current block 320 of a current image 300 may have an intra flow 330 indicating different prediction directions for reference pixels in a neighboring reference region 310. Intra flow may be referred to as "intra-flow".

[0073] The intra-frame stream 330 may be represented by a first vector representing the difference between the x component and the y component between the current pixel and the reference pixel, a second vector representing the distance r component between the current pixel and the reference pixel and the angle θ component between the current pixel and the reference pixel relative to the x-axis, or an angle between the current pixel and the reference pixel. According to an embodiment of the present disclosure, when the intra-frame stream 330 is represented only by the angle between the current pixel and the reference pixel, a reference pixel that is first located on the boundary of the current block according to the angle relative to the current pixel may be determined as the reference pixel. However, the present disclosure is not limited thereto, and therefore, according to an embodiment of the present disclosure, the intra-frame stream 330 may include one or more of the first vector, the second vector, and the angle. According to an embodiment of the present disclosure, the intra-frame stream 330 may include one or more of the first vector, the second vector, and the angle as well as other information.

[0074] Figure 4 is a diagram for describing an intra-frame flow according to an embodiment of the present disclosure.

[0075] refer to Figure 4 , an intra stream may be a stream of reference pixels representing each pixel to be predicted.

[0076] According to an embodiment of the present disclosure, a current pixel 420 of a current block 400 may have an intra-frame stream 430 indicating a reference pixel of an adjacent reference area 410 corresponding to the current pixel 420 of the current block 400. That is, each pixel in the current block 400 may have a reference pixel indicated by the intra-frame stream. According to an embodiment of the present disclosure, each pixel in the current block 400 may have an independent or separate intra-frame stream 430 indicating a reference pixel corresponding to a corresponding pixel in the current block 400. For example, a first pixel in the current block 400 may have a first intra-frame stream 430 indicating a first reference pixel corresponding to the first pixel, and a second pixel in the current block 400 may have a second intra-frame stream 430 indicating a second reference pixel corresponding to the second pixel. Although the first intra-frame stream and the second intra-frame stream are used for explanation, the present disclosure is not limited thereto, and therefore, the first intra-frame stream and the second intra-frame stream may be referred to as the first stream and the second stream contained in the intra-frame stream. That is, the intra-frame stream may contain a pointer to each pixel (pixel) to be predicted in the current block. i ) of the reference pixel flow (f i ), where i is an integer.

[0077] According to an embodiment of the present disclosure, pixels on the block boundary may not require intra-frame streaming, but intra-frame streaming can be inside the reference block when necessary. As an example, in the case where the block is replicated in the screen content, intra-frame streaming may not be necessary for pixels on the boundary of the reference block.

[0078] According to an embodiment of the present disclosure, in an image encoding process, the image encoding device may determine an intra-frame stream representing a reference pixel corresponding to each pixel in the current block, obtain a predicted pixel of the original pixel by using the intra-frame stream, reconstruct the current block based on the predicted pixel, entropy encode the information about the intra-frame stream by cost calculation, and send the obtained bit stream to the image decoding device. The cost calculation operation may be a sum of transform differences (SATD) or a rate-distortion optimization (RDO) calculation. However, the present disclosure is not limited thereto, and therefore, according to an embodiment of the present disclosure, other calculations may be performed. In addition, in an image decoding process, the image decoding device may obtain information about the intra-frame stream from the bit stream, determine an intra-frame stream representing a reference pixel corresponding to each pixel in the current block based on the information about the intra-frame stream, obtain a predicted pixel of the current pixel in the current block by using the intra-frame stream, and reconstruct the current block based on the predicted pixel.

[0079] Figure 5 is a diagram illustrating weights of reference pixels indicated by an intra stream and neighboring pixels of the reference pixels according to an embodiment of the present disclosure.

[0080] refer to Figure 5 , the reference pixel of the current pixel can be determined according to the intra-frame stream 330 indicating the reference pixels in the neighboring reference area 310 of the current pixel in the current block 320 of the current image 300. In this case, the neighboring pixels of the reference pixel represented by the intra-frame stream 330 may include a plurality of references similar to the pixel to be predicted. The final predicted pixel can be obtained by mixing and using the reference pixel and the neighboring pixels 510 of the reference pixel, which will be referred to below. Figure 6 Describe in more detail.

[0081] Figure 6 is a diagram for describing weights of reference pixels and neighboring pixels of the reference pixels indicated by an intra stream according to an embodiment of the present disclosure.

[0082] refer to Figure 6 , the reference pixel in the adjacent reference area 410 of the current block 400 can be determined according to the intra-frame stream 430 indicating the reference pixel of the current pixel 420 of the current block 400. In this case, the reference pixel and the adjacent pixel 610 of the reference pixel can be determined. The final predicted pixel can be obtained by calculating the weighted average using the determined reference pixel and the adjacent pixel 610. For example, in Figure 6 In the example, the reference pixel may be pixel W. 11 , and the neighboring pixels can be W 00 , W 01 , W 02 , W 10 , W 12 , W 20 , W 21 and W22 The weight of the reference pixel is W 11 , the weight of the upper left neighboring pixel of the reference pixel is W 00 , the weight of the upper neighboring pixel of the reference pixel is W 01 , the weight of the upper right neighboring pixel of the reference pixel is W 02 , the weight of the left neighboring pixel of the reference pixel is W 10 , the weight of the right neighboring pixel of the reference pixel is W 12 , the weight of the lower left neighboring pixel of the reference pixel is W 20 , the weight of the next neighboring pixel of the reference pixel is W 21 , the weight of the lower right neighboring pixel of the reference pixel is W 22 .

[0083] Therefore, the final predicted pixel (or pixel value) It can be calculated as a weighted average according to Equation 1 below.

[0084] [Equation 1]

[0085]

[0086] Here, W is a reference window representing the pixels around the reference pixel, and Figure 6 has a size of 3×3, W ij is the weight of row i and column j, and p̂ ij is the sample value of the pixel in row i and column j.

[0087] The weight may represent the contribution of each reference pixel to the final predicted pixel, where the weight may be large when the contribution is large, and the weight may be small when the contribution is small.

[0088] The reference window may have a size of 5×5 or 7×7 instead of 3×3 and may have a rectangular shape such as 3×5 or 5×3 instead of a square shape; however, the present disclosure is not limited thereto. Therefore, the size and shape of the reference window may vary.

[0089] The reference window may have a predetermined size, and the weight information may be used for weighted averaging based on the reference window of the predetermined size.

[0090] According to an embodiment of the present disclosure, in an image encoding process, an image encoding device may determine an intra-frame stream representing a reference pixel corresponding to each pixel in a current block and weight information about the reference pixel represented by the intra-frame stream and the neighboring pixels of the reference pixel, obtain a predicted pixel of the original pixel by using the intra-frame stream and the weight information, reconstruct the current block based on the predicted pixel, entropy encode the information about the intra-frame stream and the weight information by cost calculation, and send the obtained bit stream to an image decoding device. The cost calculation may be a sum of transform differences (SATD) or a rate-distortion optimization (RDO) calculation. In addition, in an image decoding process, the image decoding device may obtain information and weight information about the intra-frame stream from a bit stream, determine an intra-frame stream representing a reference pixel corresponding to each pixel in the current block based on the information and weight information about the intra-frame stream, obtain a predicted pixel of a current pixel in the current block by using the intra-frame stream and the weight, and reconstruct the current block based on the predicted pixel.

[0091] In the following, reference will be made to Figures 7 to 23 An image encoding method and apparatus and an image decoding method and apparatus are described for determining an intra-frame stream or an intra-frame stream and weight information by using a neural network and obtaining predicted pixels based on the intra-frame stream or the intra-frame stream and weight information.

[0092] Figure 7 is a schematic diagram illustrating an image encoding method according to an embodiment of the present disclosure.

[0093] like Figure 7 As shown, according to an embodiment of the present disclosure, the original pixels 720 of the current block, the reference pixels 710 located around the current block, and the encoding context information 730 may be input to the neural network 740, and the neural network 740 may output the intra-frame stream 760. Figure 7A plurality of original pixels and a plurality of reference pixels 710 are shown, but the present disclosure is not limited thereto, and therefore, according to an embodiment of the present disclosure, the original pixels of the current block, the reference pixels located near the original pixels, and the encoding context information 730 may be input to the neural network 740. According to an embodiment of the present disclosure, only the intra-frame stream 760 may be obtained from the neural network 740. However, the present disclosure is not limited thereto, and therefore, the neural network 760 may output other information together with the intra-frame stream 760. For example, the intra-frame stream 760 and the weight information 770 may be obtained from the neural network 740. According to an embodiment of the present disclosure, the predicted pixel 790 may be obtained by performing the prediction 780 using the intra-frame stream 760. Alternatively, the intra-frame stream 760 and the weight information 770 may be obtained from the neural network 740, and the predicted pixel 790 may be obtained by performing the prediction 780 using the intra-frame stream 760 and the weight information 770. In this case, the intra-frame prediction feature data 750 may be entropy encoded from the neural network 740 through cost calculation, and the resulting bitstream may be sent to a decoding device. The cost calculation can be a sum of transformed differences (SATD) or a rate-distortion optimized (RDO) calculation.

[0094] According to an embodiment of the present disclosure, when weight information is not used, the accuracy of the intra stream 760 may be a fractional pixel, such as ½ pixel (half pixel) or ¼ pixel (quarter pixel).

[0095] The precision of the intra stream 760 may be a pixel precision directly indicating a reference pixel when using weight information.

[0096] Figure 8 is a diagram for describing a neural network used in an image encoding method according to an embodiment of the present disclosure.

[0097] refer to Figure 8 , the quantization parameter 805, the temporal layer 810 and the partition tree structure 815 can be used as encoding context information input to the context encoder 801, and the original pixels 830 of the current block, the reference pixels 820 located around (or surrounding) the current block, and the zero pixels 825 with a sampling value of 0 in the unencoded area of ​​the adjacent block can be used as pixel information input to the context encoder 801.

[0098] According to an embodiment of the present disclosure, since the encoding context information 805, 810, and 815 is in a domain different from the pixel value of the pixel information, it may need to be processed by the neural network 835 to the same size as the pixel information 820, 830, and 825 before being input to the encoding neural network 850. Here, the neural network 835 may be a general convolutional neural network (CNN). In addition, the neural network 835 may be a recurrent neural network (RNN). However, this is merely an example, and the present disclosure is not limited thereto.

[0099] The pixel information 820 , 830 , and 825 and the encoding context information 805 , 810 , and 815 output from the neural network 835 may be concatenated ( 840 ) and input to the encoding neural network 850 .

[0100] Intra-frame prediction feature data 855 in the form of feature maps can be output from the encoding neural network 850.

[0101] When the size of pixel information including reference pixels 820, zero pixels 825, and original pixels 830 is H×W×C (where H is the height of the pixel information, W is the width of the pixel information, and C is the channel of the pixel information; for example, the value of C is 1 in the case of a luminance component), the size of the intra-frame prediction feature data 855 as feature map data may be limited to h×w×c (where "h" is less than H, "w" is less than W, and "c" is a predetermined number). Typically, "h" may be H / 16, and "w" may be W / 16. Here, when the bit rate is high, "c" may be determined to be a larger "c". However, the present disclosure is not limited thereto.

[0102] According to an embodiment of the present disclosure, since the intra prediction feature data 855 is transmitted as a bitstream and since it is inefficient to transmit the intra prediction feature data 855 when the size of the intra prediction feature data 855 is large, the intra prediction feature data 855 may be limited to an appropriate size.

[0103] According to an embodiment of the present disclosure, since the intra-frame prediction feature data 855 is entropy encoded and transmitted, the intra-frame prediction feature data 855 may include feature map data output as a result of the encoding neural network 850 and additional information representing the probability distribution of the feature map data required for entropy decoding. The intra-frame prediction feature data 855 may be input to the context decoder 802 together with the encoding context information 805, 810, and 815 and the pixel information including the reference pixel 820 and the zero pixel 825. Here, the original pixel 830 is not input to the context decoder 802, but the additional zero pixel 825 may be input to the context decoder 802. According to an embodiment of the present disclosure, since the pixel information 820 and 825 and the encoding context information 805, 810, and 815 are in a different domain from the intra-frame prediction feature data 855, they may be processed by the neural network 860 to be the same size as the intra-frame prediction feature data 855, and then concatenated (865) with the intra-frame prediction feature data 855 and input to the decoding neural network 870. Here, the neural network 860 may be a general convolutional neural network (CNN). In addition, the neural network 860 may be a recurrent neural network (RNN). However, this is merely an example, and the present disclosure is not limited thereto.

[0104] The x component 875 of the intra-frame stream, the y component 880 of the intra-frame stream, and the weight information 885 may be obtained from the decoding neural network 870 .

[0105] According to an embodiment of the present disclosure, instead of the x component 875 of the intra stream and the y component 880 of the intra stream, the angle component and the size component of the intra stream may be obtained.

[0106] According to an embodiment of the present disclosure, the angle component of the intra-frame stream may be obtained only from the decoding neural network 870 .

[0107] According to an embodiment of the present disclosure, the x component 875 of the intra-frame stream and the y component 880 of the intra-frame stream may be obtained only from the decoding neural network 870 .

[0108] According to an embodiment of the present disclosure, the intra-frame prediction feature data 855 output from the encoding neural network 850 may be transmitted to a decoding device in the form of a bit stream.

[0109] Fig. 9 Show Figure 8 Examples of the structure of the encoding neural network and the structure of the decoding neural network.

[0110] refer to Fig. 9 ,exist Figure 8In the encoding neural network 850, the input data as the encoding context information 805, 810 and 815 and the pixel information 820, 825 and 830 may be input to the first convolution layer 905. In the first convolution layer 905, "N×5×5 / 2↓" indicates that the convolution process is a convolution process of downsampling by using N filter kernels of size 5×5 at a stride interval of 2. Here, N is an integer.

[0111] The feature data generated by the first convolutional layer 905 may represent unique characteristics of the input data. For example, each feature data may represent a vertical direction characteristic, a horizontal direction characteristic, or an edge characteristic of the input data.

[0112] The feature data of the first convolution layer 905 may be input to a first generalized divisor normalization (GDN) layer 910. The first GDN layer 910 may be replaced with a sigmoid function, a tanh function, a rectified linear unit (ReLU) function, etc.; however, the present disclosure is not limited thereto.

[0113] The first GDN layer 910 may assign nonlinear characteristics to each feature data.

[0114] Assigning nonlinear characteristics by the first GDN layer 910 may mean changing and outputting some sample values ​​of feature data. In this case, the change may be performed by applying the nonlinear characteristics.

[0115] The first GDN layer 910 may determine whether to send the sample values ​​of the feature data to the second convolution layer 915. For example, some of the sample values ​​of the feature data may be activated by the first GDN layer 910 and sent to the second convolution layer 915, and some other sample values ​​of the feature data may be deactivated by the first GDN layer 910 and not sent to the second convolution layer 915. The unique characteristics of the input data represented by the feature data may be emphasized by the first GDN layer 910.

[0116] The feature data output from the first GDN layer 910 may be input to the second convolution layer 915. In the second convolution layer 915, "N×5×5 / 2↓" indicates that the convolution process is a convolution process of downsampling by using N filter kernels of a size of 5×5 at a stride interval of 2.

[0117] The output of the second convolution layer 915 may be input to the second GDN layer 920. The second GDN layer 920 may assign nonlinear characteristics to the input feature data.

[0118] The feature data output from the second GDN layer 920 may be input to the third convolution layer 925. In the third convolution layer 925, "N×5×5 / 2↓" indicates that the convolution process is a convolution process of downsampling by using N filter kernels of a size of 5×5 at a stride interval of 2.

[0119] The output of the third convolution layer 925 may be input to the third GDN layer 930. The third GDN layer 930 may assign nonlinear characteristics to the input feature data.

[0120] The feature data output from the third GDN layer 930 may be input to the fourth convolution layer 935. In the fourth convolution layer 935, "M×5×5 / 2↓" indicates that the convolution process is a convolution process of downsampling by using M filter kernels of a size of 5×5 at a stride interval of 2.

[0121] The intra-frame prediction feature data 855 in the form of a feature map can be output from the fourth convolutional layer 935.

[0122] Fig. 9 It is shown that according to an embodiment of the present disclosure, the encoding neural network 850 includes four convolution layers (a first convolution layer 905, a second convolution layer 915, a third convolution layer 925, and a fourth convolution layer 935) and three GDN layers (a first GDN layer 910, a second GDN layer 920, and a third GDN layer 930). However, this is merely an example, and therefore, according to an embodiment of the present disclosure, the number of convolution layers and GDN layers included in the encoding neural network 850 may be variously modified.

[0123] In addition, reference Fig. 9 ,exist Figure 8 In the decoding neural network 870, the input data as the encoding context information 805, 810 and 815, the pixel information 820 and 825, and the intra-frame prediction feature data 855 can be input to the fifth convolution layer 945. In the fifth convolution layer 945, "N×5×5 / 2↑" indicates that the convolution process is a convolution process of upsampling by using N filter kernels of size 5×5 with a stride interval of 2.

[0124] The feature data generated by the fifth convolutional layer 945 may represent unique characteristics of the input data. For example, each feature data may represent a vertical direction characteristic, a horizontal direction characteristic, or an edge characteristic of the input data.

[0125] The feature data of the fifth convolution layer 945 may be input to the first inverse generalized divide normalization (IGDN) layer 940. The first IGDN layer 940 may be replaced with a sigmoid function, a tanh function, a rectified linear unit (ReLU) function, etc.; however, the present disclosure is not limited thereto.

[0126] The first IGDN layer 940 may assign nonlinear characteristics to each feature data.

[0127] Assigning a nonlinear characteristic by the first IGDN layer 940 may mean changing and outputting some sample values ​​of the feature data. In this case, the change may be performed by applying a nonlinear characteristic.

[0128] The first IGDN layer 940 may determine whether to send the sampled values ​​of the feature data to the sixth convolution layer 955. For example, some of the sampled values ​​of the feature data may be activated by the first IGDN layer 940 and sent to the sixth convolution layer 955, and some other sampled values ​​thereof may be deactivated by the first IGDN layer 940 and not sent to the sixth convolution layer 955. The unique characteristics of the input data represented by the feature data may be emphasized by the first IGDN layer 940.

[0129] The feature data output from the first IGDN layer 940 may be input to the sixth convolution layer 955. In the sixth convolution layer 955, "N×5×5 / 2↑" indicates that the convolution process is a convolution process of up-sampling at a stride interval of 2 by using N filter kernels of a size of 5×5.

[0130] The output of the sixth convolutional layer 955 may be input to the second IGDN layer 950. The second IGDN layer 950 may assign nonlinear characteristics to the input feature data.

[0131] The feature data output from the second IGDN layer 950 may be input to the seventh convolution layer 965. In the seventh convolution layer 965, "N×5×5 / 2↑" indicates that the convolution process is a convolution process of up-sampling by using N filter kernels of size 5×5 at a stride interval of 2.

[0132] The output of the seventh convolutional layer 965 may be input to the third IGDN layer 960. The third IGDN layer 960 may assign nonlinear characteristics to the input feature data.

[0133] The feature data output from the third IGDN layer 960 may be input to the eighth convolution layer 975. In the eighth convolution layer 945, "3×5×5 / 2↑" indicates that the convolution process is a convolution process of up-sampling with a stride interval of 2 by using 3 filter kernels of a size of 5×5.

[0134] An x component 875 of the intra stream, a y component 880 of the intra stream, and weight information 885 may be output from the eighth convolutional layer 975 . Fig. 9The decoding neural network 870 is shown to include four convolutional layers (a fifth convolutional layer 945, a sixth convolutional layer 955, a seventh convolutional layer 965, and an eighth convolutional layer 975) and three IGDN layers (a first IGDN layer 940, a second IGDN layer 950, and a third IGDN layer 960); however, this is merely an example, and in an embodiment of the present disclosure, the number of convolutional layers and IGDN layers included in the decoding neural network 870 may be modified differently.

[0135] Fig. 9 The above-mentioned structure of the neural network is merely an example, and the present disclosure is not limited thereto.

[0136] Fig.10 is a schematic diagram illustrating an image decoding method according to an embodiment of the present disclosure.

[0137] like Fig.10 As shown, intra prediction feature data 1020 of the current block, reference pixels 1010 located around the current block, and encoding context information 1030 may be input to the neural network 1040. According to an embodiment of the present disclosure, the neural network 1040 may output only the intra stream 1060. However, the present disclosure is not limited thereto, and therefore, according to an embodiment of the present disclosure, the neural network 1040 may output the intra stream 1060 and other information. For example, the neural network 1040 may output the intra stream 1060 and weight information 1070. According to an embodiment of the present disclosure, in the case where only the intra stream 1060 is obtained from the neural network 1040, the predicted pixel 1090 may be obtained by performing prediction 1080 using the intra stream 1060. Alternatively, in the case where the intra stream 1060 and the weight information 1070 are obtained from the neural network 1040, the predicted pixel 1090 may be obtained by performing prediction 1080 using the intra stream 1060 and the weight information 1070. In this case, the intra-frame prediction feature data 1020 obtained from the bitstream may be entropy decoded and input to the neural network 1040 .

[0138] When weight information is not used, the precision of the intra stream 1060 may be a fractional pixel, such as ½ pixel (half pixel) or ¼ pixel (quarter pixel).

[0139] The precision of the intra stream 1060 may be a pixel precision directly indicating a reference pixel when using weight information.

[0140] Fig.11 is a diagram for describing a neural network used in an image decoding method according to an embodiment of the present disclosure.

[0141] According to an embodiment of the present disclosure, Fig.11 The context decoder 1101 can be used with Figure 8 The context decoder 802 is the same as .

[0142] refer to Fig.11 , the intra-frame prediction feature data 1130 in the form of a feature map can be obtained from the bitstream. The intra-frame prediction feature data 1130 can be input to the context decoder 1101 together with the encoding context information 1105, 1110 and 1115 and the pixel information corresponding to the reference pixel 1120 and the zero pixel 1125. Here, the encoding context information 1105, 1110 and 1115 and the pixel information 1120 and 1125 can be processed by the neural network 1160 to the same size as the intra-frame prediction feature data 1130, and then cascaded (1140) with the intra-frame prediction feature data 1130 and input to the decoding neural network 1150. Here, the neural network 1160 can be a general convolutional neural network (CNN). In addition, the neural network 1160 can be a recurrent neural network (RNN). However, this is just an example, and the present disclosure is not limited thereto.

[0143] The x component 1160 of the intra-frame stream, the y component 1170 of the intra-frame stream, and weight information 1180 may be obtained from the decoding neural network 1150 .

[0144] Furthermore, instead of the x component 1160 of the intra stream and the y component 1170 of the intra stream, an angle component and a size component of the intra stream may be obtained.

[0145] Furthermore, the angular component of the intra-frame stream may be obtained only from the decoding neural network 1150 .

[0146] Furthermore, the intra-stream x component 1160 and the intra-stream y component 1170 may be obtained only from the decoding neural network 1150 .

[0147] Fig.12 Show Fig.11 An example of the structure of a decoding neural network.

[0148] Fig.11 The structure of the decoding neural network 1150 can be similar to Figure 8 The structure of the decoding neural network 870 is the same.

[0149] refer to Fig.12 ,exist Fig.11 In the decoding neural network 1150, the input data as the encoding context information 1105, 1110 and 1115, the pixel information 1120 and 1125, and the intra-frame prediction feature data 1130 may be input to the first convolution layer 1205. In the first convolution layer 1205, "N×5×5 / 2↑" indicates that the convolution process is a convolution process of upsampling by using N filter kernels of size 5×5 with a stride interval of 2. Here, N is an integer.

[0150] The feature data generated by the first convolutional layer 1205 may represent unique characteristics of the input data. For example, each feature data may represent a vertical direction characteristic, a horizontal direction characteristic, or an edge characteristic of the input data.

[0151] The feature data of the first convolution layer 1205 may be input to a first inverse generalized divide normalization (IGDN) layer 1210. A sigmoid function, a tanh function, a rectified linear unit (ReLU) function, etc. may be used instead of the first IGDN layer 1210; however, the present disclosure is not limited thereto.

[0152] The first IGDN layer 1210 may assign nonlinear characteristics to each feature data.

[0153] Assigning a non-linear characteristic by the first IGDN layer 1210 may mean changing and outputting some sample values ​​of the feature data. In this case, the change may be performed by applying a non-linear characteristic.

[0154] The first IGDN layer 1210 may determine whether to send the sampled values ​​of the feature data to the second convolution layer 1215. For example, some of the sampled values ​​of the feature data may be activated by the first IGDN layer 1210 and sent to the second convolution layer 1215, and some other sampled values ​​thereof may be deactivated by the first IGDN layer 1210 and not sent to the second convolution layer 1215. The unique characteristics of the input data represented by the feature data may be emphasized by the first IGDN layer 1210.

[0155] The feature data output from the first IGDN layer 1210 may be input to the second convolution layer 1215. In the second convolution layer 1215, "N×5×5 / 2↑" indicates that the convolution process is a convolution process of up-sampling with a stride interval of 2 by using N filter kernels of a size of 5×5.

[0156] The output of the second convolution layer 1215 may be input to the second IGDN layer 1220. The second IGDN layer 1220 may assign nonlinear characteristics to the input feature data.

[0157] The feature data output from the second IGDN layer 1220 may be input to the third convolution layer 1225. In the third convolution layer 1225, "N×5×5 / 2↑" indicates that the convolution process is a convolution process of up-sampling with a stride interval of 2 by using N filter kernels of a size of 5×5.

[0158] The output of the third convolution layer 1225 may be input to the third IGDN layer 1230. The third IGDN layer 1230 may assign nonlinear characteristics to the input feature data.

[0159] The feature data output from the third IGDN layer 1230 may be input to the fourth convolution layer 1235. In the fourth convolution layer 1234, "3×5×5 / 2↑" indicates that the convolution process is a convolution process of up-sampling with a stride interval of 2 by using 3 filter kernels of a size of 5×5.

[0160] An x component 1160 of the intra stream, a y component 1170 of the intra stream, and weight information 1180 may be output from the fourth convolutional layer 1235 . Fig.12 The decoding neural network 1150 is shown to include four convolutional layers (a first convolutional layer 1205, a second convolutional layer 1215, a third convolutional layer 1225, and a fourth convolutional layer 1235) and three IGDN layers (a first IGDN layer 1210, a second IGDN layer 1220, and a third IGDN layer 1230); however, this is merely an example, and in an embodiment of the present disclosure, the number of convolutional layers and IGDN layers included in the decoding neural network 1150 may be modified differently.

[0161] Fig.12 The above-mentioned structure of the neural network is merely an example, and the present disclosure is not limited thereto.

[0162] Fig.13 is a diagram for describing a method of training a neural network used in an image encoding method and an image decoding method according to an embodiment of the present disclosure.

[0163] refer to Fig.13 , the encoding neural network 1340 and the decoding neural network 1345 can be trained by using the original training pixels 1320, the reference training pixels 1310 and the training encoding context information 1330.

[0164] Specifically, the original training pixels 1320, the reference training pixels 1310, and the training encoding context information 1330 may be input to the encoding neural network 1340 to obtain the training intra-frame prediction feature data 1350. The training intra-frame prediction feature data 1350, the reference training pixels 1310, and the training encoding context information 1330 may be input to the decoding neural network 1345 to obtain the intra-frame stream 1360 and the weight information 1370. When the prediction 1380 is performed based on the intra-frame stream 1360 and the weight information 1370, the predicted training pixels 1390 may be obtained. Alternatively, the training intra-frame prediction feature data 1350, the reference training pixels 1310, and the training encoding context information 1330 may be input to the decoding neural network 1345 to obtain only the intra-frame stream 1360, and when the prediction 1380 is performed based on the intra-frame stream 1360, the predicted training pixels 1390 may be obtained.

[0165] exist Fig.13During the training process, by comparing 1355, the neural network can be trained so that the predicted training pixels 1390 become as similar as possible to the original training pixels 1320, and the bit rate of the bit stream generated by encoding the training intra-frame prediction feature data 1350 is minimized. To this end, Fig.13 As shown, the first loss information 1365 and the second loss information 1375 may be used to train a neural network.

[0166] The first loss information 1365 may correspond to the difference between the original training pixel 1320 and the predicted training pixel 1390. In an embodiment of the present disclosure, the difference between the original training pixel 1320 and the predicted training pixel 1390 may include at least one of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, or a video multi-method assessment fusion (VMAF) value between the original training pixel 1320 and the predicted training pixel 1390.

[0167] Since the first loss information 1365 is related to the quality of a reconstructed image using the predicted training pixels 1390, it may also be referred to as quality loss information.

[0168] The second loss information 1375 may be calculated according to a bit rate of a bitstream generated as a result of encoding the training intra-prediction feature data.

[0169] Since the second loss information 1375 is related to the encoding efficiency of the training intra-frame prediction feature data, the second loss information may also be referred to as compression loss information.

[0170] The encoding neural network 1340 and the decoding neural network 1345 may be trained to reduce or minimize final loss information derived from at least one of the first loss information 1365 or the second loss information 1375 .

[0171] In an embodiment of the present disclosure, the encoding neural network 1340 and the decoding neural network 1345 can reduce or minimize the final loss information while changing the values ​​of preset parameters.

[0172] In an embodiment of the present disclosure, the final loss information may be calculated according to Equation 2 below.

[0173] [Equation 2]

[0174] Final loss information = a × first loss information + b × second loss information

[0175] In Formula 2, “a” and “b” are weights applied to the first loss information 1365 and the second loss information 1375, respectively.

[0176] According to Equation 2, it can be seen that the encoding neural network 1340 and the decoding neural network 1345 are trained so that the training predicted pixels become as similar as possible to the training original pixels, and the size of the bitstream is minimized.

[0177] The intra-frame stream can be trained to adapt to the coding scheme and video content. The weight information can also be dynamically trained to adapt to the coding scheme and video content.

[0178] refer to Fig.13 The described training process may be performed by a training device. The training device may be, for example, the image encoding device 1900 or a separate server. Parameters obtained as a result of the training may be stored in the image encoding device 1900 and the image decoding device 1700.

[0179] Fig.14 Schematic diagrams illustrating an image encoding method and an image decoding method according to an embodiment of the present disclosure.

[0180] and Figure 7 Image decoding method and Fig.10 The image encoding method is different because Fig.14 The image decoding method does not have information obtained from the bit stream and Fig.14 The image coding method does not have signaling information, so it can be performed in the same way Fig.14 An image decoding method and an image encoding method.

[0181] like Fig.14 As shown, reference pixels 1410 and encoding context information 1420 located around the current block may be input to a neural network 1430. An intra-frame stream 1440 and weight information 1450 may be obtained from the neural network 1430, and a predicted pixel 1470 may be obtained by performing prediction 1460 using the intra-frame stream 1440 and the weight information 1450.

[0182] The neural network 1430 may be a convolutional neural network. In addition, the neural network 1430 may be a recurrent neural network (RNN). However, this is merely an example, and the present disclosure is not limited thereto.

[0183] Fig.15 It is used to describe training Fig.14 Graph of the neural network approach.

[0184] refer to Fig.15 , a neural network 1530 may be trained by using reference training pixels 1510 and training encoding context information 1520 .

[0185] For example, reference training pixels 1510 and training encoding context information 1520 may be input to a neural network 1530 to obtain an intra-stream 1540 and weight information 1550. When prediction 1560 is performed based on the intra-stream 1540 and the weight information 1550, predicted training pixels 1570 may be obtained.

[0186] exist Fig.15 During the training process, by comparing 1575, the neural network 1530 can be trained so that the predicted training pixels 1570 become as similar as possible to the original training pixels 1580. To this end, Fig.15 As shown, the neural network 1530 may be trained using the first loss information 1590. According to an embodiment of the present disclosure, the reference training pixel 1510, the predicted training pixel 1570, and the original training pixel 1580 may include one or more pixels.

[0187] The first loss information 1590 may correspond to the difference between the original training pixel 1580 and the predicted training pixel 1570. In an embodiment of the present disclosure, the difference between the original training pixel 1580 and the predicted training pixel 1570 may include at least one of an L1 norm value, an L2 norm value, a structural similarity (SSIM) value, a peak signal-to-noise ratio-human visual system (PSNR-HVS) value, a multi-scale SSIM (MS-SSIM) value, a variance inflation factor (VIF) value, or a video multi-method assessment fusion (VMAF) value between the original training pixel 1580 and the predicted training pixel 1570.

[0188] Since the first loss information 1590 is related to the quality of a reconstructed image using the predicted training pixels 1570, it may also be referred to as quality loss information.

[0189] Neural network 1530 may be trained to reduce or minimize final loss information derived from first loss information 1590 .

[0190] In an embodiment of the present disclosure, the neural network 1530 can reduce or minimize the final loss information while changing the value of the preset parameter.

[0191] In an embodiment of the present disclosure, the final loss information may be calculated according to Equation 3 below.

[0192] [Equation 3]

[0193] Final loss information = c × first loss information

[0194] In Equation 3, “c” is a weight applied to the first loss information 1590 .

[0195] According to Equation 3, it can be seen that the neural network 1530 is trained so that the training predicted pixels become as similar as possible to the training original pixels.

[0196] refer to Fig.15 The described training process may be performed by a training device. The training device may be, for example, the image encoding device 2300 or a separate server. Parameters obtained as a result of the training may be stored in the image encoding device 2300 and the image decoding device 2100.

[0197] Fig.16 is a flowchart illustrating an image decoding method according to an embodiment of the present disclosure.

[0198] refer to Fig.16 , in operation S1610, the image decoding device 1700 may obtain intra-frame prediction feature data of a current block from a bitstream.

[0199] In operation S1630, the image decoding device 1700 may determine an intra stream indicating a reference pixel of a current pixel in the current block by applying the intra prediction feature data, neighboring pixels of the current block, and encoding context information of the current block to a neural network.

[0200] According to an embodiment of the present disclosure, by applying intra-frame prediction feature data, neighboring pixels of the current block and encoding context information of the current block to a neural network, the image decoding device 1700 can determine an intra-frame flow of a reference pixel indicating a current pixel in the current block and weight information about the reference pixel and multiple neighboring pixels adjacent to the reference pixel.

[0201] According to an embodiment of the present disclosure, a neural network may output an intra-frame stream using inputs of intra-frame prediction feature data, reference pixels, and encoding context information.

[0202] According to an embodiment of the present disclosure, the intra stream may be represented by one of a vector including an x ​​component and a y component, a vector including a size component and an angle component, or an angle indicated by a current pixel relative to a reference pixel.

[0203] According to an embodiment of the present disclosure, intra-frame prediction feature data may be determined through neural network-based encoding.

[0204] According to an embodiment of the present disclosure, the coding context information may include at least one of a quantization parameter of the current block, a temporal layer of the current block, a partition structure of the current block, a partition structure of adjacent pixels, or a transform type of adjacent pixels. According to an embodiment of the present disclosure, the partition structure information of the current block and the partition structure information of the adjacent pixels may indicate how the current block and the adjacent pixels are partitioned (or divided) relative to a frame of the image. In addition, the partition structure information may also include hierarchical information indicating a level of partitioning (or division) within the tree structure, such as Figure 2 shown.

[0205] According to an embodiment of the present disclosure, encoding context information may be obtained through neural network-based encoding.

[0206] In operation S1650, the image decoding device 1700 may obtain a predicted pixel of a current pixel based on an intra stream of a current block.

[0207] According to an embodiment of the present disclosure, the image decoding device 1700 can further obtain weight information about a reference pixel indicated by an intra-frame stream and multiple neighboring pixels adjacent to the reference pixel from a neural network, and obtain a predicted pixel based on a weighted average using the weight information, the reference pixel, and multiple neighboring pixels.

[0208] According to an embodiment of the present disclosure, when weight information is not obtained, the accuracy of the intra stream may be a fractional pixel, such as ½ pixel (half pixel) or ¼ pixel (quarter pixel).

[0209] According to an embodiment of the present disclosure, when weight information is additionally obtained, the accuracy of the intra stream may be a pixel accuracy directly indicating a reference pixel.

[0210] In operation S1670, the image decoding device 1700 may reconstruct the current block based on the predicted pixels.

[0211] Fig.17 is a diagram showing a configuration of an image decoding device according to an embodiment of the present disclosure.

[0212] refer to Fig.17 , the image decoding device 1700 may include a receiver 1710, an artificial intelligence (AI) processor 1720, and a decoder 1730. However, the present disclosure is not limited thereto, and thus, the image decoding device 1700 may omit Fig.17 Some components shown in FIG. 17 and FIG. 18 may be different from those shown in FIG. 17 and FIG. 19 may include other components. For example, the image decoding apparatus 1700 may include a memory.

[0213] The receiver 1710, the AI ​​processor 1720, and the decoder 1730 may be implemented as a processor. The receiver 1710, the AI ​​processor 1720, and the decoder 1730 may operate according to instructions stored in a memory.

[0214] Fig.17The receiver 1710, the AI ​​processor 1720, and the decoder 1730 are shown separately; however, the receiver 1710, the AI ​​processor 1720, and the decoder 1730 may be implemented by a single processor. In this case, the receiver 1710, the AI ​​processor 1720, and the decoder 1730 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). In addition, the dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processor for using an external memory.

[0215] The receiver 1710, the AI ​​processor 1720, and the decoder 1730 may be implemented as a plurality of processors. In this case, the receiver 1710, the AI ​​processor 1720, and the decoder 1730 may be implemented by a combination of dedicated processors or by a combination of software and a plurality of general-purpose processors such as an AP, a CPU, or a GPU. In addition, the processor may include a dedicated AI processor. As another example, the dedicated AI processor may be configured as a chip separate from the processor.

[0216] The receiver 1710 may obtain the intra-frame prediction feature data of the current block from the bitstream. In addition, the receiver 1710 may entropy decode the intra-frame prediction feature data and send the result thereof to the AI ​​processor 1720.

[0217] In addition, the AI ​​processor 1720 may obtain reference pixels and encoding context information.

[0218] The AI ​​processor 1720 may determine an intra stream of reference pixels indicating a current pixel in the current block by applying the intra prediction feature data, neighboring pixels of the current block, and encoding context information of the current block to a neural network.

[0219] By applying intra-frame prediction feature data, neighboring pixels of the current block, and encoding context information of the current block to a neural network, the AI ​​processor 1720 can determine an intra-frame flow of a reference pixel indicating a current pixel in the current block and weight information about the reference pixel and multiple neighboring pixels adjacent to the reference pixel.

[0220] The AI ​​processor 1720 may transmit the intra stream to the decoder 1730 .

[0221] The AI ​​processor 1720 may transmit the intra-frame stream and weight information to the decoder 1730 .

[0222] The decoder 1730 may obtain a predicted pixel of the current pixel based on the intra stream of the current block. In addition, the decoder 1730 may reconstruct the current block based on the predicted pixel.

[0223] The decoder 1730 may obtain a predicted pixel of a current pixel based on an intra stream and weight information of a current block.

[0224] The bitstream may be sent over a network from the image encoding apparatus 1900. In an embodiment of the present disclosure, the bitstream may be stored in a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, and / or a magneto-optical medium such as an optical magneto-floppy disk.

[0225] Fig.18 is a flowchart illustrating an image encoding method according to an embodiment of the present disclosure.

[0226] refer to Fig.18 In operation S1810, the image encoding device 1900 may determine intra-frame prediction feature data of the current block and an intra-frame stream of reference pixels indicating the original pixels of the current block by applying the original pixels of the current block, the neighboring pixels of the current block, and the encoding context information of the current block to a neural network.

[0227] According to an embodiment of the present disclosure, by applying the original pixels of the current block, the neighboring pixels of the current block, and the encoding context information of the current block to a neural network, the image encoding device 1900 can determine the intra-frame prediction feature data of the current block, the intra-frame stream of the reference pixels indicating the original pixels of the current block, and the weight information about the reference pixels indicated by the intra-frame stream and multiple neighboring pixels adjacent to the reference pixels.

[0228] According to an embodiment of the present disclosure, a neural network may include: an encoding neural network that uses inputs of original pixels, reference pixels, and encoding context information to output intra-frame prediction feature data, and a decoding neural network that uses inputs of intra-frame prediction feature data, reference pixels, and encoding context information to output intra-frame streams.

[0229] According to an embodiment of the present disclosure, the decoding neural network can also output weight information.

[0230] According to an embodiment of the present disclosure, the intra stream may be represented by one of a vector including an x ​​component and a y component, a vector including a size component and an angle component, or an angle indicated by a current pixel relative to a reference pixel.

[0231] According to an embodiment of the present disclosure, the encoding context information may include at least one of a quantization parameter of a current block, a temporal layer of a current block, a partition structure of a current block, a partition structure of adjacent pixels, or a transform type of adjacent pixels.

[0232] In operation S1830, the image encoding device 1900 may obtain predicted pixels of original pixels based on an intra stream of the current block.

[0233] According to an embodiment of the present disclosure, the image encoding device 1900 can further obtain weight information about a reference pixel indicated by an intra-frame stream and multiple neighboring pixels adjacent to the reference pixel from a neural network, and obtain a predicted pixel based on a weighted average using the weight information, the reference pixel, and multiple neighboring pixels.

[0234] According to an embodiment of the present disclosure, when weight information is not obtained, the accuracy of the intra stream may be a fractional pixel, such as ½ pixel (half pixel) or ¼ pixel (quarter pixel).

[0235] According to an embodiment of the present disclosure, when weight information is additionally obtained, the accuracy of the intra stream may be a pixel accuracy directly indicating a reference pixel.

[0236] In operation S1850, the image encoding device 1900 may encode intra prediction feature data by reconstructing the current block based on the predicted pixels.

[0237] Fig.19 is a diagram showing the configuration of an image encoding device according to an embodiment of the present disclosure.

[0238] refer to Fig.19 , the image encoding device 1900 may include a first AI processor 1910, a second AI processor 1920, and an encoder 1930. However, the present disclosure is not limited thereto, and therefore, the image encoding device 1900 may omit Fig.19 Some components shown in the figure may be different from those shown in the figure, or other components may be included. For example, the image encoding device 1900 may include a memory.

[0239] The first AI processor 1910, the second AI processor 1920, and the encoder 1930 may be implemented as processors. The first AI processor 1910, the second AI processor 1920, and the encoder 1930 may operate according to instructions stored in a memory.

[0240] Fig.19 The first AI processor 1910, the second AI processor 1920, and the encoder 1930 are shown separately; however, the first AI processor 1910, the second AI processor 1920, and the encoder 1930 may be implemented by a single processor. In this case, the first AI processor 1910, the second AI processor 1920, and the encoder 1930 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). In addition, the dedicated processor may include a memory processor for implementing an embodiment of the present disclosure or for using an external memory.

[0241] The first AI processor 1910, the second AI processor 1920, and the encoder 1930 may be implemented as a plurality of processors. In this case, the first AI processor 1910, the second AI processor 1920, and the encoder 1930 may be implemented by a combination of dedicated processors or by a combination of software and a plurality of general-purpose processors such as an AP, a CPU, or a GPU. In addition, the processor may include a dedicated AI processor. As another example, the dedicated AI processor may be configured as a chip separate from the processor.

[0242] The first AI processor 1910 may determine intra-prediction feature data of the current block by applying original pixels of the current block, neighboring pixels of the current block, and encoding context information of the current block to an encoding neural network.

[0243] The first AI processor 1910 may transmit the intra prediction feature data to the second AI processor 1920 and the encoder 1930 .

[0244] In addition, the second AI processor 1920 may obtain reference pixels and encoding context information.

[0245] The second AI processor 1920 may determine an intra stream indicating a reference pixel of a current pixel in the current block by applying the intra prediction feature data, neighboring pixels of the current block, and encoding context information of the current block to a decoding neural network.

[0246] By applying intra-frame prediction feature data, neighboring pixels of the current block, and encoding context information of the current block to a neural network, the second AI processor 1920 can determine an intra-frame flow of a reference pixel indicating a current pixel in the current block and weight information about the reference pixel and multiple neighboring pixels adjacent to the reference pixel.

[0247] The second AI processor 1920 may transmit the intra stream to the encoder 1930 .

[0248] The second AI processor 1920 may transmit the intra-stream and weight information to the encoder 1930 .

[0249] The encoder 1930 may obtain a predicted pixel of a current pixel based on an intra stream of a current block. In addition, the encoder 1930 may encode intra prediction feature data by reconstructing the current block based on the predicted pixel.

[0250] The encoder 1930 may obtain a predicted pixel of a current pixel based on the intra stream and weight information of the current block. In addition, the encoder 1930 may encode intra prediction feature data by reconstructing the current block based on the predicted pixel.

[0251] The intra-prediction feature data may be entropy encoded and transmitted to the image decoding device 1700 .

[0252] Fig. 20 is a flowchart illustrating an image decoding method according to an embodiment of the present disclosure.

[0253] refer to Fig. 20 In operation S2010, the image decoding device 2100 may determine an intra-stream of a reference pixel indicating a current pixel in the current block and weight information of the reference pixel and a plurality of neighboring pixels adjacent to the reference pixel by applying neighboring pixels of the current block and encoding context information to a neural network.

[0254] According to an embodiment of the present disclosure, the neural network may be a convolutional neural network.

[0255] According to an embodiment of the present disclosure, the image decoding device 2100 may determine an intra-stream of reference pixels indicating a current pixel in the current block by applying neighboring pixels of the current block and encoding context information to a neural network.

[0256] According to an embodiment of the present disclosure, when weight information is not obtained, the accuracy of the intra stream may be a fractional pixel, such as ½ pixel (half pixel) or ¼ pixel (quarter pixel).

[0257] According to an embodiment of the present disclosure, when weight information is additionally obtained, the accuracy of the intra stream may be a pixel accuracy directly indicating a reference pixel.

[0258] In operation S2030, the image decoding device 2100 may determine a predicted pixel of a current pixel based on the intra stream and weight information.

[0259] According to an embodiment of the present disclosure, the image decoding device 2100 may determine a predicted pixel of a current pixel based on an intra stream.

[0260] In operation S2050, the image decoding device 2100 may reconstruct the current block based on the predicted pixels.

[0261] Fig.21 is a diagram showing a configuration of an image decoding device according to an embodiment of the present disclosure.

[0262] refer to Fig.21 , the image decoding device 2100 may include an AI processor 2110 and a decoder 2120. However, the present disclosure is not limited thereto, and thus, the image decoding device 2100 may include other components. For example, the image decoding device 2100 may include a memory.

[0263] The AI ​​processor 2110 and the decoder 2120 may be implemented as processors. The AI ​​processor 2110 and the decoder 2120 may operate according to instructions stored in a memory.

[0264] Fig.21 The AI ​​processor 2110 and the decoder 2120 are shown separately; however, the AI ​​processor 2110 and the decoder 2120 may be implemented by a single processor. In this case, the AI ​​processor 2110 and the decoder 2120 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). In addition, the dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processor for using an external memory.

[0265] The AI ​​processor 2110 and the decoder 2120 may be implemented as a plurality of processors. In this case, the AI ​​processor 2110 and the decoder 2120 may be implemented by a combination of dedicated processors or by a combination of software and a plurality of general-purpose processors such as an AP, a CPU, or a GPU. In addition, the processor may include a dedicated AI processor. As another example, the dedicated AI processor may be configured as a chip separate from the processor.

[0266] The AI ​​processor 2110 may obtain reference pixels of the current block and encoding context information of the current block.

[0267] By applying neighboring pixels of the current block and encoding context information of the current block to a neural network, the AI ​​processor 2110 can determine an intra-frame flow of reference pixels indicating the current pixel in the current block and weight information about the reference pixel and the neighboring pixels of the reference pixel.

[0268] The AI ​​processor 2110 may transmit the intra-frame stream and weight information to the decoder 2120 .

[0269] The decoder 2120 may obtain a predicted pixel of the current pixel based on the intra stream and weight information of the current block. In addition, the decoder 2120 may reconstruct the current block based on the predicted pixel.

[0270] Fig. 22 is a flowchart illustrating an image encoding method according to an embodiment of the present disclosure.

[0271] refer to Fig. 22 In operation S2210, the image encoding device 2300 may determine an intra-stream of a reference pixel indicating a current pixel in a current block and weight information of the reference pixel and a plurality of neighboring pixels adjacent to the reference pixel by applying neighboring pixels of the current block and encoding context information to a neural network.

[0272] According to an embodiment of the present disclosure, the neural network may be a convolutional neural network.

[0273] According to an embodiment of the present disclosure, the image encoding device 2300 may determine an intra-stream of reference pixels indicating a current pixel in the current block by applying neighboring pixels of the current block and encoding context information to a neural network.

[0274] According to an embodiment of the present disclosure, when weight information is not obtained, the accuracy of the intra stream may be a fractional pixel, such as ½ pixel (half pixel) or ¼ pixel (quarter pixel).

[0275] According to an embodiment of the present disclosure, when weight information is additionally obtained, the accuracy of the intra stream may be a pixel accuracy directly indicating a reference pixel.

[0276] In operation S2230 , the image encoding device 2300 may determine a predicted pixel of a current pixel based on the intra stream and weight information.

[0277] According to an embodiment of the present disclosure, the image encoding device 2300 may determine a predicted pixel of a current pixel based on an intra stream.

[0278] In operation S2250, the image encoding device 2300 may encode the current block based on the predicted pixels.

[0279] Fig.23 is a diagram showing the configuration of an image encoding device according to an embodiment of the present disclosure.

[0280] refer to Fig.23 , the image encoding device 2300 may include an AI processor 2310 and an encoder 2320. However, the present disclosure is not limited thereto, and therefore, the image encoding device 2300 may include other components. For example, the image encoding device 2300 may include a memory.

[0281] The AI ​​processor 2310 and the encoder 2320 may be implemented as processors. The AI ​​processor 2310 and the encoder 2320 may operate according to instructions stored in a memory.

[0282] Fig.23 The AI ​​processor 2310 and the encoder 2320 are shown separately; however, the AI ​​processor 2310 and the encoder 2320 may be implemented by a single processor. In this case, the AI ​​processor 2310 and the encoder 2320 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). In addition, the dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processor for using an external memory.

[0283] The AI ​​processor 2310 and the encoder 2320 may be implemented as a plurality of processors. In this case, the AI ​​processor 2310 and the encoder 2320 may be implemented by a combination of dedicated processors or by a combination of software and a plurality of general-purpose processors such as an AP, a CPU, or a GPU. In addition, the processor may include a dedicated AI processor. As another example, the dedicated AI processor may be configured as a chip separate from the processor.

[0284] The AI ​​processor 2310 may obtain reference pixels of the current block and encoding context information of the current block.

[0285] By applying neighboring pixels of the current block and encoding context information of the current block to a neural network, the AI ​​processor 2310 may determine an intra-stream of reference pixels indicating a current pixel in the current block and weight information about the reference pixels and neighboring pixels of the reference pixels.

[0286] The AI ​​processor 2310 may transmit the intra-frame stream and weight information to the encoder 2320 .

[0287] The encoder 2320 may obtain a predicted pixel of a current pixel based on the intra stream and weight information of the current block. In addition, the encoder 2320 may encode the current block based on the predicted pixel.

[0288] According to an embodiment of the present disclosure, an image decoding method may include: obtaining intra-frame prediction feature data of a current block from a bitstream, determining an intra-frame stream of reference pixels indicating a current pixel in the current block by applying the intra-frame prediction feature data, neighboring pixels of the current block, and encoding context information of the current block to a neural network, obtaining predicted pixels of the current pixel based on the intra-frame stream of the current block, and reconstructing the current block based on the predicted pixels.

[0289] Unlike the intra-frame prediction in the prior art, according to the image decoding method of the embodiment of the present disclosure, since each prediction direction is applied to each pixel instead of applying one prediction direction to all pixels in the block to be predicted, and a trained intra-frame stream is obtained by using the input of intra-frame prediction feature data, adjacent pixels and encoding context information through a neural network, and predicted pixels are obtained by using the intra-frame stream instead of directly generating predicted pixels through the input of adjacent pixels through a neural network, the efficiency of intra-frame prediction can be improved by using an intra-frame stream with a unique spatial vector indicating the reference pixels available for each pixel of the block to be predicted.

[0290] According to an embodiment of the present disclosure, the step of obtaining a predicted pixel of a current pixel based on an intra-frame flow of a current block may include: additionally obtaining weight information about a reference pixel indicated by the intra-frame flow and a plurality of neighboring pixels adjacent to the reference pixel from a neural network, and obtaining a predicted pixel based on a weighted average using the weight information, the reference pixel, and the plurality of neighboring pixels.

[0291] Unlike the intra-frame prediction in the related art, according to the image decoding method according to the embodiment of the present disclosure, since a predetermined filter is not used, a trained intra-frame stream is obtained through a neural network, and predicted pixels are obtained by using the intra-frame stream and trained weight information, the efficiency of intra-frame prediction can be improved by using a weighted average based on the weight information and the intra-frame stream.

[0292] According to an embodiment of the present disclosure, a neural network may output an intra-frame stream using inputs of intra-frame prediction feature data, reference pixels, and encoding context information.

[0293] According to an embodiment of the present disclosure, the intra stream may be represented by one of a vector including an x ​​component and a y component, a vector including a size component and an angle component, or an angle indicated by a current pixel relative to a reference pixel.

[0294] According to an embodiment of the present disclosure, intra-frame prediction feature data may be determined through neural network-based encoding.

[0295] According to an embodiment of the present disclosure, the encoding context information may include at least one of a quantization parameter of a current block, a temporal layer of a current block, a partition structure of a current block, a partition structure of adjacent pixels, or a transform type of adjacent pixels.

[0296] According to an embodiment of the present disclosure, encoding context information may be obtained through neural network-based encoding.

[0297] According to an embodiment of the present disclosure, an image decoding device may include a memory storing one or more instructions, and at least one processor configured to operate according to the one or more instructions, wherein the at least one processor may be configured to obtain intra-frame prediction feature data of a current block from a bitstream, determine an intra-frame stream of reference pixels indicating a current pixel in the current block by applying the intra-frame prediction feature data, neighboring pixels of the current block, and encoding context information of the current block to a neural network, obtain predicted pixels of the current pixel based on the intra-frame stream of the current block, and reconstruct the current block based on the predicted pixels.

[0298] Unlike the intra-frame prediction in the related art, the image decoding device according to the embodiment of the present disclosure, since each prediction direction is applied to each pixel instead of applying one prediction direction to all pixels in the block to be predicted, and a trained intra-frame stream is obtained by using the input of intra-frame prediction feature data, adjacent pixels and encoding context information through a neural network, and predicted pixels are obtained by using the intra-frame stream instead of directly generating predicted pixels through the input of adjacent pixels through a neural network, the efficiency of intra-frame prediction can be improved by using the intra-frame stream with a unique spatial vector indicating the reference pixels available for each pixel of the block to be predicted.

[0299] According to an embodiment of the present disclosure, obtaining a predicted pixel of a current pixel based on an intra-frame flow of a current block may include: additionally obtaining weight information about a reference pixel indicated by the intra-frame flow and a plurality of neighboring pixels adjacent to the reference pixel from a neural network, and obtaining a predicted pixel based on a weighted average using the weight information, the reference pixel, and the plurality of neighboring pixels.

[0300] Unlike the intra-frame prediction in the related art, according to the image decoding device according to the embodiment of the present disclosure, since a predetermined filter is not used, a trained intra-frame stream is obtained through a neural network, and predicted pixels are obtained by using the intra-frame stream and trained weight information, the efficiency of intra-frame prediction can be improved by using a weighted average based on the weight information and the intra-frame stream.

[0301] According to an embodiment of the present disclosure, a neural network may output an intra-frame stream using inputs of intra-frame prediction feature data, reference pixels, and encoding context information.

[0302] According to an embodiment of the present disclosure, the intra stream may be represented by one of a vector including an x ​​component and a y component, a vector including a size component and an angle component, or an angle indicated by a current pixel relative to a reference pixel.

[0303] According to an embodiment of the present disclosure, intra-frame prediction feature data may be determined through neural network-based encoding.

[0304] According to an embodiment of the present disclosure, the encoding context information may include at least one of a quantization parameter of a current block, a temporal layer of a current block, a partition structure of a current block, a partition structure of adjacent pixels, or a transform type of adjacent pixels.

[0305] According to an embodiment of the present disclosure, encoding context information may be obtained through neural network-based encoding.

[0306] According to an embodiment of the present disclosure, an image encoding method may include: determining intra-frame prediction feature data of the current block and an intra-frame stream of reference pixels indicating the original pixels of the current block by applying original pixels of the current block, adjacent pixels of the current block, and encoding context information of the current block to a neural network, obtaining predicted pixels of the original pixels based on the intra-frame stream of the current block, and encoding the intra-frame prediction feature data by reconstructing the current block based on the predicted pixels.

[0307] Unlike the intra-frame prediction in the related art, the image encoding method according to the embodiment of the present disclosure, since each prediction direction is applied to each pixel instead of applying one prediction direction to all pixels in the block to be predicted, and a trained intra-frame stream is obtained by a neural network using the input of original pixels, adjacent pixels and encoding context information, and predicted pixels are obtained by using the intra-frame stream instead of directly generating predicted pixels by a neural network using the input of adjacent pixels, the efficiency of intra-frame prediction can be improved by using an intra-frame stream with a unique spatial vector indicating the reference pixels available for each pixel of the block to be predicted.

[0308] According to an embodiment of the present disclosure, obtaining a predicted pixel of a current pixel based on an intra-frame flow of a current block may include: additionally obtaining weight information about a reference pixel indicated by the intra-frame flow and a plurality of neighboring pixels adjacent to the reference pixel from a neural network, and obtaining a predicted pixel based on a weighted average using the weight information, the reference pixel, and the plurality of neighboring pixels.

[0309] Unlike the intra-frame prediction in the related art, according to the image encoding method according to the embodiment of the present disclosure, since a predetermined filter is not used, a trained intra-frame stream is obtained through a neural network, and predicted pixels are obtained by using the intra-frame stream and trained weight information, the efficiency of intra-frame prediction can be improved by using a weighted average based on the weight information and the intra-frame stream.

[0310] According to an embodiment of the present disclosure, a neural network may include: an encoding neural network that uses inputs of original pixels, reference pixels, and encoding context information to output intra-frame prediction feature data, and a decoding neural network that uses inputs of intra-frame prediction feature data, reference pixels, and encoding context information to output intra-frame streams.

[0311] According to an embodiment of the present disclosure, the decoding neural network can also output weight information.

[0312] According to an embodiment of the present disclosure, the intra stream may be represented by one of a vector including an x ​​component and a y component, a vector including a size component and an angle component, or an angle indicated by a current pixel relative to a reference pixel.

[0313] According to an embodiment of the present disclosure, the encoding context information may include at least one of a quantization parameter of a current block, a temporal layer of a current block, a partition structure of a current block, a partition structure of adjacent pixels, or a transform type of adjacent pixels.

[0314] According to an embodiment of the present disclosure, an image encoding device may include a memory storing one or more instructions, and at least one processor configured to operate according to the one or more instructions, wherein the at least one processor may be configured to determine intra-frame prediction feature data of the current block and an intra-frame stream of reference pixels indicating the original pixels of the current block by applying the original pixels of the current block, the neighboring pixels of the current block, and the encoding context information of the current block to a neural network, obtain predicted pixels of the original pixels based on the intra-frame stream of the current block, and encode the intra-frame prediction feature data by reconstructing the current block based on the predicted pixels.

[0315] Unlike the intra-frame prediction in the related art, the image encoding device according to the embodiment of the present disclosure, since each prediction direction is applied to each pixel instead of applying one prediction direction to all pixels in the block to be predicted, and a trained intra-frame stream is obtained by using the input of original pixels, adjacent pixels and encoding context information through a neural network, and predicted pixels are obtained by using the intra-frame stream instead of directly generating predicted pixels through the input of adjacent pixels through a neural network, the efficiency of intra-frame prediction can be improved by using the intra-frame stream with a unique spatial vector indicating the reference pixels available for each pixel of the block to be predicted.

[0316] According to an embodiment of the present disclosure, obtaining a predicted pixel of a current pixel based on an intra-frame flow of a current block may include: additionally obtaining weight information about a reference pixel indicated by the intra-frame flow and a plurality of neighboring pixels adjacent to the reference pixel from a neural network, and obtaining a predicted pixel based on a weighted average using the weight information, the reference pixel, and the plurality of neighboring pixels.

[0317] Unlike the intra-frame prediction in the related art, according to the image encoding device according to the embodiment of the present disclosure, since a predetermined filter is not used, a trained intra-frame stream is obtained through a neural network, and predicted pixels are obtained by using the intra-frame stream and trained weight information, the efficiency of intra-frame prediction can be improved by using a weighted average based on the weight information and the intra-frame stream.

[0318] According to an embodiment of the present disclosure, a neural network may include: an encoding neural network that uses inputs of original pixels, reference pixels, and encoding context information to output intra-frame prediction feature data, and a decoding neural network that uses inputs of intra-frame prediction feature data, reference pixels, and encoding context information to output intra-frame streams.

[0319] According to an embodiment of the present disclosure, the decoding neural network can also output weight information.

[0320] According to an embodiment of the present disclosure, the intra stream may be represented by one of a vector including an x ​​component and a y component, a vector including a size component and an angle component, or an angle indicated by a current pixel relative to a reference pixel.

[0321] According to an embodiment of the present disclosure, the encoding context information may include at least one of a quantization parameter of a current block, a temporal layer of a current block, a partition structure of a current block, a partition structure of adjacent pixels, or a transform type of adjacent pixels.

[0322] According to an embodiment of the present disclosure, an image decoding method may include: determining an intra-frame stream of a reference pixel indicating a current pixel in the current block and weight information of the reference pixel and a plurality of adjacent pixels adjacent to the reference pixel by applying adjacent pixels of the current block and encoding context information to a neural network, determining a predicted pixel of the current pixel based on the intra-frame stream and the weight information, and reconstructing the current block based on the predicted pixel.

[0323] According to the image decoding method of the embodiment of the present disclosure, since the trained intra-frame stream and weight information are obtained by using the input of adjacent pixels and encoding context information through a neural network, and the predicted pixel is obtained by using the intra-frame stream and the weight information, the efficiency of intra-frame prediction can be improved by using the weight information and the intra-frame stream with a unique spatial vector, which indicates the reference pixels available for each pixel of the block to be predicted.

[0324] According to an embodiment of the present disclosure, the neural network may be a convolutional neural network.

[0325] According to an embodiment of the present disclosure, an intra-frame flow of a reference pixel indicating a current pixel in the current block may be determined by applying neighboring pixels of the current block and encoding context information to a neural network.

[0326] According to an embodiment of the present disclosure, a predicted pixel of a current pixel may be determined based on an intra-frame stream.

[0327] According to an embodiment of the present disclosure, an image decoding device may include a memory storing one or more instructions, and at least one processor configured to operate according to the one or more instructions, wherein the at least one processor may be configured to determine an intra-frame stream of a reference pixel indicating a current pixel in the current block and weight information of the reference pixel and multiple neighboring pixels adjacent to the reference pixel by applying neighboring pixels of the current block and encoding context information to a neural network, determine a predicted pixel of the current pixel based on the intra-frame stream and the weight information, and reconstruct the current block based on the predicted pixel.

[0328] According to the image decoding device of the embodiment of the present disclosure, since the trained intra-frame stream and weight information are obtained by using the input of adjacent pixels and encoding context information through a neural network, and the predicted pixel is obtained by using the intra-frame stream and the weight information, the efficiency of intra-frame prediction can be improved by using the weight information and the intra-frame stream with a unique spatial vector, which indicates the reference pixels available for each pixel of the block to be predicted.

[0329] According to an embodiment of the present disclosure, the neural network may be a convolutional neural network.

[0330] According to an embodiment of the present disclosure, an intra-frame flow of a reference pixel indicating a current pixel in the current block may be determined by applying neighboring pixels of the current block and encoding context information to a neural network.

[0331] According to an embodiment of the present disclosure, a predicted pixel of a current pixel may be determined based on an intra-frame stream.

[0332] According to an embodiment of the present disclosure, an image encoding method may include: determining an intra-frame flow of a reference pixel indicating a current pixel in the current block and weight information of the reference pixel and a plurality of adjacent pixels adjacent to the reference pixel by applying adjacent pixels of the current block and encoding context information to a neural network, determining a predicted pixel of the current pixel based on the intra-frame flow and the weight information, and encoding the current block based on the predicted pixel.

[0333] According to the image encoding method of the embodiment of the present disclosure, since the trained intra-frame flow and weight information are obtained by using the input of adjacent pixels and encoding context information through a neural network, and the predicted pixel is obtained by using the intra-frame flow and weight information, the efficiency of intra-frame prediction can be improved by using weight information and an intra-frame flow with a unique spatial vector, which indicates the reference pixels available for each pixel of the block to be predicted.

[0334] According to an embodiment of the present disclosure, the neural network may be a convolutional neural network.

[0335] According to an embodiment of the present disclosure, an intra-frame flow of a reference pixel indicating a current pixel in the current block may be determined by applying neighboring pixels of the current block and encoding context information to a neural network.

[0336] According to an embodiment of the present disclosure, a predicted pixel of a current pixel may be determined based on an intra-frame stream.

[0337] According to an embodiment of the present disclosure, an image encoding device may include a memory storing one or more instructions, and at least one processor configured to operate according to the one or more instructions, wherein the at least one processor may be configured to determine an intra-frame stream of a reference pixel indicating a current pixel in the current block and weight information of the reference pixel and multiple neighboring pixels adjacent to the reference pixel by applying neighboring pixels of the current block and encoding context information to a neural network, determine a predicted pixel of the current pixel based on the intra-frame stream and the weight information, and encode the current block based on the predicted pixel.

[0338] According to the image encoding device of the embodiment of the present disclosure, since the trained intra-frame flow and weight information are obtained by using the input of adjacent pixels and encoding context information through a neural network, and the predicted pixel is obtained by using the intra-frame flow and weight information, the efficiency of intra-frame prediction can be improved by using weight information and an intra-frame flow with a unique spatial vector indicating the reference pixels available for each pixel of the block to be predicted.

[0339] According to an embodiment of the present disclosure, the neural network may be a convolutional neural network.

[0340] According to an embodiment of the present disclosure, an intra-frame flow of a reference pixel indicating a current pixel in the current block may be determined by applying neighboring pixels of the current block and encoding context information to a neural network.

[0341] According to an embodiment of the present disclosure, a predicted pixel of a current pixel may be determined based on an intra-frame stream.

[0342] The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" may mean that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), and may mean that data may be stored in the storage medium semi-permanently or temporarily. For example, the "non-transitory storage medium" may include a buffer that temporarily stores data.

[0343] According to an embodiment of the present disclosure, the method according to the various embodiments described herein may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or distributed directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer server, a memory of an application store server, or a memory of a relay server.

Claims

1. An image decoding method, comprising: Obtaining intra-frame prediction feature data of a current block from a bitstream (S1610); Obtaining intra-frame flow information indicating a reference pixel of a current pixel in the current block based on the intra-frame prediction feature data, a first neighboring pixel of the current block, and encoding context information of the current block (S1630); Obtaining a predicted pixel of a current pixel based on intra-frame stream information of a current block (S1650); and The current block is reconstructed based on the predicted pixels ( S1670 ).

2. The image decoding method according to claim 1, further comprising: Applying intra-prediction feature data, first neighboring pixels, and coding context information of the current block to a neural network; obtaining, from a neural network, an intra-frame flow and weight information about a reference pixel indicated by the intra-frame flow information and a plurality of second neighboring pixels adjacent to the reference pixel; as well as The predicted pixel is obtained based on a weighted average, where the weighted average is obtained based on the weight information, the reference pixel and the plurality of second neighboring pixels.

3. The image decoding method according to claim 1 or 2, wherein: The neural network outputs intra-frame flow information based on the intra-frame prediction feature data, the first neighboring pixels of the current block and the encoding context information provided as input to the neural network.

4. The image decoding method according to any one of claims 1 to 3, wherein: The intra-frame stream information includes one of a first vector including an x ​​component and a y component, a second vector including a size component and an angle component, or an angle indicated by a current pixel relative to a reference pixel.

5. The image decoding method according to any one of claims 1 to 4, wherein: The intra-frame prediction feature data is obtained through neural network-based encoding.

6. The image decoding method according to any one of claims 1 to 5, wherein: The encoding context information includes at least one of a quantization parameter of the current block, a temporal layer of the current block, a partition structure of the current block, a partition structure of the first neighboring pixels, or a transform type of the first neighboring pixels.

7. The image decoding method according to any one of claims 1 to 6, wherein: The encoded context information is obtained through neural network based encoding.

8. An image encoding method, comprising: Obtaining intra prediction feature data of the current block and intra stream information indicating reference pixels corresponding to the original pixels of the current block based on original pixels of the current block, first neighboring pixels of the current block, and encoding context information of the current block (S1810); Obtain predicted pixels of original pixels based on intra-frame stream information of the current block (S1830); and The intra-prediction feature data is encoded by reconstructing the current block based on the predicted pixels ( S1850 ).

9. The image encoding method according to claim 8, further comprising: Applying original pixels of the current block, first neighboring pixels of the current block, and encoded context information of the current block to a neural network; Obtaining, from a neural network, intra-frame prediction feature data of a current block, intra-frame flow information, and weight information about a reference pixel indicated by the intra-frame flow information and a plurality of second adjacent pixels adjacent to the reference pixel; as well as The predicted pixel is obtained based on a weighted average, where the weighted average is obtained based on the weight information, the reference pixel and the plurality of second neighboring pixels.

10. The image encoding method according to claim 8 or 9, further comprising: Outputting intra-frame prediction feature data by inputting original pixels, reference pixels, and coding context information into the coding neural network, and The intra-frame flow information is output by inputting the intra-frame prediction feature data, reference pixels and coding context information into the decoding neural network.

11. The image encoding method according to any one of claims 8 to 10, wherein: The decoding neural network also outputs weight information.

12. The image encoding method according to any one of claims 8 to 11, wherein: The intra stream information is represented by one of a vector including an x ​​component and a y component, a vector including a size component and an angle component, or an angle indicated by a current pixel with respect to a reference pixel.

13. The image encoding method according to any one of claims 8 to 12, wherein: The encoding context information includes at least one of a quantization parameter of the current block, a temporal layer of the current block, a partition structure of the current block, a partition structure of the first neighboring pixels, or a transform type of the first neighboring pixels.

14. An image decoding device, comprising: a memory storing one or more instructions; and At least one processor configured to operate according to the one or more instructions to: Get the intra prediction feature data of the current block from the bitstream, Based on the intra prediction feature data, the first neighboring pixel of the current block and the encoding context information of the current block, obtaining intra flow information indicating a reference pixel of the current pixel in the current block, Obtaining a predicted pixel of a current pixel based on the intra-frame flow information of the current block, and Reconstruct the current block based on the predicted pixels.

15. An image decoding method, comprising: Based on the first neighboring pixel of the current block and the encoding context information, obtaining intra-frame stream information indicating a reference pixel corresponding to a current pixel in the current block and weight information of the reference pixel and a plurality of second neighboring pixels adjacent to the reference pixel (S2010); Obtaining a predicted pixel of a current pixel based on the intra-frame flow information and the weight information (S2030); and The current block is reconstructed based on the predicted pixels ( S2050 ).

Images