Image decoding device, image decoding method, image encoding device and image encoding method for optimized quantization and dequantization

Abstract

Provided are an image decoding method and device comprising: acquiring, from a bitstream, second feature data for first feature data acquired through neural network-based encoding for the current image and a quantization index indicating one of a plurality of quantization steps; acquiring the quantization step on the basis of the quantization index; acquiring probability data by applying the second feature data to a first neural network; modifying the probability data on the basis of the quantization step; acquiring quantized first feature data by applying entropy decoding based on the modified probability data to bits included in the bitstream; acquiring dequantized first feature data by dequantizing the quantized first feature data according to the quantization step; and reconstructing the current image through neural network-based decoding for the dequantized first feature data.

Description

Image decoding device, image decoding method, image encoding device, and image encoding method for optimized quantization and dequantization

[0001] The present disclosure relates to image encoding and decoding. More specifically, the present disclosure relates to a technique for encoding and decoding images using artificial intelligence (AI), such as a neural network.

[0002] In codecs such as H.264 AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding), an image can be divided into blocks, and each block can be predicted and decoded through inter prediction or intra prediction.

[0003] Intra prediction is a method of compressing images by removing spatial redundancy within the image, and inter prediction is a method of compressing images by removing temporal redundancy between images.

[0004] A representative example of inter prediction is motion estimation coding. Motion estimation coding uses a reference image to predict blocks in the current image. Using a predetermined evaluation function, the reference block most similar to the current block can be searched within a predetermined range. The current block is predicted based on the reference block, and the predicted block generated as a result of the prediction is subtracted from the current block to generate and encode a residual block.

[0005] To derive a motion vector pointing to a reference block within a reference image, the motion vectors of previously encoded blocks can be used as motion vector predictors of the current block. The differential motion vector, which is the difference between the motion vector of the current block and the motion vector predictor, is signaled to the decoder side through a predetermined method.

[0006] Recently, technologies for encoding / decoding images using AI (Artificial Intelligence) have been proposed, and a method for effectively encoding / decoding images using AI, such as neural networks, is required.

[0007] An image decoding method according to one embodiment of the present disclosure may include the steps of: obtaining, from a bitstream, second feature data for first feature data obtained through neural network-based encoding of a current image and a quantization index indicating one quantization step among a plurality of quantization steps; obtaining the quantization step based on the quantization index; obtaining probability data by applying the second feature data to a first neural network; modifying the probability data based on the quantization step; obtaining quantized first feature data by applying entropy decoding based on the modified probability data to bits included in the bitstream; obtaining dequantized first feature data by dequantizing the quantized first feature data according to the quantization step; and restoring the current image through neural network-based decoding of the dequantized first feature data.

[0008] An image decoding device according to one embodiment of the present disclosure may include an acquisition unit that acquires, from a bitstream, second feature data for first feature data acquired through neural network-based encoding for a current image and a quantization index indicating one quantization step among a plurality of quantization steps.

[0009] In one embodiment of the present disclosure, the acquisition unit may include an AI control unit that acquires a quantization step based on a quantization index, applies second feature data to a first neural network to acquire probability data, and modifies the probability data based on the quantization step.

[0010] In one embodiment of the present disclosure, the acquisition unit may include an entropy decoding unit that acquires quantized first feature data by applying entropy decoding based on modified probability data to bits included in a bitstream.

[0011] In one embodiment of the present disclosure, the acquisition unit may include a dequantization unit that dequantizes the quantized first feature data according to the quantization step to obtain the dequantized first feature data.

[0012] In one embodiment of the present disclosure, an image decoding device may include a predictive decoding unit that restores a current image through neural network-based decoding of inverse quantized first feature data.

[0013] An image encoding method according to one embodiment of the present disclosure may include: a step of applying first feature data obtained through neural network-based encoding of a current image to a first neural network to obtain second feature data for the first feature data; a step of applying the second feature data to the second neural network to obtain probability data; a step of modifying the probability data based on one quantization step among a plurality of predetermined quantization steps; a step of quantizing the first feature data according to the quantization step to obtain quantized first feature data; and a step of applying entropy decoding based on the modified probability data for the quantized first feature data and applying entropy encoding to a quantization index corresponding to the quantization step to generate a bitstream including the quantized first feature data and bits corresponding to the quantization index.

[0014] In one embodiment of the present disclosure, the bitstream may include bits corresponding to second feature data.

[0015] An image encoding device according to one embodiment of the present disclosure may include a predictive encoding unit that obtains first feature data through neural network-based encoding of a current image.

[0016] An image encoding device according to one embodiment of the present disclosure may include a generation unit that generates a bitstream.

[0017] In one embodiment of the present disclosure, the generation unit may include an AI control unit that applies first feature data to a first neural network to obtain second feature data for the first feature data, applies the second feature data to a second neural network to obtain probability data, and modifies the probability data based on one quantization step among a plurality of predetermined quantization steps.

[0018] In one embodiment of the present disclosure, the generation unit may include a quantization unit that quantizes the first feature data according to a quantization step to obtain quantized first feature data.

[0019] In one embodiment of the present disclosure, the generation unit may include an entropy encoding unit that applies entropy decoding based on modified probability data for the quantized first feature data and applies entropy encoding to a quantization index corresponding to a quantization step to generate a bitstream including the quantized first feature data and bits corresponding to the quantization index.

[0020] FIG. 1 is a diagram for explaining a process of encoding and decoding a current image based on intra prediction according to one embodiment of the present disclosure.

[0021] FIG. 2 is a diagram for explaining a process of encoding and decoding a current image based on inter prediction according to one embodiment of the present disclosure.

[0022] FIG. 3 is a diagram for explaining a method for obtaining probability data used for entropy encoding and entropy decoding according to one embodiment of the present disclosure.

[0023] FIG. 4 is a diagram for explaining a method of obtaining quantized data used for quantization and dequantization, and modified probability data used for entropy encoding and entropy decoding according to one embodiment of the present disclosure.

[0024] FIG. 5 is a diagram for explaining a method for obtaining modified probability data used for entropy encoding and entropy decoding and a quantization step determined by a quantization index used for quantization and dequantization according to one embodiment of the present disclosure.

[0025] FIG. 6 is a diagram for explaining a method for obtaining quantization data used for quantization and dequantization and a quantization step determined by a quantization index, and modified probability data used for entropy encoding and entropy decoding according to one embodiment of the present disclosure.

[0026] FIG. 7 is a diagram illustrating the structure of a neural network according to one embodiment of the present disclosure.

[0027] FIG. 8 is a diagram for explaining a method for modifying probability data according to one embodiment of the present disclosure.

[0028] FIG. 9 is a diagram illustrating probability data, modified probability data, and a probability model according to one embodiment of the present disclosure.

[0029] FIG. 10 is a diagram for explaining a method for modifying a plurality of probability data according to one embodiment of the present disclosure.

[0030] FIG. 11 is a diagram for explaining probability data, modified probability data, and a probability model according to one embodiment of the present disclosure.

[0031] FIG. 12 is a diagram illustrating a configuration of an image decoding device according to one embodiment of the present disclosure.

[0032] FIG. 13 is a diagram illustrating the configuration of an acquisition unit according to one embodiment of the present disclosure.

[0033] FIG. 14 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0034] FIG. 15 is a diagram illustrating the configuration of an acquisition unit according to one embodiment of the present disclosure.

[0035] FIG. 16 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0036] FIG. 17 is a diagram for explaining an image decoding method according to one embodiment of the present disclosure.

[0037] FIG. 18 is a diagram illustrating a configuration of an image encoding device according to one embodiment of the present disclosure.

[0038] FIG. 19 is a diagram illustrating the configuration of a generation unit according to one embodiment of the present disclosure.

[0039] FIG. 20 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0040] FIG. 21 is a diagram illustrating the configuration of a generation unit according to one embodiment of the present disclosure.

[0041] FIG. 22 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0042] FIG. 23 is a drawing for explaining an image encoding method according to one embodiment of the present disclosure.

[0043] FIG. 24 is a diagram for explaining a method for training neural networks according to one embodiment of the present disclosure.

[0044] FIG. 25 is a diagram for explaining a method for training neural networks according to one embodiment of the present disclosure.

[0045] The present disclosure may be subject to various modifications and various embodiments. Specific embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments of the present disclosure, and the present disclosure may include all modifications, equivalents, and alternatives that fall within the spirit and technical scope of the various embodiments.

[0046] When describing embodiments, detailed descriptions of related known technologies may be omitted if they are deemed to unnecessarily obscure the gist of the present disclosure. Furthermore, numbers (e.g., "first," "second," etc.) used in the description of the specification serve as identifiers to distinguish one component from another.

[0047] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0048] In the present disclosure, when a component is referred to as being “connected” or “connected” to another component, the component may be directly connected or connected to the other component, but unless there is a specific description to the contrary, the component may also be connected or connected via another component in between.

[0049] In this disclosure, components expressed as "units", "modules", etc. may be two or more components combined into one component, or one component may be divided into two or more components with more detailed functions. Each component described below may additionally perform some or all of the functions performed by other components in addition to its own main function, and some of the main functions performed by each component may be exclusively performed by other components.

[0050] A processor may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. At least one processor, one or more processors may be configured to perform various functions described herein, individually and / or collectively, in a distributed fashion. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0051] In the present disclosure, 'image' may refer to a still image, a picture, a frame, a moving image composed of a plurality of consecutive still images, or a video.

[0052] In this disclosure, "neural network" refers to a representative example of an artificial neural network model that mimics brain neurons, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

[0053] In this disclosure, a "parameter" refers to a value used in the computational process of each layer forming a neural network. For example, it can be used when applying an input value to a given computational formula. A parameter is a value set as a result of training and can be updated using separate training data as needed.

[0054] In this disclosure, "feature data" may refer to data obtained by a neural network or neural network-based encoder processing input data. The feature data may be one-dimensional or two-dimensional data containing multiple samples. The feature data may also be referred to as a latent representation. The feature data may represent latent features in the data output by a neural network-based decoder.

[0055] In this disclosure, the term "current image" refers to the image currently being processed, and the term "previous image" refers to the image being processed prior to the current image. The "current image" or "previous image" may be a block segmented from the current image or the previous image.

[0056] In this disclosure, a "sample" refers to data assigned to a sampling location within one-dimensional or two-dimensional data, such as an image, feature data, probability data, or quantized data, and is the data to be processed. For example, a sample may include a pixel within a two-dimensional image. Two-dimensional data may also be referred to as a "map."

[0057] An AI-based end-to-end encoding / decoding system can be understood as a system that utilizes a neural network in the encoding and decoding processes of an image.

[0058] In AI-based end-to-end encoding / decoding systems, such as codecs such as HEVC and VVC, intra prediction or inter prediction can be used to encode and decode images.

[0059] As mentioned above, intra prediction is a method of compressing an image by removing spatial redundancy within the image, and inter prediction is a method of compressing an image by removing temporal redundancy between images.

[0060] In one embodiment of the present disclosure, intra prediction may be applied to a first frame among several frames, a frame that is a random access point, and a frame in which a scene change occurs.

[0061] In one embodiment of the present disclosure, inter prediction may be applied to frames subsequent to a frame to which intra prediction is applied among multiple frames.

[0062] Referring to FIGS. 1 and 2, intra prediction and inter prediction performed by an AI-based end-to-end encoding / decoding system according to one embodiment of the present disclosure are described.

[0063] FIG. 1 is a diagram for explaining a process of encoding and decoding a current image based on intra prediction according to one embodiment of the present disclosure.

[0064] In intra prediction, a video encoder (12) and a video decoder (14) can be used. The video encoder (12) and the video decoder (14) can be implemented as a neural network.

[0065] The image encoder (12) can process the current image (100) according to parameters set through training and output feature data (k) for the current image (100).

[0066] A bitstream is generated by applying quantization (22) and entropy encoding (32) to the feature data (k) of the current image (100), and the bitstream can be transmitted from an image encoding device to an image decoding device.

[0067] Entropy decoding (34) and inverse quantization (24) are applied to the bitstream to obtain restored feature data (k'), and the restored feature data (k') can be input to the image decoder (14).

[0068] The image decoder (14) can process feature data (k') according to parameters set through training and output the current restored image (300).

[0069] In intra prediction, spatial features within the current image (100) are considered, so unlike inter prediction illustrated in FIG. 2, only the current image (100) can be input to the image encoder (12).

[0070] FIG. 2 is a diagram for explaining a process of encoding and decoding a current image based on inter prediction according to one embodiment of the present disclosure.

[0071] In inter prediction, an optical flow encoder (42), an optical flow decoder (44), a residual encoder (52), and a residual decoder (54) can be used.

[0072] The optical flow encoder (42), optical flow decoder (44), residual encoder (52), and residual decoder (54) can be implemented as a neural network.

[0073] The optical flow encoder (42) and the optical flow decoder (44) can be understood as a neural network for extracting optical flow (g) from the current image (100) and the previously restored image (200).

[0074] The residual encoder (52) and the residual decoder (54) can be understood as neural networks for encoding and decoding the residual image (r).

[0075] As described above, inter prediction is a process of encoding and decoding a current image (100) by utilizing the temporal redundancy between the current image (100) and a previously restored image (200). The previously restored image (200) may be an image obtained through decoding a previous image that was the subject of processing prior to processing the current image (100).

[0076] The positional differences (or motion vectors) between blocks or samples in the current image (100) and reference blocks or reference samples in the previously restored image (200) can be used for encoding and decoding the current image (100). These positional differences can be referred to as optical flow. Optical flow can also be defined as a set of motion vectors corresponding to samples or blocks in the image.

[0077] Optical flow (g) can indicate how the positions of samples in the previous restored image (200) have changed in the current image (100), or where samples identical / similar to the samples in the current image (100) are located in the previous restored image (200).

[0078] For example, if a sample that is identical to or most similar to a sample located at (1, 1) in the current image (100) is located at (2, 1) in the previous restored image (200), the optical flow (g) or motion vector for the sample can be derived as (1(=2-1), 0(=1-1)).

[0079] For encoding the current image (100), the previous restored image (200) and the current image (100) can be input to the optical flow encoder (42).

[0080] The optical flow encoder (42) can process the current image (100) and the previously restored image (200) according to the parameters set as a result of training to output feature data (w) of the optical flow (g).

[0081] As described in Fig. 1, quantization (22) and entropy encoding (32) are applied to the feature data (w) of the optical flow (g) to generate a bitstream, and entropy decoding (34) and inverse quantization (24) are applied to the bitstream to restore the feature data (w) of the optical flow (g).

[0082] The feature data (w) of the optical flow (g) can be input to the optical flow decoder (44). The optical flow decoder (44) can process the input feature data (w) according to the parameters set as a result of training and output the optical flow (g).

[0083] The previous restored image (200) is warped through warping (60) based on optical flow (g), and a current predicted image (x') can be obtained as a result of the warping (60). Warping (60) is a type of geometric deformation that moves the positions of samples within an image.

[0084] A current predicted image (x') similar to the current image (100) can be obtained by applying warping (60) to the previous restored image (200) according to the optical flow (g) indicating the relative positional relationship between samples in the previous restored image (200) and samples in the current image (100).

[0085] For example, if the sample located at (1, 1) in the previous restored image (200) is most similar to the sample located at (2, 1) in the current image (100), the location of the sample located at (1, 1) in the previous restored image (200) can be changed to (2, 1) through warping (60).

[0086] Since the current predicted image (x') generated from the previous restored image (200) is not the current image (100) itself, a residual image (r) between the current predicted image (x') and the current image (100) can be obtained.

[0087] For example, a residual image (r) can be obtained by subtracting sample values ​​in the current predicted image (x') from sample values ​​in the current image (100).

[0088] The residual image (r) can be input to a residual encoder (52). The residual encoder (52) can process the residual image (r) according to parameters set as a result of training and output feature data (v) of the residual image (r).

[0089] As described in Fig. 1, quantization (22) and entropy encoding (32) are applied to the feature data (v) of the residual image (r) to generate a bitstream, and entropy decoding (34) and inverse quantization (24) are applied to the bitstream to restore the feature data (v) of the residual image (r).

[0090] The feature data (v) of the residual image (r) can be input to the residual decoder (54). The residual decoder (54) can process the input feature data (v) according to the parameters set as a result of training and output the restored residual image (r').

[0091] The current restored image (300) can be obtained by combining the current predicted image (x') and the restored residual image (r').

[0092] Meanwhile, as described above, entropy encoding (32) and entropy decoding (34) may be applied to the feature data (k) of the current image (100), the feature data (w) of the optical flow (g), and the feature data (v) of the residual image (r). Since entropy coding is a encoding method that varies the average length of a code representing a symbol according to the probability of the symbol, the probabilities of values ​​that the samples of the first feature data may have may be required in entropy encoding (32) and entropy decoding (34) the first feature data.

[0093] In one embodiment of the present disclosure, probability data may be acquired based on a neural network to improve the efficiency of entropy encoding (32) / entropy decoding (34) for at least one of feature data (k) of a current image (100), feature data (w) of an optical flow (g), or feature data (v) of a residual image (r) (hereinafter, first feature data).

[0094] Probability data is one-dimensional or two-dimensional data, and a sample of the probability data can represent the probability of a value that a sample of the first feature data can have.

[0095] In one embodiment of the present disclosure, by applying sample values ​​of probability data to a predetermined probability model (e.g., a Laplacian probability model or a Gaussian probability model, etc.), probabilities of values ​​that samples of first feature data may have can be derived.

[0096] In one embodiment of the present disclosure, the probability data may include, as sample values, a mean and a standard deviation (or variance) corresponding to a sample of the first feature data.

[0097] A method for obtaining probability data using a neural network is described with reference to Fig. 3.

[0098] FIG. 3 is a diagram for explaining a method for obtaining probability data used for entropy encoding and entropy decoding according to one embodiment of the present disclosure.

[0099] A hyperprior encoder (310) and a probabilistic neural network (330) can be used to obtain probability data used for entropy encoding (32) / entropy decoding (34).

[0100] The hyperflash encoder (310) may be a neural network for obtaining feature data from feature data.

[0101] Referring to FIG. 3, first feature data is input to a hyperfryer encoder (310), and the hyperfryer encoder (310) can process the first feature data according to parameters set as a result of training and output second feature data.

[0102] The second feature data can represent latent features in the first feature data, and thus can be referred to as hyper-priority feature data.

[0103] The second feature data is input to a probability neural network (330), and the probability neural network (330) can process the second feature data according to parameters set as a result of training and output probability data.

[0104] Probability data can be used for entropy encoding (32) and entropy decoding (34) described with reference to FIGS. 1 and 2.

[0105] In one embodiment of the present disclosure, in the encoding process of the current image (100), a bitstream can be obtained by applying entropy encoding (32) based on probability data to at least one of quantized first feature data, for example, quantized feature data of the current image (100), quantized feature data of an optical flow (g), or quantized feature data of a residual image (r).

[0106] In one embodiment of the present disclosure, in the decoding process of the current image (100), entropy decoding (34) based on probability data is applied to bits included in the bitstream, thereby obtaining quantized first feature data, for example, at least one of quantized feature data of the current image (100), quantized feature data of an optical flow (g), or quantized feature data of a residual image (r).

[0107] The process of acquiring probability data illustrated in FIG. 3 can be useful when quantization (22) and dequantization (24) of the first feature data are performed uniformly. This is because probability data acquired from the second feature data representing the latent features of the first feature data can be applied as is to the quantized first feature data.

[0108] The fact that quantization (22) and inverse quantization (24) are performed uniformly may mean that the quantization step sizes used for quantization (22) and inverse quantization (24) of the samples of the first feature data are all the same. For example, when the quantization step size is 2, the first feature data can be quantized by dividing all sample values ​​of the first feature data by 2 and rounding the resulting value. In addition, the quantized first feature data can be inverse quantized by multiplying all sample values ​​of the quantized first feature data by 2.

[0109] When quantization (22) is performed based on a single quantization step size, the distribution of sample values ​​of the first feature data before quantization (22) can be similarly maintained in the first feature data after quantization (22). This is because the sample values ​​of the first feature data are all divided and rounded based on the same value. That is, since the distribution of sample values ​​of the first feature data before quantization (22) is maintained in the first feature data after quantization (22), the probability data obtained from the second feature data can be applied as is to the quantized first feature data.

[0110] Uniform quantization and uniform dequantization can be useful when the sample values ​​of the first feature data follow a Laplacian distribution. However, since the distribution of the sample values ​​of the first feature data can vary depending on the characteristics of the image, uniform quantization and uniform dequantization may have certain limitations.

[0111] In one embodiment of the present disclosure, the efficiency of quantization (22) and inverse quantization (24) can be improved by obtaining data related to quantization (22), for example, a quantization step size, from hyperpriority feature data of the first feature data to be subjected to quantization (22) based on a neural network.

[0112] FIG. 4 is a diagram for explaining a method of obtaining quantized data used for quantization and dequantization, and modified probability data used for entropy encoding and entropy decoding according to one embodiment of the present disclosure.

[0113] Referring to FIG. 4, first feature data is input to a hyperfryer encoder (310), and the hyperfryer encoder (310) can process the first feature data according to parameters set as a result of training and output second feature data.

[0114] The second feature data can be input to a probabilistic neural network (330) and a quantization neural network (410). The probabilistic neural network (330) can process the second feature data according to parameters set as a result of training and output probability data. The quantization neural network (410) can process the second feature data according to parameters set as a result of training and output quantized data.

[0115] Quantized data may include quantization step sizes or quantization parameters as sample values.

[0116] The quantization step size is a value used for quantizing a sample (22). A sample value can be quantized by dividing the sample value by the quantization step size and rounding the result of the division. Conversely, a quantized sample value can be dequantized by multiplying the quantized sample value by the quantization step size.

[0117] The quantization step size can be approximated by the following mathematical expression 1.

[0118] [Mathematical Formula 1]

[0119] Quantization step size = 2^(quantization parameter / n) / quantization scale[quantization parameter%n]

[0120] In mathematical expression 1, the quantization scale [quantization parameter % n] represents the scale value indicated by the quantization parameter among n predetermined scale values. Since the HEVC codec defines six scale values ​​(26214, 23302, 20560, 18396, 16384, and 14564), n is 6 according to the HEVC codec.

[0121] In one embodiment of the present disclosure, when quantization data includes quantization parameters as samples, a quantization step size can be obtained from the quantization parameters for quantization (22) and dequantization (24) of the first feature data. For example, the aforementioned mathematical expression 1 can be used to derive the quantization step size.

[0122] When a quantization step size is obtained from quantized data, samples of the first feature data can be quantized according to the quantization step size, and samples of the quantized first feature data can be dequantized according to the quantization step size.

[0123] According to one embodiment of the present disclosure, since the quantization step size for samples of the first feature data is adaptively acquired for each sample from the trained quantization neural network (410), a high-quality current restored image (300) can be acquired at a lower bit rate compared to uniform quantization and uniform inverse quantization.

[0124] As the first feature data is quantized according to the quantization data obtained through the process illustrated in Fig. 4, the distribution of the sample values ​​of the first feature data before quantization (22) and the distribution of the sample values ​​of the first feature data after quantization (22) may differ. Accordingly, it may be necessary to modify the probability data suitable for the uniform quantization and uniform dequantization described above.

[0125] In one embodiment of the present disclosure, a correction process (430) based on quantized data is applied to probability data output from a probabilistic neural network (330), thereby obtaining corrected probability data. The corrected probability data can be used for entropy encoding (32) for quantized first feature data and entropy decoding (34) for a bitstream.

[0126] The method of modifying probability data is described below with reference to FIGS. 8 to 11.

[0127] In the embodiment described with reference to FIG. 4, quantization data and probability data are obtained from second feature data corresponding to potential features of first feature data that are the target of quantization (22) and entropy encoding (32), and the probability data is modified according to the quantization data, so that the efficiency of sequentially performed quantization (22) and entropy encoding (32) can be improved.

[0128] Meanwhile, during the training process of the quantization neural network (410), random uniform noise is added to imitate the quantization process as shown in mathematical expression 2 below.

[0129] [Equation 2]

[0130] Q(x) = x + n, n~U[-0.5 ~ 0.5]

[0131] Here, x is the training data, n is random uniform noise, and Q(x) is the quantized training data.

[0132] All channels of training data acquired during training may have large quantization errors.

[0133] However, when testing a quantized neural network, quantization is performed by rounding as in Equation 3 below.

[0134] [Equation 3]

[0135] Q(x) = [x + 0.5]

[0136] Here, x is the test data, n is random uniform noise, Q(x) is the quantized test data, and [] represents the floor function. The floor function is a function that, for an input real number x, produces a result value of n when n is the largest integer among integers less than or equal to x.

[0137] For test data, some channels of the test data may have very small quantization errors, while the remaining channels of the test data may have large quantization errors.

[0138] That is, for feature data with large variance, if there is a large quantization error during training, there may also be a large quantization error during testing.

[0139] However, for feature data with small variance, it may have a large quantization error during training and a small quantization error during testing.

[0140] In image coding, that is, when coding still images, in order to minimize bitrate, most channels of feature data have small variance values, resulting in small quantization errors, and there are few channels with large variance values.

[0141] Even in video coding, or video encoding, to minimize bitrate, most feature data channels have small variances, resulting in small quantization errors. Channels with large variances are rare. In particular, when motion compensation is effective, there are no channels with large variances.

[0142] As above, for feature data with channels with small variance values, a mismatch may occur between training and testing of the quantized neural network.

[0143] Since neural network-based decoders are trained or optimized for large quantization errors, coarser quantization, i.e., less precise quantization, may be required during testing to address this mismatch. That is, quantization using a quantization step size of 1 or greater may be required.

[0144] In the following, FIG. 5 describes a method of selecting a quantization step from among a plurality of predetermined quantization steps through rate-distortion optimization rather than quantization data based on a neural network, and obtaining modified probability data based on the selected quantization step.

[0145] FIG. 5 is a diagram for explaining a method for obtaining modified probability data used for entropy encoding and entropy decoding and a quantization step determined by a quantization index used for quantization and dequantization according to one embodiment of the present disclosure.

[0146] Referring to FIG. 5, first feature data is input to a hyperfryer encoder (310), and the hyperfryer encoder (310) can process the first feature data according to parameters set as a result of training and output second feature data.

[0147] The second feature data can be input to a probabilistic neural network (410). The probabilistic neural network (410) can process the second feature data according to parameters set as a result of training and output probability data.

[0148] A quantization index (510) representing one of a plurality of quantization steps included in a predefined quantization list can be obtained.

[0149] The quantization step may be determined by Rate-Distortion Optimization (RDO) during the encoding process and signaled as a quantization index, and may be determined based on the quantization index transmitted during the decoding process.

[0150] Specifically, in the encoding process, an optimal quantization step is determined from among a plurality of predetermined quantization steps through RDO calculation, and a quantization index (510) indicating the optimal quantization step is included in the bitstream and signaled. For example, among the predetermined quantization step values ​​of q1, q2, q3, ..., qN, the most optimal quantization step value q3 according to RDO calculation can be determined as the value indicated by the quantization index. The probability distribution of the feature data is modified according to the optimal quantization step, and entropy encoding is performed according to the modified probability distribution.

[0151] In the decoding process, inverse quantization is performed according to the quantization step indicated by the quantization index (510) included in the bitstream, and entropy decoding is performed according to a probability distribution modified based on the quantization step.

[0152] The quantization step size is a value used for quantizing a sample (22). The sample value can be quantized by dividing the sample value by the quantization step size and rounding the result of the division. Conversely, the quantized sample value can be dequantized by multiplying the quantized sample value by the quantization step size. In addition, the quantization step size can be 1 or greater.

[0153] The quantization step size can be approximated by the mathematical expression 1 described above.

[0154] In one embodiment of the present disclosure, when the quantization index (510) represents one quantization parameter among a plurality of quantization parameters, a quantization step size can be obtained from the quantization parameter for quantization (22) and inverse quantization (24) of the first feature data. For example, the aforementioned mathematical expression 1 can be used to derive the quantization step size.

[0155] According to the quantization index, when the quantization step size is obtained, samples of the first feature data can be quantized according to the quantization step size, and samples of the quantized first feature data can be dequantized according to the quantization step size.

[0156] As the first feature data is quantized according to the quantization step size obtained through the process illustrated in Fig. 5, the distribution of the sample values ​​of the first feature data before quantization (22) and the distribution of the sample values ​​of the first feature data after quantization (22) may differ. Therefore, it may be necessary to modify the probability data suitable for the uniform quantization and uniform dequantization described above.

[0157] In one embodiment of the present disclosure, modified probability data can be obtained by applying a quantization step-based modification process (430) to probability data output from a probabilistic neural network (330). The modified probability data can be used for entropy encoding (32) for quantized first feature data and entropy decoding (34) for a bitstream.

[0158] In the embodiment described with reference to FIG. 5, probability data is obtained from second feature data corresponding to a potential feature of first feature data that is the target of quantization (22) and entropy encoding (32), a quantization step is obtained according to a quantization index (510), and the probability data is modified according to the quantization step, so that the efficiency of sequentially performed quantization (22) and entropy encoding (32) can be improved.

[0159] In addition, compared to Fig. 4, by performing quantization and entropy encoding by modifying the probability data based on the quantization step determined through RDO calculation instead of the quantization neural network, the incompatibility between training and testing can be flexibly resolved by applying each optimal quantization step to each feature data.

[0160] Moreover, since many channels of feature data have small variances and the probability model is not always accurate, entropy coding for these small variance channels may not be accurate. Since skipping these small variance channels may not affect the image quality of the restored image, this skipping process can reduce bitrate. Therefore, while quantizing and entropy-encoding channels with small variances based on a quantization step size greater than 1, some of the small variance channels can be skipped. In other words, better coding efficiency can be achieved by quantizing and entropy-encoding channels with small variances based on a quantization step size together with an appropriate skipping process.

[0161] In the following, FIG. 6 describes a method of obtaining final quantization step values ​​using quantization data based on a neural network and quantization steps based on quantization indices, and obtaining modified probability data based on the final quantization step values.

[0162] FIG. 6 is a diagram for explaining a method for obtaining quantization data used for quantization and dequantization and a quantization step determined by a quantization index, and modified probability data used for entropy encoding and entropy decoding according to one embodiment of the present disclosure.

[0163] Referring to FIG. 6, first feature data is input to a hyperfryer encoder (310), and the hyperfryer encoder (310) can process the first feature data according to parameters set as a result of training and output second feature data.

[0164] The second feature data can be input to a probabilistic neural network (330) and a quantization neural network (410). The probabilistic neural network (330) can process the second feature data according to parameters set as a result of training and output probability data. The quantization neural network (410) can process the second feature data according to parameters set as a result of training and output quantized data.

[0165] Quantized data may include quantization step sizes or quantization parameters as sample values.

[0166] Additionally, a quantization index (510) indicating one quantization step among multiple quantization steps included in a predefined quantization list can be obtained.

[0167] The quantization step may be determined by Rate-Distortion Optimization (RDO) during the encoding process and signaled as a quantization index (510), and may be determined based on the quantization index (510) transmitted during the decoding process.

[0168] Final quantization step values ​​can be obtained based on quantization data obtained from a quantization neural network (410) and a quantization step determined by a quantization index (510).

[0169] Specifically, if the sample values ​​included in the quantization data represent quantization steps, the final quantization step values ​​can be obtained by multiplying (610) the quantization step values ​​included in the quantization data by the quantization step determined by the quantization index (510).

[0170] The final quantization step value is a value used for quantizing a sample (22). The sample value can be quantized by dividing the sample value by the final quantization step size and rounding the result of the division. Conversely, the quantized sample value can be dequantized by multiplying the quantized sample value by the quantization step size.

[0171] The maximum quantization step size can be approximated by the mathematical expression 1 described above.

[0172] In one embodiment of the present disclosure, when sample values ​​included in quantized data represent quantization parameters or a quantization index represents a quantization parameter, a quantization step size can be obtained from the quantization parameter for quantization (22) and inverse quantization (24) of the first feature data. For example, the aforementioned mathematical expression 1 can be used to derive the quantization step size.

[0173] According to one embodiment of the present disclosure, when quantized data is obtained from a trained quantized neural network (410), a quantization step size is obtained according to a quantization index (510), and a final quantization step size is obtained based on the quantized data and the quantized step size, samples of the first feature data may be quantized according to the final quantization step size, and samples of the quantized first feature data may be dequantized according to the final quantization step size.

[0174] As the first feature data is quantized according to the final quantization step values ​​obtained through the process illustrated in Fig. 6, the distribution of the sample values ​​of the first feature data before quantization (22) and the distribution of the sample values ​​of the first feature data after quantization (22) may differ. Therefore, it may be necessary to modify the probability data suitable for the uniform quantization and uniform dequantization described above.

[0175] In one embodiment of the present disclosure, modified probability data can be obtained by applying a modification process (430) based on the final quantization step to probability data output from a probabilistic neural network (330). The modified probability data can be used for entropy encoding (32) for quantized first feature data and entropy decoding (34) for a bitstream.

[0176] In the embodiment described with reference to FIG. 6, probability data is obtained from second feature data corresponding to potential features of first feature data that are the target of quantization (22) and entropy encoding (32), final quantization step values ​​are obtained based on quantization data obtained from a quantization neural network and a quantization step determined by a quantization index, and the probability data is modified according to the final quantization step values, so that the efficiency of sequentially performed quantization (22) and entropy encoding (32) can be improved.

[0177] In addition, compared to FIGS. 4 and 5, the final quantization step values ​​are obtained based on the quantization data obtained from the quantization neural network (410) and the quantization step determined through RDO calculation, and the probability data is modified based on the final quantization step values ​​to perform quantization and entropy encoding, thereby flexibly resolving the incompatibility between training and testing, and adaptively quantizing each sample of feature data can be performed, thereby improving the quality of the restored image.

[0178] An exemplary structure of the hyperflash encoder (310), the probabilistic neural network (330), and the quantization neural network (410) described above in FIGS. 4 to 6 will be described with reference to FIG. 7.

[0179] In one embodiment of the present disclosure, the modification (430) process may also be performed based on a neural network. An exemplary structure of a neural network for the modification (430) process is also described with reference to FIG. 7.

[0180] FIG. 7 is a diagram illustrating the structure of a neural network according to one embodiment of the present disclosure.

[0181] As illustrated in Fig. 7, input data (705) can be input to the first convolution layer (710). 3X3X4 indicated in the first convolution layer (710) exemplifies convolution processing on the input data (705) using four filter kernels of size 3x3. As a result of the convolution processing, four feature data can be generated by the four filter kernels.

[0182] In one embodiment of the present disclosure, the process of modifying probability data may be performed based on a neural network. If the neural network (700) corresponds to a neural network for the modification process, the input data (705) may include two channels of data, namely probability data and quantization data. In addition, the input data (705) may include the probability data and a quantization step indicated by the quantization index. In addition, the input data (705) may include the quantization step indicated by the quantization index and final quantization step values ​​and probability data obtained based on the quantization data.

[0183] In one embodiment of the present disclosure, if the neural network (700) corresponds to a probabilistic neural network (330) or a quantized neural network (410), the input data (705) may include second feature data.

[0184] In one embodiment of the present disclosure, if the neural network (700) corresponds to a hyperfriction encoder (310), the input data (805) may include first feature data.

[0185] The feature data generated by the first convolution layer (710) may represent unique characteristics of the input data (705). For example, each feature data may represent vertical characteristics, horizontal characteristics, or edge characteristics of the input data (705).

[0186] Feature data of the first convolution layer (710) can be input to the first activation layer (720).

[0187] The first activation layer (720) can provide non-linear characteristics to each feature data. The first activation layer (820) can include, but is not limited to, a sigmoid function, a Tanh function, a ReLU (Rectified Linear Unit) function, etc.

[0188] Imparting nonlinear characteristics in the first activation layer (720) may mean changing some sample values ​​of the feature data and outputting them. In this case, the change may be performed by applying nonlinear characteristics.

[0189] The first activation layer (720) can determine whether to pass the sample values ​​of the feature data to the second convolution layer (730). For example, some sample values ​​of the feature data may be activated by the first activation layer (720) and passed to the second convolution layer (730), and some sample values ​​may be deactivated by the first activation layer (720) and not passed to the second convolution layer (730). The unique characteristics of the input data (705) represented by the feature data may be emphasized by the first activation layer (720).

[0190] Feature data output from the first activation layer (720) can be input to the second convolution layer (730). 3X3X4 displayed in the second convolution layer (730) exemplifies convolution processing on the input feature data using four filter kernels of size 3x3.

[0191] The output of the second convolution layer (730) can be input to the second activation layer (740). The second activation layer (740) can impart nonlinear characteristics to the input feature data.

[0192] Feature data output from the second activation layer (740) can be input to the third convolution layer (750). 3X3X1 displayed in the third convolution layer (750) exemplifies convolution processing to create one output data (755) using one filter kernel of size 3x3.

[0193] The output data (755) varies depending on whether the neural network (700) is a hyperparameter encoder (310), a probabilistic neural network (330), a quantization neural network (410), or a neural network for the correction process.

[0194] For example, if the neural network (700) is a probabilistic neural network (330), the output data (755) may be probabilistic data, and if the neural network (700) is a quantized neural network (410), the output data (755) may be quantized data.

[0195] In one embodiment of the present disclosure, the number of output data (755) can be adjusted by adjusting the number of filter kernels used in the third convolution layer (750).

[0196] For example, if the neural network (700) is a probabilistic neural network (330) and the probabilistic data includes the mean data and standard deviation data described below, two filter kernels may be used so that two channels of data can be output from the third convolution layer (750).

[0197] Additionally, for example, if the neural network (700) is a probabilistic neural network (330), the probability data includes the average data and standard deviation data described below, and the number of channels of the first feature data is M, 2M filter kernels can be used in the third convolution layer (750) so that M average data and M standard deviation data can be output.

[0198] Additionally, for example, if the neural network (700) is a quantization neural network (410) and the number of channels of the first feature data is M, M filter kernels can be used in the third convolution layer (750) so that M quantization data can be output.

[0199] Although FIG. 7 illustrates that the neural network (700) includes three convolution layers (a first convolution layer (710), a second convolution layer (730), and a third convolution layer (750)) and two activation layers (a first activation layer (720) and a second activation layer (740)), this is only an example, and in one embodiment of the present disclosure, the number of convolution layers and activation layers included in the neural network (700) may vary.

[0200] In one embodiment of the present disclosure, the size and number of filter kernels used in convolutional layers included in the neural network (700) may also be varied.

[0201] In one embodiment of the present disclosure, the neural network (700) may be implemented via a recurrent neural network (RNN). This means changing the CNN structure of the neural network (700) to an RNN structure.

[0202] In one embodiment of the present disclosure, the image decoding device (1200) and the image encoding device (1800) may include at least one ALU (Arithmetic logic unit) for convolution operation and activation layer operation.

[0203] The ALU may be implemented as a processor. For convolution operations, the ALU may include a multiplier that performs a multiplication operation between sample values ​​of input data or feature data output from a previous layer and sample values ​​of the filter kernel, and an adder that adds the resulting values ​​of the multiplication.

[0204] For the operation of the activation layer, the ALU may include a multiplier that multiplies the input sample value by a weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and a comparator that compares the multiplication result with a predetermined value to determine whether to pass the input sample value to the next layer.

[0205] Hereinafter, with reference to FIGS. 8 to 11, the probability data used for entropy encoding and entropy decoding will be described.

[0206] FIG. 8 is a diagram for explaining a method for modifying probability data according to one embodiment of the present disclosure.

[0207] In one embodiment of the present disclosure, the probability data output by the probabilistic neural network (330) may represent probabilities of values ​​that samples of the first feature data may have.

[0208] In one embodiment of the present disclosure, the probability data may include samples of means and standard deviations corresponding to samples of the first feature data. In this case, the probability data may include mean data (810) including samples of means corresponding to samples of the first feature data, and standard deviation data (830) including samples of standard deviations corresponding to samples of the first feature data.

[0209] In one embodiment of the present disclosure, the probabilistic neural network (330) may include mean data including as samples means corresponding to samples of the first feature data and variance data including as samples variances corresponding to samples of the first feature data.

[0210] In one embodiment of the present disclosure, modified probability data can be obtained by dividing sample values ​​of probability data by sample values ​​of quantized data.

[0211] Referring to FIG. 8, the average data (810) may include sample values ​​of μ(0,0) to μ(1,1), the standard deviation data (830) may include sample values ​​of σ(0,0) to σ(1,1), and the quantized data (850) may include sample values ​​of q(0,0) to q(1,1). The q(0,0) to q(1,1) of the quantized data (850) may be quantization step sizes. In FIG. 9, (a, b) may represent the positions of samples within the data.

[0212] In addition, when a quantization step is determined based on a quantization index, the sample values ​​of q(0,0) to q(1,1) of the quantization data (850) may be one quantization step indicated by the quantization index. That is, the sample values ​​of q(0,0) to q(1,1) may be the same quantization step value.

[0213] In addition, when final quantization step values ​​based on quantization data and quantization index are used, the sample values ​​of q(0,0) to q(1,1) of the quantization data (850) may be values ​​obtained by multiplying the quantization step sizes output from the quantization neural network (410) by the quantization step indicated by the quantization index.

[0214] By dividing μ(0,0) to μ(1,1) in the average data (810) by q(0,0) to q(1,1) in the quantized data (850), modified average data (870) including μ(0,0) / q(0,0) to μ(1,1) / q(1,1) as samples can be obtained.

[0215] Additionally, by dividing σ(0,0) to σ(1,1) in the standard deviation data (830) by q(0,0) to q(1,1) in the quantized data (850), modified standard deviation data (890) including σ(0,0) / q(0,0) to σ(1,1) / q(1,1) as samples can be obtained.

[0216] Here, division is an example, and in one embodiment of the present disclosure, the sample values ​​of the average data (810) and the standard deviation data (830) may be multiplied by the sample values ​​of the quantized data (850) or values ​​derived from the sample values ​​of the quantized data (850) to obtain the modified average data (870) and the modified standard deviation data (890). The quantized data (850) may be determined by a quantization index, in which case all of the sample values ​​may be one quantization step value. In addition, the quantized data (850) may be final quantization step values ​​determined based on the quantized data output from the quantization neural network (410) and the quantization step indicated by the quantization index.

[0217] In one embodiment of the present disclosure, bit shift operations may be used for division or multiplication operations on sample values ​​of average data (810) and standard deviation data (830).

[0218] By applying the sample values ​​of the modified average data (870) and the sample values ​​of the modified standard deviation data (890) to a predetermined probability model, probability values ​​that the samples of the quantized first feature data can have according to the quantized data (850) can be derived.

[0219] The reason for dividing the sample values ​​of the average data (810) and the sample values ​​of the standard deviation data (830) into the sample values ​​of the quantized data (850) is that when the sample values ​​of the first feature data are divided based on the quantized data (850) and the divided result values ​​are rounded, the sample values ​​of the first feature data become larger or smaller depending on the size of the quantization step size, and accordingly, the probability model (e.g., probability density function) for the first feature data needs to be changed. Therefore, by downscaling the average data (810) and the standard deviation data (830) according to the quantized data (850), a probability model suitable for the quantized first feature data can be derived.

[0220] FIG. 9 is a diagram illustrating probability data, modified probability data, and a probability model according to one embodiment of the present disclosure.

[0221] Referring to Figure 9, the probability data is averaged (μ a ) and standard deviation (σ a ) can be included, and the mean (μ) of the probability data a ) and standard deviation (σ a ) is divided into the quantization step size (q) of the quantized data, the quantization step size (q) indicated by the quantization index, and the final quantization step size (q) based on the quantized data and the quantization index, and the modified probability data is the modified mean (μ b ) and adjusted standard deviation (σ b ) may be included.

[0222] Corrected mean (μ b ) and adjusted standard deviation (σ b ) can be applied to a predetermined probability model to determine the probability of a value that a sample of the quantized first feature data can have.

[0223] Referring to Fig. 9, a Laplacian probability model or a Gaussian probability model can be used as a predetermined probability model.

[0224] The Laplacian probability model or Gaussian probability model illustrated in FIG. 9 is an example. In one embodiment of the present disclosure, the types of probability models used for entropy encoding and entropy decoding may vary.

[0225] The type of probability model to be used for entropy encoding and decoding of the first feature data may be predetermined. For example, the type of probability model used for entropy encoding may be predetermined by the image encoding device (1800).

[0226] In one embodiment of the present disclosure, the type of probability model used for entropy encoding can be independently determined for each image or each block included in the image.

[0227] When the Laplacian model is used for entropy encoding, the modified mean (μ b ) and adjusted standard deviation (σ b ) can be applied to the Laplacian probability model to derive the probability that a sample of the quantized first feature data can have.

[0228] Additionally, when a Gaussian model is used for entropy encoding, the modified mean (μ b ) and adjusted standard deviation (σ b ) can be applied to the Gaussian probability model to derive the probability that a sample of the quantized first feature data can have.

[0229] In one embodiment of the present disclosure, the probabilistic neural network (330) may output a plurality of probability data and a plurality of weight data as a result of processing the second feature data.

[0230] In one embodiment of the present disclosure, each of the plurality of probability data may include mean data and standard deviation data. In one embodiment of the present disclosure, each of the plurality of probability data may also include mean data and variance data.

[0231] A plurality of modified probability data can be obtained by modifying a plurality of probability data according to quantization data obtained from a quantization neural network (410), a quantization step indicated by a quantization index, or final quantization step values ​​determined based on the quantization data obtained from the quantization neural network (410) and the quantization step indicated by the quantization index.

[0232] As multiple modified probability data are combined according to multiple weight data, the probability that a sample of the quantized first feature data can have can be derived.

[0233] FIG. 10 is a diagram for explaining a method for modifying a plurality of probability data according to one embodiment of the present disclosure.

[0234] Referring to FIG. 10, N average data (1010-1, 1010-2, ..., 1010-N) can be obtained from a probabilistic neural network (330).

[0235] Although not shown in Fig. 10, N standard deviation data and N weight data can also be obtained from the probabilistic neural network (330).

[0236] The size or number of samples of N mean data (1010-1, 1010-2, ..., 1010-N), N standard deviation data, and N weight data may be the same as the size or number of samples of the first feature data.

[0237] In one embodiment of the present disclosure, if the number of first feature data (or the number of channels) is M, M*N average data, M*N standard deviation data, and M*N weight data can be obtained from the probabilistic neural network (330).

[0238] N modified average data (1070-1, 1070-2, ..., 1070-N) can be obtained by dividing the sample values ​​of N average data (1010-1, 1010-2, ..., 1010-N) by the sample values ​​of quantized data (1050).

[0239] N modified standard deviation data can be obtained by dividing the sample values ​​of N standard deviation data by the sample values ​​of quantized data (1050).

[0240] In one embodiment of the present disclosure, the sample values ​​of the quantized data (1050) may all be one quantization step value when determined by a quantization index. Additionally, the sample values ​​of the quantized data (1050) may be final quantization step values ​​determined based on the quantization data output from the quantization neural network (410) and the quantization step indicated by the quantization index.

[0241] FIG. 11 is a diagram for explaining probability data, modified probability data, and a probability model according to one embodiment of the present disclosure.

[0242] Referring to Fig. 11, N average data (μ) from a probabilistic neural network (330) a ), N standard deviation data (σ a ) and N weight data (w) are acquired, N average data (μ a ) and N standard deviation data (σ a ) is divided into sample values ​​of quantized data (q) and N modified average data (μ b ) and N adjusted standard deviation data (σ b ) can be obtained.

[0243] In one embodiment of the present disclosure, N average data (μ) shown in FIG. 11 a ) may include N average data (1010-1, 1010-2, ..., 1010-N) as shown in Fig. 10. For example, μ of Fig. 11 a1 may correspond to the average data (1010-1) of Fig. 10. Also, for example, μ of Fig. 11 b1 is μ a1 It may correspond to the modified average data (1070-1), which is the result of dividing by the quantized data (1050).

[0244] In one embodiment of the present disclosure, the sample values ​​of the quantized data (1050) may all be one quantization step value when determined by a quantization index. Additionally, the sample values ​​of the quantized data (1050) may be final quantization step values ​​determined based on the quantization data output from the quantization neural network (410) and the quantization step indicated by the quantization index.

[0245] N corrected average data (μ b ), N corrected standard deviation data (σ b ) and N weight data (w) can be applied to a predetermined probability model, for example, the Laplacian probability model or the Gaussian probability model shown in FIG. 11, to derive the probabilities of values ​​that the samples of the quantized first feature data can have.

[0246] According to one embodiment of the present disclosure, N modified average data (μ) for one first feature data b ) and N adjusted standard deviation data (σ b ) are acquired, and N corrected average data (μ b ) and N adjusted standard deviation data (σ b ) are combined according to N weight data (w), so the probability that a sample of the first feature data can have can be derived more accurately and stably.

[0247] FIG. 12 is a diagram illustrating a configuration of an image decoding device according to one embodiment of the present disclosure.

[0248] Referring to FIG. 12, an image decoding device (1200) according to one embodiment of the present disclosure may include an acquisition unit (1210) and a prediction decoding unit (1230).

[0249] The acquisition unit (1210) and the prediction decoding unit (1230) may be implemented with at least one processor. The acquisition unit (1210) and the prediction decoding unit (1230) may operate according to at least one instruction stored in a memory.

[0250] Although FIG. 12 illustrates the acquisition unit (1210) and the prediction decoding unit (1230) separately, the acquisition unit (1210) and the prediction decoding unit (1230) may be implemented through a single processor. In this case, the acquisition unit (1210) and the prediction decoding unit (1230) may be implemented through a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software. In addition, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or a memory processing unit for utilizing external memory.

[0251] The acquisition unit (1210) and the prediction decoding unit (1230) may be implemented using multiple processors. In this case, the acquisition unit (1210) and the prediction decoding unit (1230) may be implemented using a combination of dedicated processors, or may be implemented using a combination of multiple general-purpose processors, such as an AP, CPU, or GPU, and software.

[0252] The acquisition unit (1210) can acquire a bitstream generated through neural network-based encoding of the current image (100). The bitstream can be generated through intra prediction as described in relation to Fig. 1 or inter prediction as described in relation to Fig. 2.

[0253] The acquisition unit (1210) can receive a bitstream from the image encoding device (1800) via a network. In one embodiment of the present disclosure, the acquisition unit (1210) can also acquire a bitstream from a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, etc.

[0254] The acquisition unit (1210) can acquire first feature data dequantized from a bitstream.

[0255] In one embodiment of the present disclosure, the acquisition unit (1210) can acquire a quantization index indicating one of a plurality of quantization steps included in a predetermined quantization list from a bitstream.

[0256] The first feature data may include at least one of feature data (k) of the current image (100) output from the image encoder (12), feature data (w) of the optical flow (g) output from the optical flow encoder (42), or feature data (v) of the residual image (r) output from the residual encoder (52).

[0257] In one embodiment of the present disclosure, the acquisition unit (1210) can acquire second feature data for first feature data from a bitstream and acquire modified probability data using the second feature data. In addition, the acquisition unit (510) can acquire dequantized first feature data through entropy decoding and dequantization of bits included in the bitstream.

[0258] In one embodiment of the present disclosure, the acquisition unit (1210) can acquire second feature data for first feature data from a bitstream and acquire modified probability data using the second feature data. In addition, the acquisition unit (1210) can acquire a quantization step indicated by a quantization index. In addition, the acquisition unit (1210) can acquire dequantized first feature data through entropy decoding and dequantization of bits included in the bitstream.

[0259] In one embodiment of the present disclosure, the acquisition unit (1210) can acquire quantized data using the second feature data, acquire a quantization step indicated by a quantization index, and acquire final quantized step values ​​based on the quantized data and the quantization step.

[0260] The dequantized first feature data is transmitted to the prediction decoding unit (1230), and the prediction decoding unit (1230) can apply the dequantized first feature data to a neural network to obtain a current restored image (300). The current restored image (300) can be output to a display device for playback.

[0261] In one embodiment of the present disclosure, the prediction decoding unit (1230) can obtain a current restored image (300) by applying the dequantized first feature data to the image decoder (14). In this case, the prediction decoding unit (1230) can be understood as restoring the current image (100) through intra prediction.

[0262] In one embodiment of the present disclosure, the prediction decoding unit (1230) can obtain the optical flow (g) by applying the inverse quantized first feature data, for example, the inverse quantized feature data of the optical flow (g), to the optical flow decoder (44). Then, the prediction decoding unit (1230) can obtain the reconstructed residual image (r') by applying the inverse quantized feature data of the residual image (r) to the residual decoder (54). The prediction decoding unit (1230) can obtain the current reconstructed image (300) by combining the current prediction image (x') obtained from the previous reconstructed image (200) and the reconstructed residual image (r') based on the optical flow (g). In this case, it can be understood that the prediction decoding unit (1230) reconstructs the current image (100) through inter prediction.

[0263] FIG. 13 is a diagram illustrating the configuration of an acquisition unit according to one embodiment of the present disclosure.

[0264] Referring to FIG. 13, the acquisition unit (1210) may include an entropy decoding unit (1310), an inverse quantization unit (1330), and an AI control unit (1350).

[0265] A bitstream is input to an entropy decoding unit (1310), and the entropy decoding unit (1310) can obtain quantized second feature data by applying entropy decoding to bits included in the bitstream.

[0266] The entropy decoding unit (1310) can obtain a quantization index indicating one of a plurality of quantization steps included in a predefined quantization list from the bitstream.

[0267] The entropy decoding unit (1310) can transmit the quantization index to the AI ​​control unit (1350).

[0268] The second feature data may be data obtained by processing the first feature data by the hyper-primer encoder (310). The image encoding device (1800) may quantize the second feature data and entropy-encode the quantized second feature data to generate a bitstream including bits corresponding to the quantized second feature data.

[0269] In one embodiment of the present disclosure, quantization may not be applied to the second feature data. In this case, the entropy decoding unit (1310) may apply entropy decoding to bits included in the bitstream to obtain the second feature data, and transmit the obtained second feature data to the AI ​​control unit (1350).

[0270] The quantized second feature data can be transmitted to the dequantization unit (1330). The dequantization unit (1330) can dequantize the quantized second feature data and transmit the dequantized second feature data to the AI ​​control unit (1350).

[0271] In one embodiment of the present disclosure, the quantized second feature data obtained by the entropy decoding unit (1310) may be provided from the entropy decoding unit (1310) to the AI ​​control unit (1350). This means that dequantization is skipped for the quantized second feature data.

[0272] In one embodiment of the present disclosure, the entropy decoding unit (1310) may use predetermined probability data to obtain second feature data (non-quantized second feature data or quantized second feature data) from a bitstream.

[0273] The probability data used to obtain the second feature data can be determined based on a rule. For example, the entropy decoding unit (1310) can determine the probability data used to obtain the second feature data based on a predefined rule without using a neural network.

[0274] In one embodiment of the present disclosure, the entropy decoding unit (1310) may obtain probability data used to obtain the second feature data based on a pre-trained neural network.

[0275] In one embodiment of the present disclosure, the inverse quantization unit (1330) can use predetermined quantization data to inverse quantize the quantized second feature data.

[0276] The quantization data used to obtain the second feature data can be determined based on rules. For example, the inverse quantization unit (1330) can determine the quantization data used to inverse quantize the quantized second feature data according to predefined rules without using a neural network. For example, the inverse quantization unit (1330) can inverse quantize the quantized second feature data according to a predefined quantization step size.

[0277] In one embodiment of the present disclosure, the inverse quantization unit (1330) can inverse quantize sample values ​​of the quantized second feature data according to the same quantization step size.

[0278] In one embodiment of the present disclosure, the inverse quantization unit (1330) may obtain quantized data used to inverse quantize the quantized second feature data based on a pre-trained neural network.

[0279] The AI ​​control unit (1350) can obtain probability data using second feature data, specifically, non-quantized second feature data, quantized second feature data, or inversely quantized second feature data.

[0280] The AI ​​control unit (1350) can obtain a quantization step indicated by the quantization index.

[0281] In one embodiment of the present disclosure, the AI ​​control unit (1350) may use a neural network to obtain probability data.

[0282] The AI ​​control unit (1350) can obtain modified probability data based on the probability data and quantization step.

[0283] The modified probability data can be transmitted to the entropy decoding unit (1310), and the quantization step can be transmitted to the inverse quantization unit (1330).

[0284] The entropy decoding unit (1310) can obtain quantized first feature data by applying entropy decoding based on modified probability data to bits included in the bitstream. The quantized first feature data can be transmitted to the dequantization unit (1330).

[0285] The dequantization unit (1330) can dequantize the quantized first feature data based on the quantization step transmitted from the AI ​​control unit (1350) and transmit the dequantized first feature data to the prediction decoding unit (1230).

[0286] The operation of the AI ​​control unit (1350) will be described in more detail with reference to FIG. 14.

[0287] FIG. 14 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0288] The AI ​​control unit (1350) can use a probabilistic neural network (330) to obtain probability data.

[0289] The probabilistic neural network (330) may be stored in memory. In one embodiment of the present disclosure, the probabilistic neural network (330) may be implemented as an AI processor.

[0290] The second feature data (specifically, non-quantized second feature data, quantized second feature data, or dequantized second feature data) can be input into the probabilistic neural network (330).

[0291] The probabilistic neural network (330) can process the second feature data according to the parameters set as a result of training and output probability data.

[0292] One quantization step among a plurality of quantization steps included in a predetermined quantization list can be determined from a quantization index (510) transmitted in a bitstream.

[0293] The quantization index may represent a quantization parameter instead of a quantization step, and the probability data may include values ​​representing probabilities of values ​​that samples of the first feature data may have. In one embodiment of the present disclosure, the probability data may include the mean, standard deviation, and / or variance for each sample of the first feature data as a sample.

[0294] In one embodiment of the present disclosure, the size or number of samples of the probability data may be the same as the size or number of samples of the first feature data.

[0295] As described above, since the distribution of sample values ​​of the first feature data can be changed through quantization based on the optimal quantization step indicated by the quantization index, the probability data can be modified through a modification process (430) based on the quantization step.

[0296] In one embodiment of the present disclosure, the AI ​​control unit (1350) may obtain modified probability data by dividing the sample values ​​of probability data by a quantization step value. Here, the division is an example, and in one embodiment of the present disclosure, the AI ​​control unit (1350) may also obtain modified probability data by multiplying the sample values ​​of probability data by a value derived from a quantization step value or a quantization parameter.

[0297] In one embodiment of the present disclosure, the AI ​​control unit (1350) may use a bit shift operation for a division operation or a multiplication operation on sample values ​​of probability data.

[0298] In one embodiment of the present disclosure, the correction process (430) may be performed based on a neural network. For example, corrected probability data may be obtained by applying probability data and quantization steps to a neural network for the correction process (430).

[0299] The AI ​​control unit (1350) can transmit the modified probability data to the entropy decoding unit (1310) and transmit the quantization step to the dequantization unit (1330).

[0300] The entropy decoding unit (1310) can obtain quantized first feature data by applying entropy decoding based on modified probability data to bits of the bitstream. The dequantization unit (1330) can obtain dequantized first feature data by dequantizing the quantized first feature data according to a quantization step.

[0301] In addition to the quantization step indicated by the quantization index, the configuration of an image decoding device (1200) that additionally utilizes quantization data obtained through a quantization neural network (410) is described later in FIGS. 15 and 16.

[0302] FIG. 15 is a diagram illustrating the configuration of an acquisition unit according to one embodiment of the present disclosure.

[0303] Referring to FIG. 15, the acquisition unit (1210) may include an entropy decoding unit (1310), an inverse quantization unit (1330), and an AI control unit (1350).

[0304] A bitstream is input to an entropy decoding unit (1310), and the entropy decoding unit (1310) can obtain quantized second feature data by applying entropy decoding to bits included in the bitstream.

[0305] The entropy decoding unit (1310) can obtain a quantization index indicating one of a plurality of quantization steps included in a predefined quantization list from the bitstream.

[0306] The entropy decoding unit (1310) can transmit the quantization index to the AI ​​control unit (1350).

[0307] The second feature data may be data obtained by processing the first feature data by the hyper-primer encoder (310). The image encoding device (1800) may quantize the second feature data and entropy-encode the quantized second feature data to generate a bitstream including bits corresponding to the quantized second feature data.

[0308] In one embodiment of the present disclosure, quantization may not be applied to the second feature data. In this case, the entropy decoding unit (1310) may apply entropy decoding to bits included in the bitstream to obtain the second feature data, and transmit the obtained second feature data to the AI ​​control unit (1350).

[0309] The quantized second feature data can be transmitted to the dequantization unit (1330). The dequantization unit (1330) can dequantize the quantized second feature data and transmit the dequantized second feature data to the AI ​​control unit (1350).

[0310] In one embodiment of the present disclosure, the quantized second feature data obtained by the entropy decoding unit (1310) may be provided from the entropy decoding unit (1310) to the AI ​​control unit (1350). This means that dequantization is skipped for the quantized second feature data.

[0311] In one embodiment of the present disclosure, the entropy decoding unit (1310) may use predetermined probability data to obtain second feature data (non-quantized second feature data or quantized second feature data) from a bitstream.

[0312] The probability data used to obtain the second feature data can be determined based on a rule. For example, the entropy decoding unit (1310) can determine the probability data used to obtain the second feature data based on a predefined rule without using a neural network.

[0313] In one embodiment of the present disclosure, the entropy decoding unit (1310) may obtain probability data used to obtain the second feature data based on a pre-trained neural network.

[0314] In one embodiment of the present disclosure, the inverse quantization unit (1330) can use predetermined quantization data to inverse quantize the quantized second feature data.

[0315] The quantization data used to obtain the second feature data can be determined based on a rule. For example, the inverse quantization unit (1330) can determine the quantization data used to inverse quantize the quantized second feature data according to a predefined rule without using a neural network. For example, the inverse quantization unit (1330) can inverse quantize the quantized second feature data according to a predefined quantization step size.

[0316] In one embodiment of the present disclosure, the inverse quantization unit (1330) can inverse quantize sample values ​​of the quantized second feature data according to the same quantization step size.

[0317] In one embodiment of the present disclosure, the inverse quantization unit (1330) may obtain quantized data used to inverse quantize the quantized second feature data based on a pre-trained neural network.

[0318] The AI ​​control unit (1350) can obtain probability data and quantized data using second feature data, specifically, non-quantized second feature data, quantized second feature data, or inversely quantized second feature data.

[0319] The AI ​​control unit (1350) can obtain a quantization step indicated by a quantization index. In one embodiment of the present disclosure, the AI ​​control unit (1350) can utilize a neural network to obtain probability data and quantization data.

[0320] The AI ​​control unit (1350) can determine final quantization step values ​​based on the quantization step and quantization data indicated by the quantization index.

[0321] The AI ​​control unit (1350) can obtain modified probability data based on the final quantization step values ​​and probability data.

[0322] The modified probability data can be transmitted to the entropy decoding unit (1310), and the final quantization step values ​​can be transmitted to the dequantization unit (1330).

[0323] The entropy decoding unit (1310) can obtain quantized first feature data by applying entropy decoding based on modified probability data to bits included in the bitstream. The quantized first feature data can be transmitted to the dequantization unit (1330).

[0324] The dequantization unit (1330) can dequantize the quantized first feature data based on the final quantization step values ​​transmitted from the AI ​​control unit (1350) and transmit the dequantized first feature data to the prediction decoding unit (530).

[0325] The operation of the AI ​​control unit (1350) will be described in more detail with reference to FIG. 16.

[0326] FIG. 16 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0327] The AI ​​control unit (1350) can use a probability neural network (330) and a quantization neural network (410) to obtain probability data and quantization data.

[0328] The probabilistic neural network (330) and the quantized neural network (410) may be stored in memory. In one embodiment of the present disclosure, the probabilistic neural network (330) and the quantized neural network (410) may be implemented as an AI processor.

[0329] The second feature data (specifically, non-quantized second feature data, quantized second feature data, or inversely quantized second feature data) can be input to the probabilistic neural network (330) and the quantized neural network (410).

[0330] The probabilistic neural network (330) can process the second feature data according to the parameters set as a result of training and output probability data.

[0331] The quantization neural network (410) can process the second feature data according to the parameters set as a result of training and output quantized data.

[0332] One quantization step among a plurality of quantization steps included in a predetermined quantization list can be determined from a quantization index (510) transmitted in a bitstream.

[0333] The quantization data may include a quantization parameter or a quantization step size, the quantization index may indicate a quantization parameter instead of a quantization step, and the probability data may include values ​​indicating probabilities of values ​​that samples of the first feature data may have. In one embodiment of the present disclosure, the probability data may include a mean, standard deviation, and / or variance for each sample of the first feature data as a sample.

[0334] In one embodiment of the present disclosure, the size or number of samples of the probability data and the quantization data may be the same as the size or number of samples of the first feature data.

[0335] As described above, the final quantization step values ​​can be determined by multiplying (1610) the optimal quantization step indicated by the quantization index (510) and the quantization step values ​​included in the quantization data obtained from the quantization neural network (410).

[0336] Since the distribution of sample values ​​of the first feature data can be changed through quantization based on the final quantization step values, the probability data can be modified through a modification process (430) based on the quantization data.

[0337] In one embodiment of the present disclosure, the AI ​​control unit (1350) can obtain modified probability data by dividing the sample values ​​of the probability data by the final quantization step values. Here, the division is an example, and in one embodiment of the present disclosure, when the sample values ​​included in the quantization data and the value indicated by the quantization index (510) are quantization parameters, the AI ​​control unit (1350) can derive quantization step values ​​from the sample values ​​included in the quantization data, derive quantization step values ​​from the quantization parameter indicated by the quantization index (510), and multiply (1610) the quantization step values ​​derived from the quantization data by the quantization step values ​​derived from the quantization index (510) to obtain final quantization step values. Then, the AI ​​control unit (1350) can obtain modified probability data by multiplying the sample values ​​of the probability data by the final quantization step values.

[0338] In one embodiment of the present disclosure, the AI ​​control unit (1350) may use a bit shift operation for a division operation or a multiplication operation on sample values ​​of probability data.

[0339] In one embodiment of the present disclosure, the correction process (430) may be performed based on a neural network. For example, corrected probability data may be obtained by applying probability data and final quantization step values ​​to a neural network for the correction process (430).

[0340] The AI ​​control unit (1350) can transmit the modified probability data to the entropy decoding unit (1310) and transmit the final quantization step values ​​to the dequantization unit (1330).

[0341] The entropy decoding unit (1310) can obtain quantized first feature data by applying entropy decoding based on modified probability data to bits of the bitstream. The dequantization unit (1330) can obtain dequantized first feature data by dequantizing the quantized first feature data according to final quantization step values.

[0342] FIG. 17 is a diagram for explaining an image decoding method according to one embodiment of the present disclosure.

[0343] In step S1710, the image decoding device (1200) can obtain second feature data for first feature data obtained through neural network-based encoding for the current image (100) and a quantization index indicating one quantization step among a plurality of quantization steps from the bitstream.

[0344] In one embodiment of the present disclosure, the first feature data may include feature data (k) obtained by applying the current image (100) to the image encoder (12), feature data (w) obtained by applying the current image (100) and the previously restored image (200) to the optical flow encoder (42), or feature data (v) obtained by applying the residual image (r) corresponding to the current image (100) to the residual encoder (52).

[0345] In one embodiment of the present disclosure, the image decoding device (500) can obtain second feature data by applying entropy decoding to bits included in a bitstream.

[0346] In one embodiment of the present disclosure, the image decoding device (500) can obtain quantized second feature data by applying entropy decoding to bits included in a bitstream, and dequantize the quantized second feature data.

[0347] At step S1720, the image decoding device (1200) obtains a quantization step based on the quantization index.

[0348] In one embodiment of the present disclosure, the size of the quantization step indicated by the quantization index may be 1 or greater.

[0349] In one embodiment of the present disclosure, a quantization index may indicate one quantization step among a plurality of quantization steps included in a predetermined quantization list.

[0350] In step S1730, the image decoding device (1200) applies the second feature data to the neural network to obtain probability data.

[0351] At step S1740, the image decoding device (1200) can modify the probability data based on the quantization step.

[0352] In one embodiment of the present disclosure, the image decoding device (1200) can divide sample values ​​of probability data into quantization steps. If the value indicated by the quantization index corresponds to a quantization parameter, the image decoding device (1200) can determine a quantization step size from the quantization parameter and divide the sample values ​​of the probability data into the determined quantization step size.

[0353] In one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent the probability of the values ​​that the samples of the quantized first feature data may have.

[0354] In one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent a mean and a standard deviation corresponding to samples of the quantized first feature data.

[0355] In one embodiment of the present disclosure, the probability of a value that a sample of quantized first feature data can have can be derived by applying the mean and standard deviation represented by the sample values ​​of the modified probability data to a predetermined probability model.

[0356] In step S1750, the image decoding device (1200) applies entropy decoding based on modified probability data to bits included in the bitstream to obtain quantized first feature data.

[0357] In step S1760, the image decoding device (1200) dequantizes the quantized first feature data according to the quantization step to obtain dequantized first feature data.

[0358] At step S1770, the image decoding device (1200) restores the current image (100) through neural network-based decoding of the dequantized first feature data.

[0359] In one embodiment of the present disclosure, the image decoding device (1200) can apply the inverse quantized first feature data to the image decoder (14), the optical flow decoder (44), and / or the residual decoder (54) to restore the current image (100).

[0360] In one embodiment of the present disclosure, the image decoding device (1200) can apply second feature data to a second neural network to further obtain quantized data, modify probability data based on sample values ​​and quantization steps of the quantized data, and inversely quantize the quantized first feature data based on the sample values ​​and quantization steps of the quantized data.

[0361] In one embodiment of the present disclosure, final quantization step values ​​are determined by multiplying sample values ​​of quantized data by a quantization step size, and probability data can be modified based on the final quantization step values. The image decoding device (1200) can divide each sample value of the probability data into a respective final quantization step value corresponding to each sample.

[0362] In one embodiment of the present disclosure, the quantized data may include a quantization parameter or a quantization step size as a sample.

[0363] In one embodiment of the present disclosure, by applying second feature data to a first neural network, a plurality of probability data and a plurality of weights are obtained, the plurality of probability data are modified based on a quantization step indicated by a quantization index, and the plurality of modified probability data are combined according to the plurality of weights, so that the probability of a value that a sample of the quantized first feature data can have can be determined.

[0364] In one embodiment of the present disclosure, by applying second feature data to a first neural network, a plurality of probability data and a plurality of weights are obtained, the plurality of probability data are modified based on final quantization step values, and the plurality of modified probability data are combined according to the plurality of weights, so that the probability of a value that a sample of the quantized first feature data can have can be determined.

[0365] FIG. 18 is a diagram illustrating a configuration of an image encoding device according to one embodiment of the present disclosure.

[0366] Referring to FIG. 18, the image encoding device (1800) may include a prediction encoding unit (1810), a generation unit (1820), an acquisition unit (1830), and a prediction decoding unit (1840).

[0367] The prediction encoding unit (1810), the generation unit (1820), the acquisition unit (1830), and the prediction decoding unit (1840) may be implemented as a processor. The prediction encoding unit (1810), the generation unit (1820), the acquisition unit (1830), and the prediction decoding unit (1840) may operate according to instructions stored in a memory.

[0368] Although FIG. 18 illustrates the prediction encoding unit (1810), the generation unit (1820), the acquisition unit (1830), and the prediction decoding unit (1840) individually, the prediction encoding unit (1810), the generation unit (1820), the acquisition unit (1830), and the prediction decoding unit (1840) may be implemented through a single processor. In this case, the prediction encoding unit (1810), the generation unit (1820), the acquisition unit (1830), and the prediction decoding unit (1840) may be implemented as a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software. In addition, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or may include a memory processing unit for utilizing an external memory.

[0369] The prediction encoding unit (1810), the generation unit (1820), the acquisition unit (1830), and the prediction decoding unit (1840) may be implemented with multiple processors. In this case, the prediction encoding unit (1810), the generation unit (1820), the acquisition unit (1830), and the prediction decoding unit (1840) may be implemented with a combination of dedicated processors, or may be implemented with a combination of multiple general-purpose processors such as an AP, CPU, or GPU, and software.

[0370] The predictive encoding unit (1810) can obtain first feature data by applying neural network-based encoding to the current image (100). The first feature data can include at least one of feature data (k) of the current image (100), feature data (w) of optical flow (g), or feature data (v) of a residual image (r).

[0371] In one embodiment of the present disclosure, the prediction encoding unit (1810) can obtain feature data (k) of the current image (100) by applying the current image (100) to the image encoder (12).

[0372] In one embodiment of the present disclosure, the prediction encoding unit (1810) can obtain feature data (w) of optical flow (g) by applying the current image (100) and the previously restored image (200) to the optical flow encoder (42).

[0373] In one embodiment of the present disclosure, the prediction encoding unit (1810) can obtain feature data (v) of the residual image (r) by applying the residual image (r) corresponding to the difference between the current prediction image (x') and the current image (100) to the residual encoder (52).

[0374] The first feature data acquired by the prediction encoding unit (1810) can be transmitted to the generation unit (1820).

[0375] The generation unit (1820) can generate a bitstream based on the first feature data.

[0376] In one embodiment of the present disclosure, the generation unit (1820) may obtain second feature data representing potential features of the first feature data, and apply the second feature data to a neural network to obtain probability data. In addition, the generation unit (1820) may quantize the first feature data according to an optimal quantization step among a plurality of quantization steps predetermined through RDO calculation. The generation unit (1820) may obtain modified probability data based on the optimal quantization step and the probability data. The generation unit (1820) may generate a bitstream by entropy encoding the quantized first feature data according to the modified probability data.

[0377] In one embodiment of the present disclosure, the generation unit (1820) can generate a bitstream by entropy encoding the second feature data or the quantized second feature data according to predetermined probability data.

[0378] In one embodiment of the present disclosure, the bitstream may include bits corresponding to quantized first feature data, bits corresponding to second feature data or quantized second feature data, and bits corresponding to a quantization index indicating an optimal quantization step.

[0379] The bitstream may be transmitted to the image decoding device (1200) via a network. In one embodiment of the present disclosure, the bitstream may be recorded on a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, etc.

[0380] The acquisition unit (1830) can acquire the dequantized first feature data and quantization index from the bitstream generated by the generation unit (1820).

[0381] The dequantized first feature data can be transmitted to the prediction decoding unit (1840).

[0382] The prediction decoding unit (1840) can obtain a current restored image (300) by applying neural network-based decoding to the dequantized first feature data.

[0383] The configuration and operation of the acquisition unit (1830) and the prediction decoding unit (1840) may be the same as the acquisition unit (1210) and the prediction decoding unit (1230) of the image decoding device (1200).

[0384] FIG. 19 is a diagram illustrating the configuration of a generation unit according to one embodiment of the present disclosure.

[0385] Referring to FIG. 19, the generation unit (1820) may include an AI control unit (1910), a quantization unit (1930), and an entropy encoding unit (1950).

[0386] The first feature data acquired by the prediction encoding unit (1810) can be transmitted to the AI ​​control unit (1910) and the quantization unit (1930).

[0387] The AI ​​control unit (1910) can obtain second feature data from first feature data and obtain probability data based on the second feature data. The AI ​​control unit (1910) can obtain one quantization step among a plurality of quantization steps. The AI ​​control unit (1910) can obtain modified probability data based on the probability data and the quantization step. The second feature data and the quantization step can be transmitted to the quantization unit (1930), and the modified probability data can be transmitted to the entropy encoding unit (1950).

[0388] In one embodiment of the present disclosure, the AI ​​control unit (1910) may obtain probability data based on second feature data to which quantization and dequantization have been applied. Predetermined quantization data, for example, quantization data determined based on rules, may be used for quantization and dequantization of the second feature data.

[0389] The quantization unit (1930) can quantize the first feature data according to the quantization step to obtain quantized first feature data.

[0390] In one embodiment of the present disclosure, the quantization unit (1930) can quantize the second feature data according to rule-based generated quantization data to obtain quantized second feature data. In one embodiment of the present disclosure, the second feature data may be transmitted from the AI ​​control unit (1910) to the entropy encoding unit (1950). This means that quantization of the second feature data is skipped.

[0391] The quantization unit (1930) can transmit the quantized first feature data and the quantized second feature data to the entropy encoding unit (1950).

[0392] The entropy encoding unit (1950) can generate a bitstream by entropy encoding the quantized first feature data according to the modified probability data.

[0393] The entropy encoding unit (1950) can receive a quantization step from the AI ​​control unit (1910) and entropy encode a quantization index representing the quantization step to generate a bitstream.

[0394] In one embodiment of the present disclosure, the entropy encoding unit (1950) can generate a bitstream by entropy encoding the second feature data or the quantized second feature data according to predetermined probability data.

[0395] The bitstream may include bits corresponding to quantized first feature data, bits corresponding to second feature data or quantized second feature data, and bits corresponding to a quantization index.

[0396] Hereinafter, the operation of the AI ​​control unit (1910) will be described in more detail with reference to FIG. 20.

[0397] FIG. 20 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0398] The AI ​​control unit (1910) can utilize a hyperflash encoder (310) and a probabilistic neural network (330).

[0399] The hyperfryer encoder (310) and the probabilistic neural network (330) may be stored in memory. In one embodiment of the present disclosure, the hyperfryer encoder (310) and the probabilistic neural network (330) may be implemented as an AI processor.

[0400] Referring to FIG. 20, the first feature data can be input to the hyperfly encoder (310).

[0401] The hyperflash encoder (310) can process first feature data according to parameters set as a result of training to obtain second feature data. The second feature data can be input into a probabilistic neural network (330).

[0402] In one embodiment of the present disclosure, second feature data to which quantization and inverse quantization have been applied may be input to a probabilistic neural network (330). The reason for quantizing and inverse quantizing the second feature data is to take into account the case where the bitstream transmitted to the image decoding device (1200) includes the quantized second feature data. In other words, since the image decoding device (1200) can obtain probability data by using the inverse quantized second feature data, the image encoding device (1800) also uses the second feature data to which quantization and inverse quantization have been applied in the same manner as the image decoding device (1200).

[0403] The probabilistic neural network (330) can process the second feature data according to the parameters set as a result of training and output probability data.

[0404] One quantization step (2010) among a plurality of predetermined quantization steps can be determined.

[0405] The probability data may include values ​​representing probabilities of values ​​that samples of the first feature data may have. In one embodiment of the present disclosure, the probability data may include the mean, standard deviation, and / or variance for each sample of the first feature data as a sample.

[0406] In one embodiment of the present disclosure, the size or number of samples of the probability data may be the same as the size or number of samples of the first feature data.

[0407] In one embodiment of the present disclosure, the probability data can be modified through a modification process (430) based on a quantization step.

[0408] In one embodiment of the present disclosure, the AI ​​control unit (1910) can obtain modified probability data by dividing the sample values ​​of probability data by quantization steps. Here, the division is an example, and the modified probability data can be obtained by multiplying the sample values ​​of the probability data by a quantization step value. Furthermore, if the value indicated by the quantization index is a quantization parameter, the modified probability data can be obtained by multiplying the sample values ​​by a quantization step value derived from the quantization parameter.

[0409] In one embodiment of the present disclosure, the AI ​​control unit (1910) may use bit shift operations for division or multiplication operations on sample values ​​of probability data.

[0410] In one embodiment of the present disclosure, the correction process (430) may be performed based on a neural network. For example, corrected probability data may be obtained by applying probability data and quantization steps to a neural network for the correction process (430).

[0411] The AI ​​control unit (1910) can transmit the modified probability data to the entropy encoding unit (1950) and transmit the quantization step to the quantization unit (1930).

[0412] The quantization unit (1930) can obtain quantized first feature data by quantizing the first feature data according to the quantization step. In addition, the quantization unit (1930) can obtain second feature data by quantizing the second feature data.

[0413] In one embodiment of the present disclosure, the quantization unit (1930) may use predetermined quantization data to quantize the second feature data. The quantization data used to quantize the second feature data may be determined based on rules. In other words, the quantization unit (1930) may determine the quantization data used to quantize the second feature data according to predefined rules without using a neural network. For example, the quantization unit (1930) may quantize the second feature data according to a predetermined quantization step size. In one embodiment of the present disclosure, the quantization unit (1930) may quantize the sample values ​​of the second feature data according to the same quantization step size.

[0414] The entropy encoding unit (1950) can generate a bitstream by applying entropy encoding based on modified probability data to the quantized first feature data. The entropy encoding unit (1950) can generate a bitstream including a quantization index indicating an optimal quantization step among a plurality of quantization steps through RDO calculation.

[0415] Additionally, the entropy encoding unit (1950) can entropy encode the second feature data or the quantized second feature data.

[0416] In one embodiment of the present disclosure, the entropy encoding unit (1950) may utilize predetermined probability data to apply entropy encoding to the second feature data or the quantized second feature data. The probability data used to entropy encode the second feature data or the quantized second feature data may be determined based on rules. In other words, the entropy encoding unit (1950) may determine the probability data used to entropy encode the second feature data or the quantized second feature data according to predefined rules without using a neural network.

[0417] FIG. 21 is a diagram illustrating the configuration of a generation unit according to one embodiment of the present disclosure.

[0418] Referring to FIG. 21, the generation unit (1820) may include an AI control unit (1910), a quantization unit (1930), and an entropy encoding unit (1950).

[0419] The first feature data acquired by the prediction encoding unit (1810) can be transmitted to the AI ​​control unit (1910) and the quantization unit (1930).

[0420] The AI ​​control unit (1910) can obtain second feature data from first feature data, and obtain quantization data and probability data based on the second feature data. The AI ​​control unit (1910) can obtain one quantization step among a plurality of quantization steps. The AI ​​control unit (1910) can obtain final quantization step values ​​based on the quantization step and the quantization data. The AI ​​control unit (1910) can obtain modified probability data based on the probability data and the final quantization step values. The second feature data and the final quantization step values ​​can be transmitted to the quantization unit (1930), and the modified probability data can be transmitted to the entropy encoding unit (1950).

[0421] In one embodiment of the present disclosure, the AI ​​control unit (1910) may obtain quantized data and probability data based on second feature data to which quantization and dequantization have been applied. Predetermined quantized data, for example, quantized data determined based on rules, may be used for quantization and dequantization of the second feature data.

[0422] The quantization unit (1930) can quantize the first feature data according to the final quantization step values ​​to obtain quantized first feature data.

[0423] In one embodiment of the present disclosure, the quantization unit (1930) can quantize the second feature data according to rule-based generated quantization data to obtain quantized second feature data. In one embodiment of the present disclosure, the second feature data may be transmitted from the AI ​​control unit (1910) to the entropy encoding unit (1950). This means that quantization of the second feature data is skipped.

[0424] The quantization unit (1930) can transmit the quantized first feature data and the quantized second feature data to the entropy encoding unit (1950).

[0425] The entropy encoding unit (1950) can generate a bitstream by entropy encoding the quantized first feature data according to the modified probability data.

[0426] The entropy encoding unit (1950) can generate a bitstream by entropy encoding a quantization index representing one quantization step selected from among a plurality of quantization steps.

[0427] In one embodiment of the present disclosure, the entropy encoding unit (1950) can generate a bitstream by entropy encoding the second feature data or the quantized second feature data according to predetermined probability data.

[0428] The bitstream may include bits corresponding to quantized first feature data, bits corresponding to second feature data or quantized second feature data, and bits corresponding to a quantization index.

[0429] Hereinafter, the operation of the AI ​​control unit (1910) will be described in more detail with reference to FIG. 22.

[0430] FIG. 22 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0431] The AI ​​control unit (1910) can utilize a hyper-primer encoder (310), a quantization neural network (410), and a probabilistic neural network (330).

[0432] The hyperfriction encoder (310), the quantization neural network (410), and the probabilistic neural network (330) may be stored in memory. In one embodiment of the present disclosure, the hyperfriction encoder (310), the quantization neural network (410), and the probabilistic neural network (330) may be implemented as an AI processor.

[0433] Referring to FIG. 22, the first feature data can be input to the hyperfly encoder (310).

[0434] The hyper-primary encoder (310) can process first feature data according to parameters set as a result of training to obtain second feature data. The second feature data can be input to a probabilistic neural network (330) and a quantization neural network (410).

[0435] In one embodiment of the present disclosure, second feature data to which quantization and inverse quantization have been applied may be input to a probability neural network (330) and a quantization neural network (410). The reason for quantizing and inverse quantizing the second feature data is to take into account the case where the bitstream transmitted to the image decoding device (1200) includes the quantized second feature data. In other words, since the image decoding device (1200) can obtain probability data and quantized data by using the inverse quantized second feature data, the image encoding device (1800) also uses the second feature data to which quantization and inverse quantization have been applied in the same manner as the image decoding device (1200).

[0436] The probabilistic neural network (330) can process the second feature data according to the parameters set as a result of training and output probability data.

[0437] The quantization neural network (410) can process the second feature data according to the parameters set as a result of training and output quantized data.

[0438] One quantization step (2010) among a plurality of predetermined quantization steps can be determined.

[0439] The probability data may include values ​​representing probabilities of values ​​that samples of the first feature data may have. In one embodiment of the present disclosure, the probability data may include the mean, standard deviation, and / or variance for each sample of the first feature data as a sample.

[0440] In one embodiment of the present disclosure, the size or number of samples of the probability data and the quantization data may be the same as the size or number of samples of the first feature data.

[0441] In one embodiment of the present disclosure, final quantization step values ​​may be determined based on quantization data and quantization steps (2010). The final quantization step values ​​may be determined by multiplying (2210) the quantization step values ​​of the quantization data by the quantization steps (2010).

[0442] In one embodiment of the present disclosure, the probability data can be modified through a modification process (430) based on the final quantization step values.

[0443] In one embodiment of the present disclosure, the AI ​​control unit (1910) may obtain modified probability data by dividing the sample values ​​of probability data by the final quantization step value for each sample. Here, the division may be performed as an example, and the modified probability data may be obtained by multiplying the sample values ​​of probability data by the final quantization step value for each sample.

[0444] In one embodiment of the present disclosure, the AI ​​control unit (1910) may use bit shift operations for division or multiplication operations on sample values ​​of probability data.

[0445] In one embodiment of the present disclosure, the correction process (430) may be performed based on a neural network. For example, corrected probability data may be obtained by applying probability data and final quantization step values ​​to a neural network for the correction process (430).

[0446] The AI ​​control unit (1910) can transmit the modified probability data to the entropy encoding unit (1950) and transmit the final quantization step values ​​to the quantization unit (1930).

[0447] The quantization unit (1930) can obtain quantized first feature data by quantizing the first feature data according to the final quantization step values. In addition, the quantization unit (1930) can obtain second feature data by quantizing the second feature data.

[0448] In one embodiment of the present disclosure, the quantization unit (1930) may use predetermined quantization data to quantize the second feature data. The quantization data used to quantize the second feature data may be determined based on rules. In other words, the quantization unit (1930) may determine the quantization data used to quantize the second feature data according to predefined rules without using a neural network. For example, the quantization unit (1930) may quantize the second feature data according to a predetermined quantization step size. In one embodiment of the present disclosure, the quantization unit (1930) may quantize the sample values ​​of the second feature data according to the same quantization step size.

[0449] The entropy encoding unit (1950) can generate a bitstream by applying entropy encoding based on modified probability data to the quantized first feature data.

[0450] The entropy encoding unit (1950) can generate a bitstream including a quantization index indicating an optimal quantization step among a plurality of quantization steps through RDO calculation.

[0451] Additionally, the entropy encoding unit (1950) can entropy encode the second feature data or the quantized second feature data.

[0452] In one embodiment of the present disclosure, the entropy encoding unit (1950) may utilize predetermined probability data to apply entropy encoding to the second feature data or the quantized second feature data. The probability data used to entropy encode the second feature data or the quantized second feature data may be determined based on rules. In other words, the entropy encoding unit (1950) may determine the probability data used to entropy encode the second feature data or the quantized second feature data according to predefined rules without using a neural network.

[0453] FIG. 23 is a drawing for explaining an image encoding method according to one embodiment of the present disclosure.

[0454] In step S2310, the image encoding device (1800) applies the first feature data obtained through neural network-based encoding of the current image (100) to the first neural network to obtain second feature data for the first feature data.

[0455] According to one embodiment of the present disclosure, the first neural network may be a hyperfryer encoder (310).

[0456] In one embodiment of the present disclosure, the first feature data may include feature data (k) obtained by applying the current image (100) to the image encoder (12), feature data (w) obtained by applying the current image (100) and the previously restored image (200) to the optical flow encoder (42), or feature data (v) obtained by applying the residual image (r) corresponding to the current image (100) to the residual encoder (52).

[0457] In step S2320, the image encoding device (1800) applies the second feature data to a second neural network (e.g., a probabilistic neural network (330)) to obtain probability data.

[0458] In one embodiment of the present disclosure, the image encoding device (1800) can obtain probability data by applying quantized and dequantized second feature data to a neural network.

[0459] In step S2330, the image encoding device (1800) modifies the probability data based on one quantization step among a plurality of predetermined quantization steps. In one embodiment of the present disclosure, the image encoding device (1800) may divide the sample values ​​of the probability data into quantization steps. In addition, the image encoding device (1800) may determine the quantization step from one quantization parameter among a plurality of quantization parameters, and divide the sample values ​​of the probability data into the determined quantization steps.

[0460] In one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent the probability of the values ​​that the samples of the quantized first feature data may have. In one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent the mean and standard deviation corresponding to the samples of the quantized first feature data.

[0461] In one embodiment of the present disclosure, the probability of a value that a sample of quantized first feature data can have can be derived by applying the mean and standard deviation represented by the sample values ​​of the modified probability data to a predetermined probability model.

[0462] In step S2340, the image encoding device (1400) can quantize the first feature data according to the quantization step to obtain quantized first feature data.

[0463] In one embodiment of the present disclosure, the image encoding device (1800) can apply the second feature data to a third neural network (e.g., a quantization neural network (410)) to further obtain quantized data, modify probability data based on sample values ​​and quantization steps of the quantized data, and quantize the first feature data based on the sample values ​​and quantization steps of the quantized data.

[0464] In one embodiment of the present disclosure, final quantization step values ​​are determined by multiplying sample values ​​of quantized data by a quantization step size, and probability data can be modified based on the final quantization step values. The image decoding device (1200) can divide sample values ​​of probability data into final quantization step values ​​for each sample.

[0465] In one embodiment of the present disclosure, the quantized data may include a quantization parameter or a quantization step size as a sample.

[0466] In one embodiment of the present disclosure, by applying second feature data to a second neural network, a plurality of probability data and a plurality of weights are obtained, the plurality of probability data are modified based on a quantization step indicated by a quantization index, and the plurality of modified probability data are combined according to the plurality of weights, so that the probability of a value that a sample of the quantized first feature data can have can be determined.

[0467] In one embodiment of the present disclosure, by applying second feature data to a second neural network, a plurality of probability data and a plurality of weights are obtained, the plurality of probability data are modified based on final quantization step values, and the plurality of modified probability data are combined according to the plurality of weights, so that the probability of a value that a sample of the quantized first feature data can have can be determined.

[0468] In one embodiment of the present disclosure, the image encoding device (1800) can obtain quantized second feature data by quantizing second feature data according to predetermined quantization data.

[0469] In step S2350, the image encoding device (1800) can apply entropy encoding based on modified probability data for the quantized first feature data, and apply entropy encoding to a quantization index corresponding to a quantization step to generate a bitstream including the quantized first feature data and bits corresponding to the quantization index.

[0470] In one embodiment of the present disclosure, the size of the quantization step indicated by the quantization index may be 1 or greater.

[0471] In one embodiment of the present disclosure, a quantization index may indicate one quantization step among a plurality of quantization steps included in a predetermined quantization list.

[0472] In one embodiment of the present disclosure, the image encoding device (1800) may entropy encode non-quantized second feature data or quantized second feature data according to predetermined probability data. In this case, the bitstream may include bits corresponding to the quantized first feature data, bits corresponding to the non-quantized second feature data or the quantized second feature data, and bits corresponding to a quantization index.

[0473] Hereinafter, with reference to FIG. 24, a method for training the aforementioned neural networks, hyper-precision encoder (310) and probabilistic neural network (330) will be described.

[0474] FIG. 24 is a diagram for explaining a method for training neural networks according to one embodiment of the present disclosure.

[0475] Referring to FIG. 24, the current training image may correspond to the current image (100) described above, and the current restored training image may correspond to the current restored image (300) described above.

[0476] In a training process according to one embodiment of the present disclosure, neural networks can be trained so that the current reconstructed training image is as similar as possible to the current training image, and the bitrate of the bitstream generated through encoding the current training image is minimized. To this end, as illustrated in FIG. 24, first loss information (2480) and second loss information (2490) can be used to train the neural networks.

[0477] Specifically describing the training process of neural networks, first, a neural network-based encoding process (2410) is applied to the current training image so that first feature data can be obtained.

[0478] The neural network-based encoding process (2410) may be a process of encoding the current training image based on an image encoder (12), an optical flow encoder (42), and / or a residual encoder (52).

[0479] The first feature data may include at least one of feature data obtained by processing the current training image by the image encoder (12), feature data obtained by processing the current training image and the previous reconstructed training image by the optical flow encoder (42), or feature data obtained by processing the residual training image corresponding to the difference between the current training image and the current prediction training image by the residual encoder (52). Here, the current prediction training image may be obtained by transforming the previous reconstructed training image according to the optical flow (g).

[0480] The first feature data can be input to the hyperfryer encoder (310). The hyperfryer encoder (310) can process the first feature data according to preset parameters and output the second feature data.

[0481] The second feature data can be input into a probabilistic neural network (330). The probabilistic neural network (330) can process the second feature data according to preset parameters and output probability data.

[0482] The quantization step (2010) may be selected from among a plurality of predetermined quantization steps. The quantization step (2010) may be selected through RDO calculation.

[0483] Probability data can be modified through a modification process (2420) depending on the quantization step. The modification process (2420) has been described in FIGS. 8 to 11, etc., so a detailed description thereof will be omitted.

[0484] A quantization process (2430) according to a quantization step may be applied to the first feature data to obtain quantized first feature data. In addition, a bitstream may be generated by applying an entropy encoding process (2440) based on modified probability data to the quantized first feature data. In the entropy encoding process (2440), a quantization index indicating a quantization step may also be included in the bitstream.

[0485] In one embodiment of the present disclosure, the bitstream may include bits corresponding to second feature data or quantized second feature data.

[0486] An entropy decoding process (2450) based on modified probability data is performed on the bitstream to obtain quantized first feature data, and an inverse quantization process (2460) according to a quantization step is performed on the quantized first feature data to obtain inverse quantized first feature data.

[0487] The current restored training image can be obtained by processing the dequantized first feature data according to a neural network-based decoding process (2470).

[0488] The neural network-based decoding process may be a process of restoring the current image (100) based on an image decoder (14), an optical flow decoder (44), and / or a residual decoder (54).

[0489] For training of the neural network, hyper-prior encoder (310), probabilistic neural network (330) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process, at least one of the first loss information (2480) or the second loss information (2490) can be obtained.

[0490] The first loss information (2480) can be derived from the bitrate of the bitstream generated as an encoding result for the current training image.

[0491] Since the first loss information (2480) is related to the encoding efficiency for the current training image, the first loss information may be referred to as compression loss information.

[0492] The second loss information (2490) may correspond to a difference between the current training image and the current reconstructed training image. In one embodiment of the present disclosure, the difference between the current training image and the current reconstructed training image may include at least one of an L1-norm value, an L2-norm value, a Structural Similarity (SSIM) value, a Peak Signal-To-Noise Ratio-Human Vision System (PSNR-HVS) value, a Multiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF) value, or a Video Multimethod Assessment Fusion (VMAF) value between the current training image and the current reconstructed training image.

[0493] The second loss information (2490) is related to the quality of the current restoration training image, and therefore may be referred to as quality loss information.

[0494] The neural network, hyper-priority encoder (310), probabilistic neural network (330) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process can be trained so that the final loss information derived from at least one of the first loss information (2480) or the second loss information (2490) is reduced or minimized.

[0495] In one embodiment of the present disclosure, the neural network, hyper-priority encoder (310), probabilistic neural network (330) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process can reduce or minimize final loss information by changing the values ​​of preset parameters.

[0496] In one embodiment of the present disclosure, the final loss information can be calculated according to the following mathematical expression 4.

[0497] [Equation 4]

[0498] Final loss information = a*first loss information+b*second loss information

[0499] In mathematical expression 4, a and b are weights applied to the first loss information (2480) and the second loss information (2490), respectively.

[0500] According to mathematical expression 4, it can be seen that the neural network, hyper-priority encoder (310), probabilistic neural network (330) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process are trained in a direction in which the current restoration training image becomes as similar to the current training image as possible and the size of the bitstream is minimized.

[0501] The training process described with reference to FIG. 24 can be performed by a training device. The training device may be, for example, an image encoding device (1800) or a separate server. The parameters obtained as a result of the training may be stored in the image encoding device (1800) and the image decoding device (1200).

[0502] Hereinafter, with reference to FIG. 25, a method for training the aforementioned neural networks, a hyper-priority encoder (310), a probabilistic neural network (330), and a quantization neural network (410) will be described.

[0503] FIG. 25 is a diagram for explaining a method for training neural networks according to one embodiment of the present disclosure.

[0504] Referring to FIG. 25, the current training image may correspond to the current image (100) described above, and the current restored training image may correspond to the current restored image (300) described above.

[0505] In a training process according to one embodiment of the present disclosure, neural networks can be trained so that the current reconstructed training image is as similar as possible to the current training image, and the bitrate of the bitstream generated through encoding the current training image is minimized. To this end, as illustrated in FIG. 25, first loss information (2580) and second loss information (2590) can be used to train the neural networks.

[0506] Specifically describing the training process of neural networks, first, a neural network-based encoding process (2510) is applied to the current training image, thereby obtaining first feature data.

[0507] The neural network-based encoding process (2510) may be a process of encoding the current training image based on an image encoder (12), an optical flow encoder (42), and / or a residual encoder (52).

[0508] The first feature data may include at least one of feature data obtained by processing the current training image by the image encoder (12), feature data obtained by processing the current training image and the previous reconstructed training image by the optical flow encoder (42), or feature data obtained by processing the residual training image corresponding to the difference between the current training image and the current prediction training image by the residual encoder (52). Here, the current prediction training image may be obtained by transforming the previous reconstructed training image according to the optical flow (g).

[0509] The first feature data can be input to the hyperfryer encoder (310). The hyperfryer encoder (310) can process the first feature data according to preset parameters and output the second feature data.

[0510] The second feature data can be input to a probabilistic neural network (330) and a quantization neural network (410). Each of the probabilistic neural network (330) and the quantization neural network (410) can process the second feature data according to preset parameters and output probability data and quantization data.

[0511] The quantization step (2010) can be selected from among a plurality of predetermined quantization steps.

[0512] The final quantization step values ​​can be obtained by multiplying the quantization data and the quantization step (2010) (2210).

[0513] Probability data can be modified through a modification process (2520) according to the final quantization step values. The modification process (2520) has been described in FIGS. 8 to 11, etc., so a detailed description thereof will be omitted.

[0514] A quantization process (2530) based on final quantization step values ​​may be applied to the first feature data to obtain quantized first feature data. In addition, a bitstream may be generated by applying an entropy encoding process (2540) based on modified probability data to the quantized first feature data. In the entropy encoding process (2540), a quantization index indicating a quantization step may also be included in the bitstream.

[0515] In one embodiment of the present disclosure, the bitstream may include bits corresponding to second feature data or quantized second feature data.

[0516] An entropy decoding process (2550) based on modified probability data is performed on the bitstream to obtain quantized first feature data, and an inverse quantization process (2560) according to final quantization step values ​​is performed on the quantized first feature data to obtain inverse quantized first feature data.

[0517] The current restored training image can be obtained by processing the dequantized first feature data according to a neural network-based decoding process (2570).

[0518] The neural network-based decoding process may be a process of restoring the current image (100) based on an image decoder (14), an optical flow decoder (44), and / or a residual decoder (54).

[0519] For training of a neural network, a hyper-priority encoder (310), a probabilistic neural network (330), a quantization neural network (410) used in a neural network-based encoding process and a neural network used in a neural network-based decoding process, at least one of the first loss information (2580) or the second loss information (2590) can be obtained.

[0520] The first loss information (2580) can be derived from the bitrate of the bitstream generated as an encoding result for the current training image.

[0521] Since the first loss information (2580) is related to the encoding efficiency for the current training image, the first loss information may be referred to as compression loss information.

[0522] The second loss information (2590) may correspond to a difference between the current training image and the current reconstructed training image. In one embodiment of the present disclosure, the difference between the current training image and the current reconstructed training image may include at least one of an L1-norm value, an L2-norm value, a Structural Similarity (SSIM) value, a Peak Signal-To-Noise Ratio-Human Vision System (PSNR-HVS) value, a Multiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF) value, or a Video Multimethod Assessment Fusion (VMAF) value between the current training image and the current reconstructed training image.

[0523] The second loss information (2590) is related to the quality of the current restoration training image, and therefore may be referred to as quality loss information.

[0524] The neural network, hyper-priority encoder (310), probabilistic neural network (330), quantization neural network (410) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process can be trained so that the final loss information derived from at least one of the first loss information (2580) or the second loss information (2590) is reduced or minimized.

[0525] In one embodiment of the present disclosure, the neural network, hyper-priority encoder (310), probabilistic neural network (330), quantization neural network (410) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process can reduce or minimize final loss information by changing the values ​​of preset parameters.

[0526] In one embodiment of the present disclosure, the final loss information can be calculated according to the following mathematical expression 5.

[0527] [Equation 5]

[0528] Final loss information = a*first loss information+b*second loss information

[0529] In mathematical expression 5, a and b are weights applied to the first loss information (2580) and the second loss information (2590), respectively.

[0530] According to mathematical expression 5, it can be seen that the neural network, hyper-priority encoder (310), probabilistic neural network (330), quantization neural network (410) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process are trained in a direction in which the current restoration training image becomes as similar to the current training image as possible and the size of the bitstream is minimized.

[0531] The training process described with reference to FIG. 25 can be performed by a training device. The training device may be, for example, an image encoding device (1800) or a separate server. The parameters obtained as a result of the training may be stored in the image encoding device (1800) and the image decoding device (1200).

[0532] An image decoding method according to one embodiment of the present disclosure may include the steps of: obtaining, from a bitstream, second feature data for first feature data obtained through neural network-based encoding of a current image and a quantization index indicating one quantization step among a plurality of quantization steps; obtaining the quantization step based on the quantization index; obtaining probability data by applying the second feature data to a first neural network; modifying the probability data based on the quantization step; obtaining quantized first feature data by applying entropy decoding based on the modified probability data to bits included in the bitstream; obtaining dequantized first feature data by dequantizing the quantized first feature data according to the quantization step; and restoring the current image through neural network-based decoding of the dequantized first feature data.

[0533] An image decoding method according to one embodiment of the present disclosure can resolve a mismatch between training data and test data that may occur in a neural network for image decoding that is trained or optimized for a large quantization error by using one quantization step among a plurality of quantization steps predetermined through RDO calculation.

[0534] In addition, the image decoding method according to one embodiment of the present disclosure can efficiently dequantize and entropy decode feature data generated through AI-based encoding of an image.

[0535] In addition, the image decoding method according to one embodiment of the present disclosure can reduce the bitrate of a bitstream and improve the quality of a restored image.

[0536] According to one embodiment of the present disclosure, the size of a quantization step indicated by a quantization index may be 1 or greater.

[0537] An image decoding method according to one embodiment of the present disclosure can resolve mismatch by using less precise quantization.

[0538] According to one embodiment of the present disclosure, a quantization index may indicate one quantization step among a plurality of quantization steps included in a predetermined quantization list.

[0539] An image decoding method according to one embodiment of the present disclosure can reduce the time required for RDO calculation by selecting one quantization step from among a plurality of quantization steps included in a predefined quantization list.

[0540] According to one embodiment of the present disclosure, an image decoding method further includes a step of applying second feature data to a second neural network to obtain quantized data; wherein the probability data is modified based on sample values ​​of the quantized data and the quantization step, and the quantized first feature data can be inversely quantized based on the sample values ​​of the quantized data and the quantization step.

[0541] According to one embodiment of the present disclosure, final quantization step values ​​are obtained by multiplying sample values ​​of quantized data by the quantization step, and the probability data can be modified based on the final quantization step values.

[0542] An image decoding method according to one embodiment of the present disclosure can improve the quality of a restored image by adaptively quantizing each sample of feature data while resolving mismatch.

[0543] According to one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent the probability of a value that a sample of the quantized first feature data may have.

[0544] According to one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent a mean and a standard deviation corresponding to a sample of the quantized first feature data.

[0545] According to one embodiment of the present disclosure, the probability of a value that a sample of the quantized first feature data can have can be derived by applying the mean and standard deviation indicated by the sample values ​​of the modified probability data to a predetermined probability model.

[0546] According to one embodiment of the present disclosure, the step of modifying the probability data may include the step of dividing sample values ​​of the probability data into the quantization steps.

[0547] According to one embodiment of the present disclosure, by applying the second feature data to the first neural network, a plurality of probability data and a plurality of weights are obtained, the plurality of probability data are modified based on the quantization step, and the plurality of modified probability data are combined according to the plurality of weights, so that the probability of a value that a sample of the quantized first feature data can have can be determined.

[0548] An image decoding method according to one embodiment of the present disclosure can perform entropy decoding more effectively by modifying probability data for feature data.

[0549] According to one embodiment of the present disclosure, the first feature data may include feature data (k) obtained by applying the current image (100) to an image encoder (12), feature data (w) obtained by applying the current image (100) and a previously restored image (200) to an optical flow encoder (42), or feature data (v) obtained by applying a residual image (r) corresponding to the current image (100) to a residual encoder (52).

[0550] An image decoding method according to one embodiment of the present disclosure can more efficiently restore a current image by decoding based on a neural network.

[0551] An image decoding device according to one embodiment of the present disclosure may include a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may obtain, from a bitstream, second feature data for first feature data obtained through neural network-based encoding of a current image and a quantization index indicating one quantization step among a plurality of quantization steps. The at least one processor may obtain the quantization step based on the quantization index. The at least one processor may obtain probability data by applying the second feature data to a first neural network. The at least one processor may modify the probability data based on the quantization step. The at least one processor may obtain quantized first feature data by applying entropy decoding based on the modified probability data to bits included in the bitstream. The at least one processor may dequantize the quantized first feature data according to the quantization step to obtain dequantized first feature data. The at least one processor can restore the current image through neural network-based decoding of the dequantized first feature data.

[0552] An image decoding device according to one embodiment of the present disclosure can resolve a mismatch between training data and test data that may occur in a neural network for image decoding that is trained or optimized for a large quantization error by using one quantization step among a plurality of quantization steps predetermined through RDO calculation.

[0553] In addition, an image decoding device according to one embodiment of the present disclosure can efficiently dequantize and entropy decode feature data generated through AI-based encoding of an image.

[0554] In addition, the image decoding device according to one embodiment of the present disclosure can reduce the bitrate of a bitstream and improve the quality of a restored image.

[0555] According to one embodiment of the present disclosure, the size of a quantization step indicated by a quantization index may be 1 or greater.

[0556] An image decoding device according to one embodiment of the present disclosure can resolve mismatch by using less precise quantization.

[0557] According to one embodiment of the present disclosure, a quantization index may indicate one quantization step among a plurality of quantization steps included in a predetermined quantization list.

[0558] An image decoding device according to one embodiment of the present disclosure can reduce the time required for RDO calculation by selecting one quantization step from among a plurality of quantization steps included in a predefined quantization list.

[0559] According to one embodiment of the present disclosure, the at least one processor of the image decoding device may apply second feature data to a second neural network to obtain quantized data. The probability data may be modified based on sample values ​​of the quantized data and the quantization step, and the quantized first feature data may be dequantized based on the sample values ​​of the quantized data and the quantization step.

[0560] According to one embodiment of the present disclosure, final quantization step values ​​are obtained by multiplying sample values ​​of quantized data by the quantization step, and the probability data can be modified based on the final quantization step values.

[0561] An image decoding device according to one embodiment of the present disclosure can improve the quality of a restored image by adaptively quantizing each sample of feature data while resolving mismatch.

[0562] According to one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent the probability of a value that a sample of the quantized first feature data may have.

[0563] According to one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent a mean and a standard deviation corresponding to a sample of the quantized first feature data.

[0564] According to one embodiment of the present disclosure, the probability of a value that a sample of the quantized first feature data can have can be derived by applying the mean and standard deviation indicated by the sample values ​​of the modified probability data to a predetermined probability model.

[0565] According to one embodiment of the present disclosure, modifying the probability data may include dividing sample values ​​of the probability data into the quantization steps.

[0566] According to one embodiment of the present disclosure, by applying the second feature data to the first neural network, a plurality of probability data and a plurality of weights are obtained, the plurality of probability data are modified based on the quantization step, and the plurality of modified probability data are combined according to the plurality of weights, so that the probability of a value that a sample of the quantized first feature data can have can be determined.

[0567] An image decoding device according to one embodiment of the present disclosure can perform entropy decoding more effectively by modifying probability data for feature data.

[0568] According to one embodiment of the present disclosure, the first feature data may include feature data (k) obtained by applying the current image (100) to an image encoder (12), feature data (w) obtained by applying the current image (100) and a previously restored image (200) to an optical flow encoder (42), or feature data (v) obtained by applying a residual image (r) corresponding to the current image (100) to a residual encoder (52).

[0569] An image decoding device according to one embodiment of the present disclosure can more efficiently restore a current image by decoding based on a neural network.

[0570] An image encoding method according to one embodiment of the present disclosure may include: a step of applying first feature data obtained through neural network-based encoding of a current image to a first neural network to obtain second feature data for the first feature data; a step of applying the second feature data to the second neural network to obtain probability data; a step of modifying the probability data based on one quantization step among a plurality of predetermined quantization steps; a step of quantizing the first feature data according to the quantization step to obtain quantized first feature data; and a step of applying entropy encoding based on the modified probability data for the quantized first feature data and applying entropy encoding to a quantization index corresponding to the quantization step to generate a bitstream including the quantized first feature data and bits corresponding to the quantization index.

[0571] According to one embodiment of the present disclosure, the bitstream may include bits corresponding to the second feature data.

[0572] An image encoding method according to one embodiment of the present disclosure can resolve a mismatch between training data and test data that may occur in a neural network for image encoding that is trained or optimized for a large quantization error by using one quantization step among a plurality of quantization steps predetermined through RDO calculation.

[0573] In addition, the image encoding method according to one embodiment of the present disclosure can efficiently dequantize and entropy encode feature data generated through AI-based encoding of an image.

[0574] In addition, the image encoding method according to one embodiment of the present disclosure can reduce the bit rate of a bitstream and improve the quality of a restored image.

[0575] According to one embodiment of the present disclosure, the size of a quantization step indicated by a quantization index may be 1 or greater.

[0576] An image encoding method according to one embodiment of the present disclosure can resolve mismatch by using less precise quantization.

[0577] According to one embodiment of the present disclosure, a quantization index may indicate one quantization step among a plurality of quantization steps included in a predetermined quantization list.

[0578] An image encoding method according to one embodiment of the present disclosure can reduce the time required for RDO calculation by selecting one quantization step from among a plurality of quantization steps included in a predetermined quantization list.

[0579] According to one embodiment of the present disclosure, an image encoding method further includes a step of applying second feature data to a third neural network to obtain quantized data; wherein the probability data is modified based on sample values ​​of the quantized data and the quantization step, and the first feature data can be quantized based on sample values ​​of the quantized data and the quantization step.

[0580] According to one embodiment of the present disclosure, final quantization step values ​​are obtained by multiplying sample values ​​of quantized data by the quantization step, and the probability data can be modified based on the final quantization step values.

[0581] An image encoding method according to one embodiment of the present disclosure can improve the quality of a restored image by adaptively quantizing each sample of feature data while resolving mismatch.

[0582] According to one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent the probability of a value that a sample of the quantized first feature data may have.

[0583] According to one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent a mean and a standard deviation corresponding to a sample of the quantized first feature data.

[0584] According to one embodiment of the present disclosure, the probability of a value that a sample of the quantized first feature data can have can be derived by applying the mean and standard deviation indicated by the sample values ​​of the modified probability data to a predetermined probability model.

[0585] According to one embodiment of the present disclosure, the step of modifying the probability data may include the step of dividing sample values ​​of the probability data into the quantization steps.

[0586] According to one embodiment of the present disclosure, by applying the second feature data to the second neural network, a plurality of probability data and a plurality of weights are obtained, the plurality of probability data are modified based on the quantization step, and the plurality of modified probability data are combined according to the plurality of weights, so that the probability of a value that a sample of the quantized first feature data can have can be determined.

[0587] An image encoding method according to one embodiment of the present disclosure can perform entropy encoding more effectively by modifying probability data for feature data.

[0588] According to one embodiment of the present disclosure, the first feature data may include feature data (k) obtained by applying the current image (100) to an image encoder (12), feature data (w) obtained by applying the current image (100) and a previously restored image (200) to an optical flow encoder (42), or feature data (v) obtained by applying a residual image (r) corresponding to the current image (100) to a residual encoder (52).

[0589] An image encoding method according to one embodiment of the present disclosure can more efficiently restore a current image by encoding it based on a neural network.

[0590] An image encoding device according to one embodiment of the present disclosure may include a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may apply first feature data obtained through neural network-based encoding of a current image to a first neural network to obtain second feature data for the first feature data. The at least one processor may apply the second feature data to the second neural network to obtain probability data. The at least one processor may modify the probability data based on one quantization step among a plurality of predetermined quantization steps. The at least one processor may quantize the first feature data according to the quantization step to obtain quantized first feature data. The at least one processor may apply entropy encoding based on the modified probability data for the quantized first feature data and apply entropy encoding to a quantization index corresponding to the quantization step to generate a bitstream including the quantized first feature data and bits corresponding to the quantization index.

[0591] According to one embodiment of the present disclosure, the bitstream may include bits corresponding to the second feature data.

[0592] An image encoding device according to one embodiment of the present disclosure can resolve a mismatch between training data and test data that may occur in a neural network for image encoding that is trained or optimized for a large quantization error by using one quantization step among a plurality of quantization steps predetermined through RDO calculation.

[0593] In addition, an image encoding device according to one embodiment of the present disclosure can efficiently dequantize and entropy encode feature data generated through AI-based encoding of an image.

[0594] In addition, the image encoding device according to one embodiment of the present disclosure can reduce the bit rate of a bitstream and improve the quality of a restored image.

[0595] According to one embodiment of the present disclosure, the size of a quantization step indicated by a quantization index may be 1 or greater.

[0596] An image encoding device according to one embodiment of the present disclosure can resolve mismatch by using less precise quantization.

[0597] According to one embodiment of the present disclosure, a quantization index may represent one quantization step among a plurality of quantization steps included in a predetermined quantization list.

[0598] An image encoding device according to one embodiment of the present disclosure can reduce the time required for RDO calculation by selecting one quantization step from among a plurality of quantization steps included in a predetermined quantization list.

[0599] According to one embodiment of the present disclosure, the at least one processor of the image encoding device may obtain quantized data by applying the second feature data to a third neural network. The probability data may be modified based on sample values ​​of the quantized data and the quantization step, and the first feature data may be quantized based on the sample values ​​of the quantized data and the quantization step.

[0600] According to one embodiment of the present disclosure, final quantization step values ​​are obtained by multiplying sample values ​​of quantized data by the quantization step, and the probability data can be modified based on the final quantization step values.

[0601] An image encoding device according to one embodiment of the present disclosure can improve the quality of a restored image by adaptively quantizing each sample of feature data while resolving mismatch.

[0602] According to one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent the probability of a value that a sample of the quantized first feature data may have.

[0603] According to one embodiment of the present disclosure, the sample values ​​of the modified probability data may represent a mean and a standard deviation corresponding to a sample of the quantized first feature data.

[0604] According to one embodiment of the present disclosure, the probability of a value that a sample of the quantized first feature data can have can be derived by applying the mean and standard deviation indicated by the sample values ​​of the modified probability data to a predetermined probability model.

[0605] According to one embodiment of the present disclosure, modifying the probability data may include dividing sample values ​​of the probability data into the quantization steps.

[0606] According to one embodiment of the present disclosure, by applying the second feature data to the first neural network, a plurality of probability data and a plurality of weights are obtained, the plurality of probability data are modified based on the quantization step, and the plurality of modified probability data are combined according to the plurality of weights, so that the probability of a value that a sample of the quantized first feature data can have can be determined.

[0607] An image encoding device according to one embodiment of the present disclosure can perform entropy encoding more effectively by modifying probability data for feature data.

[0608] According to one embodiment of the present disclosure, the first feature data may include feature data (k) obtained by applying the current image (100) to an image encoder (12), feature data (w) obtained by applying the current image (100) and a previously restored image (200) to an optical flow encoder (42), or feature data (v) obtained by applying a residual image (r) corresponding to the current image (100) to a residual encoder (52).

[0609] An image encoding device according to one embodiment of the present disclosure can more efficiently restore a current image by encoding it based on a neural network.

[0610] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0611] According to one embodiment of the present disclosure, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. A step of obtaining second feature data for first feature data obtained through neural network-based encoding for a current image (100) and a quantization index indicating one quantization step among a plurality of quantization steps from a bitstream (S1710); A step of obtaining the quantization step based on the quantization index (S1720); A step of applying the above second feature data to the first neural network (330) to obtain probability data (S1730); A step of modifying the probability data (S1740) based on the above quantization step; A step of obtaining quantized first feature data by applying entropy decoding based on the modified probability data to bits included in the bitstream (S1750); A step of obtaining dequantized first feature data by dequantizing the quantized first feature data according to the above quantization step (S1760); and An image decoding method, comprising a step of restoring the current image (S1770) through neural network-based decoding of the dequantized first feature data.

2. In paragraph 1, An image decoding method, wherein the size of the quantization step indicated by the quantization index is 1 or greater.

3. In paragraph 1 or 2, An image decoding method, wherein the above quantization index indicates one quantization step among a plurality of quantization steps included in a predefined quantization list.

4. In any one of paragraphs 1 to 3, Further comprising a step of applying the above second feature data to a second neural network (410) to obtain quantized data; The above probability data is modified based on the sample values ​​of the quantized data and the quantization step, An image decoding method, wherein the quantized first feature data is inversely quantized based on sample values ​​of the quantized data and the quantization step.

5. In paragraph 4, The final quantization step values ​​are obtained by multiplying the sample values ​​of the above quantized data and the above quantization step, An image decoding method, wherein the probability data is modified based on the final quantization step values.

6. In paragraph 4, The above quantized data is, An image decoding method comprising a quantization parameter or a quantization step size as a sample.

7. In any one of paragraphs 1 to 6, The sample values ​​of the above modified probability data are, An image decoding method, which represents the probability of a value that a sample of the above quantized first feature data can have.

8. In any one of paragraphs 1 to 7, The sample values ​​of the above modified probability data are, An image decoding method, wherein the mean and standard deviation corresponding to samples of the above quantized first feature data are represented.

9. In any one of paragraphs 1 to 8, An image decoding method, wherein the probability of a value that a sample of the quantized first feature data can have is derived by applying the average and standard deviation indicated by the sample values ​​of the modified probability data to a predetermined probability model.

10. In any one of paragraphs 1 to 9, The steps for modifying the above probability data are: An image decoding method, comprising a step of dividing sample values ​​of the above probability data into the above quantization steps.

11. In any one of paragraphs 1 to 10, The above first feature data is, An image decoding method, comprising feature data (k) obtained by applying the current image (100) to an image encoder (12), feature data (w) obtained by applying the current image (100) and a previous restored image (200) to an optical flow encoder (42), or feature data (v) obtained by applying a residual image (r) corresponding to the current image (100) to a residual encoder (52).

12. In any one of paragraphs 1 to 11, By applying the above second feature data to the above first neural network (330), multiple probability data and multiple weights are obtained. Based on the above quantization step, the plurality of probability data are modified, An image decoding method, wherein the plurality of modified probability data are combined according to the plurality of weights to determine the probability of a value that a sample of the quantized first feature data can have.

13. A step of applying the first feature data obtained through neural network-based encoding for the current image (100) to the first neural network (310) to obtain second feature data for the first feature data (S2310); A step of applying the above second feature data to a second neural network (330) to obtain probability data (S2320); A step of modifying the probability data based on one of a plurality of predetermined quantization steps (S2330); A step of obtaining quantized first feature data by quantizing the first feature data according to the above quantization step (S2340); and A step of applying entropy encoding based on the modified probability data for the quantized first feature data and applying entropy encoding to a quantization index corresponding to the quantization step to generate a bitstream including bits corresponding to the quantized first feature data and the quantization index (S2350), A method for encoding an image, wherein the bitstream includes bits corresponding to the second feature data.

14. In paragraph 13, An image encoding method, wherein the size of the quantization step indicated by the quantization index is 1 or greater.

15. In paragraph 13 or 14, A method for encoding an image, wherein the above quantization index indicates one quantization step among a plurality of quantization steps included in a predetermined quantization list.

Images