Image decoding apparatus and method, and image encoding apparatus and method

Abstract

Proposed are an image decoding method and apparatus which: acquire first and second sub feature maps from among a plurality of sub feature maps obtained by rearranging the current feature map; acquire first and second latent feature maps obtained through neural network-based encoding on the first and second sub feature maps; acquire initial first probability information for entropy coding of the first sub feature map on the basis of the first latent feature map; decode the first sub feature map by using the initial first probability information; acquire initial second probability information for entropy coding of the second sub feature map on the basis of the second latent feature map; acquire modified second probability information for entropy coding of the second sub feature map by using the initial first probability information, the decoded first sub feature map, and the initial second probability information; decode the second sub feature map by using the modified second probability information; and reconstruct the current image by using the decoded first and second sub feature maps.

Description

Image decoding device and method, and image encoding device and method

[0001] The present disclosure relates to image encoding and decoding. More specifically, the present disclosure relates to a technique for encoding and decoding images using artificial intelligence (AI), such as a neural network.

[0002] In codecs such as H.264 AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding), an image can be divided into blocks, and each block can be predicted and decoded through inter prediction or intra prediction.

[0003] Intra prediction is a method of compressing images by removing spatial redundancy within the image, and inter prediction is a method of compressing images by removing temporal redundancy between images.

[0004] A representative example of inter prediction is motion estimation coding. Motion estimation coding uses a reference image to predict blocks in the current image. Using a predetermined evaluation function, the reference block most similar to the current block can be searched within a predetermined range. The current block is predicted based on the reference block, and the predicted block generated as a result of the prediction is subtracted from the current block to generate and encode a residual block.

[0005] To derive a motion vector pointing to a reference block within a reference image, the motion vectors of previously encoded blocks can be used as motion vector predictors of the current block. The differential motion vector, which is the difference between the motion vector of the current block and the motion vector predictor, is signaled to the decoder side through a predetermined method.

[0006] Recently, technologies for encoding / decoding images using AI (Artificial Intelligence) have been proposed, and a method for effectively encoding / decoding images using AI, such as neural networks, is required.

[0007] According to one embodiment of the present disclosure, an image decoding method comprises the steps of: obtaining a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps obtained by rearranging a current feature map corresponding to a current image; obtaining a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map; obtaining initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map; decoding the first sub-feature map using the initial first probability information; obtaining initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map; obtaining modified second probability information for entropy coding of the second sub-feature map using the initial first probability information, the decoded first sub-feature map, and the initial second probability information; decoding the second sub-feature map using the modified second probability information; It may include a step of restoring the current image using the decrypted first sub-feature map and the decrypted second sub-feature map.

[0008] An image decoding device according to one embodiment of the present disclosure may include a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may obtain a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps that rearrange a current feature map corresponding to a current image. The at least one processor may obtain a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map. The at least one processor may obtain initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map. The at least one processor may decode the first sub-feature map using the initial first probability information. The at least one processor may obtain initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map. At least one processor can obtain modified second probability information for entropy coding of a second sub-feature map using the initial first probability information, the decoded first sub-feature map, and the initial second probability information. At least one processor can decode the second sub-feature map using the modified second probability information. At least one processor can reconstruct the current image using the decoded first sub-feature map and the decoded second sub-feature map.

[0009] According to one embodiment of the present disclosure, an image encoding method comprises the steps of: obtaining a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps that rearrange a current feature map corresponding to a current image; obtaining a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map; obtaining initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map; encoding the first sub-feature map using the initial first probability information; obtaining initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map; obtaining modified second probability information for entropy coding of the second sub-feature map using the initial first probability information, the encoded first sub-feature map, and the initial second probability information; encoding the second sub-feature map using the modified second probability information; It may include a step of encoding the current feature map using the encoded first sub-feature map and the encoded second sub-feature map.

[0010] An image encoding device according to one embodiment of the present disclosure may include a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may obtain a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps that are rearranged from a current feature map corresponding to a current image. The at least one processor may obtain a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map. The at least one processor may obtain initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map. The at least one processor may encode the first sub-feature map using the initial first probability information. The at least one processor may obtain initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map. At least one processor can obtain modified second probability information for entropy coding of a second sub-feature map using the initial first probability information, the encoded first sub-feature map, and the initial second probability information. The at least one processor can encode the second sub-feature map using the modified second probability information. The at least one processor can encode the current feature map using the encoded first sub-feature map and the encoded second sub-feature map.

[0011] FIG. 1 is a diagram for explaining a process of encoding and decoding a current image based on intra prediction according to one embodiment of the present disclosure.

[0012] FIG. 2 is a diagram for explaining a process of encoding and decoding a current image based on inter prediction according to one embodiment of the present disclosure.

[0013] FIG. 3 is a diagram for explaining a method for obtaining probability information used in entropy encoding and entropy decoding according to one embodiment of the present disclosure.

[0014] FIG. 4 is a diagram illustrating a method for obtaining modified probability information used in entropy encoding and entropy decoding according to one embodiment of the present disclosure.

[0015] FIG. 5 is a diagram illustrating initial probability information, modified probability information, and a probability model according to one embodiment of the present disclosure.

[0016] FIG. 6 is a diagram illustrating progressive entropy coding using modified probability information according to one embodiment of the present disclosure.

[0017] FIG. 7 is a diagram illustrating the structure of a neural network according to one embodiment of the present disclosure.

[0018] FIG. 8 is a diagram for explaining a method for modifying probability information according to one embodiment of the present disclosure.

[0019] FIG. 9 is a diagram illustrating a configuration of an image decoding device according to one embodiment of the present disclosure.

[0020] FIG. 10 is a diagram illustrating the configuration of an acquisition unit according to one embodiment of the present disclosure.

[0021] FIG. 11 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0022] FIG. 12 is a diagram for explaining an image decoding method according to one embodiment of the present disclosure.

[0023] FIG. 13 is a diagram illustrating a configuration of an image encoding device according to one embodiment of the present disclosure.

[0024] FIG. 14 is a diagram illustrating the configuration of a generation unit according to one embodiment of the present disclosure.

[0025] FIG. 15 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0026] FIG. 16 is a drawing for explaining an image encoding method according to one embodiment of the present disclosure.

[0027] FIG. 17 is a diagram for explaining a method for training neural networks according to one embodiment of the present disclosure.

[0028] The present disclosure may be subject to various modifications and various embodiments. Specific embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments of the present disclosure, and the present disclosure may include all modifications, equivalents, and alternatives that fall within the spirit and technical scope of the various embodiments.

[0029] When describing embodiments, detailed descriptions of related known technologies may be omitted if they are deemed to unnecessarily obscure the gist of the present disclosure. Furthermore, numbers (e.g., "first," "second," etc.) used in the description of the specification serve as identifiers to distinguish one component from another.

[0030] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0031] In the present disclosure, when a component is referred to as being “connected” or “connected” to another component, the component may be directly connected or connected to the other component, but unless there is a specific description to the contrary, the component may also be connected or connected via another component in between.

[0032] In this disclosure, components expressed as "units", "modules", etc. may be two or more components combined into one component, or one component may be divided into two or more components with more detailed functions. Each component described below may additionally perform some or all of the functions performed by other components in addition to its own main function, and some of the main functions performed by each component may be exclusively performed by other components.

[0033] According to one embodiment of the present disclosure, a "unit" may be implemented as a processor and a memory. The term "processor" should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some circumstances, a "processor" may also refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), and the like. The term "processor" may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0034] A processor may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to perform various functions described herein, individually and / or collectively, in a distributed fashion. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0035] The term "memory" should be interpreted broadly to include any electronic component capable of storing electronic information. The term memory may also refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. Memory is said to be in electronic communication with the processor if the processor can read information from and / or write information to the memory. Memory integrated in a processor is in electronic communication with the processor.

[0036] In the present disclosure, 'image' may refer to a still image, a picture, a frame, a moving image composed of a plurality of consecutive still images, or a video.

[0037] In this disclosure, "neural network" refers to a representative example of an artificial neural network model that mimics brain neurons, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

[0038] In this disclosure, a "parameter" refers to a value used in the computational process of each layer forming a neural network. For example, it can be used when applying an input value to a given computational formula. A parameter is a value set as a result of training and can be updated using separate training data as needed.

[0039] In the present disclosure, "feature map" or "feature data" may refer to data obtained by a neural network or a neural network-based encoder processing input data. The feature data may be one-dimensional or two-dimensional data containing multiple samples. The feature map may be two-dimensional data containing multiple samples. The feature map or feature data may also be referred to as a latent representation. The feature map or feature data may represent latent features in data output by a neural network-based decoder.

[0040] In this disclosure, the term "current image" refers to the image currently being processed, and the term "previous image" refers to the image being processed prior to the current image. The "current image" or "previous image" may be a block segmented from the current image or the previous image.

[0041] In this disclosure, a "sample" refers to data assigned to a sampling location within one-dimensional or two-dimensional data, such as an image, feature map, feature data, or probability information, and is the data to be processed. For example, a sample may include pixels within a two-dimensional image. Two-dimensional data may also be referred to as a "map."

[0042] An AI-based end-to-end encoding / decoding system can be understood as a system that utilizes a neural network in the encoding and decoding processes of an image.

[0043] In AI-based end-to-end encoding / decoding systems, such as codecs such as HEVC and VVC, intra prediction or inter prediction can be used to encode and decode images.

[0044] As mentioned above, intra prediction is a method of compressing an image by removing spatial redundancy within the image, and inter prediction is a method of compressing an image by removing temporal redundancy between images.

[0045] In one embodiment of the present disclosure, intra prediction may be applied to a first frame among several frames, a frame that is a random access point, and a frame in which a scene change occurs.

[0046] In one embodiment of the present disclosure, inter prediction may be applied to frames subsequent to a frame to which intra prediction is applied among multiple frames.

[0047] Referring to FIGS. 1 and 2, intra prediction and inter prediction performed by an AI-based end-to-end encoding / decoding system according to one embodiment of the present disclosure are described.

[0048] FIG. 1 is a diagram for explaining a process of encoding and decoding a current image based on intra prediction according to one embodiment of the present disclosure.

[0049] In intra prediction, a video encoder (12) and a video decoder (14) can be used. The video encoder (12) and the video decoder (14) can be implemented as a neural network.

[0050] The image encoder (12) can process the current image (100) according to parameters set through training and output feature data (k) for the current image (100).

[0051] A bitstream is generated by applying quantization (22) and entropy encoding (32) to the feature data (k) of the current image (100), and the bitstream can be transmitted from an image encoding device to an image decoding device.

[0052] Entropy decoding (34) and inverse quantization (24) are applied to the bitstream to obtain restored feature data (k'), and the restored feature data (k') can be input to the image decoder (14).

[0053] The image decoder (14) can process feature data (k') according to parameters set through training and output the current restored image (300).

[0054] In intra prediction, spatial features within the current image (100) are considered, so unlike inter prediction illustrated in FIG. 2, only the current image (100) can be input to the image encoder (12).

[0055] FIG. 2 is a diagram for explaining a process of encoding and decoding a current image based on inter prediction according to one embodiment of the present disclosure.

[0056] In inter prediction, an optical flow encoder (42), an optical flow decoder (44), a residual encoder (52), and a residual decoder (54) can be used.

[0057] The optical flow encoder (42), optical flow decoder (44), residual encoder (52), and residual decoder (54) can be implemented as a neural network.

[0058] The optical flow encoder (42) and the optical flow decoder (44) can be understood as a neural network for extracting optical flow (g) from the current image (100) and the previously restored image (200).

[0059] The residual encoder (52) and the residual decoder (54) can be understood as neural networks for encoding and decoding the residual image (r).

[0060] As described above, inter prediction is a process of encoding and decoding a current image (100) by utilizing the temporal redundancy between the current image (100) and a previously restored image (200). The previously restored image (200) may be an image obtained through decoding a previous image that was the subject of processing prior to processing the current image (100).

[0061] The positional differences (or motion vectors) between blocks or samples in the current image (100) and reference blocks or reference samples in the previously restored image (200) can be used for encoding and decoding the current image (100). These positional differences can be referred to as optical flow. Optical flow can also be defined as a set of motion vectors corresponding to samples or blocks in the image.

[0062] Optical flow (g) can indicate how the positions of samples in the previous restored image (200) have changed in the current image (100), or where samples identical / similar to the samples in the current image (100) are located in the previous restored image (200).

[0063] For example, if a sample that is identical to or most similar to a sample located at (1, 1) in the current image (100) is located at (2, 1) in the previous restored image (200), the optical flow (g) or motion vector for the sample can be derived as (1(=2-1), 0(=1-1)).

[0064] For encoding the current image (100), the previous restored image (200) and the current image (100) can be input to the optical flow encoder (42).

[0065] The optical flow encoder (42) can process the current image (100) and the previously restored image (200) according to the parameters set as a result of training to output feature data (w) of the optical flow (g).

[0066] As described in Fig. 1, quantization (22) and entropy encoding (32) are applied to the feature data (w) of the optical flow (g) to generate a bitstream, and entropy decoding (34) and inverse quantization (24) are applied to the bitstream to restore the feature data (w) of the optical flow (g).

[0067] The feature data (w) of the optical flow (g) can be input to the optical flow decoder (44). The optical flow decoder (44) can process the input feature data (w) according to the parameters set as a result of training and output the optical flow (g).

[0068] The previous restored image (200) is warped through warping (60) based on optical flow (g), and a current predicted image (x') can be obtained as a result of the warping (60). Warping (60) is a type of geometric deformation that moves the positions of samples within an image.

[0069] A current predicted image (x') similar to the current image (100) can be obtained by applying warping (60) to the previous restored image (200) according to the optical flow (g) indicating the relative positional relationship between samples in the previous restored image (200) and samples in the current image (100).

[0070] For example, if the sample located at (1, 1) in the previous restored image (200) is most similar to the sample located at (2, 1) in the current image (100), the location of the sample located at (1, 1) in the previous restored image (200) can be changed to (2, 1) through warping (60).

[0071] Since the current predicted image (x') generated from the previous restored image (200) is not the current image (100) itself, a residual image (r) between the current predicted image (x') and the current image (100) can be obtained.

[0072] For example, a residual image (r) can be obtained by subtracting sample values ​​in the current predicted image (x') from sample values ​​in the current image (100).

[0073] The residual image (r) can be input to a residual encoder (52). The residual encoder (52) can process the residual image (r) according to parameters set as a result of training and output feature data (v) of the residual image (r).

[0074] As described in Fig. 1, quantization (22) and entropy encoding (32) are applied to the feature data (v) of the residual image (r) to generate a bitstream, and entropy decoding (34) and inverse quantization (24) are applied to the bitstream to restore the feature data (v) of the residual image (r).

[0075] The feature data (v) of the residual image (r) can be input to the residual decoder (54). The residual decoder (54) can process the input feature data (v) according to the parameters set as a result of training and output the restored residual image (r').

[0076] The current restored image (300) can be obtained by combining the current predicted image (x') and the restored residual image (r').

[0077] Meanwhile, as described above, entropy encoding (32) and entropy decoding (34) may be applied to the feature data (k) of the current image (100), the feature data (w) of the optical flow (g), and the feature data (v) of the residual image (r). Since entropy coding is a encoding method that varies the average length of a code representing a symbol according to the probability of the symbol, the probabilities of values ​​that samples of the current feature map may have may be required in entropy encoding (32) and entropy decoding (34) the current feature map.

[0078] In one embodiment of the present disclosure, probability information can be acquired based on a neural network to improve the efficiency of entropy encoding (32) / entropy decoding (34) for at least one of feature data (k) of a current image (100), feature data (w) of an optical flow (g), or feature data (v) of a residual image (r) (hereinafter, current feature map).

[0079] Probability information is one-dimensional or two-dimensional data, and a sample of probability information can represent the probability of the value that the sample of the current feature map can have.

[0080] In one embodiment of the present disclosure, by applying sample values ​​of probability information to a predetermined probability model (e.g., a Laplacian probability model or a Gaussian probability model, etc.), probabilities of values ​​that samples of the current feature map can have can be derived.

[0081] In one embodiment of the present disclosure, the probability information may include the mean and standard deviation (or variance) corresponding to the samples of the current feature map as sample values.

[0082] A method for obtaining probability information using a neural network is described with reference to Fig. 3.

[0083] FIG. 3 is a diagram for explaining a method for obtaining probability information used in entropy encoding and entropy decoding according to one embodiment of the present disclosure.

[0084] A hyperprior encoder (310) and a probabilistic neural network (330) can be used to obtain probability information used in entropy encoding (32) / entropy decoding (34).

[0085] The hyper-primer encoder (310) may be a neural network for obtaining a feature map from a feature map.

[0086] Referring to FIG. 3, the current feature map is input to the hyperfriar encoder (310), and the hyperfriar encoder (310) can process the current feature map according to the parameters set as a result of training and output a latent feature map.

[0087] The latent feature map can represent the latent features in the current feature map, and thus can be referred to as a hyperpriority feature map.

[0088] The previous feature map may be a feature map decoded before the current feature map. Additionally, the previous feature map may be a feature map neighboring the current feature map. For example, the previous feature map may be a feature map spatially neighboring the current feature map.

[0089] The latent feature map and the previous feature map are input to a probability neural network (330), and the probability neural network (330) can process the latent feature map and the previous feature map according to the parameters set as a result of training to output probability information of the current feature map.

[0090] Probability information can be used for entropy encoding (32) and entropy decoding (34) described with reference to FIGS. 1 and 2.

[0091] In one embodiment of the present disclosure, in the encoding process of the current image (100), a bitstream can be obtained by applying entropy encoding (32) based on probability information to at least one of the current feature map, for example, the feature map of the current image (100), the feature map of the optical flow (g), or the feature map of the residual image (r).

[0092] In one embodiment of the present disclosure, in the decoding process of the current image (100), entropy decoding (34) based on probability information is applied to bits included in the bitstream, so that at least one of the current feature map, for example, the feature map of the current image (100), the quantized feature map of the optical flow (g), or the feature map of the residual image (r) can be obtained.

[0093] The probability information obtained through the probability neural network (330) can be expressed as in the mathematical expression 1 below.

[0094] [Mathematical Formula 1]

[0095]

[0096] Here, CMF is an abbreviation for Cumulative Mass Function and means probability information, NeuralNetwork means probability neural network, decodedHyperPrior means decoded latent feature map, and previousDecodedLatent means decoded previous feature map.

[0097] The accuracy of the probability information for entropy encoding and decoding may vary due to reasons such as constraints on the bitrate for the latent feature map or overfitting of a probabilistic neural network trained on a particular dataset.

[0098] To address this, it may be better to modify the probability information if necessary through Rate Distortion Optimization (RDO) on the encoding side.

[0099] Below, a method for modifying probability information is described in FIG. 4.

[0100] FIG. 4 is a diagram illustrating a method for obtaining modified probability information used in entropy encoding and entropy decoding according to one embodiment of the present disclosure.

[0101] Referring to FIG. 4, the current feature map is input to the hyperfriar encoder (310), and the hyperfriar encoder (310) can process the current feature map according to the parameters set as a result of training and output a latent feature map.

[0102] The latent feature map and the previous feature map can be input into a probabilistic neural network (330). The probabilistic neural network (330) can process the latent feature map and the previous feature map according to the parameters set as a result of training to output initial probability information of the current feature map.

[0103] To assess the accuracy of probability information estimation, probability information from feature maps neighboring the current feature map, such as previous feature maps decoded prior to the current feature map, can be utilized. Specifically, the accuracy of the previous feature map can be estimated using the previous feature map and its probability information, and the accuracy of the previous feature map can be applied to the current feature map or not. Whether or not to apply the accuracy can be determined based on the RDO calculation.

[0104] Referring again to FIG. 4, the probability information of the current feature map can be modified (430) based on the initial probability information of the current feature map, the previous feature map, and the initial probability information of the previous feature map, thereby obtaining the modified probability information of the current feature map.

[0105] Specifically, the previous feature map and the initial probability information of the previous feature map are used to modify the probability information (430). As shown in Equation 2 below, the scaling variable can be estimated from the estimated accuracy of the decoded previous feature map.

[0106] [Equation 2]

[0107]

[0108] Here, α denotes a scaling variable, σpreviousDecodedLatent denotes the standard deviation of the previous feature map, and σ initial refers to the initial probability information of the previous feature map.

[0109] The scaling variable is a factor used for correction and can therefore be referred to as a correction factor.

[0110] σpreviousDecodedLatent is the original energy change value of the previous feature map, and σ initial Since is the predicted energy change value of the previous feature map, the scaling variable indicates whether the energy of the component of the latent feature map is overestimated, underestimated, or well estimated.

[0111] That is, if the scaling variable is close to 1 (α 1), the probability information of the previous feature map is well estimated, and if the scaling variable is less than 1 (α<1), that is, if the initial variance or initial standard deviation of the previous feature map is greater than the actual variance or standard deviation of the previous feature map, the probability information of the previous feature map is incorrectly overestimated, and if the scaling variable is greater than 1 (α>1), that is, if the initial variance or initial standard deviation of the previous feature map is less than the actual variance or standard deviation of the previous feature map, the probability information of the previous feature map is incorrectly underestimated.

[0112] These measurements may share the same correction factor for all components of the feature map, i.e. all samples, or may share the same correction factor for all samples of the same channel of the feature map, or each sample of the feature map may have its own correction factor.

[0113] These scaling variables can be applied to either Gaussian or Laplacian models.

[0114] The modification (430) process can be expressed as in mathematical formula 3 below.

[0115] [Equation 3]

[0116]

[0117] Here, μ initial , σ initial means the initial probability information of the current feature map which is the target of entropy encoding and decoding, previousDecodedLatent means the previous feature map, previousInitialCMF means the initial probability information of the previous feature map, and μ corrected , σ corrected means the corrected probability information of the current feature map, i.e., the mean and standard deviation, and Correction means the correction process.

[0118] By adjusting the variance estimation, i.e. the standard deviation estimation, using the scaling variable, the probability information is modified as in Equation 4 below.

[0119] [Equation 4]

[0120]

[0121] It can also be applied to a mixture model of Gaussian and Laplacian models. In this case, the scaling parameter estimated for the previous feature map can be used as shown in Equation 5 below.

[0122] [Equation 5]

[0123]

[0124] Here, m=1, ..., M, and M is the number of models used in the mixture model.

[0125] The modified probability information obtained through this process can be used for entropy encoding (32) for the current feature map and entropy decoding (34) for the bitstream.

[0126] The method of modifying probability information is described later with reference to Fig. 8.

[0127] In the embodiment described with reference to FIG. 4, the initial probability information of the current feature map is obtained from a latent feature map corresponding to a latent feature of the current feature map that is the target of quantization (22) and entropy encoding (32), and the probability information is modified according to the initial probability information of the current feature map, the previous feature map, and the initial probability information of the previous feature map, so that the accuracy of the probability information is improved and the current image can be efficiently restored.

[0128] An exemplary structure of the hyperfryer encoder (310) and the probabilistic neural network (330) described above in FIG. 4 is described with reference to FIG. 7.

[0129] In one embodiment of the present disclosure, the modification (430) process may also be performed based on a neural network. An exemplary structure of a neural network for the modification (430) process is also described with reference to FIG. 7.

[0130] FIG. 5 is a diagram illustrating initial probability information, modified probability information, and a probability model according to one embodiment of the present disclosure.

[0131] Referring to FIG. 5, the initial probability information may include a mean (μ) and a standard deviation (σ), and the mean (μ) and the standard deviation (σ) of the initial probability information may be modified according to a scaling variable (α), and the modified probability information may include a modified mean (μ') and a modified standard deviation (σ').

[0132] By applying the modified mean (μ') and modified standard deviation (σ') to a predetermined probability model, the probability of the values ​​that a sample of the current feature map can have can be determined.

[0133] Referring to Fig. 5, a Laplacian probability model or a Gaussian probability model can be used as a predetermined probability model.

[0134] The Laplacian probability model or Gaussian probability model illustrated in FIG. 5 is an example. In one embodiment of the present disclosure, the types of probability models used for entropy encoding and entropy decoding may vary.

[0135] The probabilistic model to be used for entropy encoding and decoding of the current feature map may be predetermined. For example, the type of probabilistic model used for entropy encoding may be predetermined by the image encoding device (1300).

[0136] In one embodiment of the present disclosure, the type of probability model used for entropy encoding can be independently determined for each image or each block included in the image.

[0137] When the Laplacian model is used for entropy encoding, the probability that a sample of the current feature map can have can be derived by applying the modified mean (μ') and modified standard deviation (σ') to the Laplacian probability model.

[0138] Additionally, when a Gaussian model is used for entropy encoding, the probability that a sample of both current feature maps can have can be derived by applying the modified mean (μ') and modified standard deviation (σ') to the Gaussian probability model.

[0139] FIG. 6 is a diagram illustrating progressive entropy coding using modified probability information according to one embodiment of the present disclosure.

[0140] Referring to FIG. 6, the current feature map (600) for the current image is divided into 2x2 sample group units. In each sample group, the samples of A located at the upper left are collected into one sub-feature map to obtain a first sub-feature map (610). In each sample group, the samples of B located at the upper right are collected into one sub-feature map to obtain a second sub-feature map (620). In each sample group, the samples of C located at the lower left are collected into one sub-feature map to obtain a third sub-feature map (630). In each sample group, the samples of D located at the lower right are collected into one sub-feature map to obtain a fourth sub-feature map (640). Accordingly, the feature map of size 8x8 is divided into four sub-feature maps of size 4x4.

[0141] Performing entropy encoding and decoding on a per-component basis can cause significant delay problems because it creates spatial dependencies.

[0142] To reduce this delay, the feature map can be divided into a plurality of sub-feature maps, and the encoded or decoded sub-feature maps can be used as references while encoding or decoding the sub-feature maps consecutively. Specifically, if the encoding or decoding order of the plurality of sub-feature maps is in the order of the first sub-feature map (610), the second sub-feature map (620), the third sub-feature map (630), and the fourth sub-feature map (640), the first sub-feature map (610) can be used as a reference for the second sub-feature map (620), the third sub-feature map (630), and the fourth sub-feature map (640), the second sub-feature map (620) can be used as a reference for the third sub-feature map (630) and the fourth sub-feature map (640), and the third sub-feature map (630) can be used as a reference for the fourth sub-feature map (640). Accordingly, the first sub-feature map (610), the second sub-feature map (620), the third sub-feature map (630), and the fourth sub-feature map (640) can be progressively encoded or decoded.

[0143] The method of utilizing the modified probability information described above in FIGS. 4 and 5 can also be applied to sub-feature maps. Since multiple sub-feature maps are separated from the same feature map, they are highly related to each other, and thus the probability information of a sub-feature map can be modified using information from other sub-feature maps.

[0144] Specifically, the first sub-feature map (610) is input to the hyper-priority encoder (310) to output the first latent feature map, and the first latent feature map is input to the probabilistic neural network (330) to obtain the initial probability information of the first sub-feature map. In the case of the first sub-feature map (610), since there is no previous sub-feature map to use as a reference, the first sub-feature map (610) can be encoded or decoded using the initial probability information of the first sub-feature map without the modification (430) process.

[0145] The second sub-feature map (620) is input to the hyper-priority encoder (310) to output the second latent feature map, and the first sub-feature map and the second latent feature map are input to the probabilistic neural network (330) to obtain initial probability information of the second sub-feature map. The modified probability information of the second sub-feature map (620) can be obtained through a modification process (430) that uses the initial probability information of the second sub-feature map, the first sub-feature map encoded or decoded before the second sub-feature map, and the initial probability information of the first sub-feature map. The second sub-feature map (620) can be encoded or decoded using the modified probability information.

[0146] The third sub-feature map (630) may be input to the hyper-priority encoder (310) to output the third latent feature map, and the first sub-feature map, the second sub-feature map, and the third latent feature map may be input to the probabilistic neural network (330) to obtain initial probability information of the third sub-feature map. The modified probability information of the third sub-feature map (630) may be obtained through a modification process (430) that utilizes the initial probability information of the third sub-feature map, the first sub-feature map encoded or decoded before the third sub-feature map, the initial probability information of the first sub-feature map, the second sub-feature map encoded or decoded before the third sub-feature map, and the initial probability information of the second sub-feature map. The third sub-feature map (630) may be encoded or decoded using the modified probability information.

[0147] In addition, instead of using both the first sub-feature map and the second sub-feature map, only one of them can be used. For example, if the index is 0, the first sub-feature map can be used, and if the index is 1, the second sub-feature map can be used. The first sub-feature map and the third latent feature map can be input into the probability neural network (330) to obtain initial probability information of the third sub-feature map, and the modified probability information of the third sub-feature map (630) can be obtained through a modification process (430) that uses the initial probability information of the third sub-feature map, the first sub-feature map, and the initial probability information of the first sub-feature map, or the second sub-feature map and the third latent feature map can be input into the probability neural network (330) to obtain initial probability information of the third sub-feature map, and the modified probability information of the third sub-feature map (630) can be obtained through a modification process (430) that uses the initial probability information of the third sub-feature map, the second sub-feature map, and the initial probability information of the second sub-feature map. At this time, an index indicating a sub-feature map selected on the encoding side is generated and signaled, and the decoding side receives the index and determines a sub-feature map to be used according to the index.

[0148] The fourth sub-feature map (640) is input to the hyper-priority encoder (310) to output the fourth latent feature map, and the first sub-feature map, the second sub-feature map, the third sub-feature map, and the fourth latent feature map are input to the probabilistic neural network (330) to obtain initial probability information of the fourth sub-feature map. The modified probability information of the fourth sub-feature map (640) can be obtained through a modification process (430) that uses the initial probability information of the fourth sub-feature map, the first sub-feature map encoded or decoded before the fourth sub-feature map, the initial probability information of the first sub-feature map, the second sub-feature map encoded or decoded before the fourth sub-feature map, the initial probability information of the second sub-feature map, the third sub-feature map encoded or decoded before the fourth sub-feature map, and the initial probability information of the third sub-feature map. The fourth sub-feature map (640) can be encoded or decoded using the modified probability information.

[0149] Additionally, instead of using all of the first, second, and third sub-feature maps, only one of the three can be used. For example, if the index is 0, the first sub-feature map can be used, if the index is 1, the second sub-feature map can be used, and if the index is 2, the third sub-feature map can be used. The first sub-feature map and the fourth latent feature map are input into the probability neural network (330) to obtain the initial probability information of the fourth sub-feature map, and the modified probability information of the fourth sub-feature map (640) is obtained through a modification process (430) using the initial probability information of the fourth sub-feature map, the first sub-feature map, and the initial probability information of the first sub-feature map, or the second sub-feature map and the fourth latent feature map are input into the probability neural network (330) to obtain the initial probability information of the fourth sub-feature map, and the modified probability information of the fourth sub-feature map (640) is obtained through a modification process (430) using the initial probability information of the fourth sub-feature map, the second sub-feature map, and the initial probability information of the second sub-feature map, or the third sub-feature map and the fourth latent feature map are input into the probability neural network (330) to obtain the initial probability information of the fourth sub-feature map, and the modified probability information of the fourth sub-feature map (430) is obtained through a modification process (430) using the initial probability information of the fourth sub-feature map, the third sub-feature map, and the initial probability information of the third sub-feature map. Modified probability information of the fourth sub-feature map (640) can be obtained. At this time, an index indicating a sub-feature map selected on the encoding side is generated and signaled, and the decoding side receives the index and determines a sub-feature map to be used based on the index.

[0150] When modifying probability information using one sub-feature map among multiple sub-feature maps by a signaled index, the scaling variable can be obtained as in mathematical expression 6 below.

[0151] [Equation 6]

[0152]

[0153] Here, α k is a scaling variable for the kth sub-feature map, i is a number smaller than k, is the mean of the ith sub-feature map, is the initial mean value of the i-th sub-feature map, is the initial standard deviation value of the ith sub-feature map. The ith sub-feature map is the sub-feature map determined by the index.

[0154] When modifying probability information using multiple sub-feature maps as references, the scaling variable can be obtained as in Equation 7 below. In this case, since all available sub-feature maps are used as references, the index is not signaled separately.

[0155] [Equation 7]

[0156]

[0157] Here, α k is a scaling variable for the kth sub-feature map, i is a number greater than or equal to 1 and less than or equal to k-1, is the mean of the ith sub-feature map, is the initial mean value of the i-th sub-feature map, is the initial standard deviation value of the ith sub-feature map. Since all available sub-feature maps are used as reference, the scaling variables of all sub-feature maps are summed and then averaged, and the final scaling variable is determined.

[0158] The current feature map being divided into four sub-feature maps is an example and is not limited thereto. That is, the current feature map can be divided into two, six, eight, nine, etc. sub-feature maps. For example, if the current feature map is divided into 2x1 sample groups, the samples located at the top of each sample group can be collected into one sub-feature map to obtain a first sub-feature map, and the pixels located at the bottom of each sample group can be collected into one sub-feature map to obtain a second sub-feature map.

[0159] In addition, if the current feature map is divided into 1x2 sample group units, the samples located on the left side of each sample group can be collected into one sub-feature map to obtain a first sub-feature map, and the samples located on the right side of each sample group can be collected into one sub-feature map to obtain a second sub-feature map.

[0160] Additionally, if the current feature map is divided into 3x2 or 2x3 sample group units, samples located at the same location in each sample group can be collected into one sub-feature map, thereby obtaining six sub-feature maps.

[0161] Additionally, if the current feature map is divided into sample group units of 4x2 or 2x4, samples located at the same location in each sample group can be collected into one sub-feature map, thereby obtaining eight sub-feature maps.

[0162] Additionally, if the current feature map is divided into 3x3 sample group units, samples located at the same location in each sample group can be collected into one sub-feature map, thereby obtaining 9 sub-feature maps.

[0163] The method of modifying the aforementioned probability information can be adaptively applied according to the RDO calculation.

[0164] The method of modifying the aforementioned probability information is optimized as in the mathematical expression 8 below, and the optimized parameter sets ( ) can be transmitted to the decryption side.

[0165] [Equation 8]

[0166]

[0167] Additionally, the method of modifying the aforementioned probability information can be turned on and off at the sequence level or the frame level.

[0168] Additionally, in the method of modifying the aforementioned probability information, the encoding or decoding order can be implicitly determined by flipping or rotation.

[0169] Additionally, in the method of modifying the aforementioned probability information, the encoding or decoding order can be explicitly determined according to the indices representing the sub-feature maps.

[0170] Additionally, the method of modifying the aforementioned probability information can be turned on and off at the sub-feature map level. As described above, all decoded sub-feature maps can be used, or only the signaled reference sub-feature maps can be used.

[0171] FIG. 7 is a diagram illustrating the structure of a neural network according to one embodiment of the present disclosure.

[0172] As illustrated in Fig. 7, input data (705) can be input to the first convolution layer (710). 3X3X4 indicated in the first convolution layer (710) exemplifies convolution processing on the input data (705) using four filter kernels of size 3x3. As a result of the convolution processing, four feature data can be generated by the four filter kernels.

[0173] In one embodiment of the present disclosure, the process of modifying probability information may be performed based on a neural network. If the neural network (700) corresponds to a neural network for the modification process, the input data (705) may include initial probability information of the current feature map, the previous feature map, and initial probability information of the previous feature map. In addition, the input data (705) may include initial probability information of the current sub-feature map, the previous sub-feature map, and initial probability information of the previous sub-feature map.

[0174] In one embodiment of the present disclosure, if the neural network (700) corresponds to a probabilistic neural network (330), the input data (705) may include a latent feature map and a previous feature map. In addition, the input data (705) may include a latent sub-feature map and a previous sub-feature map.

[0175] In one embodiment of the present disclosure, if the neural network (700) corresponds to a hyperparameter encoder (310), the input data (705) may include a current feature map. Additionally, the input data (705) may include a current sub-feature map.

[0176] The feature data generated by the first convolution layer (710) may represent unique characteristics of the input data (705). For example, each feature data may represent vertical characteristics, horizontal characteristics, or edge characteristics of the input data (705).

[0177] Feature data of the first convolution layer (710) can be input to the first activation layer (720).

[0178] The first activation layer (720) can provide non-linear characteristics to each feature data. The first activation layer (820) can include, but is not limited to, a sigmoid function, a Tanh function, a ReLU (Rectified Linear Unit) function, etc.

[0179] Imparting nonlinear characteristics in the first activation layer (720) may mean changing some sample values ​​of feature data and outputting them. In this case, the change may be performed by applying nonlinear characteristics.

[0180] The first activation layer (720) can determine whether to pass the sample values ​​of the feature data to the second convolution layer (730). For example, some sample values ​​of the feature data may be activated by the first activation layer (720) and passed to the second convolution layer (730), and some sample values ​​may be deactivated by the first activation layer (720) and not passed to the second convolution layer (730). The unique characteristics of the input data (705) represented by the feature data may be emphasized by the first activation layer (720).

[0181] Feature data output from the first activation layer (720) can be input to the second convolution layer (730). 3X3X4 displayed in the second convolution layer (730) exemplifies convolution processing on the input feature data using four filter kernels of size 3x3.

[0182] The output of the second convolution layer (730) can be input to the second activation layer (740). The second activation layer (740) can impart nonlinear characteristics to the input feature data.

[0183] Feature data output from the second activation layer (740) can be input to the third convolution layer (750). 3X3X1 displayed in the third convolution layer (750) exemplifies convolution processing to create one output data (755) using one filter kernel of size 3x3.

[0184] The output data (755) depends on whether the neural network (700) is a hyperparameter encoder (310), a probabilistic neural network (330), or a neural network for the correction process (430).

[0185] For example, if the neural network (700) is a hyperparameter encoder (310), the output data (755) may be a latent feature map or a latent sub-feature map.

[0186] Additionally, when the neural network (700) is a probabilistic neural network (330), the output data (755) may be initial probability information of the current feature map or initial probability information of the current sub-feature map.

[0187] Additionally, if the neural network (700) is a neural network for the modification process (430), the output data (755) may be modified probability information of the current feature map or modified probability information of the current sub-feature map.

[0188] In one embodiment of the present disclosure, the number of output data (755) can be adjusted by adjusting the number of filter kernels used in the third convolution layer (750).

[0189] For example, if the neural network (700) is a probabilistic neural network (330) and the initial probability information includes mean data and standard deviation data, two filter kernels may be used so that two channels of data can be output from the third convolution layer (750).

[0190] Although FIG. 7 illustrates that the neural network (700) includes three convolution layers (a first convolution layer (710), a second convolution layer (730), and a third convolution layer (750)) and two activation layers (a first activation layer (720) and a second activation layer (740)), this is only an example, and in one embodiment of the present disclosure, the number of convolution layers and activation layers included in the neural network (700) may vary.

[0191] In one embodiment of the present disclosure, the size and number of filter kernels used in convolutional layers included in the neural network (700) may also be varied.

[0192] In one embodiment of the present disclosure, the neural network (700) may be implemented via a recurrent neural network (RNN). This means changing the CNN structure of the neural network (700) to an RNN structure.

[0193] In one embodiment of the present disclosure, the image decoding device (900) and the image encoding device (1300) may include at least one ALU (Arithmetic logic unit) for convolution operation and activation layer operation.

[0194] The ALU may be implemented as a processor. For convolution operations, the ALU may include a multiplier that performs a multiplication operation between sample values ​​of input data or feature data output from a previous layer and sample values ​​of the filter kernel, and an adder that adds the resulting values ​​of the multiplication.

[0195] For the operation of the activation layer, the ALU may include a multiplier that multiplies the input sample value by a weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and a comparator that compares the multiplication result with a predetermined value to determine whether to pass the input sample value to the next layer.

[0196] Hereinafter, with reference to FIG. 8, the probability information used for entropy encoding and entropy decoding will be described.

[0197] FIG. 8 is a diagram for explaining a method for modifying probability information according to one embodiment of the present disclosure.

[0198] In one embodiment of the present disclosure, the initial probability information output by the probabilistic neural network (330) may represent probabilities of values ​​that samples of the current feature map or the current sub-feature map may have.

[0199] In one embodiment of the present disclosure, the initial probability information may include samples of means and standard deviations corresponding to samples of the current feature map or the current sub-feature map. In this case, the initial probability information may include mean data (810) including samples of means corresponding to samples of the current feature map or the current sub-feature map, and standard deviation data (830) including samples of standard deviations corresponding to samples of the current feature map or the current sub-feature map.

[0200] Referring to FIG. 8, the average data (810) may include sample values ​​from μ(0,0) to μ(1,1), and the standard deviation data (830) may include sample values ​​from σ(0,0) to σ(1,1).

[0201] By applying the scaling variable a, μ(0,0) to μ(1,1) in the average data (810) can be maintained as is, so that modified average data (870) including μ(0,0) to μ(1,1) as samples can be obtained.

[0202] In addition, by applying the scaling variable a, modified standard deviation data (890) including aσ(0,0) to aσ(1,1) as samples can be obtained by multiplying σ(0,0) to σ(1,1) in the standard deviation data (830) by the scaling variable.

[0203] Additionally, when the scaling variable is not applied equally to the entire feature map but is applied to each sample, modified standard deviation data (890) including a(0,0)σ(0,0) to a(1,1)σ(1,1) as samples can be obtained. Here, a(x,y) is a scaling variable corresponding to the x, y positions.

[0204] In one embodiment of the present disclosure, bit shift operations may be used for multiplication operations on sample values ​​of standard deviation data (830).

[0205] By applying the sample values ​​of the modified average data (870) and the sample values ​​of the modified standard deviation data (890) to a predetermined probability model, probability values ​​that the samples of the current feature map or the current sub-feature map can have can be derived.

[0206] FIG. 9 is a diagram illustrating a configuration of an image decoding device according to one embodiment of the present disclosure.

[0207] Referring to FIG. 9, an image decoding device (900) according to one embodiment of the present disclosure may include an acquisition unit (910) and a prediction decoding unit (930).

[0208] The acquisition unit (910) and the prediction decoding unit (930) may be implemented with at least one processor. The acquisition unit (910) and the prediction decoding unit (930) may operate according to at least one instruction stored in a memory.

[0209] Although FIG. 9 illustrates the acquisition unit (910) and the prediction decoding unit (930) separately, the acquisition unit (910) and the prediction decoding unit (930) may be implemented through a single processor. In this case, the acquisition unit (910) and the prediction decoding unit (930) may be implemented through a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software. In addition, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or a memory processing unit for utilizing an external memory.

[0210] The acquisition unit (910) and the prediction decoding unit (930) may be implemented using multiple processors. In this case, the acquisition unit (910) and the prediction decoding unit (930) may be implemented using a combination of dedicated processors, or may be implemented using a combination of multiple general-purpose processors, such as an AP, CPU, or GPU, and software.

[0211] The acquisition unit (910) can acquire a bitstream generated through neural network-based encoding of the current image (100). The bitstream can be generated through intra prediction as described in relation to Fig. 1 or inter prediction as described in relation to Fig. 2.

[0212] The acquisition unit (910) can receive a bitstream from the image encoding device (1300) via a network. In one embodiment of the present disclosure, the acquisition unit (910) can also acquire a bitstream from a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, etc.

[0213] The acquisition unit (910) can acquire a current feature map corresponding to the current image from the bitstream.

[0214] The current feature map may include at least one of feature data (k) of the current image (100) output from the image encoder (12), feature data (w) of the optical flow (g) output from the optical flow encoder (42), or feature data (v) of the residual image (r) output from the residual encoder (52).

[0215] In one embodiment of the present disclosure, the acquisition unit (910) can acquire a latent feature map for the current feature map from the bitstream and acquire modified probability information using the latent feature map. Furthermore, the acquisition unit (910) can acquire the current feature map through entropy decoding of bits included in the bitstream.

[0216] The current feature map is transmitted to the prediction decoding unit (930), and the prediction decoding unit (930) can apply the current feature map to a neural network to obtain a current restored image (300). The current restored image (300) can be output to a display device for playback.

[0217] In one embodiment of the present disclosure, the prediction decoding unit (930) can obtain a current restored image (300) by applying the current feature map to the image decoder (14). In this case, the prediction decoding unit (90) can be understood as restoring the current image (100) through intra prediction.

[0218] In one embodiment of the present disclosure, the prediction decoding unit (930) can obtain the optical flow (g) by applying a current feature map, for example, a feature map of the optical flow (g), to the optical flow decoder (44). Then, the prediction decoding unit (930) can obtain a restored residual image (r') by applying a feature map of the residual image (r) to the residual decoder (54). The prediction decoding unit (930) can obtain the current restored image (300) by combining the current predicted image (x') obtained from the previous restored image (200) and the restored residual image (r') based on the optical flow (g). In this case, it can be understood that the prediction decoding unit (930) restores the current image (100) through inter prediction.

[0219] FIG. 10 is a diagram illustrating the configuration of an acquisition unit according to one embodiment of the present disclosure.

[0220] Referring to FIG. 10, the acquisition unit (910) may include an entropy decryption unit (1010) and an AI control unit (1050).

[0221] A bitstream is input to an entropy decoding unit (1010), and the entropy decoding unit (1010) can obtain a decoded current feature map by applying entropy decoding to bits included in the bitstream.

[0222] The entropy decoding unit (1010) can obtain a latent feature map for the current feature map from the bitstream.

[0223] The entropy decoding unit (1010) can transmit the latent feature map to the AI ​​control unit (1050).

[0224] The latent feature map may be a feature map obtained by processing the current feature map by the hyper-primer encoder (310). The image encoding device (1300) may entropy encode the latent feature map to generate a bitstream including bits corresponding to the latent feature map.

[0225] In one embodiment of the present disclosure, the entropy decoding unit (1010) can use predetermined probability information to obtain a latent feature map from a bitstream.

[0226] The probability information used to obtain a latent feature map can be determined based on rules. For example, the entropy decoding unit (1010) can determine the probability information used to obtain a latent feature map based on predefined rules without using a neural network.

[0227] In one embodiment of the present disclosure, the entropy decoding unit (1010) may obtain probability information used to obtain a latent feature map based on a pre-trained neural network.

[0228] The AI ​​control unit (1050) can obtain initial probability information for the current feature map using the latent feature map and the previous feature map.

[0229] In one embodiment of the present disclosure, the AI ​​control unit (1050) may use a neural network to obtain initial probability information for the current feature map.

[0230] The AI ​​control unit (1350) can obtain modified probability information for the current feature map by using the initial probability information for the current feature map, the previous feature map, and the initial probability information for the previous feature map.

[0231] The previous feature map and the initial probability information for the previous feature map may be obtained during the decoding process of the feature map before the current feature map and may be stored in a buffer or memory.

[0232] In one embodiment of the present disclosure, the AI ​​control unit (1050) can utilize a neural network to obtain modified probability information for the current feature map.

[0233] The modified probability information for the current feature map can be transmitted to the entropy decoding unit (1010).

[0234] The entropy decoding unit (1010) can obtain a decoded current feature map by applying entropy decoding based on modified probability information to bits included in the bitstream.

[0235] The entropy decoding unit (1010) can transfer the decoded current feature map to the prediction decoding unit (930).

[0236] The operation of the AI ​​control unit (1050) will be described in more detail with reference to FIG. 11.

[0237] FIG. 11 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0238] The AI ​​control unit (1050) can use a probabilistic neural network (330) to obtain initial probability information for the current feature map.

[0239] The probabilistic neural network (330) may be stored in memory. In one embodiment of the present disclosure, the probabilistic neural network (330) may be implemented as an AI processor.

[0240] The latent feature map and the previous feature map can be input into a probabilistic neural network (330).

[0241] The probabilistic neural network (330) can process the latent feature map and the previous feature map according to the parameters set as a result of training to output initial probability information for the current feature map.

[0242] The AI ​​control unit (1050) can perform a modification (430) process based on initial probability information for the current feature map, the previous feature map, and initial probability information for the previous feature map.

[0243] The AI ​​control unit (1050) can obtain modified probability information for the current feature map through the modification (430) process.

[0244] In one embodiment of the present disclosure, the AI ​​control unit (1050) may use a bit shift operation for a multiplication operation on sample values ​​of probability information.

[0245] In one embodiment of the present disclosure, the modification process (430) may be performed based on a neural network. For example, modified probability information may be obtained by applying initial probability information for the current feature map, the previous feature map, and initial probability information for the previous feature map to a neural network for the modification process (430).

[0246] The AI ​​control unit (1050) can transmit the modified probability information to the entropy decoding unit (1010).

[0247] The entropy decoding unit (1010) can obtain a decoded current feature map by applying entropy decoding based on modified probability information to bits of the bitstream.

[0248] In FIGS. 9 to 11, the 'current feature map' may become a 'sub-feature map', the 'latent feature map' may become a latent feature map for the 'sub-feature map', and the 'previous feature map' may become a 'sub-feature map' decoded before the 'sub-feature map'. For example, initial first probability information for the first sub-feature map may be obtained based on the first latent feature map for the first sub-feature map, and a decoded first sub-feature map may be obtained based on the initial first probability information. Initial second probability information for the second sub-feature map may be obtained based on the second latent feature map for the second sub-feature map, and modified second probability information for the second sub-feature map may be obtained using the initial first probability information, the decoded first sub-feature map, and the initial second probability information. The second sub-feature map may be decoded using the modified second probability information. The current image may be reconstructed using the decoded first sub-feature map and the decoded second sub-feature map.

[0249] FIG. 12 is a diagram for explaining an image decoding method according to one embodiment of the present disclosure.

[0250] In step S1210, the image decoding device (900) can obtain a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps that rearrange the current feature map corresponding to the current image (100).

[0251] In one embodiment of the present disclosure, the current feature map may include feature data (k) obtained by applying the current image (100) to the image encoder (12), feature data (w) obtained by applying the current image (100) and the previous restored image (200) to the optical flow encoder (42), or feature data (v) obtained by applying the residual image (r) corresponding to the current image (100) to the residual encoder (52).

[0252] In one embodiment of the present disclosure, the plurality of sub-feature maps may be rearranged such that the current feature map is divided into a plurality of sample groups and samples at the same location within the plurality of sample groups are included in the same sub-feature map.

[0253] In step S1220, the image decoding device (900) can obtain a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map.

[0254] At step S1230, the image decoding device (900) can obtain initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map.

[0255] At step S1240, the image decoding device (900) can decode the first sub-feature map using the initial first probability information.

[0256] At step S1250, the image decoding device (900) can obtain initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map.

[0257] In one embodiment of the present disclosure, the image decoding device (900) can obtain initial second probability information using the decoded first sub-feature map and the second latent feature map.

[0258] At step S1260, the image decoding device (900) can obtain modified second probability information for entropy coding of the second sub-feature map using the initial first probability information, the decoded first sub-feature map, and the initial second probability information.

[0259] In one embodiment of the present disclosure, the image decoding device (900) can determine a scaling variable using the initial first probability information and the decoded first sub-feature map, and apply the determined scaling variable to the initial second probability information to obtain modified second probability information.

[0260] In one embodiment of the present disclosure, the scaling variable may represent the prediction accuracy of the decoded first sub-feature map.

[0261] At step S1270, the image decoding device (900) can decode the second sub-feature map using the modified second probability information.

[0262] At step S1280, the image decoding device (900) can restore the current image using the decoded first sub-feature map and the decoded second sub-feature map.

[0263] In one embodiment of the present disclosure, the sample values ​​of the modified probability information may represent the probability of the values ​​that the samples of the sub-feature map may have.

[0264] In one embodiment of the present disclosure, the sample values ​​of the modified probability information may represent the mean and standard deviation corresponding to the samples of the sub-feature map.

[0265] In one embodiment of the present disclosure, the probability of a value that a sample of a sub-feature map can have can be derived by applying the mean and standard deviation indicated by the sample values ​​of the modified probability information to a predetermined probability model.

[0266] In one embodiment of the present disclosure, the image decoding device (900) can restore the current image (100) by applying the decoded first sub-feature map and the decoded second sub-feature map to the image decoder (14), the optical flow decoder (44), and / or the residual decoder (54).

[0267] In one embodiment of the present disclosure, when a plurality of sub-feature maps include a third sub-feature map, the image decoding device (900) obtains an index indicating one sub-feature map to be used as a reference sub-feature map from among the decoded first sub-feature map and the decoded second sub-feature map, obtains initial probability information of one of the decoded first sub-feature map and the decoded second sub-feature map according to the index, obtains initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map, obtains modified third probability information for entropy coding of the third sub-feature map using the obtained one initial probability information, the decoded sub-feature map indicated by the index, and the initial third probability information, decodes the third sub-feature map using the modified third probability information, and reconstructs the current image using the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map.

[0268] In one embodiment of the present disclosure, when a plurality of sub-feature maps include a fourth sub-feature map, the image decoding device (900) obtains an index indicating one sub-feature map to be used as a reference sub-feature map among the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map, obtains initial probability information of one of the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map according to the index, obtains initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map, obtains modified fourth probability information for entropy coding of the fourth sub-feature map using the obtained one initial probability information, the decoded sub-feature map indicated by the index, and the initial fourth probability information, and decodes the fourth sub-feature map using the modified fourth probability information, and decodes the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map. The current image can be restored using the sub-feature map and the decrypted fourth sub-feature map.

[0269] In one embodiment of the present disclosure, when a plurality of sub-feature maps include a third sub-feature map, the image decoding device (900) obtains initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map, obtains modified third probability information for entropy coding of the third sub-feature map using the initial first probability information, the decoded first sub-feature map, the initial second probability information, the decoded second sub-feature map, and the initial third probability information, decodes the third sub-feature map using the modified third probability information, and reconstructs the current image using the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map.

[0270] In one embodiment of the present disclosure, when a plurality of sub-feature maps include a fourth sub-feature map, the image decoding device (900) obtains initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map, obtains modified fourth probability information for entropy coding of the fourth sub-feature map using the initial first probability information, the decoded first sub-feature map, the initial second probability information, the decoded second sub-feature map, the initial third probability information, the decoded third sub-feature map, and the initial fourth probability information, decodes the fourth sub-feature map using the modified fourth probability information, and reconstructs the current image using the decoded first sub-feature map, the decoded second sub-feature map, the decoded third sub-feature map, and the decoded fourth sub-feature map.

[0271] FIG. 13 is a diagram illustrating a configuration of an image encoding device according to one embodiment of the present disclosure.

[0272] Referring to FIG. 13, the image encoding device (1300) may include a prediction encoding unit (1310), a generation unit (1320), an acquisition unit (1330), and a prediction decoding unit (1340).

[0273] The prediction encoding unit (1310), the generation unit (1320), the acquisition unit (1330), and the prediction decoding unit (1340) may be implemented as a processor. The prediction encoding unit (1310), the generation unit (1320), the acquisition unit (1330), and the prediction decoding unit (1340) may operate according to instructions stored in a memory.

[0274] Although FIG. 13 illustrates the prediction encoding unit (1310), the generation unit (1320), the acquisition unit (1330), and the prediction decoding unit (1340) individually, the prediction encoding unit (1310), the generation unit (1320), the acquisition unit (1330), and the prediction decoding unit (134040) may be implemented through a single processor. In this case, the prediction encoding unit (1310), the generation unit (1320), the acquisition unit (1330), and the prediction decoding unit (1340) may be implemented as a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software. In addition, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or may include a memory processing unit for utilizing an external memory.

[0275] The prediction encoding unit (1310), the generation unit (1320), the acquisition unit (1330), and the prediction decoding unit (1340) may be implemented with multiple processors. In this case, the prediction encoding unit (1310), the generation unit (1320), the acquisition unit (1330), and the prediction decoding unit (1340) may be implemented with a combination of dedicated processors, or may be implemented through a combination of software and a plurality of general-purpose processors such as an AP, a CPU, or a GPU.

[0276] The predictive encoding unit (1310) can obtain a current feature map by applying neural network-based encoding to the current image (100). The current feature map can include at least one of feature data (k) of the current image (100), feature data (w) of the optical flow (g), or feature data (v) of the residual image (r).

[0277] In one embodiment of the present disclosure, the prediction encoding unit (1310) can obtain feature data (k) of the current image (100) by applying the current image (100) to the image encoder (12).

[0278] In one embodiment of the present disclosure, the prediction encoding unit (1310) can obtain feature data (w) of optical flow (g) by applying the current image (100) and the previously restored image (200) to the optical flow encoder (42).

[0279] In one embodiment of the present disclosure, the prediction encoding unit (1310) can obtain feature data (v) of the residual image (r) by applying the residual image (r) corresponding to the difference between the current prediction image (x') and the current image (100) to the residual encoder (52).

[0280] The current feature map obtained by the prediction encoding unit (1310) can be transmitted to the generation unit (1820).

[0281] The generation unit (1320) can generate a bitstream based on the current feature map.

[0282] In one embodiment of the present disclosure, the generation unit (1320) may obtain a latent feature map representing a latent feature of a current feature map, and apply the latent feature map to a neural network to obtain initial probability information of the current feature map. The generation unit (1320) may obtain modified probability information based on a previous feature map, the initial probability information of the previous feature map, and the initial probability information of the current feature map. The generation unit (1320) may generate a bitstream by entropy encoding the current feature map according to the modified probability information.

[0283] In one embodiment of the present disclosure, the generation unit (1320) can generate a bitstream by entropy encoding a latent feature map according to predetermined probability information.

[0284] In one embodiment of the present disclosure, the bitstream may include bits corresponding to a current feature map and bits corresponding to a latent feature map.

[0285] The bitstream may be transmitted to the image decoding device (900) via a network. In one embodiment of the present disclosure, the bitstream may be recorded on a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, etc.

[0286] The acquisition unit (1330) can acquire the current feature map from the bitstream generated by the generation unit (1320).

[0287] The current feature map can be passed to the prediction decoding unit (1340).

[0288] The prediction decoding unit (1340) can obtain the current restored image (300) by applying neural network-based decoding to the current feature map.

[0289] The configuration and operation of the acquisition unit (1330) and the prediction decoding unit (1340) may be the same as the acquisition unit (910) and the prediction decoding unit (930) of the image decoding device (900).

[0290] FIG. 14 is a diagram illustrating the configuration of a generation unit according to one embodiment of the present disclosure.

[0291] Referring to FIG. 14, the generation unit (1320) may include an AI control unit (1410) and an entropy encoding unit (1450).

[0292] The current feature map acquired by the prediction encoding unit (1310) may be transmitted to the AI ​​control unit (1410). Additionally, the previous feature map encoded prior to the encoding process of the current feature map and the initial probability information for the previous feature map may be transmitted to the AI ​​control unit (1410). The previous feature map and the initial probability information for the previous feature map may be acquired during the encoding process and stored in a buffer or memory.

[0293] The AI ​​control unit (1410) can obtain a latent feature map from the current feature map and obtain initial probability information for the current feature map using the previous feature map and the latent feature map. The AI ​​control unit (1410) can obtain modified probability information for the current feature map based on the previous feature map, the initial probability information for the previous feature map, and the initial probability information for the current feature map.

[0294] The latent feature map, the current feature map, and the modified probability information for the current feature map can be transmitted to the entropy encoding unit (1450).

[0295] The entropy encoding unit (1450) can generate a bitstream by entropy encoding the current feature map according to modified probability information for the current feature map.

[0296] In one embodiment of the present disclosure, the entropy encoding unit (1450) can generate a bitstream by entropy encoding a latent feature map according to predetermined probability information.

[0297] The bitstream may include bits corresponding to a current feature map and bits corresponding to a latent feature map.

[0298] Hereinafter, the operation of the AI ​​control unit (1410) will be described in more detail with reference to FIG. 15.

[0299] FIG. 15 is a diagram for explaining the operation of an AI control unit according to one embodiment of the present disclosure.

[0300] The AI ​​control unit (1410) can utilize a hyperflash encoder (310) and a probabilistic neural network (330).

[0301] The hyperfryer encoder (310) and the probabilistic neural network (330) may be stored in memory. In one embodiment of the present disclosure, the hyperfryer encoder (310) and the probabilistic neural network (330) may be implemented as an AI processor.

[0302] Referring to FIG. 15, the current feature map can be input to the hyperflash encoder (310).

[0303] The hyperparameter encoder (310) can process the current feature map according to parameters set as a result of training to obtain a latent feature map. The latent feature map and the previous feature map can be input into a probabilistic neural network (330).

[0304] The probabilistic neural network (330) can process the latent feature map and the previous feature map according to the parameters set as a result of training to output initial probability information for the current feature map.

[0305] In one embodiment of the present disclosure, the AI ​​control unit (1410) may use a bit shift operation for a multiplication operation on sample values ​​of probability information.

[0306] In one embodiment of the present disclosure, the modification process (430) may be performed based on a neural network. For example, the previous feature map, the initial probability information of the previous feature map, and the initial probability information for the current feature map may be applied to a neural network for the modification process (430), thereby obtaining modified probability information for the current feature map.

[0307] The AI ​​control unit (1410) can transmit the modified probability information to the entropy encoding unit (1450).

[0308] The entropy encoding unit (1450) can generate a bitstream by applying entropy encoding based on modified probability information for the current feature map.

[0309] Additionally, the entropy encoding unit (1450) can entropy encode the latent feature map.

[0310] In one embodiment of the present disclosure, the entropy encoding unit (1450) may utilize predetermined probability information to apply entropy encoding to a latent feature map. The probability information used to entropy encode the latent feature map may be determined based on rules. In other words, the entropy encoding unit (1450) may determine the probability information used to entropy encode the latent feature map according to predefined rules without using a neural network.

[0311] In FIGS. 13 to 15, the 'current feature map' may become a 'sub-feature map', the 'latent feature map' may become a latent feature map for the 'sub-feature map', and the 'previous feature map' may become a 'sub-feature map' decoded before the 'sub-feature map'. For example, initial first probability information for the first sub-feature map may be obtained based on the first latent feature map for the first sub-feature map, and an encoded first sub-feature map may be obtained based on the initial first probability information. Initial second probability information for the second sub-feature map may be obtained based on the second latent feature map for the second sub-feature map, and modified second probability information for the second sub-feature map may be obtained using the initial first probability information, the encoded first sub-feature map, and the initial second probability information. The second sub-feature map may be encoded using the modified second probability information. The current image may be encoded using the encoded first sub-feature map and the encoded second sub-feature map.

[0312] FIG. 16 is a drawing for explaining an image encoding method according to one embodiment of the present disclosure.

[0313] In step S1610, the image encoding device (1300) can obtain a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps that rearrange the current feature map corresponding to the current image (100).

[0314] In one embodiment of the present disclosure, the current feature map may include feature data (k) obtained by applying the current image (100) to the image encoder (12), feature data (w) obtained by applying the current image (100) and the previous restored image (200) to the optical flow encoder (42), or feature data (v) obtained by applying the residual image (r) corresponding to the current image (100) to the residual encoder (52).

[0315] In one embodiment of the present disclosure, the plurality of sub-feature maps may be rearranged such that the current feature map is divided into a plurality of sample groups and samples at the same location within the plurality of sample groups are included in the same sub-feature map.

[0316] In step S1620, the image encoding device (1300) can obtain a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map.

[0317] At step S1630, the image encoding device (1300) can obtain initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map.

[0318] At step S1640, the image encoding device (1300) can encode the first sub-feature map using the initial first probability information.

[0319] At step S1650, the image encoding device (1300) can obtain initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map.

[0320] In one embodiment of the present disclosure, the image encoding device (1300) can obtain initial second probability information using the encoded first sub-feature map and the second latent feature map.

[0321] In step S1660, the image encoding device (1300) can obtain modified second probability information for entropy coding of the second sub-feature map using the initial first probability information, the encoded first sub-feature map, and the initial second probability information.

[0322] In one embodiment of the present disclosure, the image encoding device (1300) can determine a scaling variable using the initial first probability information and the encoded first sub-feature map, and apply the determined scaling variable to the initial second probability information to obtain modified second probability information.

[0323] In one embodiment of the present disclosure, the scaling variable may represent the prediction accuracy of the encoded first sub-feature map.

[0324] At step S1670, the image encoding device (1300) can encode a second sub-feature map using the modified second probability information.

[0325] At step S1680, the image encoding device (1300) can encode the current feature map using the encoded first sub-feature map and the encoded second sub-feature map.

[0326] In one embodiment of the present disclosure, the sample values ​​of the modified probability information may represent the probability of the values ​​that the samples of the sub-feature map may have.

[0327] In one embodiment of the present disclosure, the sample values ​​of the modified probability information may represent the mean and standard deviation corresponding to the samples of the sub-feature map.

[0328] In one embodiment of the present disclosure, the probability of a value that a sample of a sub-feature map can have can be derived by applying the mean and standard deviation indicated by the sample values ​​of the modified probability information to a predetermined probability model.

[0329] In one embodiment of the present disclosure, when a plurality of sub-feature maps include a third sub-feature map, the image encoding device (1300) generates an index indicating one sub-feature map to be used as a reference sub-feature map from among the encoded first sub-feature map and the encoded second sub-feature map, obtains initial probability information of one of the encoded first sub-feature map and the encoded second sub-feature map according to the index, obtains initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map, obtains modified third probability information for entropy coding of the third sub-feature map using the obtained one initial probability information, the encoded sub-feature map indicated by the index, and the initial third probability information, encodes the third sub-feature map using the modified third probability information, and encodes a current feature map using the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map.

[0330] In one embodiment of the present disclosure, when a plurality of sub-feature maps include a fourth sub-feature map, the image encoding device (1300) generates an index indicating one sub-feature map to be used as a reference sub-feature map among the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map, obtains initial probability information of one of the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map according to the index, obtains initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map, obtains modified fourth probability information for entropy coding of the fourth sub-feature map using the obtained one initial probability information, the encoded sub-feature map indicated by the index, and the initial fourth probability information, and encodes the fourth sub-feature map using the modified fourth probability information, and generates the encoded first sub-feature map, the encoded second sub-feature map, the encoded third sub-feature map, and the encoded fourth sub-feature map. The current feature map can be encoded using a map.

[0331] In one embodiment of the present disclosure, when a plurality of sub-feature maps include a third sub-feature map, the image encoding device (1300) may obtain initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map, obtain modified third probability information for entropy coding of the third sub-feature map using the initial first probability information, the encoded first sub-feature map, the initial second probability information, the encoded second sub-feature map, and the initial third probability information, encode the third sub-feature map using the modified third probability information, and encode the current feature map using the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map.

[0332] In one embodiment of the present disclosure, when a plurality of sub-feature maps include a fourth sub-feature map, the image encoding device (1300) may obtain initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map, obtain modified fourth probability information for entropy coding of the fourth sub-feature map using the initial first probability information, the encoded first sub-feature map, the initial second probability information, the encoded second sub-feature map, the initial third probability information, the encoded third sub-feature map, and the initial fourth probability information, encode the fourth sub-feature map using the modified fourth probability information, and encode a current feature map using the encoded first sub-feature map, the encoded second sub-feature map, the encoded third sub-feature map, and the encoded fourth sub-feature map.

[0333] Hereinafter, with reference to FIG. 17, a method for training the aforementioned neural networks, hyper-precision encoder (310) and probabilistic neural network (330) will be described.

[0334] FIG. 17 is a diagram for explaining a method for training neural networks according to one embodiment of the present disclosure.

[0335] Referring to FIG. 17, the current training image may correspond to the current image (100) described above, and the restored training image may correspond to the current restored image (300) described above.

[0336] In a training process according to one embodiment of the present disclosure, neural networks can be trained so that the reconstructed training images are as similar as possible to the current training images, and the bitrate of the bitstream generated through encoding the current training images is minimized. To this end, as illustrated in FIG. 17, first loss information (1730) and second loss information (1770) can be used to train the neural networks.

[0337] Specifically describing the training process of neural networks, first, a neural network-based encoding process (1710) is applied to the current training image, thereby obtaining the current training feature map.

[0338] The neural network-based encoding process (1710) may be a process of encoding the current training image based on an image encoder (12), an optical flow encoder (42), and / or a residual encoder (52).

[0339] The current feature map may include at least one of feature data obtained by processing the current training image by the image encoder (12), feature data obtained by processing the current training image and the previous reconstructed training image by the optical flow encoder (42), or feature data obtained by processing the residual training image corresponding to the difference between the current training image and the current prediction training image by the residual encoder (52). Here, the current prediction training image may be obtained by transforming the previous reconstructed training image according to the optical flow (g).

[0340] The current training feature map can be input to the hyperfryer encoder (310). The hyperfryer encoder (310) can process the current training feature map according to preset parameters and output a latent training feature map.

[0341] The latent training feature map can be input to a probabilistic neural network (330). Additionally, the previous training feature map can be input to the probabilistic neural network (330). The probabilistic neural network (330) can process the latent training feature map and the previous training feature map according to preset parameters to output initial probability information for the current training feature map.

[0342] Modified probability information for the current training feature map can be obtained through a modification process (430) using the initial probability information, the previous training feature map, and the initial probability information of the previous training feature map.

[0343] A bitstream can be generated by applying an entropy encoding process (1720) based on modified probability information to the current training feature map.

[0344] In one embodiment of the present disclosure, the bitstream may include bits corresponding to a latent training feature map.

[0345] The current training feature map can be obtained by performing an entropy decoding process (1740) based on the modified probability information on the bitstream.

[0346] A restored training image can be obtained by processing the current training feature map according to a neural network-based decoding process (1750).

[0347] The neural network-based decoding process may be a process of restoring the current image (100) based on an image decoder (14), an optical flow decoder (44), and / or a residual decoder (54).

[0348] For training of the neural network, hyper-prior encoder (310), probabilistic neural network (330) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process, at least one of the first loss information (1730) or the second loss information (1770) can be obtained.

[0349] The first loss information (1730) can be derived from the bitrate of the bitstream generated as an encoding result for the current training image.

[0350] Since the first loss information (1730) is related to the encoding efficiency for the current training image, the first loss information may be referred to as compression loss information.

[0351] The second loss information (1770) may correspond to a difference between the current training image and the reconstructed training image. In one embodiment of the present disclosure, the difference between the current training image and the reconstructed training image may include at least one of an L1-norm value, an L2-norm value, a Structural Similarity (SSIM) value, a Peak Signal-To-Noise Ratio-Human Vision System (PSNR-HVS) value, a Multiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF) value, or a Video Multimethod Assessment Fusion (VMAF) value between the current training image and the reconstructed training image.

[0352] The second loss information (1770) is related to the quality of the restored training image, so it can also be referred to as quality loss information.

[0353] The neural network, hyper-priority encoder (310), probabilistic neural network (330) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process can be trained so that the final loss information derived from at least one of the first loss information (1730) or the second loss information (1770) is reduced or minimized.

[0354] In one embodiment of the present disclosure, the neural network, hyper-priority encoder (310), probabilistic neural network (330) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process can reduce or minimize final loss information by changing the values ​​of preset parameters.

[0355] In one embodiment of the present disclosure, the final loss information can be calculated according to the following mathematical expression 9.

[0356] [Equation 9]

[0357] Final loss information = a*first loss information + b*second loss information

[0358] In mathematical expression 9, a and b are weights applied to the first loss information (1730) and the second loss information (1770), respectively.

[0359] According to mathematical expression 9, it can be seen that the neural network, hyper-priority encoder (310), probabilistic neural network (330) used in the neural network-based encoding process and the neural network used in the neural network-based decoding process are trained in a direction in which the restored training image becomes as similar as possible to the current training image and the size of the bitstream is minimized.

[0360] According to one embodiment of the present disclosure, if the modification process (430) is performed based on a neural network, the neural network for the modification process (430) may also be trained.

[0361] The training process described with reference to FIG. 17 can be performed by a training device. The training device may be, for example, an image encoding device (1300) or a separate server. The parameters obtained as a result of the training may be stored in the image encoding device (1300) and the image decoding device (900).

[0362] According to one embodiment of the present disclosure, an image decoding method comprises the steps of: obtaining a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps obtained by rearranging a current feature map corresponding to a current image; obtaining a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map; obtaining initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map; decoding the first sub-feature map using the initial first probability information; obtaining initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map; obtaining modified second probability information for entropy coding of the second sub-feature map using the initial first probability information, the decoded first sub-feature map, and the initial second probability information; decoding the second sub-feature map using the modified second probability information; It may include a step of restoring the current image using the decrypted first sub-feature map and the decrypted second sub-feature map.

[0363] An image decoding method according to one embodiment of the present disclosure can efficiently restore an image by dividing a current feature map into a plurality of sub-feature maps and gradually decoding the feature map, modifying probability information of a sub-feature map to be decoded using an already decoded sub-feature map that is related to the sub-feature map to be decoded, and decoding the sub-feature map using the modified probability information.

[0364] According to one embodiment of the present disclosure, the plurality of sub-feature maps may be rearranged such that the current feature map is divided into a plurality of sample groups and samples at the same location within the plurality of sample groups are included in the same sub-feature map.

[0365] An image decoding method according to one embodiment of the present disclosure can efficiently restore an image by rearranging samples within a current feature map and utilizing sub-feature maps that are related to each other.

[0366] According to one embodiment of the present disclosure, the step of obtaining modified second probability information may include: determining a scaling variable using the initial first probability information and the decrypted first sub-feature map; and applying the determined scaling variable to the initial second probability information to obtain modified second probability information.

[0367] An image decoding method according to one embodiment of the present disclosure can efficiently restore an image by obtaining a scaling variable using initial probability information of a decoded sub-feature map and an already decoded sub-feature map, applying the scaling variable to initial probability information of a sub-feature map to be decoded that is related to the already decoded sub-feature map, obtaining modified probability information, and decoding the sub-feature map to be decoded according to the modified probability information.

[0368] According to one embodiment of the present disclosure, the scaling variable may represent the prediction accuracy of the decoded first sub-feature map.

[0369] An image decoding method according to one embodiment of the present disclosure can efficiently restore an image by decoding a decoded sub-feature map that is related to the decoded sub-feature map using a scaling variable indicating the prediction accuracy of the decoded sub-feature map.

[0370] An image decoding method according to an embodiment of the present disclosure may further include the steps of: obtaining an index indicating one sub-feature map to be used as a reference sub-feature map from among a decoded first sub-feature map and a decoded second sub-feature map, when a plurality of sub-feature maps include a third sub-feature map; obtaining initial probability information of one of the decoded first sub-feature map and the decoded second sub-feature map according to the index; obtaining initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map; obtaining modified third probability information for entropy coding of the third sub-feature map using the obtained one initial probability information, the decoded sub-feature map indicated by the index, and the initial third probability information; decoding the third sub-feature map using the modified third probability information; and restoring a current image using the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map.

[0371] According to one embodiment of the present disclosure, an image decoding method comprises: when a plurality of sub-feature maps include a fourth sub-feature map, obtaining an index indicating one sub-feature map to be used as a reference sub-feature map from among a decoded first sub-feature map, a decoded second sub-feature map, and a decoded third sub-feature map; obtaining initial probability information of one of the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map according to the index; obtaining initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map; obtaining modified fourth probability information for entropy coding of the fourth sub-feature map using the obtained one initial probability information, the decoded sub-feature map indicated by the index, and the initial fourth probability information; decoding the fourth sub-feature map using the modified fourth probability information; The step of restoring the current image using the decrypted first sub-feature map, the decrypted second sub-feature map, the decrypted third sub-feature map, and the decrypted fourth sub-feature map may be further included.

[0372] An image decoding method according to one embodiment of the present disclosure can efficiently restore an image by, when there are two or more sub-feature maps that have already been decoded, obtaining modified probability information of a decoded sub-feature map using one of them and decoding the sub-feature map using the modified probability information.

[0373] An image decoding method according to an embodiment of the present disclosure may further include, when a plurality of sub-feature maps include a third sub-feature map, a step of obtaining initial third probability information for entropy coding of a third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map; a step of obtaining modified third probability information for entropy coding of the third sub-feature map using the initial first probability information, the decoded first sub-feature map, the initial second probability information, the decoded second sub-feature map, and the initial third probability information; a step of decoding the third sub-feature map using the modified third probability information; and a step of restoring a current image using the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map.

[0374] An image decoding method according to an embodiment of the present disclosure may further include, when a plurality of sub-feature maps include a fourth sub-feature map, a step of obtaining initial fourth probability information for entropy coding of a fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map; a step of obtaining modified fourth probability information for entropy coding of the fourth sub-feature map using the initial first probability information, the decoded first sub-feature map, the initial second probability information, the decoded second sub-feature map, the initial third probability information, the decoded third sub-feature map, and the initial fourth probability information; a step of decoding the fourth sub-feature map using the modified fourth probability information; and a step of restoring a current image using the decoded first sub-feature map, the decoded second sub-feature map, the decoded third sub-feature map, and the decoded fourth sub-feature map.

[0375] An image decoding method according to one embodiment of the present disclosure can efficiently restore an image by, when there are two or more sub-feature maps that have already been decoded, obtaining modified probability information of a decoded sub-feature map using all of the decoded sub-feature maps and decoding the sub-feature map using the modified probability information.

[0376] An image decoding device according to one embodiment of the present disclosure comprises: a memory storing one or more instructions; And at least one processor operating according to one or more instructions, wherein the at least one processor obtains a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps that rearrange a current feature map corresponding to a current image, obtains a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map, obtains initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map, decodes the first sub-feature map using the initial first probability information, obtains initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map, obtains modified second probability information for entropy coding of the second sub-feature map using the initial first probability information, the decoded first sub-feature map, and the initial second probability information, decodes the second sub-feature map using the modified second probability information, and decodes the decoded first sub-feature map and The current image can be restored using the decrypted second sub-feature map.

[0377] An image decoding device according to one embodiment of the present disclosure can efficiently restore an image by dividing a current feature map into a plurality of sub-feature maps and gradually decoding the feature map, modifying probability information of a sub-feature map to be decoded using an already decoded sub-feature map that is related to the sub-feature map to be decoded, and decoding the sub-feature map using the modified probability information.

[0378] According to one embodiment of the present disclosure, the plurality of sub-feature maps may be rearranged such that the current feature map is divided into a plurality of sample groups and samples at the same location within the plurality of sample groups are included in the same sub-feature map.

[0379] An image decoding device according to one embodiment of the present disclosure can efficiently restore an image by rearranging samples within a current feature map and utilizing sub-feature maps that are related to each other.

[0380] According to one embodiment of the present disclosure, at least one processor can determine a scaling variable using the initial first probability information and the decrypted first sub-feature map, and apply the determined scaling variable to the initial second probability information to obtain modified second probability information.

[0381] An image decoding device according to one embodiment of the present disclosure can efficiently restore an image by obtaining a scaling variable using initial probability information of a decoded sub-feature map and an already decoded sub-feature map, applying the scaling variable to initial probability information of a sub-feature map to be decoded that is related to the already decoded sub-feature map, obtaining modified probability information, and decoding the sub-feature map to be decoded according to the modified probability information.

[0382] According to one embodiment of the present disclosure, the scaling variable may represent the prediction accuracy of the decoded first sub-feature map.

[0383] An image decoding device according to one embodiment of the present disclosure can efficiently restore an image by decoding a decoded sub-feature map that is related to the decoded sub-feature map using a scaling variable indicating the prediction accuracy of the decoded sub-feature map.

[0384] According to one embodiment of the present disclosure, when a plurality of sub-feature maps include a third sub-feature map, at least one processor may obtain an index indicating one sub-feature map to be used as a reference sub-feature map from among the decoded first sub-feature map and the decoded second sub-feature map, obtain initial probability information of one of the decoded first sub-feature map and the decoded second sub-feature map according to the index, obtain initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map, obtain modified third probability information for entropy coding of the third sub-feature map using the obtained one initial probability information, the decoded sub-feature map indicated by the index, and the initial third probability information, decode the third sub-feature map using the modified third probability information, and reconstruct a current image using the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map.

[0385] According to one embodiment of the present disclosure, when a plurality of sub-feature maps include a fourth sub-feature map, at least one processor obtains an index indicating one sub-feature map to be used as a reference sub-feature map among the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map, obtains initial probability information of one of the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map according to the index, obtains initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map, obtains modified fourth probability information for entropy coding of the fourth sub-feature map using the obtained one initial probability information, the decoded sub-feature map indicated by the index, and the initial fourth probability information, decodes the fourth sub-feature map using the modified fourth probability information, and decodes the decoded first sub-feature map, the decoded second sub-feature map, the decoded third sub-feature map, and The current image can be restored using the decrypted fourth sub-feature map.

[0386] An image decoding device according to one embodiment of the present disclosure can efficiently restore an image by, when there are two or more sub-feature maps that have already been decoded, obtaining modified probability information of a decoded sub-feature map using one of them and decoding the sub-feature map using the modified probability information.

[0387] According to one embodiment of the present disclosure, when a plurality of sub-feature maps include a third sub-feature map, at least one processor may obtain initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map, obtain modified third probability information for entropy coding of the third sub-feature map using the initial first probability information, the decoded first sub-feature map, the initial second probability information, the decoded second sub-feature map, and the initial third probability information, decode the third sub-feature map using the modified third probability information, and reconstruct the current image using the decoded first sub-feature map, the decoded second sub-feature map, and the decoded third sub-feature map.

[0388] According to one embodiment of the present disclosure, when a plurality of sub-feature maps include a fourth sub-feature map, at least one processor may obtain initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map, obtain modified fourth probability information for entropy coding of the fourth sub-feature map using the initial first probability information, the decoded first sub-feature map, the initial second probability information, the decoded second sub-feature map, the initial third probability information, the decoded third sub-feature map, and the initial fourth probability information, decode the fourth sub-feature map using the modified fourth probability information, and reconstruct the current image using the decoded first sub-feature map, the decoded second sub-feature map, the decoded third sub-feature map, and the decoded fourth sub-feature map.

[0389] An image decoding device according to one embodiment of the present disclosure can efficiently restore an image by, when there are two or more sub-feature maps that have already been decoded, obtaining modified probability information of a decoded sub-feature map using all of the decoded sub-feature maps and decoding the sub-feature map using the modified probability information.

[0390] According to one embodiment of the present disclosure, an image encoding method comprises the steps of: obtaining a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps that rearrange a current feature map corresponding to a current image; obtaining a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map; obtaining initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map; encoding the first sub-feature map using the initial first probability information; obtaining initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map; obtaining modified second probability information for entropy coding of the second sub-feature map using the initial first probability information, the encoded first sub-feature map, and the initial second probability information; encoding the second sub-feature map using the modified second probability information; It may include a step of encoding the current feature map using the encoded first sub-feature map and the encoded second sub-feature map.

[0391] An image encoding method according to one embodiment of the present disclosure can efficiently encode an image by dividing a current feature map into a plurality of sub-feature maps and gradually encoding the feature map, modifying probability information of a sub-feature map to be encoded using an already encoded sub-feature map that is related to the sub-feature map to be encoded, and encoding the sub-feature map using the modified probability information.

[0392] According to one embodiment of the present disclosure, the plurality of sub-feature maps may be rearranged such that the current feature map is divided into a plurality of sample groups and samples at the same location within the plurality of sample groups are included in the same sub-feature map.

[0393] An image encoding method according to one embodiment of the present disclosure can efficiently encode an image by rearranging samples within a current feature map and utilizing sub-feature maps that are related to each other.

[0394] According to one embodiment of the present disclosure, the step of obtaining modified second probability information may include: determining a scaling variable using the initial first probability information and the encoded first sub-feature map; and applying the determined scaling variable to the initial second probability information to obtain modified second probability information.

[0395] An image encoding method according to one embodiment of the present disclosure can efficiently encode an image by obtaining a scaling variable using initial probability information of an encoded sub-feature map and an already encoded sub-feature map, applying the scaling variable to initial probability information of a sub-feature map to be encoded that is related to the already encoded sub-feature map to obtain modified probability information, and encoding the sub-feature map to be encoded according to the modified probability information.

[0396] According to one embodiment of the present disclosure, the scaling variable may represent the prediction accuracy of the encoded first sub-feature map.

[0397] An image encoding method according to one embodiment of the present disclosure can efficiently encode an image by encoding an encoded sub-feature map that is related to an encoded sub-feature map using a scaling variable indicating the prediction accuracy of the encoded sub-feature map.

[0398] An image encoding method according to an embodiment of the present disclosure may further include, when a plurality of sub-feature maps include a third sub-feature map, a step of generating an index indicating one sub-feature map to be used as a reference sub-feature map from among an encoded first sub-feature map and an encoded second sub-feature map; a step of obtaining initial probability information of one of the encoded first sub-feature map and the encoded second sub-feature map according to the index; a step of obtaining initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map; a step of obtaining modified third probability information for entropy coding of the third sub-feature map using the obtained one initial probability information, the encoded sub-feature map indicated by the index, and the initial third probability information; a step of encoding the third sub-feature map using the modified third probability information; and a step of encoding a current feature map using the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map.

[0399] According to one embodiment of the present disclosure, a method for encoding an image comprises: generating an index indicating one sub-feature map to be used as a reference sub-feature map from among an encoded first sub-feature map, an encoded second sub-feature map, and an encoded third sub-feature map, when a plurality of sub-feature maps include a fourth sub-feature map; obtaining initial probability information of one of the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map according to the index; obtaining initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map; obtaining modified fourth probability information for entropy coding of the fourth sub-feature map using the obtained one initial probability information, the encoded sub-feature map indicated by the index, and the initial fourth probability information; encoding the fourth sub-feature map using the modified fourth probability information; The step of encoding the current feature map using the encoded first sub-feature map, the encoded second sub-feature map, the encoded third sub-feature map, and the encoded fourth sub-feature map may further be included.

[0400] An image encoding method according to one embodiment of the present disclosure can efficiently encode an image by, when there are two or more sub-feature maps that have already been encoded, obtaining modified probability information of an encoded sub-feature map using one of them and encoding the sub-feature map using the modified probability information.

[0401] An image encoding method according to one embodiment of the present disclosure may further include, when a plurality of sub-feature maps include a third sub-feature map, a step of obtaining initial third probability information for entropy coding of a third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map; a step of obtaining modified third probability information for entropy coding of the third sub-feature map using the initial first probability information, the encoded first sub-feature map, the initial second probability information, the encoded second sub-feature map, and the initial third probability information; a step of encoding the third sub-feature map using the modified third probability information; and a step of encoding a current feature map using the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map.

[0402] An image encoding method according to an embodiment of the present disclosure may further include, when a plurality of sub-feature maps include a fourth sub-feature map, a step of obtaining initial fourth probability information for entropy coding of a fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map; a step of obtaining modified fourth probability information for entropy coding of the fourth sub-feature map using the initial first probability information, the encoded first sub-feature map, the initial second probability information, the encoded second sub-feature map, the initial third probability information, the encoded third sub-feature map, and the initial fourth probability information; a step of encoding the fourth sub-feature map using the modified fourth probability information; and a step of encoding a current feature map using the encoded first sub-feature map, the encoded second sub-feature map, the encoded third sub-feature map, and the encoded fourth sub-feature map.

[0403] An image encoding method according to one embodiment of the present disclosure can efficiently encode an image by, when there are two or more sub-feature maps that have already been encoded, obtaining modified probability information of an encoded sub-feature map using all of the encoded sub-feature maps and encoding the sub-feature map using the modified probability information.

[0404] An image encoding device according to one embodiment of the present disclosure comprises: a memory storing one or more instructions; And at least one processor operating according to one or more instructions, wherein the at least one processor obtains a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps that rearrange a current feature map corresponding to a current image, obtains a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map, obtains initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map, encodes the first sub-feature map using the initial first probability information, obtains initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map, obtains modified second probability information for entropy coding of the second sub-feature map using the initial first probability information, the encoded first sub-feature map, and the initial second probability information, and encodes the second sub-feature map using the modified second probability information, and The current feature map can be encoded using the feature map.

[0405] An image encoding device according to one embodiment of the present disclosure can efficiently encode an image by dividing a current feature map into a plurality of sub-feature maps and gradually encoding the feature map, modifying probability information of a sub-feature map to be encoded using an already encoded sub-feature map that is related to the sub-feature map to be encoded, and encoding the sub-feature map using the modified probability information.

[0406] According to one embodiment of the present disclosure, the plurality of sub-feature maps may be rearranged such that the current feature map is divided into a plurality of sample groups and samples at the same location within the plurality of sample groups are included in the same sub-feature map.

[0407] An image encoding device according to one embodiment of the present disclosure can efficiently encode an image by rearranging samples within a current feature map and utilizing sub-feature maps that are related to each other.

[0408] According to one embodiment of the present disclosure, at least one processor can determine a scaling variable using the initial first probability information and the encoded first sub-feature map, and apply the determined scaling variable to the initial second probability information to obtain modified second probability information.

[0409] An image encoding device according to one embodiment of the present disclosure can efficiently encode an image by obtaining a scaling variable using initial probability information of an encoded sub-feature map and an already encoded sub-feature map, applying the scaling variable to initial probability information of a sub-feature map to be encoded that is related to the already encoded sub-feature map, obtaining modified probability information, and encoding the sub-feature map to be encoded according to the modified probability information.

[0410] According to one embodiment of the present disclosure, the scaling variable may represent the prediction accuracy of the encoded first sub-feature map.

[0411] An image encoding device according to one embodiment of the present disclosure can efficiently encode an image by encoding an encoded sub-feature map that is related to an encoded sub-feature map using a scaling variable indicating the prediction accuracy of the encoded sub-feature map.

[0412] According to an embodiment of the present disclosure, when a plurality of sub-feature maps include a third sub-feature map, an image encoding device may be configured to generate an index indicating one sub-feature map to be used as a reference sub-feature map from among an encoded first sub-feature map and an encoded second sub-feature map, obtain initial probability information of one of the encoded first sub-feature map and the encoded second sub-feature map according to the index, obtain initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map, obtain modified third probability information for entropy coding of the third sub-feature map using the obtained initial probability information, the encoded sub-feature map indicated by the index, and the initial third probability information, encode the third sub-feature map using the modified third probability information, and encode a current feature map using the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map.

[0413] According to an embodiment of the present disclosure, an image encoding device, when a plurality of sub-feature maps include a fourth sub-feature map, at least one processor generates an index indicating one sub-feature map to be used as a reference sub-feature map among an encoded first sub-feature map, an encoded second sub-feature map, and an encoded third sub-feature map, obtains initial probability information of one of the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map according to the index, obtains initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map, obtains modified fourth probability information for entropy coding of the fourth sub-feature map using the obtained one initial probability information, the encoded sub-feature map indicated by the index, and the initial fourth probability information, and encodes the fourth sub-feature map using the modified fourth probability information, and generates an index indicating one of the encoded first sub-feature map, the encoded second sub-feature map, the encoded third sub-feature map, and the encoded fourth sub-feature map. The current feature map can be encoded using a map.

[0414] An image encoding device according to one embodiment of the present disclosure can efficiently encode an image by, when there are two or more sub-feature maps that have already been encoded, obtaining modified probability information of an encoded sub-feature map using one of them and encoding the sub-feature map using the modified probability information.

[0415] According to one embodiment of the present disclosure, when a plurality of sub-feature maps include a third sub-feature map, an image encoding device may be configured to: obtain initial third probability information for entropy coding of a third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map; obtain modified third probability information for entropy coding of the third sub-feature map using the initial first probability information, the encoded first sub-feature map, the initial second probability information, the encoded second sub-feature map, and the initial third probability information; encode the third sub-feature map using the modified third probability information; and encode a current feature map using the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map.

[0416] According to one embodiment of the present disclosure, when a plurality of sub-feature maps include a fourth sub-feature map, an image encoding device may be configured to: obtain initial fourth probability information for entropy coding of a fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map; obtain modified fourth probability information for entropy coding of the fourth sub-feature map using the initial first probability information, the encoded first sub-feature map, the initial second probability information, the encoded second sub-feature map, the initial third probability information, the encoded third sub-feature map, and the initial fourth probability information; encode the fourth sub-feature map using the modified fourth probability information; and encode a current feature map using the encoded first sub-feature map, the encoded second sub-feature map, the encoded third sub-feature map, and the encoded fourth sub-feature map.

[0417] An image encoding device according to one embodiment of the present disclosure can efficiently encode an image by, when there are two or more already encoded sub-feature maps, obtaining modified probability information of an encoded sub-feature map using all of the encoded sub-feature maps and encoding the sub-feature map using the modified probability information.

[0418] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0419] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. A step of obtaining a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps that rearrange the current feature map corresponding to the current image (S1210); A step of obtaining a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map (S1220); A step of obtaining initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map (S1230); A step of decoding the first sub-feature map using the above initial first probability information (S1240); A step of obtaining initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map (S1250); A step of obtaining modified second probability information for entropy coding of the second sub-feature map using the initial first probability information, the decrypted first sub-feature map, and the initial second probability information (S1260); A step of decoding the second sub-feature map using the modified second probability information (S1270); An image decoding method, comprising a step of restoring a current image (S1280) using the decrypted first sub-feature map and the decrypted second sub-feature map.

2. In paragraph 1, An image decoding method, wherein the plurality of sub-feature maps are rearranged such that the current feature map is divided into a plurality of sample groups and samples at the same location within the plurality of sample groups are included in the same sub-feature map.

3. In paragraph 1 or 2, The steps of obtaining the above modified second probability information are: A step of determining a scaling variable using the initial first probability information and the decoded first sub-feature map; and An image decoding method, comprising: a step of obtaining the modified second probability information by applying the determined scaling variable to the initial second probability information.

4. In paragraph 3, A method for decoding an image, wherein the above scaling variable represents the prediction accuracy of the decoded first sub-feature map.

5. In any one of paragraphs 1 to 4, If the above multiple sub-feature maps include a third sub-feature map, A step of obtaining an index indicating one sub-feature map to be used as a reference sub-feature map among the decrypted first sub-feature map and the decrypted second sub-feature map; A step of obtaining initial probability information of one of the decrypted first sub-feature map and the decrypted second sub-feature map according to the index; A step of obtaining initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map; A step of obtaining modified third probability information for entropy coding of the third sub-feature map using the obtained initial probability information, the decoded sub-feature map indicated by the index, and the initial third probability information; A step of decoding the third sub-feature map using the modified third probability information; An image decoding method further comprising a step of restoring a current image using the decrypted first sub-feature map, the decrypted second sub-feature map, and the decrypted third sub-feature map.

6. In paragraph 5, If the above multiple sub-feature maps include a fourth sub-feature map, A step of obtaining an index indicating one sub-feature map to be used as a reference sub-feature map among the decrypted first sub-feature map, the decrypted second sub-feature map, and the decrypted third sub-feature map; A step of obtaining initial probability information of one of the decrypted first sub-feature map, the decrypted second sub-feature map, and the decrypted third sub-feature map according to the index; A step of obtaining initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map; A step of obtaining modified fourth probability information for entropy coding of the fourth sub-feature map using the obtained initial probability information, the decoded sub-feature map indicated by the index, and the initial fourth probability information; A step of decoding the fourth sub-feature map using the modified fourth probability information; An image decoding method further comprising a step of restoring a current image using the decrypted first sub-feature map, the decrypted second sub-feature map, the decrypted third sub-feature map, and the decrypted fourth sub-feature map.

7. In any one of paragraphs 1 to 4, If the above multiple sub-feature maps include a third sub-feature map, A step of obtaining initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map; A step of obtaining modified third probability information for entropy coding of the third sub-feature map using the initial first probability information, the decoded first sub-feature map, the initial second probability information, the decoded second sub-feature map, and the initial third probability information; A step of decoding the third sub-feature map using the modified third probability information; An image decoding method further comprising a step of restoring a current image using the decrypted first sub-feature map, the decrypted second sub-feature map, and the decrypted third sub-feature map.

8. In paragraph 7, If the above multiple sub-feature maps include a fourth sub-feature map, A step of obtaining initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map; A step of obtaining modified fourth probability information for entropy coding of the fourth sub-feature map using the initial first probability information, the decoded first sub-feature map, the initial second probability information, the decoded second sub-feature map, the initial third probability information, the decoded third sub-feature map, and the initial fourth probability information; A step of decoding the fourth sub-feature map using the modified fourth probability information; An image decoding method further comprising a step of restoring a current image using the decrypted first sub-feature map, the decrypted second sub-feature map, the decrypted third sub-feature map, and the decrypted fourth sub-feature map.

9. A step of obtaining a first sub-feature map and a second sub-feature map from among a plurality of sub-feature maps that rearrange the current feature map corresponding to the current image (S1610); A step of obtaining a first latent feature map obtained through neural network-based encoding for the first sub-feature map and a second latent feature map obtained through neural network-based encoding for the second sub-feature map (S1620); A step of obtaining initial first probability information for entropy coding of the first sub-feature map based on the first latent feature map (S1630); A step of encoding a first sub-feature map using the above initial first probability information (S1640); A step of obtaining initial second probability information for entropy coding of the second sub-feature map based on the second latent feature map (S1650); A step of obtaining modified second probability information for entropy coding of the second sub-feature map using the initial first probability information, the encoded first sub-feature map, and the initial second probability information (S1660); A step of encoding the second sub-feature map using the modified second probability information (S1670); An image encoding method, comprising a step of encoding (S1680) the current feature map using the encoded first sub-feature map and the encoded second sub-feature map.

10. In paragraph 9, A method for encoding an image, wherein the plurality of sub-feature maps are rearranged such that the current feature map is divided into a plurality of sample groups and samples at the same location within the plurality of sample groups are included in the same sub-feature map.

11. In paragraph 9 or 10, The steps of obtaining the above modified second probability information are: A step of determining a scaling variable using the initial first probability information and the encoded first sub-feature map; and An image encoding method, comprising: a step of obtaining the modified second probability information by applying the determined scaling variable to the initial second probability information; 12. In any one of paragraphs 9 to 11, If the above multiple sub-feature maps include a third sub-feature map, A step of generating an index indicating one sub-feature map to be used as a reference sub-feature map among the encoded first sub-feature map and the encoded second sub-feature map; A step of obtaining initial probability information of one of the encoded first sub-feature map and the encoded second sub-feature map according to the index; A step of obtaining initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map; A step of obtaining modified third probability information for entropy coding of the third sub-feature map using the obtained initial probability information, the encoded sub-feature map indicated by the index, and the initial third probability information; A step of encoding the third sub-feature map using the modified third probability information; An image encoding method further comprising a step of encoding the current feature map using the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map.

13. In paragraph 12, If the above multiple sub-feature maps include a fourth sub-feature map, A step of generating an index indicating one sub-feature map to be used as a reference sub-feature map among the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map; A step of obtaining initial probability information of one of the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map according to the index; A step of obtaining initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map; A step of obtaining modified fourth probability information for entropy coding of the fourth sub-feature map using the obtained one initial probability information, the encoded sub-feature map indicated by the index, and the initial fourth probability information; A step of encoding the fourth sub-feature map using the modified fourth probability information; An image encoding method further comprising a step of encoding the current feature map using the encoded first sub-feature map, the encoded second sub-feature map, the encoded third sub-feature map, and the encoded fourth sub-feature map.

14. In any one of paragraphs 9 to 11, If the above multiple sub-feature maps include a third sub-feature map, A step of obtaining initial third probability information for entropy coding of the third sub-feature map based on a third latent feature map obtained through neural network-based encoding for the third sub-feature map; A step of obtaining modified third probability information for entropy coding of the third sub-feature map using the initial first probability information, the encoded first sub-feature map, the initial second probability information, the encoded second sub-feature map, and the initial third probability information; A step of encoding the third sub-feature map using the modified third probability information; An image encoding method further comprising a step of encoding the current feature map using the encoded first sub-feature map, the encoded second sub-feature map, and the encoded third sub-feature map.

15. In paragraph 14, If the above multiple sub-feature maps include a fourth sub-feature map, A step of obtaining initial fourth probability information for entropy coding of the fourth sub-feature map based on a fourth latent feature map obtained through neural network-based encoding for the fourth sub-feature map; A step of obtaining modified fourth probability information for entropy coding of the fourth sub-feature map using the initial first probability information, the encoded first sub-feature map, the initial second probability information, the encoded second sub-feature map, the initial third probability information, the encoded third sub-feature map, and the initial fourth probability information; A step of encoding the fourth sub-feature map using the modified fourth probability information; An image encoding method further comprising a step of encoding the current feature map using the encoded first sub-feature map, the encoded second sub-feature map, the encoded third sub-feature map, and the encoded fourth sub-feature map.

Images