Image decoding method, image encoding method, image encoding device, and image decoding device using ai

Abstract

Proposed is an image decoding method, comprising: obtaining feature data of a current optical flow and feature data for a residual image of a current image from a bitstream; obtaining a plurality of remembering gate values corresponding to a plurality of resolutions of the current optical flow and the current image by applying the feature data of the current optical flow to a neural network-based first decoder; obtaining a prediction image of the current image from a previously restored image on the basis of the current optical flow; obtaining a plurality of prediction tensors corresponding to the plurality of resolutions on the basis of the prediction image and the plurality of remembering gate values; and obtaining a current restored image corresponding to the current image by applying the plurality of prediction tensors, the feature data of the current optical flow, and the feature data for the residual image to a neural network-based second decoder.

Description

Image decoding method using AI, image encoding method, image encoding device, and image decoding device

[0001] The present disclosure relates to image encoding and decoding. More specifically, the present disclosure relates to a technique for encoding and decoding images using artificial intelligence (AI), such as a neural network.

[0002] In codecs such as H.264 AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding), an image can be divided into blocks, and each block can be predicted and decoded through inter prediction or intra prediction.

[0003] Intra prediction is a method of compressing images by removing spatial redundancy within the image, and inter prediction is a method of compressing images by removing temporal redundancy between images.

[0004] A representative example of inter prediction is motion estimation coding. Motion estimation coding uses a reference image to predict blocks in the current image. Using a predetermined evaluation function, the reference block most similar to the current block can be searched within a predetermined search range. The current block is predicted based on the reference block, and the resulting predicted block is subtracted from the current block to generate and encode a residual block.

[0005] To derive a motion vector pointing to a reference block within a reference image, the motion vectors of previously encoded blocks can be used as motion vector predictors of the current block. The differential motion vector, which is the difference between the motion vector of the current block and the motion vector predictor, is signaled to the decoder side through a predetermined method.

[0006] Recently, technologies for encoding / decoding images using AI (Artificial Intelligence) have been proposed, and a method for effectively encoding / decoding images using AI, such as neural networks, is required.

[0007] An image decoding method according to one embodiment of the present disclosure may include the steps of: obtaining feature data of a current optical flow and feature data for a residual image of a current image from a bitstream; obtaining a plurality of remembering gate values ​​corresponding to a plurality of resolutions of the current optical flow and the current image by applying the feature data of the current optical flow to a neural network-based first decoder; obtaining a predicted image of the current image from a previous restored image based on the current optical flow; obtaining a plurality of prediction tensors corresponding to the plurality of resolutions based on the predicted image and the plurality of remembering gate values; and obtaining a current restored image corresponding to the current image by applying the plurality of prediction tensors, the feature data of the current optical flow, and the feature data for the residual image to a neural network-based second decoder.

[0008] An image decoding device according to one embodiment of the present disclosure may include a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may obtain feature data of a current optical flow and feature data of a residual image of a current image from a bitstream. The at least one processor may obtain a plurality of remembering gate values ​​corresponding to a plurality of resolutions of the current optical flow and the current image by applying the feature data of the current optical flow to a neural network-based first decoder. The at least one processor may obtain a predicted image of the current image from a previously restored image based on the current optical flow. The at least one processor may obtain a plurality of prediction tensors corresponding to the plurality of resolutions based on the predicted image and the plurality of remembering gate values. At least one processor can apply the plurality of prediction tensors, feature data of the current optical flow, and feature data for the residual image to a neural network-based second decoder to obtain a current reconstructed image corresponding to the current image.

[0009] An image decoding method according to one embodiment of the present disclosure comprises the steps of: obtaining feature data of a current optical flow by applying a current image and a previous reconstructed image to a neural network-based first encoder; obtaining a plurality of remembering gate values ​​corresponding to a plurality of resolutions of the current optical flow and the current image by applying the feature data of the current optical flow to a neural network-based first decoder; obtaining a predicted image of the current image from a previous reconstructed image based on the current optical flow; obtaining a plurality of prediction tensors corresponding to the plurality of resolutions based on the predicted image and the plurality of remembering gate values; obtaining feature data for a residual image by applying the plurality of prediction tensors and the current image to a neural network-based second encoder; obtaining a current reconstructed image corresponding to the current image by applying the plurality of prediction tensors, the feature data of the current optical flow, and the feature data for the residual image to a neural network-based second decoder; and a step of generating a bitstream including feature data of the current optical flow and feature data for the residual image.

[0010] An image encoding device according to one embodiment of the present disclosure may include a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may obtain feature data of a current optical flow by applying a current image and a previously reconstructed image to a neural network-based first encoder. The at least one processor may obtain a plurality of remembering gate values ​​corresponding to a plurality of resolutions of the current optical flow and the current image by applying the feature data of the current optical flow to a neural network-based first decoder. The at least one processor may obtain a prediction image of the current image from a previously reconstructed image based on the current optical flow. The at least one processor may obtain a plurality of prediction tensors corresponding to the plurality of resolutions based on the prediction image and the plurality of remembering gate values. The at least one processor may obtain feature data for a residual image by applying the plurality of prediction tensors and the current image to a neural network-based second encoder. At least one processor can apply the plurality of prediction tensors, the feature data of the current optical flow, and the feature data for the residual image to a neural network-based second decoder to obtain a current reconstructed image corresponding to the current image. At least one processor can generate a bitstream including the feature data of the current optical flow and the feature data for the residual image.

[0011] Figure 1 is a diagram illustrating an image encoding and decoding process based on AI.

[0012] Figure 2 is a diagram illustrating the current optical flow, the current predicted image, and the residual image obtained from the current image and the previously restored image.

[0013] FIG. 3 is a diagram for explaining an image encoding and decoding process according to one embodiment of the present disclosure.

[0014] FIG. 4 is a diagram for explaining a deep prediction decomposition process according to one embodiment of the present disclosure.

[0015] FIG. 5 is a diagram for explaining a deep prediction decomposition process according to one embodiment of the present disclosure.

[0016] FIG. 6 is a diagram for explaining a deep prediction decomposition process according to one embodiment of the present disclosure.

[0017] FIG. 7 is a diagram for explaining the structure of a motion encoder, a motion decoder, a multi-compensation pixel encoder, and a multi-compensation pixel decoder according to one embodiment of the present disclosure.

[0018] FIG. 8 is a diagram for explaining the structure of a motion decoder according to one embodiment of the present disclosure.

[0019] FIG. 9 is a diagram illustrating the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0020] FIG. 10 is a diagram illustrating the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0021] FIG. 11 is a diagram illustrating the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0022] FIG. 12 is a diagram illustrating the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0023] FIG. 13 is a diagram illustrating the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0024] FIG. 14 is a diagram for explaining the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0025] FIG. 15 is a diagram for explaining the optical flow and prediction error, memory gate value, and forget gate value of an original frame according to one embodiment of the present disclosure.

[0026] FIG. 16 is a flowchart of an image encoding method according to one embodiment of the present disclosure.

[0027] FIG. 17 is a diagram illustrating a configuration of an image encoding device according to one embodiment of the present disclosure.

[0028] FIG. 18 is a flowchart of an image decoding method according to one embodiment of the present disclosure.

[0029] FIG. 19 is a diagram illustrating a configuration of an image decoding device according to one embodiment of the present disclosure.

[0030] FIG. 20 is a diagram illustrating a training method of a neural network of a motion encoder, a motion decoder, a deep prediction decomposition, a multi-compensation pixel encoder, and a multi-compensation pixel decoder.

[0031] The present disclosure may be subject to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the embodiments of the present disclosure, and it should be understood that the present disclosure encompasses all modifications, equivalents, and alternatives that fall within the spirit and technical scope of the various embodiments.

[0032] In describing the embodiments, detailed descriptions of related known technologies will be omitted if they are deemed to unnecessarily obscure the gist of the present disclosure. Furthermore, numbers (e.g., "first," "second," etc.) used in the description of the present disclosure are merely identifiers used to distinguish one component from another.

[0033] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0034] Additionally, in the present disclosure, when a component is referred to as being “connected” or “connected” to another component, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless there is a specific description to the contrary.

[0035] In addition, components expressed as 'unit', 'module', etc. in the present disclosure may be two or more components combined into one component, or one component may be divided into two or more components with more detailed functions. In addition, each component described below may additionally perform some or all of the functions performed by other components in addition to its own main function, and of course, some of the main functions performed by each component may be exclusively performed by other components.

[0036] A processor may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. At least one processor, one or more processors may be configured to perform various functions described herein, individually and / or collectively, in a distributed fashion. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0037] Additionally, in the present disclosure, 'image' may refer to a still image, a picture, a frame, a moving image composed of a plurality of consecutive still images, or a video.

[0038] Additionally, in this disclosure, "neural network" is a representative example of an artificial neural network model that mimics brain neurons, and is not limited to an artificial neural network model using a specific algorithm. The neural network may also be referred to as a deep neural network.

[0039] Additionally, in the present disclosure, a "parameter" refers to a value used in the computational process of each layer forming a neural network, and may be used, for example, when applying an input value to a predetermined computational formula. A parameter is a value set as a result of training and may be updated through separate training data as needed.

[0040] Additionally, in this disclosure, "feature data" refers to data obtained by a neural network-based encoder processing input data. The feature data may be one-dimensional or two-dimensional data containing multiple samples. The feature data may also be referred to as a latent representation. The feature data may represent latent features in the data output by the decoder described below.

[0041] Additionally, in the present disclosure, the term "current image" refers to an image that is currently being processed, and the term "previous image" refers to an image that is being processed prior to the current image. Additionally, the term "current motion vector" refers to a motion vector acquired for processing the current image.

[0042] Additionally, in the present disclosure, a "sample" refers to data to be processed, which is data assigned to a sampling location within an image, feature map, or feature data. For example, a sample may include a pixel within a two-dimensional image.

[0043] Additionally, in the present disclosure, a "tensor" refers to data in the form of a multi-dimensional array. A tensor may refer to image data. Furthermore, a tensor may be data obtained after performing addition, multiplication, or subtraction operations on image data. Furthermore, a tensor may be feature data obtained by processing image data through a neural network.

[0044] Figure 1 is a diagram illustrating an image encoding and decoding process based on AI.

[0045] Figure 1 illustrates an inter prediction process, in which an optical flow encoder (110), a video encoder (130), an optical flow decoder (150), and a video decoder (170) can be used.

[0046] The optical flow encoder (110), video encoder (130), optical flow decoder (150), and video decoder (170) can be implemented as a neural network.

[0047] The optical flow encoder (110) and the optical flow decoder (150) generate the current optical flow (g) from the current image (10) and the previously restored image (30). i ) can be understood as a neural network for extracting the data.

[0048] The video encoder (130) and the video decoder (170) process the input video (e.g., the residual image (r) i )) can be a neural network that extracts feature data and restores an image from the feature data.

[0049] Inter prediction is a process of encoding and decoding the current image (10) by utilizing the temporal redundancy between the current image (10) and the previously restored image (30).

[0050] The positional differences (or motion vectors) between blocks or samples in the current image (10) and reference blocks or reference samples in the previously restored image (30) are used for encoding and decoding the current image (10). These positional differences may be referred to as optical flow. Optical flow may also be defined as a set of motion vectors corresponding to samples or blocks in the image.

[0051] Optical flow, particularly the current optical flow, can indicate how the positions of samples in the previous restored image (30) have changed within the current image (10), or where samples identical / similar to the samples in the current image (10) are located within the previous restored image (30).

[0052] For example, if a sample identical to or most similar to a sample located at (1, 1) in the current image (10) is located at (2, 1) in the previous restored image (30), the optical flow or motion vector for the sample can be derived as (1(=2-1), 0(=1-1)).

[0053] In the process of encoding and decoding images using AI, the current optical flow (g) for the current image (10) i ) can be obtained by using an optical flow encoder (110) and an optical flow decoder (150).

[0054] Specifically, the previous restored image (30) and the current image (10) can be input to the optical flow encoder (110). The optical flow encoder (110) processes the current image (10) and the previous restored image (30) according to the parameters set as a result of training to generate feature data (w) for the current optical flow. i ) can be printed.

[0055] Feature data for current optical flow (w i ) can be input to the optical flow decoder (150). The optical flow decoder (150) outputs the input feature data (w i ) is processed according to the parameters set as a result of training to obtain the current optical flow (g i ) can be printed.

[0056] The previous restored image (30) is currently optical flow (g i ) is warped through warping (190) based on the current predicted image (x') as a result of warping (190). i ) can be obtained. Warping (190) is a type of geometric deformation that moves the positions of samples within an image.

[0057] Optical flow (g) indicating the relative positional relationship between samples in the previous restored image (30) and samples in the current image (10)i ) is applied to the previous restored image (30) so that the current predicted image (x') is similar to the current image (10). i ) can be obtained.

[0058] For example, if the sample located at (1, 1) in the previous restored image (30) is most similar to the sample located at (2, 1) in the current image (10), the location of the sample located at (1, 1) in the previous restored image (30) can be changed to (2, 1) through warping (190).

[0059] Current predicted image (x') generated from the previous restored image (30) i ) is not the current image (10) itself, so the current predicted image (x' i ) and the residual image (r) between the current image (10) i ) can be obtained. For example, the current predicted image (x') can be obtained from the sample values ​​in the current image (10). i ) by subtracting my sample values ​​to obtain the residual image (r i ) can be obtained.

[0060] Residual image (r i ) can be input to the image encoder (130). The image encoder (130) generates a residual image (r) according to the parameters set as a result of training. i ) to process the feature data (v) for the residual image. i ) can be printed.

[0061] Feature data for residual images (v i ) can be input to the image decoder (170). The image decoder (170) can input the input feature data (v i ) is processed according to the parameters set as a result of training and the restored residual image (r') is i ) can be printed.

[0062] Current predicted image (x') generated through warping (190) of the previous restored image (30) i ) and the restored residual image (r'i ) can be combined to obtain the current restored image (50).

[0063] When the encoding and decoding process of the image shown in Fig. 1 is implemented by an encoding device and a decoding device, the encoding device obtains the feature data (w) of the current optical flow through the encoding process for the current image (10). i ) and feature data of residual images (v i ) can be quantized, and a bitstream including the quantized feature data can be generated and transmitted to a decoding device. The decoding device dequantizes the quantized feature data extracted from the bitstream to generate the feature data (w) of the current optical flow. i ) and feature data of residual images (v i ) can be obtained. And, the decryption device can obtain the feature data (w) of the current optical flow. i ) and feature data of residual images (v i ) can be processed by an optical flow decoder (150) and an image decoder (170) to obtain a current restored image (50).

[0064] As mentioned above, the current image (10) and the current predicted image (x' i ) residual image (r) between i ) can be input to the video encoder (130), where the current predicted image (x' i ) is currently the optical flow (g i ) is generated from the previous restored image (30), so the current optical flow (g i ) if there is an error in the current predicted image (x' i ) and residual image (r i ) is also likely to contain errors.

[0065] Residual image with error (r i ) is input to the video encoder (130), the bit rate of the bitstream may increase unnecessarily. In addition, the current prediction image (x') with an errori ) is the restored residual image (r') i ) may also deteriorate the quality of the current restored image (50).

[0066] The occurrence and propagation process of errors is explained with reference to Figure 2.

[0067] Figure 2 is a diagram illustrating the current optical flow, the current predicted image, and the residual image obtained from the current image and the previously restored image.

[0068] Referring to FIG. 2, a current optical flow (23) representing the movement of samples in the current image (22) can be obtained from the current image (22) and the previously restored image (21).

[0069] As described with respect to FIG. 1, the current optical flow (23) can be obtained through a processing process by the optical flow encoder (110) and the optical flow decoder (150), and a quantization and dequantization process for the feature data of the current optical flow. This may cause an error to occur within the current optical flow (23), for example, within the A region.

[0070] Specifically, the cause of the error occurrence can be explained as follows. First, an error may occur in the current optical flow (23) due to the limitation of the processing capability of the optical flow encoder (110) and the optical flow decoder (150). Since the computational capabilities of the encoding device and the decoding device are limited, the number of layers and the size of the filter kernel of the optical flow encoder (110) and the optical flow decoder (150) are also inevitably limited. In other words, an error may occur in the current optical flow (23) by processing the current image (22) and the previously restored image (21) with the optical flow encoder (110) and the optical flow decoder (150) having limited capabilities.

[0071] Next, quantization errors may occur in the current optical flow (23) through the quantization and dequantization processes for the feature data of the current optical flow (23). In particular, when the value of the quantization parameter is increased to increase compression efficiency, the bit rate of the bitstream decreases, but at the same time, the amount of quantization error increases.

[0072] Finally, if the movement of the objects included in the current image (22) and the previously restored image (21) is fast, the possibility of an error occurring in the current optical flow (23) increases.

[0073] If an error exists in area A of the current optical flow (23), an error may also occur in area B of the current predicted image (24) generated from the previous restored image (21) based on this, and an error may also occur in area C of the residual image (25) obtained between the current predicted image (24) and the current image (22).

[0074] The residual image (25) is processed by the image encoder (130) and converted into feature data of the residual image (25), and the feature data of the residual image (25) is included in the bitstream after undergoing a predetermined processing, so it can be seen that the error currently present in the optical flow (23) is transmitted to the decoding device.

[0075] In general, since errors have high-frequency characteristics, when a residual image (25) containing errors is processed by an image encoder (130), this may unnecessarily increase the bit rate of the bitstream.

[0076] Below, we describe the image encoding and decoding process to prevent the propagation of errors currently present in the optical flow.

[0077] FIG. 3 is a diagram for explaining an image encoding and decoding process according to one embodiment of the present disclosure.

[0078] Referring to FIG. 3, a motion encoder (310), a motion decoder (320), a multi-compensation pixel encoder (350), and a multi-compensation pixel decoder (360) may be used for encoding and decoding of an image, and deep prediction decomposition (340) may be performed.

[0079] In this disclosure, "deep prediction decomposition" refers to a process of converting a predicted image into multiple predicted images or multiple prediction tensors corresponding to multiple resolutions based on a neural network. The multiple resolutions include the original resolution of the predicted image and multiple resolutions downscaled from the original resolution.

[0080] In one embodiment of the present disclosure, the motion encoder (310), the motion decoder (320), the multi-compensation pixel encoder (350), and the multi-compensation pixel decoder (360) may be implemented as neural networks. Additionally, the deep prediction decomposition (340) may also be implemented as a neural network.

[0081] For encoding of the current image (300), a previous restored image (305) and the current image (300) can be input to a motion encoder (310). The motion encoder (310) can process the current image (300) and the previous restored image (305) according to parameters set as a result of training to output feature data (311) for the current optical flow.

[0082] Feature data (311) for the current optical flow can be input to a motion decoder (320). The motion decoder (320) can process the input feature data (311) according to parameters set as a result of training and output the decoded current optical flow (321) and memory gate values ​​and forget gate values ​​(322) corresponding to multiple resolutions. Exemplary structures of the motion encoder (310) and the motion decoder (320) are described below with reference to FIGS. 7 and 8 .

[0083] The previous restored image (305) is warped through warping (330) based on the current optical flow (321), and the current predicted image (331) can be obtained as a result of the warping (330).

[0084] A deep prediction decomposition (340) process using remembering gate values ​​and forgetting gate values ​​(322) corresponding to multiple resolutions and a current prediction image (331) is performed, and a plurality of prediction tensors (341) corresponding to multiple resolutions can be obtained as a result of the deep prediction decomposition (340). The remembering gate value represents a value for maintaining main information of the image, for example, an edge or detail of a well-compensated area, in order to maintain information useful for encoding or decoding the original image, and the forgetting gate value represents a value for removing unnecessary information or noise of the image, for example, an edge or detail of a poorly compensated area (an occluded area or a non-occluded area), in order to encode or decode the original image. The remembering gate value and the forgetting gate value are set to values ​​from 0 to 1. The closer the remembering gate value is to 1, the more important the information is, and the closer it is to 0, the less important the information is. The closer the forget gate value is to 1, the more information needs to be removed, and the closer it is to 0, the less information needs to be removed. The neural networks used in the deep prediction decomposition (340) process can process the current prediction image (331) according to the parameters set as a result of training and output multiple prediction tensors (341). The pixels that are well predicted in the prediction image are very useful for residual coding to suppress temporal redundancy, and the pixels that are not well predicted in the prediction image are not useful and seriously deteriorate the coding efficiency. Accordingly, the prediction image is decomposed into well-predicted parts and poorly predicted parts, and the remember gate values ​​are flexibly used for residual coding for the well-predicted pixels, and the forget gate values ​​are used for residual coding to squeeze out information from the poorly predicted pixels.Furthermore, to maximize the utilization of the predicted image, downsampling neural network layers are applied to the extracted information, for example, the remaining useful information after applying the forget gate. Multiple useful pieces of information about the predicted image are obtained at various resolutions. That is, to better utilize the predicted image, the predicted image is decomposed into prediction tensors of multiple resolutions. These prediction tensors of multiple resolutions are used as reference for encoding and decoding the original image.

[0085] An exemplary structure of deep prediction decomposition (340) is described below with reference to FIGS. 4 to 6.

[0086] A plurality of prediction tensors (341) and a current image (300) can be input to a multi-compensation pixel encoder (350). The multi-compensation pixel encoder (350) can process the current image (300) and the plurality of prediction tensors (341) according to parameters set as a result of training to output residual image feature data (351). The residual image feature data (351) can be feature data extracted from the plurality of prediction tensors (341) and the current image (300).

[0087] The feature data (311) for the current optical flow and the residual image feature data (351) can be input to the multi-compensation pixel decoder (360). For example, the result of concatenating the feature data (311) for the current optical flow and the residual image feature data (351) can be input to the multi-compensation pixel decoder (390). Here, concatenation may mean a process of combining two or more feature data in the channel direction. In addition, a plurality of prediction tensors (341) can also be input to the multi-compensation pixel decoder (360).

[0088] A multi-compensation pixel decoder (360) can obtain a restored image (360) by processing feature data (311) for the current optical flow, residual image feature data (351), and multiple prediction tensors (341) according to parameters set as a result of training.

[0089] An exemplary structure of a multi-compensation pixel encoder (350) and a multi-compensation pixel decoder (360) is described below with reference to FIGS. 7, 9 to 14.

[0090] When the encoding and decoding process of the image illustrated in FIG. 3 is implemented by an encoding device and a decoding device, the encoding device can generate a bitstream including feature data (311) for the current optical flow obtained through encoding the current image (300) and residual image feature data (351) and transmit the bitstream to the decoding device.

[0091] The decoding device can obtain feature data (311) for optical flow and residual image feature data (351) from the bitstream. Then, the decoding device can obtain a restored image (370) based on the feature data (311) for optical flow and residual image feature data (351).

[0092] Below, the differences between the encoding and decoding process of the image illustrated in FIG. 3 and the encoding and decoding process of the image illustrated in FIG. 1 will be described.

[0093] The motion decoder (320) of FIG. 3, compared to the optical flow decoder (150) of FIG. 1, additionally outputs memory gate values ​​and forget gate values ​​(322) corresponding to multiple resolutions in addition to the current optical flow (321).

[0094] In addition, in FIG. 3, a deep prediction decomposition (340) process is additionally performed to obtain a plurality of prediction tensors (341) based on the current prediction image (311) and memory gate values ​​and forget gate values ​​(322) corresponding to a plurality of resolutions.

[0095] By using the remember gate values ​​and forget gate values, the main information of the well-predicted pixels is maintained, unnecessary information of the poorly predicted pixels is removed, and by using the prediction tensors of multiple resolutions through the downsampling neural network layer, the remaining useful information after applying the forget gate in the resolution before downscaling is utilized in the downscaled resolution, and useful information about the predicted image is obtained in various resolutions.

[0096] Additionally, the multi-compensation pixel encoder (350) and multi-compensation pixel decoder (360) of FIG. 3 utilize multiple prediction tensors obtained through the deep prediction decomposition (340) process for residual coding. Through this, a restored image is obtained in which the main information of the image is maintained and unnecessary errors are removed.

[0097] Hereinafter, the addition and multiplication operations performed are referred to as element-wise sum and element-wise multiplication, respectively.

[0098] FIG. 4 is a diagram for explaining a deep prediction decomposition process according to one embodiment of the present disclosure.

[0099] Referring to FIG. 4, the current prediction image (400) is composed of an image of three channels of RGB (Red Green Blue) with a size of height (H) x width (W). Memory gate values ​​(402, 412, 422, 432) corresponding to multiple resolutions are values ​​corresponding to resolution sizes of H x W, H / 2 x W / 2, H / 4 x W / 4, and H / 8 x W / 8, respectively. Forget gate values ​​(403, 413, 423, 433) corresponding to multiple resolutions are values ​​corresponding to resolution sizes of H x W, H / 2 x W / 2, H / 4 x W / 4, and H / 8 x W / 8, respectively. Memory gate values ​​(402, 412, 422, 432) and forget gate values ​​(403, 413, 423, 433) may be feature maps of one channel that derive spatial differences. Additionally, the memory gate values ​​(402, 412, 422, 432) and forget gate values ​​(403, 413, 423, 433) may be feature maps of one or more channels depending on the characteristics of the image.

[0100] First, by multiplying (404) the memory gate value (402) corresponding to the original resolution H x W of the current predicted image (400) with respect to the current predicted image (400), a prediction tensor (401) corresponding to the resolution of H x W in which information of well-predicted pixels is maintained is obtained.

[0101] A prediction residual tensor (407) corresponding to a resolution of H x W is obtained by subtracting (405) a prediction tensor (401) corresponding to a resolution of H x W from a current prediction image (400) and then multiplying (406) a forget gate value (403) corresponding to the original resolution of H x W. By applying the prediction residual tensor (407) to a neural network (408), an intermediate prediction tensor (410) corresponding to a resolution of H / 2 x W / 2 is obtained. A prediction tensor (411) corresponding to a resolution of H / 2 x W / 2 is obtained by multiplying (414) the intermediate prediction tensor (410) corresponding to a resolution of H / 2 x W / 2 by a remember gate value (412) corresponding to a resolution of H / 2 x W / 2.

[0102] A prediction residual tensor (417) corresponding to a resolution of H / 2 x W / 2 is obtained by subtracting (415) a prediction tensor (411) corresponding to a resolution of H / 2 x W / 2 from an intermediate prediction tensor (410) corresponding to a resolution of H / 2 x W / 2 and then multiplying (416) a forget gate value (413) corresponding to a resolution of H / 2 x W / 2. An intermediate prediction tensor (420) corresponding to a resolution of H / 4 x W / 4 is obtained by applying the prediction residual tensor (417) to a neural network (418). A prediction tensor (421) corresponding to a resolution of H / 4 x W / 4 is obtained by multiplying (424) the intermediate prediction tensor (420) corresponding to a resolution of H / 4 x W / 4 by a remember gate value (422) corresponding to a resolution of H / 4 x W / 4.

[0103] A prediction residual tensor (427) corresponding to a resolution of H / 4 x W / 4 is obtained by subtracting (425) a prediction tensor (421) corresponding to a resolution of H / 4 x W / 4 from an intermediate prediction tensor (420) corresponding to a resolution of H / 4 x W / 4 and then multiplying (426) a forget gate value (423) corresponding to a resolution of H / 4 x W / 4. An intermediate prediction tensor (430) corresponding to a resolution of H / 8 x W / 8 is obtained by applying the prediction residual tensor (427) to a neural network (428). A prediction tensor (431) corresponding to a resolution of H / 8 x W / 8 is obtained by multiplying (434) the intermediate prediction tensor (430) corresponding to a resolution of H / 8 x W / 8 by a remember gate value (432) corresponding to a resolution of H / 8 x W / 8.

[0104] A prediction residual tensor (437) corresponding to a resolution of H / 8 x W / 8 is obtained by subtracting (435) a prediction tensor (431) corresponding to a resolution of H / 8 x W / 8 from an intermediate prediction tensor (430) corresponding to a resolution of H / 8 x W / 8 and then multiplying (436) a forget gate value (433) corresponding to a resolution of H / 8 x W / 8. An intermediate prediction tensor (440) corresponding to a resolution of H / 16 x W / 16 is obtained by applying the prediction residual tensor (437) to a neural network (438).

[0105] Multiple prediction tensors (401, 411, 421, 431) corresponding to the four resolutions obtained through this process can be used for residual coding.

[0106] In addition, residual coding may additionally utilize a prediction residual tensor (407) corresponding to a resolution of H x W, a prediction residual tensor (417) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (427) corresponding to a resolution of H / 4 x W / 4, a prediction residual tensor (437) corresponding to a resolution of H / 8 x W / 8, an intermediate residual tensor (410) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (420) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (430) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (440) corresponding to a resolution of H / 16 x W / 16. This is described later in FIG. 9.

[0107] Additionally, residual coding may additionally utilize a prediction residual tensor (407) corresponding to a resolution of H x W, a prediction residual tensor (417) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (427) corresponding to a resolution of H / 4 x W / 4, and a prediction residual tensor (437) corresponding to a resolution of H / 8 x W / 8. This is described later in Fig. 10.

[0108] Additionally, residual coding may additionally utilize an intermediate residual tensor (410) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (420) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (430) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (440) corresponding to a resolution of H / 16 x W / 16. This is described later in Fig. 11.

[0109] In addition, instead of multiple prediction tensors (401, 411, 421, 431), the residual coding may use a prediction residual tensor (407) corresponding to a resolution of H x W, a prediction residual tensor (417) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (427) corresponding to a resolution of H / 4 x W / 4, a prediction residual tensor (437) corresponding to a resolution of H / 8 x W / 8, an intermediate residual tensor (410) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (420) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (430) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (440) corresponding to a resolution of H / 16 x W / 16. This is described later in FIG. 12.

[0110] Additionally, instead of multiple prediction tensors (401, 411, 421, 431), residual coding may use a prediction residual tensor (407) corresponding to a resolution of H x W, a prediction residual tensor (417) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (427) corresponding to a resolution of H / 4 x W / 4, and a prediction residual tensor (437) corresponding to a resolution of H / 8 x W / 8. This is described later in Fig. 13.

[0111] Additionally, instead of multiple prediction tensors (401, 411, 421, 431), the residual coding may use an intermediate residual tensor (410) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (420) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (430) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (440) corresponding to a resolution of H / 16 x W / 16. This is described later in Fig. 14.

[0112] Although one embodiment of the present disclosure describes prediction tensors of four resolutions, the present invention is not limited thereto, and prediction tensors of less than four or more than four resolutions may be obtained.

[0113] Since multiple prediction tensors have a lot of information at the corresponding resolution, they can be referred to as high-frequency data, or since they contain prediction information that is ultimately used at the corresponding resolution, they can be referred to as prediction data.

[0114] Prediction residual tensors may be referred to as residual data or residual tensors, since they subtract the tensor with memory gate values ​​applied from the image or tensor without memory gate values ​​applied at the corresponding resolution, or as entire data or entire tensors, since they contain information that is available at all resolutions downscaled from the corresponding resolution.

[0115] Intermediate prediction tensors are tensors of a resolution downscaled from the corresponding resolution and therefore contain relatively little information, and may be referred to as low-frequency data or low-frequency tensors, or may contain leftover information utilized at the downscaled resolution and may be referred to as surplus data or surplus tensors.

[0116] The prediction tensors correspond to the remember gate values ​​of the corresponding resolution, the prediction residual tensors correspond to the forget gate values ​​of the corresponding resolution, and the intermediate prediction tensors correspond to the downscaling neural network.

[0117] The prediction tensor may be referred to as a predicted image feature map or predicted image feature data.

[0118] The predicted residual tensor may be referred to as the predicted residual image feature map or the predicted residual image feature data.

[0119] The intermediate prediction tensor may be referred to as an intermediate prediction image feature map or intermediate prediction image feature data.

[0120] FIG. 5 is a diagram for explaining a deep prediction decomposition process according to one embodiment of the present disclosure.

[0121] Referring to FIG. 5, the current prediction image (500) is composed of an image of three channels of RGB (Red Green Blue) with a size of height (H) x width (W). Memory gate values ​​(502, 512, 522, 532) corresponding to multiple resolutions are values ​​corresponding to resolution sizes of H x W, H / 2 x W / 2, H / 4 x W / 4, and H / 8 x W / 8, respectively. Forget gate values ​​(503, 513, 523, 533) corresponding to multiple resolutions are values ​​corresponding to resolution sizes of H x W, H / 2 x W / 2, H / 4 x W / 4, and H / 8 x W / 8, respectively. Memory gate values ​​(502, 512, 522, 532) and forget gate values ​​(503, 513, 523, 533) may be feature maps of one channel that derive spatial differences. Additionally, the memory gate values ​​(502, 512, 522, 532) and forget gate values ​​(503, 513, 523, 533) may be feature maps of one or more channels depending on the characteristics of the image.

[0122] The forget gate values ​​(403, 413, 423, 433) in the embodiment of Fig. 4 are G f,k , k=0, 1, 2, 3, and the resolution is H / (2 k ) x W / (2 k ) and the memory gate values ​​(502, 512, 522, 532) of Fig. 5 are G r,k If the memory gate values ​​(402, 412, 422, 432) of Fig. 4 are the same as the forget gate values ​​(503, 513, 523, 533) of Fig. 5, then the forget gate values ​​(503, 513, 523, 533) of Fig. 5 are G f,k (1-G r,k ) becomes.

[0123] First, by multiplying (504) the memory gate value (502) corresponding to the original resolution H x W of the current predicted image (500), a prediction tensor (501) corresponding to the resolution H x W in which information of well-predicted pixels is maintained is obtained.

[0124] A prediction residual tensor (507) corresponding to a resolution of H x W is obtained by multiplying (506) the current prediction image (500) by a forget gate value (503) corresponding to the original resolution of H x W. An intermediate prediction tensor (510) corresponding to a resolution of H / 2 x W / 2 is obtained by applying the prediction residual tensor (507) to a neural network (508). A prediction tensor (511) corresponding to a resolution of H / 2 x W / 2 is obtained by multiplying (514) the intermediate prediction tensor (510) corresponding to a resolution of H / 2 x W / 2 by a remember gate value (512) corresponding to a resolution of H / 2 x W / 2.

[0125] A prediction residual tensor (517) corresponding to a resolution of H / 2 x W / 2 is obtained by multiplying (516) an intermediate prediction tensor (510) corresponding to a resolution of H / 2 x W / 2 by a forget gate value (513) corresponding to a resolution of H / 2 x W / 2. An intermediate prediction tensor (520) corresponding to a resolution of H / 4 x W / 4 is obtained by applying the prediction residual tensor (517) to a neural network (518). A prediction tensor (521) corresponding to a resolution of H / 4 x W / 4 is obtained by multiplying (524) an intermediate prediction tensor (520) corresponding to a resolution of H / 4 x W / 4 by a remember gate value (522) corresponding to a resolution of H / 4 x W / 4.

[0126] A prediction residual tensor (527) corresponding to a resolution of H / 4 x W / 4 is obtained by multiplying (526) an intermediate prediction tensor (520) corresponding to a resolution of H / 4 x W / 4 by a forget gate value (523) corresponding to a resolution of H / 4 x W / 4. An intermediate prediction tensor (530) corresponding to a resolution of H / 8 x W / 8 is obtained by applying the prediction residual tensor (527) to a neural network (528). A prediction tensor (531) corresponding to a resolution of H / 8 x W / 8 is obtained by multiplying (534) an intermediate prediction tensor (530) corresponding to a resolution of H / 8 x W / 8 by a remember gate value (532) corresponding to a resolution of H / 8 x W / 8.

[0127] A prediction residual tensor (537) corresponding to a resolution of H / 8 x W / 8 is obtained by multiplying (536) an intermediate prediction tensor (530) corresponding to a resolution of H / 8 x W / 8 by a forget gate value (533) corresponding to a resolution of H / 8 x W / 8. An intermediate prediction tensor (540) corresponding to a resolution of H / 16 x W / 16 is obtained by applying the prediction residual tensor (537) to a neural network (538).

[0128] Multiple prediction tensors (501, 511, 521, 531) corresponding to the four resolutions obtained through this process can be used for residual coding.

[0129] Since the forget gate values ​​(503, 513, 523, 533) of FIG. 5 are obtained by multiplying the value obtained by subtracting each of the memory gate values ​​(502, 512, 522, 532) of FIG. 5 from 1 by each of the forget gate values ​​(403, 413, 423, 433) of FIG. 4, each of the prediction residual tensors (507, 517, 527, 537) of FIG. 5 can be ultimately identical to each of the prediction residual tensors (407, 417, 427, 437) of FIG. 4.

[0130] In addition, residual coding may additionally utilize a prediction residual tensor (507) corresponding to a resolution of H x W, a prediction residual tensor (517) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (527) corresponding to a resolution of H / 4 x W / 4, a prediction residual tensor (537) corresponding to a resolution of H / 8 x W / 8, an intermediate residual tensor (510) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (520) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (530) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (540) corresponding to a resolution of H / 16 x W / 16. This is described later in FIG. 9.

[0131] Additionally, residual coding may additionally utilize a prediction residual tensor (507) corresponding to a resolution of H x W, a prediction residual tensor (517) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (527) corresponding to a resolution of H / 4 x W / 4, and a prediction residual tensor (537) corresponding to a resolution of H / 8 x W / 8. This is described later in Fig. 10.

[0132] Additionally, residual coding may additionally utilize an intermediate residual tensor (510) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (520) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (530) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (540) corresponding to a resolution of H / 16 x W / 16. This is described later in Fig. 11.

[0133] In addition, instead of multiple prediction tensors (501, 511, 521, 531), residual coding may use a prediction residual tensor (507) corresponding to a resolution of H x W, a prediction residual tensor (517) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (527) corresponding to a resolution of H / 4 x W / 4, a prediction residual tensor (537) corresponding to a resolution of H / 8 x W / 8, an intermediate residual tensor (510) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (520) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (530) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (540) corresponding to a resolution of H / 16 x W / 16. This is described later in FIG. 12.

[0134] Additionally, instead of multiple prediction tensors (501, 511, 521, 531), residual coding may use a prediction residual tensor (507) corresponding to a resolution of H x W, a prediction residual tensor (517) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (527) corresponding to a resolution of H / 4 x W / 4, and a prediction residual tensor (537) corresponding to a resolution of H / 8 x W / 8. This is described later in Fig. 13.

[0135] Additionally, instead of multiple prediction tensors (501, 511, 521, 531), residual coding may use an intermediate residual tensor (510) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (520) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (530) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (540) corresponding to a resolution of H / 16 x W / 16. This is described later in Fig. 14.

[0136] Although one embodiment of the present disclosure describes prediction tensors of four resolutions, the present invention is not limited thereto, and prediction tensors of less than four or more than four resolutions may be obtained.

[0137] FIG. 6 is a diagram for explaining a deep prediction decomposition process according to one embodiment of the present disclosure.

[0138] Referring to Fig. 6, the current prediction image (600) is composed of an image of three channels of RGB (Red Green Blue) with a size of height (H) x width (W). Memory gate values ​​(602, 612, 422, 632) corresponding to multiple resolutions are values ​​corresponding to resolution sizes of H x W, H / 2 x W / 2, H / 4 x W / 4, and H / 8 x W / 8, respectively. The memory gate values ​​(602, 612, 622, 632) may be feature maps of one channel that derive spatial differences. In addition, the memory gate values ​​(602, 612, 622, 632) may be feature maps of one or more channels depending on the characteristics of the image.

[0139] First, by multiplying (604) the memory gate value (602) corresponding to the original resolution H x W of the current predicted image (600), a prediction tensor (601) corresponding to the resolution H x W in which information of well-predicted pixels is maintained is obtained.

[0140] A prediction residual tensor (607) corresponding to a resolution of H x W is obtained by subtracting (605) a prediction tensor (601) corresponding to a resolution of H x W from a current prediction image (600). By applying the prediction residual tensor (607) to a neural network (608), an intermediate prediction tensor (610) corresponding to a resolution of H / 2 x W / 2 is obtained. The convolution kernel of the first layer of the neural network (608) may be linearly mixed with forget gate values ​​corresponding to a resolution of H x W. Therefore, the intermediate prediction tensor (610) may be an image from which information about poorly predicted pixels is removed. A prediction tensor (611) corresponding to a resolution of H / 2 x W / 2 is obtained by multiplying (614) an intermediate prediction tensor (610) corresponding to a resolution of H / 2 x W / 2 by a memory gate value (612) corresponding to a resolution of H / 2 x W / 2.

[0141] A prediction residual tensor (617) corresponding to a resolution of H / 2 x W / 2 is obtained by subtracting (615) a prediction tensor (611) corresponding to a resolution of H / 2 x W / 2 from an intermediate prediction tensor (610) corresponding to a resolution of H / 2 x W / 2. By applying the prediction residual tensor (617) to a neural network (618), an intermediate prediction tensor (620) corresponding to a resolution of H / 4 x W / 4 is obtained. The convolution kernel of the first layer of the neural network (618) may be linearly mixed with forget gate values ​​corresponding to a resolution of H / 2 x W / 2. Therefore, the intermediate prediction tensor (620) may be an image from which information about pixels that are not predicted well is removed. A prediction tensor (621) corresponding to a resolution of H / 4 x W / 4 is obtained by multiplying (624) an intermediate prediction tensor (620) corresponding to a resolution of H / 4 x W / 4 by a memory gate value (622) corresponding to a resolution of H / 4 x W / 4.

[0142] A prediction residual tensor (627) corresponding to a resolution of H / 4 x W / 4 is obtained by subtracting (425) a prediction tensor (621) corresponding to a resolution of H / 4 x W / 4 from an intermediate prediction tensor (620) corresponding to a resolution of H / 4 x W / 4. By applying the prediction residual tensor (627) to a neural network (628), an intermediate prediction tensor (630) corresponding to a resolution of H / 8 x W / 8 is obtained. The convolution kernel of the first layer of the neural network (628) may be linearly mixed with forget gate values ​​corresponding to a resolution of H / 4 x W / 4. Therefore, the intermediate prediction tensor (630) may be an image from which information about pixels that are not predicted well is removed. A prediction tensor (631) corresponding to a resolution of H / 8 x W / 8 is obtained by multiplying (634) an intermediate prediction tensor (630) corresponding to a resolution of H / 8 x W / 8 by a memory gate value (632) corresponding to a resolution of H / 8 x W / 8.

[0143] A prediction residual tensor (637) corresponding to a resolution of H / 8 x W / 8 is obtained by subtracting (635) a prediction tensor (631) corresponding to a resolution of H / 8 x W / 8 from an intermediate prediction tensor (630) corresponding to a resolution of H / 8 x W / 8. By applying the prediction residual tensor (637) to a neural network (638), an intermediate prediction tensor (640) corresponding to a resolution of H / 16 x W / 16 is obtained. The convolution kernel of the first layer of the neural network (638) may be linearly mixed with forget gate values ​​corresponding to a resolution of H / 8 x W / 8. Therefore, the intermediate prediction tensor (630) may be an image from which information about pixels that are not predicted well is removed.

[0144] Multiple prediction tensors (601, 611, 621, 631) corresponding to the four resolutions obtained through this process can be used for residual coding.

[0145] Since the convolution kernel of the first layer of each of the neural networks (608, 618, 628, 638) of FIG. 6 is mixed with forget gate values ​​corresponding to each resolution, each of the prediction residual tensors (607, 617, 627, 637) of FIG. 6 can be consequently identical to each of the prediction residual tensors (407, 417, 427, 437) of FIG. 4.

[0146] In addition, residual coding may additionally utilize a prediction residual tensor (607) corresponding to a resolution of H x W, a prediction residual tensor (617) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (627) corresponding to a resolution of H / 4 x W / 4, a prediction residual tensor (637) corresponding to a resolution of H / 8 x W / 8, an intermediate residual tensor (610) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (620) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (630) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (640) corresponding to a resolution of H / 16 x W / 16. This is described later in FIG. 9.

[0147] Additionally, residual coding may additionally utilize a prediction residual tensor (607) corresponding to a resolution of H x W, a prediction residual tensor (617) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (627) corresponding to a resolution of H / 4 x W / 4, and a prediction residual tensor (637) corresponding to a resolution of H / 8 x W / 8. This is described later in Fig. 10.

[0148] Additionally, residual coding may additionally utilize an intermediate residual tensor (610) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (620) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (630) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (640) corresponding to a resolution of H / 16 x W / 16. This is described later in Fig. 11.

[0149] In addition, instead of multiple prediction tensors (601, 611, 621, 631), the residual coding may use a prediction residual tensor (607) corresponding to a resolution of H x W, a prediction residual tensor (617) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (627) corresponding to a resolution of H / 4 x W / 4, a prediction residual tensor (637) corresponding to a resolution of H / 8 x W / 8, an intermediate residual tensor (610) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (620) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (630) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (640) corresponding to a resolution of H / 16 x W / 16. This is described later in FIG. 12.

[0150] Additionally, instead of multiple prediction tensors (601, 611, 621, 631), residual coding may use a prediction residual tensor (607) corresponding to a resolution of H x W, a prediction residual tensor (617) corresponding to a resolution of H / 2 x W / 2, a prediction residual tensor (627) corresponding to a resolution of H / 4 x W / 4, and a prediction residual tensor (637) corresponding to a resolution of H / 8 x W / 8. This is described later in Fig. 13.

[0151] Additionally, instead of multiple prediction tensors (601, 611, 621, 631), the residual coding may use an intermediate residual tensor (610) corresponding to a resolution of H / 2 x W / 2, an intermediate residual tensor (620) corresponding to a resolution of H / 4 x W / 4, an intermediate residual tensor (630) corresponding to a resolution of H / 8 x W / 8, and an intermediate residual tensor (640) corresponding to a resolution of H / 16 x W / 16. This is described later in Fig. 14.

[0152] Although one embodiment of the present disclosure describes prediction tensors of four resolutions, the present invention is not limited thereto, and prediction tensors of less than four or more than four resolutions may be obtained.

[0153] FIG. 7 is a diagram for explaining the structure of a motion encoder, a motion decoder, a multi-compensation pixel encoder, and a multi-compensation pixel decoder according to one embodiment of the present disclosure.

[0154] Referring to FIG. 7, a reference image (705), which is a previously restored image, and an original image (700) are input to a motion encoder (710). Feature data (715) for the current optical flow is output by sequentially downscaling through a plurality of neural networks (711, 712, 713, 716) within the motion encoder (710). Each of the plurality of neural networks (711, 712, 713, 716) within the motion encoder (710) may include at least one convolutional layer. In addition, the plurality of neural networks (711, 712, 713, 716) may downscale input data and output it.

[0155] Feature data (715) for the current optical flow output from the motion encoder (710) is input to the motion decoder (720). The feature data (715) for the current optical flow is input to a neural network (721) to obtain first intermediate data. In addition, the feature data (715) for the current optical flow is input to a convolutional layer (725) and a sigmoid function (726) to obtain a memory gate value (733) of the first resolution and a forget gate value (734) of the first resolution. The sigmoid function is one of the activation functions used in neural networks and is a nonlinear function that outputs input data as a value between 0 and 1. Therefore, the memory gate value and forget gate values ​​obtained through the sigmoid function are values ​​between 0 and 1. The first intermediate data is input to a neural network (722) to obtain second intermediate data. In addition, the first intermediate data is input to a convolution layer (727) and a sigmoid function (728) to obtain a memory gate value (735) of the second resolution and a forget gate value (736) of the second resolution. The second intermediate data is input to a neural network (723) to obtain third intermediate data. In addition, the second intermediate data is input to a convolution layer (729) and a sigmoid function (730) to obtain a memory gate value (737) of the third resolution and a forget gate value (738) of the third resolution. The third intermediate data is input to a neural network (724) to obtain the current optical flow. In addition, the third intermediate data is input to a convolution layer (731) and a sigmoid function (732) to obtain a memory gate value (739) of the fourth resolution and a forget gate value (740) of the fourth resolution. For example, the fourth resolution may correspond to the resolution of the original image, the third resolution may correspond to 1 / 2 the resolution of the original resolution, the second resolution may correspond to 1 / 4 the resolution of the original resolution, and the first resolution may correspond to 1 / 8 the resolution of the original resolution.The neural networks (721, 722, 723, 724) within the motion decoder (720) may include at least one convolutional layer. Additionally, the neural networks (721, 722, 723, 724) may upscale input data. Additionally, the convolutional layers (725, 727, 729, 731) within the motion decoder (720) may upscale input data.

[0156] A predicted image (743) for the current image is obtained by warping (742) using the current optical flow (741) and the reference image (705).

[0157] Prediction tensors (745, 746, 747, 748) corresponding to multiple resolutions are obtained through a deep prediction decomposition (744) process using memory gate values ​​(733, 735, 737, 739) corresponding to multiple resolutions, forget gate values ​​(734, 736, 738, 740) corresponding to multiple resolutions, and a prediction image (743). The deep prediction decomposition (744) process is omitted as it has been described above in FIGS. 4 to 6.

[0158] In the multi-compensation pixel encoder (750), a first subtraction tensor obtained by subtracting (751) a prediction tensor (745) of a fourth resolution corresponding to the original resolution from the original image (700) is input to a neural network (752). An intermediate encoding tensor of a third resolution is output through the neural network (752). A second subtraction tensor obtained by subtracting (753) a prediction tensor (746) of a third resolution from the intermediate encoding tensor of the third resolution is input to a neural network (754). An intermediate encoding tensor of a second resolution is output through the neural network (754). A third subtraction tensor obtained by subtracting (755) a prediction tensor (747) of a second resolution from the intermediate encoding tensor of the second resolution is input to a neural network (756). An intermediate encoding tensor of a first resolution is output through the neural network (754). A fourth subtracted tensor, which is obtained by subtracting (757) the prediction tensor (748) of the first resolution from the intermediate encoding tensor of the first resolution, is input to a neural network (758). Residual image feature data (759) is output through the neural network (758). Each of the plurality of neural networks (752, 754, 756, 758) within the multi-compensation pixel encoder (750) may include at least one convolutional layer. In addition, the plurality of neural networks (752, 754, 756, 758) may downscale input data and output it.

[0159] In the multi-compensation pixel decoder (760), residual image feature data (759) is first input into a neural network (761). A residual tensor of a first resolution is obtained through the neural network (761). The residual tensor of the first resolution is summed (762) with a prediction tensor (748) of the first resolution and input into a neural network (763). A residual tensor of a second resolution is obtained through the neural network (763). The residual tensor of the second resolution is summed (764) with a prediction tensor (747) of the second resolution and input into a neural network (765). A residual tensor of a third resolution is obtained through the neural network (765). The residual tensor of the third resolution is summed (766) with a prediction tensor (746) of the third resolution and input into a neural network (767). A residual tensor of the first resolution is obtained through a neural network (767). The residual tensor of the first resolution is combined (768) with the prediction tensor (745) of the first resolution. A restored image (770) is output as a result of the combination (768). The neural networks (761, 763, 765, 767) within the multi-compensation pixel decoder (760) may include at least one convolutional layer. In addition, the neural networks (761, 763, 765, 767) may upscale input data.

[0160] Additionally, the multi-compensation pixel decoder (760) may be input with feature data (715) of the current optical flow in addition to residual image feature data (759). The residual image feature data (759) and feature data (715) of the current optical flow may be input in a concatenated manner.

[0161] A multi-compensation pixel encoder (750) and a multi-compensation pixel decoder (760) sequentially perform residual coding multiple times according to multiple resolutions in the pixel domain.

[0162] FIG. 8 is a diagram for explaining the structure of a motion decoder according to one embodiment of the present disclosure.

[0163] Referring to FIG. 8, an original frame (800) and a reference frame (805) are input to a motion encoder (820). Additional information (810) about the current image may be input to the motion encoder (820). Feature data (825) of the current optical flow is output from the motion encoder (820). The structure of the motion encoder (820) may be identical to the structure (710) of the motion encoder described above in FIG. 7.

[0164] The feature data (825) of the current optical flow is input to the neural network (831) of the motion decoder (830). First intermediate data and first gate data are obtained from the neural network (831). The first gate data is applied to a sigmoid function (832) to obtain a memory gate value (847) of the first resolution and a forget gate value (848) of the first resolution. The first intermediate data is input to the neural network (833). Second intermediate data and second gate data are obtained from the neural network (833). The second gate data is applied to a sigmoid function (834) to obtain a memory gate value (845) of the second resolution and a forget gate value (846) of the second resolution. The second intermediate data is input to the neural network (835). Third intermediate data and third gate data are obtained from the neural network (835). The third gate data is applied to a sigmoid function (836) to obtain a memory gate value (843) of the third resolution and a forget gate value (844) of the third resolution. The third intermediate data is input to a neural network (837). The current optical flow (850) and the fourth gate data are obtained from the neural network (837). The fourth gate data is applied to a sigmoid function (838) to obtain a memory gate value (841) of the fourth resolution and a forget gate value (842) of the fourth resolution. For example, the fourth resolution may correspond to the resolution of the original image, the third resolution may correspond to 1 / 2 the resolution of the original resolution, the second resolution may correspond to 1 / 4 the resolution of the original resolution, and the first resolution may correspond to 1 / 8 the resolution of the original resolution. The neural networks (831, 833, 835, 837) within the motion decoder (830) may include at least one convolutional layer. Additionally, the neural networks (831, 833, 835, 837) may upscale input data.

[0165] The motion decoder (720) of FIG. 7 and the motion decoder (830) of FIG. 8 output the current optical flow, memory gate values, and forget gate values ​​based on the feature data (715, 825) of the current optical flow, and thus may be referred to as a multi-purpose decoder. That is, optical flow encoding and decoding and gate generation may be efficiently merged. In addition, the corresponding motion encoder (710) of FIG. 7 and the motion encoder (820) of FIG. 8 may be referred to as a multi-purpose encoder. The memory gate values ​​and the forget gate values ​​may also be referred to as a decomposition gate or a decomposition weight.

[0166] According to one embodiment of the present disclosure, the memory gate values ​​and the forget gate values ​​may not be output based on the motion encoder and the motion decoder, but may be independently generated from each encoder and decoder. For example, an original image and a reference image may be input to a memory gate encoder, memory gate feature data may be output from the memory gate encoder, the memory gate feature data may be input to the memory gate decoder, and memory gate values ​​corresponding to a plurality of resolutions may be output from the memory gate decoder. In addition, an original image and a reference image may be input to a forget gate encoder, forget gate feature data may be output from the forget gate encoder, the forget gate feature data may be input to the forget gate decoder, and forget gate values ​​corresponding to a plurality of resolutions may be output from the forget gate decoder.

[0167] FIG. 9 is a diagram illustrating the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0168] Referring to FIG. 9, in the multi-compensation pixel encoder (910), a first subtraction tensor obtained by subtracting (911) the prediction tensor (901) of the original resolution from the original image (900) and a prediction residual tensor (944) of the original resolution are input to the neural network (912). The first subtraction tensor and the prediction residual tensor (944) may be input in a concatenated manner. The prediction residual tensor (944) of the original resolution corresponds to the prediction residual tensor (407) of FIG. 4, the prediction residual tensor (507) of FIG. 5, or the prediction residual tensor (607) of FIG. 6.

[0169] An intermediate encoding tensor of a first downscaled resolution, in which the original resolution is downscaled, is output through a neural network (912). A second subtraction tensor obtained by subtracting (913) the prediction tensor (902) of the first downscaled resolution from the intermediate encoding tensor of the first downscaled resolution and a prediction residual tensor (943) of the first downscaled resolution are input to the neural network (914). The second subtraction tensor and the prediction residual tensor (943) may be input in a concatenated manner. The prediction residual tensor (943) of the first downscaled resolution corresponds to the prediction residual tensor (417) of FIG. 4, the prediction residual tensor (517) of FIG. 5, or the prediction residual tensor (617) of FIG. 6.

[0170] An intermediate encoding tensor of a second downscaled resolution, in which the first downscaled resolution is downscaled, is output through a neural network (914). A third subtraction tensor obtained by subtracting (915) the prediction tensor (903) of the second downscaled resolution from the intermediate encoding tensor of the second downscaled resolution and a prediction residual tensor (942) of the second downscaled resolution are input to the neural network (916). The third subtraction tensor and the prediction residual tensor (942) may be input in a concatenated manner. The prediction residual tensor (942) of the second downscaled resolution corresponds to the prediction residual tensor (427) of FIG. 4, the prediction residual tensor (527) of FIG. 5, or the prediction residual tensor (627) of FIG. 6.

[0171] An intermediate encoding tensor of a third downscaled resolution, in which the second downscaled resolution is downscaled, is output through a neural network (916). A fourth subtraction tensor obtained by subtracting (917) the prediction tensor (904) of the third downscaled resolution from the intermediate encoding tensor of the third downscaled resolution and a prediction residual tensor (941) of the third downscaled resolution are input to a neural network (918). The fourth subtraction tensor and the prediction residual tensor (941) may be input in a concatenated manner. The prediction residual tensor (941) of the third downscaled resolution corresponds to the prediction residual tensor (437) of FIG. 4, the prediction residual tensor (537) of FIG. 5, or the prediction residual tensor (637) of FIG. 6.

[0172] Residual image feature data (930) is output through a neural network (918). Each of the multiple neural networks (912, 914, 916, 918) within the multi-compensation pixel encoder (910) may include at least one convolutional layer. In addition, the multiple neural networks (912, 914, 916, 918) may downscale input data and output it.

[0173] In the multi-compensation pixel decoder (920), residual image feature data (930) and an intermediate prediction tensor (931) of a fourth downscaled resolution, which is a downscaled third downscaled resolution, are first input to the neural network (921). The residual image feature data (930) and the intermediate prediction tensor (931) may be input in a concatenated manner. The intermediate prediction tensor (931) of the fourth downscaled resolution corresponds to the intermediate prediction tensor (440) of FIG. 4, the intermediate prediction tensor (540) of FIG. 5, or the intermediate prediction tensor (640) of FIG. 6.

[0174] A residual tensor of a third downscale resolution is obtained through a neural network (921). The residual tensor of the third downscale resolution is summed (922) with the prediction tensor (904) of the third downscale resolution and input to a neural network (923). In addition, an intermediate prediction tensor (932) of the third downscale resolution is also input to the neural network (923). The result data of the summation (922) and the intermediate prediction tensor (932) may be concatenated and input. The intermediate prediction tensor (932) of the third downscale resolution corresponds to the intermediate prediction tensor (430) of FIG. 4, the intermediate prediction tensor (530) of FIG. 5, or the intermediate prediction tensor (630) of FIG. 6.

[0175] A residual tensor of the second downscale resolution is obtained through a neural network (923). The residual tensor of the second downscale resolution is summed (924) with the prediction tensor (903) of the second downscale resolution and input to the neural network (925). In addition, an intermediate prediction tensor (933) of the second downscale resolution is also input to the neural network (925). The result data of the summation (924) and the intermediate prediction tensor (933) may be concatenated and input. The intermediate prediction tensor (933) of the second downscale resolution corresponds to the intermediate prediction tensor (420) of FIG. 4, the intermediate prediction tensor (520) of FIG. 5, or the intermediate prediction tensor (620) of FIG. 6.

[0176] A residual tensor of the first downscale resolution is obtained through a neural network (925). The residual tensor of the first downscale resolution is summed (926) with the prediction tensor (902) of the first downscale resolution and input to the neural network (927). In addition, an intermediate prediction tensor (934) of the first downscale resolution is also input to the neural network (927). The result data of the summation (926) and the intermediate prediction tensor (934) may be concatenated and input. The intermediate prediction tensor (934) of the first downscale resolution corresponds to the intermediate prediction tensor (410) of FIG. 4, the intermediate prediction tensor (510) of FIG. 5, or the intermediate prediction tensor (610) of FIG. 6.

[0177] A residual tensor of the original resolution is obtained through a neural network (927). The residual tensor of the original resolution is combined (928) with the prediction tensor (901) of the original resolution. A restored image (950) is output as a result of the combination (928).

[0178] The neural networks (921, 923, 925, 927) within the multi-compensation pixel decoder (920) may include at least one convolutional layer. Additionally, the neural networks (921, 923, 925, 927) may upscale input data.

[0179] The prediction tensors (901, 902, 903, 904) correspond to the prediction tensors (401, 402, 403, 404) of FIG. 4, the prediction tensors (501, 502, 503, 504) of FIG. 5, or the prediction tensors (601, 602, 603, 604) of FIG. 6.

[0180] Additionally, the multi-compensation pixel decoder (920) may be input with feature data of the current optical flow in addition to residual image feature data (930). The residual image feature data (930) and the feature data of the current optical flow may be input in a concatenated manner.

[0181] FIG. 10 is a diagram illustrating the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0182] Referring to FIG. 10, in a multi-compensation pixel encoder (1010), a first subtraction tensor obtained by subtracting (1011) a prediction tensor (1001) of the original resolution from an original image (1000) and a prediction residual tensor (1044) of the original resolution are input to a neural network (1012). The first subtraction tensor and the prediction residual tensor (1044) may be input in a concatenated manner. The prediction residual tensor (1044) of the original resolution corresponds to the prediction residual tensor (407) of FIG. 4, the prediction residual tensor (507) of FIG. 5, or the prediction residual tensor (607) of FIG. 6.

[0183] An intermediate encoding tensor of a first downscaled resolution, in which the original resolution is downscaled, is output through a neural network (1012). A second subtraction tensor obtained by subtracting (1013) the prediction tensor (1002) of the first downscaled resolution from the intermediate encoding tensor of the first downscaled resolution and a prediction residual tensor (1043) of the first downscaled resolution are input to the neural network (1014). The second subtraction tensor and the prediction residual tensor (1043) may be input in a concatenated manner. The prediction residual tensor (1043) of the first downscaled resolution corresponds to the prediction residual tensor (417) of FIG. 4, the prediction residual tensor (517) of FIG. 5, or the prediction residual tensor (617) of FIG. 6.

[0184] An intermediate encoding tensor of a second downscaled resolution, which is a downscaled version of the first downscaled resolution, is output through a neural network (1014). A third subtraction tensor obtained by subtracting (1015) the prediction tensor (1003) of the second downscaled resolution from the intermediate encoding tensor of the second downscaled resolution and a prediction residual tensor (1042) of the second downscaled resolution are input to the neural network (1016). The third subtraction tensor and the prediction residual tensor (1042) may be input in a concatenated manner. The prediction residual tensor (1042) of the second downscaled resolution corresponds to the prediction residual tensor (427) of FIG. 4, the prediction residual tensor (527) of FIG. 5, or the prediction residual tensor (627) of FIG. 6.

[0185] An intermediate encoding tensor of a third downscaled resolution, which is a downscaled version of the second downscaled resolution, is output through a neural network (1016). A fourth subtraction tensor, which is obtained by subtracting (1017) the prediction tensor (1004) of the third downscaled resolution from the intermediate encoding tensor of the third downscaled resolution, and a prediction residual tensor (1041) of the third downscaled resolution are input to a neural network (1018). The fourth subtraction tensor and the prediction residual tensor (1041) may be input in a concatenated manner. The prediction residual tensor (1041) of the third downscaled resolution corresponds to the prediction residual tensor (437) of FIG. 4, the prediction residual tensor (537) of FIG. 5, or the prediction residual tensor (637) of FIG. 6.

[0186] Residual image feature data (1030) is output through a neural network (1018). Each of the multiple neural networks (1012, 1014, 1016, 1018) within the multi-compensation pixel encoder (1010) may include at least one convolutional layer. In addition, the multiple neural networks (1012, 1014, 1016, 1018) may downscale input data and output it.

[0187] In the multi-compensation pixel decoder (1020), residual image feature data (1030) is first input into the neural network (1021).

[0188] A residual tensor of the third downscale resolution is obtained through a neural network (1021). The residual tensor of the third downscale resolution is combined (1022) with the prediction tensor (1004) of the third downscale resolution and input to the neural network (1023).

[0189] A residual tensor of the second downscale resolution is obtained through a neural network (1023). The residual tensor of the second downscale resolution is combined (1024) with the prediction tensor (1003) of the second downscale resolution and input to the neural network (1025).

[0190] A residual tensor of the first downscale resolution is obtained through a neural network (1025). The residual tensor of the first downscale resolution is combined (1026) with the prediction tensor (1002) of the first downscale resolution and input to the neural network (1027).

[0191] A residual tensor of the original resolution is obtained through a neural network (1027). The residual tensor of the original resolution is combined (1028) with the predicted tensor of the original resolution (1001). A restored image (1050) is output as a result of the combination (1028).

[0192] The neural networks (1021, 1023, 1025, 1027) within the multi-compensation pixel decoder (1020) may include at least one convolutional layer. Additionally, the neural networks (1021, 1023, 1025, 1027) may upscale input data.

[0193] The prediction tensors (1001, 1002, 1003, 1004) correspond to the prediction tensors (401, 402, 403, 404) of FIG. 4, the prediction tensors (501, 502, 503, 504) of FIG. 5, or the prediction tensors (601, 602, 603, 604) of FIG. 6.

[0194] Additionally, the multi-compensation pixel decoder (1020) may be input with feature data of the current optical flow in addition to residual image feature data (1030). The residual image feature data (1030) and the feature data of the current optical flow may be input in a concatenated manner.

[0195] FIG. 11 is a diagram illustrating the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0196] Referring to FIG. 11, in a multi-compensation pixel encoder (1110), a first subtraction tensor obtained by subtracting (1111) a prediction tensor (1101) of the original resolution from the original image (1100) is input to a neural network (1112).

[0197] An intermediate encoding tensor of a first downscaled resolution, in which the original resolution is downscaled, is output through a neural network (1112). A second subtracted tensor, which is obtained by subtracting (1113) the prediction tensor (1102) of the first downscaled resolution from the intermediate encoding tensor of the first downscaled resolution, is input to a neural network (1114).

[0198] An intermediate encoding tensor of a second downscaled resolution, in which the first downscaled resolution is downscaled, is output through a neural network (1114). A third subtracted tensor, which is obtained by subtracting (1115) the prediction tensor (1103) of the second downscaled resolution from the intermediate encoding tensor of the second downscaled resolution, is input to a neural network (1116).

[0199] An intermediate encoding tensor of a third downscaled resolution, in which the second downscaled resolution is downscaled, is output through a neural network (1116). A fourth subtracted tensor, which is obtained by subtracting (1117) the prediction tensor (1104) of the third downscaled resolution from the intermediate encoding tensor of the third downscaled resolution, is input to a neural network (1118).

[0200] Residual image feature data (1130) is output through a neural network (1118). Each of the multiple neural networks (1112, 1114, 1116, 1118) within the multi-compensation pixel encoder (1110) may include at least one convolutional layer. In addition, the multiple neural networks (1112, 1114, 1116, 1118) may downscale input data and output it.

[0201] In the multi-compensation pixel decoder (1120), residual image feature data (1130) and an intermediate prediction tensor (1131) of a fourth downscaled resolution, which is a downscaled third downscaled resolution, are first input to the neural network (1121). The residual image feature data (1130) and the intermediate prediction tensor (1131) may be input in a concatenated manner. The intermediate prediction tensor (1131) of the fourth downscaled resolution corresponds to the intermediate prediction tensor (440) of FIG. 4, the intermediate prediction tensor (540) of FIG. 5, or the intermediate prediction tensor (640) of FIG. 6.

[0202] A residual tensor of a third downscale resolution is obtained through a neural network (1121). The residual tensor of the third downscale resolution is summed (1122) with the prediction tensor (1104) of the third downscale resolution and input to a neural network (1123). In addition, an intermediate prediction tensor (1132) of the third downscale resolution is also input to the neural network (1123). The result data of the summation (1122) and the intermediate prediction tensor (1132) may be concatenated and input. The intermediate prediction tensor (1132) of the third downscale resolution corresponds to the intermediate prediction tensor (430) of FIG. 4, the intermediate prediction tensor (530) of FIG. 5, or the intermediate prediction tensor (630) of FIG. 6.

[0203] A residual tensor of the second downscale resolution is obtained through a neural network (1123). The residual tensor of the second downscale resolution is summed (1124) with the prediction tensor (1103) of the second downscale resolution and input to the neural network (1125). In addition, an intermediate prediction tensor (1133) of the second downscale resolution is also input to the neural network (1125). The result data of the summation (1124) and the intermediate prediction tensor (1133) may be concatenated and input. The intermediate prediction tensor (1133) of the second downscale resolution corresponds to the intermediate prediction tensor (420) of FIG. 4, the intermediate prediction tensor (520) of FIG. 5, or the intermediate prediction tensor (620) of FIG. 6.

[0204] A residual tensor of the first downscale resolution is obtained through a neural network (1125). The residual tensor of the first downscale resolution is summed (1126) with the prediction tensor (1102) of the first downscale resolution and input to a neural network (1127). In addition, an intermediate prediction tensor (1134) of the first downscale resolution is also input to the neural network (1127). The result data of the summation (1126) and the intermediate prediction tensor (1134) may be concatenated and input. The intermediate prediction tensor (1134) of the first downscale resolution corresponds to the intermediate prediction tensor (410) of FIG. 4, the intermediate prediction tensor (510) of FIG. 5, or the intermediate prediction tensor (610) of FIG. 6.

[0205] A residual tensor of the original resolution is obtained through a neural network (1127). The residual tensor of the original resolution is added (1128) to the predicted tensor of the original resolution (1101). A restored image (1150) is output as a result of the addition (1128).

[0206] The neural networks (1121, 1123, 1125, 1127) within the multi-compensation pixel decoder (1120) may include at least one convolutional layer. Additionally, the neural networks (1121, 1123, 1125, 1127) may upscale input data.

[0207] The prediction tensors (1101, 1102, 1103, 1104) correspond to the prediction tensors (401, 402, 403, 404) of FIG. 4, the prediction tensors (501, 502, 503, 504) of FIG. 5, or the prediction tensors (601, 602, 603, 604) of FIG. 6.

[0208] Additionally, the multi-compensation pixel decoder (1120) may be input with feature data of the current optical flow in addition to residual image feature data (1130). The residual image feature data (1130) and the feature data of the current optical flow may be input in a concatenated manner.

[0209] FIG. 12 is a diagram illustrating the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0210] Referring to Fig. 12, in the multi-compensation pixel encoder (1210), the original image (1200) and the predicted residual tensor (1244) of the original resolution are first input to the neural network (1212). The original image (1200) and the predicted residual tensor (1244) may be input in a concatenated manner. The predicted residual tensor (1244) of the original resolution corresponds to the predicted residual tensor (407) of Fig. 4, the predicted residual tensor (507) of Fig. 5, or the predicted residual tensor (607) of Fig. 6.

[0211] An intermediate encoding tensor of a first downscaled resolution, in which the original resolution is downscaled, is output through a neural network (1212). The intermediate encoding tensor of the first downscaled resolution and the prediction residual tensor (1243) of the first downscaled resolution are input to the neural network (1214). The intermediate encoding tensor of the first downscaled resolution and the prediction residual tensor (1243) may be input in a concatenated manner. The prediction residual tensor (1243) of the first downscaled resolution corresponds to the prediction residual tensor (417) of FIG. 4, the prediction residual tensor (517) of FIG. 5, or the prediction residual tensor (617) of FIG. 6.

[0212] An intermediate encoding tensor of a second downscaled resolution, which is a downscaled version of the first downscaled resolution, is output through a neural network (1214). The intermediate encoding tensor of the second downscaled resolution and the prediction residual tensor (1242) of the second downscaled resolution are input to the neural network (1216). The intermediate encoding tensor of the second downscaled resolution and the prediction residual tensor (1242) may be input in a concatenated manner. The prediction residual tensor (1242) of the second downscaled resolution corresponds to the prediction residual tensor (427) of FIG. 4, the prediction residual tensor (527) of FIG. 5, or the prediction residual tensor (627) of FIG. 6.

[0213] An intermediate encoding tensor of a third downscaled resolution, which is a downscaled version of the second downscaled resolution, is output through a neural network (1216). The intermediate encoding tensor of the third downscaled resolution and the prediction residual tensor (1241) of the third downscaled resolution are input to the neural network (1218). The intermediate encoding tensor of the third downscaled resolution and the prediction residual tensor (1241) may be input in a concatenated manner. The prediction residual tensor (1241) of the third downscaled resolution corresponds to the prediction residual tensor (437) of FIG. 4, the prediction residual tensor (537) of FIG. 5, or the prediction residual tensor (637) of FIG. 6.

[0214] Residual image feature data (1230) is output through a neural network (1218). Each of the multiple neural networks (1212, 1214, 1216, 1218) within the multi-compensation pixel encoder (1210) may include at least one convolutional layer. Additionally, the multiple neural networks (1212, 1214, 1216, 1218) may downscale input data and output it.

[0215] In the multi-compensation pixel decoder (1220), residual image feature data (1230) and an intermediate prediction tensor (1231) of a fourth downscaled resolution, which is a downscaled third downscaled resolution, are first input to the neural network (1221). The residual image feature data (1230) and the intermediate prediction tensor (1231) may be input in a concatenated manner. The intermediate prediction tensor (1231) of the fourth downscaled resolution corresponds to the intermediate prediction tensor (440) of FIG. 4, the intermediate prediction tensor (540) of FIG. 5, or the intermediate prediction tensor (640) of FIG. 6.

[0216] A residual tensor of a third downscale resolution is obtained through a neural network (1221). The residual tensor of the third downscale resolution and the intermediate prediction tensor (1232) of the third downscale resolution are input to the neural network (1223). The residual tensor of the third downscale resolution and the intermediate prediction tensor (1232) may be input in a concatenated manner. The intermediate prediction tensor (1232) of the third downscale resolution corresponds to the intermediate prediction tensor (430) of FIG. 4, the intermediate prediction tensor (530) of FIG. 5, or the intermediate prediction tensor (630) of FIG. 6.

[0217] A residual tensor of the second downscale resolution is obtained through a neural network (1223). The residual tensor of the second downscale resolution and the intermediate prediction tensor (1233) of the second downscale resolution are input to the neural network (1225). The residual tensor of the second downscale resolution and the intermediate prediction tensor (1233) may be input in a concatenated form. The intermediate prediction tensor (1233) of the second downscale resolution corresponds to the intermediate prediction tensor (420) of FIG. 4, the intermediate prediction tensor (520) of FIG. 5, or the intermediate prediction tensor (620) of FIG. 6.

[0218] A residual tensor of the first downscale resolution is obtained through a neural network (1225). The residual tensor of the first downscale resolution and the intermediate prediction tensor (1234) of the first downscale resolution are input to a neural network (1227). The residual tensor of the first downscale resolution and the intermediate prediction tensor (1234) may be input in a concatenated manner. The intermediate prediction tensor (1234) of the first downscale resolution corresponds to the intermediate prediction tensor (410) of FIG. 4, the intermediate prediction tensor (510) of FIG. 5, or the intermediate prediction tensor (610) of FIG. 6.

[0219] A restored image (1250) of the original resolution is output through the neural network (1227).

[0220] The neural networks (1221, 1223, 1225, 1227) within the multi-compensation pixel decoder (1220) may include at least one convolutional layer. Additionally, the neural networks (1221, 1223, 1225, 1227) may upscale input data.

[0221] Additionally, the multi-compensation pixel decoder (1220) may be input with feature data of the current optical flow in addition to residual image feature data (1230). The residual image feature data (1230) and the feature data of the current optical flow may be input in a concatenated manner.

[0222] FIG. 13 is a diagram illustrating the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0223] Referring to Fig. 13, in the multi-compensation pixel encoder (1310), the original image (1300) and the predicted residual tensor (1344) of the original resolution are first input to the neural network (1312). The original image (1300) and the predicted residual tensor (1344) may be input in a concatenated manner. The predicted residual tensor (1344) of the original resolution corresponds to the predicted residual tensor (407) of Fig. 4, the predicted residual tensor (507) of Fig. 5, or the predicted residual tensor (607) of Fig. 6.

[0224] An intermediate encoding tensor of a first downscaled resolution, in which the original resolution is downscaled, is output through a neural network (1312). The intermediate encoding tensor of the first downscaled resolution and the prediction residual tensor (1343) of the first downscaled resolution are input to the neural network (1314). The intermediate encoding tensor of the first downscaled resolution and the prediction residual tensor (1343) may be input in a concatenated manner. The prediction residual tensor (1343) of the first downscaled resolution corresponds to the prediction residual tensor (417) of FIG. 4, the prediction residual tensor (517) of FIG. 5, or the prediction residual tensor (617) of FIG. 6.

[0225] An intermediate encoding tensor of a second downscaled resolution, which is a downscaled version of the first downscaled resolution, is output through a neural network (1314). The intermediate encoding tensor of the second downscaled resolution and the prediction residual tensor (1342) of the second downscaled resolution are input to the neural network (1316). The intermediate encoding tensor of the second downscaled resolution and the prediction residual tensor (1342) may be input in a concatenated manner. The prediction residual tensor (1342) of the second downscaled resolution corresponds to the prediction residual tensor (427) of FIG. 4, the prediction residual tensor (527) of FIG. 5, or the prediction residual tensor (627) of FIG. 6.

[0226] An intermediate encoding tensor of a third downscaled resolution, which is a downscaled version of the second downscaled resolution, is output through a neural network (1316). The intermediate encoding tensor of the third downscaled resolution and the prediction residual tensor (1341) of the third downscaled resolution are input to a neural network (1318). The intermediate encoding tensor of the third downscaled resolution and the prediction residual tensor (1341) may be input in a concatenated manner. The prediction residual tensor (1341) of the third downscaled resolution corresponds to the prediction residual tensor (437) of FIG. 4, the prediction residual tensor (537) of FIG. 5, or the prediction residual tensor (637) of FIG. 6.

[0227] Residual image feature data (1330) is output through a neural network (1318). Each of the multiple neural networks (1312, 1314, 1316, 1318) within the multi-compensation pixel encoder (1310) may include at least one convolutional layer. In addition, the multiple neural networks (1312, 1314, 1316, 1318) may downscale input data and output it.

[0228] In the multi-compensation pixel decoder (1320), residual image feature data (1330) is output as a restored image (1350) of the original resolution through multiple neural networks (1321, 1323, 1325, 1327).

[0229] The neural networks (1321, 1323, 1325, 1327) within the multi-compensation pixel decoder (1320) may include at least one convolutional layer. Additionally, the neural networks (1321, 1323, 1325, 1327) may upscale input data.

[0230] Additionally, the multi-compensation pixel decoder (1320) may be input with feature data of the current optical flow in addition to residual image feature data (1330). The residual image feature data (1330) and the feature data of the current optical flow may be input in a concatenated manner.

[0231] FIG. 14 is a diagram for explaining the structure of a multi-pixel compensation encoder and a multi-pixel compensation decoder according to one embodiment of the present disclosure.

[0232] Referring to FIG. 14, in a multi-compensation pixel encoder (1410), an original image (1400) is output as residual image feature data (1430) through multiple neural networks (1412, 1414, 1416, 1418).

[0233] Each of the multiple neural networks (1412, 1414, 1416, 1418) within the multi-compensation pixel encoder (1410) may include at least one convolutional layer. Additionally, the multiple neural networks (1412, 1414, 1416, 1418) may downscale input data and output it.

[0234] In the multi-compensation pixel decoder (1420), residual image feature data (1430) and an intermediate prediction tensor (1431) of a fourth downscaled resolution, which is a downscaled third downscaled resolution, are first input to the neural network (1421). The residual image feature data (1430) and the intermediate prediction tensor (1431) may be input in a concatenated manner. The intermediate prediction tensor (1431) of the fourth downscaled resolution corresponds to the intermediate prediction tensor (440) of FIG. 4, the intermediate prediction tensor (540) of FIG. 5, or the intermediate prediction tensor (640) of FIG. 6.

[0235] A residual tensor of a third downscale resolution is obtained through a neural network (1421). The residual tensor of the third downscale resolution and the intermediate prediction tensor (1432) of the third downscale resolution are input to the neural network (1423). The residual tensor of the third downscale resolution and the intermediate prediction tensor (1432) may be input in a concatenated manner. The intermediate prediction tensor (1432) of the third downscale resolution corresponds to the intermediate prediction tensor (430) of FIG. 4 , the intermediate prediction tensor (530) of FIG. 5 , or the intermediate prediction tensor (630) of FIG. 6 .

[0236] A residual tensor of the second downscale resolution is obtained through a neural network (1423). The residual tensor of the second downscale resolution and the intermediate prediction tensor (1433) of the second downscale resolution are input to the neural network (1425). The residual tensor of the second downscale resolution and the intermediate prediction tensor (1433) may be input in a concatenated form. The intermediate prediction tensor (1433) of the second downscale resolution corresponds to the intermediate prediction tensor (420) of FIG. 4, the intermediate prediction tensor (520) of FIG. 5, or the intermediate prediction tensor (620) of FIG. 6.

[0237] A residual tensor of the first downscale resolution is obtained through a neural network (1425). The residual tensor of the first downscale resolution and the intermediate prediction tensor (1434) of the first downscale resolution are input to a neural network (1427). The residual tensor of the first downscale resolution and the intermediate prediction tensor (1434) may be input in a concatenated manner. The intermediate prediction tensor (1434) of the first downscale resolution corresponds to the intermediate prediction tensor (410) of FIG. 4, the intermediate prediction tensor (510) of FIG. 5, or the intermediate prediction tensor (610) of FIG. 6.

[0238] A restored image (1250) of the original resolution is output through a neural network (1427).

[0239] The neural networks (1421, 1423, 1425, 1427) within the multi-compensation pixel decoder (1420) may include at least one convolutional layer. Additionally, the neural networks (1421, 1423, 1425, 1427) may upscale input data.

[0240] Additionally, the multi-compensation pixel decoder (1420) may be input with feature data of the current optical flow in addition to residual image feature data (1430). The residual image feature data (1430) and the feature data of the current optical flow may be input in a concatenated manner.

[0241] FIG. 15 is a diagram for explaining the optical flow and prediction error, memory gate value, and forget gate value of an original frame according to one embodiment of the present disclosure.

[0242] Referring to FIG. 15, a prediction frame for the original frame (1500) is obtained based on the optical flow (1505) of the original frame, and a prediction error (1510) may occur between the original frame (1500) and the prediction frame.

[0243] To prevent these prediction errors (1510), memory gate values ​​(1515, 1525, 1535, 1545) and forget gate values ​​(1520, 1530, 1540, 1550) are used.

[0244] In the area where a prediction error (1510) occurred, it is necessary to retain the well-predicted portion and remove the poorly predicted portion. Therefore, the memory gate values ​​(1515, 1525, 1535, 1545) have relatively large values ​​for portions where prediction errors occurred and where information that needs to be retained is contained, and the forget gate values ​​(1520, 1530, 1540, 1550) have relatively large values ​​for portions where prediction errors occurred and where information that needs to be removed is contained.

[0245] FIG. 16 is a flowchart of an image encoding method according to one embodiment of the present disclosure.

[0246] In step S1610, the image encoding device (1700) applies the current image and the previously restored image to a neural network-based first encoder to obtain feature data of the current optical flow.

[0247] In step S1620, the image encoding device (1700) obtains a plurality of memory gate values ​​corresponding to the current optical flow and a plurality of resolutions of the current image by applying the feature data of the current optical flow to a neural network-based first decoder.

[0248] According to one embodiment of the present disclosure, a plurality of memory gate values ​​may represent values ​​for maintaining information within the current image.

[0249] At step S1630, the image encoding device (1700) obtains a predicted image of the current image from the previously restored image based on the current optical flow.

[0250] In step S1640, the image encoding device (1700) obtains a plurality of prediction tensors corresponding to a plurality of resolutions based on the predicted image and a plurality of memory gate values.

[0251] According to one embodiment of the present disclosure, a plurality of forget gate values ​​corresponding to a plurality of resolutions of a current image are additionally obtained from a neural network-based first decoder, and a plurality of prediction tensors are obtained based on the predicted image, the plurality of memory gate values, and the forget gate values, and the plurality of forget gate values ​​may represent values ​​for removing a region in the current image in which a prediction error is greater than a predetermined value.

[0252] According to one embodiment of the present disclosure, a prediction tensor of the original resolution of the current image among a plurality of prediction tensors can be determined based on memory gate values ​​corresponding to the predicted image and the original resolution.

[0253] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a subtraction tensor obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image and a forget gate value corresponding to the original resolution, and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on an intermediate prediction tensor of a resolution downscaled from the original resolution obtained by applying the prediction residual tensor to a downscale neural network and a remember gate value corresponding to the downscaled resolution.

[0254] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a memory gate value corresponding to an original resolution of a current image, a forget gate value corresponding to the original resolution, and a prediction image, and by applying the prediction residual tensor to a downscaled neural network, an intermediate prediction tensor of a downscaled resolution from the original resolution obtained and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on the memory gate value corresponding to the downscaled resolution.

[0255] According to one embodiment of the present disclosure, a prediction residual tensor is obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image, and a prediction tensor of a downscaled resolution is determined among a plurality of prediction tensors based on an intermediate prediction tensor of a downscaled resolution from an original resolution obtained by applying the prediction residual tensor to a downscaled neural network and a memory gate value corresponding to the downscaled resolution, and a convolution kernel of a first layer of the downscaled neural network may be linearly mixed with a forget gate value corresponding to the original resolution.

[0256] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a subtraction tensor obtained by subtracting a prediction tensor of the first downscale resolution from an intermediate prediction tensor of the first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a forget gate value corresponding to the first downscale resolution, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution.

[0257] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a memory gate value corresponding to the first downscale resolution, a forget gate value corresponding to the first downscale resolution, and an intermediate prediction tensor of the first downscale resolution before the memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among the plurality of prediction tensors, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to the downscale neural network, and the memory gate value corresponding to the second downscale resolution.

[0258] According to one embodiment of the present disclosure, a prediction residual tensor of a first downscale resolution is obtained by subtracting a prediction tensor of a first downscale resolution from an intermediate prediction tensor of a first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a prediction tensor of a second downscale resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of a second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution, and a convolution kernel of a first layer of the downscale neural network may be linearly mixed with a forget gate value corresponding to the first downscale resolution.

[0259] At step S1650, the image encoding device (1700) applies a plurality of prediction tensors and the current image to a neural network-based second encoder to obtain feature data for the residual image.

[0260] In step S1660, the image encoding device (1700) applies a plurality of prediction tensors, feature data of the current optical flow, and feature data for the residual image to a neural network-based second decoder to obtain a current restored image corresponding to the current image.

[0261] According to one embodiment of the present disclosure, the predicted residual tensor may be additionally applied to a neural network-based second encoder to obtain feature data for a residual image, and the intermediate predicted tensor may be additionally applied to a neural network-based second decoder to obtain a current restored image.

[0262] At step S1670, the image encoding device (1700) generates a bitstream including feature data of the current optical flow and feature data for the residual image.

[0263] FIG. 17 is a diagram illustrating a configuration of an image encoding device according to one embodiment of the present disclosure.

[0264] Referring to FIG. 17, the image encoding device (1700) may include a prediction encoding unit (1710), a generation unit (1720), an acquisition unit (1730), and a prediction decoding unit (1740).

[0265] The prediction encoding unit (1710), the generation unit (1720), the acquisition unit (1730), and the prediction decoding unit (1740) may be implemented as a processor. The processor may include at least one processing circuit and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to individually and / or collectively perform the various functions described herein in a distributed manner. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. The at least one processor may execute program instructions to achieve or perform various functions. The prediction encoding unit (1710), the generation unit (1720), the acquisition unit (1730), and the prediction decoding unit (1740) can operate according to instructions stored in a memory (not shown).

[0266] Although FIG. 17 illustrates the prediction encoding unit (1710), the generation unit (1720), the acquisition unit (1730), and the prediction decoding unit (1740) individually, the prediction encoding unit (1710), the generation unit (1720), the acquisition unit (1730), and the prediction decoding unit (1740) may be implemented through a single processor. In this case, the prediction encoding unit (1710), the generation unit (1720), the acquisition unit (1730), and the prediction decoding unit (1740) may be implemented as a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software. In addition, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or may include a memory processing unit for utilizing an external memory.

[0267] The prediction encoding unit (1710), the generation unit (1720), the acquisition unit (1730), and the prediction decoding unit (1740) may be configured with multiple processors. In this case, the prediction encoding unit (1710), the generation unit (1720), the acquisition unit (1730), and the prediction decoding unit (1740) may be implemented as a combination of dedicated processors, or may be implemented through a combination of software and a plurality of general-purpose processors such as an AP, a CPU, or a GPU.

[0268] The prediction encoder (1710) may include a motion encoder (1711) and a multi-compensation pixel encoder (1712).

[0269] The motion encoder (1711) and the multi-compensation pixel encoder (1712) can be implemented as a neural network including one or more layers (e.g., convolutional layers).

[0270] The motion encoder (1711) and the multi-compensation pixel encoder (1712) may be stored in memory. The motion encoder (1711) and the multi-compensation pixel encoder (1712) may also be implemented with at least one dedicated processor for AI.

[0271] The prediction encoding unit (1710) can obtain feature data of the current optical flow using the current image and the previous restored image, and can obtain residual image feature data using the current image and a plurality of prediction tensors received from the deep prediction decomposition unit (1743) of the prediction decoding unit (1740). Specifically, the motion encoder (1711) can receive the current image and the previous restored image as input and output feature data of the current optical flow. In addition, the multi-compensation pixel encoder (1712) can receive a plurality of prediction tensors from the current image and the deep prediction decomposition unit (1743) of the prediction decoding unit (1740) as input and output residual image feature data.

[0272] The current optical flow feature data and residual image feature data acquired by the prediction encoding unit (1710) can be transmitted to the generation unit (1720).

[0273] The generation unit (1720) can generate a bitstream including feature data of the current optical flow and residual image feature data.

[0274] In one embodiment, the generation unit (1720) may generate a first bitstream corresponding to feature data of the current optical flow and a second bitstream corresponding to residual image feature data.

[0275] The bitstream may be transmitted to the image decoding device (1900) over a network. Additionally, in one embodiment, the bitstream may be recorded on a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and the like.

[0276] The acquisition unit (1730) can acquire feature data of the current optical flow and residual image feature data from the bitstream generated by the generation unit (1720).

[0277] In one embodiment, the acquisition unit (1730) may receive feature data of the current optical flow and residual image feature data from the prediction encoding unit (1710).

[0278] The current optical flow feature data and residual image feature data can be transmitted to the prediction decoding unit (1740).

[0279] The prediction decoding unit (1740) may include a motion decoder (1741), a motion compensation unit (1742), a deep prediction decomposition unit (1743), and a multi-compensation pixel decoder (1744).

[0280] The motion decoder (1741), the deep prediction decomposition unit (1743), and the multi-compensation pixel decoder (1744) can be implemented as a neural network including one or more layers (e.g., convolutional layers).

[0281] The motion decoder (1741), the deep prediction decomposition unit (1743), and the multi-compensation pixel decoder (1744) may be stored in memory. The motion decoder (1741), the deep prediction decomposition unit (1743), and the multi-compensation pixel decoder (1744) may be implemented as at least one dedicated processor for AI.

[0282] The prediction decoding unit (1740) can obtain a current reconstructed image using the feature data of the current optical flow and the residual image feature data. Specifically, the motion decoder (1741) can receive the current optical flow feature data and output the memory gate values ​​and forget gate values ​​corresponding to the current optical flow and multiple resolutions. The current optical flow can be transmitted to the motion compensation unit (1742), and the memory gate values ​​and forget gate values ​​corresponding to the multiple resolutions can be transmitted to the deep prediction decomposition unit (1743). The motion compensation unit (1742) can obtain a predicted image by performing warping using the previous reconstructed image and the current optical flow. The predicted image can be transmitted to the deep prediction decomposition unit (1743). The deep prediction decomposition unit (1743) can obtain a plurality of prediction tensors using the memory gate values ​​and forget gate values ​​corresponding to the predicted image and multiple resolutions. A plurality of prediction tensors can be transmitted to a multi-compensation pixel encoder (1712) of a prediction encoding unit (1710) and a multi-compensation pixel decoder (1744) of a prediction decoding unit (1740). The multi-compensation pixel decoder (1744) can obtain a current restored image using feature data of the current optical flow, residual image feature data, and a plurality of prediction tensors.

[0283] The specific operations of the motion encoder (1711), the multi-compensation pixel encoder (1712), the motion decoder (1741), the deep prediction decomposition unit (1743), and the multi-compensation pixel decoder (1744) are omitted as they have been described above in FIGS. 3 to 14.

[0284] FIG. 18 is a flowchart of an image decoding method according to one embodiment of the present disclosure.

[0285] In step S1810, the image decoding device (1900) obtains feature data of the current optical flow and feature data for the residual image of the current image from the bitstream.

[0286] In step S1820, the image decoding device (1900) obtains a plurality of memory gate values ​​corresponding to the current optical flow and a plurality of resolutions of the current image by applying the feature data of the current optical flow to a neural network-based first decoder.

[0287] According to one embodiment of the present disclosure, a plurality of memory gate values ​​may represent values ​​for maintaining information within the current image.

[0288] At step S1830, the image decoding device (1900) obtains a predicted image of the current image from the previously restored image based on the current optical flow.

[0289] In step S1840, the image decoding device (1900) obtains a plurality of prediction tensors corresponding to a plurality of resolutions based on the predicted image and a plurality of memory gate values.

[0290] According to one embodiment of the present disclosure, a plurality of forget gate values ​​corresponding to a plurality of resolutions of a current image are additionally obtained from a neural network-based first decoder, and a plurality of prediction tensors are obtained based on the predicted image, the plurality of memory gate values, and the forget gate values, and the plurality of forget gate values ​​may represent values ​​for removing a region in the current image in which a prediction error is greater than a predetermined value.

[0291] According to one embodiment of the present disclosure, a prediction tensor of the original resolution of the current image among a plurality of prediction tensors can be determined based on memory gate values ​​corresponding to the predicted image and the original resolution.

[0292] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a subtraction tensor obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image and a forget gate value corresponding to the original resolution, and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on an intermediate prediction tensor of a resolution downscaled from the original resolution obtained by applying the prediction residual tensor to a downscale neural network and a remember gate value corresponding to the downscaled resolution.

[0293] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a memory gate value corresponding to an original resolution of a current image, a forget gate value corresponding to the original resolution, and a prediction image, and by applying the prediction residual tensor to a downscaled neural network, an intermediate prediction tensor of a downscaled resolution from the original resolution obtained and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on the memory gate value corresponding to the downscaled resolution.

[0294] According to one embodiment of the present disclosure, a prediction residual tensor is obtained by subtracting a second prediction tensor of an original resolution of a current image from a prediction image, and a prediction tensor of a downscaled resolution is determined among a plurality of prediction tensors based on an intermediate prediction tensor of a downscaled resolution from the original resolution obtained by applying the prediction residual tensor to a downscaled neural network and a memory gate value corresponding to the downscaled resolution, and a convolution kernel of a first layer of the downscaled neural network may be linearly mixed with a forget gate value corresponding to the original resolution.

[0295] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a subtraction tensor obtained by subtracting a prediction tensor of the first downscale resolution from an intermediate prediction tensor of the first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a forget gate value corresponding to the first downscale resolution, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution.

[0296] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a memory gate value corresponding to the first downscale resolution, a forget gate value corresponding to the first downscale resolution, and an intermediate prediction tensor of the first downscale resolution before the memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among the plurality of prediction tensors, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to the downscale neural network, and the memory gate value corresponding to the second downscale resolution.

[0297] According to one embodiment of the present disclosure, a prediction residual tensor of a first downscale resolution is obtained by subtracting a prediction tensor of a first downscale resolution from an intermediate prediction tensor of a first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a prediction tensor of a second downscale resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of a second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution, and a convolution kernel of a first layer of the downscale neural network may be linearly mixed with a forget gate value corresponding to the first downscale resolution.

[0298] At step S1850, the image decoding device (1900) applies a plurality of prediction tensors, feature data of the current optical flow, and feature data for the residual image to a neural network-based second decoder to obtain a current restored image corresponding to the current image.

[0299] According to one embodiment of the present disclosure, the intermediate prediction tensor can be additionally applied to a neural network-based second decoder to obtain a current restored image.

[0300] FIG. 19 is a diagram illustrating a configuration of an image decoding device according to one embodiment of the present disclosure.

[0301] Referring to FIG. 19, the image decoding device (1900) may include an acquisition unit (1910) and a prediction decoding unit (1920).

[0302] The acquisition unit (1910) and the prediction decoding unit (1920) may be implemented as a processor. The processor may include at least one processing circuit and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to individually and / or collectively perform the various functions described herein in a distributed manner. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. The at least one processor may execute program instructions to achieve or perform various functions. The acquisition unit (1910) and the prediction decoding unit (1920) can operate according to instructions stored in memory (not shown).

[0303] Although FIG. 19 illustrates the acquisition unit (1910) and the prediction decoding unit (1920) separately, the acquisition unit (1910) and the prediction decoding unit (1920) may be implemented through a single processor. In this case, the acquisition unit (1910) and the prediction decoding unit (1920) may be implemented through a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software. In addition, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or a memory processing unit for utilizing an external memory.

[0304] The acquisition unit (1910) and the prediction decoding unit (1920) may be configured with multiple processors. In this case, the acquisition unit (1910) and the prediction decoding unit (1920) may be implemented as a combination of dedicated processors, or may be implemented through a combination of multiple general-purpose processors, such as an AP, CPU, or GPU, and software.

[0305] The acquisition unit (1910) can acquire feature data of the current optical flow and residual image feature data from the bitstream.

[0306] The current optical flow feature data and residual image feature data can be transmitted to the prediction decoding unit (1920).

[0307] The prediction decoding unit (1920) may include a motion decoder (1921), a motion compensation unit (1922), a deep prediction decomposition unit (1923), and a multi-compensation pixel decoder (1924).

[0308] The motion decoder (1921), the deep prediction decomposition unit (1923), and the multi-compensation pixel decoder (1924) can be implemented as a neural network including one or more layers (e.g., convolutional layers).

[0309] The motion decoder (1921), the deep prediction decomposition unit (1923), and the multi-compensation pixel decoder (1924) may be stored in memory. The motion decoder (1921), the deep prediction decomposition unit (1923), and the multi-compensation pixel decoder (1924) may be implemented as at least one dedicated processor for AI.

[0310] The prediction decoding unit (1920) can obtain a current restored image using the feature data of the current optical flow and the residual image feature data. Specifically, the motion decoder (1921) can receive the current optical flow feature data and output the memory gate values ​​and the forget gate values ​​corresponding to the current optical flow and multiple resolutions. The current optical flow can be transmitted to the motion compensation unit (1922), and the memory gate values ​​and the forget gate values ​​corresponding to the multiple resolutions can be transmitted to the deep prediction decomposition unit (1743). The motion compensation unit (1922) can obtain a predicted image by performing warping using the previous restored image and the current optical flow. The predicted image can be transmitted to the deep prediction decomposition unit (1923). The deep prediction decomposition unit (1923) can obtain a plurality of prediction tensors using the predicted image and the memory gate values ​​and the forget gate values ​​corresponding to the multiple resolutions. A multi-compensation pixel decoder (1924) can obtain a current restored image using feature data of the current optical flow, residual image feature data, and multiple prediction tensors.

[0311] The specific operations of the motion decoder (1921), the deep prediction decomposition unit (1923), and the multi-compensation pixel decoder (1924) are omitted as they have been described above in FIGS. 3 to 14.

[0312] FIG. 20 is a diagram illustrating a training method of a neural network of a motion encoder, a motion decoder, a deep prediction decomposition, a multi-compensation pixel encoder, and a multi-compensation pixel decoder.

[0313] In Fig. 20, the current training image (2000), the previous restored training image (2005), and the restored training image (2070) correspond to the current image, the previous restored image, and the restored image described above, respectively.

[0314] In training the neural networks of the motion encoder (2010), motion decoder (2020), deep predictive decomposition (2040), multi-compensation pixel encoder (2050), and multi-compensation pixel decoder (2060), it should be considered how similar the reconstructed training image (2070) is to the current training image (2000), and how large the bitrate of the bitstream generated through encoding the current training image (2000) is.

[0315] To this end, in one embodiment, the neural networks of the motion encoder (2010), the motion decoder (2020), the deep prediction decomposition (2040), the multi-compensation pixel encoder (2050), and the multi-compensation pixel decoder (2060) can be trained according to the first loss information (20800) and the second loss information (2085) corresponding to the size of the bitstream, and the third loss information (2090) corresponding to the similarity between the current training image (2000) and the reconstructed training image (2070).

[0316] Referring to FIG. 20, the current training image (2000) and the previous restored training image (2005) can be input to a motion encoder (2010). The optical flow encoder (2010) can process the current training image (2000) and the previous restored training image (2005) to output feature data (2011) of the current optical flow.

[0317] The feature data (2011) of the current optical flow can be input to a motion decoder (2020), and the motion decoder (2020) can process the feature data (2011) of the current optical flow to output the memory gate values ​​and forget gate values ​​(2022) corresponding to the current optical flow (2021) and multiple resolutions.

[0318] The current prediction training image (2031) can be generated by warping (2030) the previous restoration training image (2005) according to the current optical flow (2021).

[0319] The current prediction training image (2031) and the memory gate values ​​and forget gate values ​​(2022) corresponding to multiple resolutions can be used in a deep prediction decomposition process (2040) based on a neural network. Through the deep prediction decomposition process (2040), multiple prediction training tensors (2041) corresponding to multiple resolutions can be output.

[0320] The current training image (2420) and multiple prediction training tensors (2041) are input to a multi-compensation pixel encoder (2050), and residual image feature data (2051) can be output from the multi-compensation pixel encoder (2050).

[0321] The current optical flow feature data (2011), residual image feature data (2051), and multiple prediction training tensors (2041) are input to a multi-compensation pixel decoder (2060), and a restored training image (2070) can be output from the multi-compensation pixel decoder (2060).

[0322] For training the neural networks of the motion encoder (2010), the motion decoder (2020), the deep prediction decomposition (2040), the multi-compensation pixel encoder (2050), and the multi-compensation pixel decoder (2060), at least one of the first loss information (2080), the second loss information (2085), and the third loss information (2090) can be obtained.

[0323] The first loss information (2080) can be derived from the entropy of the feature data (2011) of the current optical flow or the bitrate of the bitstream corresponding to the feature data (2011) of the current optical flow.

[0324] The second loss information (2085) can be derived from the entropy of the residual image feature data (2051) or the bitrate of the bitstream corresponding to the residual image feature data (2051).

[0325] Since the first loss information (2080) and the second loss information (2085) are related to the encoding efficiency for the current training image (2000), the first loss information (2080) and the second loss information (2085) can be referred to as compression loss information.

[0326] In one embodiment, FIG. 20 illustrates that first loss information (2080) and second loss information (2085) are derived in relation to the bitrate of the bitstream, but it is also possible to derive one loss information corresponding to the bitrate of one bitstream generated through encoding of the current training image (2000).

[0327] The third loss information (2090) may correspond to a difference between the current training image (2000) and the restored training image (2070). That is, it may be obtained through a comparison (2075) between the current training image (2000) and the restored training image (2070). The difference between the current training image (2000) and the restored training image (2070) may include at least one of an L1-norm value, an L2-norm value, an SSIM (Structural Similarity) value, a PSNR-HVS (Peak Signal-To-Noise Ratio-Human Vision System) value, an MS-SSIM (Multiscale SSIM) value, a VIF (Variance Inflation Factor) value, and a VMAF (Video Multimethod Assessment Fusion) value between the current training image (2000) and the restored training image (2070).

[0328] Since the third loss information (2090) is related to the quality of the restored training image (2070), the third loss information (2090) may also be referred to as quality loss information.

[0329] The neural networks of the motion encoder (2010), the motion decoder (2020), the deep prediction decomposition (2040), the multi-compensation pixel encoder (2050), and the multi-compensation pixel decoder (2060) can be trained such that the final loss information derived from at least one of the first loss information (2080), the second loss information (2085), and the third loss information (2090) is reduced or minimized.

[0330] Specifically, the neural networks of the motion encoder (2010), the motion decoder (2020), the deep prediction decomposition (2040), the multi-compensation pixel encoder (2050), and the multi-compensation pixel decoder (2060) can reduce or minimize the final loss information while changing the values ​​of the preset parameters.

[0331] In one embodiment, the final loss information can be calculated according to the following mathematical expression 1.

[0332] [Mathematical Formula 1]

[0333] Final loss information = a*first loss information+b*second loss information+c*third loss information

[0334] In mathematical expression 1, a, b, and c are weights applied to the first loss information (2080), the second loss information (2085), and the third loss information (2090), respectively.

[0335] According to mathematical expression 1, it can be seen that the neural networks of the motion encoder (2010), the motion decoder (2020), the deep prediction decomposition (2040), the multi-compensation pixel encoder (2050), and the multi-compensation pixel decoder (2060) are trained in a direction such that the restored training image (2070) becomes as similar as possible to the current training image (2000), and the size of the bitstream corresponding to the data output from the motion encoder (2010) and the multi-compensation pixel encoder (2050) is minimized.

[0336] According to one embodiment, an image decoding method may include the steps of: obtaining feature data of a current optical flow and feature data for a residual image of the current image from a bitstream; obtaining a plurality of remembering gate values ​​corresponding to a plurality of resolutions of the current optical flow and the current image by applying the feature data of the current optical flow to a neural network-based first decoder; obtaining a predicted image of the current image from a previously restored image based on the current optical flow; obtaining a plurality of prediction tensors corresponding to a plurality of resolutions based on the predicted image and the plurality of remembering gate values; and obtaining a current restored image corresponding to the current image by applying the plurality of prediction tensors, the feature data of the current optical flow, and the feature data for the residual image to a neural network-based second decoder.

[0337] According to one embodiment of the present disclosure, the plurality of memory gate values ​​may represent values ​​for maintaining information within the current image.

[0338] An image decoding method according to one embodiment of the present disclosure restores an image based on a plurality of prediction tensors corresponding to a plurality of resolutions obtained using a plurality of memory gate values ​​representing values ​​for maintaining key information in an image, thereby maintaining key information of the image, thereby preventing errors in the restored image and improving the accuracy and coding efficiency of the image.

[0339] According to one embodiment of the present disclosure, a plurality of forget gate values ​​corresponding to a plurality of resolutions of a current image are additionally obtained from a neural network-based first decoder, a plurality of prediction tensors are obtained based on the first predicted image, the plurality of memory gate values, and the forget gate values, and the plurality of forget gate values ​​may represent values ​​for removing a region in the current image in which a prediction error is greater than a predetermined value.

[0340] An image decoding method according to one embodiment of the present disclosure obtains a plurality of prediction tensors by additionally using a plurality of forget gate values ​​representing values ​​for removing a region in which a prediction error of an image is greater than a predetermined value, i.e., a region in which prediction is poorly made, and restores an image based on a plurality of prediction tensors corresponding to a plurality of resolutions, thereby removing unnecessary information of the image, thereby preventing an error in a restored image and improving the accuracy and coding efficiency of the image.

[0341] According to one embodiment of the present disclosure, a prediction tensor of the original resolution of the current image among a plurality of prediction tensors can be determined based on memory gate values ​​corresponding to the predicted image and the original resolution.

[0342] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a subtraction tensor obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image and a forget gate value corresponding to the original resolution, and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on an intermediate prediction tensor of a resolution downscaled from the original resolution obtained by applying the prediction residual tensor to a downscale neural network and a remember gate value corresponding to the downscaled resolution.

[0343] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a memory gate value corresponding to an original resolution of a current image, a forget gate value corresponding to the original resolution, and a prediction image, and by applying the prediction residual tensor to a downscaled neural network, an intermediate prediction tensor of a downscaled resolution from the original resolution obtained and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on the memory gate value corresponding to the downscaled resolution.

[0344] According to one embodiment of the present disclosure, a prediction residual tensor is obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image, and a prediction tensor of a downscaled resolution is determined among a plurality of prediction tensors based on an intermediate prediction tensor of a downscaled resolution from an original resolution obtained by applying the prediction residual tensor to a downscaled neural network and a memory gate value corresponding to the downscaled resolution, and a convolution kernel of a first layer of the downscaled neural network may be linearly mixed with a forget gate value corresponding to the original resolution.

[0345] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a subtraction tensor obtained by subtracting a prediction tensor of the first downscale resolution from an intermediate prediction tensor of the first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a forget gate value corresponding to the first downscale resolution, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution.

[0346] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a memory gate value corresponding to the first downscale resolution, a forget gate value corresponding to the first downscale resolution, and an intermediate prediction tensor of the first downscale resolution before the memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among the plurality of prediction tensors, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to the downscale neural network, and the memory gate value corresponding to the second downscale resolution.

[0347] According to one embodiment of the present disclosure, a prediction residual tensor of a first downscale resolution is obtained by subtracting a prediction tensor of a first downscale resolution from an intermediate prediction tensor of a first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a prediction tensor of a second downscale resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of a second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution, and a convolution kernel of a first layer of the downscale neural network may be linearly mixed with a forget gate value corresponding to the first downscale resolution.

[0348] An image decoding method according to one embodiment of the present disclosure decomposes a prediction image of a current image into prediction tensors of multiple resolutions using memory gate values ​​and forget gate values ​​corresponding to multiple resolutions, and restores the image based on the multiple prediction tensors corresponding to the multiple resolutions, thereby maintaining the main information of the image and removing unnecessary information to prevent errors, and improving the accuracy and coding efficiency of the image.

[0349] According to one embodiment of the present disclosure, the intermediate prediction tensor can be additionally applied to a neural network-based second decoder to obtain a current restored image.

[0350] An image decoding method according to one embodiment of the present disclosure can improve image accuracy and coding efficiency by utilizing additional information in residual coding.

[0351] According to one embodiment, an image decoding device includes a memory storing one or more instructions; and at least one processor operating according to the one or more instructions, wherein the at least one processor obtains feature data of a current optical flow and feature data for a residual image of a current image from a bitstream, applies the feature data of the current optical flow to a neural network-based first decoder to obtain a plurality of remembering gate values ​​corresponding to a plurality of resolutions of the current optical flow and the current image, obtains a predicted image of the current image from a previous restored image based on the current optical flow, obtains a plurality of prediction tensors corresponding to a plurality of resolutions based on the predicted image and the plurality of remembering gate values, and applies the plurality of prediction tensors, the feature data of the current optical flow, and the feature data for the residual image to a neural network-based second decoder to obtain a current restored image corresponding to the current image.

[0352] According to one embodiment of the present disclosure, the plurality of memory gate values ​​may represent values ​​for maintaining information within the current image.

[0353] An image decoding device according to one embodiment of the present disclosure restores an image based on a plurality of prediction tensors corresponding to a plurality of resolutions obtained using a plurality of memory gate values ​​representing values ​​for maintaining key information in an image, thereby maintaining key information of the image, thereby preventing errors in the restored image and improving the accuracy and coding efficiency of the image.

[0354] According to one embodiment of the present disclosure, a plurality of forget gate values ​​corresponding to a plurality of resolutions of a current image are additionally obtained from a neural network-based first decoder, a plurality of prediction tensors are obtained based on the predicted image, the plurality of memory gate values, and the forget gate values, and the plurality of forget gate values ​​may represent values ​​for removing a region in the current image in which a prediction error is greater than a predetermined value.

[0355] An image decoding device according to one embodiment of the present disclosure obtains a plurality of prediction tensors by additionally using a plurality of forget gate values ​​representing values ​​for removing an area in which the prediction error of the image is greater than a predetermined value, and restores the image based on the plurality of prediction tensors corresponding to a plurality of resolutions, thereby removing unnecessary information of the image, thereby preventing the occurrence of errors in the restored image, and improving the accuracy and coding efficiency of the image.

[0356] According to one embodiment of the present disclosure, a prediction tensor of the original resolution of the current image among a plurality of prediction tensors can be determined based on memory gate values ​​corresponding to the predicted image and the original resolution.

[0357] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a subtraction tensor obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image and a forget gate value corresponding to the original resolution, and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on an intermediate prediction tensor of a resolution downscaled from the original resolution obtained by applying the prediction residual tensor to a downscale neural network and a remember gate value corresponding to the downscaled resolution.

[0358] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a memory gate value corresponding to an original resolution of a current image, a forget gate value corresponding to the original resolution, and a prediction image, and by applying the prediction residual tensor to a downscaled neural network, an intermediate prediction tensor of a downscaled resolution from the original resolution obtained and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on the memory gate value corresponding to the downscaled resolution.

[0359] According to one embodiment of the present disclosure, a prediction residual tensor is obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image, and the prediction residual tensor is applied to a downscaled neural network, and a prediction tensor of a downscaled resolution is determined among a plurality of prediction tensors based on an intermediate prediction tensor of a downscaled resolution from the original resolution obtained by applying the prediction residual tensor to the downscaled neural network and a memory gate value corresponding to the downscaled resolution, and a convolution kernel of a first layer of the downscaled neural network may be linearly mixed with a forget gate value corresponding to the original resolution.

[0360] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a subtraction tensor obtained by subtracting a prediction tensor of the first downscale resolution from an intermediate prediction tensor of the first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a forget gate value corresponding to the first downscale resolution, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution.

[0361] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a memory gate value corresponding to the first downscale resolution, a forget gate value corresponding to the first downscale resolution, and an intermediate prediction tensor of the first downscale resolution before the memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among the plurality of prediction tensors, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to the downscale neural network, and the memory gate value corresponding to the second downscale resolution.

[0362] According to one embodiment of the present disclosure, a prediction residual tensor of a first downscale resolution is obtained by subtracting a prediction tensor of a first downscale resolution from an intermediate prediction tensor of a first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a prediction tensor of a second downscale resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of a second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution, and a convolution kernel of a first layer of the downscale neural network may be linearly mixed with a forget gate value corresponding to the first downscale resolution.

[0363] An image decoding device according to one embodiment of the present disclosure decomposes a prediction image of a current image into prediction tensors of a plurality of resolutions using memory gate values ​​and forget gate values ​​corresponding to a plurality of resolutions, and restores the image based on a plurality of prediction tensors corresponding to a plurality of resolutions, thereby maintaining main information of the image and removing unnecessary information to prevent the occurrence of errors, and improving the accuracy and coding efficiency of the image.

[0364] According to one embodiment of the present disclosure, the intermediate prediction tensor can be additionally applied to a neural network-based second decoder to obtain a current restored image.

[0365] An image decoding device according to one embodiment of the present disclosure can improve image accuracy and coding efficiency by utilizing additional information in residual coding.

[0366] According to one embodiment, a method for encoding an image may include: obtaining feature data of a current optical flow by applying a current image and a previous reconstructed image to a neural network-based first encoder; obtaining a plurality of remembering gate values ​​corresponding to a plurality of resolutions of the current optical flow and the current image by applying the feature data of the current optical flow to a neural network-based first decoder; obtaining a predicted image of the current image from a previous reconstructed image based on the current optical flow; obtaining a plurality of prediction tensors corresponding to a plurality of resolutions based on the predicted image and the plurality of remembering gate values; obtaining feature data for a residual image by applying the plurality of prediction tensors and the current image to a neural network-based second encoder; obtaining a current reconstructed image corresponding to the current image by applying the plurality of prediction tensors, the feature data of the current optical flow, and the feature data for the residual image to a neural network-based second decoder; and generating a bitstream including the feature data of the current optical flow and the feature data for the residual image.

[0367] According to one embodiment of the present disclosure, the plurality of memory gate values ​​may represent values ​​for maintaining information within the current image.

[0368] An image encoding method according to one embodiment of the present disclosure restores an image based on a plurality of prediction tensors corresponding to a plurality of resolutions obtained using a plurality of memory gate values ​​representing values ​​for maintaining key information in an image, thereby maintaining key information of the image, thereby preventing errors in a restored image and improving the accuracy and coding efficiency of the image.

[0369] According to one embodiment of the present disclosure, a plurality of forget gate values ​​corresponding to a plurality of resolutions of a current image are additionally obtained from a neural network-based first decoder, a plurality of prediction tensors are obtained based on the predicted image, the plurality of memory gate values, and the forget gate values, and the plurality of forget gate values ​​may represent values ​​for removing a region in the current image in which a prediction error is greater than a predetermined value.

[0370] An image encoding method according to one embodiment of the present disclosure obtains a plurality of prediction tensors by additionally using a plurality of forget gate values ​​representing values ​​for removing an area in which a prediction error of an image is greater than a predetermined value, and restores an image based on a plurality of prediction tensors corresponding to a plurality of resolutions, thereby removing unnecessary information of the image, thereby preventing an error from occurring in the restored image, and improving the accuracy and coding efficiency of the image.

[0371] According to one embodiment of the present disclosure, a prediction tensor of the original resolution of the current image among a plurality of prediction tensors can be determined based on memory gate values ​​corresponding to the predicted image and the original resolution.

[0372] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a subtraction tensor obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image and a forget gate value corresponding to the original resolution, and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on an intermediate prediction tensor of a resolution downscaled from the original resolution obtained by applying the prediction residual tensor to a downscale neural network and a remember gate value corresponding to the downscaled resolution.

[0373] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a memory gate value corresponding to an original resolution of a current image, a forget gate value corresponding to the original resolution, and a prediction image, and by applying the prediction residual tensor to a downscaled neural network, an intermediate prediction tensor of a downscaled resolution from the original resolution obtained and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on the memory gate value corresponding to the downscaled resolution.

[0374] According to one embodiment of the present disclosure, a prediction residual tensor is obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image, and a prediction tensor of a downscaled resolution is determined among a plurality of prediction tensors based on an intermediate prediction tensor of a downscaled resolution from an original resolution obtained by applying the prediction residual tensor to a downscaled neural network and a memory gate value corresponding to the downscaled resolution, and a convolution kernel of a first layer of the downscaled neural network may be linearly mixed with a forget gate value corresponding to the original resolution.

[0375] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a subtraction tensor obtained by subtracting a prediction tensor of the first downscale resolution from an intermediate prediction tensor of the first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a forget gate value corresponding to the first downscale resolution, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution.

[0376] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a memory gate value corresponding to the first downscale resolution, a forget gate value corresponding to the first downscale resolution, and an intermediate prediction tensor of the first downscale resolution before the memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among the plurality of prediction tensors, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to the downscale neural network, and the memory gate value corresponding to the second downscale resolution.

[0377] According to one embodiment of the present disclosure, a prediction residual tensor of a first downscale resolution is obtained by subtracting a prediction tensor of a first downscale resolution from an intermediate prediction tensor of a first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a prediction tensor of a second downscale resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of a second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution, and a convolution kernel of a first layer of the downscale neural network may be linearly mixed with a forget gate value corresponding to the first downscale resolution.

[0378] An image encoding method according to one embodiment of the present disclosure decomposes a prediction image of a current image into prediction tensors of multiple resolutions using memory gate values ​​and forget gate values ​​corresponding to multiple resolutions, and restores the image based on the multiple prediction tensors corresponding to the multiple resolutions, thereby maintaining the main information of the image and removing unnecessary information to prevent the occurrence of errors, and improving the accuracy and coding efficiency of the image.

[0379] According to one embodiment of the present disclosure, the predicted residual tensor may be additionally applied to a neural network-based second encoder to obtain feature data for the residual image, and the intermediate predicted tensor may be additionally applied to a neural network-based second decoder to obtain a current restored image.

[0380] An image encoding method according to one embodiment of the present disclosure can improve image accuracy and coding efficiency by utilizing additional information in residual coding.

[0381] An image encoding device according to one embodiment comprises: a memory storing one or more instructions; And at least one processor operating according to the one or more instructions, wherein the at least one processor applies the current image and the previous restored image to a neural network-based first encoder to obtain feature data of the current optical flow, applies the feature data of the current optical flow to a neural network-based first decoder to obtain a plurality of remembering gate values ​​corresponding to the current optical flow and a plurality of resolutions of the current image, obtains a predicted image of the current image from the previous restored image based on the current optical flow, obtains a plurality of prediction tensors corresponding to a plurality of resolutions based on the predicted image and the plurality of remembering gate values, applies the plurality of second prediction tensors and the current image to a neural network-based second encoder to obtain feature data for a residual image, applies the plurality of second prediction tensors, the feature data of the current optical flow, and the feature data for the residual image to a neural network-based second decoder to obtain a current restored image corresponding to the current image, and includes the feature data of the current optical flow and the feature data for the residual image. It can generate a bitstream.

[0382] According to one embodiment of the present disclosure, the plurality of memory gate values ​​may represent values ​​for maintaining information within the current image.

[0383] An image encoding device according to one embodiment of the present disclosure restores an image based on a plurality of prediction tensors corresponding to a plurality of resolutions obtained using a plurality of memory gate values ​​representing values ​​for maintaining key information in an image, thereby maintaining key information of the image, thereby preventing errors in the restored image and improving the accuracy and coding efficiency of the image.

[0384] According to one embodiment of the present disclosure, a plurality of forget gate values ​​corresponding to a plurality of resolutions of a current image are additionally obtained from a neural network-based first decoder, a plurality of prediction tensors are obtained based on the first predicted image, the plurality of memory gate values, and the forget gate values, and the plurality of forget gate values ​​may represent values ​​for removing a region in the current image in which a prediction error is greater than a predetermined value.

[0385] An image encoding device according to one embodiment of the present disclosure obtains a plurality of prediction tensors by additionally using a plurality of forget gate values ​​representing values ​​for removing an area in which a prediction error of an image is greater than a predetermined value, and restores an image based on a plurality of prediction tensors corresponding to a plurality of resolutions, thereby removing unnecessary information of the image, thereby preventing an error from occurring in the restored image, and improving the accuracy and coding efficiency of the image.

[0386] According to one embodiment of the present disclosure, a prediction tensor of the original resolution of the current image among a plurality of prediction tensors can be determined based on memory gate values ​​corresponding to the predicted image and the original resolution.

[0387] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a subtraction tensor obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image and a forget gate value corresponding to the original resolution, and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on an intermediate prediction tensor of a resolution downscaled from the original resolution obtained by applying the prediction residual tensor to a downscale neural network and a remember gate value corresponding to the downscaled resolution.

[0388] According to one embodiment of the present disclosure, a prediction residual tensor is obtained based on a memory gate value corresponding to an original resolution of a current image, a forget gate value corresponding to the original resolution, and a prediction image, and by applying the prediction residual tensor to a downscaled neural network, an intermediate prediction tensor of a downscaled resolution from the original resolution obtained and a prediction tensor of a downscaled resolution among a plurality of prediction tensors can be determined based on the memory gate value corresponding to the downscaled resolution.

[0389] According to one embodiment of the present disclosure, a prediction residual tensor is obtained by subtracting a prediction tensor of an original resolution of a current image from a prediction image, and a prediction tensor of a downscaled resolution is determined among a plurality of prediction tensors based on an intermediate prediction tensor of a downscaled resolution from an original resolution obtained by applying the prediction residual tensor to a downscaled neural network and a memory gate value corresponding to the downscaled resolution, and a convolution kernel of a first layer of the downscaled neural network may be linearly mixed with a forget gate value corresponding to the original resolution.

[0390] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a subtraction tensor obtained by subtracting a prediction tensor of the first downscale resolution from an intermediate prediction tensor of the first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a forget gate value corresponding to the first downscale resolution, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution.

[0391] According to one embodiment of the present disclosure, a prediction residual tensor of the first downscale resolution may be obtained based on a memory gate value corresponding to the first downscale resolution, a forget gate value corresponding to the first downscale resolution, and an intermediate prediction tensor of the first downscale resolution before the memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among the plurality of prediction tensors, and a prediction tensor of the second downscale resolution among the plurality of prediction tensors may be determined based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to the downscale neural network, and the memory gate value corresponding to the second downscale resolution.

[0392] According to one embodiment of the present disclosure, a prediction residual tensor of a first downscale resolution is obtained by subtracting a prediction tensor of a first downscale resolution from an intermediate prediction tensor of a first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among a plurality of prediction tensors, and a prediction tensor of a second downscale resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of a second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to a downscale neural network, and a memory gate value corresponding to the second downscale resolution, and a convolution kernel of a first layer of the downscale neural network may be linearly mixed with a forget gate value corresponding to the first downscale resolution.

[0393] An image encoding device according to one embodiment of the present disclosure decomposes a prediction image of a current image into prediction tensors of multiple resolutions using memory gate values ​​and forget gate values ​​corresponding to multiple resolutions, and restores the image based on the multiple prediction tensors corresponding to the multiple resolutions, thereby maintaining main information of the image and removing unnecessary information to prevent the occurrence of errors, and improving the accuracy and coding efficiency of the image.

[0394] According to one embodiment of the present disclosure, the predicted residual tensor may be additionally applied to a neural network-based second encoder to obtain feature data for a residual image, and the intermediate predicted tensor may be additionally applied to a neural network-based second decoder to obtain a current restored image.

[0395] An image encoding method according to one embodiment of the present disclosure can improve image accuracy and coding efficiency by utilizing additional information in residual coding.

[0396] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0397] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In the video decryption method, A step of obtaining feature data of a current optical flow and feature data for a residual image of a current image from a bitstream (S1810); A step of obtaining a plurality of remembering gate values ​​corresponding to a plurality of resolutions of the current optical flow and the current image by applying the feature data of the current optical flow to a neural network-based first decoder (S1820); A step of obtaining a predicted image of a current image from a previously restored image based on the current optical flow (S1830); A step of obtaining a plurality of prediction tensors corresponding to the plurality of resolutions based on the above predicted image and the plurality of memory gate values ​​(S1840); An image decoding method, comprising: a step of applying the plurality of prediction tensors, feature data of the current optical flow, and feature data for the residual image to a neural network-based second decoder to obtain a current restored image corresponding to the current image (S1850).

2. In paragraph 1, An image decoding method, wherein the above plurality of memory gate values ​​represent values ​​for maintaining information within the current image.

3. In paragraph 1 or 2, From the first decoder based on the neural network, a plurality of forgetting gate values ​​corresponding to the plurality of resolutions of the current image are additionally obtained, Based on the above predicted image, the plurality of memory gate values, and the forget gate values, the plurality of prediction tensors are obtained, A method for decoding an image, wherein the above plurality of forget gate values ​​represent values ​​for removing an area in the current image in which the prediction error is greater than a predetermined value.

4. In any one of paragraphs 1 to 3, Among the above multiple prediction tensors, the prediction tensor of the original resolution of the current image is An image decoding method, wherein the image decoding method is determined based on a memory gate value corresponding to the predicted image and the original resolution.

5. In any one of paragraphs 1 to 4, A prediction residual tensor is obtained based on a subtraction tensor obtained by subtracting a prediction tensor of the original resolution of the current image from the predicted image and a forget gate value corresponding to the original resolution. An image decoding method, wherein a prediction tensor of the downscaled resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of the downscaled resolution from the original resolution obtained by applying the prediction residual tensor to a downscaled neural network and a memory gate value corresponding to the downscaled resolution.

6. In any one of paragraphs 1 to 4, A prediction residual tensor is obtained based on a memory gate value corresponding to the original resolution of the current image, a forget gate value corresponding to the original resolution, and the predicted image. An image decoding method, wherein a prediction tensor of the downscaled resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of the downscaled resolution from the original resolution obtained by applying the prediction residual tensor to a downscaled neural network and a memory gate value corresponding to the downscaled resolution.

7. In any one of paragraphs 1 to 4, A prediction residual tensor is obtained by subtracting the prediction tensor of the original resolution of the current image from the predicted image, A prediction tensor of the downscaled resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of the downscaled resolution from the original resolution obtained by applying the prediction residual tensor to a downscaled neural network and a memory gate value corresponding to the downscaled resolution, A method for decoding an image, wherein the convolution kernel of the first layer of the above downscaled neural network is linearly mixed with a forget gate value corresponding to the original resolution.

8. In any one of paragraphs 1 to 4, A prediction residual tensor of the first downscale resolution is obtained based on a subtraction tensor obtained by subtracting the prediction tensor of the first downscale resolution from an intermediate prediction tensor of the first downscale resolution before a memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among the plurality of prediction tensors, and a forget gate value corresponding to the first downscale resolution. An image decoding method, wherein a prediction tensor of the second downscaled resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of the second downscaled resolution, which is a resolution downscaled from the first downscaled resolution obtained by applying the prediction residual tensor to a downscaled neural network, and a memory gate value corresponding to the second downscaled resolution.

9. In any one of paragraphs 1 to 4, A prediction residual tensor of the first downscale resolution is obtained based on a memory gate value corresponding to the first downscale resolution, a forget gate value corresponding to the first downscale resolution, and an intermediate prediction tensor of the first downscale resolution before the memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among the plurality of prediction tensors. An image decoding method, wherein a prediction tensor of the second downscaled resolution among the plurality of prediction tensors is determined based on an intermediate prediction tensor of the second downscaled resolution, which is a resolution downscaled from the first downscaled resolution obtained by applying the prediction residual tensor to a downscaled neural network, and a memory gate value corresponding to the second downscaled resolution.

10. In any one of paragraphs 1 to 4, A prediction residual tensor of the first downscale resolution is obtained by subtracting the prediction tensor of the first downscale resolution from the intermediate prediction tensor of the first downscale resolution before the memory gate value corresponding to the first downscale resolution is applied to the prediction tensor of the first downscale resolution among the plurality of prediction tensors, A prediction tensor of the second downscale resolution is determined among the plurality of prediction tensors based on an intermediate prediction tensor of the second downscale resolution, which is a resolution downscaled from the first downscale resolution obtained by applying the prediction residual tensor to the downscale neural network, and a memory gate value corresponding to the second downscale resolution, An image decoding method, wherein the convolution kernel of the first layer of the above downscaled neural network is linearly mixed with a forget gate value corresponding to the first downscaled resolution.

11. In any one of paragraphs 5 to 10, An image decoding method, wherein the intermediate prediction tensor is additionally applied to the second decoder based on the neural network, thereby obtaining the current restored image.

12. In the video encoding method, A step of obtaining feature data of the current optical flow by applying the current image and the previous restored image to a first encoder based on a neural network (S1610); A step of obtaining a plurality of remembering gate values ​​corresponding to the current optical flow and a plurality of resolutions of the current image by applying the feature data of the current optical flow to a neural network-based first decoder (S1620); A step of obtaining a predicted image of a current image from a previously restored image based on the current optical flow (S1630); A step of obtaining a plurality of prediction tensors corresponding to the plurality of resolutions based on the above predicted image and the plurality of memory gate values ​​(S1640); A step of applying the above-mentioned plurality of prediction tensors and the current image to a neural network-based second encoder to obtain feature data for the residual image (S1650); A step of applying the above plurality of prediction tensors, feature data of the current optical flow, and feature data for the residual image to a neural network-based second decoder to obtain a current restored image corresponding to the current image (S1660); and An image encoding method, comprising: a step of generating a bitstream including feature data of the current optical flow and feature data for the residual image (S1670).

13. In paragraph 12, An image encoding method, wherein the plurality of memory gate values ​​represent values ​​for maintaining information in the current image.

14. In paragraph 12 or 13, A plurality of forgetting gate values ​​corresponding to the plurality of resolutions of the current image are additionally obtained from the first decoder based on the neural network, Based on the above predicted image, the plurality of memory gate values, and the forget gate values, the plurality of prediction tensors are obtained, An image encoding method, wherein the above plurality of forget gate values ​​represent values ​​for removing an area in the current image in which the prediction error is greater than a predetermined value.

15. In any one of paragraphs 12 to 14, Among the above multiple prediction tensors, the prediction tensor of the original resolution of the current image is An image encoding method, wherein the image is determined based on a memory gate value corresponding to the predicted image and the original resolution.

Images