Image decoding method, image encoding method, image encoding device, and image decoding device using ai

Abstract

Provided is an image decoding method, comprising: obtaining feature data of a current optical flow and feature data of a residual image of a current image from a bit stream; applying the feature data of the current optical flow to a first decoder based on a neural network to obtain the current optical flow and a plurality of memory thresholds corresponding to a plurality of resolutions of the current image; obtaining a predicted image of the current image from the previously reconstructed image based on the current optical flow; obtaining a plurality of prediction tensors corresponding to the plurality of resolutions based on the prediction image and the plurality of memory thresholds; a current reconstructed image corresponding to the current image is obtained by applying the plurality of prediction tensors, the feature data of the current optical flow, and the feature data of the residual image to a second decoder based on a neural network.

Description

TECHNICAL FIELD

[0001] The disclosure relates to image encoding and decoding. More specifically, the disclosure relates to a technology of encoding and decoding an image by using artificial intelligence (AI) (e.g., a neural network). BACKGROUND

[0002] A codec such as H.264 Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC) can divide an image into blocks and predictively encode and decode each block through inter prediction or intra prediction.

[0003] Intra prediction is a method of compressing an image by removing spatial redundancy in the image, and inter prediction is a method of compressing an image by removing temporal redundancy between images.

[0004] A representative example of inter prediction is motion estimation encoding. Motion estimation encoding predicts a block of a current image by using a reference image. A reference block most similar to the current block can be found in a certain search range by using a certain evaluation function. The current block is predicted based on the reference block, and a residual block is generated by subtracting a prediction block generated as a result of prediction from the current block. The residual block is then encoded.

[0005] In order to derive a motion vector indicating a reference block in a reference image, a motion vector of a previously encoded block can be used as a motion vector predictor of a current block. A differential motion vector corresponding to a difference between the motion vector of the current block and the motion vector predictor of the current block is signaled to the decoder side through a predetermined method.

[0006] Recently, a technology for encoding / decoding an image by using artificial intelligence (AI) has been proposed, and there is a need for a method for efficiently encoding / decoding an image using AI (e.g., a neural network). SUMMARY

[0007] Solution to the problem An image decoding method according to an embodiment of the disclosure can include obtaining feature data of a current optical flow and feature data of a residual image of a current image from a bitstream; obtaining the current optical flow and a plurality of memory gate values corresponding to a plurality of resolutions of the current image by applying the feature data of the current optical flow to a first decoder based on a neural network; obtaining a prediction image of the current image from a previously reconstructed image based on the current optical flow; obtaining a plurality of prediction tensors corresponding to the plurality of resolutions based on the prediction image and the plurality of memory gate values; and obtaining a current reconstructed image corresponding to the current image by applying the plurality of prediction tensors, the feature data of the current optical flow, and the feature data of the residual image to a second decoder based on a neural network.

[0008] An image decoding apparatus according to an embodiment of the disclosure can include a memory storing one or more instructions, and at least one processor operating according to the one or more instructions. The at least one processor can obtain feature data of a current optical flow and feature data of a residual image of a current image from a bitstream. The at least one processor can obtain the current optical flow and a plurality of memory gate values corresponding to a plurality of resolutions of the current image by applying the feature data of the current optical flow to a first decoder based on a neural network. The at least one processor can obtain a predicted image of the current image from a previous reconstructed image based on the current optical flow. The at least one processor can obtain a plurality of predicted tensors corresponding to the plurality of resolutions based on the predicted image and the plurality of memory gate values. The at least one processor can obtain a current reconstructed image corresponding to the current image by applying the plurality of predicted tensors, the feature data of the current optical flow, and the feature data of the residual image to a second decoder based on a neural network.

[0009] An image encoding method according to an embodiment of the disclosure can include obtaining feature data of a current optical flow by applying a current image and a previous reconstructed image to a first encoder based on a neural network, obtaining a current optical flow and a plurality of memory gate values corresponding to a plurality of resolutions of the current image by applying the feature data of the current optical flow to a first decoder based on a neural network, obtaining a predicted image of the current image from the previous reconstructed image based on the current optical flow, obtaining a plurality of predicted tensors corresponding to the plurality of resolutions based on the predicted image and the plurality of memory gate values, obtaining feature data of a residual image by applying the plurality of predicted tensors and the current image to a second encoder based on a neural network, obtaining a current reconstructed image corresponding to the current image by applying the plurality of predicted tensors, the feature data of the current optical flow, and the feature data of the residual image to a second decoder based on a neural network, and generating a bitstream including the feature data of the current optical flow and the feature data of the residual image.

[0010] An image encoding apparatus according to an embodiment of the disclosure can include a memory storing one or more instructions, and at least one processor operating according to the one or more instructions. The at least one processor can obtain feature data of a current optical flow by applying a current image and a previously reconstructed image to a first encoder based on a neural network. The at least one processor can obtain the current optical flow and a plurality of memory gate values corresponding to a plurality of resolutions of the current image by applying the feature data of the current optical flow to a first decoder based on a neural network. The at least one processor can obtain a predicted image of the current image from the previously reconstructed image based on the current optical flow. The at least one processor can obtain a plurality of predicted tensors corresponding to the plurality of resolutions based on the predicted image and the plurality of memory gate values. The at least one processor can obtain feature data of a residual image by applying the plurality of predicted tensors and the current image to a second encoder based on a neural network. The at least one processor can obtain a current reconstructed image corresponding to the current image by applying the plurality of predicted tensors, the feature data of the current optical flow, and the feature data of the residual image to a second decoder based on a neural network. The at least one processor can generate a bitstream including the feature data of the current optical flow and the feature data of the residual image. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a diagram illustrating an artificial intelligence (AI)-based image encoding and decoding process.

[0012] Figure 2 is a diagram illustrating a current optical flow, a current predicted image, and a residual image obtained from a current image and a previously reconstructed image.

[0013] Figure 3 is a diagram for describing an image encoding and decoding process according to an embodiment of the disclosure.

[0014] Figure 4 is a diagram for describing a depth prediction decomposition process according to an embodiment of the disclosure.

[0015] Figure 5 is a diagram for describing a depth prediction decomposition process according to an embodiment of the disclosure.

[0016] Figure 6 is a diagram for describing a depth prediction decomposition process according to an embodiment of the disclosure.

[0017] Figure 7 is a diagram for describing a structure of a motion encoder, a motion decoder, a multi-compensation pixel encoder, and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0018] Figure 8 is a diagram for describing a structure of a motion decoder according to an embodiment of the disclosure.

[0019] Figure 9 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0020] Figure 10 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0021] Figure 11 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0022] Figure 12 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0023] Figure 13 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0024] Figure 14 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0025] Figure 15 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0026] Figure 16 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0027] Figure 17 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0028] Figure 18 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0029] Figure 19 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure.

[0030] Figure 20 FIG. 1 is a diagram for describing a structure of a multi-compensation pixel encoder and a multi-compensation pixel decoder according to an embodiment of the disclosure. DETAILED DESCRIPTION

[0031] Since the present disclosure allows various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present disclosure to particular modes of practice, and it is to be appreciated that all changes, equivalents, and substitutes not departing from the spirit and technical scope of the present disclosure are encompassed in the present disclosure.

[0032] In the description of embodiments of the present disclosure, certain detailed explanations of related art are omitted when it is deemed that such certain detailed explanations of related art can unnecessarily obscure the essence of the present disclosure. Although terms such as "first," "second," etc. can be used to describe various components, the components are not necessarily limited to the above terms. The above terms are used only to distinguish one component from another.

[0033] Throughout the disclosure, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0034] When an element (for example, a first element) "is coupled to" or "is connected to" another element (for example, a second element), the first element can be directly coupled to or connected to the second element, or, unless otherwise described, there can be another element therebetween.

[0035] With respect to components denoted as "part (unit)" or "module" as used herein, two or more components can be combined into one component, or one component can be divided into two or more components according to a subdivided function. Furthermore, each component described below can additionally perform some or all of the functions performed by another component, in addition to its own main function, and some of the main functions of each component can be completely performed by another component.

[0036] The processor can include various processing circuitry and / or a plurality of processors. For example, the term "processor" used herein and in claims can include various processing circuitry including at least one processor. One or more processors of the at least one processor can be configured to individually and / or collectively perform various functions described herein in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" can be configured to perform several functions. However, these terms encompass, without limitation, cases where one processor performs some functions while other (one or more) processors perform other functions, and cases where a single processor is capable of performing all functions. Furthermore, the at least one processor can include a combination of processors of various functions performing the disclosed functions in a distributed manner. The at least one processor can execute program instructions in order to accomplish or perform various functions.

[0037] An "image" as used herein can indicate a still image, a picture, a frame, a moving picture consisting of a plurality of consecutive still images, or a video.

[0038] A "neural network" as used herein is a representative example of an artificial neural network model that imitates a brain nerve, and is not limited to an artificial neural network model using a specific algorithm. The neural network can also be referred to as a deep neural network.

[0039] For example, a "parameter" as used herein, which is a value used in a calculation process of each layer included in a neural network, can be used when an input value is applied to a predetermined calculation formula. The parameter of the value set as a result of training can be updated by separate training data as necessary.

[0040] "Feature data" as used herein refers to data obtained by processing input data by an encoder based on a neural network. The feature data can be one-dimensional or two-dimensional (1D or 2D) data including a number of samples. The feature data can also be referred to as a latent representation. The feature data can represent a latent feature of data output by a decoder described below.

[0041] A "current image" as used herein refers to an image that is currently to be processed, and a "previous image" as used herein refers to an image that is to be processed before the current image. A "current motion vector" refers to a motion vector obtained for processing the current image.

[0042] A "sample" used herein as data assigned to a sampling position in an image, a feature map, or feature data refers to data to be processed. For example, a sample can include a pixel in a 2D image.

[0043] In addition, in the present disclosure, the term "tensor" refers to data in the form of a multi-dimensional array. The tensor can refer to image data. Further, the tensor can be data after performing, for example, an addition operation, a multiplication operation, or a subtraction operation on the image data. Further, the tensor can be feature data processed by a neural network.

[0044] Figure 1 is a diagram illustrating an artificial intelligence (AI)-based image encoding and decoding process.

[0045] Figure 1 Inter-prediction processing is illustrated. In inter-prediction, an optical flow encoder 110, an image encoder 130, an optical flow decoder 150, and an image decoder 170 can be used.

[0046] The optical flow encoder 110, the image encoder 130, the optical flow decoder 150, and the image decoder 170 can be implemented as a neural network.

[0047] Optical flow encoder 110 and optical flow decoder 150 can be understood as being used to extract the current optical flow g from the current image 10 and the previously reconstructed image 30. i Neural networks.

[0048] Image encoder 130 and image decoder 170 can be used to extract input images (e.g., residual images r). i The neural network reconstructs images from feature data and features data.

[0049] Inter-frame prediction is the process of encoding and decoding the current image 10 by using the temporal redundancy between the current image 10 and the previously reconstructed image 30.

[0050] The positional differences (or motion vectors) between a block or sample in the current image 10 and a reference block or sample in the previously reconstructed image 30 are used to encode and decode the current image 10. These positional differences can be referred to as optical flow. Optical flow can be defined as a set of motion vectors corresponding to a sample or block in the image.

[0051] Optical flow, especially current optical flow, can indicate how the position of a sample point in the previously reconstructed image 30 has been changed in the current image 10, or the position of a sample point that is the same as or similar to a sample point in the current image 10 in the previously reconstructed image 30.

[0052] For example, when the same or most similar sample point located at (1, 1) in the current image 10 is located at (2, 1) in the previously reconstructed image 30, the optical flow or motion vector of the sample point can be derived as (1(1(=2-1), 0(=1-1)).

[0053] In the process of image encoding and decoding using AI, the optical flow encoder 110 and the optical flow decoder 150 can be used to obtain the current optical flow g of the current image 10. i .

[0054] In detail, the previously reconstructed image 30 and the current image 10 can be input into the optical flow encoder 110. The optical flow encoder 110 can output the feature data w of the current optical flow by processing the current image 10 and the previously reconstructed image 30 according to parameters set as training results. i .

[0055] Current optical flow characteristic data w i It can be input to the optical flow decoder 150. The optical flow decoder 150 can process the input feature data w according to the parameters set as a training result. i Process to output the current optical flow g i .

[0056] Based on the current optical flow g iThe previously reconstructed image 30 is warped by 190 degrees, and the current predicted image x' is obtained as a result of the 190-degree warp. i Distortion 190 is a geometric transformation used to change the position of sample points in an image.

[0057] It can be determined based on the current optical flow g i A distortion of 190 is applied to the previously reconstructed image 30 to obtain a current predicted image x' that is similar to the current image 10. i Current optical flow g i This indicates the relative positional relationship between the sample points in the previously reconstructed image 30 and the sample points in the current image 10.

[0058] For example, when the sample point located at (1, 1) in the previously reconstructed image 30 is most similar to the sample point located at (2, 1) in the current image 10, the position of the sample point located at (1, 1) in the previously reconstructed image 30 can be changed to (2, 1) by warping 190.

[0059] Because the current predicted image x' is generated from the previously reconstructed image 30 i It is not the current image 10 itself, so it is possible to obtain the image x' with the current prediction image. i The residual image r corresponding to the difference between the current image 10 and the image 10 i For example, this can be achieved by subtracting the current predicted image x' from the sample values ​​in the current image I0. i The residual image r is obtained by taking the sample values ​​from the sample points. i .

[0060] residual image r i It can be input into image encoder 130. Image encoder 130 can process the residual image r according to parameters set as a result of training. i To output the residual image r i Feature data v i .

[0061] Feature data v of residual image i It can be input to image decoder 170. Image decoder 170 can process the input feature data v according to parameters set as a result of training. i Processing is performed to output the reconstructed residual image r' i .

[0062] The currently predicted image x' can be generated by warping the previously reconstructed image 30 by 190 degrees. i With the reconstructed residual image data r' i Combine them to obtain the current reconstructed image 50.

[0063] When implemented by encoding and decoding devicesFigure 1 During the image encoding and decoding process shown, the encoding device can process the feature data w of the current optical flow obtained by encoding the current image 10. i and feature data v of residual image i Quantization is performed to generate a bitstream containing quantized feature data, and this bitstream is sent to the decoding device. The decoding device can obtain the feature data w of the current optical flow by inverse quantizing the quantized feature data extracted from the bitstream. i and feature data v of residual image i The decoding device can use the optical flow decoder 150 and the image decoder 170 to analyze the feature data of the current optical flow. i and feature data v of residual image i Processing is performed to obtain the current reconstructed image 50.

[0064] As described above, the current image 10 and the current predicted image x' i The residual image r between i This can be input into the image encoder 130. Because the currently predicted image x' i Based on the current optical flow g i Generated from the previously reconstructed image 30, so in the current optical flow g i When there is an error, the current predicted image x' i and residual image r i There is also a high probability that errors exist.

[0065] When the residual image r has errors i When input to the image encoder 130, the bit rate of the bitstream may increase unnecessarily. Furthermore, this is because the current predicted image x' has errors. i With the reconstructed residual image r' i Therefore, the quality of the currently reconstructed image 50 may also be degraded due to the combination of factors.

[0066] Reference Figure 2 Explain the process of error occurrence and propagation.

[0067] Figure 2 This is a diagram showing the current optical flow, the current predicted image, and the residual image obtained from the current image and the previously reconstructed image.

[0068] Reference Figure 2 The current optical flow 23, which indicates the motion of the sample points in the current image 22, can be obtained from the current image 22 and the previously reconstructed image 21.

[0069] As referred above Figure 1The current optical flow 23 can be obtained through the processing of the optical flow encoder 110 and the optical flow decoder 150, as well as the quantization and dequantization of the feature data of the current optical flow. Therefore, errors may be generated in the current optical flow 23 (e.g., in region A).

[0070] To describe in detail the causes of the errors, firstly, errors may occur in the current optical flow 23 due to the limitations of the processing capabilities of the optical flow encoder 110 and optical flow decoder 150. Because the computing power of the encoding and decoding devices is limited, the number of layers and the size of the filter kernels in the optical flow encoder 110 and optical flow decoder 150 may also be limited. In other words, because both the optical flow encoder 110 and optical flow decoder 150 have limited capabilities in processing the current image 22 and the previously reconstructed image 21, errors may occur in the current optical flow 23.

[0071] Next, through the quantization and dequantization of the feature data of the current optical flow 23, quantization errors may occur in the current optical flow 23. Specifically, when the value of the quantization parameter is increased to improve compression efficiency, the bit rate of the bitstream decreases, but the number of quantization errors increases.

[0072] Finally, when the objects included in the current image 22 and the previously reconstructed image 21 move rapidly, the likelihood of errors occurring in the current optical flow 23 increases.

[0073] When an error exists in region A of the current optical flow 23, the error may also occur in region B of the current predicted image 24 generated from the previously reconstructed image 21, based on the presence of the error in region A, and the error may also occur in region C of the residual image 25 obtained between the current predicted image 24 and the current image 22.

[0074] Because the residual image 25 is processed by the image encoder 130 and transformed into feature data of the residual image 25, and the feature data of the residual image 25 is included in the bitstream after undergoing a preset process, it can be seen that the error present in the current optical stream 23 is transmitted to the decoding device.

[0075] Typically, because errors have high-frequency characteristics, when the residual image 25 including errors is processed by the image encoder 130, the errors may cause an unnecessary increase in the bit rate of the bit stream.

[0076] The image encoding and decoding process used to prevent the spread of errors present in the current optical flow will now be described.

[0077] Figure 3 This is a diagram used to describe the image encoding and decoding process according to embodiments of the present disclosure.

[0078] Reference Figure 3The motion encoder 310, motion decoder 320, multi-compensated pixel encoder 350, and multi-compensated pixel decoder 360 can be used to encode and decode images, and depth prediction decomposition 340 can be performed.

[0079] In this disclosure, "depth predictive decomposition" refers to the process of transforming a predicted image into multiple predicted images or multiple predicted tensors corresponding to multiple resolutions based on a neural network. The multiple resolutions include the original resolution of the predicted image and multiple resolutions scaled down from the original resolution.

[0080] According to embodiments of this disclosure, the motion encoder 310, motion decoder 320, multi-compensated pixel encoder 350, and multi-compensated pixel decoder 360 can be implemented as neural networks. Furthermore, the depth predictive decomposition 340 can also be implemented as a neural network.

[0081] To encode the current image 300, the previously reconstructed image 305 and the current image 300 can be input into the motion encoder 310. The motion encoder 310 can output feature data 311 of the current optical flow by processing the current image 300 and the previously reconstructed image 305 according to parameters set as a result of training.

[0082] The current optical flow feature data 311 can be input to the motion decoder 320. The motion decoder 320 can process the input feature data 311 according to parameters set as training results to output the decoded current optical flow 321 and memory gate values ​​and forget gate values ​​322 corresponding to multiple resolutions. See below. Figure 7 and Figure 8 An exemplary structure of the motion encoder 310 and the motion decoder 320 is described.

[0083] The previously reconstructed image 305 can be warped based on the current optical flow 321 via warp 330, and the current predicted image 331 can be obtained as a result of warp 330.

[0084] A depth prediction decomposition 340 can be performed using memory and forget gates 322 corresponding to multiple resolutions and the current prediction image 331, and as a result of the depth prediction decomposition 340, multiple prediction tensors 341 corresponding to multiple resolutions can be obtained. The memory gate represents a value used to preserve the main information of the image (e.g., edges or details of well-compensated regions) to retain information useful for encoding or decoding the original image. The forget gate represents a value used to remove unnecessary information for encoding or decoding the original image (e.g., edges or details of poorly compensated regions (i.e., occluded or de-occluded regions)) or noise in the image. The memory and forget gates are set to values ​​between 0 and 1. The closer the memory gate is to 1, the more important the information becomes, and the closer it is to 0, the less important the information becomes. The closer the forget gate is to 1, the more information needs to be removed; the closer the forget gate is to 0, the less information needs to be removed. The neural network used in the depth prediction decomposition 340 can output multiple prediction tensors 341 by processing the current prediction image 331 according to parameters set as a training result. Well-predicted pixels in the predicted image are very useful for residual coding to suppress temporal redundancy, while poorly predicted pixels are useless and severely degrade coding efficiency. Therefore, the predicted image is decomposed into well-predicted and poorly predicted parts. A memory gate is flexibly used for residual coding in well-predicted pixels, and a forget gate is used for residual coding in poorly predicted pixels to extract multiple pieces of information. Furthermore, to fully utilize the predicted image, downsampling neural network layers are applied to the extracted information, such as the multiple pieces of useful information remaining after applying the forget gate. Multiple pieces of useful information about the predicted image are obtained at various resolutions. That is, the predicted image is decomposed into prediction tensors of multiple resolutions to achieve better utilization of the predicted image. These prediction tensors of multiple resolutions are used as references for encoding and decoding the original image.

[0085] The following reference Figure 4 to Figure 6 An exemplary structure for depth prediction decomposition 340 is described.

[0086] Multiple prediction tensors 341 and the current image 300 can be input to a multi-compensated pixel encoder 350. The multi-compensated pixel encoder 350 can output residual image feature data 351 by processing the current image 300 and the multiple prediction tensors 341 according to parameters set as a result of training. The residual image feature data 351 can be feature data extracted from the multiple prediction tensors 341 and the current image 300.

[0087] The current optical flow feature data 311 and the residual image feature data 351 can be input to the multi-compensated pixel decoder 360. For example, the result of concatenating the current optical flow feature data 311 and the residual image feature data 351 can be input to the multi-compensated pixel decoder 390. Concatenation can refer to the process of combining two or more feature data in the channel direction. Multiple prediction tensors 341 can also be input to the multi-compensated pixel decoder 360.

[0088] The multi-compensated pixel decoder 360 can obtain the reconstructed image 360 ​​by processing the feature data 311 of the current optical flow, the feature data 351 of the residual image, and multiple prediction tensors 341 according to the parameters set as a result of training.

[0089] The following reference Figure 7 and Figure 9 to Figure 14 An exemplary structure of the multi-compensated pixel encoder 350 and the multi-compensated pixel decoder 360 is described.

[0090] When implemented by encoding and decoding devices Figure 3 During the image encoding and decoding process shown, the encoding device can generate a bitstream including feature data 311 of the current optical flow and residual image feature data 351 obtained by encoding the current image 300, and can send the generated bitstream to the decoding device.

[0091] The decoding device can obtain the current optical flow feature data 311 and residual image feature data 351 from the bitstream. The decoding device can also obtain the reconstructed image 370 based on the current optical flow feature data 311 and residual image feature data 351.

[0092] Now will describe Figure 3 The image encoding and decoding process shown in the figure is similar to Figure 1 The image encoding and decoding processes shown in the figure differ from those in the figure.

[0093] and Figure 1 Compared to the optical flow decoder 150, besides the current optical flow 321, Figure 3 The motion decoder 320 also outputs memory gates and forget gates 322 corresponding to multiple resolutions.

[0094] In addition, Figure 3 In addition, depth prediction decomposition 340 is performed to obtain multiple prediction tensors 341 based on memory and forget gates 322 corresponding to multiple resolutions and the current prediction image 311.

[0095] By utilizing memory and forget gates, the main information of well-predicted pixels is preserved, while unnecessary information of poorly predicted pixels is removed. Furthermore, by using prediction tensors at multiple resolutions through downsampling neural network layers, multiple remaining useful information after applying the forget gate at the original resolution is utilized at the downsized resolution, and useful information about the predicted image is obtained at various resolutions.

[0096] also, Figure 3 The multi-compensated pixel encoder 350 and multi-compensated pixel decoder 360 use multiple prediction tensors obtained through depth predictive decomposition 340 for residual coding. Thus, a reconstructed image is obtained, in which the main information of the image is preserved and unnecessary errors have been removed.

[0097] In the following text, the addition and multiplication operations performed are referred to as element-wise summation and element-wise multiplication, respectively.

[0098] Figure 4 This is a diagram used to describe the depth prediction decomposition process according to embodiments of the present disclosure.

[0099] Reference Figure 4 The currently predicted image 400 consists of three channels: red, green, and blue (RGB), with a height (H) × width (W) dimension. The memory gate values ​​402, 412, 422, and 432, corresponding to multiple resolutions, are values ​​corresponding to resolutions of H×W, H / 2×W / 2, H / 4×W / 4, and H / 8×W / 8, respectively. The forget gate values ​​403, 413, 423, and 433, corresponding to multiple resolutions, are values ​​corresponding to resolutions of H×W, H / 2×W / 2, H / 4×W / 4, and H / 8×W / 8, respectively. The memory gate values ​​402, 412, 422, and 432, and the forget gate values ​​403, 413, 423, and 433 can be feature maps for deriving spatial differences from one channel. In addition, the memory gates 402, 412, 422 and 432 and the forget gates 403, 413, 423 and 433 can be feature maps of one or more channels depending on the characteristics of the image.

[0100] First, a prediction tensor 401 is obtained by multiplying the current prediction image 400 by (as shown by reference numeral 404) a memory gate value 402 corresponding to the original resolution H×W of the current prediction image 400. This prediction tensor 401 corresponds to the resolution H×W and retains information about the pixels that are well predicted.

[0101] By subtracting the prediction tensor 401 corresponding to the H×W resolution (as shown in reference numeral 405) from the current predicted image 400, and then multiplying the result of the subtraction by the forget gate value 403 corresponding to the original resolution H×W (as shown in reference numeral 406), a prediction residual tensor 407 corresponding to the H×W resolution is obtained. The prediction residual tensor 407 is applied to the neural network 408 to obtain an intermediate prediction tensor 410 corresponding to the H / 2×W / 2 resolution. The intermediate prediction tensor 410 corresponding to the H / 2×W / 2 resolution is multiplied by the memory gate value 412 corresponding to the H / 2×W / 2 resolution (as shown in reference numeral 414) to obtain a prediction tensor 411 corresponding to the H / 2×W / 2 resolution.

[0102] By subtracting the prediction tensor 411 corresponding to the resolution H / 2×W / 2 (as shown in reference numeral 415) from the intermediate prediction tensor 410 corresponding to the resolution H / 2×W / 2, and then multiplying the result of the subtraction by the forget gate value 413 corresponding to the resolution H / 2×W / 2 (as shown in reference numeral 416), the prediction residual tensor 417 corresponding to the resolution H / 2×W / 2 is obtained. The intermediate prediction tensor 420 corresponding to the resolution H / 4×W / 4 is obtained by applying the prediction residual tensor 417 to the neural network 418. The prediction tensor 421 corresponding to the resolution H / 4×W / 4 is obtained by multiplying the intermediate prediction tensor 420 corresponding to the resolution H / 4×W / 4 by the memory gate value 422 corresponding to the resolution H / 4×W / 4 (as shown in reference numeral 424).

[0103] By subtracting the prediction tensor 421 corresponding to the H / 4×W / 4 resolution (as shown in reference numeral 425) from the intermediate prediction tensor 420 corresponding to the H / 4×W / 4 resolution, and then multiplying the result of the subtraction by the forget gate value 423 corresponding to the H / 4×W / 4 resolution (as shown in reference numeral 426), the prediction residual tensor 427 corresponding to the H / 4×W / 4 resolution is obtained. The intermediate prediction tensor 430 corresponding to the H / 8×W / 8 resolution is obtained by applying the prediction residual tensor 427 to the neural network 428. The prediction tensor 431 corresponding to the H / 8×W / 8 resolution is obtained by multiplying the intermediate prediction tensor 430 corresponding to the H / 8×W / 8 resolution by the memory gate value 432 corresponding to the H / 8×W / 8 resolution (as shown in reference numeral 434).

[0104] By subtracting the prediction tensor 431 corresponding to the H / 8×W / 8 resolution (as shown in reference numeral 435) from the intermediate prediction tensor 430 corresponding to the H / 8×W / 8 resolution, and then multiplying the result of the subtraction by the forget gate value 433 corresponding to the H / 8×W / 8 resolution (as shown in reference numeral 436), the prediction residual tensor 437 corresponding to the H / 8×W / 8 resolution is obtained. The intermediate prediction tensor 440 corresponding to the H / 16×W / 16 resolution is obtained by applying the prediction residual tensor 437 to the neural network 438.

[0105] The multiple prediction tensors 401, 411, 421, and 431 obtained through this process, corresponding to the four resolutions, can be used for residual coding.

[0106] In residual coding, additionally, the following tensors can be used: prediction residual tensor 407 corresponding to H×W resolution; prediction residual tensor 417 corresponding to H / 2×W / 2 resolution; prediction residual tensor 427 corresponding to H / 4×W / 4 resolution; prediction residual tensor 437 corresponding to H / 8×W / 8 resolution; intermediate residual tensor 410 corresponding to H / 2×W / 2 resolution; intermediate residual tensor 420 corresponding to H / 4×W / 4 resolution; intermediate residual tensor 430 corresponding to H / 8×W / 8 resolution; and intermediate residual tensor 440 corresponding to H / 16×W / 16 resolution. (See below for further details.) Figure 9 This will be described.

[0107] In residual coding, additionally, prediction residual tensors 407 corresponding to H×W resolution, 417 corresponding to H / 2×W / 2 resolution, 427 corresponding to H / 4×W / 4 resolution, and 437 corresponding to H / 8×W / 8 resolution can be used. (See below for further details.) Figure 10 This will be described.

[0108] In residual coding, intermediate residual tensors 410 corresponding to a resolution of H / 2×W / 2, 420 corresponding to a resolution of H / 4×W / 4, 430 corresponding to a resolution of H / 8×W / 8, and 440 corresponding to a resolution of H / 16×W / 16 can also be used. (See below for further details.) Figure 11 This will be described.

[0109] In residual coding, the following tensors can be used to replace multiple prediction tensors 401, 411, 421, and 431: prediction residual tensor 407 corresponding to H×W resolution, prediction residual tensor 417 corresponding to H / 2×W / 2 resolution, prediction residual tensor 427 corresponding to H / 4×W / 4 resolution, prediction residual tensor 437 corresponding to H / 8×W / 8 resolution, intermediate residual tensor 410 corresponding to H / 2×W / 2 resolution, intermediate residual tensor 420 corresponding to H / 4×W / 4 resolution, intermediate residual tensor 430 corresponding to H / 8×W / 8 resolution, and intermediate residual tensor 440 corresponding to H / 16×W / 16 resolution. (See below for further details.) Figure 12 This will be described.

[0110] In residual coding, multiple prediction tensors 401, 411, 421, and 431 can be replaced by prediction residual tensor 407 corresponding to H×W resolution, prediction residual tensor 417 corresponding to H / 2×W / 2 resolution, prediction residual tensor 427 corresponding to H / 4×W / 4 resolution, and prediction residual tensor 437 corresponding to H / 8×W / 8 resolution. See below for further details. Figure 13 This will be described.

[0111] In residual coding, intermediate residual tensors 410 (corresponding to a resolution of H / 2×W / 2), 420 (corresponding to a resolution of H / 4×W / 4), 430 (corresponding to a resolution of H / 8×W / 8), and 440 (corresponding to a resolution of H / 16×W / 16) can be used to replace multiple prediction tensors 401, 411, 421, and 431. See below for further details. Figure 14 This will be described.

[0112] Although embodiments of this disclosure describe prediction tensors at four resolutions, this disclosure is not limited thereto, and prediction tensors at fewer or more resolutions may be obtained.

[0113] Because multiple prediction tensors contain a large amount of information at their respective resolutions, they can be referred to as high-frequency data. Alternatively, because multiple prediction tensors include prediction information that is ultimately used at their respective resolutions, they can be referred to as prediction data.

[0114] Since the prediction residual tensor is the tensor with the applied memory gate subtracted from the image or tensor with the memory gate never applied at the corresponding resolution, the prediction residual tensor can be referred to as residual data or residual tensor. Alternatively, because the prediction residual tensor includes information available at all resolutions down from the corresponding resolution, the prediction residual tensor can be referred to as whole data or whole tensor.

[0115] Because the intermediate prediction tensor is a tensor scaled down from the corresponding resolution and therefore includes relatively little information, it can be referred to as low-frequency data or a low-frequency tensor. Alternatively, because the intermediate prediction tensor includes the remaining information used at the scaled-down resolution, it can be referred to as redundant data or a redundant tensor.

[0116] The prediction tensor corresponds to the memory gate value at the corresponding resolution, the prediction residual tensor corresponds to the forget gate value at the corresponding resolution, and the intermediate prediction tensor corresponds to the shrinking neural network.

[0117] The prediction tensor can be referred to as the prediction image feature map or the prediction image feature data.

[0118] The prediction residual tensor can be referred to as the prediction residual image feature map or the prediction residual image feature data.

[0119] The intermediate prediction tensor can be referred to as the intermediate prediction image feature map or intermediate prediction image feature data.

[0120] Figure 5 This is a diagram used to describe the depth prediction decomposition process according to embodiments of the present disclosure.

[0121] Reference Figure 5 The currently predicted image 500 consists of three channels: red, green, and blue (RGB), with a height (H) × width (W). The memory gate values ​​502, 512, 522, and 532, corresponding to multiple resolutions, are values ​​corresponding to resolutions of H×W, H / 2×W / 2, H / 4×W / 4, and H / 8×W / 8, respectively. The forget gate values ​​503, 513, 523, and 533, corresponding to multiple resolutions, are values ​​corresponding to resolutions of H×W, H / 2×W / 2, H / 4×W / 4, and H / 8×W / 8, respectively. The memory gate values ​​502, 512, 522, and 532, and the forget gate values ​​503, 513, 523, and 533 can be feature maps for deriving spatial differences from one channel. In addition, the memory gates 502, 512, 522 and 532 and the forget gates 503, 513, 523 and 533 can be feature maps of one or more channels depending on the characteristics of the image.

[0122] when Figure 4 In the embodiments, the forget gate values ​​403, 413, 423, and 433 are G f,k k=0, 1, 2, 3, resolution is H / (2 k ) x W / (2 k ),and Figure 5 The memory gate values ​​502, 512, 522, and 532 are G r,k And with Figure 4 When the memory gate values ​​402, 412, 422, and 432 are the same,Figure 5 The forgetting gate values ​​503, 513, 523, and 533 become G. f,k (1-G r,k ).

[0123] First, a prediction tensor 501 is obtained by multiplying the current predicted image 500 by (as shown by reference numeral 504) a memory gate value 502 corresponding to the original resolution H×W of the current predicted image 500. This prediction tensor 501 corresponds to the resolution H×W and retains information about the pixels that are well predicted.

[0124] By multiplying the current predicted image 500 by (as shown in reference numeral 506) and the forget gate value 503 corresponding to the original resolution H×W, a prediction residual tensor 507 corresponding to the resolution H×W is obtained. The prediction residual tensor 507 is applied to the neural network 508 to obtain an intermediate prediction tensor 510 corresponding to the resolution H / 2×W / 2. The intermediate prediction tensor 510 corresponding to the resolution H / 2×W / 2 is multiplied by (as shown in reference numeral 514) and the memory gate value 512 corresponding to the resolution H / 2×W / 2 to obtain a prediction tensor 511 corresponding to the resolution H / 2×W / 2.

[0125] By multiplying the intermediate prediction tensor 510 corresponding to a resolution of H / 2×W / 2 by (as shown in reference numeral 516) the forget gate value 513 corresponding to a resolution of H / 2×W / 2, a prediction residual tensor 517 corresponding to a resolution of H / 2×W / 2 is obtained. The prediction residual tensor 517 is applied to the neural network 518 to obtain the intermediate prediction tensor 520 corresponding to a resolution of H / 4×W / 4. The intermediate prediction tensor 520 corresponding to a resolution of H / 4×W / 4 is multiplied by (as shown in reference numeral 524) the memory gate value 522 corresponding to a resolution of H / 4×W / 4 to obtain the prediction tensor 521 corresponding to a resolution of H / 4×W / 4.

[0126] By multiplying the intermediate prediction tensor 520 corresponding to a resolution of H / 4×W / 4 by (as shown in reference numeral 526) the forget gate value 523 corresponding to a resolution of H / 4×W / 4, a prediction residual tensor 527 corresponding to a resolution of H / 4×W / 4 is obtained. The prediction residual tensor 527 is applied to the neural network 528 to obtain the intermediate prediction tensor 530 corresponding to a resolution of H / 8×W / 8. The intermediate prediction tensor 530 corresponding to a resolution of H / 8×W / 8 is multiplied by (as shown in reference numeral 534) the memory gate value 532 corresponding to a resolution of H / 8×W / 8 to obtain the prediction tensor 531 corresponding to a resolution of H / 8×W / 8.

[0127] By multiplying the intermediate prediction tensor 530 corresponding to the H / 8×W / 8 resolution by (as shown in reference numeral 536) the forget gate value 533 corresponding to the H / 8×W / 8 resolution, the prediction residual tensor 537 corresponding to the H / 8×W / 8 resolution is obtained. The intermediate prediction tensor 540 corresponding to the H / 16×W / 16 resolution is obtained by applying the prediction residual tensor 537 to the neural network 538.

[0128] Multiple prediction tensors 501, 511, 521 and 531 corresponding to the four resolutions obtained through this process can be used for residual coding.

[0129] because Figure 5 The forgetting gate values ​​503, 513, 523, and 533 are obtained by subtracting 1 from the threshold value. Figure 5 The value obtained by multiplying each of the memory gate values ​​502, 512, 522, and 532 by... Figure 4 The forgetting gate values ​​403, 413, 423, and 433 were obtained, so... Figure 5 The predicted residual tensors 507, 517, 527, and 537 can therefore be correlated with... Figure 4 The predicted residual tensors 407, 417, 427 and 437 are the same.

[0130] In residual coding, additionally, the following tensors can be used: a prediction residual tensor 507 corresponding to a resolution of H×W; a prediction residual tensor 517 corresponding to a resolution of H / 2×W / 2; a prediction residual tensor 527 corresponding to a resolution of H / 4×W / 4; a prediction residual tensor 537 corresponding to a resolution of H / 8×W / 8; an intermediate residual tensor 510 corresponding to a resolution of H / 2×W / 2; an intermediate residual tensor 520 corresponding to a resolution of H / 4×W / 4; an intermediate residual tensor 530 corresponding to a resolution of H / 8×W / 8; and an intermediate residual tensor 540 corresponding to a resolution of H / 16×W / 16. (See below for further details.) Figure 9 This will be described.

[0131] In residual coding, additionally, prediction residual tensors 507 corresponding to H×W resolution, 517 corresponding to H / 2×W / 2 resolution, 527 corresponding to H / 4×W / 4 resolution, and 537 corresponding to H / 8×W / 8 resolution can be used. (See below for further details.) Figure 10 This will be described.

[0132] In residual coding, intermediate residual tensors 510 corresponding to a resolution of H / 2×W / 2, 520 corresponding to a resolution of H / 4×W / 4, 530 corresponding to a resolution of H / 8×W / 8, and 540 corresponding to a resolution of H / 16×W / 16 can also be used. (See below for further details.) Figure 11 This will be described.

[0133] In residual coding, the following tensors can be used to replace multiple prediction tensors 501, 511, 521, and 531: prediction residual tensor 507 corresponding to H×W resolution, prediction residual tensor 517 corresponding to H / 2×W / 2 resolution, prediction residual tensor 527 corresponding to H / 4×W / 4 resolution, prediction residual tensor 537 corresponding to H / 8×W / 8 resolution, intermediate residual tensor 510 corresponding to H / 2×W / 2 resolution, intermediate residual tensor 520 corresponding to H / 4×W / 4 resolution, intermediate residual tensor 530 corresponding to H / 8×W / 8 resolution, and intermediate residual tensor 540 corresponding to H / 16×W / 16 resolution. (See below for further details.) Figure 12 This will be described.

[0134] In residual coding, multiple prediction tensors 501, 511, 521, and 531 can be replaced by prediction residual tensor 507 corresponding to H×W resolution, prediction residual tensor 517 corresponding to H / 2×W / 2 resolution, prediction residual tensor 527 corresponding to H / 4×W / 4 resolution, and prediction residual tensor 537 corresponding to H / 8×W / 8 resolution. See below for further details. Figure 13 This will be described.

[0135] In residual coding, intermediate residual tensors 510 (corresponding to a resolution of H / 2×W / 2), 520 (corresponding to a resolution of H / 4×W / 4), 530 (corresponding to a resolution of H / 8×W / 8), and 540 (corresponding to a resolution of H / 16×W / 16) can be used to replace multiple prediction tensors 501, 511, 521, and 531. See below for further details. Figure 14 This will be described.

[0136] Although embodiments of this disclosure describe prediction tensors at four resolutions, this disclosure is not limited thereto, and prediction tensors at fewer or more resolutions may be obtained.

[0137] Figure 6 This is a diagram used to describe the depth prediction decomposition process according to embodiments of the present disclosure.

[0138] Reference Figure 6The currently predicted image 600 consists of three channels: red, green, and blue (RGB), with a height (H) × width (W) dimension. The memory gate values ​​602, 612, 422, and 632, corresponding to multiple resolutions, are values ​​corresponding to resolution sizes of H×W, H / 2×W / 2, H / 4×W / 4, and H / 8×W / 8, respectively. The memory gate values ​​602, 612, 622, and 632 can be feature maps of one channel for deriving spatial differences. Furthermore, the memory gate values ​​602, 612, 622, and 632 can be feature maps of one or more channels, depending on the characteristics of the image.

[0139] First, a prediction tensor 601 is obtained by multiplying the current predicted image 600 by (as shown by reference numeral 604) a memory gate value 602 corresponding to the original resolution H×W of the current predicted image 600. This prediction tensor 601 corresponds to the resolution H×W and retains information about the pixels that are well predicted.

[0140] By subtracting the prediction tensor 601 corresponding to the H×W resolution (as shown in reference numeral 605) from the current predicted image 600, a prediction residual tensor 607 corresponding to the H×W resolution is obtained. The prediction residual tensor 607 is applied to a neural network 608 to obtain an intermediate prediction tensor 610 corresponding to the H / 2×W / 2 resolution. The convolutional kernel of the first layer of the neural network 608 can be linearly mixed with a forgetting gate corresponding to the H×W resolution. Therefore, the intermediate prediction tensor 610 can be an image from which information about pixels with poor predictions has been removed. By multiplying the intermediate prediction tensor 610 corresponding to the H / 2×W / 2 resolution by a memory gate 612 corresponding to the H / 2×W / 2 resolution (as shown in reference numeral 614), a prediction tensor 611 corresponding to the H / 2×W / 2 resolution is obtained.

[0141] By subtracting the prediction tensor 611 (as shown in reference numeral 615) corresponding to the H / 2×W / 2 resolution from the intermediate prediction tensor 610 corresponding to the H / 2×W / 2 resolution, a prediction residual tensor 617 corresponding to the H / 2×W / 2 resolution is obtained. The intermediate prediction tensor 620 corresponding to the H / 4×W / 4 resolution is obtained by applying the prediction residual tensor 617 to a neural network 618. The convolutional kernel of the first layer of the neural network 618 can be linearly mixed with a forgetting gate corresponding to the H / 2×W / 2 resolution. Therefore, the intermediate prediction tensor 620 can be an image from which information about pixels with poor predictions has been removed. The prediction tensor 621 corresponding to the H / 4×W / 4 resolution is obtained by multiplying the intermediate prediction tensor 620 corresponding to the H / 4×W / 4 resolution by a memory gate 622 (as shown in reference numeral 624).

[0142] By subtracting the prediction tensor 621 corresponding to the H / 4×W / 4 resolution (as shown in reference numeral 425) from the intermediate prediction tensor 620 corresponding to the H / 4×W / 4 resolution, a prediction residual tensor 627 corresponding to the H / 4×W / 4 resolution is obtained. The intermediate prediction tensor 630 corresponding to the H / 8×W / 8 resolution is obtained by applying the prediction residual tensor 627 to a neural network 628. The convolutional kernel of the first layer of the neural network 628 can be linearly mixed with a forgetting gate corresponding to the H / 4×W / 4 resolution. Therefore, the intermediate prediction tensor 630 can be an image from which information about pixels with poor predictions has been removed. The prediction tensor 631 corresponding to the H / 8×W / 8 resolution is obtained by multiplying the intermediate prediction tensor 630 corresponding to the H / 8×W / 8 resolution by a memory gate 632 (as shown in reference numeral 634).

[0143] By subtracting the prediction tensor 631 (as shown by reference numeral 635) corresponding to the H / 8×W / 8 resolution from the intermediate prediction tensor 630 corresponding to the H / 8×W / 8 resolution, a prediction residual tensor 637 corresponding to the H / 8×W / 8 resolution is obtained. An intermediate prediction tensor 640 corresponding to the H / 16×W / 16 resolution is obtained by applying the prediction residual tensor 637 to a neural network 638. The convolutional kernel of the first layer of the neural network 638 can be linearly mixed with a forgetting gate corresponding to the H / 8×W / 8 resolution. Therefore, the intermediate prediction tensor 630 can be an image from which information about pixels with poor predictions has been removed.

[0144] The multiple prediction tensors 601, 611, 621 and 631 obtained through this process, corresponding to the four resolutions, can be used for residual coding.

[0145] because Figure 6 The convolutional kernels of the first layer of neural networks 608, 618, 628, and 638 are mixed with forgetting gates corresponding to their respective resolutions, so... Figure 6 The predicted residual tensors 607, 617, 627, and 637 can therefore be correlated with... Figure 4 The predicted residual tensors 407, 417, 427 and 437 are the same.

[0146] In residual coding, additionally, the following tensors can be used: a prediction residual tensor 607 corresponding to a resolution of H×W; a prediction residual tensor 617 corresponding to a resolution of H / 2×W / 2; a prediction residual tensor 627 corresponding to a resolution of H / 4×W / 4; a prediction residual tensor 637 corresponding to a resolution of H / 8×W / 8; an intermediate residual tensor 610 corresponding to a resolution of H / 2×W / 2; an intermediate residual tensor 620 corresponding to a resolution of H / 4×W / 4; an intermediate residual tensor 630 corresponding to a resolution of H / 8×W / 8; and an intermediate residual tensor 640 corresponding to a resolution of H / 16×W / 16. (See below for further details.) Figure 9 This will be described.

[0147] In residual coding, additionally, prediction residual tensors 607 corresponding to H×W resolution, 617 corresponding to H / 2×W / 2 resolution, 627 corresponding to H / 4×W / 4 resolution, and 637 corresponding to H / 8×W / 8 resolution can be used. (See below for further details.) Figure 10 This will be described.

[0148] In residual coding, intermediate residual tensors 610 corresponding to a resolution of H / 2×W / 2, 620 corresponding to a resolution of H / 4×W / 4, 630 corresponding to a resolution of H / 8×W / 8, and 640 corresponding to a resolution of H / 16×W / 16 can also be used. (See below for further details.) Figure 11 This will be described.

[0149] In residual coding, the following tensors can be used to replace multiple prediction tensors 601, 611, 621, and 631: prediction residual tensor 607 corresponding to H×W resolution, prediction residual tensor 617 corresponding to H / 2×W / 2 resolution, prediction residual tensor 627 corresponding to H / 4×W / 4 resolution, prediction residual tensor 637 corresponding to H / 8×W / 8 resolution, intermediate residual tensor 610 corresponding to H / 2×W / 2 resolution, intermediate residual tensor 620 corresponding to H / 4×W / 4 resolution, intermediate residual tensor 630 corresponding to H / 8×W / 8 resolution, and intermediate residual tensor 640 corresponding to H / 16×W / 16 resolution. (See below for further details.) Figure 12 This will be described.

[0150] In residual coding, multiple prediction tensors 601, 611, 621, and 631 can be replaced by prediction residual tensor 607 corresponding to H×W resolution, prediction residual tensor 617 corresponding to H / 2×W / 2 resolution, prediction residual tensor 627 corresponding to H / 4×W / 4 resolution, and prediction residual tensor 637 corresponding to H / 8×W / 8 resolution. See below for further details. Figure 13 This will be described.

[0151] In residual coding, intermediate residual tensors 610 (corresponding to a resolution of H / 2×W / 2), 620 (corresponding to a resolution of H / 4×W / 4), 630 (corresponding to a resolution of H / 8×W / 8), and 640 (corresponding to a resolution of H / 16×W / 16) can be used to replace multiple prediction tensors 601, 611, 621, and 631. See below for further details. Figure 14 This will be described.

[0152] Although embodiments of this disclosure describe prediction tensors at four resolutions, this disclosure is not limited thereto, and prediction tensors at fewer or more resolutions may be obtained.

[0153] Figure 7 This is a diagram illustrating the structure of a motion encoder, motion decoder, multi-compensated pixel encoder, and multi-compensated pixel decoder according to embodiments of the present disclosure.

[0154] Reference Figure 7 A reference image 705 and the original image 700, used as previously reconstructed images, are input to the motion encoder 710. Feature data 715 for the current optical flow is sequentially downscaled and output through multiple neural networks 711, 712, 713, and 716 within the motion encoder 710. Each of the multiple neural networks 711, 712, 713, and 716 within the motion encoder 710 may include at least one convolutional layer. Furthermore, the multiple neural networks 711, 712, 713, and 716 can downscale the input data and output the downscaled result.

[0155] Feature data 715 for the current optical flow, output by motion encoder 710, is input to motion decoder 720. The feature data 715 for the current optical flow is then input to network 721, thus obtaining first intermediate data. Additionally, the feature data 715 for the current optical flow is input to convolutional layer 725 and sigmoid function 726 to obtain a first-resolution memory gate value 733 and a first-resolution forget gate value 734. The sigmoid function, one of the activation functions used in the neural network, is a non-linear function that outputs values ​​between 0 and 1 from the input data. Therefore, the memory gate value and forget gate value obtained by the sigmoid function are values ​​between 0 and 1. The first intermediate data is input to neural network 722, thus obtaining second intermediate data. Furthermore, the first intermediate data is input to convolutional layer 727 and sigmoid function 728 to obtain a second-resolution memory gate value 735 and a second-resolution forget gate value 736. The second intermediate data is input to neural network 723, thus obtaining third intermediate data. Furthermore, the second intermediate data is input to convolutional layer 729 and sigmoid function 730 to obtain a memory gate value 737 and a forget gate value 738 at a third resolution. The third intermediate data is input to neural network 724, thus obtaining the current optical flow. Additionally, the third intermediate data is input to convolutional layer 731 and sigmoid function 732 to obtain a memory gate value 739 and a forget gate value 740 at a fourth resolution. For example, the fourth resolution may correspond to the resolution of the original image, the third resolution may correspond to half the resolution of the original image, the second resolution may correspond to a quarter the resolution of the original image, and the first resolution may correspond to an eighth the resolution of the original image. Each of the neural networks 721, 722, 723, and 724 within the motion decoder 720 may include at least one convolutional layer. Furthermore, neural networks 721, 722, 723, and 724 may amplify the input data. In addition, the convolutional layers 725, 727, 729 and 731 within the motion decoder 720 can amplify the input data.

[0156] A predicted image 743 for the current image is obtained by distorting (as shown by reference numeral 742) the current optical flow 741 and the reference image 705.

[0157] By using memory gates 733, 735, 737, and 739 corresponding to multiple resolutions, forget gates 734, 736, 738, and 740 corresponding to multiple resolutions, and a prediction image 743, prediction tensors 745, 746, 747, and 748 corresponding to multiple resolutions are obtained through depth prediction decomposition 744. (The above has already been referred to...) Figure 4 to Figure 6 The depth prediction decomposition 744 is described, so its description will be omitted.

[0158] In the multi-compensated pixel encoder 750, firstly, a first subtraction tensor obtained by subtracting (as shown by reference numeral 751) the prediction tensor 745 corresponding to the original resolution from the original image 700 is input into neural network 752. Neural network 752 outputs an intermediate encoded tensor of the third resolution. A second subtraction tensor obtained by subtracting (as shown by reference numeral 753) the prediction tensor 746 of the third resolution from the intermediate encoded tensor of the third resolution is input into neural network 754. Neural network 754 outputs an intermediate encoded tensor of the second resolution. A third subtraction tensor obtained by subtracting (as shown by reference numeral 755) the prediction tensor 747 of the second resolution from the intermediate encoded tensor of the second resolution is input into neural network 756. Neural network 754 outputs an intermediate encoded tensor of the first resolution. A fourth subtraction tensor obtained by subtracting (as shown by reference numeral 757) the prediction tensor 748 of the first resolution from the intermediate encoded tensor of the first resolution is input into neural network 758. The residual image feature data 759 is output through neural network 758. Each of the plurality of neural networks 752, 754, 756 and 758 within the multi-compensated pixel encoder 750 may include at least one convolutional layer. In addition, the plurality of neural networks 752, 754, 756 and 758 can reduce the input data and output the reduced result.

[0159] In the multi-compensated pixel decoder 760, firstly, residual image feature data 759 is input into neural network 761. A residual tensor at a first resolution is obtained through neural network 761. The residual tensor at the first resolution is summed with the prediction tensor at the first resolution 748 (as shown in reference numeral 762) and input into neural network 763. A residual tensor at a second resolution is obtained through neural network 763. The residual tensor at the second resolution is summed with the prediction tensor at the second resolution 747 (as shown in reference numeral 764) and input into neural network 765. A residual tensor at a third resolution is obtained through neural network 765. The residual tensor at the third resolution is summed with the prediction tensor at the third resolution 746 (as shown in reference numeral 766) and input into neural network 767. A residual tensor at the first resolution is obtained through neural network 767. The residual tensor at the first resolution is summed with the prediction tensor at the first resolution 745 (as shown in reference numeral 768). As a result of the summation 768, a reconstructed image 770 is output. Each of the neural networks 761, 763, 765, and 767 within the multi-compensated pixel decoder 760 may include at least one convolutional layer. Additionally, neural networks 761, 763, 765, and 767 may amplify the input data.

[0160] In addition to the residual image feature data 759, the multi-compensated pixel decoder 760 can also receive the feature data 715 of the current optical flow. The residual image feature data 759 and the feature data 715 of the current optical flow can be cascaded together and can be input.

[0161] The multi-compensated pixel encoder 750 and the multi-compensated pixel decoder 760 sequentially perform residual coding several times according to multiple resolutions in the pixel domain.

[0162] Figure 8 This is a diagram illustrating the structure of a motion decoder according to an embodiment of the present disclosure.

[0163] Reference Figure 8 The original frame 800 and the reference frame 805 are input to the motion encoder 820. Additional information 810 about the current image can be input to the motion encoder 820. The motion encoder 820 outputs the feature data 825 of the current optical flow. The structure of the motion encoder 820 can be the same as the above reference frame. Figure 7 The motion encoder 710 described has the same structure.

[0164] The current optical flow feature data 825 is input into the neural network 831 of the motion decoder 830. First intermediate data and first gate data are obtained from the neural network 831. The first intermediate data is applied to the sigmoid function 832 to obtain a first-resolution memory gate value 847 and a first-resolution forget gate value 848. The first intermediate data is input into the neural network 833. Second intermediate data and second gate data are obtained from the neural network 833. The second intermediate data is applied to the sigmoid function 834 to obtain a second-resolution memory gate value 845 and a second-resolution forget gate value 846. The second intermediate data is input into the neural network 835. Third intermediate data and third gate data are obtained from the neural network 835. The third intermediate data is applied to the sigmoid function 836 to obtain a third-resolution memory gate value 843 and a third-resolution forget gate value 844. The third intermediate data is input into the neural network 837. The current optical flow 850 and fourth intermediate data are obtained from the neural network 837. The fourth intermediate data is applied to the sigmoid function 838 to obtain a memory gate 841 and a forget gate 842 for the fourth resolution. For example, the fourth resolution may correspond to the resolution of the original image, the third resolution may correspond to half the resolution of the original image, the second resolution may correspond to a quarter the resolution of the original image, and the first resolution may correspond to an eighth the resolution of the original image. Each of the neural networks 831, 833, 835, and 837 within the motion decoder 830 may include at least one convolutional layer. Furthermore, the neural networks 831, 833, 835, and 837 may amplify the input data.

[0165] because Figure 7 Motion decoder 720 and Figure 8 The motion decoder 830 outputs the current optical flow, memory gate, and forget gate based on the feature data 715 and 825 of the current optical flow. Figure 7 Motion decoder 720 and Figure 8 The motion decoder 830 can be described as a multi-purpose motion decoder. That is, optical flow encoding and decoding, as well as gate generation, can be effectively combined with each other. Furthermore, Figure 7 Motion encoder 710 and its corresponding Figure 8 The motion encoder 820 can be referred to as a multi-purpose motion encoder. The memory gate and forget gate can also be referred to as decomposition gates or decomposition weights.

[0166] According to embodiments of this disclosure, memory gates and forget gates can be generated independently of the corresponding encoders and decoders, rather than being output based on motion encoders and decoders. For example, an original image and a reference image can be input to a memory gate encoder, memory gate feature data can be output by the memory gate encoder, memory gate feature data can be input to a memory gate decoder, and memory gates corresponding to multiple resolutions can be output by the memory gate decoder. Similarly, an original image and a reference image can be input to a forget gate encoder, forget gate feature data can be output by the forget gate encoder, forget gate feature data can be input to a forget gate decoder, and forget gates corresponding to multiple resolutions can be output by the forget gate decoder.

[0167] Figure 9 This is a diagram illustrating the structure of a multi-compensated pixel encoder and a multi-compensated pixel decoder according to embodiments of the present disclosure.

[0168] Reference Figure 9 In the multi-compensated pixel encoder 910, firstly, a first subtraction tensor obtained by subtracting (as shown by reference numeral 911) the prediction tensor 901 of the original resolution from the original image 900, and a prediction residual tensor 944 of the original resolution, are input into the neural network 912. The first subtraction tensor and the prediction residual tensor 944 can be cascaded together and input into each other. The prediction residual tensor 944 of the original resolution and... Figure 4 The predicted residual tensor is 407. Figure 5 The predicted residual tensor 507 or Figure 6 The predicted residual tensor corresponds to 607.

[0169] The neural network 912 outputs an intermediate encoded tensor at a first reduced resolution obtained by downscaling the original resolution. A second subtraction tensor, obtained by subtracting (as shown by reference numeral 913) the prediction tensor 902 at the first reduced resolution from the intermediate encoded tensor at the first reduced resolution, and the prediction residual tensor 943 at the first reduced resolution, are input into the neural network 914. The second subtraction tensor and the prediction residual tensor 943 can be concatenated and input together. The prediction residual tensor 943 at the first reduced resolution and... Figure 4 The predicted residual tensor 417 Figure 5 The predicted residual tensor 517 or Figure 6 The predicted residual tensor corresponds to 617.

[0170] The neural network 914 outputs an intermediate encoded tensor at a second reduced resolution obtained by reducing the first reduced resolution. A third subtraction tensor, obtained by subtracting (as shown in reference numeral 915) the prediction tensor 903 at the second reduced resolution from the intermediate encoded tensor at the second reduced resolution, and the prediction residual tensor 942 at the second reduced resolution, are input into the neural network 916. The third subtraction tensor and the prediction residual tensor 942 can be concatenated and input together. The prediction residual tensor 942 at the second reduced resolution and... Figure 4 The predicted residual tensor is 427. Figure 5 The predicted residual tensor is 527 or Figure 6 The predicted residual tensor corresponds to 627.

[0171] The neural network 916 outputs an intermediate encoded tensor at a third reduced resolution obtained by downscaling from the second reduced resolution. A fourth subtraction tensor, obtained by subtracting (as shown in reference numeral 917) the prediction tensor 904 at the third reduced resolution from the intermediate encoded tensor at the third reduced resolution, and the prediction residual tensor 941 at the third reduced resolution, are input into the neural network 918. The fourth subtraction tensor and the prediction residual tensor 941 can be concatenated and input together. The prediction residual tensor 941 at the third reduced resolution and... Figure 4 The predicted residual tensor is 437. Figure 5 The predicted residual tensor is 537 or Figure 6 The predicted residual tensor corresponds to 637.

[0172] The residual image feature data 930 is output through neural network 918. Each of the plurality of neural networks 912, 914, 916 and 918 within the multi-compensated pixel encoder 910 may include at least one convolutional layer. In addition, the plurality of neural networks 912, 914, 916 and 918 can reduce the input data and output the reduced result.

[0173] In the multi-compensated pixel decoder 920, firstly, the intermediate prediction tensor 931 obtained by reducing the third reduction resolution, along with the residual image feature data 930, is input into the neural network 921. The residual image feature data 930 and the intermediate prediction tensor 931 can be concatenated and can be input into each other. The intermediate prediction tensor 931 at the fourth reduction resolution and... Figure 4 Intermediate prediction tensor 440 Figure 5 The intermediate prediction tensor is 540 or Figure 6 The intermediate prediction tensor 640 corresponds to this.

[0174] The residual tensor at the third reduced resolution is obtained through neural network 921. The residual tensor at the third reduced resolution is summed with the prediction tensor at the third reduced resolution 904 (as shown in reference numeral 922) and then input into neural network 923. Additionally, the intermediate prediction tensor at the third reduced resolution 932 is also input into neural network 923. The data corresponding to the result of summation 922 and the intermediate prediction tensor 932 can be concatenated and input. The intermediate prediction tensor at the third reduced resolution 932 and... Figure 4 Intermediate prediction tensor 430 Figure 5 Intermediate prediction tensor 530 or Figure 6 The intermediate prediction tensor 630 corresponds to this.

[0175] The residual tensor at the second reduced resolution is obtained through neural network 923. The residual tensor at the second reduced resolution is summed with the prediction tensor at the second reduced resolution 903 (as shown in reference numeral 924) and input into neural network 925. Additionally, the intermediate prediction tensor at the second reduced resolution 933 is also input into neural network 925. The data corresponding to the result of summation 924 and the intermediate prediction tensor 933 can be concatenated and input. The intermediate prediction tensor 933 at the second reduced resolution and... Figure 4 Intermediate prediction tensor 420 Figure 5 The intermediate prediction tensor is 520 or Figure 6 The intermediate prediction tensor 620 corresponds to this.

[0176] The residual tensor at the first reduced resolution is obtained through neural network 925. The residual tensor at the first reduced resolution is summed with the prediction tensor 902 at the first reduced resolution (as shown in reference numeral 926) and input into neural network 927. Additionally, the intermediate prediction tensor 934 at the first reduced resolution is also input into neural network 927. The data corresponding to the result of summation 926 and the intermediate prediction tensor 934 can be concatenated and input. The intermediate prediction tensor 934 at the first reduced resolution and... Figure 4 Intermediate prediction tensor 410 Figure 5 Intermediate prediction tensor 510 or Figure 6 The intermediate prediction tensor 610 corresponds to this.

[0177] The residual tensor at the original resolution is obtained through neural network 927. The residual tensor at the original resolution is summed with the predicted tensor at the original resolution 901 (as shown in reference numeral 928). The reconstructed image 950 is output as the result of the summation 928.

[0178] Each of the neural networks 921, 923, 925, and 927 within the multi-compensated pixel decoder 920 may include at least one convolutional layer. Additionally, neural networks 921, 923, 925, and 927 may amplify the input data.

[0179] Predicting tensors 901, 902, 903, and 904 with Figure 10 The predicted tensors 401, 402, 403, and 404, Figure 10 The predicted tensors 501, 502, 503, and 504 or Figure 4 The predicted tensors 601, 602, 603, and 604 correspond to this.

[0180] In addition to the residual image feature data 930, the multi-compensated pixel decoder 920 can also receive the feature data of the current optical flow. The residual image feature data 930 and the feature data of the current optical flow can be cascaded together and can be input.

[0181] Figure 5 This is a diagram illustrating the structure of a multi-compensated pixel encoder and a multi-compensated pixel decoder according to embodiments of the present disclosure.

[0182] Reference Figure 6 In the multi-compensated pixel encoder 1010, firstly, a first subtraction tensor obtained by subtracting (as shown by reference numeral 1011) the prediction tensor 1001 of the original resolution from the original image 1000, and a prediction residual tensor 1044 of the original resolution, are input into the neural network 1012. The first subtraction tensor and the prediction residual tensor 1044 can be cascaded together and input into each other. The prediction residual tensor 1044 of the original resolution and... Figure 4 The predicted residual tensor is 407. Figure 5 The predicted residual tensor 507 or Figure 6 The predicted residual tensor corresponds to 607.

[0183] The neural network 1012 outputs an intermediate encoded tensor of a first reduced resolution obtained by reducing the original resolution. A second subtraction tensor, obtained by subtracting (as shown by reference numeral 1013) the prediction tensor 1002 of the first reduced resolution from the intermediate encoded tensor of the first reduced resolution, and the prediction residual tensor 1043 of the first reduced resolution, are input into the neural network 1014. The second subtraction tensor and the prediction residual tensor 1043 can be concatenated and input together. The prediction residual tensor 1043 of the first reduced resolution and... Figure 4 The predicted residual tensor 417 Figure 5 The predicted residual tensor 517 or Figure 6 The predicted residual tensor corresponds to 617.

[0184] The neural network 1014 outputs an intermediate encoded tensor at a second reduced resolution obtained by reducing the first reduced resolution. A third subtraction tensor, obtained by subtracting (as shown in reference numeral 1015) the prediction tensor 1003 at the second reduced resolution from the intermediate encoded tensor at the second reduced resolution, and the prediction residual tensor 1042 at the second reduced resolution, are input into the neural network 1016. The third subtraction tensor and the prediction residual tensor 1042 can be concatenated and input together. The prediction residual tensor 1042 at the second reduced resolution and... Figure 4 The predicted residual tensor is 427. Figure 5 The predicted residual tensor is 527 or Figure 6 The predicted residual tensor corresponds to 627.

[0185] The neural network 1016 outputs an intermediate encoding tensor at a third reduced resolution obtained by downscaling from the second reduced resolution. A fourth subtraction tensor, obtained by subtracting (as shown in reference numeral 1017) the prediction tensor 1004 at the third reduced resolution from the intermediate encoding tensor at the third reduced resolution, and the prediction residual tensor 1041 at the third reduced resolution, are input into the neural network 1018. The fourth subtraction tensor and the prediction residual tensor 1041 can be concatenated and input together. The prediction residual tensor 1041 at the third reduced resolution and... Figure 4 The predicted residual tensor is 437. Figure 5 The predicted residual tensor is 537 or Figure 6 The predicted residual tensor corresponds to 637.

[0186] The residual image feature data 1030 is output through the neural network 1018. Each of the plurality of neural networks 1012, 1014, 1016 and 1018 within the multi-compensated pixel encoder 1010 may include at least one convolutional layer. In addition, the plurality of neural networks 1012, 1014, 1016 and 1018 can reduce the input data and output the reduced result.

[0187] In the multi-compensated pixel decoder 1020, firstly, the residual image feature data 1030 is input into the neural network 1021.

[0188] The residual tensor at the third reduced resolution is obtained through neural network 1021. The residual tensor at the third reduced resolution is summed with the prediction tensor at the third reduced resolution 1004 (as shown in reference numeral 1022) and then fed into neural network 1023.

[0189] The residual tensor at the second reduced resolution is obtained through neural network 1023. The residual tensor at the second reduced resolution is summed with the prediction tensor at the second reduced resolution 1003 (as shown in reference numeral 1024) and then fed into neural network 1025.

[0190] The residual tensor at the first reduced resolution is obtained through neural network 1025. The residual tensor at the first reduced resolution is summed with the prediction tensor 1002 at the first reduced resolution (as shown in reference numeral 1026) and then input into neural network 1027.

[0191] The residual tensor at the original resolution is obtained through neural network 1027. The residual tensor at the original resolution is summed with the predicted tensor at the original resolution 1001 (as shown in reference numeral 1028). As a result of summing 1028, the reconstructed image 1050 is output.

[0192] Each of the neural networks 1021, 1023, 1025, and 1027 within the multi-compensated pixel decoder 1020 may include at least one convolutional layer. Additionally, neural networks 1021, 1023, 1025, and 1027 may amplify the input data.

[0193] Predicting tensors 1001, 1002, 1003, and 1004 with Figure 11 The predicted tensors 401, 402, 403, and 404, Figure 11 The predicted tensors 501, 502, 503, and 504 or Figure 4 The predicted tensors 601, 602, 603, and 604 correspond to this.

[0194] In addition to the residual image feature data 1030, the multi-compensated pixel decoder 1020 can also receive the feature data of the current optical flow. The residual image feature data 1030 and the feature data of the current optical flow can be cascaded together and can be input.

[0195] Figure 5 This is a diagram illustrating the structure of a multi-compensated pixel encoder and a multi-compensated pixel decoder according to embodiments of the present disclosure.

[0196] Reference Figure 6In the multi-compensated pixel encoder 1110, firstly, a first subtraction tensor obtained by subtracting the prediction tensor 1101 of the original resolution (as shown by reference numeral 1111) from the original image 1100 is input into the neural network 1112.

[0197] The neural network 1112 outputs an intermediate encoding tensor of the first reduced resolution obtained by reducing the original resolution. The second subtraction tensor obtained by subtracting the prediction tensor 1102 of the first reduced resolution (as shown in reference numeral 1113) from the intermediate encoding tensor of the first reduced resolution is input into the neural network 1114.

[0198] The neural network 1114 outputs an intermediate encoding tensor at a second reduced resolution obtained by reducing the first reduced resolution. A third subtraction tensor, obtained by subtracting (as shown in reference numeral 1115) the prediction tensor 1103 at the second reduced resolution from the intermediate encoding tensor at the second reduced resolution, is input into the neural network 1116.

[0199] The neural network 1116 outputs an intermediate encoding tensor at a third reduced resolution obtained by reducing the second reduced resolution. A fourth subtraction tensor, obtained by subtracting (as shown in reference numeral 1117) the prediction tensor 1104 at the third reduced resolution from the intermediate encoding tensor at the third reduced resolution, is input into the neural network 1118.

[0200] The residual image feature data 1130 is output through neural network 1118. Each of the plurality of neural networks 1112, 1114, 1116 and 1118 within the multi-compensated pixel encoder 1110 may include at least one convolutional layer. In addition, the plurality of neural networks 1112, 1114, 1116 and 1118 can reduce the input data and output the reduced result.

[0201] In the multi-compensated pixel decoder 1120, firstly, the intermediate prediction tensor 1131 obtained by reducing the resolution to a fourth reduced resolution, and the residual image feature data 1130, are input into the neural network 1121. The residual image feature data 1130 and the intermediate prediction tensor 1131 can be concatenated and can be input to each other. The intermediate prediction tensor 1131 at the fourth reduced resolution and... Figure 4 Intermediate prediction tensor 440 Figure 5 The intermediate prediction tensor is 540 or Figure 6 The intermediate prediction tensor 640 corresponds to this.

[0202] The residual tensor at the third reduced resolution is obtained through neural network 1121. The residual tensor at the third reduced resolution is summed with the prediction tensor at the third reduced resolution 1104 (as shown in reference numeral 1122) and input into neural network 1123. Additionally, the intermediate prediction tensor at the third reduced resolution 1132 is also input into neural network 1123. The data corresponding to the result of summing 1122 and the intermediate prediction tensor 1132 can be concatenated and input. The intermediate prediction tensor at the third reduced resolution and... Figure 4 Intermediate prediction tensor 430 Figure 5 Intermediate prediction tensor 530 or Figure 6 The intermediate prediction tensor 630 corresponds to this.

[0203] The residual tensor at the second reduced resolution is obtained through neural network 1123. The residual tensor at the second reduced resolution is summed with the prediction tensor at the second reduced resolution 1103 (as shown in reference numeral 1124) and input into neural network 1125. Additionally, the intermediate prediction tensor at the second reduced resolution 1133 is also input into neural network 1125. The data corresponding to the result of summing 1124 and the intermediate prediction tensor 1133 can be concatenated and input. The intermediate prediction tensor 1133 at the second reduced resolution and... Figure 4 Intermediate prediction tensor 420 Figure 5 The intermediate prediction tensor is 520 or Figure 6 The intermediate prediction tensor 620 corresponds to this.

[0204] The residual tensor at the first reduced resolution is obtained through neural network 1125. The residual tensor at the first reduced resolution is summed with the prediction tensor 1102 at the first reduced resolution (as shown in reference numeral 1126) and input into neural network 1127. Additionally, the intermediate prediction tensor 1134 at the first reduced resolution is also input into neural network 1127. The data corresponding to the result of summing 1126 and the intermediate prediction tensor 1134 can be concatenated and input. The intermediate prediction tensor 1134 at the first reduced resolution and... Figure 4 Intermediate prediction tensor 410 Figure 5 Intermediate prediction tensor 510 or Figure 6 The intermediate prediction tensor 610 corresponds to this.

[0205] The residual tensor at the original resolution is obtained through neural network 1127. The residual tensor at the original resolution is summed with the predicted tensor at the original resolution 1101 (as shown in reference numeral 1128). The reconstructed image 1150 is output as a result of the summation 1128.

[0206] Each of the neural networks 1121, 1123, 1125, and 1127 within the multi-compensated pixel decoder 1120 may include at least one convolutional layer. Additionally, neural networks 1121, 1123, 1125, and 1127 may amplify the input data.

[0207] Predicting tensors 1101, 1102, 1103, and 1104 with Figure 4 The predicted tensors 401, 402, 403, and 404, Figure 5 The predicted tensors 501, 502, 503, and 504 or Figure 6 The predicted tensors 601, 602, 603, and 604 correspond to this.

[0208] In addition to the residual image feature data 1130, the multi-compensated pixel decoder 1120 can also receive the feature data of the current optical flow. The residual image feature data 1130 and the feature data of the current optical flow can be concatenated and can be input.

[0209] Figure 4 This is a diagram illustrating the structure of a multi-compensated pixel encoder and a multi-compensated pixel decoder according to embodiments of the present disclosure.

[0210] Reference Figure 5 In the multi-compensated pixel encoder 1210, firstly, the original image 1200 and the prediction residual tensor 1244 at the original resolution are input into the neural network 1212. The original image 1200 and the prediction residual tensor 1244 can be cascaded together and input. The prediction residual tensor 1244 at the original resolution and... Figure 6 The predicted residual tensor is 407. Figure 4 The predicted residual tensor 507 or Figure 5 The predicted residual tensor corresponds to 607.

[0211] The neural network 1212 outputs an intermediate encoding tensor at a first reduced resolution, obtained by downscaling the original resolution. The intermediate encoding tensor at the first reduced resolution and the prediction residual tensor at the first reduced resolution 1243 are input into the neural network 1214. The intermediate encoding tensor at the first reduced resolution and the prediction residual tensor 1243 can be concatenated and input together. The prediction residual tensor 1243 at the first reduced resolution and... Figure 6 The predicted residual tensor 417 Figure 4 The predicted residual tensor 517 or Figure 5 The predicted residual tensor corresponds to 617.

[0212] The neural network 1214 outputs an intermediate encoding tensor at a second reduced resolution, obtained by reducing the first reduced resolution. The intermediate encoding tensor at the second reduced resolution and the prediction residual tensor 1242 at the second reduced resolution are input into the neural network 1216. The intermediate encoding tensor at the second reduced resolution and the prediction residual tensor 1242 can be concatenated and input together. The prediction residual tensor 1242 at the second reduced resolution and... Figure 6 The predicted residual tensor is 427. Figure 13 The predicted residual tensor is 527 or Figure 13 The predicted residual tensor corresponds to 627.

[0213] The neural network 1216 outputs an intermediate encoding tensor at a third reduced resolution, obtained by reducing the second reduction resolution. The intermediate encoding tensor at the third reduced resolution and the prediction residual tensor 1241 at the third reduced resolution are input into the neural network 1218. The intermediate encoding tensor at the third reduced resolution and the prediction residual tensor 1241 can be concatenated and input together. The prediction residual tensor 1241 at the third reduced resolution and... Figure 4 The predicted residual tensor is 437. Figure 5 The predicted residual tensor is 537 or Figure 6 The predicted residual tensor corresponds to 637.

[0214] The residual image feature data 1230 is output through the neural network 1218. Each of the plurality of neural networks 1212, 1214, 1216 and 1218 within the multi-compensated pixel encoder 1210 may include at least one convolutional layer. In addition, the plurality of neural networks 1212, 1214, 1216 and 1218 can reduce the input data and output the reduced result.

[0215] In the multi-compensated pixel decoder 1220, firstly, the intermediate prediction tensor 1231 obtained by reducing the third reduction resolution to a fourth reduction resolution, and the residual image feature data 1230, are input into the neural network 1221. The residual image feature data 1230 and the intermediate prediction tensor 1231 can be concatenated and can be input to each other. The intermediate prediction tensor 1231 at the fourth reduction resolution and... Figure 4 Intermediate prediction tensor 440 Figure 5 The intermediate prediction tensor is 540 or Figure 6 The intermediate prediction tensor 640 corresponds to this.

[0216] The residual tensor at the third reduced resolution is obtained through neural network 1221. The residual tensor at the third reduced resolution and the intermediate prediction tensor at the third reduced resolution 1232 are input into neural network 1223. The residual tensor at the third reduced resolution and the intermediate prediction tensor 1232 can be concatenated and input together. The intermediate prediction tensor 1232 at the third reduced resolution and... Figure 4Intermediate prediction tensor 430 Figure 5 Intermediate prediction tensor 530 or Figure 6 The intermediate prediction tensor 630 corresponds to this.

[0217] The residual tensor at the second reduced resolution is obtained through neural network 1223. The residual tensor at the second reduced resolution and the intermediate prediction tensor at the second reduced resolution 1233 are input into neural network 1225. The residual tensor at the second reduced resolution and the intermediate prediction tensor 1233 can be concatenated and input together. The intermediate prediction tensor 1233 at the second reduced resolution and... Figure 4 Intermediate prediction tensor 420 Figure 5 The intermediate prediction tensor is 520 or Figure 6 The intermediate prediction tensor 620 corresponds to this.

[0218] The residual tensor at the first reduced resolution is obtained through neural network 1225. The residual tensor at the first reduced resolution and the intermediate prediction tensor at the first reduced resolution 1234 are input into neural network 1227. The residual tensor at the first reduced resolution and the intermediate prediction tensor 1234 can be concatenated and input together. The intermediate prediction tensor 1234 at the first reduced resolution and... Figure 14 Intermediate prediction tensor 410 Figure 14 Intermediate prediction tensor 510 or Figure 4 The intermediate prediction tensor 610 corresponds to this.

[0219] The neural network 1227 outputs a reconstructed image 1250 at the original resolution.

[0220] Each of the neural networks 1221, 1223, 1225, and 1227 within the multi-compensated pixel decoder 1220 may include at least one convolutional layer. Additionally, neural networks 1221, 1223, 1225, and 1227 may amplify the input data.

[0221] In addition to the residual image feature data 1230, the multi-compensated pixel decoder 1220 can also receive the feature data of the current optical flow. The residual image feature data 1230 and the feature data of the current optical flow can be concatenated and can be input to each other.

[0222] Figure 5 This is a diagram illustrating the structure of a multi-compensated pixel encoder and a multi-compensated pixel decoder according to embodiments of the present disclosure.

[0223] Reference Figure 6 In the multi-compensated pixel encoder 1310, firstly, the original image 1300 and the prediction residual tensor 1344 at the original resolution are input into the neural network 1312. The original image 1300 and the prediction residual tensor 1344 can be cascaded together and input. The prediction residual tensor 1344 at the original resolution and...Figure 4 The predicted residual tensor is 407. Figure 5 The predicted residual tensor 507 or Figure 6 The predicted residual tensor corresponds to 607.

[0224] The neural network 1312 outputs an intermediate encoding tensor at a first reduced resolution, obtained by downscaling the original resolution. The intermediate encoding tensor at the first reduced resolution and the prediction residual tensor at the first reduced resolution 1343 are input into the neural network 1314. The intermediate encoding tensor at the first reduced resolution and the prediction residual tensor at the first reduced resolution 1343 can be concatenated and input together. The prediction residual tensor at the first reduced resolution 1343 and... Figure 4 The predicted residual tensor 417 Figure 5 The predicted residual tensor 517 or Figure 6 The predicted residual tensor corresponds to 617.

[0225] The neural network 1314 outputs an intermediate encoding tensor at a second reduced resolution, obtained by reducing the first reduced resolution. The intermediate encoding tensor at the second reduced resolution and the prediction residual tensor 1342 at the second reduced resolution are input into the neural network 1316. The intermediate encoding tensor at the second reduced resolution and the prediction residual tensor 1342 can be concatenated and input together. The prediction residual tensor 1342 at the second reduced resolution and... Figure 4 The predicted residual tensor is 427. Figure 5 The predicted residual tensor is 527 or Figure 6 The predicted residual tensor corresponds to 627.

[0226] The neural network 1316 outputs an intermediate encoding tensor at a third reduced resolution, obtained by reducing the second reduction resolution. The intermediate encoding tensor at the third reduced resolution and the prediction residual tensor 1341 at the third reduced resolution are input into the neural network 1318. The intermediate encoding tensor at the third reduced resolution and the prediction residual tensor 1341 can be concatenated and input together. The prediction residual tensor 1341 at the third reduced resolution and... Figure 15 The predicted residual tensor is 437. Figure 15 The predicted residual tensor is 537 or Figure 16 The predicted residual tensor corresponds to 637.

[0227] The residual image feature data 1330 is output through the neural network 1318. Each of the plurality of neural networks 1312, 1314, 1316 and 1318 within the multi-compensated pixel encoder 1310 may include at least one convolutional layer. In addition, the plurality of neural networks 1312, 1314, 1316 and 1318 can reduce the input data and output the reduced result.

[0228] In the multi-compensated pixel decoder 1320, the residual image feature data 1330 is output as a reconstructed image 1350 at the original resolution through multiple neural networks 1321, 1323, 1325 and 1327.

[0229] Each of the neural networks 1321, 1323, 1325, and 1327 within the multi-compensated pixel decoder 1320 may include at least one convolutional layer. Additionally, neural networks 1321, 1323, 1325, and 1327 may amplify the input data.

[0230] In addition to the residual image feature data 1330, the multi-compensated pixel decoder 1320 can also receive the feature data of the current optical flow. The residual image feature data 1330 and the feature data of the current optical flow can be concatenated and can be input.

[0231] Figure 17 This is a diagram illustrating the structure of a multi-compensated pixel encoder and a multi-compensated pixel decoder according to embodiments of the present disclosure.

[0232] Reference Figure 17 In the multi-compensated pixel encoder 1410, the original image 1400 is output as residual image feature data 1430 through multiple neural networks 1412, 1414, 1416 and 1418.

[0233] Each of the plurality of neural networks 1412, 1414, 1416, and 1418 within the multi-compensated pixel encoder 1410 may include at least one convolutional layer. Additionally, the plurality of neural networks 1412, 1414, 1416, and 1418 may reduce the input data and output a reduced result.

[0234] In the multi-compensated pixel decoder 1420, firstly, the intermediate prediction tensor 1431 obtained by reducing the third reduction resolution to a fourth reduction resolution, and the residual image feature data 1430, are input into the neural network 1421. The residual image feature data 1430 and the intermediate prediction tensor 1431 can be concatenated and can be input to each other. The intermediate prediction tensor 1431 at the fourth reduction resolution and... Figure 17 Intermediate prediction tensor 440 Figure 3 to Figure 14 The intermediate prediction tensor is 540 or Figure 18 The intermediate prediction tensor 640 corresponds to this.

[0235] The residual tensor at the third reduced resolution is obtained through neural network 1421. The residual tensor at the third reduced resolution and the intermediate prediction tensor at the third reduced resolution 1432 are input into neural network 1423. The residual tensor at the third reduced resolution and the intermediate prediction tensor 1432 can be concatenated and input together. The intermediate prediction tensor at the third reduced resolution and... Figure 19Intermediate prediction tensor 430 Figure 19 Intermediate prediction tensor 530 or Figure 19 The intermediate prediction tensor 630 corresponds to this.

[0236] The residual tensor at the second reduced resolution is obtained through neural network 1423. The residual tensor at the second reduced resolution and the intermediate prediction tensor at the second reduced resolution 1433 are input into neural network 1425. The residual tensor at the second reduced resolution and the intermediate prediction tensor 1433 can be concatenated and input together. The intermediate prediction tensor 1433 at the second reduced resolution and... Figure 3 to Figure 14 Intermediate prediction tensor 420 Figure 20 The intermediate prediction tensor is 520 or Figure 20 The intermediate prediction tensor 620 corresponds to this.

[0237] The residual tensor at the first reduced resolution is obtained through neural network 1425. The residual tensor at the first reduced resolution and the intermediate prediction tensor at the first reduced resolution 1434 are input into neural network 1427. The residual tensor at the first reduced resolution and the intermediate prediction tensor 1434 can be concatenated and input together. The intermediate prediction tensor 1434 at the first reduced resolution and... Figure 20 Intermediate prediction tensor 410 Figure 20 Intermediate prediction tensor 510 or ​ The intermediate prediction tensor 610 corresponds to this.

[0238] The neural network 1427 outputs a reconstructed image 1250 at the original resolution.

[0239] Each of the neural networks 1421, 1423, 1425, and 1427 within the multi-compensated pixel decoder 1420 may include at least one convolutional layer. Additionally, neural networks 1421, 1423, 1425, and 1427 may amplify the input data.

[0240] In addition to the residual image feature data 1430, the multi-compensated pixel decoder 1420 can also receive the feature data of the current optical flow. The residual image feature data 1430 and the feature data of the current optical flow can be concatenated and can be input.

[0241] ​ This is a view used to illustrate the optical flow and prediction error, memory threshold, and forget threshold of the original frame according to embodiments of the present disclosure.

[0242] Reference ​ A prediction frame for the original frame 1500 can be obtained based on the optical flow 1505 of the original frame 1500, and a prediction error 1510 may occur between the original frame 1500 and the prediction frame.

[0243] To prevent prediction error 1510, memory gates 1515, 1525, 1535 and 1545 and forget gates 1520, 1530, 1540 and 1550 are used.

[0244] In the region where prediction error 1510 occurs, the parts with good predictions need to be preserved, and the parts with bad predictions need to be removed. Therefore, the memory gates 1515, 1525, 1535, and 1545 have relatively large values ​​for the parts of the region with prediction errors that have information that needs to be preserved, and the forget gates 1520, 1530, 1540, and 1550 have relatively large values ​​for the parts of the region with prediction errors that need to be removed.

[0245] ​ This is a flowchart of an image encoding method according to an embodiment of the present disclosure.

[0246] In operation S1610, the image encoding device 1700 obtains feature data of the current optical flow by applying the current image and the previously reconstructed image to a first encoder based on a neural network.

[0247] In operation S1620, the image encoding device 1700 obtains the current optical flow and multiple memory gates corresponding to multiple resolutions of the current image by applying the feature data of the current optical flow to a first decoder based on a neural network.

[0248] According to embodiments of this disclosure, a plurality of memory gates may represent values ​​used to retain information within the current image.

[0249] In operation S1630, the image encoding device 1700 obtains a predicted image of the current image from a previously reconstructed image based on the current optical flow.

[0250] In operation S1640, the image coding device 1700 obtains multiple prediction tensors corresponding to multiple resolutions based on the predicted image and multiple memory gates.

[0251] According to embodiments of this disclosure, multiple forgetting gates corresponding to multiple resolutions of the current image can be additionally obtained from a first decoder based on a neural network. Multiple prediction tensors can be obtained based on the prediction image, multiple memory gates, and forgetting gates. The multiple forgetting gates can represent values ​​used to remove regions from the current image whose prediction errors are equal to or greater than preset values.

[0252] According to embodiments of this disclosure, a prediction tensor of the original resolution of the current image can be determined from a plurality of prediction tensors based on the predicted image and a memory gate value corresponding to the original resolution.

[0253] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a subtraction tensor obtained by subtracting the original resolution of the current image from the prediction image and a forgetting gate value corresponding to the original resolution. Furthermore, a prediction tensor at a reduced resolution from the original resolution can be determined among a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0254] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a memory gate corresponding to the original resolution of the current image, a forget gate corresponding to the original resolution, and a prediction image. Furthermore, a prediction tensor at a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0255] According to embodiments of this disclosure, a prediction residual tensor can be obtained by subtracting a prediction tensor of the original resolution of the current image from the prediction image. A prediction tensor of a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor of the reduced resolution and a memory gate corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate corresponding to the original resolution.

[0256] According to embodiments of this disclosure, a prediction residual tensor at a first reduced resolution can be obtained based on a subtraction tensor and a forgetting gate corresponding to a first reduced resolution. The subtraction tensor is obtained by subtracting the prediction tensor at the first reduced resolution from an intermediate prediction tensor at the first reduced resolution before applying the forgetting gate corresponding to the first reduced resolution to the prediction tensor at the first reduced resolution among a plurality of prediction tensors. Furthermore, a prediction tensor at a second reduced resolution obtained by reducing the first reduced resolution can be determined based on an intermediate prediction tensor at a second reduced resolution and a forgetting gate corresponding to the second reduced resolution. The intermediate prediction tensor at the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0257] According to embodiments of this disclosure, a prediction residual tensor of a first reduced resolution can be obtained based on a memory gate corresponding to a first reduced resolution, a forget gate corresponding to a first reduced resolution, and an intermediate prediction tensor of a first reduced resolution before applying the memory gate corresponding to the first reduced resolution to a plurality of prediction tensors. Furthermore, a prediction tensor of a second reduced resolution obtained by reducing the first reduced resolution can be determined based on the intermediate prediction tensor of a second reduced resolution and the memory gate corresponding to the second reduced resolution. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0258] According to embodiments of this disclosure, a prediction residual tensor of the first reduced resolution can be obtained by subtracting the prediction tensor of the first reduced resolution from an intermediate prediction tensor of the first reduced resolution before applying a memory gate corresponding to the first reduced resolution to the prediction tensor of the first reduced resolution among a plurality of prediction tensors. A prediction tensor of the second reduced resolution obtained by reducing the first reduced resolution can be determined based on the intermediate prediction tensor of the second reduced resolution and the memory gate corresponding to the second reduced resolution among a plurality of prediction tensors. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate corresponding to the first reduced resolution.

[0259] In operation S1650, the image encoding device 1700 obtains feature data of the residual image by applying multiple prediction tensors and the current image to a neural network-based second encoder.

[0260] In operation S1660, the image encoding device 1700 obtains the current reconstructed image corresponding to the current image by applying multiple prediction tensors, feature data of the current optical flow, and feature data of the residual image to a second decoder based on a neural network.

[0261] According to embodiments of this disclosure, the predictive residual tensor can be further applied to a second neural network-based encoder to obtain feature data of the residual image, and the intermediate predictive tensor can be further applied to a second neural network-based decoder to obtain the current reconstructed image.

[0262] In operation S1670, the image encoding device 1700 generates a bitstream that includes feature data of the current optical flow and feature data of the residual image.

[0263] ​ This is a block diagram of the structure of an image encoding device according to an embodiment of the present disclosure.

[0264] Reference ​The image encoding device 1700 may include a predictive encoder 1710, a generator 1720, an acquirer 1730, and a predictive decoder 1740.

[0265] The predictive encoder 1710, generator 1720, acquirer 1730, and predictive decoder 1740 may be implemented as processors. A processor may include at least one processing circuit and / or multiple processors. For example, the term "processor" as used herein and in the claims may include various processing circuits, including at least one processor. One or more of the at least one processor may be configured to perform the various functions described herein individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform several functions. However, these terms cover, but are not limited to, situations where one processor performs some functions while other processors(one or more) perform other functions, and situations where a single processor is capable of performing all functions. Additionally, at least one processor may include a combination of processors performing various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to complete or perform various functions. The predictive encoder 1710, generator 1720, acquirer 1730, and predictive decoder 1740 may operate according to instructions stored in memory (not shown).

[0266] Despite ​ The predictive encoder 1710, generator 1720, acquirer 1730, and predictive decoder 1740 are shown separately, but they can be implemented as a single processor. In this case, the predictive encoder 1710, generator 1720, acquirer 1730, and predictive decoder 1740 can be implemented as a dedicated processor, or as a combination of software and a general-purpose processor such as an application processor (AP), central processing unit (CPU), or graphics processing unit (GPU). The dedicated processor may include memory for implementing embodiments of this disclosure or a memory processing unit for using external memory.

[0267] The predictive encoder 1710, generator 1720, acquirer 1730, and predictive decoder 1740 can be implemented as multiple processors. In this case, the predictive encoder 1710, generator 1720, acquirer 1730, and predictive decoder 1740 can be implemented as a combination of dedicated processors, or they can be implemented by a combination of software and multiple general-purpose processors (such as AP, CPU, or GPU).

[0268] The predictive encoder 1710 may include a motion encoder 1711 and a multi-compensated pixel encoder 1712.

[0269] The motion encoder 1711 and the multi-compensated pixel encoder 1712 can be implemented as a neural network including one or more layers (e.g., one or more convolutional layers).

[0270] The motion encoder 1711 and the multi-compensated pixel encoder 1712 can be stored in memory. The motion encoder 1711 and the multi-compensated pixel encoder 1712 can be implemented as at least one dedicated processor for AI.

[0271] The predictive encoder 1710 can obtain feature data of the current optical flow using the current image and the previously reconstructed image, and can obtain residual image feature data using the current image and multiple prediction tensors received from the depth predictive decomposer 1743 of the predictive decoder 1740. Specifically, the motion encoder 1711 can receive the current image and the previously reconstructed image, and thus output the feature data of the current optical flow. Additionally, the multi-compensated pixel encoder 1712 can receive multiple prediction tensors from the depth predictive decomposer 1743 of the predictive decoder 1740 and the current image, and can output residual image feature data.

[0272] Both the current optical flow feature data and the residual image feature data obtained by the predictive encoder 1710 can be sent to the generator 1720.

[0273] Generator 1720 can generate a bitstream that includes feature data of the current optical flow and feature data of the residual image.

[0274] According to an embodiment, generator 1720 can generate a first bitstream corresponding to the feature data of the current optical flow and a second bitstream corresponding to the feature data of the residual image.

[0275] The bitstream can be sent to the image decoding device 1900 via a network. According to an embodiment, the bitstream can be stored in a data storage medium including magnetic media (such as hard disks, floppy disks, or magnetic tapes), optical recording media (such as CD-ROMs or DVDs), or magneto-optical media (such as optical floppy disks).

[0276] The acquirer 1730 can obtain the feature data of the current optical flow and the feature data of the residual image from the bit stream generated by the generator 1720.

[0277] According to an embodiment, the acquirer 1730 can receive feature data of the current optical flow and feature data of the residual image from the predictive encoder 1710.

[0278] The current optical flow feature data and residual image feature data can be sent to the prediction decoder 1740.

[0279] The prediction decoder 1740 may include a motion decoder 1741, a motion compensator 1742, a depth prediction decomposer 1743, and a multi-compensated pixel decoder 1744.

[0280] The motion decoder 1741, the depth prediction decomposer 1743, and the multi-compensated pixel decoder 1744 can be implemented as a neural network comprising one or more layers (e.g., one or more convolutional layers).

[0281] The motion decoder 1741, depth prediction decomposer 1743, and multi-compensated pixel decoder 1744 can be stored in memory. The motion decoder 1741, depth prediction decomposer 1743, and multi-compensated pixel decoder 1744 can be implemented as at least one dedicated processor for AI.

[0282] Predictive decoder 1740 obtains the current reconstructed image using feature data of the current optical flow and feature data of the residual image. Specifically, motion decoder 1741 receives feature data of the current optical flow and outputs the current optical flow along with memory and forget gates corresponding to multiple resolutions. The current optical flow can be sent to motion compensator 1742, and the memory and forget gates corresponding to multiple resolutions can be sent to depth predictive decomposer 1743. Motion compensator 1742 obtains a predicted image by performing warping using the previously reconstructed image and the current optical flow. The predicted image is sent to depth predictive decomposer 1743. Depth predictive decomposer 1743 obtains multiple prediction tensors using the predicted image and memory and forget gates corresponding to multiple resolutions. The multiple prediction tensors can be passed to multi-compensated pixel encoder 1712 of predictive encoder 1710 and multi-compensated pixel decoder 1744 of predictive decoder 1740. Multi-compensated pixel decoder 1744 obtains the current reconstructed image using feature data of the current optical flow, feature data of the residual image, and the multiple prediction tensors.

[0283] The detailed operation of the motion encoder 1711, multi-compensated pixel encoder 1712, motion decoder 1741, depth prediction decomposer 1743, and multi-compensated pixel decoder 1744 has been referenced above. ​ They are described, but omitted.

[0284] ​ This is a flowchart of an image decoding method according to an embodiment of the present disclosure.

[0285] In operation S1810, the image decoding device 1900 obtains feature data of the current optical flow and feature data of the residual image of the current image from the bit stream.

[0286] In operation S1820, the image decoding device 1900 obtains the current optical flow and multiple memory gates corresponding to multiple resolutions of the current image by applying the feature data of the current optical flow to a first decoder based on a neural network.

[0287] According to embodiments of this disclosure, a plurality of memory gates may represent values ​​used to retain information within the current image.

[0288] In operation S1830, the image decoding device 1900 obtains a predicted image of the current image from the previously reconstructed image based on the current optical flow.

[0289] In operation S1840, the image decoding device 1900 obtains multiple prediction tensors corresponding to multiple resolutions based on the predicted image and multiple memory gates.

[0290] According to embodiments of this disclosure, multiple forgetting gates corresponding to multiple resolutions of the current image can be additionally obtained from a first decoder based on a neural network. Multiple prediction tensors can be obtained based on the predicted image, multiple memory gates, and multiple forgetting gates. The multiple forgetting gates can represent values ​​used to remove regions from the current image whose prediction errors are equal to or greater than preset values.

[0291] According to embodiments of this disclosure, a prediction tensor of the original resolution of the current image can be determined from a plurality of prediction tensors based on the predicted image and a memory gate value corresponding to the original resolution.

[0292] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a subtraction tensor obtained by subtracting the original resolution of the current image from the prediction image and a forgetting gate value corresponding to the original resolution. Furthermore, a prediction tensor at a reduced resolution from the original resolution can be determined among a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0293] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a memory gate corresponding to the original resolution of the current image, a forget gate corresponding to the original resolution, and a prediction image. Furthermore, a prediction tensor at a reduced resolution can be determined from among a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0294] According to embodiments of this disclosure, a prediction residual tensor can be obtained by subtracting a prediction tensor of the original resolution of the current image from the prediction image. A prediction tensor of a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor of the reduced resolution and a memory gate corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate corresponding to the original resolution.

[0295] According to embodiments of this disclosure, a prediction residual tensor at a first reduced resolution can be obtained based on a subtraction tensor and a forgetting gate corresponding to a first reduced resolution. The subtraction tensor is obtained by subtracting the prediction tensor at the first reduced resolution from an intermediate prediction tensor at the first reduced resolution before applying the forgetting gate corresponding to the first reduced resolution to the prediction tensor at the first reduced resolution among a plurality of prediction tensors. Furthermore, a prediction tensor at a second reduced resolution obtained by reducing the first reduced resolution can be determined based on an intermediate prediction tensor at a second reduced resolution and a forgetting gate corresponding to the second reduced resolution. The intermediate prediction tensor at the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0296] According to embodiments of this disclosure, a prediction residual tensor of a first reduced resolution can be obtained based on a memory gate corresponding to a first reduced resolution, a forget gate corresponding to a first reduced resolution, and an intermediate prediction tensor of a first reduced resolution before applying the memory gate corresponding to the first reduced resolution to a plurality of prediction tensors. Furthermore, a prediction tensor of a second reduced resolution obtained by reducing the first reduced resolution can be determined based on the intermediate prediction tensor of a second reduced resolution and the memory gate corresponding to the second reduced resolution. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0297] According to embodiments of this disclosure, a prediction residual tensor of the first reduced resolution can be obtained by subtracting the prediction tensor of the first reduced resolution from an intermediate prediction tensor of the first reduced resolution before applying a memory gate corresponding to the first reduced resolution to the prediction tensor of the first reduced resolution among a plurality of prediction tensors. A prediction tensor of the second reduced resolution obtained by reducing the first reduced resolution can be determined based on the intermediate prediction tensor of the second reduced resolution and the memory gate corresponding to the second reduced resolution among a plurality of prediction tensors. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate corresponding to the first reduced resolution.

[0298] In operation S1850, the image decoding device 1900 obtains the current reconstructed image corresponding to the current image by applying multiple prediction tensors, feature data of the current optical flow, and feature data of the residual image to a second decoder based on a neural network.

[0299] According to embodiments of this disclosure, intermediate prediction tensors can be additionally applied to a second neural network-based decoder to obtain the current reconstructed image.

[0300] ​ This is a block diagram of the structure of an image decoding device according to an embodiment of the present disclosure.

[0301] Reference ​ The image decoding device 1900 may include an acquirer 1910 and a predictive decoder 1920.

[0302] The acquirer 1910 and the predictive decoder 1920 may be implemented as processors. A processor may include at least one processing circuit and / or multiple processors. For example, the term "processor" as used herein and in the claims may include various processing circuits, including at least one processor. One or more of the at least one processor may be configured to perform the various functions described herein individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform several functions. However, these terms cover, but are not limited to, situations where one processor performs some functions while other processors(one or more) perform other functions, and situations where a single processor is capable of performing all functions. Furthermore, at least one processor may include a combination of processors performing various functions in a distributed manner. At least one processor may execute program instructions to complete or perform various functions. The acquirer 1910 and the predictive decoder 1920 may operate according to instructions stored in memory (not shown).

[0303] Despite ​ Acquirer 1910 and predictive decoder 1920 are shown separately, but acquirer 1910 and predictive decoder 1920 can be implemented by a single processor. In this case, acquirer 1910 and predictive decoder 1920 can be implemented as a dedicated processor, or by a combination of software and a general-purpose processor (such as an application processor (AP), central processing unit (CPU), or graphics processing unit (GPU)). The dedicated processor may include memory for implementing embodiments of this disclosure or a memory processing unit for using external memory.

[0304] The acquirer 1910 and the predictive decoder 1920 can be configured with multiple processors. In this case, the acquirer 1910 and the predictive decoder 1920 can be implemented as a combination of dedicated processors, or they can be implemented by a combination of software and multiple general-purpose processors, such as an AP, CPU, or GPU.

[0305] The acquirer 1910 can acquire the feature data of the current optical flow and the feature data of the residual image from the bit stream.

[0306] The current optical flow feature data and residual image feature data can be sent to the prediction decoder 1920.

[0307] The prediction decoder 1920 may include a motion decoder 1921, a motion compensator 1922, a depth prediction decomposer 1923, and a multi-compensated pixel decoder 1924.

[0308] The motion decoder 1921, the depth prediction decomposer 1923, and the multi-compensated pixel decoder 1924 can be implemented as a neural network comprising one or more layers (e.g., one or more convolutional layers).

[0309] The motion decoder 1921, depth prediction decomposer 1923, and multi-compensated pixel decoder 1924 can be stored in memory. The motion decoder 1921, depth prediction decomposer 1923, and multi-compensated pixel decoder 1924 can be implemented as at least one dedicated processor for AI.

[0310] The predictive decoder 1920 obtains the current reconstructed image using feature data of the current optical flow and feature data of the residual image. Specifically, the motion decoder 1921 receives feature data of the current optical flow and outputs the current optical flow along with memory and forget gates corresponding to multiple resolutions. The current optical flow can be sent to the motion compensator 1922, and the memory and forget gates corresponding to multiple resolutions can be sent to the depth predictive decomposer 1743. The motion compensator 1922 obtains the predicted image by performing warping using the previously reconstructed image and the current optical flow. The predicted image can be sent to the depth predictive decomposer 1923. The depth predictive decomposer 1923 obtains multiple prediction tensors using the predicted image and memory and forget gates corresponding to multiple resolutions. The multi-compensated pixel decoder 1924 obtains the current reconstructed image using feature data of the current optical flow, feature data of the residual image, and multiple prediction tensors.

[0311] The detailed operation of the motion decoder 1921, depth prediction decomposer 1923, and multi-compensated pixel decoder 1924 has been referenced above. ​ They are described, but omitted.

[0312] ​This is a diagram used to illustrate methods for training neural networks for motion encoders, motion decoders, depth predictive decomposition, multi-compensated pixel encoders, and multi-compensated pixel decoders.

[0313] exist ​ In this context, the current training image 2000, the previously reconstructed training image 2005, and the reconstructed training image 2070 correspond to the current image, the previously reconstructed image, and the reconstructed image, respectively.

[0314] When training the neural networks of motion encoder 2010, motion decoder 2020, depth predictive decomposition 2040, multi-compensated pixel encoder 2050 and multi-compensated pixel decoder 2060, it is necessary to consider the similarity between the reconstructed training image 2070 and the current training image 2000, as well as the bit rate of the bitstream that will be generated by encoding the current training image 2000.

[0315] Therefore, according to the embodiment, the neural networks of motion encoder 2010, motion decoder 2020, depth prediction decomposition 2040, multi-compensated pixel encoder 2050 and multi-compensated pixel decoder 2060 can be trained based on first loss information 20800 and second loss information 2085 corresponding to the size of the bitstream and third loss information 2090 corresponding to the similarity between the current training image 2000 and the reconstructed training image 2070.

[0316] Reference ​ The current training image 2000 and the previously reconstructed training image 2005 can be input into the motion encoder 2010. The optical flow encoder 2010 can output the current optical flow feature data 2011 by processing the current training image 2000 and the previously reconstructed training image 2005.

[0317] The feature data 2011 of the current optical flow can be input to the motion decoder 2020, and the motion decoder 2020 can output the current optical flow 2021 and the memory gate and forget gate 2022 corresponding to multiple resolutions by processing the feature data 2011 of the current optical flow.

[0318] The previously reconstructed training image 2005 can be warped via warp 2030 according to the current optical flow 2021 to generate the current predicted training image 2031.

[0319] The current prediction training image 2031, along with memory and forget gates 2022 corresponding to multiple resolutions, can be used during the depth prediction decomposition 2040. Multiple prediction training tensors 2041 corresponding to multiple resolutions can be output through the depth prediction decomposition 2040.

[0320] The current training image 2420 and multiple prediction training tensors 2041 can be input into the multi-compensated pixel encoder 2050, and the residual image feature data 2051 can be output by the multi-compensated pixel encoder 2050.

[0321] The current optical flow feature data 2011, residual image feature data 2051, and multiple prediction training tensors 2041 can be input into the multi-compensation pixel decoder 2060, and the reconstructed training image 2070 can be output by the multi-compensation pixel decoder 2060.

[0322] To train the neural networks of motion encoder 2010, motion decoder 2020, depth prediction decomposition 2040, multi-compensated pixel encoder 2050, and multi-compensated pixel decoder 2060, at least one of first loss information 2080, second loss information 2085, or third loss information 2090 can be obtained.

[0323] The first loss information 2080 can be calculated from the entropy of the feature data 2011 of the current optical flow or the bit rate of the bit stream corresponding to the feature data 2011 of the current optical flow.

[0324] The second loss information 2085 can be calculated from the entropy of the residual image feature data 2051 or the bit rate of the bit stream corresponding to the residual image feature data 2051.

[0325] Since the first loss information 2080 and the second loss information 2085 are related to the efficiency of encoding the current training image 2000, the first loss information 2080 and the second loss information 2085 can be referred to as compressed loss information.

[0326] According to the embodiments, although in ​ The first loss information 2080 and the second loss information 2085 related to the bit rate of the bit stream are derived, but a loss information corresponding to the bit rate of a bit stream generated by encoding the current training image 2000 can be derived.

[0327] The third loss information 2090 can correspond to the difference between the current training image 2000 and the reconstructed training image 2070. That is, the third loss information 2090 can be obtained by comparing the current training image 2000 with the reconstructed training image 2070. The difference between the current training image 2000 and the reconstructed training image 2070 can include at least one of the following: L1 norm value, L2 norm value, structural similarity (SSIM) value, peak signal-to-noise ratio-human visual system (PSNR-HVS) value, multi-scale SSIM (MS-SSIM) value, variance inflation factor (VIF) value, or video multi-method evaluation fusion (VMAF) value.

[0328] Because the third loss information 2090 is related to the quality of the reconstructed training image 2070, the third loss information 2090 can be called quality loss information.

[0329] The neural networks of the motion encoder 2010, motion decoder 2020, depth prediction decomposition 2040, multi-compensated pixel encoder 2050 and multi-compensated pixel decoder 2060 can be trained such that the final loss information derived from at least one of the first loss information 2080, the second loss information 2085 and the third loss information 2090 can be reduced or minimized.

[0330] In detail, the neural network of the trainable motion encoder 2010, motion decoder 2020, depth predictive decomposition 2040, multi-compensated pixel encoder 2050 and multi-compensated pixel decoder 2060 can reduce or minimize the final loss information while the values ​​of preset parameters are changed.

[0331] According to embodiments of this disclosure, the final loss information can be calculated according to Equation 1 below.

[0332] [Equation 1] Final loss information = a First loss information + b Second loss information + c Third loss information In Equation 1, a, b, and c represent the weights applied to the first loss information 2080, the second loss information 2085, and the third loss information 2090, respectively.

[0333] According to Equation 1, it is found that the neural networks of motion encoder 2010, motion decoder 2020, depth prediction decomposition 2040, multi-compensated pixel encoder 2050 and multi-compensated pixel decoder 2060 can be trained such that the reconstructed training image 2070 is as similar as possible to the current training image 2000, and the size of the bitstream corresponding to the data output by motion encoder 2010 and multi-compensated pixel encoder 2050 is minimized.

[0334] The image decoding method according to the embodiment may include: obtaining feature data of the current optical flow and feature data of the residual image of the current image from a bitstream; obtaining the current optical flow and a plurality of memory gates corresponding to a plurality of resolutions of the current image by applying the feature data of the current optical flow to a first decoder based on a neural network; obtaining a predicted image of the current image from a previously reconstructed image based on the current optical flow; obtaining a plurality of prediction tensors corresponding to a plurality of resolutions based on the predicted image and the plurality of memory gates; and obtaining a current reconstructed image corresponding to the current image by applying the plurality of prediction tensors, the feature data of the current optical flow, and the feature data of the residual image to a second decoder based on a neural network.

[0335] According to embodiments of this disclosure, a plurality of memory gates may represent values ​​used to retain information within the current image.

[0336] In the image decoding method according to embodiments of the present disclosure, an image can be reconstructed based on multiple prediction tensors corresponding to multiple resolutions obtained using multiple memory gate values ​​representing values ​​used to preserve the main information in the image, thereby preserving the main information of the image. Therefore, errors can be prevented in the reconstructed image, and the accuracy and coding efficiency of the image can be improved.

[0337] According to embodiments of this disclosure, multiple forgetting gates corresponding to multiple resolutions of the current image can be obtained from a first decoder based on a neural network. Multiple prediction tensors can be obtained based on a first predicted image, multiple memory gates, and multiple forgetting gates. The multiple forgetting gates can represent values ​​used to remove regions from the current image whose prediction errors are equal to or greater than preset values.

[0338] In the image decoding method according to embodiments of the present disclosure, multiple prediction tensors can be obtained by additionally using values ​​for removing regions from the image whose prediction errors are equal to or greater than a preset value (i.e., poorly predicted regions), and the image can be reconstructed based on the multiple prediction tensors corresponding to multiple resolutions, thereby removing unnecessary information from the image. Therefore, errors in the reconstructed image can be prevented, and the accuracy and coding efficiency of the image can be improved.

[0339] According to embodiments of this disclosure, a prediction tensor of the original resolution of the current image can be determined from a plurality of prediction tensors based on the predicted image and a memory gate value corresponding to the original resolution.

[0340] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a subtraction tensor obtained by subtracting the original resolution of the current image from the prediction image and a forgetting gate value corresponding to the original resolution. Furthermore, a prediction tensor at a reduced resolution from the original resolution can be determined among a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0341] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a memory gate corresponding to the original resolution of the current image, a forget gate corresponding to the original resolution, and a prediction image. Furthermore, a prediction tensor at a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0342] According to embodiments of this disclosure, a prediction residual tensor can be obtained by subtracting a prediction tensor of the original resolution of the current image from the prediction image. A prediction tensor of a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor of the reduced resolution and a memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate value corresponding to the original resolution.

[0343] According to embodiments of this disclosure, a prediction residual tensor at a first reduced resolution can be obtained based on a subtraction tensor and a forgetting gate corresponding to a first reduced resolution. The subtraction tensor is obtained by subtracting the prediction tensor at the first reduced resolution from an intermediate prediction tensor at the first reduced resolution before applying the forgetting gate corresponding to the first reduced resolution to the prediction tensor at the first reduced resolution among a plurality of prediction tensors. Furthermore, a prediction tensor at a second reduced resolution obtained by reducing the first reduced resolution can be determined based on an intermediate prediction tensor at a second reduced resolution and a forgetting gate corresponding to the second reduced resolution among a plurality of prediction tensors. The intermediate prediction tensor at the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0344] According to embodiments of this disclosure, a prediction residual tensor of a first reduced resolution can be obtained based on a memory gate corresponding to a first reduced resolution, a forget gate corresponding to a first reduced resolution, and an intermediate prediction tensor of a first reduced resolution before applying the memory gate corresponding to the first reduced resolution to a plurality of prediction tensors. Furthermore, a prediction tensor of a second reduced resolution obtained by reducing the first reduced resolution can be determined based on the intermediate prediction tensor of a second reduced resolution and the memory gate corresponding to the second reduced resolution. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0345] According to embodiments of this disclosure, a prediction residual tensor of the first reduced resolution can be obtained by subtracting the prediction tensor of the first reduced resolution from an intermediate prediction tensor of the first reduced resolution before applying a memory gate corresponding to the first reduced resolution to the prediction tensor of the first reduced resolution among a plurality of prediction tensors. A prediction tensor of the second reduced resolution obtained by reducing the first reduced resolution among a plurality of prediction tensors can be determined based on an intermediate prediction tensor of the second reduced resolution and a memory gate corresponding to the second reduced resolution. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate corresponding to the first reduced resolution.

[0346] In the image decoding method according to embodiments of this disclosure, the predicted image of the current image can be decomposed into prediction tensors of multiple resolutions using memory and forget gates corresponding to multiple resolutions, and the image can be reconstructed based on the multiple prediction tensors corresponding to multiple resolutions. This preserves the main information of the image and removes unnecessary information. Therefore, errors can be prevented, and the accuracy and coding efficiency of the image can be improved.

[0347] According to embodiments of this disclosure, intermediate prediction tensors can be additionally applied to a second neural network-based decoder to obtain the current reconstructed image.

[0348] In the image decoding method according to embodiments of the present disclosure, additional information can be used in residual coding, thereby improving image accuracy and coding efficiency.

[0349] The image decoding apparatus according to an embodiment may include: a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may be configured to obtain feature data of a current optical flow and feature data of a residual image of a current image from a bitstream; obtain a current optical flow and a plurality of memory gates corresponding to a plurality of resolutions of the current image by applying the feature data of the current optical flow to a first neural network-based decoder; obtain a predicted image of the current image from a previously reconstructed image based on the current optical flow; obtain a plurality of prediction tensors corresponding to a plurality of resolutions based on the predicted image and the plurality of memory gates; and obtain a current reconstructed image corresponding to the current image by applying the plurality of prediction tensors, the feature data of the current optical flow, and the feature data of the residual image to a second neural network-based decoder.

[0350] According to embodiments of this disclosure, a plurality of memory gates may represent values ​​used to retain information within the current image.

[0351] In the image decoding apparatus according to embodiments of the present disclosure, an image can be reconstructed based on multiple prediction tensors corresponding to multiple resolutions obtained using multiple memory gate values ​​representing values ​​used to preserve the main information in the image, thereby preserving the main information of the image. Therefore, errors can be prevented in the reconstructed image, and the accuracy and coding efficiency of the image can be improved.

[0352] According to embodiments of this disclosure, multiple forgetting gates corresponding to multiple resolutions of the current image can be obtained from a first decoder based on a neural network. Multiple prediction tensors can be obtained based on the prediction image, multiple memory gates, and multiple forgetting gates. The multiple forgetting gates can represent values ​​used to remove regions from the current image whose prediction errors are equal to or greater than preset values.

[0353] In the image decoding apparatus according to embodiments of the present disclosure, multiple prediction tensors can be obtained by additionally using values ​​for removing regions from the image whose prediction errors are equal to or greater than a preset value, and the image can be reconstructed based on the multiple prediction tensors corresponding to multiple resolutions, thereby removing unnecessary information from the image. Therefore, errors can be prevented in the reconstructed image, and the accuracy and coding efficiency of the image can be improved.

[0354] According to embodiments of this disclosure, a prediction tensor of the original resolution of the current image can be determined from a plurality of prediction tensors based on the predicted image and a memory gate value corresponding to the original resolution.

[0355] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a subtraction tensor obtained by subtracting the original resolution of the current image from the prediction image and a forgetting gate value corresponding to the original resolution. Furthermore, a prediction tensor at a reduced resolution from the original resolution can be determined among a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0356] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a memory gate corresponding to the original resolution of the current image, a forget gate corresponding to the original resolution, and a prediction image. Furthermore, a prediction tensor at a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0357] According to embodiments of this disclosure, a prediction residual tensor can be obtained by subtracting a prediction tensor of the original resolution of the current image from the prediction image. A prediction tensor of a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor of the reduced resolution and a memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate value corresponding to the original resolution.

[0358] According to embodiments of this disclosure, a prediction residual tensor at a first reduced resolution can be obtained based on a subtraction tensor and a forgetting gate corresponding to a first reduced resolution. The subtraction tensor is obtained by subtracting the prediction tensor at the first reduced resolution from an intermediate prediction tensor at the first reduced resolution before applying the forgetting gate corresponding to the first reduced resolution to the prediction tensor at the first reduced resolution among a plurality of prediction tensors. Furthermore, a prediction tensor at a second reduced resolution obtained by reducing the first reduced resolution can be determined based on an intermediate prediction tensor at a second reduced resolution and a forgetting gate corresponding to the second reduced resolution among a plurality of prediction tensors. The intermediate prediction tensor at the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0359] According to embodiments of this disclosure, a prediction residual tensor of a first reduced resolution can be obtained based on a memory gate corresponding to a first reduced resolution, a forget gate corresponding to a first reduced resolution, and an intermediate prediction tensor of a first reduced resolution before applying the memory gate corresponding to the first reduced resolution to a plurality of prediction tensors. Furthermore, a prediction tensor of a second reduced resolution obtained by reducing the first reduced resolution can be determined based on the intermediate prediction tensor of a second reduced resolution and the memory gate corresponding to the second reduced resolution. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0360] According to embodiments of this disclosure, a prediction residual tensor of the first reduced resolution can be obtained by subtracting the prediction tensor of the first reduced resolution from an intermediate prediction tensor of the first reduced resolution before applying a memory gate corresponding to the first reduced resolution to the prediction tensor of the first reduced resolution among a plurality of prediction tensors. A prediction tensor of the second reduced resolution obtained by reducing the first reduced resolution among a plurality of prediction tensors can be determined based on an intermediate prediction tensor of the second reduced resolution and a memory gate corresponding to the second reduced resolution. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate corresponding to the first reduced resolution.

[0361] In the image decoding apparatus according to embodiments of the present disclosure, the predicted image of the current image can be decomposed into prediction tensors of multiple resolutions by using memory gates and forget gates corresponding to multiple resolutions, and the image can be reconstructed based on the multiple prediction tensors corresponding to multiple resolutions, thereby preserving the main information of the image and removing unnecessary information. Therefore, errors can be prevented, and the accuracy of the image and coding efficiency can be improved.

[0362] According to embodiments of this disclosure, intermediate prediction tensors can be additionally applied to a second neural network-based decoder to obtain the current reconstructed image.

[0363] In the image decoding apparatus according to embodiments of the present disclosure, additional information can be used in residual coding, thereby improving image accuracy and coding efficiency.

[0364] The image encoding method according to an embodiment may include: obtaining feature data of a current optical flow by applying a current image and a previously reconstructed image to a first encoder based on a neural network; obtaining a current optical flow and a plurality of memory gates corresponding to a plurality of resolutions of the current image by applying the feature data of the current optical flow to a first decoder based on a neural network; obtaining a predicted image of the current image from a previously reconstructed image based on the current optical flow; obtaining a plurality of prediction tensors corresponding to a plurality of resolutions based on the predicted image and the plurality of memory gates; obtaining feature data of a residual image by applying the plurality of prediction tensors and the current image to a second encoder based on a neural network; obtaining a current reconstructed image corresponding to the current image by applying the plurality of prediction tensors, the feature data of the current optical flow, and the feature data of the residual image to a second decoder based on a neural network; and generating a bitstream including the feature data of the current optical flow and the feature data of the residual image.

[0365] According to embodiments of this disclosure, a plurality of memory gates may represent values ​​used to retain information within the current image.

[0366] In the image coding method according to embodiments of the present disclosure, an image can be reconstructed based on multiple prediction tensors corresponding to multiple resolutions obtained using multiple memory gate values ​​representing values ​​used to preserve the main information in the image, thereby preserving the main information of the image. Therefore, errors can be prevented in the reconstructed image, and the accuracy and coding efficiency of the image can be improved.

[0367] According to embodiments of this disclosure, multiple forgetting gates corresponding to multiple resolutions of the current image can be additionally obtained from a first decoder based on a neural network. Multiple prediction tensors can be obtained based on the predicted image, multiple memory gates, and multiple forgetting gates. The multiple forgetting gates can represent values ​​used to remove regions from the current image whose prediction errors are equal to or greater than preset values.

[0368] In the image encoding method according to embodiments of the present disclosure, multiple prediction tensors can be obtained by additionally using values ​​for removing regions from the image whose prediction errors are equal to or greater than a preset value, and the image can be reconstructed based on the multiple prediction tensors corresponding to multiple resolutions, thereby removing unnecessary information from the image. Therefore, errors can be prevented in the reconstructed image, and the accuracy and encoding efficiency of the image can be improved.

[0369] According to embodiments of this disclosure, a prediction tensor of the original resolution of the current image can be determined from a plurality of prediction tensors based on the predicted image and a memory gate value corresponding to the original resolution.

[0370] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a subtraction tensor obtained by subtracting the original resolution of the current image from the prediction image and a forgetting gate value corresponding to the original resolution. Furthermore, a prediction tensor at a reduced resolution from the original resolution can be determined among a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0371] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a memory gate corresponding to the original resolution of the current image, a forget gate corresponding to the original resolution, and a prediction image. Furthermore, a prediction tensor at a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0372] According to embodiments of this disclosure, a prediction residual tensor can be obtained by subtracting a prediction tensor of the original resolution of the current image from the prediction image. A prediction tensor of a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor of the reduced resolution and a memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate value corresponding to the original resolution.

[0373] According to embodiments of this disclosure, a prediction residual tensor at a first reduced resolution can be obtained based on a subtraction tensor and a forgetting gate corresponding to a first reduced resolution. The subtraction tensor is obtained by subtracting the prediction tensor at the first reduced resolution from an intermediate prediction tensor at the first reduced resolution before applying the forgetting gate corresponding to the first reduced resolution to the prediction tensor at the first reduced resolution among a plurality of prediction tensors. Furthermore, a prediction tensor at a second reduced resolution obtained by reducing the first reduced resolution can be determined based on an intermediate prediction tensor at a second reduced resolution and a forgetting gate corresponding to the second reduced resolution. The intermediate prediction tensor at the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0374] According to embodiments of this disclosure, a prediction residual tensor of a first reduced resolution can be obtained based on a memory gate corresponding to a first reduced resolution, a forget gate corresponding to a first reduced resolution, and an intermediate prediction tensor of a first reduced resolution before applying the memory gate corresponding to the first reduced resolution to a plurality of prediction tensors. Furthermore, a prediction tensor of a second reduced resolution obtained by reducing the first reduced resolution can be determined based on the intermediate prediction tensor of a second reduced resolution and the memory gate corresponding to the second reduced resolution. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0375] According to embodiments of this disclosure, a prediction residual tensor of the first reduced resolution can be obtained by subtracting the prediction tensor of the first reduced resolution from an intermediate prediction tensor of the first reduced resolution before applying a memory gate corresponding to the first reduced resolution to the prediction tensor of the first reduced resolution among a plurality of prediction tensors. A prediction tensor of the second reduced resolution obtained by reducing the first reduced resolution among a plurality of prediction tensors can be determined based on an intermediate prediction tensor of the second reduced resolution and a memory gate corresponding to the second reduced resolution. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate corresponding to the first reduced resolution.

[0376] In the image coding method according to embodiments of this disclosure, the predicted image of the current image can be decomposed into prediction tensors of multiple resolutions using memory gates and forget gates corresponding to multiple resolutions, and the image can be reconstructed based on the multiple prediction tensors corresponding to multiple resolutions. This preserves the main information of the image and removes unnecessary information. Therefore, errors can be prevented, and the accuracy and coding efficiency of the image can be improved.

[0377] According to embodiments of this disclosure, the predictive residual tensor can be further applied to a second neural network-based encoder to obtain feature data of the residual image, and the intermediate predictive tensor can be further applied to a second neural network-based decoder to obtain the current reconstructed image.

[0378] In the image coding method according to embodiments of the present disclosure, additional information can be used in residual coding, thereby improving the accuracy and coding efficiency of the image.

[0379] The image encoding apparatus according to an embodiment may include: a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may be configured to: obtain feature data of a current optical flow by applying a current image and a previously reconstructed image to a first encoder based on a neural network; obtain a current optical flow and a plurality of memory gates corresponding to a plurality of resolutions of the current image by applying the feature data of the current optical flow to a first decoder based on a neural network; obtain a predicted image of the current image from a previously reconstructed image based on the current optical flow; obtain a plurality of prediction tensors corresponding to a plurality of resolutions based on the predicted image and the plurality of memory gates; obtain feature data of a residual image by applying a plurality of second prediction tensors and the current image to a second encoder based on a neural network; obtain a current reconstructed image corresponding to the current image by applying the plurality of second prediction tensors, the feature data of the current optical flow, and the feature data of the residual image to a second decoder based on a neural network; and generate a bitstream including the feature data of the current optical flow and the feature data of the residual image.

[0380] According to embodiments of this disclosure, a plurality of memory gates may represent values ​​used to retain information within the current image.

[0381] In the image coding apparatus according to embodiments of the present disclosure, an image can be reconstructed based on multiple prediction tensors corresponding to multiple resolutions obtained using multiple memory gate values ​​representing values ​​for preserving key information in the image, thereby preserving the key information of the image. Therefore, errors in the reconstructed image can be prevented, and image accuracy and coding efficiency can be improved.

[0382] According to embodiments of this disclosure, multiple forgetting gates corresponding to multiple resolutions of the current image can be obtained from a first decoder based on a neural network. Multiple prediction tensors can be obtained based on a first predicted image, multiple memory gates, and multiple forgetting gates. The multiple forgetting gates can represent values ​​used to remove regions from the current image whose prediction errors are equal to or greater than preset values.

[0383] In the image coding apparatus according to embodiments of the present disclosure, multiple prediction tensors can be obtained by additionally using values ​​for removing regions from the image whose prediction errors are equal to or greater than a preset value, and the image can be reconstructed based on the multiple prediction tensors corresponding to multiple resolutions, thereby removing unnecessary information from the image. Therefore, errors in the reconstructed image can be prevented, and the accuracy of the image and coding efficiency can be improved.

[0384] According to embodiments of this disclosure, a prediction tensor of the original resolution of the current image can be determined from a plurality of prediction tensors based on the predicted image and a memory gate value corresponding to the original resolution.

[0385] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a subtraction tensor obtained by subtracting the original resolution of the current image from the prediction image and a forgetting gate value corresponding to the original resolution. Furthermore, a prediction tensor at a reduced resolution from the original resolution can be determined among a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0386] According to embodiments of this disclosure, a prediction residual tensor can be obtained based on a memory gate corresponding to the original resolution of the current image, a forget gate corresponding to the original resolution, and a prediction image. Furthermore, a prediction tensor at a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor at a reduced resolution and a memory gate corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network.

[0387] According to embodiments of this disclosure, a prediction residual tensor can be obtained by subtracting a prediction tensor of the original resolution of the current image from the prediction image. A prediction tensor of a resolution reduced from the original resolution can be determined from a plurality of prediction tensors based on an intermediate prediction tensor of the reduced resolution and a memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate value corresponding to the original resolution.

[0388] According to embodiments of this disclosure, a prediction residual tensor at a first reduced resolution can be obtained based on a subtraction tensor and a forgetting gate corresponding to a first reduced resolution. The subtraction tensor is obtained by subtracting the prediction tensor at the first reduced resolution from an intermediate prediction tensor at the first reduced resolution before applying the forgetting gate corresponding to the first reduced resolution to the prediction tensor at the first reduced resolution among a plurality of prediction tensors. Furthermore, a prediction tensor at a second reduced resolution obtained by reducing the first reduced resolution can be determined based on an intermediate prediction tensor at a second reduced resolution and a forgetting gate corresponding to the second reduced resolution among a plurality of prediction tensors. The intermediate prediction tensor at the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0389] According to embodiments of this disclosure, a prediction residual tensor of a first reduced resolution can be obtained based on a memory gate corresponding to a first reduced resolution, a forget gate corresponding to a first reduced resolution, and an intermediate prediction tensor of a first reduced resolution before applying the memory gate corresponding to the first reduced resolution to a plurality of prediction tensors. Furthermore, a prediction tensor of a second reduced resolution obtained by reducing the first reduced resolution can be determined based on the intermediate prediction tensor of a second reduced resolution and the memory gate corresponding to the second reduced resolution. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network.

[0390] According to embodiments of this disclosure, a prediction residual tensor of the first reduced resolution can be obtained by subtracting the prediction tensor of the first reduced resolution from an intermediate prediction tensor of the first reduced resolution before applying a memory gate corresponding to the first reduced resolution to the prediction tensor of the first reduced resolution among a plurality of prediction tensors. A prediction tensor of the second reduced resolution obtained by reducing the first reduced resolution among a plurality of prediction tensors can be determined based on an intermediate prediction tensor of the second reduced resolution and a memory gate corresponding to the second reduced resolution. The intermediate prediction tensor of the second reduced resolution is obtained by applying the prediction residual tensor to a reduced neural network, and the convolutional kernel of the first layer of the reduced neural network can be linearly mixed with a forgetting gate corresponding to the first reduced resolution.

[0391] In the image coding apparatus according to embodiments of the present disclosure, the predicted image of the current image can be decomposed into prediction tensors of multiple resolutions by using memory gates and forget gates corresponding to multiple resolutions, and the image can be reconstructed based on the multiple prediction tensors corresponding to multiple resolutions, thereby preserving the main information of the image and removing unnecessary information. Therefore, errors can be prevented, and the accuracy of the image and coding efficiency can be improved.

[0392] According to embodiments of this disclosure, the predictive residual tensor can be further applied to a second neural network-based encoder to obtain feature data of the residual image, and the intermediate predictive tensor can be further applied to a second neural network-based decoder to obtain the current reconstructed image.

[0393] In the image coding method according to embodiments of the present disclosure, additional information can be used in residual coding, thereby improving the accuracy and coding efficiency of the image.

[0394] Machine-readable storage media may be provided as non-transitory storage media. "Non-transitory storage media" is a tangible device and means only that it does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is temporarily stored. For example, a "non-transitory recording medium" may include a buffer for temporarily storing data.

[0395] According to embodiments, methods according to various disclosed embodiments can be provided by being included in a computer program product. The computer program product, as a commodity, can be traded between a seller and a buyer. The computer program product is distributed in the form of a device-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)), or can be distributed directly and online (e.g., downloaded or uploaded) between two user devices (e.g., smartphones) through an app store. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be stored at least temporarily in a device-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a relay server), or may be temporarily generated.

Claims

1. An image decoding method, comprising: Obtain the feature data of the current optical flow and the feature data of the residual image of the current image from the bit stream (S1810). The current optical flow and multiple memory gates corresponding to multiple resolutions of the current image are obtained by applying the feature data of the current optical flow to a first decoder based on a neural network (S1820). Based on the current optical flow, obtain the predicted image of the current image from the previously reconstructed image (S1830). Based on the predicted image and the multiple memory gates, multiple prediction tensors corresponding to the multiple resolutions are obtained (S1840). as well as The current reconstructed image corresponding to the current image is obtained by applying the feature data of the multiple prediction tensors, the feature data of the current optical flow, and the feature data of the residual image to a second decoder based on a neural network (S1850).

2. The image decoding method according to claim 1, wherein, The plurality of memory gates represent values ​​used to retain information within the current image.

3. The image decoding method according to claim 1 or 2, wherein, The multiple forgetting gates corresponding to the multiple resolutions of the current image are additionally obtained from a first decoder based on a neural network. The plurality of prediction tensors are obtained based on the prediction image, the plurality of memory gates, and the plurality of forget gates, and The multiple forgetting gates represent values ​​used to remove regions from the current image whose prediction errors are equal to or greater than a preset value.

4. The image decoding method according to any one of claims 1 to 3, wherein, The prediction tensor of the current image at its original resolution among the plurality of prediction tensors is determined based on the prediction image and the memory gate value corresponding to the original resolution.

5. The image decoding method according to any one of claims 1 to 4, wherein, The prediction residual tensor is obtained based on a subtraction tensor obtained by subtracting the prediction tensor of the current image from the original resolution of the predicted image, and a forgetting gate value corresponding to the original resolution. The prediction tensor of the plurality of prediction tensors, which is the resolution reduced from the original resolution, is determined based on the intermediate prediction tensor of the reduced resolution and the memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to the reduced neural network.

6. The image decoding method according to any one of claims 1 to 4, wherein, The predicted residual tensor is obtained based on the memory gate corresponding to the original resolution of the current image, the forget gate corresponding to the original resolution, and the predicted image. The prediction tensor of the plurality of prediction tensors, which is the resolution reduced from the original resolution, is determined based on the intermediate prediction tensor of the reduced resolution and the memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to the reduced neural network.

7. The image decoding method according to any one of claims 1 to 4, wherein, The prediction residual tensor is obtained by subtracting the prediction tensor of the current image at its original resolution from the prediction image. The prediction tensor at the reduced resolution among the multiple prediction tensors is determined based on the intermediate prediction tensor at the reduced resolution and the memory gate value corresponding to the reduced resolution. The intermediate prediction tensor is obtained by applying the prediction residual tensor to the reduced neural network. The convolutional kernels of the first layer of the neural network are linearly mixed with a forgetting gate corresponding to the original resolution.

8. The image decoding method according to any one of claims 1 to 4, wherein, The prediction residual tensor at the first reduced resolution is obtained based on a subtraction tensor and a forgetting gate corresponding to the first reduced resolution. The subtraction tensor is obtained by subtracting the prediction tensor at the first reduced resolution from the intermediate prediction tensors at the first reduced resolution before applying the forgetting gate corresponding to the first reduced resolution to the prediction tensors among the plurality of prediction tensors. The second-reduced-resolution prediction tensor, obtained by reducing the first reduction resolution, among the plurality of prediction tensors, is determined based on the intermediate prediction tensor of the second-reduced resolution and the memory gate value corresponding to the second-reduced resolution. The intermediate prediction tensor of the second-reduced resolution is obtained by applying the prediction residual tensor to the reduced neural network.

9. The image decoding method according to any one of claims 1 to 4, wherein, The prediction residual tensor at the first reduced resolution is obtained based on the memory gate corresponding to the first reduced resolution, the forget gate corresponding to the first reduced resolution, and the intermediate prediction tensor at the first reduced resolution before applying the memory gate corresponding to the first reduced resolution to the prediction tensor at the first reduced resolution among the plurality of prediction tensors. The second-reduced-resolution prediction tensor, obtained by reducing the first reduction resolution, among the plurality of prediction tensors, is determined based on the intermediate prediction tensor of the second-reduced resolution and the memory gate value corresponding to the second-reduced resolution. The intermediate prediction tensor of the second-reduced resolution is obtained by applying the prediction residual tensor to the reduced neural network.

10. The image decoding method according to any one of claims 1 to 4, wherein, The prediction residual tensor at the first reduced resolution is obtained by subtracting the prediction tensor at the first reduced resolution from the intermediate prediction tensors at the first reduced resolution before applying the memory gate corresponding to the first reduced resolution to the prediction tensors among the plurality of prediction tensors. The second-reduced-resolution prediction tensor, obtained by reducing the first reduction resolution among the plurality of prediction tensors, is determined based on the intermediate prediction tensor of the second-reduced resolution and the memory gate value corresponding to the second-reduced resolution. The intermediate prediction tensor of the second-reduced resolution is obtained by applying the prediction residual tensor to the reduced neural network. The convolutional kernels of the first layer of the scaled-down neural network are linearly mixed with a forgetting gate corresponding to the first scaled-down resolution.

11. The image decoding method according to any one of claims 5 to 10, wherein, The intermediate prediction tensor is then applied to a second neural network-based decoder to obtain the current reconstructed image.

12. An image encoding method, comprising: Feature data of the current optical flow is obtained by applying the current image and the previously reconstructed image to a first encoder based on a neural network (S1610). The current optical flow and multiple memory gates corresponding to multiple resolutions of the current image are obtained by applying the feature data of the current optical flow to a first decoder based on a neural network (S1620). Based on the current optical flow, obtain the predicted image of the current image from the previously reconstructed image (S1630). Based on the predicted image and the multiple memory gates, multiple prediction tensors corresponding to the multiple resolutions are obtained (S1640). Feature data of the residual image is obtained by applying the plurality of predicted tensors and the current image to a second encoder based on a neural network (S1650). The current reconstructed image corresponding to the current image is obtained by applying the feature data of the plurality of prediction tensors, the feature data of the current optical flow, and the feature data of the residual image to a second decoder based on a neural network (S1660); and Generate a bitstream including feature data of the current optical flow and feature data of the residual image (S1670).

13. The image encoding method according to claim 12, wherein, The plurality of memory gates represent values ​​used to retain information within the current image.

14. The image encoding method according to claim 12 or 13, wherein, The multiple forgetting gates corresponding to the multiple resolutions of the current image are additionally obtained from a first decoder based on a neural network. The plurality of prediction tensors are obtained based on the prediction image, the plurality of memory gates, and the plurality of forget gates, and The multiple forgetting gates represent values ​​used to remove regions from the current image whose prediction errors are equal to or greater than a preset value.

15. The image encoding method according to any one of claims 12 to 14, wherein, The prediction tensor of the current image at its original resolution among the plurality of prediction tensors is determined based on the prediction image and the memory gate value corresponding to the original resolution.

Images