Image decoding device and method, and image encoding device and method

Abstract

Proposed are an image decoding method and device, the method comprising: obtaining feature data for a current frame from a bitstream; decoding a first sub-frame among a plurality of sub-frames obtained by rearranging the current frame according to a predetermined decoding order on the basis of the feature data for the current frame; obtaining at least one intra-flow for the decoded first sub-frame in order to decode a second sub-frame among the plurality of sub-frames; decoding the second sub-frame by using the at least one intra-flow and the decoded first sub-frame; and reconstructing the current frame by using the decoded first sub-frame and the decoded second sub-frame.

Description

Image decoding device and method, and image encoding device and method

[0001] The present disclosure relates to an image decoding method and device, and an image encoding method and device, and more particularly, to a method and device for performing intra prediction more efficiently.

[0002] In codecs such as H.264 AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding), an image can be divided into blocks, and each block can be predicted and decoded through inter prediction or intra prediction.

[0003] Intra prediction is a method of compressing images by removing spatial redundancy within the image, and inter prediction is a method of compressing images by removing temporal redundancy between images.

[0004] Intra prediction is performed by predicting all pixels within a specific block in a single intra prediction direction based on a single intra prediction mode. Additionally, interpolation of reference pixels is performed using fixed filter coefficients of a predefined filter.

[0005] Recently, technologies for encoding / decoding images using AI (Artificial Intelligence) have been proposed, and a method for effectively encoding / decoding images by performing intra prediction using AI, for example, neural networks, is required.

[0006] A video decoding method according to one embodiment of the present disclosure may include: obtaining feature data for a current frame from a bitstream; decoding a first subframe among a plurality of subframes in which the current frame is rearranged according to a predetermined decoding order based on the feature data for the current frame; obtaining at least one intraflow for the decoded first subframe to decode a second subframe among the plurality of subframes; decoding the second subframe using the at least one intraflow and the decoded first subframe; and reconstructing the current frame using the decoded first subframe and the decoded second subframe.

[0007] An image decoding device according to one embodiment of the present disclosure may include a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may obtain feature data for a current frame from a bitstream. The at least one processor may decode a first subframe among a plurality of subframes that rearrange the current frame according to a predetermined decoding order based on the feature data for the current frame. The at least one processor may obtain at least one intraflow for the decoded first subframe in order to decode a second subframe among the plurality of subframes. The at least one processor may decode the second subframe using the at least one intraflow and the decoded first subframe. The at least one processor may reconstruct the current frame using the decoded first subframe and the decoded second subframe.

[0008] A video encoding method according to one embodiment of the present disclosure may include: generating feature data for a current frame; encoding a first subframe among a plurality of subframes in which the current frame is rearranged according to a predetermined encoding order based on the feature data for the current frame; obtaining at least one intraflow for an encoded first subframe to encode a second subframe among the plurality of subframes; encoding the second subframe using the at least one intraflow and the encoded first subframe; encoding the current frame using the encoded first subframe and the encoded second subframe; and transmitting a bitstream including the feature data for the current frame.

[0009] An image encoding device according to one embodiment of the present disclosure may include a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may generate feature data for a current frame. The at least one processor may encode a first subframe among a plurality of subframes that rearrange the current frame according to a predetermined encoding order based on the feature data for the current frame. The at least one processor may obtain at least one intraflow for the encoded first subframe to encode a second subframe among the plurality of subframes. The at least one processor may encode the second subframe using the at least one intraflow and the encoded first subframe. The at least one processor may encode the current frame using the encoded first subframe and the encoded second subframe. The at least one processor may transmit a bitstream including the feature data for the current frame.

[0010] Figure 1 is a diagram illustrating the process of encoding and decoding an image.

[0011] Figure 2 is a diagram showing blocks divided according to a tree structure from an image.

[0012] FIG. 3 is a diagram illustrating an intra flow according to one embodiment of the present disclosure.

[0013] FIG. 4 is a diagram illustrating a method of decomposing a frame into a plurality of sub-frames to restore the frame using intra-flow according to one embodiment of the present disclosure.

[0014] FIG. 5 is a diagram for explaining an intra flow between multiple subframes according to one embodiment of the present disclosure.

[0015] FIG. 6 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network according to one embodiment of the present disclosure.

[0016] FIG. 7 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network according to one embodiment of the present disclosure.

[0017] FIG. 8 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0018] FIG. 9 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0019] FIG. 10 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0020] FIG. 11 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0021] FIG. 12 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0022] FIG. 13 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0023] FIG. 14 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0024] FIG. 15 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0025] FIG. 16 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0026] FIG. 17 is a diagram for explaining a training method of a neural network used in an image encoding method and an image decoding method according to one embodiment of the present disclosure.

[0027] FIG. 18 is a flowchart of an image decoding method according to one embodiment of the present disclosure.

[0028] FIG. 19 is a diagram illustrating a configuration of an image decoding device according to one embodiment of the present disclosure.

[0029] FIG. 20 is a flowchart of an image encoding method according to one embodiment of the present disclosure.

[0030] FIG. 21 is a diagram illustrating a configuration of an image encoding device according to one embodiment of the present disclosure.

[0031] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0032] The present disclosure may be subject to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments of the present disclosure, and it should be understood that the present disclosure encompasses all modifications, equivalents, and alternatives falling within the spirit and technical scope of the various embodiments.

[0033] In describing the embodiments, detailed descriptions of related known technologies are omitted if they are deemed to unnecessarily obscure the gist of the present disclosure. Furthermore, numbers (e.g., "first," "second," etc.) used throughout the description of the specification are merely identifiers used to distinguish one component from another.

[0034] Additionally, in the present disclosure, when a component is referred to as being “connected” or “connected” to another component, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless there is a specific description to the contrary.

[0035] In addition, components expressed as 'unit', 'module', etc. in the present disclosure may be two or more components combined into one component, or one component may be divided into two or more components with more detailed functions. In addition, each component described below may additionally perform some or all of the functions performed by other components in addition to its own main function, and of course, some of the main functions performed by each component may be exclusively performed by other components.

[0036] According to one embodiment of the present disclosure, a "unit" may be implemented as a processor and a memory. The term "processor" should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some circumstances, a "processor" may also refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), and the like. The term "processor" may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0037] A processor may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to perform various functions described herein, individually and / or collectively, in a distributed fashion. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0038] The term "memory" should be interpreted broadly to include any electronic component capable of storing electronic information. The term memory may also refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. Memory is said to be in electronic communication with the processor if the processor can read information from and / or write information to the memory. Memory integrated in a processor is in electronic communication with the processor.

[0039] Additionally, in the present disclosure, 'image or picture' may mean a still image (or frame), a moving image composed of a plurality of consecutive still images, or a video.

[0040] In this disclosure, "neural network" refers to a representative example of an artificial neural network model that mimics brain neurons, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

[0041] In this disclosure, a "parameter" refers to a value used in the computational process of each layer forming a neural network. For example, it can be used when applying an input value to a given computational formula. A parameter is a value set as a result of training and can be updated using separate training data as needed.

[0042] In the present disclosure, "feature data" or "feature map" may refer to data obtained by a neural network or a neural network-based encoder processing input data. The feature data may be one-dimensional or two-dimensional data containing multiple samples. The feature data may also be referred to as a latent tensor or latent representation. The feature data may represent latent features in the data output by a neural network-based decoder.

[0043] In this disclosure, a "sample" refers to data assigned to a sampling location within one-dimensional or two-dimensional data, such as an image, block, or feature data, and is the data to be processed. For example, a sample may include a pixel within a two-dimensional image. Two-dimensional data may also be referred to as a "map."

[0044] Additionally, in the present disclosure, the 'current frame' means the frame that is the current processing target.

[0045] Additionally, in the present disclosure, 'intra flow' means a reference pixel pointed to by each pixel within a frame.

[0046] Before describing an image decoding method, an image decoding device, an image encoding method, and an image encoding device according to one embodiment, the image encoding and decoding process will be described with reference to FIGS. 1 and 2.

[0047] Figure 1 is a diagram illustrating the process of encoding and decoding an image.

[0048] The encoding device (110) transmits a bitstream generated through encoding of an image to the decoding device (150), and the decoding device (150) receives and decodes the bitstream to restore the image.

[0049] Specifically, in the encoding device (110), the prediction encoding unit (115) outputs a prediction block through inter prediction and intra prediction, and the transformation and quantization unit (120) transforms and quantizes residual samples of a residual block between the prediction block and the current block to output quantized transformation coefficients. The entropy encoding unit (125) encodes the quantized transformation coefficients and outputs them as a bitstream.

[0050] The quantized transform coefficients are restored into a residual block containing residual samples in the spatial domain through the inverse quantization and inverse transformation unit (130). The restored block, which is a combination of the prediction block and the residual block, is output as a filtered block through the deblocking filtering unit (135) and the loop filtering unit (140). The restored image containing the filtered block can be used as a reference image for the next input image in the prediction encoding unit (115).

[0051] The bitstream received by the decoding device (150) is restored to a residual block including residual samples in the spatial domain through the entropy decoding unit (155) and the inverse quantization and inverse transformation unit (160). The prediction block and the residual block output from the prediction decoding unit (175) are combined to generate a restored block, and the restored block is output as a filtered block through the deblocking filtering unit (165) and the loop filtering unit (170). The restored image including the filtered block can be used as a reference image for the next image in the prediction decoding unit (175).

[0052] The loop filtering unit (140) of the encoding device (110) performs loop filtering using filter information input according to user input or system settings. The filter information used by the loop filtering unit (140) is transmitted to the decoding device (150) via the entropy encoding unit (125). The loop filtering unit (170) of the decoding device (150) can perform loop filtering based on the filter information input from the entropy decoding unit (155).

[0053] In the process of encoding and decoding an image, the image is hierarchically divided, and encoding and decoding are performed on the blocks divided from the image. The blocks divided from the image are described with reference to FIG. 2.

[0054] Figure 2 is a drawing showing blocks divided according to a tree structure from an image (200).

[0055] A single image (200) can be divided into one or more slices or one or more tiles. A single slice can include multiple tiles.

[0056] A slice or a tile may be a sequence of one or more Maximum Coding Units (Maximum CUs).

[0057] A single maximum coding unit may be split into one or more coding units. The coding unit may be a reference block for determining a prediction mode. In other words, it may be determined whether an intra-prediction mode or an inter-prediction mode is applied to each coding unit. In the present disclosure, a maximum coding unit may be referred to as a maximum coding block, and a coding unit may be referred to as a coding block.

[0058] The size of the coding unit may be equal to or smaller than the maximum coding unit. Since the maximum coding unit is the coding unit with the maximum size, it may also be referred to as the coding unit.

[0059] One or more prediction units for intra-prediction or inter-prediction can be determined from a coding unit. The size of the prediction unit can be the same as or smaller than the coding unit.

[0060] Additionally, one or more transformation units for transformation and quantization may be determined from the coding unit. The size of the transformation unit may be the same as or smaller than the coding unit. The transformation unit serves as a reference block for transformation and quantization, and residual samples of the coding unit may be transformed and quantized for each transformation unit within the coding unit.

[0061] In the present disclosure, a current block may be a slice, tile, maximum coding unit, encoding unit, prediction unit, or transformation unit segmented from an image (200). In addition, a lower block of the current block is a block segmented from the current block. For example, if the current block is a maximum coding unit, the lower block may be a coding unit, prediction unit, or transformation unit. In addition, an upper block of the current block is a block that includes the current block as a part. For example, if the current block is a maximum coding unit, the upper block may be a picture sequence, a picture, a slice, or a tile.

[0062] Hereinafter, a video decoding method, a video decoding device, a video encoding method, and a video encoding device according to one embodiment will be described with reference to FIGS. 3 to 23.

[0063] FIG. 3 is a diagram illustrating an intra flow according to one embodiment of the present disclosure.

[0064] Predicting all pixels within a block in a single intra prediction direction based on a single intra prediction mode cannot cover diverse video content. Each pixel within a block can have a different prediction direction, each pointing to its own best reference pixels. Exploiting this non-local similarity can improve video decoding quality.

[0065] Referring to FIG. 3, pixels within a current block (320) of a current image (300) have intra flows (330) indicating different prediction directions that point to reference pixels within a surrounding reference area (310).

[0066] Intra flow (330) can be expressed as a vector representing the difference in x and y components between the current pixel and the reference pixel, a vector of the distance r component between the current pixel and the reference pixel and the angle θ component between the current pixel and the reference pixel with respect to the x-axis, or an angle between the current pixel and the reference pixel. When expressed only by the angle between the current pixel and the reference pixel, the reference pixel that is first located at the boundary of the current block based on the angle with respect to the current pixel is determined as the reference pixel.

[0067] FIG. 4 is a diagram illustrating a method of decomposing a frame into a plurality of sub-frames to restore the frame using intra-flow according to one embodiment of the present disclosure.

[0068] Each pixel within a frame has a spatial flow that indicates surrounding pixels with similar pixel values. To effectively utilize this spatial flow, a single frame can be decomposed into multiple subframes.

[0069] The current frame can be decomposed into multiple MxN subframes by dividing the current frame into MxN pixel group units or MxN decomposition blocks, and rearranging the pixels in the same position within the MxN pixel group units so that they are included in the same subframe. Accordingly, the current HxW frame can be decomposed into K (=MxN) hxw subframes. This may be referred to as pixel unshuffle or pixel decimation.

[0070] Referring to FIG. 4, the current frame (400) is divided into 2x2 pixel group units (401). In each pixel group, pixels of A located at the upper left are collected into one sub-frame to obtain a first sub-frame (410). In each pixel group, pixels of B located at the upper right are collected into one sub-frame to obtain a second sub-frame (420). In each pixel group, pixels of C located at the lower left are collected into one sub-frame to obtain a third sub-frame (430). In each pixel group, pixels of D located at the lower right are collected into one sub-frame to obtain a fourth sub-frame (440). Accordingly, a frame of size 8x16 is decomposed into four sub-frames of size 4x8.

[0071] This is just one example and is not limited thereto. That is, the current frame can be decomposed into sub-frames such as 2, 6, 8, 9, etc. For example, if the current frame is divided into 2x1 pixel groups, the pixels located at the top of each pixel group can be collected into one sub-frame to obtain the first sub-frame, and the pixels located at the bottom of each pixel group can be collected into one sub-frame to obtain the first sub-frame.

[0072] In addition, if the current frame is divided into 1x2 pixel group units, the pixels located on the left side of each pixel group can be collected into one sub-frame to obtain the first sub-frame, and the pixels located on the right side of each pixel group can be collected into one sub-frame to obtain the first sub-frame.

[0073] Additionally, if the current frame is divided into 3x2 or 2x3 pixel group units, pixels located at the same position in each pixel group can be collected into one sub-frame to obtain six sub-frames.

[0074] Additionally, if the current frame is divided into 4x2 or 2x4 pixel group units, pixels located at the same position in each pixel group can be collected into one sub-frame to obtain 8 sub-frames.

[0075] Additionally, if the current frame is divided into 3x3 pixel group units, pixels located at the same position in each pixel group can be collected into one sub-frame to obtain 9 sub-frames.

[0076] Additionally, the current frame is divided into blocks of size PxQ, and these PxQ blocks can become subframes. In this way, the number of subframes is equal to the number of PxQ blocks, where P and Q are natural numbers.

[0077] Additionally, the current frame can be in the feature map domain as well as the pixel domain, i.e., the current frame can be a feature map.

[0078] Below, the intra flow between multiple subframes is described in FIG. 5.

[0079] FIG. 5 is a diagram for explaining an intra flow between multiple subframes according to one embodiment of the present disclosure.

[0080] A pixel within a subframe decomposed from the same frame may have multiple similar pixels within other subframes by multiple intraflows or by one intraflow.

[0081] Referring to FIG. 5, a first pixel (431) in a third subframe (430) may have two intra flows (500, 510) for the first subframe (410). Among the two intra flows, the first intra flow (500) may point to a first reference pixel (411) in the first subframe (410), and the second intra flow (510) may point to a second reference pixel (412) in the first subframe (410). In addition, the first pixel (431) in the third subframe (430) may have two intra flows (520, 530) for the second subframe (420). Among the two intra flows, the third intra flow (520) may point to the first reference pixel (421) within the second sub-frame (420), and the fourth intra flow (530) may point to the second reference pixel (422) within the second sub-frame (430).

[0082] If the number of intra flows representing reference pixels in other subframes for a pixel in a subframe is N, the number of intra gates representing the reliability of each intra flow can also be N. Here, N is a natural number. In addition, N can be different for each subframe.

[0083] Referring again to FIG. 5, when the first sub-frame (410), the second sub-frame (420), and the third sub-frame (430) are decoded in that order, the first sub-frame (410) and the second sub-frame (420) are used as reference sub-frames of the third sub-frame (430) at the time the third sub-frame (430) is decoded.

[0084] Specifically, referring to FIGS. 4 and 5, when the first subframe (410), the second subframe (420), the third subframe (430), and the fourth subframe (440) are decoded in that order, there is no reference subframe when the first subframe (410) is decoded. When the second subframe (420) is subsequently decoded, the decoded first subframe (410) is used as a reference subframe. In addition, when the third subframe (430) is decoded, two subframes, namely, the decoded first subframe (410) and the decoded second subframe (420), are used as reference subframes. In addition, when the fourth subframe (440) is decoded, three subframes, namely, the decoded first subframe (410), the decoded second subframe (420), and the decoded third subframe (430), are used as reference subframes.

[0085] When a frame is decomposed into K subframes, the kth subframe uses the 1st to k-1th subframes as reference subframes. If the number of subframes that can be used as reference subframes is J, then J is less than or equal to K.

[0086] FIG. 6 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network according to one embodiment of the present disclosure.

[0087] Referring to FIG. 6, the current frame (600) is input into an encoding neural network (605) and output as feature data for the current frame (600). As the feature data for the current frame (600) is input into a decoding neural network (610), the first sub-frame (620) is decoded first according to a predetermined decoding order. The predetermined decoding order may be determined based on the learning or training results of the decoding neural network (610) for previous frames.

[0088] The decrypted first sub-frame (620) is additionally input to the decryption neural network (610). That is, the second sub-frame (630) is decrypted through the decryption neural network (610) based on the feature data for the current frame (600) and the decrypted first sub-frame (620).

[0089] The decrypted second sub-frame (630) is additionally input to the decryption neural network (610). That is, the third sub-frame (640) is decrypted through the decryption neural network (610) based on the feature data for the current frame (600) and the decrypted second sub-frame (630).

[0090] The decrypted third sub-frame (640) is additionally input to the decryption neural network (610). That is, the fourth sub-frame (650) is decrypted through the decryption neural network (610) based on the feature data for the current frame (600) and the decrypted third sub-frame (640).

[0091] The current frame is restored based on the decrypted first sub-frame (620), the decrypted second sub-frame (630), the decrypted third sub-frame (640), and the decrypted fourth sub-frame (650).

[0092] The decryption order may be any other arbitrary order, such as the order of the first sub-frame (620), the second sub-frame (630), the third sub-frame (640), and the fourth sub-frame (650), or the order of the first sub-frame (620), the second sub-frame (630), the fourth sub-frame (650), and the third sub-frame (640).

[0093] Here, the decoding neural network (610) is described as one neural network, but this is an example, and a different set of neural networks may be used for each subframe.

[0094] In Fig. 6, the decoded sub-frames are not used in the encoding neural network, but in Fig. 7, the decoded sub-frames can also be used in the encoding neural network.

[0095] FIG. 7 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network according to one embodiment of the present disclosure.

[0096] Referring to FIG. 7, the current frame (700) is input into an encoding neural network (705) and output as first feature data for the current frame (700). As the first feature data for the current frame (700) is input into a decoding neural network (710), the first sub-frame (720) is decoded first according to a predetermined decoding order. The predetermined decoding order may be determined according to the learning or training results of the decoding neural network (710) for previous frames.

[0097] The decoded first sub-frame (720) is additionally input to the encoding neural network (705). That is, second feature data is output from the encoding neural network (705) based on the current frame (700) and the decoded first sub-frame (720). The second sub-frame (730) is decoded through the decoding neural network (710) based on the second feature data and the decoded first sub-frame (720).

[0098] The decoded second sub-frame (730) is additionally input to the encoding neural network (705). That is, third feature data is output from the encoding neural network (705) based on the current frame (700) and the decoded second sub-frame (730). The third sub-frame (740) is decoded through the decoding neural network (710) based on the third feature data and the decoded second sub-frame (730).

[0099] The decoded third sub-frame (740) is additionally input to the encoding neural network (705). That is, fourth feature data is output from the encoding neural network (705) based on the current frame (700) and the decoded third sub-frame (740). The fourth sub-frame (750) is decoded through the decoding neural network (710) based on the fourth feature data and the decoded third sub-frame (740).

[0100] The current frame is restored based on the decrypted first sub-frame (620), the decrypted second sub-frame (630), the decrypted third sub-frame (640), and the decrypted fourth sub-frame (650).

[0101] The decryption order may be any other arbitrary order, such as the order of the first sub-frame (720), the second sub-frame (730), the third sub-frame (740), and the fourth sub-frame (750), or the order of the first sub-frame (720), the second sub-frame (730), the fourth sub-frame (750), and the third sub-frame (740).

[0102] Here, the decryption neural network (710) is described as one neural network, but this is an example, and a different set of neural networks may be used for each subframe.

[0103] FIG. 8 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0104] Referring to FIG. 8, the current frame (800) is input into an encoding neural network (805) and output as feature data for the current frame (800). As the feature data for the current frame (800) is input into a decoding neural network (810), the first sub-frame (820) is decoded first according to a predetermined decoding order. The predetermined decoding order may be determined according to the learning or training results of the decoding neural network (810) for previous frames.

[0105] After the first sub-frame (820) is decoded, feature data for the current frame (800) is input into a flow / gate decoding neural network (815), thereby obtaining an intra flow and an intra gate for the first sub-frame of the second sub-frame (830). By performing intra compensation (860) using the decoded first sub-frame (820) and the intra flow and the intra gate for the first sub-frame (820) of the second sub-frame (830), a second sub-prediction frame for the second sub-frame is obtained. The feature data for the current frame (800) and the second sub-prediction frame are input into a decoding neural network (810), thereby decoding the second sub-frame (830).

[0106] In the present disclosure, 'intra compensation' may utilize 'intra flow', which is a flow that points to a sample similar to each pixel or sample position within a feature map that each pixel or sample position within a feature map may have, 'intra gate value', which is a gate value associated with the intra flow and determines how reliable the intra flow is, and 'previously decoded sub-frame' or 'previously decoded sub-feature map', to generate prediction values ​​of the current sub-frame to be decoded or the current sub-feature map.

[0107] Additionally, in the present disclosure, a "sub-feature map" may refer to data obtained by processing data related to a "frame" through a neural network. Furthermore, a "sub-feature map" may be obtained by dividing samples of a "feature map" obtained through a neural network into multiple pixel groups, and rearranging samples at the same location within the multiple pixel groups so that they are included in the same sub-feature map.

[0108] After the second sub-frame (830) is decoded, feature data for the current frame (800) is input to the flow / gate decoding neural network (815), thereby obtaining the intra flow and intra gate for the first sub-frame (820) of the third sub-frame (840) and the intra flow and intra gate for the second sub-frame (830) of the third sub-frame (840). By performing intra compensation (860) using the decoded first sub-frame (820), the decoded second sub-frame (830), the intra flow and intra gate for the first sub-frame (820) of the third sub-frame (840), and the intra flow and intra gate for the second sub-frame (830) of the third sub-frame (840), a third sub-prediction frame for the third sub-frame is obtained. The feature data for the current frame (800) and the third sub-prediction frame are input into the decoding neural network (810), thereby decoding the third sub-frame (840).

[0109] After the third sub-frame (840) is decoded, feature data for the current frame (800) is input to the flow / gate decoding neural network (815), thereby obtaining the intra flow and intra gate for the first sub-frame (820) of the fourth sub-frame (850), the intra flow and intra gate for the second sub-frame (830) of the fourth sub-frame (850), and the intra flow and intra gate for the third sub-frame (840) of the fourth sub-frame (850). A fourth sub-prediction frame for the fourth sub-frame is obtained by performing intra compensation (860) using the intra flow and intra gate for the first sub-frame (820) of the decrypted first sub-frame (820), the decrypted second sub-frame (830), the decrypted third sub-frame (840), and the fourth sub-frame (850), the intra flow and intra gate for the second sub-frame (830) of the fourth sub-frame (850), and the intra flow and intra gate for the third sub-frame (840) of the fourth sub-frame (850). The feature data for the current frame (800) and the fourth sub-prediction frame are input to the decryption neural network (810), thereby decrypting the fourth sub-frame (850).

[0110] The current frame is restored based on the decrypted first sub-frame (820), the decrypted second sub-frame (830), the decrypted third sub-frame (840), and the decrypted fourth sub-frame (850).

[0111] According to one embodiment of the present disclosure, the intra gate may not be used and only the intra flow may be used. This may correspond to a case where the intra gate value is 1. The intra flow is used in the pixel domain.

[0112] Below, the intra compensation method is described.

[0113] If a frame is decomposed into K subframes, J subframes are decoded before the kth subframe, and the number of intraflows for each subframe is 1, multiple prediction values ​​for the kth subframe can be generated as shown in the following mathematical expression 1.

[0114] [Mathematical Formula 1]

[0115]

[0116] Here, j=1,..., J. The intraflow of the kth subframe for the jth subframe decoded before the kth subframe is j ) and the k-th sub-prediction frame for the k-th sub-frame by warping based on the j-th sub-frame. j->k ) can be obtained. Accordingly, as many as J k-th sub-prediction frames (SubPrediction j->k ) can be obtained. J can be less than or equal to k-1. If all k-1 subframes before the k-th subframe are utilized, J can be equal to k-1. If all k-1 subframes cannot be utilized due to complexity constraints, J can be less than k-1.

[0117] Among these multiple sub-prediction frames, one sub-prediction frame can be used for decoding. Additionally, the multiple sub-prediction frames can be used for concatenation decoding in the case of the feature map domain.

[0118] In addition, if a frame is decomposed into K subframes, J subframes are decoded before the kth subframe, and the number of intraflows for each subframe is 1, the predicted value of the kth subframe can be generated as a single value as in the following mathematical expression 2.

[0119] [Equation 2]

[0120]

[0121] Here, j=1,..., J. The intraflow of the kth subframe for the jth subframe decoded before the kth subframe is j ) and the jth sub-frame, the kth sub-prediction frame for the kth sub-frame can be obtained by warping. By equally adding the obtained J kth sub-prediction frames with a weight of 1 and dividing the result by J, one kth sub-prediction frame (SubPrediction) for the kth sub-frame is obtained. k ) can be obtained.

[0122] These sub-prediction frames can be used for decoding, for example, for residual decoding.

[0123] If a frame is decomposed into K subframes, J subframes are decoded before the kth subframe, and the number of intraflows for each subframe is N, multiple prediction values ​​for the kth subframe can be generated as shown in the following mathematical expression 3.

[0124] [Equation 3]

[0125]

[0126] Here, j = 1,..., J. The nth intraflow (intraFlow) among the N intraflows of the kth subframe for the jth subframe decoded before the kth subframe n,j ) and the jth sub-frame, kth sub-prediction frames for N kth sub-frames can be obtained by warping. By dividing the result of adding all of these N kth sub-prediction frames by N, J kth sub-prediction frames (SubPrediction j->k ) can be obtained.

[0127] Among these multiple sub-prediction frames, one sub-prediction frame can be used for decoding. Additionally, the multiple sub-prediction frames can be used for concatenation decoding in the case of the feature map domain.

[0128] In addition, if a frame is decomposed into K subframes, J subframes are decoded before the kth subframe, and the number of intraflows for each subframe is N, the predicted value of the kth subframe can be generated as a single value as in the following mathematical expression 4.

[0129] [Equation 4]

[0130]

[0131] Here, j = 1,..., J. The nth intraflow (intraFlow) among the N intraflows of the kth subframe for the jth subframe decoded before the kth subframe n,j ) and the jth sub-frame, kth sub-prediction frames for N kth sub-frames can be obtained by warping. By adding all of these N kth sub-prediction frames, J kth sub-prediction frames can be obtained. By equally adding the obtained J kth sub-prediction frames with a weight of 1 and dividing the result by N*J, one kth sub-prediction frame (SubPrediction) for the kth sub-frame is obtained. k ) can be obtained.

[0132] These sub-prediction frames can be used for decoding, for example, for residual decoding.

[0133] If a frame is decomposed into K subframes, and there are J decoded subframes before the kth subframe, and the number of intra flows for each subframe is 1 and the number of intra gates for each subframe is also 1, multiple prediction values ​​for the kth subframe can be generated as shown in the following mathematical expression 5.

[0134] [Equation 5]

[0135]

[0136] Here, j=1,..., J. The intraflow of the kth subframe for the jth subframe decoded before the kth subframe is j ) and warp based on the j-th sub-frame, and apply the intra gate (intraGate) of the k-th sub-frame to the j-th sub-frame based on the warping result. j ) for the kth sub-prediction frame (SubPrediction j->k ) can be obtained. Accordingly, as many as J k-th sub-prediction frames (SubPrediction j->k ) can be obtained.

[0137] Among these multiple sub-prediction frames, one sub-prediction frame can be used for decoding. Additionally, the multiple sub-prediction frames can be used for concatenation decoding in the case of the feature map domain.

[0138] In addition, if a frame is decomposed into K subframes, and there are J decoded subframes before the kth subframe, and the number of intra flows for each subframe is 1 and the number of intra gates for each subframe is also 1, the predicted value of the kth subframe can be generated as a single value as in the following mathematical expression 6.

[0139] [Equation 6]

[0140]

[0141] Here, j=1,..., J. The intraflow of the kth subframe for the jth subframe decoded before the kth subframe is j ) and warp based on the j-th sub-frame, and apply the intra gate (intraGate) of the k-th sub-frame to the j-th sub-frame based on the warping result. j ) can be obtained by multiplying the kth sub-prediction frame for the kth sub-frame. By dividing the result of adding the obtained J kth sub-prediction frames with the same weight of 1 by J, one kth sub-prediction frame (SubPrediction) for the kth sub-frame is obtained. k ) can be obtained.

[0142] These sub-prediction frames can be used for decoding, for example, for residual decoding.

[0143] If a frame is decomposed into K subframes, and there are J decoded subframes before the kth subframe, and the number of intra flows for each subframe is N and the number of intra gates for each subframe is also N, then multiple prediction values ​​for the kth subframe can be generated as shown in the following mathematical expression 7.

[0144] [Equation 7]

[0145]

[0146] Here, j = 1,..., J. The nth intraflow (intraFlow) among the N intraflows of the kth subframe for the jth subframe decoded before the kth subframe n,j ) and warp based on the j-th sub-frame, and apply the n-th intra gate (intraGate) of the k-th sub-frame to the j-th sub-frame based on the warping result. n,j) can be obtained for the N k-th sub-prediction frames. By multiplying the result of adding all of these N k-th sub-prediction frames and dividing it by N, J k-th sub-prediction frames (SubPrediction j->k ) can be obtained.

[0147] Among these multiple sub-prediction frames, one sub-prediction frame can be used for decoding. Additionally, the multiple sub-prediction frames can be used for concatenation decoding in the case of the feature map domain.

[0148] In addition, if a frame is decomposed into K subframes, and there are J decoded subframes before the kth subframe, and the number of intra flows for each subframe is N and the number of intra gates for each subframe is also N, the predicted value of the kth subframe can be generated as a single value as in the following mathematical expression 8.

[0149] [Equation 8]

[0150]

[0151] Here, j = 1,..., J. The nth intraflow (intraFlow) among the N intraflows of the kth subframe for the jth subframe decoded before the kth subframe n,j ) and warp based on the j-th sub-frame, and apply the n-th intra gate (intraGate) of the k-th sub-frame to the j-th sub-frame based on the warping result. n,j) can be obtained for the N k-th sub-prediction frames. By adding all of these N k-th sub-prediction frames, J k-th sub-prediction frames can be obtained. By dividing the result of adding the obtained J k-th sub-prediction frames by N*J, one k-th sub-prediction frame (SubPrediction k ) can be obtained.

[0152] These sub-prediction frames can be used for decoding, for example, for residual decoding.

[0153] FIG. 9 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0154] Referring to FIG. 9, the current frame (900) is input into an encoding neural network (905) and output as feature data for the current frame (900). The feature data for the current frame (900) is input into a feature extraction neural network (911) within a decoding neural network (910), thereby outputting feature extraction data for the current frame (900). The feature extraction data may be data in which the features of the feature data for the current frame (900) are further enhanced.

[0155] The feature extraction data for the current frame (900) is input into the progressive intra block (912), thereby decoding the first sub-frame (920), the second sub-frame (930), the third sub-frame (940), and the fourth sub-frame (950).

[0156] Specifically, feature extraction data for the current frame (900) is input into an upscale neural network (913) within a progressive intra block (912), so that the first subframe (920) is decoded first according to a predetermined decoding order. The upscale neural network (913) is used to upscale the feature extraction data for the current frame (900) because it is of a reduced size compared to the original size of the current frame (900). The predetermined decoding order may be determined according to the learning or training results of the upscale neural network (913) for previous frames.

[0157] After the first sub-frame (920) is decoded, feature extraction data for the current frame (900) is input into a flow / gate upscale neural network (914) within an incremental intra block (912), thereby obtaining an intra flow and an intra gate for the first sub-frame (920) of the second sub-frame (930). By performing intra compensation (915) using the decoded first sub-frame (920) and the intra flow and the intra gate for the first sub-frame (920) of the second sub-frame (930), a second sub-prediction frame for the second sub-frame (930) is obtained. The feature extraction data for the current frame (900) and the second sub-prediction frame are input into an upscale neural network (913), thereby decoding the second sub-frame (930).

[0158] After the second sub-frame (930) is decoded, feature extraction data for the current frame (900) is input into a flow / gate upscale neural network (914) within the progressive intra block (912), thereby obtaining an intra flow and an intra gate for the first sub-frame (920) of the third sub-frame (940) and an intra flow and an intra gate for the second sub-frame (930) of the third sub-frame (940). By performing intra compensation (915) using the decoded first sub-frame (920), the decoded second sub-frame (930), the intra flow and the intra gate for the first sub-frame (920) of the third sub-frame (940), and the intra flow and the intra gate for the second sub-frame (930) of the third sub-frame (940), a third sub-prediction frame for the third sub-frame (940) is obtained. The feature extraction data for the current frame (900) and the third sub-prediction frame are input to the upscale neural network (913), thereby decoding the third sub-frame (940).

[0159] After the third sub-frame (940) is decoded, feature data for the current frame (900) is input into a flow / gate upscale neural network (914) within the progressive intra block (912), thereby obtaining an intra flow and an intra gate for the first sub-frame (920) of the fourth sub-frame (950), an intra flow and an intra gate for the second sub-frame (930) of the fourth sub-frame (950), and an intra flow and an intra gate for the third sub-frame (940) of the fourth sub-frame (950). A fourth sub-prediction frame for the fourth sub-frame (950) is obtained by performing intra compensation (915) using the intra flow and intra gate for the first sub-frame (920) of the decoded first sub-frame (920), the decoded second sub-frame (930), the decoded third sub-frame (940), and the fourth sub-frame (950), the intra flow and intra gate for the second sub-frame (930) of the fourth sub-frame (950), and the intra flow and intra gate for the third sub-frame (940) of the fourth sub-frame (950). The feature extraction data for the current frame (900) and the fourth sub-prediction frame are input to the upscale neural network (913), thereby decoding the fourth sub-frame (950).

[0160] The current frame is restored based on the decrypted first sub-frame (920), the decrypted second sub-frame (930), the decrypted third sub-frame (940), and the decrypted fourth sub-frame (950).

[0161] According to one embodiment of the present disclosure, the intra gate may not be used, and only the intra flow may be used. This may correspond to a case where the intra gate value is 1. Since the intra compensation is performed in the pixel domain, the intra flow is in the pixel domain.

[0162] Although Figure 9 illustrates a single feature extraction neural network, multiple feature extraction neural networks may be used. Accordingly, the feature extraction data may be output through multiple feature extraction neural networks.

[0163] Although Figure 9 illustrates a single progressive intra block, there may be multiple progressive intra blocks. A configuration with two progressive intra blocks is described below in Figure 10.

[0164] FIG. 10 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0165] Referring to FIG. 10, the current frame (1000) is input into an encoding neural network (1005) and output as feature data for the current frame (1000). The feature data for the current frame (1000) is input into a feature extraction neural network (1011) within a decoding neural network (1010), thereby outputting feature extraction data for the current frame (1000). The feature extraction data may be data in which the features of the feature data for the current frame (1000) are further enhanced.

[0166] Feature extraction data for the current frame (1000) is input into the first progressive intra block (1012) and the second progressive intra block (1022), thereby decoding the first sub-frame (1020), the second sub-frame (1030), the third sub-frame (1040), and the fourth sub-frame (1050).

[0167] Specifically, feature extraction data for the current frame (900) is input into an upscale neural network (1013) within a first progressive intra block (1012), thereby first obtaining a first sub-frame feature map according to a predetermined decoding order. The upscale neural network (1013) is used to upscale the feature extraction data for the current frame (1000) because the size is reduced compared to the original size of the current frame (1000). The predetermined decoding order may be determined according to the learning or training results of the upscale neural network (1013) for previous frames.

[0168] After the first sub-frame feature map is acquired, feature extraction data for the current frame (1000) is input into a flow / gate upscale neural network (1014) in the first progressive intra block (1012), thereby acquiring intraflow and intragate for the first sub-frame feature map of the second sub-frame feature map. By performing intra compensation (1015) using the acquired first sub-frame feature map and the intraflow and intragate for the first sub-frame feature map of the second sub-frame feature map, a second sub-prediction frame feature map for the second sub-frame feature map is acquired. The feature extraction data for the current frame (1000) and the second sub-prediction frame feature map are input into an upscale neural network (1013), thereby acquiring the second sub-frame feature map.

[0169] After the second sub-frame feature map is acquired, feature extraction data for the current frame (1000) is input into a flow / gate upscale neural network (1014) in the first progressive intra block (1012), thereby acquiring an intra flow and an intra gate for the first sub-frame feature map of the third sub-frame feature map and an intra flow and an intra gate for the second sub-frame feature map of the third sub-frame feature map. By performing intra compensation (1015) using the acquired first sub-frame feature map, the acquired second sub-frame feature map, the intra flow and the intra gate for the first sub-frame feature map of the third sub-frame feature map, and the intra flow and the intra gate for the second sub-frame feature map of the third sub-frame feature map, a third sub-prediction frame feature map for the third sub-frame feature map is acquired. The feature extraction data for the current frame (1000) and the third sub-prediction frame feature map are input into the upscale neural network (1013), thereby obtaining the third sub-frame feature map.

[0170] After the third sub-frame feature map is acquired, feature data for the current frame (1000) is input into a flow / gate upscale neural network (1014) within the first progressive intra block (1012), thereby acquiring an intra flow and an intra gate for the first sub-frame feature map of the fourth sub-frame feature map, an intra flow and an intra gate for the second sub-frame feature map of the fourth sub-frame feature map, and an intra flow and an intra gate for the third sub-frame feature map of the fourth sub-frame feature map. A fourth sub-prediction frame feature map for the fourth sub-frame feature map is obtained by performing intra compensation (1015) using the intra flow and intra gate for the first sub-frame feature map of the acquired first sub-frame feature map, the intra flow and intra gate for the second sub-frame feature map of the acquired second sub-frame feature map, and the intra flow and intra gate for the third sub-frame feature map of the acquired third sub-frame feature map. The fourth sub-frame feature map is obtained by inputting the feature extraction data for the current frame (1000) and the fourth sub-prediction frame feature map into an upscale neural network (1013).

[0171] A feature map for the current frame (1000) is obtained based on the acquired first sub-frame feature map, the acquired second sub-frame feature map, the acquired third sub-frame feature map, and the acquired fourth sub-frame feature map.

[0172] The feature map for the current frame (1000) is input into the upscale neural network (1023) within the second progressive intra block (1022), so that the first subframe (1020) is decoded first according to a predetermined decoding order. The feature map for the current frame (1000) is data representing the features of the current frame based on the neural network, and may be larger than the feature extraction data of the current frame (1000) and smaller than the current frame (1000). The predetermined decoding order may be determined according to the learning or training results of the upscale neural network (1023) for previous frames.

[0173] After the first sub-frame (1020) is decoded, the feature map for the current frame (1000) is input into a flow / gate upscale neural network (1024) in a second progressive intra block (1022), thereby obtaining an intra flow and an intra gate for the first sub-frame (1020) of the second sub-frame (1030). By performing intra compensation (1025) using the decoded first sub-frame (1020) and the intra flow and the intra gate for the first sub-frame (1020) of the second sub-frame (1030), a second sub-prediction frame for the second sub-frame (1030) is obtained. The feature map for the current frame (1000) and the second sub-prediction frame are input into an upscale neural network (1023), thereby decoding the second sub-frame (1030).

[0174] After the second sub-frame (1030) is decoded, the feature map for the current frame (1000) is input into the flow / gate upscale neural network (1024) within the second progressive intra block (1022), thereby obtaining the intra flow and intra gate for the first sub-frame (1020) of the third sub-frame (1040) and the intra flow and intra gate for the second sub-frame (1030) of the third sub-frame (1040). A third sub-prediction frame for the third sub-frame (1040) is obtained by performing intra compensation (1025) using the intra flow and intra gate for the first sub-frame (1020) of the decoded first sub-frame (1020), the decoded second sub-frame (1030), and the intra flow and intra gate for the second sub-frame (1030) of the third sub-frame (1040). The feature map for the current frame (1000) and the third sub-prediction frame are input to the upscale neural network (1023), thereby decoding the third sub-frame (1040).

[0175] After the third sub-frame (1040) is decoded, the feature map for the current frame (1000) is input into the flow / gate upscale neural network (1024) within the second progressive intra block (1022), thereby obtaining the intra flow and intra gate for the first sub-frame (1020) of the fourth sub-frame (1050), the intra flow and intra gate for the second sub-frame (1030) of the fourth sub-frame (1050), and the intra flow and intra gate for the third sub-frame (1040) of the fourth sub-frame (1050). A fourth sub-prediction frame for the fourth sub-frame (1050) is obtained by performing intra compensation (1025) using the intra flow and intra gate for the first sub-frame (1020) of the decoded first sub-frame (1020), the decoded second sub-frame (1030), the decoded third sub-frame (1040), and the fourth sub-frame (1050), the intra flow and intra gate for the second sub-frame (1030) of the fourth sub-frame (1050), and the intra flow and intra gate for the third sub-frame (1040) of the fourth sub-frame (1050). The feature map for the current frame (1000) and the fourth sub-prediction frame are input to the upscale neural network (1023), thereby decoding the fourth sub-frame (1050).

[0176] The current frame is restored based on the decrypted first sub-frame (1020), the decrypted second sub-frame (1030), the decrypted third sub-frame (1040), and the decrypted fourth sub-frame (1050).

[0177] According to one embodiment of the present disclosure, the intra gate may not be used and only the intra flow may be used. This may correspond to a case where the intra gate value is 1. In the first progressive intra block (1012), intra compensation is performed in the feature map domain, so the intra flow of the first progressive intra block (1012) is in the feature map domain, and in the second progressive intra block (1022), intra compensation is performed in the pixel domain, so the intra flow of the second progressive intra block (1022) is in the pixel domain.

[0178] Although Figure 10 illustrates a single feature extraction neural network, multiple feature extraction neural networks may be used. Accordingly, the feature extraction data may be output through multiple feature extraction neural networks.

[0179] FIG. 11 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0180] Referring to FIG. 11, the current frame (1100) is input into an encoding neural network (1105) and output as feature data for the current frame (1100).

[0181] The feature data for the current frame (1100) is input into the progressive intra block (1111) within the decoding neural network (1110), thereby decoding the first sub-frame (1120), the second sub-frame (1130), the third sub-frame (1140), and the fourth sub-frame (1150).

[0182] Specifically, feature data for the current frame (1100) is input into an upscale neural network (1112) within a progressive intra block (1111), so that the first subframe (1120) is decoded first according to a predetermined decoding order. The upscale neural network (1112) is used to upscale the feature data for the current frame (1100) because the feature data has a reduced size compared to the original size of the current frame (1100). The predetermined decoding order may be determined based on the learning or training results of the upscale neural network (1112) for previous frames.

[0183] After the first sub-frame (1120) is decoded, feature data for the current frame (1100) is input into a flow / gate upscale neural network (1113) within an incremental intra block (1111), thereby obtaining an intra flow and an intra gate for the first sub-frame (1120) of the second sub-frame (1130). By performing intra compensation (1114) using the decoded first sub-frame (1120) and the intra flow and the intra gate for the first sub-frame (1120) of the second sub-frame (1130), a second sub-prediction frame for the second sub-frame (1130) is obtained. The feature data for the current frame (1100) and the second sub-prediction frame are input into an upscale neural network (1112), thereby decoding the second sub-frame (1130).

[0184] After the second sub-frame (1130) is decoded, feature data for the current frame (1100) is input into a flow / gate upscale neural network (1113) within a progressive intra block (1111), thereby obtaining an intra flow and intra gate for the first sub-frame (1120) of the third sub-frame (1140) and an intra flow and intra gate for the second sub-frame (1130) of the third sub-frame (1140). A third sub-prediction frame for the third sub-frame (1140) is obtained by performing intra compensation (1114) using the intra flow and intra gate for the first sub-frame (1120) of the decoded first sub-frame (1120), the decoded second sub-frame (1130), and the intra flow and intra gate for the second sub-frame (1130) of the third sub-frame (1140). The feature data for the current frame (1100) and the third sub-prediction frame are input to the upscale neural network (1112), thereby decoding the third sub-frame (1140).

[0185] After the third sub-frame (1140) is decoded, feature data for the current frame (1100) is input into a flow / gate upscale neural network (1113) within a progressive intra block (1111), thereby obtaining an intra flow and an intra gate for the first sub-frame (1120) of the fourth sub-frame (1150), an intra flow and an intra gate for the second sub-frame (1130) of the fourth sub-frame (1150), and an intra flow and an intra gate for the third sub-frame (1140) of the fourth sub-frame (1150). A fourth sub-prediction frame for the fourth sub-frame (1150) is obtained by performing intra compensation (1114) using the intra flow and intra gate for the first sub-frame (1120) of the decoded first sub-frame (1120), the decoded second sub-frame (1130), the decoded third sub-frame (1140), and the fourth sub-frame (1150), the intra flow and intra gate for the second sub-frame (1130) of the fourth sub-frame (1150), and the intra flow and intra gate for the third sub-frame (1140) of the fourth sub-frame (1150). The feature data for the current frame (1100) and the fourth sub-prediction frame are input to the upscale neural network (1114), thereby decoding the fourth sub-frame (1150).

[0186] The current frame is restored based on the decrypted first sub-frame (1120), the decrypted second sub-frame (1130), the decrypted third sub-frame (1140), and the decrypted fourth sub-frame (1150).

[0187] According to one embodiment of the present disclosure, the intra gate may not be used, and only the intra flow may be used. This may correspond to a case where the intra gate value is 1. In the progressive intra block (1111), intra compensation is performed in the pixel domain, and therefore, the intra flow of the progressive intra block (1111) is in the pixel domain.

[0188] Although Figure 11 illustrates a single progressive intra block, there may be multiple progressive intra blocks. A configuration with two progressive intra blocks is described below in Figure 12.

[0189] FIG. 12 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0190] Referring to FIG. 12, the current frame (1200) is input into an encoding neural network (1205) and output as feature data for the current frame (1200).

[0191] Feature data for the current frame (1200) is input into the first progressive intra block (1211) and the second progressive intra block (1221) within the decoding neural network (1210), thereby decoding the first sub-frame (1220), the second sub-frame (1230), the third sub-frame (1240), and the fourth sub-frame (1250).

[0192] Specifically, feature data for the current frame (1200) is input into an upscale neural network (1212) within a first progressive intra block (1211), thereby first obtaining a first subframe feature map according to a predetermined decoding order. The predetermined decoding order may be determined based on the learning or training results of the upscale neural network (1212) for previous frames.

[0193] After the first sub-frame feature map is acquired, feature data for the current frame (1200) is input into a flow / gate upscale neural network (1213) in the first progressive intra block (1211), thereby acquiring an intra flow and an intra gate for the first sub-frame feature map of the second sub-frame feature map. By performing intra compensation (1214) using the acquired first sub-frame feature map and the intra flow and the intra gate for the first sub-frame feature map of the second sub-frame feature map, a second sub-prediction frame feature map for the second sub-frame feature map is acquired. The feature data for the current frame (1200) and the second sub-prediction frame feature map are input into an upscale neural network (1212), thereby acquiring the second sub-frame feature map.

[0194] After the second sub-frame feature map is acquired, feature data for the current frame (1200) is input into a flow / gate upscale neural network (1213) in the first progressive intra block (1212), thereby acquiring an intra flow and an intra gate for the first sub-frame feature map of the third sub-frame feature map and an intra flow and an intra gate for the second sub-frame feature map of the third sub-frame feature map. By performing intra compensation (1214) using the acquired first sub-frame feature map, the acquired second sub-frame feature map, the intra flow and the intra gate for the first sub-frame feature map of the third sub-frame feature map, and the intra flow and the intra gate for the second sub-frame feature map of the third sub-frame feature map, a third sub-prediction frame feature map for the third sub-frame feature map is acquired. The feature data for the current frame (1200) and the third sub-prediction frame feature map are input into the upscale neural network (1214), thereby obtaining the third sub-frame feature map.

[0195] After the third sub-frame feature map is acquired, feature data for the current frame (1200) is input into a flow / gate upscale neural network (1213) within the first progressive intra block (1211), thereby acquiring an intra flow and an intra gate for the first sub-frame feature map of the fourth sub-frame feature map, an intra flow and an intra gate for the second sub-frame feature map of the fourth sub-frame feature map, and an intra flow and an intra gate for the third sub-frame feature map of the fourth sub-frame feature map. A fourth sub-prediction frame feature map for the fourth sub-frame feature map is obtained by performing intra compensation (1214) using the intra flow and intra gate for the first sub-frame feature map of the acquired first sub-frame feature map, the intra flow and intra gate for the second sub-frame feature map of the acquired third sub-frame feature map, and the intra flow and intra gate for the third sub-frame feature map of the acquired fourth sub-frame feature map. The feature extraction data for the current frame (1200) and the fourth sub-prediction frame feature map are input to the upscale neural network (1211), thereby obtaining the fourth sub-frame feature map.

[0196] A feature map for the current frame (1200) is obtained based on the acquired first sub-frame feature map, the acquired second sub-frame feature map, the acquired third sub-frame feature map, and the acquired fourth sub-frame feature map.

[0197] The feature map for the current frame (1200) is input into the upscale neural network (1222) within the second progressive intra block (1221), so that the first subframe (1220) is decoded first according to a predetermined decoding order. The feature map for the current frame (1200) is data representing the features of the current frame based on the neural network, and may be larger than the feature data of the current frame (1200) and smaller than the current frame (1200). The predetermined decoding order may be determined according to the learning or training results of the upscale neural network (1222) for previous frames.

[0198] After the first sub-frame (1220) is decoded, the feature map for the current frame (1200) is input into a flow / gate upscale neural network (1223) in the second progressive intra block (1221), thereby obtaining an intra flow and an intra gate for the first sub-frame (1220) of the second sub-frame (1230). By performing intra compensation (1224) using the decoded first sub-frame (1220) and the intra flow and the intra gate for the first sub-frame (1220) of the second sub-frame (1230), a second sub-prediction frame for the second sub-frame (1230) is obtained. The feature map for the current frame (1200) and the second sub-prediction frame are input into an upscale neural network (1222), thereby decoding the second sub-frame (1230).

[0199] After the second sub-frame (1230) is decoded, the feature map for the current frame (1200) is input into the flow / gate upscale neural network (1223) within the second progressive intra block (1221), thereby obtaining the intra flow and intra gate for the first sub-frame (1220) of the third sub-frame (1240) and the intra flow and intra gate for the second sub-frame (1230) of the third sub-frame (1240). A third sub-prediction frame for the third sub-frame (1240) is obtained by performing intra compensation (1224) using the intra flow and intra gate for the first sub-frame (1220) of the decoded first sub-frame (1220), the decoded second sub-frame (1230), and the intra flow and intra gate for the second sub-frame (1230) of the third sub-frame (1240). The feature map for the current frame (1200) and the third sub-prediction frame are input to the upscale neural network (1222), thereby decoding the third sub-frame (1240).

[0200] After the third sub-frame (1240) is decoded, the feature map for the current frame (1200) is input into the flow / gate upscale neural network (1223) in the second progressive intra block (1221), thereby obtaining the intra flow and intra gate for the first sub-frame (1220) of the fourth sub-frame (1250), the intra flow and intra gate for the second sub-frame (1230) of the fourth sub-frame (1250), and the intra flow and intra gate for the third sub-frame (1240) of the fourth sub-frame (1250). A fourth sub-prediction frame for the fourth sub-frame (1250) is obtained by performing intra compensation (1224) using the intra flow and intra gate for the first sub-frame (1220) of the decoded first sub-frame (1220), the decoded second sub-frame (1230), the decoded third sub-frame (1240), and the fourth sub-frame (1250), the intra flow and intra gate for the second sub-frame (1230) of the fourth sub-frame (1250), and the intra flow and intra gate for the third sub-frame (1240) of the fourth sub-frame (1250). The feature map for the current frame (1200) and the fourth sub-prediction frame are input to the upscale neural network (1222), thereby decoding the fourth sub-frame (1250).

[0201] The current frame is restored based on the decrypted first sub-frame (1220), the decrypted second sub-frame (1230), the decrypted third sub-frame (1240), and the decrypted fourth sub-frame (1250).

[0202] According to one embodiment of the present disclosure, the intra gate may not be used and only the intra flow may be used. This may correspond to a case where the intra gate value is 1. In the first progressive intra block (1211), intra compensation is performed in the feature map domain, so the intra flow of the first progressive intra block (1211) is in the feature map domain, and in the second progressive intra block (1221), intra compensation is performed in the pixel domain, so the intra flow of the second progressive intra block (1221) is in the pixel domain.

[0203] FIG. 13 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0204] Referring to FIG. 13, the current frame (1300) is input into an encoding neural network (1305) and output as feature data for the current frame (1300).

[0205] The feature data for the current frame (1300) is input into the progressive intra block (1311) within the decoding neural network (1310), thereby decoding the first sub-frame (1320), the second sub-frame (1330), the third sub-frame (1340), and the fourth sub-frame (1350).

[0206] Specifically, feature data for the current frame (1300) is input into an upscale neural network (1312) within a progressive intra block (1311), so that the first subframe (1320) is decoded first according to a predetermined decoding order. The upscale neural network (1312) is used to upscale the feature data for the current frame (1300) because the feature data is of a reduced size compared to the original size of the current frame (1300). The predetermined decoding order may be determined based on the learning or training results of the upscale neural network (1312) for previous frames.

[0207] After the first sub-frame (1320) is decoded, feature data for the current frame (1300) is input into a flow / gate upscale neural network (1313) within a progressive intra block (1311), thereby obtaining an intra flow and an intra gate for the first sub-frame (1320) of the second sub-frame (1330). By performing intra compensation (1314) using the decoded first sub-frame (1320) and the intra flow and the intra gate for the first sub-frame (1320) of the second sub-frame (1330), a second sub-prediction frame for the second sub-frame (1130) is obtained. A second sub-residual frame is obtained by inputting feature data for the current frame (1300) into an upscale neural network (1312). The second sub-frame (1330) is decoded by adding the second sub-prediction frame and the second sub-residual frame (1315).

[0208] After the second sub-frame (1330) is decoded, feature data for the current frame (1300) is input into a flow / gate upscale neural network (1313) within the progressive intra block (1311), thereby obtaining the intra flow and intra gate for the first sub-frame (1320) of the third sub-frame (1340) and the intra flow and intra gate for the second sub-frame (1330) of the third sub-frame (1340). A third sub-prediction frame for the third sub-frame (1340) is obtained by performing intra compensation (1314) using the intra flow and intra gate for the first sub-frame (1320) of the decoded first sub-frame (1320), the decoded second sub-frame (1330), and the intra flow and intra gate for the second sub-frame (1330) of the third sub-frame (1340). The feature data for the current frame (1300) is input to an upscale neural network (1312) to obtain a third sub-residual frame. The third sub-prediction frame and the third sub-residual frame are added (1315) to decode the third sub-frame (1340).

[0209] After the third sub-frame (1340) is decoded, feature data for the current frame (1300) is input into a flow / gate upscale neural network (1313) within a progressive intra block (1311), thereby obtaining an intra flow and an intra gate for the first sub-frame (1320) of the fourth sub-frame (1350), an intra flow and an intra gate for the second sub-frame (1330) of the fourth sub-frame (1350), and an intra flow and an intra gate for the third sub-frame (1340) of the fourth sub-frame (1350). A fourth sub-prediction frame for the fourth sub-frame (1350) is obtained by performing intra compensation (1314) using the intra flow and intra gate for the first sub-frame (1320) of the decoded first sub-frame (1320), the decoded second sub-frame (1330), the decoded third sub-frame (1340), and the intra flow and intra gate for the third sub-frame (1340) of the fourth sub-frame (1350), the intra flow and intra gate for the second sub-frame (1330) of the fourth sub-frame (1350), and the intra flow and intra gate for the third sub-frame (1340) of the fourth sub-frame (1350). The feature data for the current frame (1300) is input to the upscale neural network (1314), thereby obtaining the fourth sub-residual frame. The fourth sub-prediction frame and the fourth sub-residual frame are added (1315) to decode the fourth sub-frame (1350).

[0210] The current frame is restored based on the decrypted first sub-frame (1320), the decrypted second sub-frame (1330), the decrypted third sub-frame (1340), and the decrypted fourth sub-frame (1350).

[0211] According to one embodiment of the present disclosure, the intra gate may not be used, and only the intra flow may be used. This may correspond to a case where the intra gate value is 1. In the progressive intra block (1311), intra compensation is performed in the pixel domain, and therefore, the intra flow of the progressive intra block (1311) is in the pixel domain.

[0212] Although Figure 13 illustrates a single progressive intra block, there may be multiple progressive intra blocks. A configuration with two progressive intra blocks is described below in Figure 14.

[0213] FIG. 14 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0214] Referring to FIG. 14, the current frame (1400) is input to an encoding neural network (1405) and output as feature data for the current frame (1400).

[0215] Feature data for the current frame (1400) is input into the first progressive intra block (1411) and the second progressive intra block (1421) within the decoding neural network (1410), thereby decoding the first sub-frame (1420), the second sub-frame (1430), the third sub-frame (1440), and the fourth sub-frame (1450).

[0216] Specifically, feature data for the current frame (1400) is input into an upscale neural network (1412) within a first progressive intra block (1411), thereby first obtaining a first subframe feature map according to a predetermined decoding order. The predetermined decoding order may be determined based on the learning or training results of the upscale neural network (1412) for previous frames.

[0217] After the first sub-frame feature map is acquired, feature data for the current frame (1400) is input into a flow / gate upscale neural network (1413) in the first progressive intra block (1411), thereby acquiring an intra flow and an intra gate for the first sub-frame feature map of the second sub-frame feature map. By performing intra compensation (1414) using the acquired first sub-frame feature map and the intra flow and the intra gate for the first sub-frame feature map of the second sub-frame feature map, a second sub-prediction frame feature map for the second sub-frame feature map is acquired. A second sub-residual frame feature map is acquired by inputting feature data for the current frame (1400) into an upscale neural network (1412). The second sub-frame feature map is acquired by adding (1415) the second sub-prediction frame feature map and the second sub-residual frame feature map.

[0218] After the second sub-frame feature map is acquired, feature data for the current frame (1400) is input into a flow / gate upscale neural network (1413) within a first progressive intra block (1412), thereby acquiring an intraflow and an intragate for the first sub-frame feature map of the third sub-frame feature map and an intraflow and an intragate for the second sub-frame feature map of the third sub-frame feature map. By performing intra compensation (1414) using the acquired first sub-frame feature map, the acquired second sub-frame feature map, the intraflow and the intragate for the first sub-frame feature map of the third sub-frame feature map, and the intraflow and the intragate for the second sub-frame feature map of the third sub-frame feature map, a third sub-prediction frame feature map for the third sub-frame feature map is acquired. The feature data for the current frame (1400) is input into an upscale neural network (1414), thereby acquiring a third sub-residual frame feature map. The third sub-frame feature map is obtained by adding the third sub-prediction frame feature map and the third sub-residual frame feature map (1415).

[0219] After the third sub-frame feature map is acquired, feature data for the current frame (1400) is input into a flow / gate upscale neural network (1413) within the first progressive intra block (1411), thereby acquiring an intra flow and an intra gate for the first sub-frame feature map of the fourth sub-frame feature map, an intra flow and an intra gate for the second sub-frame feature map of the fourth sub-frame feature map, and an intra flow and an intra gate for the third sub-frame feature map of the fourth sub-frame feature map. A fourth sub-prediction frame feature map is obtained for the fourth sub-frame feature map by performing intra compensation (1414) using the intra flow and intra gate for the first sub-frame feature map of the acquired first sub-frame feature map, the intra flow and intra gate for the second sub-frame feature map of the acquired third sub-frame feature map, the intra flow and intra gate for the third sub-frame feature map of the acquired fourth sub-frame feature map. A fourth sub-residual frame feature map is obtained by inputting feature extraction data for the current frame (1400) into an upscale neural network (1411). The fourth sub-prediction frame feature map and the fourth sub-residual frame feature map are added (1415) to obtain the fourth sub-frame feature map.

[0220] A feature map for the current frame (1400) is obtained based on the acquired first sub-frame feature map, the acquired second sub-frame feature map, the acquired third sub-frame feature map, and the acquired fourth sub-frame feature map.

[0221] The feature map for the current frame (1400) is input into the upscale neural network (1422) within the second progressive intra block (1421), so that the first subframe (1420) is decoded first according to a predetermined decoding order. The feature map for the current frame (1400) is data representing the features of the current frame based on the neural network, and may be larger than the feature data of the current frame (1400) and smaller than the current frame (1400). The predetermined decoding order may be determined according to the learning or training results of the upscale neural network (1422) for previous frames.

[0222] After the first sub-frame (1420) is decoded, the feature map for the current frame (1400) is input into a flow / gate upscale neural network (1423) in a second progressive intra block (1421), thereby obtaining an intra flow and an intra gate for the first sub-frame (1420) of the second sub-frame (1430). By performing intra compensation (1424) using the decoded first sub-frame (1420) and the intra flow and the intra gate for the first sub-frame (1420) of the second sub-frame (1430), a second sub-prediction frame for the second sub-frame (1430) is obtained. A second sub-residual frame is obtained by inputting the feature map for the current frame (1400) into an upscale neural network (1222). The second sub-frame (1430) is decoded by adding the second sub-residual frame and the second sub-prediction frame (1425).

[0223] After the second sub-frame (1430) is decoded, the feature map for the current frame (1400) is input into the flow / gate upscale neural network (1423) within the second progressive intra block (1421), thereby obtaining the intra flow and intra gate for the first sub-frame (1420) of the third sub-frame (1440) and the intra flow and intra gate for the second sub-frame (1430) of the third sub-frame (1440). A third sub-prediction frame for the third sub-frame (1440) is obtained by performing intra compensation (1424) using the intra flow and intra gate for the first sub-frame (1420) of the decoded first sub-frame (1420), the decoded second sub-frame (1430), and the intra flow and intra gate for the second sub-frame (1430) of the third sub-frame (1440). A feature map for the current frame (1400) is input to an upscale neural network (1422) to obtain a third sub-residual frame. The third sub-residual frame and the third sub-prediction frame are added (1425) to decode the third sub-frame (1440).

[0224] After the third sub-frame (1440) is decoded, the feature map for the current frame (1400) is input into the flow / gate upscale neural network (1423) in the second progressive intra block (1421), thereby obtaining the intra flow and intra gate for the first sub-frame (1420) of the fourth sub-frame (1450), the intra flow and intra gate for the second sub-frame (1430) of the fourth sub-frame (1450), and the intra flow and intra gate for the third sub-frame (1440) of the fourth sub-frame (1450). A fourth sub-prediction frame for the fourth sub-frame (1450) is obtained by performing intra compensation (1424) using the intra flow and intra gate for the first sub-frame (1420), the intra flow and intra gate for the second sub-frame (1430) of the fourth sub-frame (1450), and the intra flow and intra gate for the third sub-frame (1440) of the fourth sub-frame (1450) of the decoded first sub-frame (1420), the decoded second sub-frame (1430), the decoded third sub-frame (1440), and the fourth sub-frame (1450). A feature map for the current frame (1400) is input to an upscale neural network (1422) to obtain a fourth sub-residual frame. The fourth sub-residual frame and the fourth sub-prediction frame are added (1425) to decode the fourth sub-frame (1450).

[0225] The current frame is restored based on the decrypted first sub-frame (1420), the decrypted second sub-frame (1430), the decrypted third sub-frame (1440), and the decrypted fourth sub-frame (1450).

[0226] According to one embodiment of the present disclosure, the intra gate may not be used and only the intra flow may be used. This may correspond to a case where the intra gate value is 1. In the first progressive intra block (1411), intra compensation is performed in the feature map domain, so the intra flow of the first progressive intra block (1411) is in the feature map domain, and in the second progressive intra block (1421), intra compensation is performed in the pixel domain, so the intra flow of the second progressive intra block (1421) is in the pixel domain.

[0227] FIG. 15 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0228] Referring to FIG. 15, the current frame (1500) is input into an encoding neural network (1505) and output as feature data for the current frame (1500).

[0229] The feature data for the current frame (1500) is input into the progressive intra block (1511) within the decoding neural network (1510), thereby decoding the first sub-frame (1520), the second sub-frame (1530), the third sub-frame (1540), and the fourth sub-frame (1550).

[0230] Specifically, feature data for the current frame (1500) is input into an upscale neural network (1512) within a progressive intra block (1511), so that the first subframe (1320) is decoded first according to a predetermined decoding order. The upscale neural network (1512) is used to upscale the feature data for the current frame (1500) because the size is reduced compared to the original size of the current frame (1500). When data is output from the upscale neural network (1512), channel splitting is performed so that data is output for each channel. Data for some channels may be data for intra flows, data for some channels may be data for intra gates, and data for the remaining channels may be data for subframes. For example, if there is only one intraflow, data from two channels is data for the intraflow, and if there is only one introflow, no intra gate is required, so data from the remaining channels excluding the two channels may be data for subframes. In addition, if there are four intraflows, data from eight channels may be data for the intraflow, data from one channel may be data for the intragate, and data from the remaining channels may be data for subframes. The predetermined decoding order may be determined based on the learning or training results of the upscale neural network (1512) for previous frames.

[0231] After the first sub-frame (1520) is decoded, feature data for the current frame (1500) is input into an upscale neural network (1512) within an incremental intra block (1511), thereby obtaining an intra flow and an intra gate for the first sub-frame (1520) of the second sub-frame (1530) through channel division. By performing intra compensation (1513) using the decoded first sub-frame (1520) and the intra flow and the intra gate for the first sub-frame (1520) of the second sub-frame (1530), a second sub-prediction frame for the second sub-frame (1530) is obtained. A second sub-residual frame is obtained through channel division by inputting feature data for the current frame (1500) into an upscale neural network (1512). The second sub-frame (1530) is decoded by adding the second sub-prediction frame and the second sub-residual frame (1514).

[0232] After the second sub-frame (1530) is decoded, feature data for the current frame (1500) is input into an upscale neural network (1512) within an incremental intra block (1511), thereby obtaining an intra flow and an intra gate for the first sub-frame (1520) of the third sub-frame (1540) and an intra flow and an intra gate for the second sub-frame (1530) of the third sub-frame (1540) through channel division. A third sub-prediction frame for the third sub-frame (1540) is obtained by performing intra compensation (1513) using the intra flow and intra gate for the first sub-frame (1520) of the decoded first sub-frame (1520), the decoded second sub-frame (1530), and the intra flow and intra gate for the second sub-frame (1530) of the third sub-frame (1540) of the decoded third sub-frame (1520). The feature data for the current frame (1500) is input to an upscale neural network (1512), and a third sub-residual frame is obtained through channel division. The third sub-prediction frame and the third sub-residual frame are added (1514) to decode the third sub-frame (1540).

[0233] After the third sub-frame (1540) is decoded, feature data for the current frame (1500) is input into an upscale neural network (1512) within a progressive intra block (1511), thereby obtaining an intra flow and an intra gate for the first sub-frame (1520) of the fourth sub-frame (1550), an intra flow and an intra gate for the second sub-frame (1530) of the fourth sub-frame (1550), and an intra flow and an intra gate for the third sub-frame (1540) of the fourth sub-frame (1550) through channel division. A fourth sub-prediction frame for the fourth sub-frame (1550) is obtained by performing intra compensation (1513) using the intra flow and intra gate for the first sub-frame (1520), the intra flow and intra gate for the second sub-frame (1530) of the fourth sub-frame (1550), and the intra flow and intra gate for the third sub-frame (1540) of the fourth sub-frame (1550) of the decoded first sub-frame (1520), the decoded second sub-frame (1530), the decoded third sub-frame (1540), and the fourth sub-frame (1550). Feature data for the current frame (1500) is input to an upscale neural network (1512), and a fourth sub-residual frame is obtained through channel division. The fourth sub-prediction frame and the fourth sub-residual frame are added (1514) to decode the fourth sub-frame (1550).

[0234] The current frame is restored based on the decrypted first sub-frame (1520), the decrypted second sub-frame (1530), the decrypted third sub-frame (1540), and the decrypted fourth sub-frame (1550).

[0235] According to one embodiment of the present disclosure, the intra gate may not be used, and only the intra flow may be used. This may correspond to a case where the intra gate value is 1. In the progressive intra block (1511), intra compensation is performed in the pixel domain, and therefore, the intra flow of the progressive intra block (1511) is in the pixel domain.

[0236] Although Figure 15 illustrates a single progressive intra block, there may be multiple progressive intra blocks. A configuration with two progressive intra blocks is described below in Figure 16.

[0237] FIG. 16 is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and intraflow according to one embodiment of the present disclosure.

[0238] Referring to FIG. 16, the current frame (1600) is input to an encoding neural network (1605) and output as feature data for the current frame (1600).

[0239] Feature data for the current frame (1600) is input into the first progressive intra block (1611) and the second progressive intra block (1621) within the decoding neural network (1610), thereby decoding the first sub-frame (1620), the second sub-frame (1630), the third sub-frame (1640), and the fourth sub-frame (1650).

[0240] Specifically, feature data for the current frame (1600) is input into an upscale neural network (1612) within a first progressive intra block (1611), thereby first obtaining a first subframe feature map according to a predetermined decoding order through channel division. The predetermined decoding order may be determined based on the learning or training results of the upscale neural network (1612) for previous frames.

[0241] After the first sub-frame feature map is obtained, feature data for the current frame (1600) is input into an upscale neural network (1613) in the first progressive intra block (1611), thereby obtaining an intraflow and an intragate for the first sub-frame feature map of the second sub-frame feature map through channel division. By performing intra compensation (613) using the obtained first sub-frame feature map and the intraflow and the intragate for the first sub-frame feature map of the second sub-frame feature map, a second sub-prediction frame feature map for the second sub-frame feature map is obtained. A second sub-residual frame feature map is obtained by inputting feature data for the current frame (1600) into an upscale neural network (1612) through channel division. The second sub-frame feature map is obtained by adding (1614) the second sub-prediction frame feature map and the second sub-residual frame feature map.

[0242] After the second sub-frame feature map is acquired, feature data for the current frame (1600) is input into an upscale neural network (1612) within a first progressive intra block (1612), thereby obtaining an intraflow and an intragate for the first sub-frame feature map of the third sub-frame feature map and an intraflow and an intragate for the second sub-frame feature map of the third sub-frame feature map through channel division. By performing intra compensation (1613) using the acquired first sub-frame feature map, the acquired second sub-frame feature map, the intraflow and an intragate for the first sub-frame feature map of the third sub-frame feature map, and the intraflow and an intragate for the second sub-frame feature map of the third sub-frame feature map, a third sub-prediction frame feature map for the third sub-frame feature map is acquired. Feature data for the current frame (1600) is input into an upscale neural network (1612), thereby obtaining a third sub-residual frame feature map through channel division. The third sub-frame feature map is obtained by adding the third sub-prediction frame feature map and the third sub-residual frame feature map (1614).

[0243] After the third sub-frame feature map is acquired, feature data for the current frame (1600) is input into an upscale neural network (1612) within the first progressive intra block (1611), thereby obtaining an intraflow and an intragate for the first sub-frame feature map of the fourth sub-frame feature map, an intraflow and an intragate for the second sub-frame feature map of the fourth sub-frame feature map, and an intraflow and an intragate for the third sub-frame feature map of the fourth sub-frame feature map through channel division. By performing intra compensation (1613) using the intra flow and intra gate for the first sub-frame feature map of the acquired first sub-frame feature map, the intra flow and intra gate for the second sub-frame feature map of the acquired third sub-frame feature map, and the intra flow and intra gate for the third sub-frame feature map of the acquired fourth sub-frame feature map, a fourth sub-prediction frame feature map is acquired for the fourth sub-frame feature map. The feature extraction data for the current frame (1600) is input to the upscale neural network (1612), and the fourth sub-residual frame feature map is acquired through channel division. The fourth sub-frame feature map is acquired by adding the fourth sub-prediction frame feature map and the fourth sub-residual frame feature map (1614).

[0244] A feature map for the current frame (1600) is obtained based on the acquired first sub-frame feature map, the acquired second sub-frame feature map, the acquired third sub-frame feature map, and the acquired fourth sub-frame feature map.

[0245] The feature map for the current frame (1600) is input into the upscale neural network (1622) within the second progressive intra block (1621), so that the first subframe (1620) is decoded first according to a predetermined decoding order. The feature map for the current frame (1600) is data representing the features of the current frame based on the neural network, and may be larger than the feature data of the current frame (1600) and smaller than the current frame (1600). The predetermined decoding order may be determined according to the learning or training results of the upscale neural network (1622) for previous frames.

[0246] After the first sub-frame (1620) is decoded, the feature map for the current frame (1600) is input into an upscale neural network (1622) in a second progressive intra block (1621), thereby obtaining an intra flow and an intra gate for the first sub-frame (1620) of the second sub-frame (1630) through channel division. By performing intra compensation (1623) using the decoded first sub-frame (1620) and the intra flow and the intra gate for the first sub-frame (1620) of the second sub-frame (1630), a second sub-prediction frame for the second sub-frame (1630) is obtained. The feature map for the current frame (1600) is input into an upscale neural network (1622), thereby obtaining a second sub-residual frame through channel division. The second sub-frame (1630) is decoded by adding the second sub-residual frame and the second sub-prediction frame (1624).

[0247] After the second sub-frame (1630) is decoded, the feature map for the current frame (1600) is input into an upscale neural network (1622) within the second progressive intra block (1621), thereby obtaining the intra flow and intra gate for the first sub-frame (1620) of the third sub-frame (1640) and the intra flow and intra gate for the second sub-frame (1630) of the third sub-frame (1640) through channel division. A third sub-prediction frame for the third sub-frame (1640) is obtained by performing intra compensation (1623) using the intra flow and intra gate for the first sub-frame (1620) of the decoded first sub-frame (1620), the decoded second sub-frame (1630), and the intra flow and intra gate for the second sub-frame (1630) of the third sub-frame (1640) and the intra flow and intra gate for the third sub-frame (1640). A feature map for the current frame (1600) is input to an upscale neural network (1622) to obtain a third sub-residual frame through channel division. The third sub-residual frame and the third sub-prediction frame are added (1624) to decode the third sub-frame (1640).

[0248] After the third sub-frame (1640) is decoded, the feature map for the current frame (1600) is input into the upscale neural network (1622) within the second progressive intra block (1621), thereby obtaining the intra flow and intra gate for the first sub-frame (1620) of the fourth sub-frame (1650), the intra flow and intra gate for the second sub-frame (1630) of the fourth sub-frame (1650), and the intra flow and intra gate for the third sub-frame (1640) of the fourth sub-frame (1650) through channel division. A fourth sub-prediction frame for the fourth sub-frame (1650) is obtained by performing intra compensation (1623) using the intra flow and intra gate for the first sub-frame (1620), the intra flow and intra gate for the second sub-frame (1630) of the fourth sub-frame (1650), and the intra flow and intra gate for the third sub-frame (1640) of the fourth sub-frame (1650) of the decoded first sub-frame (1620), the decoded second sub-frame (1630), the decoded third sub-frame (1640), and the fourth sub-frame (1650). The feature map for the current frame (1600) is input to the upscale neural network (1622), and the fourth sub-residual frame is obtained through channel division. The fourth sub-frame (1650) is decoded by adding the fourth sub-residual frame and the fourth sub-prediction frame (1624).

[0249] The current frame is restored based on the decrypted first sub-frame (1620), the decrypted second sub-frame (1630), the decrypted third sub-frame (1640), and the decrypted fourth sub-frame (1650).

[0250] According to one embodiment of the present disclosure, the intra gate may not be used and only the intra flow may be used. This may correspond to a case where the intra gate value is 1. In the first progressive intra block (1611), intra compensation is performed in the feature map domain, so the intra flow of the first progressive intra block (1611) is in the feature map domain, and in the second progressive intra block (1621), intra compensation is performed in the pixel domain, so the intra flow of the second progressive intra block (1621) is in the pixel domain.

[0251] The method of utilizing the intra flow described in FIGS. 8 to 16 can be applied to learned image coding.

[0252] Furthermore, the method of utilizing intraflow described in FIGS. 8 to 16 can be applied in the same manner to a motion coding network that utilizes optical flow of learned video coding. This is because optical flow has strong spatial correlation since the motion of all pixels of an object must be identical, and strong non-local similarity since the motion of all pixels of an object's boundary must be identical.

[0253] Additionally, the method of utilizing the intraflow described in FIGS. 8 to 16 can also be applied to the pixel coding network of learned video coding, since pixels have strong spatial correlation and strong non-local similarity.

[0254] FIG. 17 is a diagram for explaining a training method of a neural network used in an image encoding method and an image decoding method according to one embodiment of the present disclosure.

[0255] Referring to FIG. 17, an encoding neural network (1705), a decoding neural network (1710), and a flow / gate decoding neural network (1715) can be trained using a current frame (1700) for training.

[0256] Specifically, the current frame for training (1700) is input to an encoding neural network (1705) to obtain feature data for the current frame for training (1700). The first sub-frame for training (1720) is obtained by inputting the feature data for the current frame for training (1700) to a decoding neural network (1710).

[0257] By inputting feature data for the current training frame (1700) into the flow / gate decoding neural network (1715), the intra flow and intra gate for the first training sub-frame (1720) of the second training sub-frame (1730) are obtained. Intra compensation (1760) is performed using the intra flow and intra gate for the first training sub-frame (1720) of the second training sub-frame (1730) and the obtained first training sub-frame (1720). The second training sub-prediction frame is obtained through the intra compensation (1760). The second training sub-frame (1730) is obtained by inputting the feature data for the current training frame (1700) and the second training sub-prediction frame into the decoding neural network (1710).

[0258] Feature data for the current training frame (1700) is input to the flow / gate decoding neural network (1715), thereby obtaining the intra flow and intra gate for the first training sub-frame (1720) of the third training sub-frame (1740) and the intra flow and intra gate for the second training sub-frame (1730) of the third training sub-frame (1740). Intra compensation (1760) is performed using the intra flow and intra gate for the first training sub-frame (1720) of the third training sub-frame (1740), the intra flow and intra gate for the second training sub-frame (1730) of the third training sub-frame (1740), the obtained first training sub-frame (1720), and the obtained second training sub-frame (1730). The third training sub-prediction frame is obtained through the intra compensation (1760). The feature data for the current training frame (1700) and the third sub-prediction frame for training are input into the decoding neural network (1710), thereby obtaining the third sub-frame for training (1740).

[0259] By inputting feature data for the current training frame (1700) into the flow / gate decoding neural network (1715), the intra flow and intra gate for the first training sub-frame (1720) of the fourth training sub-frame (1750), the intra flow and intra gate for the second training sub-frame (1730) of the fourth training sub-frame (1750), and the intra flow and intra gate for the third training sub-frame (1740) of the fourth training sub-frame (1750) are obtained. Intra compensation (1760) is performed using the intra flow and intra gate for the first training sub-frame (1720) of the fourth training sub-frame (1750), the intra flow and intra gate for the second training sub-frame (1730) of the fourth training sub-frame (1750), the intra flow and intra gate for the third training sub-frame (1740) of the fourth training sub-frame (1750), the acquired first training sub-frame (1720), the acquired second training sub-frame (1730), and the acquired third training sub-frame (1740). The fourth training sub-prediction frame is acquired through the intra compensation (1760). The fourth training sub-frame (1750) is acquired by inputting the feature data for the current training frame (1700) and the fourth training sub-prediction frame into the decoding neural network (1710).

[0260] A training restoration frame (1770) is obtained using the acquired first training sub-frame (1720), the acquired second training sub-frame (1730), the acquired third training sub-frame (1740), and the acquired fourth training sub-frame (1750).

[0261] In the training process of FIG. 17, the encoding neural network (1705), the decoding neural network (1710), and the flow / gate decoding neural network (1715) can be trained so that the training restoration frame (1770) becomes as similar as possible to the training current frame (1700) through comparison (1775). To this end, as illustrated in FIG. 17, the first loss information (1780) can be used to train the encoding neural network (1705), the decoding neural network (1710), and the flow / gate decoding neural network (1715).

[0262] The first loss information (1780) may correspond to a difference between a current training frame (1700) and a reconstructed training frame (1770). In one embodiment, the difference between the current training frame (1700) and the reconstructed training frame (1770) may include at least one of an L1-norm value, an L2-norm value, a Structural Similarity (SSIM) value, a Peak Signal-To-Noise Ratio-Human Vision System (PSNR-HVS) value, a Multiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF) value, or a Video Multimethod Assessment Fusion (VMAF) value between the current training frame (1700) and the reconstructed training frame (1770).

[0263] The first loss information (1780) is related to the quality of the restored image, so it can also be referred to as quality loss information.

[0264] The second loss information (1790) can be derived from the bit rate of the bitstream generated as an encoding result for the feature data of the current frame (1700) for training.

[0265] Since the second loss information (1790) is related to the encoding efficiency for the feature data of the current frame (1700) for training, the second loss information may be referred to as compression loss information.

[0266] The encoding neural network (1705), the decoding neural network (1710), and the flow / gate decoding neural network (1715) can be trained such that the final loss information derived from the first loss information (1780) and / or the second loss information (1790) is reduced or minimized.

[0267] In one embodiment, the encoding neural network (1705), the decoding neural network (1710), and the flow / gate decoding neural network (1715) can reduce or minimize the final loss information by changing the values ​​of preset parameters.

[0268] In one embodiment, the final loss information can be calculated according to the following mathematical expression (9).

[0269] [Equation 9]

[0270] Final loss information = a*first loss information + b*second loss information

[0271] In mathematical expression 9, a and b are weights applied to the first loss information (1780) and the second loss information (1790), respectively.

[0272] According to mathematical expression 9, the encoding neural network (1705), the decoding neural network (1710), and the flow / gate decoding neural network (1715) can be trained so that the training reconstruction frame is as similar as possible to the training current frame, and the bit rate of the bitstream generated through encoding the feature data of the training current frame is minimized.

[0273] The training process described with reference to FIG. 17 can be performed by a training device. The training device may be, for example, an image encoding device (2100) or a separate server. The parameters obtained as a result of the training may be stored in the image encoding device (2100) and the image decoding device (1900).

[0274] Although Fig. 17 describes a training method for the neural networks described in Fig. 8, the neural networks described in Figs. 9 to 16 can also be trained in a similar manner.

[0275] FIG. 18 is a flowchart of an image decoding method according to one embodiment of the present disclosure.

[0276] Referring to FIG. 18, in step S1810, the image decoding device (1900) can obtain feature data for the current frame from the bitstream.

[0277] According to one embodiment of the present disclosure, the current frame may be a pixel map containing pixels.

[0278] According to one embodiment of the present disclosure, the current frame may be a feature map.

[0279] In step S1820, the image decoding device (1900) can decode a first subframe among a plurality of subframes that rearrange the current frame according to a predetermined decoding order based on feature data for the current frame.

[0280] According to one embodiment of the present disclosure, the plurality of sub-frames may be rearranged such that the current frame is divided into a plurality of pixel groups and pixels at the same position within the plurality of pixel groups are included in the same sub-frame.

[0281] At step S1830, the video decoding device (1900) can obtain at least one intra flow for the decoded first subframe in order to decode the second subframe among the plurality of subframes.

[0282] According to one embodiment of the present disclosure, at least one intra flow for a decoded first sub-frame may represent a reference pixel within the decoded first sub-frame for a first pixel within a second sub-frame.

[0283] At step S1840, the video decoding device (1900) can decode the second subframe using at least one intraflow and the decoded first subframe.

[0284] According to one embodiment of the present disclosure, the image decoding device (1900) obtains an intra gate value for at least one intra flow, and a second subframe is decoded using the at least one intra flow, the intra gate value, and the decoded first subframe, wherein the intra gate value may represent a weight for the at least one intra flow.

[0285] According to one embodiment of the present disclosure, when at least one intra flow is a plurality of intra flows, the image decoding device (1900) obtains an intra gate value corresponding to one of the plurality of intra flows, and a second sub-frame can be decoded using one of the plurality of intra flows, the intra gate value corresponding to one of the plurality of intra flows, and the decoded first sub-frame.

[0286] According to one embodiment of the present disclosure, the video decoding device (1900) may obtain a second sub-residual frame for a second sub-frame, obtain a second sub-prediction frame using at least one intra flow and the decoded first sub-frame, and decode the second sub-frame using the second sub-residual frame and the second sub-prediction frame.

[0287] At step S1850, the video decoding device (1900) can restore the current frame using the decoded first sub-frame and the decoded second sub-frame.

[0288] According to one embodiment of the present disclosure, when a plurality of sub-frames include a third sub-frame, the video decoding device (1900) may obtain at least one second intra flow for the decoded first sub-frame and at least one third intra flow for the decoded second sub-frame to decode the third sub-frame, decode the third sub-frame using the at least one second intra flow, the decoded first sub-frame, the at least one third intra flow, and the decoded second sub-frame, and reconstruct the current frame using the decoded first sub-frame, the decoded second sub-frame, and the decoded third sub-frame.

[0289] According to one embodiment of the present disclosure, when a plurality of sub-frames include a fourth sub-frame, the video decoding device (1900) may obtain at least one fourth intra flow for a decoded first sub-frame, at least one fifth intra flow for a decoded second sub-frame, and at least one sixth intra flow for a decoded third sub-frame to decode the fourth sub-frame, and may decode the fourth sub-frame using the at least one fourth intra flow, the decoded first sub-frame, the at least one fifth intra flow, the decoded second sub-frame, the at least one sixth intra flow, and the decoded third sub-frame, and may reconstruct the current frame using the decoded first sub-frame, the decoded second sub-frame, the decoded third sub-frame, and the decoded fourth sub-frame.

[0290] FIG. 19 is a diagram illustrating a configuration of an image decoding device according to one embodiment of the present disclosure.

[0291] Referring to FIG. 19, the image decoding device (1900) may include a receiving unit (1910), an AI processing unit (1920), an intra flow / gate acquisition unit (1930), an intra compensation unit (1940), and a decoding unit (1950).

[0292] The receiver (1910), AI processing unit (1920), intraflow / gate acquisition unit (1930), intracompensation unit (1940), and decoder (1950) may be implemented as a processor. The processor may include at least one processing circuit and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to individually and / or collectively perform the various functions described herein in a distributed manner. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions. The receiving unit (1910), the AI ​​processing unit (1920), the intraflow / gate acquisition unit (1930), the intracompensation unit (1940), and the decryption unit (1950) may operate according to instructions stored in a memory (not shown).

[0293] Although FIG. 19 illustrates the receiver (1910), the AI ​​processing unit (1920), the intra flow / gate acquisition unit (1930), the intra compensation unit (1940), and the decoding unit (1950) individually, the receiver (1910), the AI ​​processing unit (1920), the intra flow / gate acquisition unit (1930), the intra compensation unit (1940), and the decoding unit (1950) may be implemented through a single processor. In this case, the receiver (1910), the AI ​​processing unit (1920), the intra flow / gate acquisition unit (1930), the intra compensation unit (1940), and the decoding unit (1950) may be implemented as a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software. Additionally, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure or a memory processing unit for utilizing external memory.

[0294] The receiving unit (1910), the AI ​​processing unit (1920), the intra flow / gate acquisition unit (1930), the intra compensation unit (1940), and the decoding unit (1950) may be configured with multiple processors. In this case, the receiving unit (1910), the AI ​​processing unit (1920), the intra flow / gate acquisition unit (1930), the intra compensation unit (1940), and the decoding unit (1950) may be implemented by a combination of dedicated processors, or may be implemented by a combination of multiple general-purpose processors such as an AP, a CPU, or a GPU, and software.

[0295] The receiver (1910) can obtain feature data for the current frame from the bitstream.

[0296] Feature data for the current frame can be transmitted to the AI ​​processing unit (1920) and the intraflow / gate acquisition unit (1930).

[0297] The AI ​​processing unit (1920) can decode the first subframe among multiple subframes according to a predetermined decoding order.

[0298] The intra flow / gate acquisition unit (1930) can output the intra flow and intra gate for the first sub-frame of the second sub-frame.

[0299] The intra compensation unit (1940) can receive the intra flow and intra gate for the first sub-frame of the second sub-frame from the intra flow / gate acquisition unit (1930), and can receive the first sub-frame from the AI ​​processing unit (1920).

[0300] The intra compensation unit (1940) can obtain a second sub prediction frame based on the intra flow and intra gate for the first sub frame of the second sub frame and the first sub frame.

[0301] The intra compensation unit (1940) can transmit the second sub-prediction frame to the AI ​​processing unit (1920).

[0302] The AI ​​processing unit (1920) can decode the second sub-frame based on feature data for the current frame and the second sub-prediction frame.

[0303] According to one embodiment of the present disclosure, the AI ​​processing unit (1920) may include an intra flow / gate acquisition unit (1930) and an intra compensation unit (1940). Accordingly, the AI ​​processing unit (1920) may decode a first subframe among a plurality of subframes according to a predefined decoding order, output an intra flow and an intra gate for the first subframe of the second subframe, acquire a second sub-prediction frame based on the intra flow and the intra gate for the first subframe of the second subframe and the first subframe, and decode the second subframe based on feature data for the current frame and the second sub-prediction frame.

[0304] The decryption unit (1950) can obtain a restored frame of the current frame based on the first sub-frame and the second sub-frame.

[0305] According to one embodiment of the present disclosure, when a plurality of sub-frames include a third sub-frame, the intra-flow / gate acquisition unit (1930) acquires at least one second intra-flow for a first sub-frame and at least one third intra-flow for a second sub-frame to decode the third sub-frame, the intra-compensation unit (1940) acquires a third sub-prediction frame based on the at least one second intra-flow, the first sub-frame, the at least one third intra-flow, and the second sub-frame, and transfers the third sub-prediction frame to the AI ​​processing unit (1920), the AI ​​processing unit (1920) decodes the third sub-frame based on feature data for the current frame and the third sub-prediction frame, and the decoding unit (1950) acquires a reconstructed frame of the current frame based on the first sub-frame, the second sub-frame, and the third sub-frame.

[0306] According to one embodiment of the present disclosure, when a plurality of sub-frames include a fourth sub-frame, the intra-flow / gate acquisition unit (1930) acquires at least one fourth intra-flow for the first sub-frame, at least one fifth intra-flow for the second sub-frame, and at least one sixth intra-flow for the third sub-frame to decode the fourth sub-frame, and the intra-compensation unit (1940) acquires at least one fourth intra-flow, the first sub-frame, the at least one fifth intra-flow, and the second sub-frame. At least one sixth intra flow, a fourth sub-prediction frame is obtained based on the third sub-frame, and the fourth sub-prediction frame is transmitted to the AI ​​processing unit (1920), the AI ​​processing unit (1920) decodes the fourth sub-frame based on feature data for the current frame and the fourth sub-prediction frame, and the decoding unit (1950) can obtain a restored frame of the current frame based on the first sub-frame, the second sub-frame, the third sub-frame, and the fourth sub-frame.

[0307] The AI ​​processing unit (1920) intraflow / gate acquisition unit (1930) can be implemented as a neural network including one or more layers (e.g., convolutional layers).

[0308] The AI ​​processing unit (1920) intraflow / gate acquisition unit (1930) may be stored in memory. The AI ​​processing unit (1920) intraflow / gate acquisition unit (1930) may be implemented with at least one dedicated processor for AI.

[0309] FIG. 20 is a flowchart of an image encoding method according to one embodiment of the present disclosure.

[0310] Referring to FIG. 20, in step S2010, the image encoding device (2100) can generate feature data for the current frame.

[0311] According to one embodiment of the present disclosure, the current frame may be a pixel map containing pixels.

[0312] According to one embodiment of the present disclosure, the current frame may be a feature map.

[0313] In step S2020, the video encoding device (2100) can encode a first subframe among a plurality of subframes that rearrange the current frame according to a predetermined encoding order based on feature data for the current frame.

[0314] According to one embodiment of the present disclosure, the plurality of sub-frames may be rearranged such that the current frame is divided into a plurality of pixel groups and pixels at the same position within the plurality of pixel groups are included in the same sub-frame.

[0315] In step S2030, the video encoding device (2100) can obtain at least one intra flow for the encoded first subframe in order to encode the second subframe among the plurality of subframes.

[0316] According to one embodiment of the present disclosure, at least one intra flow for an encoded first subframe may represent a reference pixel within the encoded first subframe for a first pixel within a second subframe.

[0317] At step S2040, the video encoding device (2100) can encode the second subframe using at least one intraflow and the decoded first subframe.

[0318] According to one embodiment of the present disclosure, a video encoding device (2100) obtains an intra gate value for at least one intra flow, and a second subframe is encoded using the at least one intra flow, the intra gate value, and the encoded first subframe, wherein the intra gate value may represent a weight for the at least one intra flow.

[0319] According to one embodiment of the present disclosure, when at least one intra flow is a plurality of intra flows, the image encoding device (2100) obtains an intra gate value corresponding to one of the plurality of intra flows, and a second sub-frame can be encoded using one of the plurality of intra flows, the intra gate value corresponding to one of the plurality of intra flows, and the encoded first sub-frame.

[0320] According to one embodiment of the present disclosure, the video encoding device (2100) may obtain a second sub-residual frame for a second sub-frame, obtain a second sub-prediction frame using at least one intraflow and an encoded first sub-frame, and encode the second sub-frame using the second sub-residual frame and the second sub-prediction frame.

[0321] At step S2050, the video encoding device (2100) can encode the current frame using the encoded first sub-frame and the encoded second sub-frame.

[0322] According to one embodiment of the present disclosure, when a plurality of sub-frames include a third sub-frame, the video encoding device (2100) may obtain at least one second intra flow for an encoded first sub-frame and at least one third intra flow for an encoded second sub-frame to encode the third sub-frame, encode the third sub-frame using the at least one second intra flow, the encoded first sub-frame, the at least one third intra flow, and the encoded second sub-frame, and encode the current frame using the encoded first sub-frame, the encoded second sub-frame, and the encoded third sub-frame.

[0323] According to one embodiment of the present disclosure, when a plurality of sub-frames include a fourth sub-frame, the video encoding device (2100) may obtain at least one fourth intra flow for an encoded first sub-frame, at least one fifth intra flow for an encoded second sub-frame, and at least one sixth intra flow for an encoded third sub-frame to encode the fourth sub-frame, and may encode the fourth sub-frame using the at least one fourth intra flow, the encoded first sub-frame, the at least one fifth intra flow, the encoded second sub-frame, the at least one sixth intra flow, and the encoded third sub-frame, and may encode the current frame using the encoded first sub-frame, the encoded second sub-frame, the encoded third sub-frame, and the encoded fourth sub-frame.

[0324] At step S2060, the video encoding device (2100) can transmit a bitstream including feature data for the current frame.

[0325] FIG. 21 is a diagram illustrating a configuration of an image encoding device according to one embodiment of the present disclosure.

[0326] Referring to FIG. 21, the image encoding device (2100) may include a first AI processing unit (2110), a second AI processing unit (2120), an intra flow / gate acquisition unit (2130), an intra compensation unit (2140), a decoding unit (2150), and an encoding unit (2160).

[0327] The first AI processing unit (2110), the second AI processing unit (2120), the intra flow / gate acquisition unit (2130), the intra compensation unit (2140), the decoding unit (2150), and the encoding unit (2160) may be implemented as a processor. The processor may include at least one processing circuit and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to individually and / or collectively perform the various functions described herein in a distributed fashion. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor can perform all of the functions. Additionally, at least one processor may comprise a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions. The receiving unit (1910), the AI ​​processing unit (1920), the intraflow / gate acquisition unit (1930), the intracompensation unit (1940), and the decryption unit (1950) may operate according to instructions stored in a memory (not shown).

[0328] Although FIG. 21 individually illustrates the first AI processing unit (2110), the second AI processing unit (2120), the intra flow / gate acquisition unit (2130), the intra compensation unit (2140), the decoding unit (2150), and the encoding unit (2160), the first AI processing unit (2110), the second AI processing unit (2120), the intra flow / gate acquisition unit (2130), the intra compensation unit (2140), the decoding unit (2150), and the encoding unit (2160) may be implemented through one processor. In this case, the first AI processing unit (2110), the second AI processing unit (2120), the intra flow / gate acquisition unit (2130), the intra compensation unit (2140), the decoding unit (2150), and the encoding unit (2160) may be implemented as a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU) and software. In addition, in the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or a memory processing unit for utilizing an external memory.

[0329] The first AI processing unit (2110), the second AI processing unit (2120), the intra flow / gate acquisition unit (2130), the intra compensation unit (2140), the decoding unit (2150), and the encoding unit (2160) may be configured with multiple processors. In this case, the first AI processing unit (2110), the second AI processing unit (2120), the intra flow / gate acquisition unit (2130), the intra compensation unit (2140), the decoding unit (2150), and the encoding unit (2160) may be implemented by a combination of dedicated processors, or may be implemented by a combination of multiple general-purpose processors such as an AP, a CPU, or a GPU, and software.

[0330] The first AI processing unit (2110) can take the current frame as input and output feature data for the current frame.

[0331] Feature data for the current frame can be transmitted to the second AI processing unit (2120) and the intraflow / gate acquisition unit (2130).

[0332] The second AI processing unit (2120) can encode the first subframe among a plurality of subframes according to a predetermined decoding order.

[0333] The intra flow / gate acquisition unit (2130) can output the intra flow and intra gate for the first sub-frame of the second sub-frame.

[0334] The intra compensation unit (2140) can receive the intra flow and intra gate for the first sub-frame of the second sub-frame from the intra flow / gate acquisition unit (2130), and can receive the first sub-frame from the AI ​​processing unit (2120).

[0335] The intra compensation unit (2140) can obtain a second sub prediction frame based on the intra flow and intra gate for the first sub frame of the second sub frame and the first sub frame.

[0336] The intra compensation unit (2140) can transmit the second sub-prediction frame to the second AI processing unit (2120).

[0337] The second AI processing unit (2120) can encode the second sub-frame based on feature data for the current frame and the second sub-prediction frame.

[0338] According to one embodiment of the present disclosure, the second AI processing unit (2120) may include an intra flow / gate acquisition unit (2130) and an intra compensation unit (2140). Accordingly, the second AI processing unit (2120) may decode a first subframe among a plurality of subframes according to a predefined decoding order, output an intra flow and an intra gate for the first subframe of the second subframe, acquire a second sub-prediction frame based on the intra flow and the intra gate for the first subframe of the second subframe and the first subframe, and decode the second subframe based on feature data for the current frame and the second sub-prediction frame.

[0339] The decryption unit (2150) can obtain a restored frame of the current frame based on the first sub-frame and the second sub-frame.

[0340] According to one embodiment of the present disclosure, when a plurality of sub-frames include a third sub-frame, the intra-flow / gate acquisition unit (2130) acquires at least one second intra-flow for the first sub-frame and at least one third intra-flow for the second sub-frame to decode the third sub-frame, the intra-compensation unit (2140) acquires a third sub-prediction frame based on the at least one second intra-flow, the first sub-frame, the at least one third intra-flow, and the second sub-frame, and transfers the third sub-prediction frame to the second AI processing unit (2120), the second AI processing unit (2120) decodes the third sub-frame based on feature data for the current frame and the third sub-prediction frame, and the decoding unit (2150) acquires a restored frame of the current frame based on the first sub-frame, the second sub-frame, and the third sub-frame.

[0341] According to one embodiment of the present disclosure, when a plurality of sub-frames include a fourth sub-frame, the intra-flow / gate acquisition unit (2130) acquires at least one fourth intra-flow for the first sub-frame, at least one fifth intra-flow for the second sub-frame, and at least one sixth intra-flow for the third sub-frame to decode the fourth sub-frame, and the intra-compensation unit (2140) acquires at least one fourth intra-flow, the first sub-frame, the at least one fifth intra-flow, and the second sub-frame. At least one sixth intra flow, a fourth sub-prediction frame is obtained based on the third sub-frame, and the fourth sub-prediction frame is transmitted to the second AI processing unit (2120), the second AI processing unit (2120) decodes the fourth sub-frame based on feature data for the current frame and the fourth sub-prediction frame, and the decoding unit (2150) can obtain a restored frame of the current frame based on the first sub-frame, the second sub-frame, the third sub-frame, and the fourth sub-frame.

[0342] The encoding unit (2160) can generate a bitstream including data for the current frame based on the restored frame.

[0343] The first AI processing unit (2110), the second AI processing unit (2120), and the intraflow / gate acquisition unit (2130) can be implemented as a neural network including one or more layers (e.g., convolutional layers).

[0344] The first AI processing unit (2110), the second AI processing unit (2120), and the intraflow / gate acquisition unit (2130) may be stored in memory. The first AI processing unit (2110), the second AI processing unit (2120), and the intraflow / gate acquisition unit (2130) may be implemented as at least one dedicated processor for AI.

[0345] A video decoding method according to one embodiment of the present disclosure may include: obtaining feature data for a current frame from a bitstream; decoding a first subframe among a plurality of subframes in which the current frame is rearranged according to a predetermined decoding order based on the feature data for the current frame; obtaining at least one intraflow for the decoded first subframe to decode a second subframe among the plurality of subframes; decoding the second subframe using the at least one intraflow and the decoded first subframe; and reconstructing the current frame using the decoded first subframe and the decoded second subframe.

[0346] A video decoding method according to one embodiment of the present disclosure can effectively restore an image by dividing a current frame into sub-frames, gradually decoding each sub-frame, using the decoded sub-frame as a reference sub-frame, and utilizing intraflow between the sub-frames.

[0347] According to one embodiment of the present disclosure, the plurality of sub-frames may be rearranged such that the current frame is divided into a plurality of pixel groups and pixels at the same position within the plurality of pixel groups are included in the same sub-frame.

[0348] A video decoding method according to one embodiment of the present disclosure can effectively restore an image by dividing a current frame into a plurality of pixel groups, rearranging pixels at the same position within the plurality of pixel groups, using a decoded subframe among the rearranged subframes as a reference subframe, and utilizing intraflow between the subframes.

[0349] According to one embodiment of the present disclosure, at least one intra flow for a decoded first sub-frame may represent a reference pixel within the decoded first sub-frame for a first pixel within a second sub-frame.

[0350] A video decoding method according to one embodiment of the present disclosure can effectively restore an image by utilizing an intra flow between a plurality of sub-frames in which a current frame is rearranged.

[0351] A video decoding method according to one embodiment of the present disclosure further includes a step of obtaining an intra gate value for at least one intra flow, wherein a second subframe is decoded using the at least one intra flow, the intra gate value, and the decoded first subframe, and the intra gate value may represent a weight for the at least one intra flow.

[0352] An image decoding method according to one embodiment of the present disclosure can effectively restore an image by using an intra flow between a plurality of sub-frames in which a current frame is rearranged and an intra gate value indicating a degree of confidence in the intra flow.

[0353] According to one embodiment of the present disclosure, if at least one intra flow is a plurality of intra flows, an intra gate value corresponding to one of the plurality of intra flows is obtained, and a second subframe can be decoded using one of the plurality of intra flows, the intra gate value corresponding to one of the plurality of intra flows, and the decrypted first subframe.

[0354] A video decoding method according to one embodiment of the present disclosure can effectively restore an image by using a plurality of intra flows between a plurality of sub-frames in which a current frame is rearranged and a plurality of intra gate values ​​indicating a degree of confidence for the plurality of intra flows.

[0355] According to one embodiment of the present disclosure, the step of decoding a second sub-frame using at least one intra-flow and the decoded first sub-frame may further include: obtaining a second sub-residual frame for the second sub-frame; obtaining a second sub-prediction frame using at least one intra-flow and the decoded first sub-frame; and decoding the second sub-frame using the second sub-residual frame and the second sub-prediction frame.

[0356] A video decoding method according to one embodiment of the present disclosure can effectively restore an image by obtaining a sub-prediction frame using an intra flow between sub-frames and adding a sub-prediction frame and a sub-residual frame.

[0357] A video decoding method according to one embodiment of the present disclosure may further include, when a plurality of sub-frames include a third sub-frame, obtaining at least one second intra flow for a decoded first sub-frame and at least one third intra flow for the decoded second sub-frame to decode the third sub-frame; decoding the third sub-frame using the at least one second intra flow, the decoded first sub-frame, the at least one third intra flow, and the decoded second sub-frame; and reconstructing a current frame using the decoded first sub-frame, the decoded second sub-frame, and the decoded third sub-frame.

[0358] A video decoding method according to one embodiment of the present disclosure may further include, when a plurality of sub-frames include a fourth sub-frame, obtaining at least one fourth intra flow for a decoded first sub-frame, at least one fifth intra flow for a decoded second sub-frame, and at least one sixth intra flow for a decoded third sub-frame to decode the fourth sub-frame; decoding the fourth sub-frame using the at least one fourth intra flow, the decoded first sub-frame, the at least one fifth intra flow, the decoded second sub-frame, the at least one sixth intra flow, and the decoded third sub-frame; and reconstructing a current frame using the decoded first sub-frame, the decoded second sub-frame, the decoded third sub-frame, and the decoded fourth sub-frame.

[0359] A video decoding method according to one embodiment of the present disclosure can effectively restore an image by dividing a current frame into sub-frames, gradually decoding each sub-frame, using the decoded sub-frame as a reference sub-frame, and utilizing intraflow between the sub-frames.

[0360] An image decoding device according to one embodiment of the present disclosure comprises: a memory storing one or more instructions; and at least one processor operating according to the one or more instructions, wherein the at least one processor obtains feature data for a current frame from a bitstream, decodes a first subframe among a plurality of subframes in which the current frame is rearranged according to a predetermined decoding order based on the feature data for the current frame, obtains at least one intraflow for the decoded first subframe to decode a second subframe among the plurality of subframes, decodes the second subframe using the at least one intraflow and the decoded first subframe, and reconstructs the current frame using the decoded first subframe and the decoded second subframe.

[0361] An image decoding device according to one embodiment of the present disclosure can effectively restore an image by dividing a current frame into sub-frames, gradually decoding each sub-frame, using the decoded sub-frame as a reference sub-frame, and utilizing intraflow between the sub-frames.

[0362] According to one embodiment of the present disclosure, the plurality of sub-frames may be rearranged such that the current frame is divided into a plurality of pixel groups and pixels at the same position within the plurality of pixel groups are included in the same sub-frame.

[0363] An image decoding device according to one embodiment of the present disclosure can effectively restore an image by dividing a current frame into a plurality of pixel groups, rearranging pixels at the same position within the plurality of pixel groups, and using a decoded subframe among the rearranged subframes as a reference subframe, and utilizing an intraflow between the subframes.

[0364] According to one embodiment of the present disclosure, at least one intra flow for a decoded first sub-frame may represent a reference pixel within the decoded first sub-frame for a first pixel within a second sub-frame.

[0365] An image decoding device according to one embodiment of the present disclosure can effectively restore an image by utilizing an intra flow between a plurality of sub-frames in which a current frame is rearranged.

[0366] According to one embodiment of the present disclosure, at least one processor is configured to obtain an intra gate value for at least one intra flow, a second subframe is decoded using the at least one intra flow, the intra gate value, and the decoded first subframe, wherein the intra gate value may represent a weight for the at least one intra flow.

[0367] An image decoding device according to one embodiment of the present disclosure can effectively restore an image by using an intra flow between a plurality of sub-frames in which a current frame is rearranged and an intra gate value indicating a degree of confidence in the intra flow.

[0368] According to one embodiment of the present disclosure, if at least one intra flow is a plurality of intra flows, an intra gate value corresponding to one of the plurality of intra flows is obtained, and a second subframe can be decoded using one of the plurality of intra flows, the intra gate value corresponding to one of the plurality of intra flows, and the decrypted first subframe.

[0369] An image decoding device according to one embodiment of the present disclosure can effectively restore an image by using a plurality of intra flows between a plurality of sub-frames in which a current frame is rearranged and a plurality of intra gate values ​​indicating a degree of confidence for the plurality of intra flows.

[0370] According to one embodiment of the present disclosure, at least one processor may obtain a second sub-residual frame for a second sub-frame, obtain a second sub-prediction frame using at least one intra flow and the decoded first sub-frame, and decode the second sub-frame using the second sub-residual frame and the second sub-prediction frame.

[0371] An image decoding device according to one embodiment of the present disclosure can obtain a sub-prediction frame by using an intra flow between sub-frames, and effectively restore an image by adding a sub-prediction frame and a sub-residual frame.

[0372] According to one embodiment of the present disclosure, when the plurality of sub-frames include a third sub-frame, at least one processor may obtain at least one second intra flow for a decoded first sub-frame and at least one third intra flow for the decoded second sub-frame to decode the third sub-frame, decode the third sub-frame using the at least one second intra flow, the decoded first sub-frame, the at least one third intra flow, and the decoded second sub-frame, and reconstruct the current frame using the decoded first sub-frame, the decoded second sub-frame, and the decoded third sub-frame.

[0373] According to one embodiment of the present disclosure, when a plurality of sub-frames include a fourth sub-frame, at least one processor may obtain at least one fourth intra flow for a decoded first sub-frame, at least one fifth intra flow for a decoded second sub-frame, and at least one sixth intra flow for a decoded third sub-frame to decode the fourth sub-frame, and may decode the fourth sub-frame using the at least one fourth intra flow, the decoded first sub-frame, the at least one fifth intra flow, the decoded second sub-frame, the at least one sixth intra flow, and the decoded third sub-frame, and may reconstruct the current frame using the decoded first sub-frame, the decoded second sub-frame, the decoded third sub-frame, and the decoded fourth sub-frame.

[0374] An image decoding device according to one embodiment of the present disclosure can effectively restore an image by dividing a current frame into sub-frames, gradually decoding each sub-frame, using the decoded sub-frame as a reference sub-frame, and utilizing intraflow between the sub-frames.

[0375] A video encoding method according to one embodiment of the present disclosure may include: generating feature data for a current frame; encoding a first subframe among a plurality of subframes in which the current frame is rearranged according to a predetermined encoding order based on the feature data for the current frame; obtaining at least one intraflow for an encoded first subframe to encode a second subframe among the plurality of subframes; encoding the second subframe using the at least one intraflow and the encoded first subframe; encoding the current frame using the encoded first subframe and the encoded second subframe; and transmitting a bitstream including the feature data for the current frame.

[0376] A video encoding method according to one embodiment of the present disclosure can effectively encode a video by dividing a current frame into sub-frames, gradually encoding each sub-frame, using the encoded sub-frame as a reference sub-frame, and utilizing intraflow between the sub-frames.

[0377] According to one embodiment of the present disclosure, the plurality of sub-frames may be rearranged such that the current frame is divided into a plurality of pixel groups and pixels at the same position within the plurality of pixel groups are included in the same sub-frame.

[0378] A video encoding method according to one embodiment of the present disclosure can effectively encode a video by dividing a current frame into a plurality of pixel groups, rearranging pixels at the same position within the plurality of pixel groups, using an encoded subframe among the rearranged subframes as a reference subframe, and utilizing intraflow between the subframes.

[0379] According to one embodiment of the present disclosure, at least one intra flow for an encoded first subframe may represent a reference pixel within the encoded first subframe for a first pixel within a second subframe.

[0380] A video encoding method according to one embodiment of the present disclosure can effectively encode a video by utilizing an intra flow between a plurality of sub-frames in which a current frame is rearranged.

[0381] A video encoding method according to one embodiment of the present disclosure further includes a step of obtaining an intra gate value for at least one intra flow, wherein a second subframe is encoded using the at least one intra flow, the intra gate value, and the encoded first subframe, and the intra gate value may represent a weight for the at least one intra flow.

[0382] A video encoding method according to one embodiment of the present disclosure can effectively encode a video by using an intra flow between a plurality of sub-frames that rearrange a current frame and an intra gate value indicating a degree of confidence in the intra flow.

[0383] According to one embodiment of the present disclosure, if at least one intra flow is a plurality of intra flows, an intra gate value corresponding to one of the plurality of intra flows is obtained, and a second subframe can be encoded using one of the plurality of intra flows, the intra gate value corresponding to one of the plurality of intra flows, and the encoded first subframe.

[0384] A video encoding method according to one embodiment of the present disclosure can effectively encode a video by using a plurality of intra flows between a plurality of sub-frames in which a current frame is rearranged and a plurality of intra gate values ​​indicating a degree of confidence for the plurality of intra flows.

[0385] According to one embodiment of the present disclosure, the step of encoding a second subframe using at least one intraflow and the encoded first subframe may further include: obtaining a second sub-residual frame for the second subframe; obtaining a second sub-prediction frame using at least one intraflow and the encoded first subframe; and encoding the second subframe using the second sub-residual frame and the second sub-prediction frame.

[0386] A video encoding method according to one embodiment of the present disclosure can effectively encode a video by obtaining a sub-prediction frame using an intra flow between sub-frames and adding a sub-prediction frame and a sub-residual frame.

[0387] A video encoding method according to one embodiment of the present disclosure may further include, when a plurality of sub-frames include a third sub-frame, obtaining at least one second intra flow for an encoded first sub-frame and at least one third intra flow for an encoded second sub-frame to encode the third sub-frame; encoding the third sub-frame using the at least one second intra flow, the encoded first sub-frame, the at least one third intra flow, and the encoded second sub-frame; and encoding the current frame using the encoded first sub-frame, the encoded second sub-frame, and the encoded third sub-frame.

[0388] A video encoding method according to an embodiment of the present disclosure may further include, when a plurality of sub-frames include a fourth sub-frame, obtaining at least one fourth intraflow for an encoded first sub-frame, at least one fifth intraflow for an encoded second sub-frame, and at least one sixth intraflow for an encoded third sub-frame to encode the fourth sub-frame; encoding the fourth sub-frame using the at least one fourth intraflow, the encoded first sub-frame, the at least one fifth intraflow, the encoded second sub-frame, the at least one sixth intraflow, and the encoded third sub-frame; and encoding a current frame using the encoded first sub-frame, the encoded second sub-frame, the encoded third sub-frame, and the encoded fourth sub-frame.

[0389] A video encoding method according to one embodiment of the present disclosure can effectively encode a video by dividing a current frame into sub-frames, gradually encoding each sub-frame, using the encoded sub-frame as a reference sub-frame, and utilizing intraflow between the sub-frames.

[0390] An image encoding device according to one embodiment of the present disclosure comprises: a memory storing one or more instructions; and at least one processor operating according to the one or more instructions, wherein the at least one processor generates feature data for a current frame, encodes a first subframe among a plurality of subframes in which the current frame is rearranged according to a predetermined encoding order based on the feature data for the current frame, obtains at least one intraflow for the encoded first subframe to encode a second subframe among the plurality of subframes, encodes the second subframe using the at least one intraflow and the encoded first subframe, encodes the current frame using the encoded first subframe and the encoded second subframe, and transmits a bitstream including the feature data for the current frame.

[0391] An image encoding device according to one embodiment of the present disclosure can effectively encode an image by dividing a current frame into sub-frames, gradually encoding each sub-frame, using the encoded sub-frame as a reference sub-frame, and utilizing intraflow between the sub-frames.

[0392] According to one embodiment of the present disclosure, the plurality of sub-frames may be rearranged such that the current frame is divided into a plurality of pixel groups and pixels at the same position within the plurality of pixel groups are included in the same sub-frame.

[0393] An image encoding device according to one embodiment of the present disclosure can effectively encode an image by dividing a current frame into a plurality of pixel groups, rearranging pixels at the same position within the plurality of pixel groups, using an encoded subframe among the rearranged subframes as a reference subframe, and utilizing intraflow between the subframes.

[0394] According to one embodiment of the present disclosure, at least one intra flow for an encoded first subframe may represent a reference pixel within the encoded first subframe for a first pixel within a second subframe.

[0395] An image encoding device according to one embodiment of the present disclosure can effectively encode an image by utilizing an intra flow between a plurality of sub-frames that rearrange a current frame.

[0396] According to one embodiment of the present disclosure, at least one processor obtains an intra gate value for at least one intra flow, a second subframe is encoded using the at least one intra flow, the intra gate value, and the encoded first subframe, and the intra gate value may represent a weight for the at least one intra flow.

[0397] An image encoding device according to one embodiment of the present disclosure can effectively encode an image by using an intra flow between a plurality of sub-frames that rearrange a current frame and an intra gate value indicating a degree of confidence in the intra flow.

[0398] According to one embodiment of the present disclosure, if at least one intra flow is a plurality of intra flows, an intra gate value corresponding to one of the plurality of intra flows is obtained, and a second subframe can be encoded using one of the plurality of intra flows, the intra gate value corresponding to one of the plurality of intra flows, and the encoded first subframe.

[0399] An image encoding device according to one embodiment of the present disclosure can effectively encode an image by using a plurality of intra flows between a plurality of sub-frames in which a current frame is rearranged and a plurality of intra gate values ​​indicating a degree of confidence for the plurality of intra flows.

[0400] According to one embodiment of the present disclosure, at least one processor may obtain a second sub-residual frame for a second sub-frame, obtain a second sub-prediction frame using at least one intraflow and the encoded first sub-frame, and encode the second sub-frame using the second sub-residual frame and the second sub-prediction frame.

[0401] An image encoding device according to one embodiment of the present disclosure can effectively encode an image by obtaining a sub-prediction frame using an intra flow between sub-frames and adding a sub-prediction frame and a sub-residual frame.

[0402] According to one embodiment of the present disclosure, at least one processor may, when a plurality of sub-frames include a third sub-frame, obtain at least one second intra flow for an encoded first sub-frame and at least one third intra flow for an encoded second sub-frame to encode the third sub-frame, encode the third sub-frame using the at least one second intra flow, the encoded first sub-frame, the at least one third intra flow, and the encoded second sub-frame, and encode the current frame using the encoded first sub-frame, the encoded second sub-frame, and the encoded third sub-frame.

[0403] According to one embodiment of the present disclosure, at least one processor may obtain at least one fourth intraflow for an encoded first subframe, at least one fifth intraflow for an encoded second subframe, and at least one sixth intraflow for an encoded third subframe, to encode the fourth subframe when the plurality of subframes include a fourth subframe; encode the fourth subframe using the at least one fourth intraflow, the encoded first subframe, the at least one fifth intraflow, the encoded second subframe, the at least one sixth intraflow, and the encoded third subframe, and encode the current frame using the encoded first subframe, the encoded second subframe, the encoded third subframe, and the encoded fourth subframe.

[0404] An image encoding device according to one embodiment of the present disclosure can effectively encode an image by dividing a current frame into sub-frames, gradually encoding each sub-frame, using the encoded sub-frame as a reference sub-frame, and utilizing intraflow between the sub-frames.

[0405] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0406] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. A step of obtaining feature data for the current frame from a bitstream (S1810); A step of decoding (S1820) a first sub-frame among a plurality of sub-frames in which the current frame is rearranged according to a predetermined decoding order based on feature data for the current frame; A step of obtaining at least one intra flow for the decrypted first sub-frame to decrypt a second sub-frame among the plurality of sub-frames (S1830); A step of decoding the second sub-frame using at least one intra flow and the decoded first sub-frame (S1840); An image decoding method comprising: a step of restoring a current frame (S1850) using the decrypted first sub-frame and the decrypted second sub-frame.

2. In paragraph 1, A method for decoding an image, wherein the plurality of sub-frames are rearranged such that the current frame is divided into a plurality of pixel groups and pixels at the same position within the plurality of pixel groups are included in the same sub-frame.

3. In paragraph 1 or 2, A method for decoding a video signal, wherein at least one intra flow for the decoded first sub-frame indicates a reference pixel within the decoded first sub-frame for a first pixel within the second sub-frame.

4. In any one of paragraphs 1 to 3, Further comprising a step of obtaining an intra gate value for at least one intra flow, The second sub-frame is decoded using the at least one intra flow, the intra gate value, and the decoded first sub-frame, A method for decoding an image, wherein the intra gate value represents a weight for at least one intra flow.

5. In any one of paragraphs 1 to 4, If at least one of the above intra flows is a plurality of intra flows, Obtain an intra gate value corresponding to one of the above multiple intra flows, A method for decoding a video, wherein the second sub-frame is decoded using one of the plurality of intra flows, an intra gate value corresponding to one of the plurality of intra flows, and the decoded first sub-frame.

6. In any one of paragraphs 1 to 5, The step of decoding the second subframe using the at least one intra flow and the decoded first subframe comprises: A step of obtaining a second sub-residual frame for the second sub-frame; and A step of obtaining the second sub-prediction frame using the at least one intra flow and the decoded first sub-frame; and A method for decoding an image, further comprising the step of decoding the second sub-frame using the second sub-residual frame and the second sub-prediction frame.

7. In any one of paragraphs 1 to 6, If the above multiple sub-frames include a third sub-frame, To decrypt the third sub-frame, a step of obtaining at least one second intra flow for the decrypted first sub-frame and at least one third intra flow for the decrypted second sub-frame; A step of decoding the third sub-frame using the at least one second intra flow, the decrypted first sub-frame, the at least one third intra flow, and the decrypted second sub-frame; and A video decoding method further comprising the step of restoring the current frame using the decoded first sub-frame, the decoded second sub-frame, and the decoded third sub-frame.

8. In paragraph 7, If the above multiple sub-frames include a fourth sub-frame, To decrypt the fourth sub-frame, a step of obtaining at least one fourth intra flow for the decrypted first sub-frame, at least one fifth intra flow for the decrypted second sub-frame, and at least one sixth intra flow for the decrypted third sub-frame; A step of decoding the fourth subframe using the at least one fourth intra flow, the decoded first subframe, the at least one fifth intra flow, the decoded second subframe, the at least one sixth intra flow, and the decoded third subframe; and A video decoding method further comprising the step of restoring the current frame using the decoded first sub-frame, the decoded second sub-frame, the decoded third sub-frame, and the decoded fourth sub-frame.

9. Step of generating feature data for the current frame (S2010); A step of encoding (S2020) a first subframe among a plurality of subframes in which the current frame is rearranged according to a predetermined encoding order based on feature data for the current frame; A step of obtaining at least one intra flow for the encoded first subframe to encode a second subframe among the plurality of subframes (S2030); A step of encoding (S2040) the second sub-frame using at least one intra flow and the encoded first sub-frame; A step of encoding the current frame (S2050) using the encoded first sub-frame and the encoded second sub-frame; and A method for encoding an image, comprising: a step of transmitting (S2060) a bitstream including feature data for the current frame.

10. In paragraph 9, A video encoding method, wherein the plurality of sub-frames are rearranged such that the current frame is divided into a plurality of pixel groups and pixels at the same position within the plurality of pixel groups are included in the same sub-frame.

11. In clause 9 or 10, A method for encoding a video, wherein at least one intra flow for the encoded first sub-frame represents a reference pixel within the encoded first sub-frame for a first pixel within the second sub-frame.

12. In any one of paragraphs 9 to 11, Further comprising a step of obtaining an intra gate value for at least one intra flow, The second sub-frame is encoded using the at least one intra flow, the intra gate value, and the encoded first sub-frame, A method for encoding an image, wherein the intra gate value represents a weight for at least one intra flow.

13. In any one of paragraphs 9 to 12, If at least one of the above intra flows is a plurality of intra flows, Obtain an intra gate value corresponding to one of the above multiple intra flows, A video encoding method, wherein the second sub-frame is encoded using one of the plurality of intra flows, an intra gate value corresponding to one of the plurality of intra flows, and the encoded first sub-frame.

14. In any one of paragraphs 9 to 13, If the above multiple sub-frames include a third sub-frame, To encode the third sub-frame, a step of obtaining at least one second intra flow for the encoded first sub-frame and at least one third intra flow for the encoded second sub-frame; A step of encoding the third subframe using the at least one second intra flow, the encoded first subframe, the at least one third intra flow, and the encoded second subframe; and A video encoding method further comprising the step of encoding the current frame using the encoded first sub-frame, the encoded second sub-frame, and the encoded third sub-frame.

15. In paragraph 14, If the above multiple sub-frames include a fourth sub-frame, To encode the fourth sub-frame, a step of obtaining at least one fourth intra flow for the encoded first sub-frame, at least one fifth intra flow for the encoded second sub-frame, and at least one sixth intra flow for the encoded third sub-frame; A step of encoding the fourth subframe using the at least one fourth intraflow, the encoded first subframe, the at least one fifth intraflow, the encoded second subframe, the at least one sixth intraflow, and the encoded third subframe; and A video encoding method further comprising the step of encoding the current frame using the encoded first sub-frame, the encoded second sub-frame, the encoded third sub-frame, and the encoded fourth sub-frame.

Images