Image decoding apparatus and method, and image encoding apparatus and method

Abstract

An image decoding method and apparatus are proposed. The method includes: obtaining feature data of a current frame from a bitstream; decoding a first subframe of a plurality of subframes obtained by rearranging the current frame according to a predetermined decoding order based on the feature data of the current frame; obtaining at least one intra-frame stream for the decoded first subframe to decode a second subframe of the plurality of subframes; decoding the second subframe using the at least one intra-frame stream and the decoded first subframe; and reconstructing the current frame using the decoded first subframe and the decoded second subframe.

Description

Technical Field

[0001] This disclosure relates to an image decoding method and apparatus, as well as an image encoding method and apparatus, and more specifically, to a method and apparatus for performing intra-frame prediction more efficiently. Background Technology

[0002] In codecs such as H.264 Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC), images can be divided into blocks, and each block can be predictively encoded and predictedly decoded via inter-frame prediction or intra-frame prediction.

[0003] Intra-frame prediction is a method of compressing images by removing spatial redundancy, while inter-frame prediction is a method of compressing images by removing temporal redundancy between images.

[0004] Intra-prediction is performed by predicting all pixels within a specific block in an intra-prediction direction based on an intra-prediction mode. Furthermore, interpolation of reference pixels is performed using fixed filter coefficients of a predetermined filter.

[0005] Recently, techniques for encoding / decoding images using artificial intelligence (AI) have been proposed, and there is a need for a method to efficiently encode / decode images by performing intra-frame prediction using AI (e.g., neural networks). Summary of the Invention

[0006] Solution to the problem An image decoding method according to an embodiment of the present disclosure may include: obtaining feature data of a current frame from a bitstream; decoding a first subframe of a plurality of subframes rearranged from the current frame according to a predetermined decoding order based on the feature data of the current frame; obtaining at least one intra-frame stream for the decoded first subframe to decode a second subframe of the plurality of subframes; decoding the second subframe using the at least one intra-frame stream and the decoded first subframe; and reconstructing the current frame using the decoded first subframe and the decoded second subframe.

[0007] An image decoding apparatus according to embodiments of the present disclosure may include: a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may obtain feature data of a current frame from a bitstream. The at least one processor may decode a first subframe of a plurality of subframes rearranged from the current frame according to a predetermined decoding order based on the feature data of the current frame. The at least one processor may obtain at least one intra-frame stream for the decoded first subframe to decode a second subframe of the plurality of subframes. The at least one processor may decode the second subframe using the at least one intra-frame stream and the decoded first subframe. The at least one processor may reconstruct the current frame using the decoded first subframe and the decoded second subframe.

[0008] An image encoding method according to an embodiment of the present disclosure may include: generating feature data of a current frame; encoding a first subframe of a plurality of subframes rearranged from the current frame according to a predetermined encoding order based on the feature data of the current frame; obtaining at least one intra-frame stream for the encoded first subframe to encode a second subframe of the plurality of subframes; encoding the second subframe using the at least one intra-frame stream and the encoded first subframe; encoding the current frame using the encoded first subframe and the encoded second subframe; and transmitting a bit stream including the feature data of the current frame.

[0009] An image encoding apparatus according to embodiments of the present disclosure may include: a memory storing one or more instructions; and at least one processor operating according to the one or more instructions. The at least one processor may generate feature data for a current frame. The at least one processor may encode a first subframe of a plurality of subframes rearranged from the current frame according to a predetermined encoding order based on the feature data of the current frame. The at least one processor may obtain at least one intra-frame stream for the encoded first subframe in order to encode a second subframe of the plurality of subframes. The at least one processor may encode the second subframe using the at least one intra-frame stream and the encoded first subframe. The at least one processor may encode the current frame using the encoded first subframe and the encoded second subframe. The at least one processor may transmit a bitstream including the feature data of the current frame. Attached Figure Description

[0010] Figure 1 This diagram illustrates the encoding and decoding processes for an image.

[0011] Figure 2 This is a diagram showing the blocks divided from the image according to the tree structure.

[0012] Figure 3 This is a diagram illustrating an intra-frame stream according to an embodiment of the present disclosure.

[0013] Figure 4 This is a diagram illustrating a method for reconstructing a frame by using an intra-frame stream to decompose a frame into multiple subframes according to embodiments of the present disclosure.

[0014] Figure 5 It is a diagram used to describe the intra-frame flow between multiple subframes according to embodiments of the present disclosure.

[0015] Figure 6 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network according to embodiments of the present disclosure.

[0016] Figure 7 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network according to embodiments of the present disclosure.

[0017] Figure 8 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0018] Figure 9 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0019] Figure 10 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0020] Figure 11 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0021] Figure 12 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0022] Figure 13 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0023] Figure 14 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0024] Figure 15 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0025] Figure 16 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0026] Figure 17 This is a diagram illustrating a method for training a neural network used in an image encoding method and an image decoding method according to embodiments of the present disclosure.

[0027] Figure 18 This is a flowchart illustrating an image decoding method according to an embodiment of the present disclosure.

[0028] Figure 19 This is a diagram illustrating the configuration of an image decoding apparatus according to an embodiment of the present disclosure.

[0029] Figure 20 This is a flowchart illustrating an image encoding method according to an embodiment of the present disclosure.

[0030] Figure 21 This is a diagram illustrating the configuration of an image encoding apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0031] Throughout this disclosure, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0032] Because this disclosure allows for various changes and numerous examples, specific embodiments will be shown in the accompanying drawings and described in detail in the written description. However, this is not intended to limit the embodiments of this disclosure, and it should be understood that this disclosure includes all modifications, equivalents, and alternatives contained within the spirit and technical scope of the various embodiments.

[0033] In describing embodiments, detailed descriptions of relevant known technologies will be omitted where it is determined that such descriptions may unnecessarily obscure the gist of this disclosure. Furthermore, the numbers used in the description (e.g., first and second) are merely identifier codes used to distinguish one component from another.

[0034] Furthermore, it will be understood in this disclosure that, unless otherwise stated, when components are “connected” or “coupled” to each other, the components may be directly connected or coupled to each other, but may optionally be connected or coupled to each other with an intermediate component in between.

[0035] Furthermore, in this disclosure, regarding components referred to as "...unit" or "module," two or more components may be combined into one element, or a component may be divided into two or more components according to subdivided functions. Additionally, besides their own primary functions, each component described below may additionally perform some or all of the functions performed by another component, and some of the primary functions of each component may be entirely performed by another component.

[0036] According to embodiments of this disclosure, "...unit" may include a processor and memory. The term "processor" should be broadly interpreted to encompass general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some cases, "processor" may refer to application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc. The term "processor" may refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other combination of such components.

[0037] A processor may include various processing circuitry and / or multiple processors. For example, the term "processor" as used in the specification including the claims may include various processing circuitry comprising at least one processor. In at least one processor, one or more processors may be configured to perform the various functions described herein individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms cover, but are not limited to, situations where one processor performs some functions while other processors perform other functions, and situations where a single processor can perform all functions. Furthermore, at least one processor may include a combination of processors with various functions for performing the disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions.

[0038] The term "memory" should be broadly interpreted to encompass any electronic component capable of storing electronic information. "Memory" can refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. Memory is said to be in electronic communication with the processor when the processor can read information from and / or write information to it. Memory integrated within the processor is in electronic communication with the processor.

[0039] Furthermore, in this disclosure, "image" or "picture" may refer to a still image (or frame), a moving image comprising multiple consecutive still images, or a video.

[0040] In this disclosure, "neural network (DNN)" is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to artificial neural network models using a specific algorithm. Neural networks can also be called deep neural networks.

[0041] In this disclosure, "parameter" refers to a value used in the operation of each layer of a neural network, and may be used, for example, when applying input values ​​to an operational expression. Parameters are values ​​set as a result of training and may be updated as needed using separate training data.

[0042] In this disclosure, "feature data" or "feature map" can refer to data obtained when a neural network or a neural network-based encoder processes input data. Feature data can be one-dimensional or two-dimensional data comprising a number of samples. Feature data can also be referred to as a latent tensor or latent representation. Feature data can represent latent features in the data output by a neural network-based decoder.

[0043] In this disclosure, a "sample point" is data assigned to a sampling location in one-dimensional or two-dimensional data (such as an image, block, or feature data), and refers to the data to be processed. For example, a sample point may include pixels in a two-dimensional image. Two-dimensional data may also be referred to as a "graph".

[0044] In addition, in this disclosure, "current frame" refers to the frame that is currently being processed.

[0045] Additionally, in this disclosure, "intra-frame stream" refers to a reference pixel indicated by each pixel in a frame.

[0046] Before describing the image decoding method, image decoding apparatus, image encoding method, and image encoding apparatus according to embodiments, reference will be made to... Figure 1 and Figure 2 Describes the process of encoding and decoding images.

[0047] Figure 1 This diagram illustrates the encoding and decoding processes for an image.

[0048] The encoding device 110 sends the bit stream generated by encoding the image to the decoding device 150, and the decoding device 150 reconstructs the image by receiving the bit stream and decoding the bit stream.

[0049] Specifically, in the encoding apparatus 110, the prediction encoder 115 outputs prediction blocks through inter-frame prediction and intra-frame prediction, and the transform and quantization unit 120 outputs quantized transform coefficients by transforming and quantizing the residual samples of the residual block between the prediction block and the current block. The entropy encoder 125 encodes the quantized transform coefficients and outputs a bit stream.

[0050] The quantized transform coefficients are reconstructed into a residual block including residual samples in the spatial domain by the inverse quantization and inverse transform unit 130. The reconstructed block obtained by combining the prediction block with the residual block is output as a filtered block by the deblocking filter unit 135 and the loop filter unit 140. The reconstructed image including the filtered block can be used as a reference image for the next input image in the predictive encoder 115.

[0051] The bitstream received by the decoding device 150 is reconstructed into residual blocks including residual samples in the spatial domain by the entropy decoder 155 and the inverse quantization and inverse transform unit 160. The reconstructed blocks are generated by combining the residual blocks and the prediction blocks output from the prediction decoder 175, and are output as filtered blocks by the deblocking filtering unit 165 and the loop filtering unit 170. The reconstructed image including the filtered blocks can be used as a reference image for the next image in the prediction decoder 175.

[0052] The loop filtering unit 140 of the encoding device 110 performs loop filtering by using filter information input according to user input or system settings. The filter information used by the loop filtering unit 140 is sent to the decoding device 150 via the entropy encoder 125. The loop filtering unit 170 of the decoding device 150 can perform loop filtering based on the filter information input from the entropy decoder 155.

[0053] In image encoding and decoding, the image is divided into layers, and encoding and decoding are performed on the blocks derived from the image. (Refer to...) Figure 2 Describes the blocks that are divided from the image.

[0054] Figure 2 This is a diagram showing the blocks divided from image 200 according to the tree structure.

[0055] An image 200 can be divided into one or more stripes or one or more parallel blocks. A strip may include multiple parallel blocks.

[0056] A stripe or a parallel block can be a sequence of one or more maximum coding units (CUs).

[0057] A maximum coding unit can be divided into one or more coding units. A coding unit can be a reference block used to determine the prediction mode. In other words, it can be determined whether an intra-frame prediction mode or an inter-frame prediction mode is applied to each coding unit. In this disclosure, the maximum coding unit may be referred to as the maximum coding block, and the coding unit may be referred to as a coding block.

[0058] The size of a coding unit can be equal to or smaller than the size of the maximum coding unit. Since the maximum coding unit is the coding unit with the largest size, it can be called a coding unit.

[0059] One or more prediction units for intra-frame or inter-frame prediction can be determined from the coding unit. The size of the prediction unit can be equal to or smaller than the size of the coding unit.

[0060] Furthermore, one or more transform units for transforming and quantizing can be determined from the coding unit. The size of the transform unit can be equal to or smaller than the size of the coding unit. The transform unit is a reference block for transforming and quantizing, and the residual samples of the coding unit can be transformed and quantized for each transform unit within the coding unit.

[0061] In this disclosure, the current block can be a stripe, a parallel block, a maximum coding unit, a coding unit, a prediction unit, or a transform unit divided from image 200. Furthermore, the lower-level block of the current block is a block divided from the current block, and for example, when the current block is a maximum coding unit, the lower-level block can be a coding unit, a prediction unit, or a transform unit. Additionally, the upper-level block of the current block is a block that includes the current block as a part, and for example, when the current block is a maximum coding unit, the upper-level block can be a frame sequence, a frame, a stripe, or a parallel block.

[0062] In the following text, reference will be made to Figures 3 to 2 3. A video decoding method, a video decoding apparatus, a video encoding method, and a video encoding apparatus according to embodiments are described.

[0063] Figure 3 This is a diagram illustrating an intra-frame stream according to an embodiment of the present disclosure.

[0064] Performing predictions for all pixels in a block according to an intra-prediction mode in one intra-prediction direction may not cover all video content. Each pixel in a block can have a different prediction direction indicating the best reference pixel for that pixel. When using this non-local similarity, image decoding quality can be improved.

[0065] Reference Figure 3 The pixels in the current block 320 of the current image 300 have an intra-frame stream 330, which indicates different prediction directions for reference pixels in the neighboring reference region 310.

[0066] The intra stream 330 can be represented as a vector indicating the difference between the x and y components between the current pixel and a reference pixel, a vector representing the distance r between the current pixel and a reference pixel and an angle θ between the current pixel and a reference pixel based on the x-axis, or an angle between the current pixel and a reference pixel. When the intra stream is represented solely by the angle between the current pixel and the reference pixel, the reference pixel that is first located at the boundary of the current block based on the angle is determined as the reference pixel.

[0067] Figure 4 This is a diagram illustrating a method for reconstructing a frame by using an intra-frame stream to decompose a frame into multiple subframes according to embodiments of the present disclosure.

[0068] Each pixel in a frame has a spatial flow that indicates neighboring pixels with similar pixel values. To use this spatial flow effectively, a frame can be divided into multiple subframes.

[0069] The current frame can be divided into M×N pixel groups or M×N decomposition blocks, and can be further decomposed into M×N subframes by rearranging them so that pixels at the same position in the M×N pixel groups are included in the same subframe. Therefore, the current frame of H×W can be decomposed into K (=M×N) h×w subframes. This can be called pixel de-washing or pixel extraction.

[0070] Reference Figure 4 The current frame 400 is divided into 2×2 pixel group units 401. A first subframe 410 is obtained by collecting the pixel A located on the top left side of each pixel group into a subframe. A second subframe 420 is obtained by collecting the pixel B located on the top right side of each pixel group into a subframe. A third subframe 430 is obtained by collecting the pixel C located on the bottom left side of each pixel group into a subframe. A fourth subframe 440 is obtained by collecting the pixel D located on the bottom right side of each pixel group into a subframe. Therefore, a frame with an 8×16 size is decomposed into four subframes with a 4×8 size.

[0071] This is an example, and it is not limited to this. That is, the current frame can be divided into two, six, eight, or nine subframes. For example, when the current frame is divided into 2×1 pixel groups, the first subframe can be obtained by collecting the upper pixels in each pixel group into a subframe, and the second subframe can be obtained by collecting the lower pixels in each pixel group into a subframe.

[0072] Furthermore, when the current frame is divided into 1×2 pixel group units, a first subframe can be obtained by collecting the left pixels in each pixel group into a subframe, and a second subframe can be obtained by collecting the right pixels in each pixel group into a subframe.

[0073] Furthermore, when the current frame is divided into 3×2 or 2×3 pixel groups, six subframes can be obtained by collecting pixels at the same position in each pixel group into a single subframe.

[0074] Furthermore, when the current frame is divided into 4×2 or 2×4 pixel groups, eight subframes can be obtained by collecting pixels at the same position in each pixel group into a single subframe.

[0075] Furthermore, when the current frame is divided into 3×3 pixel groups, nine subframes can be obtained by collecting pixels at the same position in each pixel group into a single subframe.

[0076] Furthermore, the current frame can be divided into blocks of size P×Q, and each P×Q block can be a subframe. In this method, the number of subframes is equal to the number of P×Q blocks. Here, P and Q are natural numbers.

[0077] Furthermore, the current frame can be either a feature map domain or a pixel domain. In other words, the current frame can be a feature map.

[0078] In the following text, reference will be made to Figure 5 Describes the intra-frame flow between multiple subframes.

[0079] Figure 5 It is a diagram used to describe the intra-frame flow between multiple subframes according to embodiments of the present disclosure.

[0080] A pixel in a subframe derived from the same frame may have multiple similar pixels in another subframe via multiple intrastreams or one intrastream.

[0081] Reference Figure 5 The first pixel 431 in the third subframe 430 may have two intra-streams (e.g., 500 and 510) for the first subframe 410. In the two intra-streams, the first intra-stream 500 may indicate a first reference pixel 411 in the first subframe 410, and the second intra-stream 510 may indicate a second reference pixel 412 in the first subframe 410. Furthermore, the first pixel 431 in the third subframe 430 may have two intra-streams (e.g., 520 and 530) for the second subframe 420. In the two intra-streams, the third intra-stream 520 may indicate a first reference pixel 421 in the second subframe 420, and the fourth intra-stream 530 may indicate a second reference pixel 422 in the second subframe 430.

[0082] When the number of intrastreams in which a pixel in an indicator subframe has a reference pixel in another subframe is N, the number of intrastream gates indicating the reliability of each intrastream can also be N. Here, N is a natural number. Furthermore, N can vary depending on the subframe.

[0083] Return to reference Figure 5 When decoding is performed in the order of first subframe 410, second subframe 420 and third subframe 430, at the time point when the third subframe 430 is decoded, the first subframe 410 and the second subframe 420 are used as reference subframes for the third subframe 430.

[0084] Specifically, refer to Figure 4 and Figure 5 When decoding is performed in the order of first subframe 410, second subframe 420, third subframe 430, and fourth subframe 440, no reference subframe exists when the first subframe 410 is decoded. Subsequently, when the second subframe 420 is decoded, the decoded first subframe 410 is used as the reference subframe. When the third subframe 430 is decoded, two subframes (i.e., the decoded first subframe 410 and the decoded second subframe 420) are used as reference subframes. Furthermore, when the fourth subframe 440 is decoded, three subframes (i.e., the decoded first subframe 410, the decoded second subframe 420, and the decoded third subframe 430) are used as reference subframes.

[0085] When a frame is divided into K subframes, the first to (k-1)th subframes are used as reference subframes for the kth subframe. When the number of subframes that can be used as reference subframes is J, J is equal to or less than K.

[0086] Figure 6 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network according to embodiments of the present disclosure.

[0087] Reference Figure 6 The current frame 600 is input into the encoding neural network 605 and output as feature data for the current frame 600. When the feature data of the current frame 600 is input into the decoding neural network 610, the first subframe 620 is decoded first according to a predetermined decoding order. The predetermined decoding order can be determined based on the learning or training results of the decoding neural network 610 for previous frames.

[0088] The first subframe 620, which is decoded, is additionally input into the decoding neural network 610. That is, based on the feature data of the current frame 600 and the first subframe 620, the second subframe 630 is decoded by the decoding neural network 610.

[0089] The decoded second subframe 630 is additionally input into the decoding neural network 610. That is, based on the feature data of the current frame 600 and the decoded second subframe 630, the third subframe 640 is decoded by the decoding neural network 610.

[0090] The decoded third subframe 640 is additionally input into the decoding neural network 610. That is, based on the feature data of the current frame 600 and the decoded third subframe 640, the fourth subframe 650 is decoded by the decoding neural network 610.

[0091] The current frame is reconstructed based on the first subframe 620, the second subframe 630, the third subframe 640, and the fourth subframe 650.

[0092] The decoding order can be any other order, such as the order of first subframe 620, second subframe 630, fourth subframe 650 and third subframe 640, or the order of first subframe 620, second subframe 630, third subframe 640 and fourth subframe 650.

[0093] Here, although the decoding neural network 610 is described as a neural network, this is an example, and a different set of neural networks can be used for each subframe.

[0094] Despite Figure 6 In the encoding neural network, the decoded subframes are not used, but in Figure 7 Decoded subframes can also be used in the encoded neural network.

[0095] Figure 7 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network according to embodiments of the present disclosure.

[0096] Reference Figure 7 The current frame 700 is input into the encoding neural network 705 and output as the first feature data of the current frame 700. When the first feature data of the current frame 700 is input into the decoding neural network 710, the first subframe 720 is decoded first according to a predetermined decoding order. The predetermined decoding order can be determined based on the learning or training results of the decoding neural network 710 for previous frames.

[0097] The decoded first subframe 720 is additionally input into the encoding neural network 705. That is, based on the current frame 700 and the decoded first subframe 720, the encoding neural network 705 outputs second feature data. Based on the second feature data and the decoded first subframe 720, the second subframe 730 is decoded by the decoding neural network 710.

[0098] The decoded second subframe 730 is additionally input into the encoding neural network 705. That is, based on the current frame 700 and the decoded second subframe 730, the encoding neural network 705 outputs third feature data. Based on the third feature data and the decoded second subframe 730, the third subframe 740 is decoded by the decoding neural network 710.

[0099] The decoded third subframe 740 is additionally input into the encoding neural network 705. That is, based on the current frame 700 and the decoded third subframe 740, the encoding neural network 705 outputs fourth feature data. Based on the fourth feature data and the decoded third subframe 740, the fourth subframe 750 is decoded by the decoding neural network 710.

[0100] The current frame is reconstructed based on the first subframe 620, the second subframe 630, the third subframe 640, and the fourth subframe 650.

[0101] The decoding order can be any other order, such as the order of first subframe 720, second subframe 730, fourth subframe 750 and third subframe 740, or the order of first subframe 720, second subframe 730, third subframe 740 and fourth subframe 750.

[0102] Here, although the decoding neural network 710 is described as a neural network, this is an example, and a set of different neural networks can be used for each subframe.

[0103] Figure 8 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0104] Reference Figure 8 The current frame 800 is input into the encoding neural network 805 and output as the feature data of the current frame 800. When the feature data of the current frame 800 is input into the decoding neural network 810, the first subframe 820 is decoded first according to a predetermined decoding order. The predetermined decoding order can be determined based on the learning or training results of the decoding neural network 810 for previous frames.

[0105] After the first subframe 820 is decoded, when the feature data of the current frame 800 is input into the stream / gated decoding neural network 815, the intra-stream and intra-gated parameters of the second subframe 830 relative to the first subframe are obtained. Intra-compensation 860 is performed using the decoded first subframe 820 and the intra-stream and intra-gated parameters of the second subframe 830 relative to the first subframe 820 to obtain the second subframe prediction frame. The second subframe 830 is decoded when the feature data of the current frame 800 and the second subframe prediction frame are input into the decoding neural network 810.

[0106] In this disclosure, "intra-frame compensation" may use "intra-frame stream", "intra-frame gating value" and "previously decoded subframe" or "previously decoded sub-feature map" to generate a predicted value for the current subframe or current sub-feature map to be decoded, wherein "intra-frame stream" is a stream that indicates the number of pixels or samples that may be similar to each pixel or sample location in the feature map at each pixel or sample location in the feature map, and "intra-frame gating value" is a gating value associated with the intra-frame stream and used to determine how reliable the intra-frame stream is.

[0107] Furthermore, in this disclosure, "sub-feature map" can refer to data obtained by processing data related to a "frame" through a neural network. Additionally, a "sub-feature map" can be obtained by dividing the samples of the "feature map" obtained through the neural network into multiple pixel groups and rearranging the samples at the same position in the multiple pixel groups to be included in the same sub-feature map.

[0108] After the second subframe 830 is decoded, when the feature data of the current frame 800 is input into the stream / gated decoding neural network 815, the intra-frame stream and intra-frame gating of the third subframe 840 relative to the first subframe 820, and the intra-frame stream and intra-frame gating of the third subframe 840 relative to the second subframe 830 are obtained. Intra-frame compensation 860 is performed using the decoded first subframe 820, the decoded second subframe 830, and the intra-frame stream and intra-frame gating of the third subframe 840 relative to the first subframe 820, and the intra-frame stream and intra-frame gating of the third subframe 840 relative to the second subframe 830 to obtain the third subframe prediction frame. The third subframe 840 is decoded when the feature data of the current frame 800 and the third subframe prediction frame are input into the decoding neural network 810.

[0109] After the third subframe 840 is decoded, when the feature data of the current frame 800 is input into the stream / gated decoding neural network 815, the intra-frame stream and intra-frame gating of the fourth subframe 850 relative to the first subframe 820, the intra-frame stream and intra-frame gating of the fourth subframe 850 relative to the second subframe 830, and the intra-frame gating and intra-frame gating of the fourth subframe 850 relative to the third subframe 840 are obtained. Intra-frame compensation 860 is performed using the decoded first subframe 820, the decoded second subframe 830, the decoded third subframe 840, and the intra-frame stream and intra-frame gating of the fourth subframe 850 relative to the first subframe 820, the intra-frame stream and intra-frame gating of the fourth subframe 850 relative to the second subframe 830, and the intra-frame stream and intra-frame gating of the fourth subframe 850 relative to the third subframe 840 to obtain the fourth subframe prediction frame of the fourth subframe. When the feature data of the current frame 800 and the fourth sub-predicted frame are input into the decoding neural network 810, the fourth sub-frame 850 is decoded.

[0110] The current frame is reconstructed based on the first subframe 820, the second subframe 830, the third subframe 840, and the fourth subframe 850.

[0111] According to embodiments of this disclosure, intra-frame streaming can be used without intra-frame gating. This corresponds to the case where the intra-frame gating value is 1. Intra-frame streaming is used in the pixel domain.

[0112] The intra-frame compensation method will be described below.

[0113] When a frame is decomposed into K subframes, the number of subframes decoded before the k-th subframe is J, and the number of intra-streams for each subframe is 1, multiple prediction values ​​for the k-th subframe can be generated as shown in Equation 1 below.

[0114] [Equation 1]

[0115] Where j = 1, …, J. It can be achieved through the intraflow of the j-th subframe decoded before the k-th subframe, based on the j-th and k-th subframes. j The distortion is performed to obtain the k-th sub-prediction frame of the k-th sub-frame. j->k Therefore, J k-th subprediction frames can be obtained. j->k J can be equal to or less than k-1. When all k-1 subframes preceding the k-th subframe are used, J can be equal to k-1. When all k-1 subframes may not be used due to complexity constraints, J can be less than k-1.

[0116] One of multiple sub-predicted frames can be used for decoding. Additionally, when multiple sub-predicted frames are in the feature map domain, they can be used for concatenated decoding.

[0117] Furthermore, when a frame is decomposed into K subframes, the number of subframes decoded before the k-th subframe is J, and the number of intra-streams for each subframe is 1, as shown in Equation 2 below, the predicted value of the k-th subframe can be generated as a single value.

[0118] [Equation 2]

[0119] Where j = 1, …, J. It can be achieved through the intraflow of the j-th subframe decoded before the k-th subframe, based on the j-th and k-th subframes. jThe Kth sub-prediction frame of the Kth subframe is obtained by warping the J obtained Kth sub-prediction frames with weight 1 and dividing the result by J. k ).

[0120] Such sub-predicted frames can be used for decoding. For example, they can be used for residual decoding.

[0121] When a frame is decomposed into K subframes, the number of subframes decoded before the k-th subframe is J, and the number of intra-frame streams for each subframe is N, as shown in Equation 3 below, multiple prediction values ​​for the k-th subframe can be generated.

[0122] [Equation 3]

[0123] Where j = 1, …, J. The nth intraflow (intraflow) can be generated from N intraflows of the j-th subframe decoded before the k-th subframe, based on the j-th and k-th subframes. n,j The distortion is performed to obtain N k-th sub-predicted frames of the k-th subframe. The J k-th sub-predicted frames can be obtained by summing all N k-th sub-predicted frames and dividing by N. j->k ).

[0124] One of multiple sub-predicted frames can be used for decoding. Additionally, when multiple sub-predicted frames are in the feature map domain, they can be used for concatenated decoding.

[0125] Furthermore, when a frame is decomposed into K subframes, the number of subframes decoded before the k-th subframe is J, and the number of intra-frame streams for each subframe is N, as shown in Equation 4 below, the predicted value of the k-th subframe can be generated as a value.

[0126] [Equation 4]

[0127] Where j = 1, …, J. The nth intraflow (intraflow) can be generated from N intraflows of the j-th subframe decoded before the k-th subframe, based on the j-th and k-th subframes. n,j The distortion is performed to obtain N k-th sub-predicted frames of the k-th sub-frame. J k-th predicted frames can be obtained by adding all N k-th predicted frames together. The J k-th predicted frames can be obtained by equally adding them with weight 1 and dividing the result by N. J is used to obtain a subprediction frame (SubPrediction) of the k-th subframe. k ).

[0128] Such sub-predicted frames can be used for decoding. For example, they can be used for residual decoding.

[0129] When a frame is decomposed into K subframes, the number of subframes decoded before the k-th subframe is J, the number of intra-streams for each subframe is 1, and the number of intra-gatings for each subframe is 1, multiple prediction values ​​for the k-th subframe can be generated as shown in Equation 5 below.

[0130] [Equation 5]

[0131] Where j = 1, …, J. It can be achieved through the intraflow of the j-th subframe decoded before the k-th subframe, based on the j-th and k-th subframes. j The distortion is performed, and the distortion result is multiplied by the intra-gate of the k-th subframe for the j-th subframe. j To obtain the k-th sub-prediction frame of the k-th sub-frame. j->k Therefore, J k-th subprediction frames can be obtained. j->k ).

[0132] One of multiple sub-predicted frames can be used for decoding. Additionally, when multiple sub-predicted frames are in the feature map domain, they can be used for concatenated decoding.

[0133] Furthermore, when a frame is decomposed into K subframes, the number of subframes decoded before the k-th subframe is J, the number of intra-streams for each subframe is 1, and the number of intra-gatings for each subframe is 1, the predicted value of the k-th subframe can be generated as a single value, as shown in Equation 6 below.

[0134] [Equation 6]

[0135] Where j = 1, …, J. It can be achieved through the intraflow of the j-th subframe decoded before the k-th subframe, based on the j-th and k-th subframes. j The distortion is performed, and the distortion result is multiplied by the intra-gate of the k-th subframe for the j-th subframe. j The k-th sub-prediction frame of the k-th subframe can be obtained by summing the J obtained k-th sub-prediction frames equally with weight 1 and dividing the result by J. k ).

[0136] Such sub-predicted frames can be used for decoding. For example, they can be used for residual decoding.

[0137] When a frame is decomposed into K subframes, the number of subframes decoded before the k-th subframe is J, the number of intra-streams for each subframe is N, and the number of intra-gatings for each subframe is N, as shown in Equation 7 below, multiple prediction values ​​for the k-th subframe can be generated.

[0138] [Equation 7]

[0139] Where j = 1, …, J. The nth intraflow (intraflow) can be generated from N intraflows of the j-th subframe decoded before the k-th subframe, based on the j-th and k-th subframes. n,j The distortion is performed, and the distortion result is multiplied by the intragate of the k-th subframe for the j-th subframe. n,j To obtain the N k-th sub-predicted frames of the k-th subframe, we can use the sum of all N k-th sub-predicted frames and divide by N to obtain the J k-th sub-predicted frames. j->k ).

[0140] One of multiple sub-predicted frames can be used for decoding. Additionally, when multiple sub-predicted frames are in the feature map domain, they can be used for concatenated decoding.

[0141] Furthermore, when a frame is decomposed into K subframes, the number of subframes decoded before the k-th subframe is J, the number of intra-streams for each subframe is N, and the number of intra-gatings for each subframe is N, the predicted value of the k-th subframe can be generated as a single value, as shown in Equation 8 below.

[0142] [Equation 8]

[0143] Where j = 1, …, J. The nth intraflow (intraflow) can be generated from N intraflows of the j-th subframe decoded before the k-th subframe, based on the j-th and k-th subframes. n,j The distortion is performed, and the distortion result is multiplied by the intragate of the k-th subframe for the j-th subframe. n,j We can obtain N k-th sub-predicted frames from the k-th sub-frame. We can obtain J k-th sub-predicted frames by summing all N k-th sub-predicted frames. We can then divide the sum of the J k-th sub-predicted frames by N. J is used to obtain a subprediction frame (SubPrediction) of the k-th subframe. k ).

[0144] Such sub-predicted frames can be used for decoding. For example, they can be used for residual decoding.

[0145] Figure 9 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0146] Reference Figure 9 The current frame 900 is input into the encoding neural network 905 and output as the feature data of the current frame 900. When the feature data of the current frame 900 is input into the feature extraction neural network 911 in the decoding neural network 910, the feature extraction data of the current frame 900 is output. The feature extraction data can be data that further enhances the features of the current frame 900.

[0147] When the feature extraction data of the current frame 900 is input into the progressive intra-frame block 912, the first subframe 920, the second subframe 930, the third subframe 940 and the fourth subframe 950 are decoded.

[0148] Specifically, when the feature extraction data of the current frame 900 is input into the amplification neural network 913 in the progressive intra-frame block 912, the first subframe 920 is decoded first according to a predetermined decoding order. The amplification neural network 913 is used to amplify the feature extraction data of the current frame 900 because the feature extraction data has a reduced size compared to the original size of the current frame 900. The predetermined decoding order can be determined based on the learning or training results of the amplification neural network 913 for previous frames.

[0149] After the first subframe 920 is decoded, when the feature extraction data of the current frame 900 is input into the stream / gated amplification neural network 914 in the progressive intra-frame block 912, the intra-frame stream and intra-frame gating of the second subframe 930 relative to the first subframe 920 are obtained. Intra-frame compensation 915 is performed using the decoded first subframe 920 and the intra-frame stream and intra-frame gating of the second subframe 930 relative to the first subframe 920, to obtain the second sub-predicted frame of the second subframe 930. The second subframe 930 is decoded when the feature extraction data of the current frame 900 and the second sub-predicted frame are input into the amplification neural network 913.

[0150] After the second subframe 930 is decoded, when the feature extraction data of the current frame 900 is input into the stream / gated amplification neural network 914 in the progressive intra-frame block 912, the intra-frame stream and intra-frame gating of the third subframe 940 relative to the first subframe 920, and the intra-frame stream and intra-frame gating of the third subframe 940 relative to the second subframe 930 are obtained. Intra-frame compensation 915 is performed using the decoded first subframe 920, the decoded second subframe 930, and the intra-frame stream and intra-frame gating of the third subframe 940 relative to the first subframe 920, and the intra-frame stream and intra-frame gating of the third subframe 940 relative to the second subframe 930, to obtain the third sub-predicted frame of the third subframe 940. The third subframe 940 is decoded when the feature extraction data of the current frame 900 and the third sub-predicted frame are input into the amplification neural network 913.

[0151] After the third subframe 940 is decoded, when the feature data of the current frame 900 is input into the stream / gated amplification neural network 914 in the progressive intra-frame block 912, the intra-frame stream and intra-frame gating of the fourth subframe 950 relative to the first subframe 920, the intra-frame stream and intra-frame gating of the fourth subframe 950 relative to the second subframe 930, and the intra-frame stream and intra-frame gating of the fourth subframe 950 relative to the third subframe 940 are obtained. By performing intra-frame compensation 915 using the decoded first subframe 920, the decoded second subframe 930, the decoded third subframe 940, the intra-frame stream and intra-frame gating of the fourth subframe 950 relative to the first subframe 920, the intra-frame stream and intra-frame gating of the fourth subframe 950 relative to the second subframe 930, and the intra-frame stream and intra-frame gating of the fourth subframe 950 relative to the third subframe 940, the fourth sub-predicted frame of the fourth subframe 950 is obtained. When the feature extraction data of the current frame 900 and the fourth sub-prediction frame are input into the amplification neural network 913, the fourth sub-frame 950 is decoded.

[0152] The current frame is reconstructed based on the first subframe 920, the second subframe 930, the third subframe 940, and the fourth subframe 950 that were decoded.

[0153] According to embodiments of this disclosure, intra-streams can be used without intra-gating. This corresponds to the case where the intra-gating value is 1. Here, because intra-compensation is performed in the pixel domain, the intra-stream is in the pixel domain.

[0154] Although Figure 9 There may be one feature extraction neural network, but multiple feature extraction neural networks can exist. Therefore, feature extraction data can be output through multiple feature extraction neural networks.

[0155] Despite Figure 9 There may be one progressive intra-block, but multiple progressive intra-blocks may exist. In the following text, reference will be made to... Figure 10Describes a configuration with two progressive intra-blocks.

[0156] Figure 10 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0157] Reference Figure 10 The current frame 1000 is input into the encoding neural network 1005 and output as the feature data of the current frame 1000. When the feature data of the current frame 1000 is input into the feature extraction neural network 1011 in the decoding neural network 1010, the feature extraction data of the current frame 1000 is output. The feature extraction data can be data that further enhances the features of the current frame 1000.

[0158] When the feature extraction data of the current frame 1000 is input into the first progressive intra-frame block 1012 and the second progressive intra-frame block 1022, the first subframe 1020, the second subframe 1030, the third subframe 1040 and the fourth subframe 1050 are decoded.

[0159] Specifically, when the feature extraction data of the current frame 900 is input into the amplification neural network 1013 in the first progressive intra-frame block 1012, the feature map of the first sub-frame is first obtained according to a predetermined decoding order. The amplification neural network 1013 is used to amplify the feature extraction data of the current frame 1000 because the feature extraction data has a reduced size compared to the original size of the current frame 1000. The predetermined decoding order can be determined based on the learning or training results of the amplification neural network 1013 for previous frames.

[0160] After obtaining the first subframe feature map, when the feature extraction data of the current frame 1000 is input into the stream / gated amplification neural network 1014 in the first progressive intra-frame block 1012, the intra-frame stream and intra-frame gating of the second subframe feature map relative to the first subframe feature map are obtained. Intra-frame compensation 1015 is performed using the obtained first subframe feature map and the intra-frame stream and intra-frame gating of the second subframe feature map relative to the first subframe feature map to obtain the second subframe feature map. When the feature extraction data of the current frame 1000 and the second subframe prediction feature map are input into the amplification neural network 1013, the second subframe feature map is obtained.

[0161] After obtaining the second subframe feature map, when the feature extraction data of the current frame 1000 is input into the stream / gated amplification neural network 1014 in the first progressive intra-frame block 1012, intra-frame streams and intra-frame gating of the third subframe feature map relative to the first subframe feature map, and intra-frame streams and intra-frame gating of the third subframe feature map relative to the second subframe feature map are obtained. Intra-frame compensation 1015 is performed using the obtained first subframe feature map, the obtained second subframe feature map, the intra-frame streams and intra-frame gating of the third subframe feature map relative to the first subframe feature map, and the intra-frame streams and intra-frame gating of the third subframe feature map relative to the second subframe feature map, to obtain the third subframe predicted frame feature map of the third subframe feature map. When the feature extraction data of the current frame 1000 and the third subframe predicted frame feature map are input into the amplification neural network 1013, the third subframe feature map is obtained.

[0162] After obtaining the third subframe feature map, when the feature data of the current frame 1000 is input into the stream / gated amplification neural network 1014 in the first progressive intra-frame block 1012, the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the first subframe feature map, the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the second subframe feature map, and the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the third subframe feature map are obtained. Intra-frame compensation 1015 is performed by using the obtained first subframe feature map, the obtained second subframe feature map, the obtained third subframe feature map, the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the first subframe feature map, the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the second subframe feature map, and the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the third subframe feature map, to obtain the fourth sub-predicted frame feature map of the fourth subframe feature map. When the feature extraction data of the current frame 1000 and the feature map of the fourth sub-predicted frame are input into the amplification neural network 1013, the feature map of the fourth sub-frame is obtained.

[0163] The feature map of the current frame 1000 can be obtained based on the obtained first subframe feature map, the obtained second subframe feature map, the obtained third subframe feature map, and the obtained fourth subframe feature map.

[0164] When the feature map of the current frame 1000 is input into the amplified neural network 1023 in the second progressive intra-frame block 1022, the first subframe 1020 is decoded first according to a predetermined decoding order. The feature map of the current frame 1000 is data based on the neural network representing the features of the current frame, and may be larger than the feature extraction data of the current frame 1000 but smaller than the current frame 1000. The predetermined decoding order can be determined based on the learning or training results of the amplified neural network 1023 for previous frames.

[0165] After the first subframe 1020 is decoded, when the feature map of the current frame 1000 is input into the stream / gated amplification neural network 1024 in the second progressive intra-frame block 1022, the intra-frame stream and intra-frame gating of the second subframe 1030 relative to the first subframe 1020 are obtained. Intra-frame compensation 1025 is performed using the decoded first subframe 1020 and the intra-frame gating and intra-frame stream of the second subframe 1030 relative to the first subframe 1020 to obtain the second subframe prediction frame of the second subframe 1030. The second subframe 1030 is decoded when the feature map of the current frame 1000 and the second subframe prediction frame are input into the amplification neural network 1023.

[0166] After the second subframe 1030 is decoded, when the feature map of the current frame 1000 is input into the stream / gated amplification neural network 1024 in the second progressive intra-frame block 1022, the intra-frame stream and intra-frame gating of the third subframe 1040 relative to the first subframe 1020, and the intra-frame stream and intra-frame gating of the third subframe 1040 relative to the second subframe 1030 are obtained. Intra-frame compensation 1025 is performed using the decoded first subframe 1020, the decoded second subframe 1030, and the intra-frame stream and intra-frame gating of the third subframe 1040 relative to the second subframe 1030, to obtain the third sub-predicted frame of the third subframe 1040. The third subframe 1040 is decoded when the feature map of the current frame 1000 and the third sub-predicted frame are input into the amplification neural network 1023.

[0167] After the third subframe 1040 is decoded, when the feature map of the current frame 1000 is input into the stream / gated amplification neural network 1024 in the second progressive intra-frame block 1022, the intra-frame stream and intra-frame gating of the fourth subframe 1050 for the first subframe 1020, the intra-frame stream and intra-frame gating of the fourth subframe 1050 for the second subframe 1030, and the intra-frame stream and intra-frame gating of the fourth subframe 1050 for the third subframe 1040 are obtained. Intra-frame compensation 1025 is performed using the decoded first subframe 1020, decoded second subframe 1030, decoded third subframe 1040, and fourth subframe 1050, applying intra-frame compensation to the intra-frame flow and intra-frame gating of the first subframe 1020, the intra-frame flow and intra-frame gating of the fourth subframe 1050 to the second subframe 1030, and the intra-frame flow and intra-frame gating of the fourth subframe 1050 to the third subframe 1040, to obtain the fourth subframe prediction frame of the fourth subframe 1050. The fourth subframe 1050 is decoded when the feature map of the current frame 1000 and the fourth subframe prediction frame are input into the amplification neural network 1023.

[0168] The current frame is reconstructed based on the first subframe 1020, the second subframe 1030, the third subframe 1040, and the fourth subframe 1050 that were decoded.

[0169] According to embodiments of this disclosure, intra-streams can be used without intra-gating. This corresponds to the case where the intra-gating value is 1. In the first progressive intra-block 1012, since intra-compensation is performed in the feature map domain, the intra-stream of the first progressive intra-block 1012 is in the feature map domain, and in the second progressive intra-block 1022, since intra-compensation is performed in the pixel domain, the intra-stream of the second progressive intra-block 1022 is in the pixel domain.

[0170] Although Figure 10 There may be one feature extraction neural network, but multiple feature extraction neural networks can exist. Therefore, feature extraction data can be output through multiple feature extraction neural networks.

[0171] Figure 11 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0172] Reference Figure 11 The current frame 1100 is input into the encoding neural network 1105 and output as the feature data of the current frame 1100.

[0173] When the feature data of the current frame 1100 is input into the progressive intra-frame block 1111 in the decoding neural network 1110, the first subframe 1120, the second subframe 1130, the third subframe 1140 and the fourth subframe 1150 are decoded.

[0174] Specifically, when the feature data of the current frame 1100 is input into the amplification neural network 1112 in the progressive intra-frame block 1111, the first subframe 1120 is decoded first according to a predetermined decoding order. The amplification neural network 1112 is used to amplify the feature data of the current frame 1100 because the feature data of the current frame 1100 has a reduced size compared to the original size of the current frame 1100. The predetermined decoding order can be determined based on the learning or training results of the amplification neural network 1112 for previous frames.

[0175] After the first subframe 1120 is decoded, when the feature data of the current frame 1100 is input into the stream / gated amplification neural network 1113 in the progressive intra-frame block 1111, the intra-frame stream and intra-frame gating of the second subframe 1130 relative to the first subframe 1120 are obtained. Intra-frame compensation 1114 is performed using the decoded first subframe 1120 and the intra-frame stream and intra-frame gating of the second subframe 1130 relative to the first subframe 1120, to obtain the second sub-predicted frame of the second subframe 1130. The second subframe 1130 is decoded when the feature data of the current frame 1100 and the second sub-predicted frame are input into the amplification neural network 1112.

[0176] After the second subframe 1130 is decoded, when the feature data of the current frame 1100 is input into the stream / gated amplification neural network 1113 in the progressive intra-frame block 1111, the intra-frame stream and intra-frame gating of the third subframe 1140 relative to the first subframe 1120, and the intra-frame stream and intra-frame gating of the third subframe 1140 relative to the second subframe 1130 are obtained. Intra-frame compensation 1114 is performed by using the decoded first subframe 1120, the decoded second subframe 1130, and the intra-frame stream and intra-frame gating of the third subframe 1140 relative to the first subframe 1120, and the intra-frame stream and intra-frame gating of the third subframe 1140 relative to the second subframe 1130, to obtain the third sub-predicted frame of the third subframe 1140. The third subframe 1140 is decoded when the feature data of the current frame 1100 and the third sub-predicted frame are input into the amplification neural network 1112.

[0177] After the third subframe 1140 is decoded, when the feature data of the current frame 1100 is input into the stream / gated amplification neural network 1113 in the progressive intra-frame block 1111, the intra-frame stream and intra-frame gating of the fourth subframe 1150 for the first subframe 1120, the intra-frame stream and intra-frame gating of the fourth subframe 1150 for the second subframe 1130, and the intra-frame stream and intra-frame gating of the fourth subframe 1150 for the third subframe 1140 are obtained. Intra-frame compensation 1114 is performed on the first subframe 1120, the second subframe 1130, the third subframe 1140, and the fourth subframe 1150 using intra-frame compensation 1114 for the first subframe 1120, intra-frame compensation 1114 for the second subframe 1130, and intra-frame compensation 1114 for the third subframe 1140, to obtain the fourth subframe 1150. The fourth subframe 1150 is decoded when the feature data of the current frame 1100 and the fourth subframe 1114 are input into the amplification neural network 1114.

[0178] The current frame is reconstructed based on the first subframe 1120, the second subframe 1130, the third subframe 1140, and the fourth subframe 1150 that were decoded.

[0179] According to embodiments of this disclosure, intra-streams can be used without intra-gating. This corresponds to the case where the intra-gating value is 1. In progressive intra-block 1111, because intra-compensation is performed in the pixel domain, the intra-stream of progressive intra-block 1111 is in the pixel domain.

[0180] Despite Figure 11 There may be one progressive intra-block, but multiple progressive intra-blocks may exist. In the following text, reference will be made to... Figure 12 Describes a configuration with two progressive intra-blocks.

[0181] Figure 12 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0182] Reference Figure 12 The current frame 1200 is input into the encoding neural network 1205 and output as the feature data of the current frame 1200.

[0183] When the feature data of the current frame 1200 is input into the first progressive intra-block 1211 and the second progressive intra-block 1221 in the decoding neural network 1210, the first subframe 1220, the second subframe 1230, the third subframe 1240 and the fourth subframe 1250 are decoded.

[0184] Specifically, when the feature data of the current frame 1200 is input into the amplified neural network 1212 in the first progressive intra-frame block 1211, the feature map of the first sub-frame is obtained first according to a predetermined decoding order. The predetermined decoding order can be determined based on the learning or training results of the amplified neural network 1212 for previous frames.

[0185] After obtaining the first subframe feature map, when the feature data of the current frame 1200 is input into the stream / gated amplification neural network 1213 in the first progressive intra-frame block 1211, the intra-frame stream and intra-frame gating of the second subframe feature map relative to the first subframe feature map are obtained. Intra-frame compensation 1214 is performed using the obtained first subframe feature map and the intra-frame stream and intra-frame gating of the second subframe feature map relative to the first subframe feature map to obtain the second subframe feature map. The second subframe feature map is obtained when the feature data of the current frame 1200 and the second subframe prediction feature map are input into the amplification neural network 1212.

[0186] After obtaining the second subframe feature map, when the feature data of the current frame 1200 is input into the stream / gated amplification neural network 1213 in the first progressive intra-frame block 1212, intra-frame streams and intra-frame gating of the third subframe feature map relative to the first subframe feature map, and intra-frame streams and intra-frame gating of the third subframe feature map relative to the second subframe feature map are obtained. Intra-frame compensation 1214 is performed using the obtained first subframe feature map, the obtained second subframe feature map, the intra-frame streams and intra-frame gating of the third subframe feature map relative to the first subframe feature map, and the intra-frame streams and intra-frame gating of the third subframe feature map relative to the second subframe feature map, to obtain the third subframe feature map of the third subframe feature map. The third subframe feature map is obtained when the feature data of the current frame 1200 and the third subframe prediction feature map are input into the amplification neural network 1214.

[0187] After obtaining the third subframe feature map, when the feature data of the current frame 1200 is input into the stream / gated amplification neural network 1213 in the first progressive intra-frame block 1211, the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the first subframe feature map, the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the second subframe feature map, and the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the third subframe feature map are obtained. Intra-frame compensation 1214 is performed by using the obtained first subframe feature map, the obtained second subframe feature map, the obtained third subframe feature map, the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the first subframe feature map, the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the second subframe feature map, and the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the third subframe feature map, to obtain the fourth sub-predicted frame feature map of the fourth subframe feature map. When the feature extraction data of the current frame 1200 and the feature map of the fourth sub-predicted frame are input into the amplification neural network 1211, the feature map of the fourth sub-frame is obtained.

[0188] The feature map of the current frame 1200 is obtained based on the obtained first subframe feature map, the obtained second subframe feature map, the obtained third subframe feature map, and the obtained fourth subframe feature map.

[0189] When the feature map of the current frame 1200 is input into the amplified neural network 1222 in the second progressive intra-frame block 1221, the first subframe 1220 is decoded first according to a predetermined decoding order. The feature map of the current frame 1200 is data based on the neural network representing the features of the current frame, and may be greater than the feature data of the current frame 1200 but less than the feature data of the current frame 1200. The predetermined decoding order can be determined based on the learning or training results of the amplified neural network 1222 for previous frames.

[0190] After the first subframe 1220 is decoded, when the feature map of the current frame 1200 is input into the stream / gated amplification neural network 1223 in the second progressive intra-frame block 1221, the intra-stream and intra-gated parameters of the second subframe 1230 relative to the first subframe 1220 are obtained. Intra-frame compensation 1224 is performed using the decoded first subframe 1220 and the intra-stream and intra-gated parameters of the second subframe 1230 relative to the first subframe 1220, resulting in the second sub-predicted frame of the second subframe 1230. The second subframe 1230 is decoded when the feature map of the current frame 1200 and the second sub-predicted frame are input into the amplification neural network 1222.

[0191] After the second subframe 1230 is decoded, when the feature map of the current frame 1200 is input into the stream / gated amplification neural network 1223 in the second progressive intra-frame block 1221, the intra-frame stream and intra-frame gating of the third subframe 1240 relative to the first subframe 1220, and the intra-frame stream and intra-frame gating of the third subframe 1240 relative to the second subframe 1230 are obtained. Intra-frame compensation 1224 is performed using the decoded first subframe 1220, the decoded second subframe 1230, and the intra-frame stream and intra-frame gating of the third subframe 1240 relative to the second subframe 1230, to obtain the third subframe prediction frame of the third subframe 1240. The third subframe 1240 is decoded when the feature map of the current frame 1200 and the third subframe prediction frame are input into the amplification neural network 1222.

[0192] After the third subframe 1240 is decoded, when the feature map of the current frame 1200 is input into the stream / gated amplification neural network 1223 in the second progressive intra-frame block 1221, the intra-frame stream and intra-frame gating of the fourth subframe 1250 for the first subframe 1220, the intra-frame stream and intra-frame gating of the fourth subframe 1250 for the second subframe 1230, and the intra-frame stream and intra-frame gating of the fourth subframe 1250 for the third subframe 1240 are obtained. Intra-frame compensation 1224 is performed using the decoded first subframe 1220, decoded second subframe 1230, decoded third subframe 1240, and fourth subframe 1250, applying intra-frame compensation to the intra-frame flow and intra-frame gating of the first subframe 1220, the intra-frame flow and intra-frame gating of the fourth subframe 1250 to the second subframe 1230, and the intra-frame flow and intra-frame gating of the fourth subframe 1250 to the third subframe 1240, to obtain the fourth subframe prediction frame of the fourth subframe 1250. The fourth subframe 1250 is decoded when the feature map of the current frame 1200 and the fourth subframe prediction frame are input into the amplification neural network 1222.

[0193] The current frame is reconstructed based on the first subframe 1220, the second subframe 1230, the third subframe 1240, and the fourth subframe 1250 that were decoded.

[0194] According to embodiments of this disclosure, intra-streams can be used without intra-gating. This corresponds to the case where the intra-gating value is 1. In the first progressive intra-block 1211, because intra-compensation is performed in the feature map domain, the intra-stream of the first progressive intra-block 1211 is in the feature map domain, and in the second progressive intra-block 1221, because intra-compensation is performed in the pixel domain, the intra-stream of the second progressive intra-block 1221 is in the pixel domain.

[0195] Figure 13This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0196] Reference Figure 13 The current frame 1300 is input into the encoding neural network 1305 and output as the feature data of the current frame 1300.

[0197] When the feature data of the current frame 1300 is input into the progressive intra-frame block 1311 in the decoding neural network 1310, the first subframe 1320, the second subframe 1330, the third subframe 1340 and the fourth subframe 1350 are decoded.

[0198] Specifically, when the feature data of the current frame 1300 is input into the amplifying neural network 1312 in the progressive intra-frame block 1311, the first subframe 1320 is decoded first according to a predetermined decoding order. The amplifying neural network 1312 is used to amplify the feature data of the current frame 1300 because the feature data of the current frame 1300 has a reduced size compared to the original size of the current frame 1300. The predetermined decoding order can be determined based on the learning or training results of the amplifying neural network 1312 for previous frames.

[0199] After the first subframe 1320 is decoded, when the feature data of the current frame 1300 is input into the stream / gated amplification neural network 1313 in the progressive intra-frame block 1311, the intra-stream and intra-gated parameters of the second subframe 1330 relative to the first subframe 1320 are obtained. Intra-compensation 1314 is performed by using the decoded first subframe 1320 and the intra-stream and intra-gated parameters of the second subframe 1330 relative to the first subframe 1320, resulting in the second subframe 1130's second sub-predicted frame. When the feature data of the current frame 1300 is input into the amplification neural network 1312, the second sub-residual frame is obtained. The second subframe 1330 is decoded when the second sub-predicted frame and the second sub-residual frame are added together (1315).

[0200] After the second subframe 1330 is decoded, when the feature data of the current frame 1300 is input into the stream / gated amplification neural network 1313 in the progressive intra-frame block 1311, the intra-frame stream and intra-frame gating of the third subframe 1340 relative to the first subframe 1320, and the intra-frame stream and intra-frame gating of the third subframe 1340 relative to the second subframe 1330 are obtained. Intra-frame compensation 1314 is performed using the decoded first subframe 1320, the decoded second subframe 1330, and the intra-frame stream and intra-frame gating of the third subframe 1340 relative to the first subframe 1320, and the intra-frame stream and intra-frame gating of the third subframe 1340 relative to the second subframe 1330, to obtain the third sub-predicted frame of the third subframe 1340. When the feature data of the current frame 1300 is input into the amplification neural network 1312, the third sub-residual frame is obtained. The third subframe 1340 is decoded when the third sub-predicted frame is added to the third sub-residual frame (1315).

[0201] After the third subframe 1340 is decoded, when the feature data of the current frame 1300 is input into the stream / gated amplification neural network 1313 in the progressive intra-frame block 1311, the intra-frame stream and intra-frame gating of the fourth subframe 1350 for the first subframe 1320, the intra-frame stream and intra-frame gating of the fourth subframe 1350 for the second subframe 1330, and the intra-frame stream and intra-frame gating of the fourth subframe 1350 for the third subframe 1340 are obtained. Intra-frame compensation 1314 is performed using the decoded first subframe 1320, decoded second subframe 1330, decoded third subframe 1340, and fourth subframe 1350, applying intra-frame compensation to the first subframe 1320, intra-frame compensation to the second subframe 1330, and intra-frame compensation to the third subframe 1340, respectively, to obtain the fourth subframe 1350's fourth subframe prediction frame. When the feature data of the current frame 1300 is input into the amplification neural network 1314, the fourth subframe residual frame is obtained. The fourth subframe 1350 is decoded when the fourth subframe prediction frame and the fourth subframe residual frame are added together (1315).

[0202] The current frame is reconstructed based on the first subframe 1320, the second subframe 1330, the third subframe 1340, and the fourth subframe 1350 that were decoded.

[0203] According to embodiments of this disclosure, intra-streams can be used without intra-gating. This corresponds to the case where the intra-gating value is 1. In progressive intra-block 1311, because intra-compensation is performed in the pixel domain, the intra-stream of progressive intra-block 1311 is in the pixel domain.

[0204] Despite Figure 13 There may be one progressive intra-block, but multiple progressive intra-blocks can exist. In the following text, we will... Figure 14 The description contains a configuration with two progressive intra-blocks.

[0205] Figure 14 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0206] Reference Figure 14 The current frame 1400 is input into the encoding neural network 1405 and output as the feature data of the current frame 1400.

[0207] When the feature data of the current frame 1400 is input into the first progressive intra-block 1411 and the second progressive intra-block 1421 in the decoding neural network 1410, the first subframe 1420, the second subframe 1430, the third subframe 1440 and the fourth subframe 1450 are decoded.

[0208] Specifically, when the feature data of the current frame 1400 is input into the amplified neural network 1412 in the first progressive intra-frame block 1411, the feature map of the first subframe is obtained first according to a predetermined decoding order. The predetermined decoding order can be determined based on the learning or training results of the amplified neural network 1412 for previous frames.

[0209] After obtaining the first subframe feature map, when the feature data of the current frame 1400 is input into the stream / gated amplification neural network 1413 in the first progressive intra-frame block 1411, the intra-frame stream and intra-frame gating of the second subframe feature map relative to the first subframe feature map are obtained. Intra-frame compensation 1414 is performed by using the obtained first subframe feature map and the intra-frame stream and intra-frame gating of the second subframe feature map relative to the first subframe feature map to obtain the second subframe feature map. When the feature data of the current frame 1400 is input into the amplification neural network 1412, the second sub-residual frame feature map is obtained. When the second sub-predicted frame feature map and the second sub-residual frame feature map are added (1415), the second subframe feature map is obtained.

[0210] After obtaining the second subframe feature map, when the feature data of the current frame 1400 is input into the stream / gated amplification neural network 1413 in the first progressive intra-frame block 1412, the intra-frame stream and intra-frame gating of the third subframe feature map relative to the first subframe feature map, and the intra-frame stream and intra-frame gating of the third subframe feature map relative to the second subframe feature map are obtained. Intra-frame compensation 1414 is performed by using the obtained first subframe feature map, the obtained second subframe feature map, the intra-frame stream and intra-frame gating of the third subframe feature map relative to the first subframe feature map, and the intra-frame stream and intra-frame gating of the third subframe feature map relative to the second subframe feature map, to obtain the third subframe feature map. When the feature data of the current frame 1400 is input into the amplification neural network 1414, the third sub-residual frame feature map is obtained. When the third sub-predicted frame feature map and the third sub-residual frame feature map are added (1415), the third subframe feature map is obtained.

[0211] After obtaining the third subframe feature map, when the feature data of the current frame 1400 is input into the stream / gated amplification neural network 1413 in the first progressive intra-frame block 1411, intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the first subframe feature map, intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the second subframe feature map, and intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the third subframe feature map are obtained. Intra-frame compensation 1414 is performed by using the obtained first subframe feature map, the obtained second subframe feature map, the obtained third subframe feature map, the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the first subframe feature map, the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the second subframe feature map, and the intra-frame stream and intra-frame gating of the fourth subframe feature map relative to the third subframe feature map, to obtain the fourth sub-predicted frame feature map of the fourth subframe feature map. When the feature extraction data of the current frame 1400 is input into the amplification neural network 1411, the feature map of the fourth sub-residual frame is obtained. When the feature map of the fourth sub-predicted frame is added to the feature map of the fourth sub-residual frame (1415), the feature map of the fourth sub-frame is obtained.

[0212] The feature map of the current frame 1400 is obtained based on the obtained first subframe feature map, the obtained second subframe feature map, the obtained third subframe feature map, and the obtained fourth subframe feature map.

[0213] When the feature map of the current frame 1400 is input into the amplified neural network 1422 in the second progressive intra-frame block 1421, the first subframe 1420 is decoded first according to a predetermined decoding order. The feature map of the current frame 1400 is data based on the neural network representing the features of the current frame, and may be greater than or less than the feature data of the current frame 1400. The predetermined decoding order can be determined based on the learning or training results of the amplified neural network 1422 for previous frames.

[0214] After the first subframe 1420 is decoded, when the feature map of the current frame 1400 is input into the stream / gated amplification neural network 1423 in the second progressive intra-frame block 1421, the intra-stream and intra-gated parameters of the second subframe 1430 relative to the first subframe 1420 are obtained. Intra-compensation 1424 is performed using the decoded first subframe 1420 and the intra-stream and intra-gated parameters of the second subframe 1430 relative to the first subframe 1420, resulting in the second sub-predicted frame of the second subframe 1430. When the feature map of the current frame 1400 is input into the amplification neural network 1222, the second sub-residual frame is obtained. The second subframe 1430 is decoded when the second sub-residual frame is added to the second sub-predicted frame (1425).

[0215] After the second subframe 1430 is decoded, when the feature map of the current frame 1400 is input into the stream / gated amplification neural network 1423 in the second progressive intra-frame block 1421, the intra-frame stream and intra-frame gating of the third subframe 1440 relative to the first subframe 1420, and the intra-frame stream and intra-frame gating of the third subframe 1440 relative to the second subframe 1430 are obtained. Intra-frame compensation 1424 is performed using the decoded first subframe 1420, the decoded second subframe 1430, and the intra-frame stream and intra-frame gating of the third subframe 1440 relative to the first subframe 1420, and the intra-frame stream and intra-frame gating of the third subframe 1440 relative to the second subframe 1430, to obtain the third sub-predicted frame of the third subframe 1440. When the feature map of the current frame 1400 is input into the amplification neural network 1422, the third sub-residual frame is obtained. When the third sub-residual frame is added to the third sub-predicted frame (1425), the third sub-frame 1440 is decoded.

[0216] After the third subframe 1440 is decoded, when the feature map of the current frame 1400 is input into the stream / gated amplification neural network 1423 in the second progressive intra-frame block 1421, the intra-frame stream and intra-frame gating of the fourth subframe 1450 relative to the first subframe 1420, the intra-frame stream and intra-frame gating of the fourth subframe 1450 relative to the second subframe 1430, and the intra-frame stream and intra-frame gating of the fourth subframe 1450 relative to the third subframe 1440 are obtained. Intra-frame compensation 1424 is performed using the decoded first subframe 1420, decoded second subframe 1430, decoded third subframe 1440, and fourth subframe 1450, applying intra-frame compensation to the intra-frame flow and intra-frame gating of the first subframe 1420, the intra-frame flow and intra-frame gating of the fourth subframe 1450 to the second subframe 1430, and the intra-frame flow and intra-frame gating of the fourth subframe 1450 to the third subframe 1440, to obtain the fourth sub-predicted frame of the fourth subframe 1450. The fourth sub-residual frame is obtained when the feature map of the current frame 1400 is input into the amplification neural network 1422. The fourth subframe 1450 is decoded when the fourth sub-residual frame is added to the fourth sub-predicted frame (1425).

[0217] The current frame is reconstructed based on the first subframe 1420, the second subframe 1430, the third subframe 1440, and the fourth subframe 1450 that were decoded.

[0218] According to embodiments of this disclosure, intra-streams can be used without intra-gating. This corresponds to the case where the intra-gating value is 1. In the first progressive intra-block 1411, because intra-compensation is performed in the feature map domain, the intra-stream of the first progressive intra-block 1411 is in the feature map domain, and in the second progressive intra-block 1421, because intra-compensation is performed in the pixel domain, the intra-stream of the second progressive intra-block 1421 is in the pixel domain.

[0219] Figure 15 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0220] Reference Figure 15 The current frame 1500 is input into the encoding neural network 1505 and output as the feature data of the current frame 1500.

[0221] When the feature data of the current frame 1500 is input into the progressive intra-frame block 1511 in the decoding neural network 1510, the first subframe 1520, the second subframe 1530, the third subframe 1540 and the fourth subframe 1550 are decoded.

[0222] Specifically, when the feature data of the current frame 1500 is input into the amplification neural network 1512 in the progressive intra-frame block 1511, the first subframe 1320 is decoded first according to a predetermined decoding order. The amplification neural network 1512 is used to amplify the feature data of the current frame 1500 because the feature data of the current frame 1500 has a reduced size compared to the original size of the current frame 1500. When data is output from the amplification neural network 1512, channel partitioning is performed, so that data is output for each channel. Some channels of data may be data for intra-frame streaming, some channels of data may be data for intra-frame gating, and the remaining channels of data may be data for subframes. For example, when there is one intra-frame streaming, two channels of data are data for intra-frame streaming, and because intra-frame gating is not required when there is one intra-frame streaming, the data of channels other than the two channels may be data for subframes. In addition, when there are four intra-frame streaming, eight channels of data may be data for intra-frame streaming, one channel of data may be data for intra-frame gating, and the remaining channels of data may be data for subframes. The predetermined decoding order can be determined based on the learning or training results of the amplified neural network 1512 on previous frames.

[0223] After the first subframe 1520 is decoded, when the feature data of the current frame 1500 is input into the amplifying neural network 1512 in the progressive intra-frame block 1511, the intra-frame stream and intra-frame gating of the second subframe 1530 relative to the first subframe 1520 are obtained through channel partitioning. Intra-frame compensation 1513 is performed by using the decoded first subframe 1520 and the intra-frame stream and intra-frame gating of the second subframe 1530 relative to the first subframe 1520 to obtain the second subframe 1530's second subframe prediction frame. When the feature data of the current frame 1500 is input into the amplifying neural network 1512, the second subframe residual is obtained through channel partitioning. When the second subframe prediction frame and the second subframe residual are added (1514), the second subframe 1530 is decoded.

[0224] After the second subframe 1530 is decoded, when the feature data of the current frame 1500 is input into the amplifying neural network 1512 in the progressive intra-frame block 1511, the intra-frame stream and intra-frame gating of the third subframe 1540 relative to the first subframe 1520, and the intra-frame stream and intra-frame gating of the third subframe 1540 relative to the second subframe 1530 are obtained through channel partitioning. Intra-frame compensation 1513 is performed using the decoded first subframe 1520, the decoded second subframe 1530, and the intra-frame stream and intra-frame gating of the third subframe 1540 relative to the first subframe 1520, and the intra-frame stream and intra-frame gating of the third subframe 1540 relative to the second subframe 1530, to obtain the third sub-predicted frame of the third subframe 1540. When the feature data of the current frame 1500 is input into the amplifying neural network 1512, the third sub-residual frame is obtained through channel partitioning. When the third sub-predicted frame is added to the third sub-residual frame (1514), the third sub-frame 1540 is decoded.

[0225] After the third subframe 1540 is decoded, when the feature data of the current frame 1500 is input into the amplified neural network 1512 in the progressive intra-frame block 1511, the intra-frame stream and intra-frame gating of the fourth subframe 1550 relative to the first subframe 1520, the intra-frame stream and intra-frame gating of the fourth subframe 1550 relative to the second subframe 1530, and the intra-frame stream and intra-frame gating of the fourth subframe 1550 relative to the third subframe 1540 are obtained through channel partitioning. Intra-frame compensation 1513 is performed using the decoded first subframe 1520, decoded second subframe 1530, decoded third subframe 1540, and fourth subframe 1550, applying intra-frame compensation to the intra-frame stream and intra-frame gating of the first subframe 1520, the intra-frame stream and intra-frame gating of the fourth subframe 1550 to the second subframe 1530, and the intra-frame stream and intra-frame gating of the fourth subframe 1550 to the third subframe 1540, to obtain the fourth sub-predicted frame of the fourth subframe 1550. When the feature data of the current frame 1500 is input into the amplifying neural network 1512, the fourth sub-residual frame is obtained through channel partitioning. The fourth subframe 1550 is decoded when the fourth sub-predicted frame and the fourth sub-residual frame are added together (1514).

[0226] The current frame is reconstructed based on the first subframe 1520, the second subframe 1530, the third subframe 1540, and the fourth subframe 1550.

[0227] According to embodiments of this disclosure, intra-streams can be used without intra-gating. This corresponds to the case where the intra-gating value is 1. In progressive intra-block 1511, because intra-compensation is performed in the pixel domain, the intra-stream of progressive intra-block 1511 is in the pixel domain.

[0228] Despite Figure 15 There may be one progressive intra-block, but multiple progressive intra-blocks may exist. In the following text, reference will be made to... Figure 16 Describes a configuration with two progressive intra-blocks.

[0229] Figure 16 This is a diagram illustrating a method for progressively decoding multiple subframes using a neural network and an intra-frame stream, according to embodiments of the present disclosure.

[0230] Reference Figure 16 The current frame 1600 is input into the encoding neural network 1605 and output as the feature data of the current frame 1600.

[0231] When the feature data of the current frame 1600 is input into the first progressive intra-block 1611 and the second progressive intra-block 1621 in the decoding neural network 1610, the first subframe 1620, the second subframe 1630, the third subframe 1640 and the fourth subframe 1650 are decoded.

[0232] Specifically, when the feature data of the current frame 1600 is input into the amplified neural network 1612 in the first progressive intra-frame block 1611, the feature map of the first sub-frame is first obtained according to a predetermined decoding order through channel partitioning. The predetermined decoding order can be determined based on the learning or training results of the amplified neural network 1612 for previous frames.

[0233] After obtaining the first subframe feature map, when the feature data of the current frame 1600 is input into the amplified neural network 1613 in the first progressive intra-block 1611, the intra-stream and intra-gating of the second subframe feature map relative to the first subframe feature map are obtained through channel partitioning. Intra-compensation 613 is performed using the obtained first subframe feature map and the intra-stream and intra-gating of the second subframe feature map relative to the first subframe feature map to obtain the second subframe feature map. When the feature data of the current frame 1600 is input into the amplified neural network 1612, the second sub-residual frame feature map is obtained through channel partitioning. The second subframe feature map is obtained by adding the second sub-predicted frame feature map and the second sub-residual frame feature map (1614).

[0234] After obtaining the second subframe feature map, when the feature data of the current frame 1600 is input into the amplification neural network 1612 in the first progressive intra-frame block 1612, the intra-frame stream and intra-frame gating of the third subframe feature map relative to the first subframe feature map, and the intra-frame stream and intra-frame gating of the third subframe feature map relative to the second subframe feature map are obtained through channel partitioning. Intra-frame compensation 1613 is performed by using the obtained first subframe feature map, the obtained second subframe feature map, the intra-frame stream and intra-frame gating of the third subframe feature map relative to the first subframe feature map, and the intra-frame stream and intra-frame gating of the third subframe feature map relative to the second subframe feature map to obtain the third subframe feature map. When the feature data of the current frame 1600 is input into the amplification neural network 1612, the third sub-residual frame feature map is obtained through channel partitioning. The third subframe feature map is obtained by adding the third sub-predicted frame feature map and the third sub-residual frame feature map (1614).

[0235] After obtaining the third subframe feature map, when the feature data of the current frame 1600 is input into the amplified neural network 1612 in the first progressive intra-frame block 1611, intra-frame flow and intra-frame gating of the fourth subframe feature map relative to the first subframe feature map, intra-frame flow and intra-frame gating of the fourth subframe feature map relative to the second subframe feature map, and intra-frame flow and intra-frame gating of the fourth subframe feature map relative to the third subframe feature map are obtained through channel partitioning. Intra-frame compensation 1613 is performed using the obtained first subframe feature map, the obtained second subframe feature map, the obtained third subframe feature map, the intra-frame flow and intra-frame gating of the fourth subframe feature map relative to the first subframe feature map, the intra-frame flow and intra-frame gating of the fourth subframe feature map relative to the second subframe feature map, and the intra-frame flow and intra-frame gating of the fourth subframe feature map relative to the third subframe feature map to obtain the fourth subframe prediction frame feature map of the fourth subframe feature map. When the feature extraction data of the current frame 1600 is input into the amplification neural network 1612, the fourth sub-residual frame feature map is obtained through channel partitioning. The fourth sub-frame feature map is obtained by adding the fourth sub-predicted frame feature map and the fourth sub-residual frame feature map (1614).

[0236] The feature map of the current frame 1600 is obtained based on the obtained first subframe feature map, the obtained second subframe feature map, the obtained third subframe feature map, and the obtained fourth subframe feature map.

[0237] When the feature map of the current frame 1600 is input into the amplified neural network 1622 in the second progressive intra-frame block 1621, the first subframe 1620 is decoded first according to a predetermined decoding order. The feature map of the current frame 1600 is data based on the neural network representing the features of the current frame, which may be greater than but less than the feature data of the current frame 1600. The predetermined decoding order can be determined based on the learning or training results of the amplified neural network 1622 for previous frames.

[0238] After the first subframe 1620 is decoded, when the feature map of the current frame 1600 is input into the amplifying neural network 1622 in the second progressive intra-frame block 1621, the intra-frame stream and intra-frame gating of the second subframe 1630 relative to the first subframe 1620 are obtained through channel partitioning. Intra-frame compensation 1623 is performed by using the decoded first subframe 1620 and the intra-frame stream and intra-frame gating of the second subframe 1630 relative to the first subframe 1620, to obtain the second sub-predicted frame of the second subframe 1630. When the feature map of the current frame 1600 is input into the amplifying neural network 1622, the second sub-residual frame is obtained through channel partitioning. When the second sub-residual frame and the second sub-predicted frame are added (1624), the second subframe 1630 is decoded.

[0239] After the second subframe 1630 is decoded, when the feature map of the current frame 1600 is input into the amplification neural network 1622 in the second progressive intra-frame block 1621, the intra-frame stream and intra-frame gating of the third subframe 1640 relative to the first subframe 1620, and the intra-frame stream and intra-frame gating of the third subframe 1640 relative to the second subframe 1630 are obtained through channel partitioning. Intra-frame compensation 1623 is performed using the decoded first subframe 1620, the decoded second subframe 1630, and the intra-frame stream and intra-frame gating of the third subframe 1640 relative to the first subframe 1620, and the intra-frame stream and intra-frame gating of the third subframe 1640 relative to the second subframe 1630, to obtain the third sub-predicted frame of the third subframe 1640. When the feature map of the current frame 1600 is input into the amplification neural network 1622, the third sub-residual frame is obtained through channel partitioning. When the third sub-residual frame is added to the third sub-predicted frame (1624), the third sub-frame 1640 is decoded.

[0240] After the third subframe 1640 is decoded, when the feature map of the current frame 1600 is input into the amplified neural network 1622 in the second progressive intra-frame block 1621, the intra-frame stream and intra-frame gating of the fourth subframe 1650 relative to the first subframe 1620, the intra-frame stream and intra-frame gating of the fourth subframe 1650 relative to the second subframe 1630, and the intra-frame stream and intra-frame gating of the fourth subframe 1650 relative to the third subframe 1640 are obtained through channel partitioning. Intra-frame compensation 1623 is performed using the decoded first subframe 1620, decoded second subframe 1630, decoded third subframe 1640, and fourth subframe 1650, applying intra-frame compensation to the intra-frame flow and intra-frame gating of the first subframe 1620, the intra-frame flow and intra-frame gating of the fourth subframe 1650 to the second subframe 1630, and the intra-frame flow and intra-frame gating of the fourth subframe 1650 to the third subframe 1640, to obtain the fourth sub-predicted frame of the fourth subframe 1650. When the feature map of the current frame 1600 is input into the amplification neural network 1622, the fourth sub-residual frame is obtained through channel partitioning. The fourth subframe 1650 is decoded when the fourth sub-residual frame is added to the fourth sub-predicted frame (1624).

[0241] The current frame is reconstructed based on the first subframe 1620, the second subframe 1630, the third subframe 1640, and the fourth subframe 1650 that were decoded.

[0242] According to embodiments of this disclosure, intra-streams can be used without intra-gating. This corresponds to the case where the intra-gating value is 1. In the first progressive intra-block 1611, because intra-compensation is performed in the feature map domain, the intra-stream of the first progressive intra-block 1611 is in the feature map domain, and in the second progressive intra-block 1621, because intra-compensation is performed in the pixel domain, the intra-stream of the second progressive intra-block 1621 is in the pixel domain.

[0243] Reference above Figures 8 to 16 The described method using intra-frame streams can be applied to image encoding during training.

[0244] In addition, the above references Figures 8 to 16 The described method using intra-frame streams can be applied in the same way to motion coding networks using optical flow learned from video coding. This is because optical flow exhibits strong spatial correlation due to the requirement that the movement of all pixels within an object must be identical, and strong nonlocal similarity due to the requirement that the movement of all pixels at the object's boundaries must be identical.

[0245] Additionally, the above reference Figures 8 to 16 The method described using intra-frame streams can also be applied to pixel-coding networks for learned video coding. This is because pixels have strong spatial correlation and strong non-local similarity.

[0246] Figure 17 This is a diagram illustrating a method for training a neural network used in an image encoding method and an image decoding method according to embodiments of the present disclosure.

[0247] Reference Figure 17 The encoding neural network 1705, the decoding neural network 1710, and the stream / gated decoding neural network 1715 can be trained using the current frame 1700.

[0248] Specifically, the current frame 1700 is input into the encoding neural network 1705 to obtain the feature data of the current frame 1700. When the feature data of the current frame 1700 is input into the decoding neural network 1710, the first training subframe 1720 is obtained.

[0249] When the feature data of the current training frame 1700 is input into the streaming / gated decoding neural network 1715, the intra-stream and intra-gated parameters for the training of the first sub-frame 1720 are obtained for the training of the second sub-frame 1730. Intra-compensation 1760 is performed using the intra-stream and intra-gated parameters for the training of the first sub-frame 1720 and the obtained training first sub-frame 1720. The training second sub-prediction frame is obtained through intra-compensation 1760. When the feature data of the current training frame 1700 and the training second sub-prediction frame are input into the decoding neural network 1710, the training second sub-frame 1730 is obtained.

[0250] When the feature data of the current training frame 1700 is input into the streaming / gated decoding neural network 1715, the intra-frame streaming and intra-frame gating for the training of the first sub-frame 1720 for the training of the third sub-frame 1740, and the intra-frame streaming and intra-frame gating for the training of the second sub-frame 1730 for the training of the third sub-frame 1740 are obtained. Intra-frame compensation 1760 is performed by using the intra-frame streaming and intra-frame gating for the training of the first sub-frame 1720, the intra-frame streaming and intra-frame gating for the training of the second sub-frame 1730 for the training of the third sub-frame 1740, the obtained training first sub-frame 1720, and the obtained training second sub-frame 1730. The training third sub-prediction frame is obtained through intra-frame compensation 1760. When the feature data of the current training frame 1700 and the training third sub-prediction frame are input into the decoding neural network 1710, the training third sub-frame 1740 is obtained.

[0251] When the feature data of the current frame 1700 is input into the stream / gated decoding neural network 1715, the intra-frame stream and intra-frame gating for the training of the first sub-frame 1720, the intra-frame stream and intra-frame gating for the training of the second sub-frame 1730, and the intra-frame stream and intra-frame gating for the training of the third sub-frame 1740 are obtained from the training of the fourth sub-frame 1750. Intra-frame compensation 1760 is performed by using the intra-frame stream and intra-frame gating for the training of the first sub-frame 1720, the training of the second sub-frame 1730, and the training of the third sub-frame 1740, as well as the obtained training of the first sub-frame 1720, the training of the second sub-frame 1730, and the training of the third sub-frame 1740. The fourth sub-prediction frame is obtained through intra-frame compensation 1760. When the feature data of the current frame 1700 and the fourth sub-prediction frame are input into the decoding neural network 1710, the fourth sub-frame 1750 is obtained.

[0252] Training reconstruction frame 1770 is obtained by using the obtained first training subframe 1720, the obtained second training subframe 1730, the obtained third training subframe 1740, and the obtained fourth training subframe 1750.

[0253] exist Figure 17 During training, the encoding neural network 1705, the decoding neural network 1710, and the stream / gated decoding neural network 1715 can be trained so that the training reconstructed frame 1770 becomes as similar as possible to the current training frame 1700 by comparison with 1775. To this end, as... Figure 17 As shown, the first loss information 1780 can be used to train the encoding neural network 1705, the decoding neural network 1710, and the stream / gated decoding neural network 1715.

[0254] The first loss information 1780 may correspond to the difference between the training current frame 1700 and the training reconstructed frame 1770. In an embodiment, the difference between the training current frame 1700 and the training reconstructed frame 1770 may include at least one of the following: L1 norm value, L2 norm value, structural similarity (SSIM) value, peak signal-to-noise ratio-human visual system (PSNR-HVS) value, multi-scale SSIM (MS-SSIM) value, variance inflation factor (VIF) value, or video multi-method evaluation fusion (VMAF) value.

[0255] Because the first loss information 1780 is related to the quality of the reconstructed image, the first loss information 1780 can be called quality loss information.

[0256] The second loss information 1790 can be calculated based on the bit rate of the bit stream generated as a result of encoding the feature data of the current training frame 1700.

[0257] Because the second loss information 1790 is related to the encoding efficiency of the feature data of the current frame 1700 during training, the second loss information can be called the compressed loss information.

[0258] The trainable encoding neural network 1705, decoding neural network 1710, and stream / gated decoding neural network 1715 reduce or minimize the final loss information derived from the first loss information 1780 and / or the second loss information 1790.

[0259] In this embodiment, the encoding neural network 1705, the decoding neural network 1710, and the stream / gated decoding neural network 1715 can be trained while changing the values ​​of preset parameters to reduce or minimize the final loss information.

[0260] In this embodiment, the final loss information can be calculated according to Equation 9 below.

[0261] [Equation 9] Final loss information = a First loss information + b Second loss information In Equation 9, a and b are the weights applied to the first loss information 1780 and the second loss information 1790, respectively.

[0262] According to Equation 9, the trainable encoding neural network 1705, decoding neural network 1710, and stream / gated decoding neural network 1715 can make the training reconstructed frame as similar as possible to the training current frame, and minimize the bit rate of the bit stream generated by encoding the feature data of the training current frame.

[0263] Reference can be executed by the training device Figure 17 The training process is described. The training device can be, for example, an image encoding device 2100 or a separate server. The parameters obtained as training results can be stored in the image encoding device 2100 and the image decoding device 1900.

[0264] Although reference Figure 17 Describes the training reference Figure 8 The method described is for neural networks, but reference systems can also be trained in a similar way. Figures 9 to 16 The neural network described.

[0265] Figure 18 This is a flowchart illustrating an image decoding method according to an embodiment of the present disclosure.

[0266] Reference Figure 18 During operation S1810, the image decoding device 1900 can obtain the feature data of the current frame from the bit stream.

[0267] According to embodiments of this disclosure, the current frame may be a pixel map including pixels.

[0268] According to embodiments of this disclosure, the current frame may be a feature map.

[0269] During operation S1820, the image decoding device 1900 can decode the first subframe among a plurality of subframes rearranged from the current frame according to a predetermined decoding order based on the feature data of the current frame.

[0270] According to embodiments of this disclosure, multiple subframes can be obtained by dividing the current frame into multiple pixel groups and rearranging pixels at the same position in the multiple pixel groups so that they are included in the same subframe.

[0271] In operation S1830, the image decoding device 1900 can obtain at least one intra-frame stream for the first subframe to be decoded, so as to decode the second subframe among the plurality of subframes.

[0272] According to embodiments of the present disclosure, at least one intra-stream of the decoded first subframe may indicate a reference pixel in the decoded first subframe for a first pixel in the second subframe.

[0273] During operation S1840, the image decoding device 1900 can decode the second subframe by using at least one intra-frame stream and the first subframe to be decoded.

[0274] According to embodiments of the present disclosure, the image decoding apparatus 1900 can obtain an intra-frame gating value for at least one intra-frame stream, and can decode a second subframe by using at least one intra-frame stream, the intra-frame gating value, and a first subframe to be decoded, wherein the intra-frame gating value can indicate the weight for at least one intra-frame stream.

[0275] According to embodiments of this disclosure, when at least one intra stream is multiple intra streams, the image decoding apparatus 1900 can obtain an intra-frame gating value corresponding to one of the multiple intra streams, and can decode a second subframe by using one of the multiple intra streams, the intra-frame gating value corresponding to one of the multiple intra streams, and the decoded first subframe.

[0276] According to embodiments of the present disclosure, the image decoding apparatus 1900 can obtain a second sub-residual frame for a second sub-frame, obtain a second sub-predicted frame by using at least one intra-frame stream and a decoded first sub-frame, and decode the second sub-frame by using the second sub-residual frame and the second sub-predicted frame.

[0277] During operation S1850, the image decoding device 1900 can reconstruct the current frame by using the first subframe and the second subframe that have been decoded.

[0278] According to embodiments of this disclosure, when multiple subframes include a third subframe, in order to decode the third subframe, the image decoding apparatus 1900 can obtain at least one second intra-frame stream for the first subframe to be decoded and at least one third intra-frame stream for the second subframe to be decoded. The third subframe can be decoded by using at least one second intra-frame stream, the first subframe to be decoded, at least one third intra-frame stream, and the second subframe to be decoded, and the current frame can be reconstructed by using the first subframe to be decoded, the second subframe to be decoded, and the third subframe to be decoded.

[0279] According to embodiments of this disclosure, when multiple subframes include a fourth subframe, in order to decode the fourth subframe, the image decoding apparatus 1900 can obtain at least one fourth intra-frame stream for the first subframe to be decoded, at least one fifth intra-frame stream for the second subframe to be decoded, and at least one sixth intra-frame stream for the third subframe to be decoded. The fourth subframe can be decoded by using at least one fourth intra-frame stream, the first subframe to be decoded, at least one fifth intra-frame stream, the second subframe to be decoded, at least one sixth intra-frame stream, and the third subframe to be decoded, and the current frame can be reconstructed by using the first subframe to be decoded, the second subframe to be decoded, the third subframe to be decoded, and the fourth subframe to be decoded.

[0280] Figure 19 This is a diagram illustrating the configuration of an image decoding apparatus according to an embodiment of the present disclosure.

[0281] Reference Figure 19 The image decoding device 1900 may include a receiver 1910, an artificial intelligence (AI) processor 1920, an intra-frame stream / gating acquirer 1930, an intra-frame compensator 1940, and a decoder 1950.

[0282] Receiver 1910, AI processor 1920, intra-stream / gating acquirer 1930, intra-compensator 1940, and decoder 1950 may be implemented as processors. A processor may include at least one processing circuit and / or multiple processors. For example, the term "processor" as used herein, including the claims, may include various processing circuits comprising at least one processor. In at least one processor, one or more processors may be configured to perform the various functions described herein individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms cover, but are not limited to, situations where one processor performs some functions while other processors perform other functions, and situations where a single processor can perform all functions. Furthermore, at least one processor may include a combination of processors for performing the disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions. Receiver 1910, AI processor 1920, intra-stream / gating acquirer 1930, intra-compensator 1940, and decoder 1950 may operate according to instructions stored in memory (not shown).

[0283] Despite Figure 19 The receiver 1910, AI processor 1920, intra-stream / gated acquirer 1930, intra-frame compensator 1940, and decoder 1950 are shown separately, but they can be implemented by a single processor. In this case, the receiver 1910, AI processor 1920, intra-stream / gated acquirer 1930, intra-frame compensator 1940, and decoder 1950 can be implemented as a dedicated processor, or by a combination of software and a general-purpose processor such as an application processor (AP), central processing unit (CPU), or graphics processing unit (GPU). Furthermore, when using a dedicated processor, it may include memory for implementing embodiments of this disclosure or a memory processor for using external memory.

[0284] The receiver 1910, AI processor 1920, intra-stream / gated acquirer 1930, intra-frame compensator 1940, and decoder 1950 may include multiple processors. In this case, the receiver 1910, AI processor 1920, intra-stream / gated acquirer 1930, intra-frame compensator 1940, and decoder 1950 may be implemented as a combination of dedicated processors, or may be implemented by a combination of software and multiple general-purpose processors (such as AP, CPU, or GPU).

[0285] Receiver 1910 can obtain the feature data of the current frame from the bit stream.

[0286] The feature data of the current frame can be sent to the AI ​​processor 1920 and the intra-frame stream / gating acquirer 1930.

[0287] The AI ​​processor 1920 can decode the first subframe of a plurality of subframes according to a predetermined decoding order.

[0288] The intra-stream / gating acquirer 1930 can output the intra-stream and intra-gating of the first subframe for the second subframe.

[0289] The intra-frame compensator 1940 can receive the intra-frame stream and intra-frame gating of the second subframe for the first subframe from the intra-frame stream / gating acquirer 1930, and can receive the first subframe from the AI ​​processor 1920.

[0290] The intra-frame compensator 1940 can obtain the second sub-predicted frame based on the first sub-frame and the intra-frame gating and intra-frame stream of the second sub-frame for the first sub-frame.

[0291] The intra-frame compensator 1940 can send the second sub-predicted frame to the AI ​​processor 1920.

[0292] The AI ​​processor 1920 can decode the second subframe based on the feature data of the current frame and the second sub-predicted frame.

[0293] According to embodiments of this disclosure, the AI ​​processor 1920 may include an intra-frame stream / gating acquirer 1930 and an intra-frame compensator 1940. Therefore, within the AI ​​processor 1920, a first subframe among multiple subframes can be decoded according to a predetermined decoding order; an intra-frame stream and intra-frame gating of a second subframe for the first subframe can be output; a second sub-predicted frame can be obtained based on the first subframe and the intra-frame stream and intra-frame gating of the second subframe for the first subframe; and the second subframe can be decoded based on the feature data of the current frame and the second sub-predicted frame.

[0294] Decoder 1950 can obtain the reconstructed frame of the current frame based on the first and second subframes.

[0295] According to embodiments of this disclosure, when multiple subframes include a third subframe, in order to decode the third subframe, the intra-stream / gating obtainr 1930 can obtain at least one second intra-stream for the first subframe and at least one third intra-stream for the second subframe. The intra-compensator 1940 can obtain a third sub-predicted frame based on at least one second intra-stream, the first subframe, at least one third intra-stream, and the second subframe, and can send the third sub-predicted frame to the AI ​​processor 1920. The AI ​​processor 1920 can decode the third subframe based on the feature data of the current frame and the third sub-predicted frame, and the decoder 1950 can obtain the reconstructed frame of the current frame based on the first subframe, the second subframe, and the third subframe.

[0296] According to embodiments of this disclosure, when multiple subframes include a fourth subframe, in order to decode the fourth subframe, the intra-stream / gating obtainr 1930 can obtain at least one fourth intra-stream for the first subframe, at least one fifth intra-stream for the second subframe, and at least one sixth intra-stream for the third subframe. The intra-compensator 1940 can obtain the fourth sub-predicted frame based on at least one fourth intra-stream, the first subframe, at least one fifth intra-stream, the second subframe, at least one sixth intra-stream, and the third subframe, and can send the fourth sub-predicted frame to the AI ​​processor 1920. The AI ​​processor 1920 can decode the fourth subframe based on the feature data of the current frame and the fourth sub-predicted frame, and the decoder 1950 can obtain the reconstructed frame of the current frame based on the first subframe, the second subframe, the third subframe, and the fourth subframe.

[0297] The AI ​​processor 1920 and the intra-stream / gated acquirer 1930 can be implemented as a neural network including one or more layers (e.g., convolutional layers).

[0298] The AI ​​processor 1920 and the intra-stream / gated acquirer 1930 can be stored in memory. The AI ​​processor 1920 and the intra-stream / gated acquirer 1930 can be implemented as at least one dedicated processor for AI.

[0299] Figure 20 This is a flowchart illustrating an image encoding method according to an embodiment of the present disclosure.

[0300] Reference Figure 20 During operation S2010, the image encoding device 2100 can generate feature data for the current frame.

[0301] According to embodiments of this disclosure, the current frame may be a pixel map including pixels.

[0302] According to embodiments of this disclosure, the current frame may be a feature map.

[0303] During operation S2020, the image encoding device 2100 can encode the first subframe among a plurality of subframes rearranged from the current frame according to a predetermined encoding order based on the feature data of the current frame.

[0304] According to embodiments of this disclosure, multiple subframes can be obtained by dividing the current frame into multiple pixel groups and rearranging pixels at the same position in the multiple pixel groups so that they are included in the same subframe.

[0305] During operation S2030, the image encoding device 2100 can obtain at least one intra-frame stream for encoding the first subframe so as to encode the second subframe among a plurality of subframes.

[0306] According to embodiments of the present disclosure, at least one intra-stream for an encoded first subframe may indicate a reference pixel in the encoded first subframe for a first pixel in a second subframe.

[0307] During operation S2040, the image encoding device 2100 can encode the second subframe by using at least one intra-frame stream and the decoded first subframe.

[0308] According to embodiments of the present disclosure, the image encoding apparatus 2100 can obtain an intra-frame gating value for at least one intra-frame stream, and can encode a second subframe using at least one intra-frame stream, the intra-frame gating value, and an encoded first subframe, wherein the intra-frame gating value can indicate the weight for at least one intra-frame stream.

[0309] According to embodiments of this disclosure, when at least one intra stream is multiple intra streams, the image encoding apparatus 2100 can obtain an intra-frame gating value corresponding to one of the multiple intra streams, and can encode a second subframe by using one of the multiple intra streams, the intra-frame gating value corresponding to one of the multiple intra streams, and the encoded first subframe.

[0310] According to embodiments of the present disclosure, the image encoding apparatus 2100 can obtain a second sub-residual frame for a second sub-frame, obtain a second sub-predicted frame by using at least one intra-frame stream and encoded first sub-frame, and encode the second sub-frame by using the second sub-residual frame and the second sub-predicted frame.

[0311] During operation S2050, the image encoding device 2100 can encode the current frame using the first encoded subframe and the second encoded subframe.

[0312] According to embodiments of this disclosure, when multiple subframes include a third subframe, in order to encode the third subframe, the image encoding apparatus 2100 can obtain at least one second intra-frame stream for the encoded first subframe and at least one third intra-frame stream for the encoded second subframe. The third subframe can be encoded by using at least one second intra-frame stream, the encoded first subframe, at least one third intra-frame stream, and the encoded second subframe. The current frame can also be encoded by using the encoded first subframe, the encoded second subframe, and the encoded third subframe.

[0313] According to embodiments of this disclosure, when multiple subframes include a fourth subframe, in order to encode the fourth subframe, the image encoding apparatus 2100 can obtain at least one fourth intra-frame stream for the first subframe to be encoded, at least one fifth intra-frame stream for the second subframe to be encoded, and at least one sixth intra-frame stream for the third subframe to be encoded. The fourth subframe can be encoded by using at least one fourth intra-frame stream, the encoded first subframe, at least one fifth intra-frame stream, the encoded second subframe, at least one sixth intra-frame stream, and the encoded third subframe. The current frame can also be encoded by using the encoded first subframe, the encoded second subframe, the encoded third subframe, and the encoded fourth subframe.

[0314] During operation S2060, the image encoding device 2100 can send a bit stream including feature data of the current frame.

[0315] Figure 21 This is a diagram illustrating the configuration of an image encoding apparatus according to an embodiment of the present disclosure.

[0316] Reference Figure 21 The image encoding device 2100 may include a first AI processor 2110, a second AI processor 2120, an intra-frame stream / gating acquirer 2130, an intra-frame compensator 2140, a decoder 2150, and an encoder 2160.

[0317] The first AI processor 2110, the second AI processor 2120, the intra-stream / gating acquirer 2130, the intra-compensator 2140, the decoder 2150, and the encoder 2160 may be implemented as processors. A processor may include at least one processing circuit and / or multiple processors. For example, the term "processor" as used herein, including the claims, may include various processing circuits that include at least one processor. In at least one processor, one or more processors may be configured to perform the various functions described herein individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms cover, but are not limited to, situations where one processor performs some functions while other processors perform other functions, and situations where a single processor can perform all functions. Furthermore, at least one processor may include a combination of processors with various functions for performing the disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions. The receiver 1910, AI processor 1920, intra stream / gating acquirer 1930, intra compensator 1940, and decoder 1950 can operate according to instructions stored in memory (not shown).

[0318] Despite Figure 21The first AI processor 2110, the second AI processor 2120, the intra-stream / gating acquirer 2130, the intra-frame compensator 2140, the decoder 2150, and the encoder 2160 are shown separately, but they can be implemented by a single processor. In this case, the first AI processor 2110, the second AI processor 2120, the intra-stream / gating acquirer 2130, the intra-frame compensator 2140, the decoder 2150, and the encoder 2160 can be implemented as a dedicated processor, or by a combination of software and a general-purpose processor (such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU)). Furthermore, when using a dedicated processor, it may include memory for implementing embodiments of this disclosure or a memory processor for using external memory.

[0319] The first AI processor 2110, the second AI processor 2120, the intra-stream / gating acquirer 2130, the intra-frame compensator 2140, the decoder 2150, and the encoder 2160 may include multiple processors. In this case, the first AI processor 2110, the second AI processor 2120, the intra-stream / gating acquirer 2130, the intra-frame compensator 2140, the decoder 2150, and the encoder 2160 may be implemented as a combination of dedicated processors, or may be implemented by a combination of software and multiple general-purpose processors (such as AP, CPU, or GPU).

[0320] The first AI processor 2110 can receive the current frame as input and output the feature data of the current frame.

[0321] The feature data of the current frame can be sent to the second AI processor 2120 and the intra-frame stream / gating acquirer 2130.

[0322] The second AI processor 2120 can encode the first subframe among multiple subframes according to a predetermined decoding order.

[0323] The intra-stream / gating acquirer 2130 can output the intra-stream and intra-gating of the second subframe for the first subframe.

[0324] The intra-frame compensator 2140 can receive the intra-frame stream and intra-frame gating of the second subframe for the first subframe from the intra-frame stream / gating acquirer 2130, and can receive the first subframe from the AI ​​processor 2120.

[0325] The intra-frame compensator 2140 can obtain the second sub-predicted frame based on the first sub-frame and the intra-frame gating and intra-frame stream of the second sub-frame for the first sub-frame.

[0326] The intra-frame compensator 2140 can send the second sub-prediction frame to the second AI processor 2120.

[0327] The second AI processor 2120 can encode the second subframe based on the feature data of the current frame and the second sub-predicted frame.

[0328] According to embodiments of this disclosure, the second AI processor 2120 may include an intra-frame stream / gating obtainr 2130 and an intra-frame compensator 2140. Therefore, within the second AI processor 2120, a first subframe among multiple subframes can be decoded according to a predetermined decoding order; the intra-frame stream and intra-frame gating of the second subframe relative to the first subframe can be output; a second sub-predicted frame can be obtained based on the first subframe and the intra-frame stream and intra-frame gating of the second subframe relative to the first subframe; and the second subframe can be decoded based on the feature data of the current frame and the second sub-predicted frame.

[0329] Decoder 2150 can obtain the reconstructed frame of the current frame based on the first subframe and the second subframe.

[0330] According to embodiments of this disclosure, when multiple subframes include a third subframe, in order to decode the third subframe, the intra-stream / gating obtainr 2130 can obtain at least one second intra-stream for the first subframe and at least one third intra-stream for the second subframe. The intra-compensator 2140 can obtain a third sub-predicted frame based on at least one second intra-stream, the first subframe, at least one third intra-stream, and the second subframe, and can send the third sub-predicted frame to the second AI processor 2120. The second AI processor 2120 can decode the third subframe based on the feature data of the current frame and the third sub-predicted frame, and the decoder 2150 can obtain the reconstructed frame of the current frame based on the first subframe, the second subframe, and the third subframe.

[0331] According to embodiments of this disclosure, when multiple subframes include a fourth subframe, in order to decode the fourth subframe, the intra-stream / gating obtainr 2130 can obtain at least one fourth intra-stream for the first subframe, at least one fifth intra-stream for the second subframe, and at least one sixth intra-stream for the third subframe. The intra-compensator 2140 can obtain the fourth sub-predicted frame based on at least one fourth intra-stream, the first subframe, at least one fifth intra-stream, the second subframe, at least one sixth intra-stream, and the third subframe, and can send the fourth sub-predicted frame to the second AI processor 2120. The second AI processor 2120 can decode the fourth subframe based on the feature data of the current frame and the fourth sub-predicted frame, and the decoder 2150 can obtain the reconstructed frame of the current frame based on the first subframe, the second subframe, the third subframe, and the fourth subframe.

[0332] Encoder 2160 can generate a bitstream that includes data from the current frame based on the reconstructed frame.

[0333] The first AI processor 2110, the second AI processor 2120, and the intra-stream / gating acquirer 2130 can be implemented as a neural network including one or more layers (e.g., convolutional layers).

[0334] The first AI processor 2110, the second AI processor 2120, and the intra-stream / gating acquirer 2130 can be stored in memory. The first AI processor 2110, the second AI processor 2120, and the intra-stream / gating acquirer 2130 can be implemented as at least one dedicated processor for AI.

[0335] An image decoding method according to an embodiment of the present disclosure may include: obtaining feature data of a current frame from a bitstream; decoding a first subframe of a plurality of subframes rearranged from the current frame according to a predetermined decoding order based on the feature data of the current frame; obtaining at least one intra-frame stream for the decoded first subframe to decode a second subframe of the plurality of subframes; decoding the second subframe using at least one intra-frame stream and the decoded first subframe; and reconstructing the current frame using the decoded first subframe and the decoded second subframe.

[0336] The image decoding method according to embodiments of the present disclosure can effectively reconstruct an image by dividing the current frame into subframes, progressively decoding each subframe, using the decoded subframes as reference subframes, and using intra-frame streams between subframes.

[0337] According to embodiments of this disclosure, multiple subframes can be obtained by dividing the current frame into multiple pixel groups and rearranging pixels at the same position in the multiple pixel groups so that they are included in the same subframe.

[0338] The image decoding method according to embodiments of the present disclosure can effectively reconstruct the image by dividing the current frame into multiple pixel groups, using a decoded subframe formed by collecting and rearranging pixels at the same position in multiple pixel groups as a reference subframe, and using intra-frame streams between subframes.

[0339] According to embodiments of the present disclosure, at least one intra-stream of the decoded first subframe may indicate a reference pixel in the decoded first subframe for a first pixel in the second subframe.

[0340] The image decoding method according to embodiments of the present disclosure can efficiently reconstruct an image by using intra-frame streams between multiple subframes rearranged from the current frame.

[0341] The image decoding method according to embodiments of the present disclosure may further include: obtaining an intra-frame gating value for at least one intra-frame stream, wherein a second subframe is decoded using at least one intra-frame stream, the intra-frame gating value, and a decoded first subframe, and the intra-frame gating value indicates a weight for at least one intra-frame stream.

[0342] The image decoding method according to embodiments of the present disclosure can effectively reconstruct an image by using an intra-frame stream between multiple sub-frames rearranged from the current frame and an intra-frame gating value indicating the reliability of the intra-frame stream.

[0343] According to embodiments of this disclosure, when at least one intra stream is multiple intra streams, an intra gating value corresponding to one of the multiple intra streams can be obtained, and a second subframe can be decoded using one of the multiple intra streams, the intra gating value corresponding to one of the multiple intra streams, and the decoded first subframe.

[0344] The image decoding method according to embodiments of the present disclosure can efficiently reconstruct an image by using multiple intra-frame streams between multiple sub-frames rearranged from the current frame and multiple intra-frame gating values ​​indicating the reliability of the multiple intra-frame streams.

[0345] According to embodiments of this disclosure, decoding a second subframe using at least one intra-frame stream and a first subframe that has been decoded may further include: obtaining a second sub-residual frame for the second subframe; obtaining a second sub-predicted frame using at least one intra-frame stream and a first subframe that has been decoded; and decoding the second subframe using the second sub-residual frame and the second sub-predicted frame.

[0346] The image decoding method according to embodiments of the present disclosure can effectively reconstruct an image by using intra-frame streams between subframes to obtain sub-predicted frames and adding the sub-predicted frames to the sub-residual frames.

[0347] The image decoding method according to embodiments of the present disclosure may further include: when a plurality of subframes include a third subframe, obtaining at least one second intra-frame stream for the first subframe to be decoded and at least one third intra-frame stream for the second subframe to be decoded, so as to decode the third subframe; decoding the third subframe by using at least one second intra-frame stream, the first subframe to be decoded, at least one third intra-frame stream and the second subframe to be decoded; and reconstructing the current frame by using the first subframe to be decoded, the second subframe to be decoded and the third subframe to be decoded.

[0348] The image decoding method according to embodiments of the present disclosure may further include: when a plurality of subframes include a fourth subframe, obtaining at least one fourth intra-frame stream for a first subframe to be decoded, at least one fifth intra-frame stream for a second subframe to be decoded, and at least one sixth intra-frame stream for a third subframe to be decoded, so as to decode the fourth subframe; decoding the fourth subframe by using at least one fourth intra-frame stream, the first subframe to be decoded, at least one fifth intra-frame stream, the second subframe to be decoded, at least one sixth intra-frame stream to be decoded, and the third subframe to be decoded; and reconstructing the current frame by using the first subframe to be decoded, the second subframe to be decoded, the third subframe to be decoded, and the fourth subframe to be decoded.

[0349] The image decoding method according to embodiments of the present disclosure can effectively reconstruct an image by dividing the current frame into subframes, progressively decoding each subframe, using the decoded subframes as reference subframes, and using intra-frame streams between subframes.

[0350] An image decoding apparatus according to an embodiment of the present disclosure may include: a memory storing one or more instructions; and at least one processor operating according to the one or more instructions, wherein the at least one processor obtains feature data of a current frame from a bitstream, decodes a first subframe of a plurality of subframes rearranged from the current frame according to a predetermined decoding order based on the feature data of the current frame, obtains at least one intra-frame stream for the decoded first subframe, decodes a second subframe of the plurality of subframes, decodes the second subframe using at least one intra-frame stream and the decoded first subframe, and reconstructs the current frame using the decoded first subframe and the decoded second subframe.

[0351] An image decoding apparatus according to embodiments of the present disclosure can effectively reconstruct an image by dividing the current frame into subframes, progressively decoding each subframe, using the decoded subframes as reference subframes, and using intra-frame streams between subframes.

[0352] According to embodiments of this disclosure, multiple subframes can be obtained by dividing the current frame into multiple pixel groups and rearranging pixels at the same position in the multiple pixel groups so that they are included in the same subframe.

[0353] An image decoding apparatus according to an embodiment of the present disclosure can effectively reconstruct an image by dividing the current frame into multiple pixel groups, using a decoded subframe formed by collecting and rearranging pixels at the same position in multiple pixel groups as a reference subframe, and using intra-frame streams between subframes.

[0354] According to embodiments of the present disclosure, at least one intra-stream of the decoded first subframe may indicate a reference pixel in the decoded first subframe for a first pixel in the second subframe.

[0355] An image decoding apparatus according to embodiments of the present disclosure can efficiently reconstruct an image by using intra-frame streams between multiple subframes rearranged from the current frame.

[0356] According to embodiments of this disclosure, at least one processor can obtain an intra-frame gating value for at least one intra-stream, a second subframe can be decoded using at least one intra-stream, the intra-frame gating value, and the decoded first subframe, and the intra-frame gating value can indicate the weight for at least one intra-stream.

[0357] An image decoding apparatus according to embodiments of the present disclosure can efficiently reconstruct an image by using an intra-frame stream between multiple subframes rearranged from the current frame and an intra-frame gating value indicating the reliability of the intra-frame stream.

[0358] According to embodiments of this disclosure, when at least one intra stream is multiple intra streams, an intra gating value corresponding to one of the multiple intra streams can be obtained, and a second subframe can be decoded using one of the multiple intra streams, the intra gating value corresponding to one of the multiple intra streams, and the decoded first subframe.

[0359] An image decoding apparatus according to embodiments of the present disclosure can efficiently reconstruct an image by using multiple intra-frame streams between multiple sub-frames rearranged from the current frame and multiple intra-frame gating values ​​indicating the reliability of the multiple intra-frame streams.

[0360] According to embodiments of the present disclosure, at least one processor can obtain a second sub-residual frame for a second sub-frame, obtain a second sub-predicted frame by using at least one intra-frame stream and a decoded first sub-frame, and decode the second sub-frame by using the second sub-residual frame and the second sub-predicted frame.

[0361] An image decoding apparatus according to embodiments of the present disclosure can effectively reconstruct an image by using intra-frame streams between subframes to obtain sub-predicted frames and adding the sub-predicted frames and sub-residual frames together.

[0362] According to embodiments of this disclosure, when multiple subframes include a third subframe, in order to decode the third subframe, at least one processor may obtain at least one second intra-frame stream for the first subframe to be decoded and at least one third intra-frame stream for the second subframe to be decoded. The third subframe may be decoded by using at least one second intra-frame stream, the first subframe to be decoded, at least one third intra-frame stream and the second subframe to be decoded, and the current frame may be reconstructed by using the first subframe to be decoded, the second subframe to be decoded and the third subframe to be decoded.

[0363] According to embodiments of this disclosure, when multiple subframes include a fourth subframe, in order to decode the fourth subframe, at least one processor may obtain at least one fourth intra-frame stream for the first subframe to be decoded, at least one fifth intra-frame stream for the second subframe to be decoded, and at least one sixth intra-frame stream for the third subframe to be decoded. The fourth subframe can be decoded by using at least one fourth intra-frame stream, the first subframe to be decoded, at least one fifth intra-frame stream, the second subframe to be decoded, at least one sixth intra-frame stream, and the third subframe to be decoded. The current frame can be reconstructed by using the first subframe to be decoded, the second subframe to be decoded, the third subframe to be decoded, and the fourth subframe to be decoded.

[0364] An image decoding apparatus according to embodiments of the present disclosure can effectively reconstruct an image by dividing the current frame into subframes, progressively decoding each subframe, using the decoded subframes as reference subframes, and using intra-frame streams between subframes.

[0365] An image encoding method according to an embodiment of the present disclosure may include: generating feature data of a current frame; encoding a first subframe of a plurality of subframes rearranged from the current frame according to a predetermined encoding order based on the feature data of the current frame; obtaining at least one intra-frame stream for the encoded first subframe to encode a second subframe of the plurality of subframes; encoding the second subframe by using at least one intra-frame stream and the encoded first subframe; encoding the current frame by using the encoded first subframe and the encoded second subframe; and transmitting a bit stream including the feature data of the current frame.

[0366] The image encoding method according to embodiments of the present disclosure can effectively encode an image by dividing the current frame into subframes, progressively encoding each subframe, using the encoded subframes as reference subframes, and using intra-frame streams between subframes.

[0367] According to embodiments of this disclosure, multiple subframes can be obtained by dividing the current frame into multiple pixel groups and rearranging pixels at the same position in the multiple pixel groups so that they are included in the same subframe.

[0368] The image encoding method according to embodiments of the present disclosure can effectively encode the image by dividing the current frame into multiple pixel groups, using an encoded subframe formed by collecting and rearranging pixels at the same position in multiple pixel groups as a reference subframe, and using intra-frame streams between subframes.

[0369] According to embodiments of the present disclosure, at least one intra-stream for an encoded first subframe may indicate a reference pixel in the encoded first subframe for a first pixel in a second subframe.

[0370] The image encoding method according to embodiments of the present disclosure can efficiently encode an image by using an intra-frame stream between multiple subframes rearranged from the current frame.

[0371] The image encoding method according to embodiments of the present disclosure may further include: obtaining an intra-frame gating value for at least one intra-frame stream, wherein a second subframe is encoded using at least one intra-frame stream, the intra-frame gating value, and an encoded first subframe, and the intra-frame gating value indicates a weight for at least one intra-frame stream.

[0372] The image encoding method according to embodiments of the present disclosure can efficiently encode an image by using an intra-frame stream between multiple sub-frames rearranged from the current frame and an intra-frame gating value indicating the reliability of the intra-frame stream.

[0373] According to embodiments of this disclosure, when at least one intra stream is multiple intra streams, an intra gating value corresponding to one of the multiple intra streams can be obtained, and a second subframe can be encoded using one of the multiple intra streams, the intra gating value corresponding to one of the multiple intra streams, and the encoded first subframe.

[0374] The image encoding method according to embodiments of the present disclosure can efficiently encode an image by using multiple intra-frame streams between multiple sub-frames rearranged from the current frame and multiple intra-frame gating values ​​indicating the reliability of the multiple intra-frame streams.

[0375] According to embodiments of this disclosure, encoding a second subframe using at least one intra-stream and encoded first subframe may further include: obtaining a second sub-residual frame for the second subframe; obtaining a second sub-predicted frame using at least one intra-stream and encoded first subframe; and encoding the second subframe using the second sub-residual frame and the second sub-predicted frame.

[0376] The image encoding method according to embodiments of the present disclosure can effectively encode an image by using intra-frame streams between subframes to obtain sub-predicted frames and adding the sub-predicted frames to the sub-residual frames.

[0377] The image encoding method according to embodiments of the present disclosure may further include: when a plurality of subframes include a third subframe: obtaining at least one second intra-frame stream for encoding a first subframe and at least one third intra-frame stream for encoding a second subframe, so as to encode the third subframe; encoding the third subframe by using at least one second intra-frame stream, the encoded first subframe, at least one third intra-frame stream, and the encoded second subframe; and encoding the current frame by using the encoded first subframe, the encoded second subframe, and the encoded third subframe.

[0378] The image encoding method according to embodiments of the present disclosure may further include: when a plurality of subframes include a fourth subframe: obtaining at least one fourth intra-frame stream for encoding a first subframe, at least one fifth intra-frame stream for encoding a second subframe, and at least one sixth intra-frame stream for encoding a third subframe, so as to encode the fourth subframe; encoding the fourth subframe by using at least one fourth intra-frame stream, the encoded first subframe, at least one fifth intra-frame stream, the encoded second subframe, at least one sixth intra-frame stream, and the encoded third subframe; and encoding the current frame by using the encoded first subframe, the encoded second subframe, the encoded third subframe, and the encoded fourth subframe.

[0379] The image encoding method according to embodiments of the present disclosure can effectively encode an image by dividing the current frame into subframes, progressively encoding each subframe, using the encoded subframes as reference subframes, and using intra-frame streams between subframes.

[0380] An image encoding apparatus according to an embodiment of the present disclosure may include: a memory storing one or more instructions; and at least one processor operating according to one or more instructions, wherein the at least one processor generates feature data of a current frame, encodes a first subframe of a plurality of subframes rearranged from the current frame according to a predetermined encoding order based on the feature data of the current frame, obtains at least one intra-frame stream for the encoded first subframe, encodes a second subframe of the plurality of subframes, encodes the second subframe using at least one intra-frame stream and the encoded first subframe, encodes the current frame using the encoded first subframe and the encoded second subframe, and transmits a bit stream including the feature data of the current frame.

[0381] The image encoding apparatus according to embodiments of the present disclosure can efficiently encode an image by dividing the current frame into subframes, progressively encoding each subframe, using the encoded subframes as reference subframes, and using intra-frame streams between subframes.

[0382] According to embodiments of this disclosure, multiple subframes can be obtained by dividing the current frame into multiple pixel groups and rearranging pixels at the same position in the multiple pixel groups so that they are included in the same subframe.

[0383] The image encoding apparatus according to embodiments of the present disclosure can efficiently encode an image by dividing the current frame into multiple pixel groups, using an encoded subframe formed by collecting and rearranging pixels at the same position in multiple pixel groups as a reference subframe, and using intra-frame streams between subframes.

[0384] According to embodiments of the present disclosure, at least one intra-stream for an encoded first subframe may indicate a reference pixel in the encoded first subframe for a first pixel in a second subframe.

[0385] The image encoding apparatus according to embodiments of the present disclosure can efficiently encode an image by using an intra-frame stream between multiple subframes rearranged from the current frame.

[0386] According to embodiments of the present disclosure, at least one processor may obtain an intra-frame gating value for at least one intra-stream, a second subframe may be encoded using at least one intra-stream, an intra-frame gating value, and an encoded first subframe, and the intra-frame gating value may indicate a weight for at least one intra-stream.

[0387] The image encoding apparatus according to embodiments of the present disclosure can efficiently encode an image by using an intra-frame stream between multiple sub-frames rearranged from the current frame and an intra-frame gating value indicating the reliability of the intra-frame stream.

[0388] According to embodiments of this disclosure, when at least one intra stream is multiple intra streams, an intra gating value corresponding to one of the multiple intra streams can be obtained, and a second subframe can be encoded using one of the multiple intra streams, the intra gating value corresponding to one of the multiple intra streams, and the encoded first subframe.

[0389] An image encoding apparatus according to embodiments of the present disclosure can efficiently encode an image by using multiple intra-frame streams between multiple sub-frames rearranged from the current frame and multiple intra-frame gating values ​​indicating the reliability of the multiple intra-frame streams.

[0390] According to embodiments of the present disclosure, at least one processor can obtain a second sub-residual frame for a second sub-frame, obtain a second sub-predicted frame by using at least one intra-frame stream and encoded first sub-frame, and encode the second sub-frame by using the second sub-residual frame and the second sub-predicted frame.

[0391] The image encoding apparatus according to embodiments of the present disclosure can efficiently encode an image by using an intra-frame stream between subframes to obtain a sub-predicted frame and adding the sub-predicted frame to the sub-residual frame.

[0392] According to embodiments of this disclosure, when multiple subframes include a third subframe, in order to encode the third subframe, at least one processor may obtain at least one second intra-frame stream for the encoded first subframe and at least one third intra-frame stream for the encoded second subframe. The third subframe may be encoded by using at least one second intra-frame stream, the encoded first subframe, at least one third intra-frame stream, and the encoded second subframe. The current frame may also be encoded by using the encoded first subframe, the encoded second subframe, and the encoded third subframe.

[0393] According to embodiments of this disclosure, when multiple subframes include a fourth subframe, in order to encode the fourth subframe, at least one processor may obtain at least one fourth intra-frame stream for the encoded first subframe, at least one fifth intra-frame stream for the encoded second subframe, and at least one sixth intra-frame stream for the encoded third subframe. The fourth subframe can be encoded by using at least one fourth intra-frame stream, the encoded first subframe, at least one fifth intra-frame stream, the encoded second subframe, at least one sixth intra-frame stream, and the encoded third subframe. The current frame can also be encoded by using the encoded first subframe, the encoded second subframe, the encoded third subframe, and the encoded fourth subframe.

[0394] The image encoding apparatus according to embodiments of the present disclosure can efficiently encode an image by dividing the current frame into subframes, progressively encoding each subframe, using the encoded subframes as reference subframes, and using intra-frame streams between subframes.

[0395] Machine-readable storage media may be provided in the form of non-transitory storage media. Here, "non-transitory" means that the storage medium does not include signals (e.g., electromagnetic waves) and is tangible, but does not distinguish whether the data is stored in the storage medium semi-permanently or temporarily. For example, "non-transitory storage media" may include buffers for temporarily storing data.

[0396] According to embodiments, methods according to various embodiments disclosed herein may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an app store, or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be stored at least temporarily in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a relay server), or may be temporarily generated.

Claims

1. An image decoding method, comprising: Obtain the feature data of the current frame from the bit stream (S1810). Based on the feature data of the current frame, the first subframe among the multiple subframes rearranged from the current frame is decoded according to a predetermined decoding order (S1820). Obtain at least one intra-frame stream for the first subframe to be decoded, so as to decode the second subframe among the plurality of subframes (S1830). Decoding the second subframe using the at least one intra-frame stream and the decoded first subframe (S1840); and The current frame is reconstructed by using the first and second subframes that have been decoded (S1850).

2. The image decoding method according to claim 1, wherein, The multiple subframes are obtained by dividing the current frame into multiple pixel groups and rearranging pixels at the same position in the multiple pixel groups so that they are included in the same subframe.

3. The image decoding method according to claim 1 or 2, wherein, The at least one intra-stream instruction for the first subframe being decoded indicates a reference pixel in the first subframe for the first pixel in the second subframe.

4. The image decoding method according to any one of claims 1 to 3, further comprising: Obtain the intra-frame gating value for the at least one intra-frame stream. The second subframe is decoded using the at least one intra-frame stream, the intra-frame gating value, and the decoded first subframe. The intra-frame gating value indicates the weight for the at least one intra-frame stream.

5. The image decoding method according to any one of claims 1 to 4, wherein, When the at least one intra stream is multiple intra streams The intra-frame gating value corresponding to one of the plurality of intra-frame streams is obtained, and The second subframe is decoded using one of the plurality of intra streams, an intra-gating value corresponding to the one of the plurality of intra streams, and the first subframe being decoded.

6. The image decoding method according to any one of claims 1 to 5, wherein, Decoding the second subframe using the at least one intra-frame stream and the decoded first subframe further includes: Obtain the second sub-residual frame for the second sub-frame; The second sub-predicted frame is obtained by using the at least one intra-frame stream and the decoded first sub-frame; and The second subframe is decoded using the second sub-residual frame and the second sub-predicted frame.

7. The image decoding method according to any one of claims 1 to 6, further comprising: When the plurality of subframes includes a third subframe: Obtain at least one second intra-frame stream for the first subframe to be decoded and at least one third intra-frame stream for the second subframe to be decoded, so as to decode the third subframe; The third subframe is decoded by using the at least one second intra-frame stream, the decoded first subframe, the at least one third intra-frame stream, and the decoded second subframe; as well as The current frame is reconstructed by using the first decoded subframe, the second decoded subframe, and the third decoded subframe.

8. The image decoding method according to claim 7, further comprising: When the plurality of subframes includes a fourth subframe: Obtain at least one fourth intra-frame stream for the first subframe to be decoded, at least one fifth intra-frame stream for the second subframe to be decoded, and at least one sixth intra-frame stream for the third subframe to be decoded. The fourth subframe is decoded by using the at least one fourth intra-frame stream, the decoded first subframe, the at least one fifth intra-frame stream, the decoded second subframe, the at least one sixth intra-frame stream, and the decoded third subframe. as well as The current frame is reconstructed by using the first decoded subframe, the second decoded subframe, the third decoded subframe, and the fourth decoded subframe.

9. An image encoding method, comprising: Generate feature data for the current frame (S2010). Based on the feature data of the current frame, the first subframe among multiple subframes rearranged from the current frame is encoded according to a predetermined encoding order (S2020). Obtain at least one intra-frame stream for encoding the first subframe, so as to encode the second subframe among the plurality of subframes (S2030). The second subframe is encoded by using the at least one intra-frame stream and the encoded first subframe (S2040). The current frame is encoded using the first encoded subframe and the second encoded subframe (S2050); and Send a bit stream including the feature data of the current frame (S2060).

10. The image encoding method according to claim 9, wherein, The multiple subframes are obtained by dividing the current frame into multiple pixel groups and rearranging pixels at the same position in the multiple pixel groups so that they are included in the same subframe.

11. The image encoding method according to claim 9 or 10, wherein, The at least one intra-stream for the encoded first subframe indicates a reference pixel in the encoded first subframe for the first pixel in the second subframe.

12. The image encoding method according to any one of claims 9 to 11, further comprising: Obtain the intra-frame gating value for the at least one intra-frame stream. The second subframe is encoded using the at least one intra-frame stream, an intra-frame gating value, and the encoded first subframe. The intra-frame gating value indicates the weight for the at least one intra-frame stream.

13. The image encoding method according to any one of claims 9 to 12, wherein, When the at least one intra stream is multiple intra streams The intra-frame gating value corresponding to one of the plurality of intra-frame streams is obtained, and The second subframe is encoded using one of the plurality of intra streams, an intra-gating value corresponding to the one of the plurality of intra streams, and the encoded first subframe.

14. The image encoding method according to any one of claims 9 to 13, further comprising: When the plurality of subframes includes a third subframe: Obtain at least one second intra-frame stream for the encoded first subframe and at least one third intra-frame stream for the encoded second subframe in order to encode the third subframe; The third subframe is encoded by using the at least one second intra-frame stream, the encoded first subframe, the at least one third intra-frame stream, and the encoded second subframe; as well as The current frame is encoded by using the first encoded subframe, the second encoded subframe, and the third encoded subframe.

15. The image encoding method according to claim 14, further comprising: When the plurality of subframes includes a fourth subframe: Obtain at least one fourth intra-frame stream for the first subframe to be encoded, at least one fifth intra-frame stream for the second subframe to be encoded, and at least one sixth intra-frame stream for the third subframe to be encoded. The fourth subframe is encoded using at least one fourth intra-frame stream, an encoded first subframe, at least one fifth intra-frame stream, an encoded second subframe, at least one sixth intra-frame stream, and an encoded third subframe. as well as The current frame is encoded by using the first encoded subframe, the second encoded subframe, the third encoded subframe, and the fourth encoded subframe.

Images