Video encoding method and apparatus using implicit neural representation model

Abstract

The embodiment discloses a video coding method and device using an implicit neural representation model. According to the present embodiment, an image decoding device obtains a video neural representation (NeRV) model and configures an input of the NeRV model. The image decoding apparatus outputs the N blocks by applying the configured input to the NeRV model. Here, the N block corresponds to a residual block, a prediction block, a reconstruction block, or a loop filtering block of the current block. After decoding the type of the N block, the image decoding apparatus applies an image decoding process based on the type of the N block to the N block.

Description

Technical Field

[0001] This disclosure relates to video coding methods and apparatus using implicit neural representation models. Background Technology

[0002] The statements in this section are provided only as background information in relation to this disclosure and do not necessarily constitute prior art.

[0003] Because video data contains a large amount of data compared to audio or still image data, it requires significant hardware resources (including memory) to store or transmit video data without compression.

[0004] Therefore, encoders are typically used to compress and store or transmit video data. Decoders receive compressed video data, decompress the received compressed video data, and play the decompressed video data. Video compression technologies include H.264 / Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), with VVC offering approximately 30% or more improvement in decoding efficiency compared to HEVC.

[0005] However, as image size, resolution, and frame rate gradually increase, the amount of data to be encoded also increases. Therefore, new compression techniques are needed that offer higher decoding efficiency and improved image enhancement compared to existing compression techniques.

[0006] Recently, deep learning-based image processing techniques have been applied to existing decoding element technologies. In existing decoding techniques, compression techniques such as inter-frame prediction, intra-frame prediction, intra-loop filtering, and transforms can be improved using deep learning-based image processing techniques to enhance decoding efficiency. Typical application examples include inter-frame prediction based on virtual reference frames generated by deep learning, and intra-loop filtering based on denoising models.

[0007] Meanwhile, recent research has focused on implicit neural representation models for representing various data in images, including those within neural network structures. Traditional video representation methods use explicit representations with RGB pixel values ​​for each pixel location. To replace this explicit representation, implicit neural representations have been introduced, using neural networks to represent the function used to transform the (x, y) coordinates of video pixel locations into (r, g, b) values. Compared to explicit representations, implicit neural representations can be used regardless of image resolution. Therefore, there is a need to apply implicit neural representations to video decoding to increase decoding efficiency and enhance video quality. Summary of the Invention

[0008] Technical issues

[0009] This disclosure seeks to provide a video decoding method and apparatus utilizing an implicit neural representation model operating in N blocks or N frames. Here, N blocks and N frames refer to blocks and frames, respectively, that are decoded using the implicit neural representation model.

[0010] This disclosure attempts to provide a method and apparatus for video decoding that uses an implicit neural representation model to perform prediction or in-loop filtering of N blocks or N frames.

[0011] Technical solution

[0012] At least one aspect of this disclosure provides a method for reconstructing a current block using a video decoding device. The method includes obtaining a video neural representation model (NeRV model). The method further includes constructing an input for the NeRV model. The method further includes applying the input to the NeRV model to output N blocks, the N blocks corresponding to a residual block, a predicted block, a reconstructed block, or a loop-filtered block of the current block. The method further includes decoding the type of the N blocks from a bitstream. The method further includes applying a video decoding process to the N blocks based on their type.

[0013] Another aspect of this disclosure provides a method for encoding a current block by a video encoding device. The method includes applying a video encoding process to the original block of the current block to generate a first block, the first block being a loop-filtered block of the current block. The method also includes constructing input for a video neural representation model (NeRV model). The method further includes constructing the NeRV model based on the input. The method also includes applying the input to the NeRV model to output N blocks, the N blocks corresponding to a residual block, a predicted block, a reconstructed block, or a loop-filtered block of the current block. The method further includes applying a video decoding process to the N blocks based on their type to generate a second block corresponding to the loop-filtered block of the current block.

[0014] Another aspect of this disclosure provides a computer-readable recording medium for storing a bitstream generated by a video coding method. The video coding method includes applying a video coding process to an original block of a current block to generate a first block, the first block being a loop-filtered block of the current block. The video coding method also includes constructing input for a video neural representation model (NeRV model). The video coding method further includes constructing the NeRV model based on the input. The video coding method further includes applying the input to the NeRV model to output N blocks, the N blocks corresponding to a residual block, a predicted block, a reconstructed block, or a loop-filtered block of the current block. The video coding method also includes applying a video decoding process to the N blocks based on their type to generate a second block corresponding to the loop-filtered block of the current block.

[0015] Beneficial effects

[0016] As described above, this disclosure provides a video decoding method and apparatus utilizing an implicit neural representation model operating in N blocks or N frames. Therefore, the video decoding method and apparatus improve video decoding efficiency by transmitting the additional information required for decoding N blocks and N frames.

[0017] Furthermore, this disclosure provides a video decoding method and apparatus that uses an implicit neural representation model to perform prediction or in-loop filtering of N blocks or N frames. Therefore, the video decoding method and apparatus enhance video quality. Attached Figure Description

[0018] Figure 1 This is a block diagram of a video encoding device that can implement the technology disclosed herein.

[0019] Figure 2 A method for partitioning blocks using a quadtree plus binary tree ternary tree (QTBTTT) structure is shown.

[0020] Figure 3a and Figure 3b Several intra-prediction modes, including the wide-angle intra-prediction mode, are shown.

[0021] Figure 4 The adjacent blocks of the current block are shown.

[0022] Figure 5 This is a block diagram of a video decoding device that can implement the technology disclosed herein.

[0023] Figure 6 This is a diagram illustrating the operations of a convolutional layer.

[0024] Figure 7 This is a diagram illustrating a Single Image Super-Resolution (SISR) network.

[0025] Figure 8 This is a diagram showing the residual blocks used in SISR.

[0026] Figure 9 This is a diagram illustrating a fixed-coefficient in-loop filter based on a convolutional neural network (CNN).

[0027] Figure 10 This is a diagram illustrating the Neural Representation of Video (NeRV) model.

[0028] Figure 11a and Figure 11b This is a diagram illustrating the concept of a NeRV model according to at least one embodiment of the present disclosure.

[0029] Figure 12 This is a diagram illustrating the coding pipeline of a NeRV model according to at least one embodiment of the present disclosure.

[0030] Figure 13 This is a diagram illustrating a decoding pipeline for a NeRV model according to at least one embodiment of the present disclosure.

[0031] Figure 14 This is a diagram illustrating a block-based NeRV model according to at least one embodiment of the present disclosure.

[0032] Figure 15 This is a diagram illustrating a block-based NeRV model according to another embodiment of the present disclosure.

[0033] Figure 16 This is a diagram illustrating a frame-based NeRV model according to at least one embodiment of the present disclosure.

[0034] Figure 17 This is a diagram illustrating the application of a NeRV model according to at least one embodiment of the present disclosure.

[0035] Figure 18 This is a diagram illustrating the application of a NeRV model according to another embodiment of this disclosure.

[0036] Figure 19 This is a diagram illustrating the application of the NeRV model according to yet another embodiment of the present disclosure.

[0037] Figure 20 This is a diagram illustrating the application of the NeRV model according to yet another embodiment of the present disclosure.

[0038] Figure 21 This is a flowchart of a method for encoding the current block by a video encoding device according to at least one embodiment of the present disclosure.

[0039] Figure 22 This is a flowchart of a method for reconstructing the current block by a video decoding device according to at least one embodiment of the present disclosure. Detailed Implementation

[0040] In the following description, some embodiments of the present disclosure are described in detail with reference to the accompanying drawings. In the following description, the same reference numerals denote the same elements, although the elements are shown in different drawings. Furthermore, in the following description of some embodiments, detailed descriptions of relevant known components and functions may be omitted for clarity and brevity, where such omissions would be considered to obscure the subject matter of the disclosure.

[0041] Figure 1 This is a block diagram of a video encoding device capable of implementing the technology disclosed herein. In the following, reference is made to... Figure 1 The diagram illustrates a video encoding device and its components.

[0042] The encoding device may include an image segmenter 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a rearrangement unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filter unit 180, and a memory 190.

[0043] Each component of the encoding device can be implemented as hardware, software, or a combination of both. Furthermore, the functionality of each component can be implemented as software, and a microprocessor can be implemented to execute the software functionality corresponding to each component.

[0044] A video consists of one or more sequences of multiple images. Each image is segmented into multiple regions, and each region is encoded. For example, an image is segmented into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile and / or slice is segmented into one or more coding tree units (CTUs). Furthermore, each CTU is segmented into one or more coding units (CUs) through a tree structure. Information applied to each coding unit (CU) is encoded as the syntax of the CU, and information commonly applied to all CUs included in a CTU is encoded as the syntax of the CTU. Furthermore, information commonly applied to all blocks in a slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more images is encoded as a picture parameter set (PPS) or picture header. Furthermore, information commonly referenced by multiple images is encoded into a sequence parameter set (SPS). Furthermore, information commonly referenced by one or more SPSs is encoded into a video parameter set (VPS). Furthermore, information commonly applied to a tile or tile group can also be encoded as the syntax of a tile or tile group header. Syntax included in SPS, PPS, slice headers, tiles, or tile group headers can be referred to as advanced syntax.

[0045] Image splitter 110 determines the size of the coding tree unit (CTU). Information about the size of the CTU (CTU size) is encoded into SPS or PPS syntax and delivered to the video decoding device.

[0046] Image segmenter 110 segments each image constituting the video into multiple coding tree units (CTUs) of predetermined size, and then recursively segments the CTUs using a tree structure. The leaf nodes in the tree structure are called coding units (CUs), which are the basic units of coding.

[0047] The tree structure can be a quadtree (QT), in which a higher node (or parent node) is divided into four lower nodes (or child nodes) of the same size. The tree structure can also be a binary tree (BT), in which a higher node is divided into two lower nodes. The tree structure can also be a ternary tree (TT), in which a higher node is divided into three lower nodes in a 1:2:1 ratio. The tree structure can also be a combination of two or more of the following structures: QT, BT, and TT. For example, a quadtree plus binary tree (QTBT) structure or a quadtree plus binary ternary tree (QTBTTT) structure can be used. Here, adding binary ternary tree (BTTT) to the tree structure is called a multi-type tree (MTT).

[0048] Figure 2 This is a diagram illustrating a method for segmenting blocks using a QTBTTT structure.

[0049] like Figure 2 As shown, firstly, the CTU can be partitioned into a QT structure. Quadtree partitioning can be recursive until the size of the partitioned block reaches the minimum block size (MinQTSize) allowed for a leaf node in QT. A first flag (QT_split_flag) indicating whether each node in the QT structure is partitioned into the four nodes below it is encoded by the entropy encoder 155 and signaled to the video decoding device. Leaf nodes can also be partitioned into at least one of a BT structure or a TT structure when the leaf node of the QT is not larger than the maximum block size (MaxBTSize) allowed for the root node in BT. Multiple partitioning directions can exist in the BT structure and / or TT structure. For example, two directions can exist: one where the block of the corresponding node is partitioned horizontally and another where the block of the corresponding node is partitioned vertically. Figure 2 As shown, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether a node has been split, a flag indicating the splitting direction (vertical or horizontal), and / or a flag indicating the splitting type (binary or ternary) if the node has been split are encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0050] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node has been split into the four lower-level nodes, the CU split flag (split_cu_flag) indicating whether a node has been split can also be encoded. When the value of the CU split flag (split_cu_flag) indicates that each node has not been split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU as the basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates that each node has been split, the video encoding device first begins encoding the first flag using the above scheme.

[0051] When QTBT is used as another example of a tree structure, two types can exist: one where the corresponding node's block is horizontally split into two blocks of the same size (i.e., symmetrical horizontal split), and another where the corresponding node's block is vertically split into two blocks of the same size (i.e., symmetrical vertical split). A split flag (split_flag) indicating whether each node in the BT structure is split into blocks of the lower level, and split type information indicating the split type, are encoded by an entropy encoder 155 and delivered to the video decoding device. Additionally, it is possible to present a type where the corresponding node's block is split into two blocks that are asymmetrically divided. Asymmetrical forms can include those where the corresponding node's block is split into two rectangular blocks with a 1:3 size ratio, or those where the corresponding node's block is split along a diagonal direction.

[0052] The CU can have different sizes depending on the QTBT or QTBTTT segmentation from the CTU. In the following text, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block". Due to the QTBTTT segmentation, the current block can be rectangular in shape in addition to a square shape.

[0053] Predictor 120 predicts the current block to generate a prediction block. Predictor 120 includes an intra-frame predictor 122 and an inter-frame predictor 124.

[0054] Typically, predictive decoding can be performed on each current block in an image. Prediction of the current block can usually be performed using either intra-frame prediction (using data from the image containing the current block) or inter-frame prediction (using data from an image decoded before the image containing the current block). Inter-frame prediction includes both one-way and two-way prediction.

[0055] Intra-predictor 122 predicts pixels in the current block using pixels (reference pixels) located around the current block in the current image, including the current block. Depending on the prediction direction, multiple intra-prediction modes exist. For example, such as... Figure 3aAs shown, multiple intra-frame prediction modes can include two non-directional modes (including a planar mode and a DC mode) and can include 65 directional modes. The neighboring pixels and the arithmetic equations to be used are defined differently for each prediction mode.

[0056] To perform efficient orientation prediction for the current block with a rectangular shape, additional methods such as... can be used. Figure 3b The dashed arrows indicate the directional modes (intra-prediction modes #67 to #80, #-1 to #-14). These directional modes can be referred to as "wide-angle intra-prediction modes." Figure 3b In the diagram, the arrow indicates the corresponding reference sample used for prediction but does not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrow. When the current block has a rectangular shape, the wide-angle intra-frame prediction mode is a mode that performs prediction in the opposite direction to the specific direction mode without additional bit transmission. In this case, in the wide-angle intra-frame prediction mode, some wide-angle intra-frame prediction modes that can be used for the current block can be determined by the ratio of the width to the height of the current block with a rectangular shape. For example, when the current block has a rectangular shape with a height less than its width, wide-angle intra-frame prediction modes with an angle less than 45 degrees (intra-frame prediction modes #67 to #80) are available. When the current block has a rectangular shape with a width greater than its height, wide-angle intra-frame prediction modes with an angle greater than -135 degrees are available.

[0057] Intra-predictor 122 can determine the intra-prediction to be used for encoding the current block. In some examples, intra-predictor 122 can encode the current block using multiple intra-prediction modes, and can also select the appropriate intra-prediction mode to be used from test modes. For example, intra-predictor 122 can calculate rate-distortion values ​​using rate-distortion analysis of intra-prediction modes for multiple tests, and can also select the intra-prediction mode with the best rate-distortion characteristics from the test modes.

[0058] Intra-predictor 122 selects one of multiple intra-prediction modes and predicts the current block using neighboring pixels (reference pixels) and an arithmetic equation determined according to the selected intra-prediction mode. Information about the selected intra-prediction mode is encoded by entropy encoder 155 and delivered to video decoding device.

[0059] Inter-frame predictor 124 generates a predicted block for the current block using motion compensation processing. Inter-frame predictor 124 searches for the most similar block to the current block in a reference image that was encoded and decoded earlier than the current image, and generates a predicted block for the current block using the searched block. Furthermore, a motion vector (MV) is generated, corresponding to the displacement between the current block in the current image and the predicted block in the reference image. Typically, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma and chroma components. Motion information, including information about the reference image and information about the motion vector used to predict the current block, is encoded by entropy encoder 155 and delivered to the video decoding device.

[0060] The inter-frame predictor 124 can also interpolate a reference image or reference block to increase prediction accuracy. In other words, it interpolates subsamples between two consecutive integer samples by applying filter coefficients to multiple consecutive integer samples, including two integer samples. When performing a process to search for the most similar block to the current block against the interpolated reference image, motion vectors can be represented with decimal precision instead of integer sample precision. The precision or resolution of the motion vectors can be set differently for each target region to be encoded (e.g., a cell such as a slice, tile, CTU, CU, etc.). When this adaptive motion vector resolution (AMVR) is applied, information about the motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, information about the motion vector resolution applied to each CU is signaled. The information about the motion vector resolution can be information representing the precision of the motion vector difference, which will be described below.

[0061] Simultaneously, the inter-frame predictor 124 can perform inter-frame prediction using bidirectional prediction. In the case of bidirectional prediction, two reference images and two motion vectors representing the location of the block most similar to the current block in each reference image are used. The inter-frame predictor 124 selects a first reference image and a second reference image from reference image list 0 (RefPicList0) and reference image list 1 (RefPicList1), respectively. The inter-frame predictor 124 also searches for the block most similar to the current block in the corresponding reference image to generate a first reference block and a second reference block. Furthermore, the predicted block for the current block is generated by averaging or weighted averaging the first reference block and the second reference block. In addition, motion information including information about the two reference images used to predict the current block and information about the two motion vectors is delivered to the entropy encoder 155. Here, reference image list 0 may consist of images in the pre-reconstructed images that appear before the current image in display order, and reference image list 1 may consist of images in the pre-reconstructed images that appear after the current image in display order. However, while not specifically limited to this, pre-reconstructed images following the current image in display order may be additionally included in reference image list 0. Conversely, pre-reconstructed images preceding the current image may also be additionally included in reference image list 1.

[0062] Various methods can be used to minimize the number of bits consumed in encoding motion information.

[0063] For example, when the reference image and motion vector of the current block are the same as those of the neighboring blocks, the information that can identify the neighboring blocks is encoded to deliver the motion information of the current block to the video decoding device. This method is called merge mode.

[0064] In merge mode, the inter-frame predictor 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as "merge candidates") from the neighboring blocks of the current block.

[0065] As adjacent blocks used to derive merge candidates, such as Figure 4 As shown, some or all of the following blocks in the current image that are adjacent to the current block: left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2 can be used. Furthermore, in addition to the current image containing the current block, blocks located in the reference image (which may be the same as or different from the reference image used to predict the current block) can also be used as merge candidates. For example, blocks located in the same position as the current block within the reference image, or blocks adjacent to blocks located in the same position, can also be used as merge candidates. If the number of merge candidates selected by the method described above is less than a preset number, a zero vector is added to the merge candidates.

[0066] The inter-frame predictor 124 uses a merge list including a predetermined number of merge candidates by configuring adjacent blocks. It selects a merge candidate from the merge candidates included in the merge list to be used as motion information for the current block, and generates merge index information to identify the selected candidate. The generated merge index information is encoded by the entropy encoder 155 and delivered to the video decoding device.

[0067] Merge skip mode is a special case of merging mode. After quantization, when all transform coefficients used for entropy coding are close to zero, only the adjacent block selection information is transmitted, and the residual signal is not transmitted. By using merge skip mode, relatively high coding efficiency can be achieved for images with slight motion, still images, and screen content images.

[0068] In the following text, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0069] Another method for encoding motion information is the Advanced Motion Vector Prediction (AMVP) pattern.

[0070] In AMVP mode, the inter-frame predictor 124 derives motion vector predictor candidates for the motion vector of the current block by using neighboring blocks of the current block. As neighboring blocks used for deriving the motion vector predictor candidates, they can use... Figure 4 The current block in the current image shown can be any or some of the following adjacent blocks: left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2. In addition to the current image containing the current block, blocks located within a reference image (which may be the same as or different from the reference image used to predict the current block) can also be used as adjacent blocks for deriving motion vector predictor candidates. For example, blocks located in the same position as the current block within the reference image, or blocks adjacent to blocks located in the same position, can be used. If the number of motion vector candidates selected by the above method is less than a preset number, a zero vector is added to the motion vector candidates.

[0071] The inter-frame predictor 124 derives motion vector predictor candidates using motion vectors from neighboring blocks, and determines the motion vector predictor for the current block using these candidates. Furthermore, the motion vector difference is calculated by subtracting the motion vector predictor from the motion vector of the current block.

[0072] Motion vector predictors can be obtained by applying a predefined function (e.g., center value and average value calculation) to motion vector predictor candidates. In this case, the video decoding device also knows the predefined function. Furthermore, since the neighboring blocks used to derive the motion vector predictor candidates are already encoded and decoded, the video decoding device may also already know the motion vectors of the neighboring blocks. Therefore, the video encoding device does not need to encode the information used to identify the motion vector predictor candidates. Thus, in this case, information about the motion vector difference and information about the reference image used to predict the current block are encoded.

[0073] Alternatively, motion vector predictors can be determined by selecting any one of the motion vector predictor candidates. In this case, the information used to identify the selected motion vector predictor candidate, along with information about the motion vector difference and information about the reference image used to predict the current block, are jointly encoded.

[0074] Subtractor 130 generates a residual block by subtracting the prediction block generated by intra-predictor 122 or inter-predictor 124 from the current block.

[0075] Transformer 140 converts the residual signal in the residual block, which has pixel values ​​in the spatial domain, into transform coefficients in the frequency domain. Transformer 140 can transform the residual signal in the residual block by using the total size of the residual block as a transform unit, or it can divide the residual block into multiple sub-blocks and perform the transformation by using the sub-blocks as transform units. Alternatively, the residual block is divided into two sub-blocks, a transform region and a non-transform region, so that the residual signal is transformed only by using the transform region sub-block as a transform unit. Here, the transform region sub-block can be one of two rectangular blocks with a 1:1 size ratio based on the horizontal (or vertical) axis. In this case, a flag (cu_sbt_flag) indicates that only the sub-block is transformed, and the orientation (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag) is encoded by entropy encoder 155 and signaled to the video decoding device. Furthermore, the size of the transform region sub-block can have a 1:3 size ratio based on the horizontal (or vertical) axis. In this case, the flag corresponding to the segmentation (cu_sbt_quad_flag) is further encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0076] Simultaneously, the transformer 140 can perform transformations on the residual block independently in the horizontal and vertical directions. Various types of transformation functions or transformation matrices can be used for the transformation. For example, a pair of transformation functions for horizontal and vertical transformations can be defined as a Multiple Transform Set (MTS). The transformer 140 can select the transformation function pair with the highest transformation efficiency in the MTS and can transform the residual block in each of the horizontal and vertical directions. Information about the transformation function pair in the MTS (mts_idx) is encoded by the entropy encoder 155 and transmitted as a signal to the video decoding device.

[0077] Quantizer 145 quantizes the transform coefficients output from transformer 140 using quantization parameters and outputs the quantized transform coefficients to entropy encoder 155. Quantizer 145 can also immediately quantize the associated residual block without applying it to any block or frame transform. Quantizer 145 can also apply different quantization coefficients (scaling values) depending on the position of the transform coefficients in the transform block. The quantization matrix applied to the quantized transform coefficients arranged in a 2D array can be encoded and signaled to the video decoding device.

[0078] The rearrangement unit 150 can perform recalibration of coefficient values ​​for quantized residual values.

[0079] The rearrangement unit 150 can transform a 2D coefficient array into a 1D coefficient sequence by using coefficient scanning. For example, the rearrangement unit 150 can output a 1D coefficient sequence by scanning DC coefficients into high-frequency domain coefficients using a zigzag scan or a diagonal scan. Depending on the size of the transform unit and the intra-frame prediction mode, a vertical scan scanning the 2D coefficient array along the column direction and a horizontal scan scanning the 2D block-type coefficients along the row direction can also be used instead of a zigzag scan. In other words, the scanning method to be used can be determined from zigzag scan, diagonal scan, vertical scan, and horizontal scan, depending on the size of the transform unit and the intra-frame prediction mode.

[0080] The entropy encoder 155 generates a bitstream by encoding a sequence of 1D quantized transform coefficients output from the rearrangement unit 150 using various encoding schemes, including context-based adaptive binary arithmetic code (CABAC) and exponential Golomb.

[0081] Furthermore, the entropy encoder 155 encodes block-segmentation related information (such as CTU size, CTU segmentation flag, QT segmentation flag, MTT segmentation type, MTT segmentation direction, etc.) to allow the video decoding device and the video encoding device to segment blocks equivalently. Additionally, the entropy encoder 155 encodes information indicating whether the current block is encoded using intra-frame prediction or inter-frame prediction. The entropy encoder 155 encodes intra-frame prediction information (i.e., information about the intra-frame prediction mode) or inter-frame prediction information (in the case of merge mode, merge index, and in the case of AMVP mode, information about the reference picture index and motion vector difference) based on the prediction type. Furthermore, the entropy encoder 155 encodes quantization-related information (i.e., information about quantization parameters and information about the quantization matrix).

[0082] Inverse quantizer 160 dequantizes the quantized transform coefficients output from quantizer 145 to generate transform coefficients. Inverse transformer 165 transforms the transform coefficients output from inverse quantizer 160 from the frequency domain to the spatial domain to reconstruct the residual block.

[0083] Adder 170 adds the reconstructed residual block to the predicted block generated by predictor 120 to reconstruct the current block. When performing intra-frame prediction for the next sequential block, the pixels in the reconstructed current block can be used as reference pixels.

[0084] The loop filter unit 180 performs filtering on the reconstructed pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc., that occur due to block-based prediction and transform / quantization. The loop filter unit 180, as an in-loop filter, may include all or some of the deblocking filter 182, the sample adaptive offset (SAO) filter 184, and the adaptive loop filter (ALF) 186.

[0085] Deblocking filter 182 filters the boundaries between reconstructed blocks to remove blocking artifacts caused by block unit encoding / decoding, and SAO filter 184 and ALF 186 perform additional filtering on the deblocked filtered video. SAO filter 184 and ALF 186 are filters used to compensate for differences between reconstructed pixels and original pixels caused by lossy decoding. SAO filter 184 applies an offset as a CTU unit to enhance subjective image quality and coding efficiency. On the other hand, ALF 186 performs block unit filtering and applies different filters to compensate for distortion by dividing the boundaries of corresponding blocks and the degree of variation. Information about the filter coefficients used for ALF can be encoded and transmitted as a signal to the video decoding device.

[0086] The reconstructed blocks, filtered by deblocking filter 182, SAO filter 184, and ALF 186, are stored in memory 190. When all blocks in an image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.

[0087] Video encoding devices can store the bitstream of encoded video data in a non-transitory storage medium or transmit the bitstream to a video decoding device via a communication network.

[0088] Figure 5 This is a functional block diagram of a video decoding device that can implement the technology of this disclosure. In the following, reference is made to... Figure 5 This describes the video decoding device and its components.

[0089] The video decoding device may include an entropy decoder 510, a rearrangement unit 515, an inverse quantizer 520, an inverse transformer 530, a predictor 540, an adder 550, a loop filter unit 560, and a memory 570.

[0090] and Figure 1 Similar to video encoding devices, each component of a video decoding device can be implemented as hardware, software, or a combination of both. Furthermore, the functionality of each component can be implemented as software, and a microprocessor can be implemented to execute the software functions corresponding to each component.

[0091] The entropy decoder 510 extracts information related to block segmentation by decoding the bitstream generated by the video encoding device to determine the current block to be decoded, and extracts the prediction information and information about the residual signal needed to reconstruct the current block.

[0092] The entropy decoder 510 determines the size of the CTU by extracting information about the CTU size from the sequence parameter set (SPS) or picture parameter set (PPS) and segments the image into CTUs of a defined size. Furthermore, the CTU is identified as the highest level of the tree structure, i.e., the root node, and segmentation information of the CTU can be extracted to segment the CTU using the tree structure.

[0093] For example, when segmenting a CTU using a QTBTTT structure, the first flag (QT_split_flag) related to the QT segmentation is first extracted to divide each node into four lower-level nodes. Furthermore, for nodes corresponding to leaf nodes of the QT, the second flag (mtt_split_flag), segmentation direction (vertical / horizontal), and / or segmentation type (binary / ternary) related to the MTT segmentation are extracted to segment the corresponding leaf node into an MTT structure. Therefore, each node below a leaf node of the QT is recursively segmented into a BT structure or a TT structure.

[0094] As another example, when splitting a CTU using a QTBTTT structure, a CU split flag (split_cu_flag) indicating whether a CU has been split is extracted. When the corresponding block is split, a first flag (QT_split_flag) can also be extracted. During the splitting process, for each node, zero or more recursive MTT splits can occur after zero or more recursive QT splits. For example, for a CTU, an MTT split might occur immediately, or conversely, only a few QT splits might occur.

[0095] As another example, when segmenting a CTU using a QTBT structure, the first flag (QT_split_flag) related to the QT split is extracted to divide each node into four nodes at the lower level. Additionally, a split flag (split_flag) indicating whether the node corresponding to a leaf node of the QT is further split into BTs, as well as splitting direction information, is extracted.

[0096] Simultaneously, when the entropy decoder 510 determines the current block to be decoded by using tree-structured segmentation, it extracts information about the prediction type, indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 510 extracts syntax elements representing the intra-prediction information (intra-prediction mode) for the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 extracts information representing syntax elements representing the inter-prediction information (i.e., motion vectors and reference images for motion vector references).

[0097] In addition, the entropy decoder 510 extracts quantization-related information of the current block and extracts information about the transform coefficients of the quantization of the current block as information about the residual signal.

[0098] The rearrangement unit 515 can again transform the sequence of 1D quantized transform coefficients entropy decoded by the entropy decoder 510 into a 2D coefficient array (i.e., a block) in the reverse order of the coefficient scan performed by the video encoding device.

[0099] The inverse quantizer 520 dequantizes the quantized transform coefficients by using quantization parameters. The inverse quantizer 520 can also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in a 2D array. The inverse quantizer 520 can perform dequantization by applying a matrix (scaling values) of quantization coefficients from the video encoding device to a 2D array of quantized transform coefficients.

[0100] The inverse transformer 530 reconstructs the residual signal by inversely transforming the dequantized transform coefficients from the frequency domain to the spatial domain to generate the residual block of the current block.

[0101] Furthermore, when the inverse transformer 530 performs an inverse transformation on a portion (sub-block) of the transform block, it extracts only the flag (cu_sbt_flag) indicating that the sub-block of the transform block has been transformed, the orientation (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or the position information (cu_sbt_pos_flag) of the sub-block. The inverse transformer 530 also inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to reconstruct the residual signal, and fills the untransformed regions with a value of "0" as the residual signal to generate the final residual block for the current block.

[0102] Furthermore, when applying MTS, the inverse transformer 530 determines the transform function or transform matrix to be applied in each direction, both horizontally and vertically, by using the MTS information (mts_idx) transmitted from the video encoding device signal. The inverse transformer 530 also performs an inverse transform on the transform coefficients in the transform block in both the horizontal and vertical directions using the determined transform function.

[0103] Predictor 540 may include intra-predictor 542 and inter-predictor 544. Intra-predictor 542 is activated when the prediction type of the current block is intra-prediction, and inter-predictor 544 is activated when the prediction type of the current block is inter-prediction.

[0104] Intra-predictor 542 determines the intra-prediction mode for the current block among multiple intra-prediction modes based on the syntax elements for the intra-prediction mode extracted from entropy decoder 510. Intra-predictor 542 also predicts the current block by using neighboring reference pixels of the current block according to the intra-prediction mode.

[0105] The inter-frame predictor 544 determines the motion vector and reference picture of the motion vector reference for the current block by using the syntax elements for the inter-frame prediction mode extracted from the entropy decoder 510, and predicts the current block by using the motion vector and the reference picture.

[0106] Adder 550 reconstructs the current block by adding the residual block output from inverse transformer 530 to the predicted block output from inter-frame predictor 544 or intra-frame predictor 542. Pixels within the reconstructed current block are used as reference pixels when performing intra-frame prediction for blocks to be decoded later.

[0107] The loop filter unit 560, serving as an in-loop filter, may include a deblocking filter 562, a SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundaries between reconstructed blocks to remove blocking artifacts caused by block unit decoding. The SAO filter 564 and ALF 566 perform additional filtering on the reconstructed blocks after deblocking filtering to compensate for differences between the reconstructed pixels and the original pixels caused by lossy decoding. The filter coefficients of the ALF are determined using information about the filter coefficients decoded from the bitstream.

[0108] The reconstructed blocks, filtered by deblocking filter 562, SAO filter 564, and ALF 566, are stored in memory 570. When all blocks in an image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.

[0109] In some embodiments, this disclosure relates to encoding and decoding video images as described above. More specifically, this disclosure provides a video decoding method and apparatus utilizing an implicit neural representation model operating on a unit of N blocks or N frames. Here, N blocks refer to blocks decoded using the implicit neural representation model, and N frames refer to frames decoded using the implicit neural representation model. Additionally, this disclosure provides a video decoding method and apparatus for performing prediction or in-loop filtering on N blocks or N frames using an implicit neural representation model.

[0110] The following implementation methods can be performed by multiple components in a video encoding device. The following implementation methods can also be performed by multiple components in a video decoding device.

[0111] When encoding the current block, the video encoding device can generate signaling information associated with this embodiment in terms of optimizing rate distortion. The video encoding device can encode the signaling information using the entropy encoder 155 and send the encoded signaling information to the video decoding device. The video decoding device can decode the signaling information associated with the decoding of the current block from the bitstream using the entropy decoder 510.

[0112] In the following description, the term "target block" may be used interchangeably with the current block or coding unit (CU), or may refer to some region of a coding unit.

[0113] Furthermore, a true value for a flag indicates when the flag is set to 1. Conversely, a false value for a flag indicates when the flag is set to 0.

[0114] I. Convolutional Neural Networks (CNN)

[0115] CNN refers to a neural network consisting of multiple convolutional and pooling layers, and is known as the most suitable deep learning technique for image processing. Convolutional layers use multiple kernels or filters to extract feature maps, also known as "features." The kernel coefficients that make up the filters are parameters determined during the learning process.

[0116] In the convolutional layers of a CNN, the front layers near the input extract feature maps that respond to simple, low-level image features (such as lines, points, or faces). The back layers near the output extract feature maps that respond to higher-level features (such as textures and object parts).

[0117] Figure 6 This is a diagram illustrating the operation of a convolutional layer according to at least one embodiment of the present disclosure.

[0118] Convolutional layers use convolution operations to generate feature maps from the input image. Figure 6 The example depicts a kernel (or filter) with a kernel size of 3×3. The kernel size is also called the filter size. The kernel has kernel parameters or filter parameters, also known as weights. Figure 6 The kernel shown has a total of nine kernel parameters. The kernel parameters can be initially set to random values, and their values ​​can be updated based on training.

[0119] Convolutional layers perform convolution operations using blocks equal to the kernel size in the input image. In this case, the block equal to the kernel size in the input image is called a window.

[0120] When filtering is performed on the input image according to the raster scan sequence, the amount of window movement is called the stride. Figure 6 In the example, the stride is 1. If the stride is set to 2, then a convolution operation that shifts the window by 2 samples is performed, resulting in the horizontal and vertical dimensions of the feature map being half the horizontal and vertical dimensions of the input image, respectively.

[0121] As mentioned above, a convolutional layer can include multiple filters. The number of filters, or kernels, is called the number of channels. That is, the number of channels equals the number of filters. The number of filters also determines the dimension of the feature map.

[0122] Padding refers to the method of expanding input data by adding specific values ​​around it before performing a convolution operation. Padding is typically used to control the spatial size of the output data. The values ​​used for padding can be determined by hyperparameters, but zero padding is commonly used. Without padding, the spatial size of the output data decreases with each pass through the convolutional layer, leading to a loss of boundary information. Therefore, padding is used to avoid this problem. Padding can be used to balance the spatial size of the output and input data of a convolutional layer.

[0123] The deconvolutional layer performs the inverse operation of the convolutional layer. It generates the desired data image as output from the feature map that serves as input.

[0124] Pooling layers perform pooling, which is the process of subsampling the feature maps generated by convolutional layers. Pooling layers use a 2×2 window to select samples such that the width and height of the output are half the width and height of the input, respectively. In other words, pooling layers are used to reduce the size of the input image or input feature map by clustering 2×2 regions into a single sample.

[0125] The opposite of a pooling layer is defined as an anti-pooling layer. In contrast to pooling layers, anti-pooling layers are used to add dimensions and are typically used after deconvolutional layers.

[0126] A convolutional encoder-decoder architecture is a network structure consisting of pairs of convolutional and deconvolutional layers. The convolutional encoder, composed of convolutional and pooling layers, outputs a feature map (or feature vector) from the input image. The final output vector of the convolutional encoder is also called the latent vector. The convolutional decoder, composed of deconvolutional and unpooling layers, generates an output image from the feature map or latent vector.

[0127] The inputs and outputs of a convolutional encoder-decoder can vary depending on the application and the purpose of the network. For example, the inputs and outputs can be optical flow maps, saliency maps, image frames, etc.

[0128] Figure 7 This is a diagram showing an SISR network.

[0129] An example application of CNNs is Single Image Super-Resolution (SISR). SISR networks generate high-resolution output images from low-resolution input images. For example... Figure 7 As shown, a SISR network can include multiple convolutional layers. Each convolutional layer includes an activation function, such as a rectified linear unit (ReLU). The parameters of the SISR network can be trained so that the resulting super-resolution (SR) image approximates the ground truth (GT).

[0130] SR methods using CNNs can improve SR performance by increasing depth (e.g., by increasing the number of convolutional layers). To overcome the overfitting problem that can occur during learning with increasing depth, SISR networks can utilize residual blocks capable of performing skipped connections and residual learning. Figure 8 As shown, residual blocks are used not only to apply convolution operations to the input feature x, but also... l The path also includes skip paths. Furthermore, when generating output x... l+1 At this time, residual blocks can choose between applying convolution operations or skipping paths based on learning efficiency. Figure 8 In the example, the residual block includes a batch normalization (BN) layer.

[0131] In one example, an enhanced deep residual network for SISR (EDSR) enhances network performance by increasing depth through continuous connection of residual blocks. Another example is Accurate Image Super-Resolution Using Very Deep Convolutional Networks (VDSR), which is based on a CNN model using a Visual Geometric Groups Network (VGG network) that employs residual learning to add residual frames to the final output. VDSR adds the residual signal to the very end of the network, thus adding the residual signal to the input signal.

[0132] As another example, CNNs can be used as in-loop filters in video encoding or decoding devices. In this case, deep learning-based in-loop filters can be employed at any location within an existing loop filter unit 180 or 560, which consists of a deblocking filter, a SAO filter, and an ALF.

[0133] Deep learning-based in-loop filters can be fixed-coefficient in-loop filters. These filters utilize common CNN kernel parameters stored on both the video encoding and decoding sides of the video encoding and decoding devices.

[0134] Figure 9 This is a diagram illustrating a fixed-coefficient in-loop filter based on a CNN.

[0135] Input blocks (or input frames) are passed through a normalized QP map to subsequent stages. The normalized QP map is used to reduce inference error during learning and inference when quantization noise of varying intensities is mixed. The kernel parameters that constitute dense residual units (DRUs) and convolutional layers can be stored and used identically in both video encoding and decoding devices. Figure 9 In the example, each DRU may include all or part of a convolutional layer, a ReLU layer, and a depthwise separable convolutional (DSC) layer.

[0136] Deep learning-based fixed-coefficient in-loop filters need to provide general performance across a wide range of video frames, and therefore they have the disadvantage of increasing the number of layers in the deeper CNN, thus correspondingly increasing the computation time.

[0137] II. Implicit Neural Manifestation Model

[0138] Recently, there has been active research into implicit neural representation models for representing various data in images, including neural network structures. Traditional video representation methods use explicit representations with RGB pixel values ​​for each pixel location. To replace this explicit representation, a new implicit neural representation method has been introduced, which uses a neural network to represent the function used to transform the (x, y) coordinates of video pixel locations into (r, g, b) values. Compared to explicit representations, implicit neural representations can be used regardless of image resolution. Therefore, for implicit neural representation models, multilayer perceptron (MLP) networks comprising multiple fully connected (FC) layers can be used.

[0139] An example of an implicit neural representation model is the implicit video neural representation. Hereinafter, the implicit video neural representation model will be referred to as the NeRV model. As mentioned above, explicit representation methods represent pixel values ​​on a (x, y, t) grid using pixel locations (x, y) in the image and the time index t of the frame. Implicit neural representation models output RGB pixel values ​​based on the input representing the (x, y, t) location. However, training a neural network at every (x, y, t) location significantly increases computational complexity, so the NeRV model uses a neural network structure that only uses the time index t as input, outputting the RGB image of the entire frame at time index t, as shown below. Figure 10 As shown.

[0140] Figure 11a and Figure 11b This is a diagram illustrating the concept of a NeRV model according to at least one embodiment of the present disclosure.

[0141] To facilitate providing image output for a time-indexed input (t), convolutional layers are more efficient than MLP (Multilayer Perceptron) networks that utilize existing implicit representation models. The NeRV model can generate output using a stacked structure of NeRV blocks consisting of multiple convolutional layers. For example... Figure 11a As shown, the outputs based on MLP and those based on NeRV blocks can be compared. Figure 11b As shown, a NeRV block includes convolutional layers, pixel rearrangement layers, and activation layers that perform convolutions. Figure 11b In the example, C represents the number of channels, W and H represent the width and height of the NeRV block input, and S represents the amplification factor.

[0142] Furthermore, the NeRV model embeds the time index t into a higher-dimensional space and uses the embedded index as input, as shown in Equation 1.

[0143] Equation 1

[0144] γ(t)=(sin(2 0 πt),cos(2 0 πt),…,sin(2 L-1 πt),cos(2 L-1 πt))

[0145] As shown in Equation 1, the time index t can be mapped to a 2L-dimensional vector γ(t). By utilizing the embedded time index t to train the NeRV model, this disclosure enables the NeRV model to better predict video data containing high-frequency variations.

[0146] As the loss function for the NeRV model, a combination of L1 loss and Structural Similarity Index (SSIM) loss can be used. During the training of the NeRV model, these losses can be computed at all pixel locations in the image estimated by the NeRV model based on the input and ground truth (GT) images (i.e., the original images). The L1 loss is computed using the absolute values ​​of the differences between pixels in the estimated image and the GT image. The SSIM loss is computed based on the mean, standard deviation, and correlation of pixels in the estimated and GT images. The weights (or parameters) constituting the NeRV model can then be updated based on the computed losses, and during the training process.

[0147] Figure 12 This is a diagram illustrating the coding pipeline of a NeRV model according to at least one embodiment of the present disclosure.

[0148] Since the NeRV model can be used to approximate the original video, end-to-end compression techniques can be used to compress and transmit the NeRV model using neural networks. The encoding pipeline for encoding the NeRV model may include all or part of a video overfitter 1210, a model trimmer 1220, a model quantizer 1230, and a weight encoder 1240, such as... Figure 12 As shown.

[0149] The video overfitter 1210 represents the input video frame using a NeRV model. The video encoding device trains the NeRV model using the aforementioned loss function to make the NeRV model represent the video frame. The model pruner 1220 uses pruning to simplify the NeRV model structure, which can be an MLP (Multilayer Perceptron) or a hybrid of MLP and convolutional layers. For example, weights less than a preset threshold are pruned to zero. The model quantizer 1230 quantizes the pruned weights, for example, weights greater than or equal to the threshold. The weight encoder 1240 applies entropy encoding to the quantized weights to generate a bitstream of weights.

[0150] exist Figure 12 The reverse order of the encoding pipeline shown in the figure, and the decoding pipeline for decoding the NeRV model, can be as follows: Figure 13 As shown. The decoding pipeline may include all or part of the weight decoder 1310, the model inverse quantizer 1320, the model reconstructor 1330, and the video generator 1340, such as Figure 13 As shown.

[0151] The weight decoder 1310 applies entropy encoding to the bitstream of the weights of the NeRV model to generate quantized weights. The model inverse quantizer 1320 dequantizes the quantized weights to generate pruned weights. The model reconstructor 1330 reconstructs the NeRV model based on the pruned weights. The video generator 1340 uses the reconstructed NeRV model according to the time index to generate reconstructed video frames.

[0152] As mentioned above, the NeRV model (Implicit Video Neural Representation model) uses the time index t of the frame as input. Therefore, for an input video with pixels of size t×h×w, the NeRV model samples the input video only t times instead of t×h×w times, which can lead to a significant gain in both encoding and decoding speed.

[0153] III. Embodiments according to this disclosure

[0154] In the following sections, this disclosure applies a NeRV-based neural network model to conventional video coding, such as VVC (Video Coding for Multipurpose Use). This disclosure includes the following to improve the performance of video decoding. First, the input format of the NeRV model is modified to be suitable for video decoding. Existing NeRV models are either trained using positional information of all pixels or trained without pixel positional information. However, video decoding, which processes signals block by block, requires training the NeRV model using a CTU (Code Tree Unit) or CU (Code Unit) and using the index of each training unit to extract the signal.

[0155] This disclosure can also be applied to coding techniques, such as in-loop filters and inter-frame prediction used by VVC. In this case, a single NeRV model can be used. Alternatively, to improve decoding efficiency, multiple NeRV models can be used, based on additional information about the decoding device, such as quantization parameters (QP), slice type, etc. This disclosure describes the use of NeRV models in the aforementioned coding techniques.

[0156] Conventional implicit neural representation models utilize pixel indices (x, y, t) as input, which significantly increases complexity. NeRV models can output the entire frame using the temporal index t as input, but may lose spatial information. Conventional video decoding devices (including VVC) segment video frames into blocks and process the signal within each block. Therefore, by utilizing the block indices to train the NeRV model, it can be applied more efficiently to existing decoding devices. In the following text, in addition to improving existing NeRV models for inputs including block indices (i, j), the term NeRV can refer to a more general approach for implicit neural representation models including block indices (i, j).

[0157] Figure 14 This is a diagram illustrating a block-based NeRV model according to at least one embodiment of the present disclosure.

[0158] In one example, the block-based NeRV model takes the index (i, j) and temporal index t of a video block containing one or more pixels as input to represent the signal in the video block corresponding to the block index. For example, in Figure 14 In the example, index (i, j) represents the index of the CTU or CU used by the video decoding device. Similar to regular NeRV, a block-based NeRV model is implemented using a combination of MLP and convolutional layers. As described above, as shown in Equation 1, the block index and temporal index are embedded in a high-dimensional space, and then the embedded block index and embedded temporal index are used as input to the NeRV model.

[0159] For example, video encoding devices are trained using the loss function described above. Figure 14 The block-based NeRV model shown is used to represent blocks using the NeRV model. The video encoding device transmits the block-based NeRV model to the video decoding device using the aforementioned encoding pipeline. The video decoding device reconstructs the block-based NeRV model using the aforementioned decoding pipeline. The video decoding device can then use... Figure 14 The block-based NeRV model shown in the figure is used to reconstruct the block corresponding to the block index.

[0160] Figure 15 This is a diagram illustrating a block-based NeRV model according to another embodiment of the present disclosure.

[0161] As another example, in addition to block indexing and temporal indexing, block-based NeRV models can also use local features of video frames extracted from deep learning-based extraction models as input. As local features, latent features including the features of the (i, j)th block or even the local features of its neighboring blocks are used. For example, neighboring blocks are blocks that have already been decoded, and in intra-frame prediction, they represent samples of blocks located to the left or above the current block. In inter-frame prediction, neighboring blocks represent blocks from earlier frames in the decoding sequence. Figure 15 In the example, the NeRV model additionally uses local features δ t As input, the local feature is the combined feature of the (i, j)th block and the features of the neighboring blocks.

[0162] For example, video encoding devices can be trained end-to-end. Figure 15 The extraction model and block-based NeRV model shown in the diagram enable the extraction model to extract local features and the NeRV model to represent blocks. A video encoding device transmits the block-based NeRV model to a video decoding device using the aforementioned encoding pipeline. Furthermore, the video encoding device encodes the local features associated with the blocks and transmits the encoded local features to the video decoding device. The video decoding device reconstructs the block-based NeRV model using the aforementioned decoding pipeline. Additionally, the video decoding device decodes the local features. The video decoding device can then input the block index, temporal index, and local features into the block-based NeRV model to reconstruct the block corresponding to the block index.

[0163] like Figure 12As shown, the NeRV model is encoded by the video encoding device using an encoding pipeline and then transmitted to the video decoding device. The parameters (i.e., weights) of the trained model are encoded after reducing their data size through model pruning, quantization, and entropy encoding. The NeRV model can be implemented using floating-point operations or fixed-point operations with N bits (e.g., 8, 16, 32, etc.) to allow versatility across heterogeneous devices such as central processing units (CPUs), graphics processing units (GPUs), etc. However, utilizing the NeRV model for each block significantly increases the overall size of the video bitstream. Therefore, the video encoding device can adaptively transmit the NeRV model for each block, rather than transmitting all NeRV models block by block. For example, in addition to blocks generated by prediction (such as I (intra-frame) blocks, P (prediction) blocks, B (bidirectional prediction) blocks, etc.), a new type of block called the N-block is defined. Instead of conventionally constructing blocks by combining prediction blocks and residual blocks based on intra-frame or inter-frame prediction, the video decoding device generates the signal of the N-block based on the received NeRV model. N blocks are represented by a block header whose type is assigned to either CU or CTU. The video decoding device decodes the parameters of the NeRV model and then uses the NeRV model to decode the N blocks.

[0164] Each N block can have parameters of the NeRV model. Furthermore, each N block can copy the parameters of the NeRV model from its neighboring N blocks, and the copied parameters can be used directly. Alternatively, the copied parameters can be used as initial values ​​for training each N block. In the merged mode of VVC inter-frame prediction, the video coding device copies the additional information from neighboring blocks and uses the copied additional information. The video coding device can then reduce the bit rate required to transmit additional information by transmitting only information about which neighboring blocks' indices will be used, instead of transmitting all the additional information needed for inter-frame prediction of the current block.

[0165] Similar to the VVC merge mode, by resolving only the indices indicating the previously decoded N blocks adjacent to the current block, the video decoding device can reconstruct the current N blocks using the parameters of the NeRV model of the previously decoded N blocks. In addition to models of spatially adjacent blocks, the video decoding device also uses models of spatially distant blocks.

[0166] The video coding device organizes and manages queues of parameters for the NeRV model associated with N blocks, and uses signaling to transmit indices for the queues used to send information about the NeRV model. Alternatively, the video coding device utilizes parameters of the NeRV model associated with the N blocks that are not adjacent but are spatially closest.

[0167] At the same time, if a frame consists entirely of N blocks, then the frame is defined as an N-frame. Alternatively, if a frame contains at least one N block, then the frame can be defined as an N-frame. Since frames can be further divided into slices, a slice consisting only of N blocks is defined as an N-slice. Alternatively, a slice containing at least one N block is defined as an N-slice.

[0168] Different NeRV models can be used for different N blocks within N frames. Alternatively, when N frames consist of only N blocks, a single NeRV model can be utilized. When using a single NeRV model, the input to the NeRV model does not require block-by-block indexing. That is, as with a regular NeRV model, the video coding device constructs the NeRV model using an input structure that includes only the time index t and not the block index (i, j), and then encodes the constructed NeRV model. In another example, while maintaining the input structure that utilizes block indexing, the video coding device constructs the input by filling all (i, j) with values ​​in the range that the existing (i, j) index cannot have (0, 0) or (e.g., negative values), and then uses the constructed input to train the NeRV model.

[0169] A video encoding device can apply one or more NeRV models to N blocks within N frames.

[0170] Video encoding devices can adaptively pass NeRV models per group of pictures (GOP). In one example, one NeRV model is used for one GOP. The video decoding device can receive the NeRV model in the first Instant Decode Refresh (IDR) frame of the GOP and apply the received model to the entire GOP. As another example, if T is the number of frames that make up the GOP, then T NeRV models can be used. When applying NeRV models to a GOP, the video encoding device uses the Picture Order Count (POC) as input to the NeRV model, which is the output order of the frames. Alternatively, the video encoding device can perform a T-modulus operation (...). Figure 16 In the example, %) is applied to the time series t of the frame to generate a remainder, and the resulting remainder is used as the time index of the NeRV model. The time series t of the frame is determined by the POC or the coding order. Figure 16 In the example described above, the remainder calculated by applying the T-modulus operation to the time series t of the frame is used as the time index of NeRV.

[0171] For example, video encoding devices are trained using the loss function described above. Figure 16 The frame-based NeRV model shown in the diagram enables the NeRV model to represent frames. Figure 16In the example, the video encoding device organizes the input by filling all (i, j) with values ​​in the range that the existing (i, j) index cannot have (0, 0) or (e.g., negative values). The video encoding device then passes the frame-based NeRV model to the video decoding device using the aforementioned encoding pipeline. The video decoding device reconstructs the frame-based NeRV model using the aforementioned decoding pipeline. The video decoding device can then use... Figure 15 The image shows a frame-based NeRV model for reconstructing frames corresponding to time indices.

[0172] Simultaneously, conventional deep learning-based in-loop filters, incorporated into video encoding or decoding devices, are trained in advance using a large amount of data. When quantization noise is sufficiently removed from typical videos, a common deep learning module, trained on the in-loop filters of the video encoding or decoding device, is assumed to be used, and video encoding / decoding is performed based on this common deep learning module. In this case, the deep learning module is constructed to make the encoded, quantized noisy video samples closely approximate the original samples and is consistently used in the video encoding or decoding devices.

[0173] However, as described above, the performance of a deep learning module trained can depend on the characteristics of the samples used for training. For example, if unused samples are input during training, the performance of the deep learning module may decrease rather than increase. This is because the video samples input during actual video encoding may have very different statistical characteristics depending on the intensity of quantization noise, the characteristics of the sensors used to create / generate content, and so on.

[0174] The following describes an example of applying the NeRV model described above to an in-loop filter using a video encoding device.

[0175] Figure 17 This is a diagram illustrating the application of a NeRV model according to at least one embodiment of the present disclosure.

[0176] exist Figure 17In one example, the video encoding device uses a conventional in-loop filter to reconstruct the first frame. Furthermore, the video encoding device trains a NeRV model using a time index as input and generates a second frame as the output of the NeRV model. The video encoding device selects the frame that is optimal in terms of rate-distortion between the first and second frames. In another example, the video encoding device uses a conventional in-loop filter to reconstruct the first block. Furthermore, the video encoding device trains a NeRV model using both a block index and a time index as input and generates a second block as the output of the NeRV model. The video encoding device selects the block that is optimal in terms of rate-distortion between the first and second blocks. The video encoding device can then explicitly signal frame / block level information about which block will be used between the first frame / first block and the second frame / second block to the video decoding device. In the following text, the signaled information is referred to as the NeRV utilization flag.

[0177] The following describes an example of applying the aforementioned NeRV model to intra-frame prediction or inter-frame prediction using video coding devices.

[0178] Figure 18 This is a diagram illustrating the application of a NeRV model according to another embodiment of this disclosure.

[0179] exist Figure 18 In the example, the video encoding device uses a NeRV model to replace all or part of the conventional inverse transformer 165, inverse quantizer 160, predictor 120, and adder 170. In conventional prediction methods, the video encoding device decodes the residual block to generate a prediction block, and then sums the residual block and the prediction block to generate a first reconstructed block. However, using the provided N blocks, the video encoding device can generate a second reconstructed block decoded using the NeRV model. After constructing the input using a combination of the temporal index and / or block index of the block / frame to be decoded, the video encoding device trains the NeRV model using the constructed input and outputs the video block at the corresponding temporal and spatial location as the output of the NeRV model to generate the second reconstructed block. The video encoding device selects the block that is optimal in terms of rate-distortion between the first and second reconstructed blocks. The video encoding device can then explicitly signal block-level information about which block to use between the first and second reconstructed blocks to the video decoding device. After the video decoding device decodes the information about whether to use the first or second reconstruction block, it can use the NeRV model instead of traditional prediction methods based on the decoded information. As mentioned above, the information conveyed by the signal is called the NeRV utilization flag.

[0180] exist Figure 18In the examples, the NeRV model generates (but is not necessarily limited to) the block to be decoded. In one example, the NeRV model can be trained to output the predicted block generated by predictor 120. The video encoding device can generate the decoded block by summing the residual block generated by inverse transform 165 and the predicted block generated by the NeRV model. As another example, the NeRV model can be trained to output the residual block generated by inverse transform 165. The video encoding device can generate the decoded block by summing the predicted block generated by predictor 120 and the residual block generated by the NeRV model. When the NeRV model is trained to output the residual block, the NeRV model can use the original block as the ground truth (GT) and can define the loss function by using the residual (i.e., the difference between the GT and the predicted block).

[0181] Therefore, the output N blocks of the NeRV model correspond to the residual block, prediction block, reconstruction block, or loop-filtered block of the current block. Regardless of the type of the N blocks, the NeRV model can be trained using the original blocks as ground truth (GT) based on an appropriate loss function. The video encoding device can also pass the type signal of the N blocks to the video decoding device. The video decoding device can decode the types of the N blocks and then apply the video decoding process to the N blocks as the output of the NeRV model based on the type of the N blocks. For example, if the N blocks correspond to the residual block, the video decoding device generates the prediction block of the current block according to the regular decoding process and generates the reconstruction block by summing the N blocks and the prediction block. If the N blocks correspond to the prediction block, the video decoding device generates the residual block of the current block according to the regular decoding process and generates the reconstruction block by summing the residual block and the N blocks. If the N blocks correspond to the reconstruction block, the video decoding device applies loop filtering to the N blocks. Furthermore, if the N blocks correspond to the loop-filtered block, the video decoding device stores the frame containing the N blocks in the decoded picture buffer (DPB). The DPB resides in memory 570.

[0182] The following describes a detailed example of applying the NeRV model to inter-frame prediction methods.

[0183] In traditional deep learning-based inter-frame prediction, a reference frame at the current time *t* is generated using reference frames from past and future time points, and motion prediction and compensation for the current block are performed based on this generated reference frame. Since motion prediction is performed using a reference frame estimated at the current time, improved decoding efficiency can be expected by reducing the number of bits used to represent motion vectors. However, the performance of the deep learning module trained to generate reference frames at the current time also depends on the characteristics of the samples used for training. This is because the performance of the deep learning module varies depending on the intensity of quantization noise, the characteristics of the sensor used to create / generate content, and so on. To address these issues, this paper describes a method for generating reference frames using a NeRV model and applying them to motion compensation.

[0184] In one example, such as Figure 19 As shown, the video decoding device generates a reference frame at the current time t using the NeRV model and stores the generated reference frame in the DPB. The DPB resides in memory 570. In the same manner as previously decoded video frames, the video decoding device uses the stored reference frame for motion prediction and compensation of the current block.

[0185] The following describes a method that utilizes multiple NeRV models.

[0186] When decoding a single N-block or a single N-frame, the video decoding device can utilize the single NeRV model described above. Alternatively, the video decoding device can select one of multiple NeRV models and utilize the selected NeRV model. In this case, the video encoding device generates multiple NeRV models, selects one of the multiple NeRV models, and performs encoding of the selected NeRV model. The manner in which various NeRV models are generated is based on additional information provided by the video encoding device. For example, different NeRV models can be used depending on the slice type or frame type. That is, different NeRV models can be used depending on whether N blocks in a P-frame or N blocks in an I-frame will be decoded. Alternatively, different NeRV models can be used depending on the quantization parameters.

[0187] In one example, Figure 12 In the NeRV model encoding pipeline shown, the video encoding device constructs multiple NeRV models by different settings of the model trimmer 1220 or the model quantizer 1230. The video encoding device selects the best model in terms of rate distortion from the constructed NeRV models. By using the above encoding pipeline, the video encoding device transmits the selected NeRV model to the video decoding device.

[0188] As another example, frame type or quantization parameters are used as input to the NeRV model. Figure 20In the example, in addition to the time index and the block index, the video encoding device uses the frame type or quantization parameter as input to the block-based NeRV model.

[0189] The following is for reference. Figure 21 and Figure 22 Describe the method of using the NeRV model to encode and decode the current block.

[0190] Figure 21 This is a flowchart of a method for encoding the current block by a video encoding device according to at least one embodiment of the present disclosure.

[0191] The video encoding device applies the video encoding process to the raw block to generate the first block of the current block (S2100). Here, the first block corresponds to the loop filter block of the current block.

[0192] The video encoding device constructs input for the NeRV model (S2102).

[0193] The video encoding device uses this input to construct the NeRV model (S2104).

[0194] In one example, a video encoding device can build a NeRV model by using the input and the original block (i.e., the ground truth block) to train the NeRV model to reduce the loss function. As the loss function, a combination of L1 loss and structural similarity index (SSIM) loss can be used.

[0195] As another example, a video encoding device can construct a NeRV model by using the parameters of the NeRV model of the N previously decoded blocks adjacent to the current block.

[0196] The video encoding device applies this input to the NeRV model to output N blocks (S2106). Here, N blocks correspond to the residual block, prediction block, reconstruction block, or loop filter block of the current block.

[0197] The video encoding device applies the video decoding process to the N blocks based on the type of the N blocks to generate a second block corresponding to the loop filter block of the current block (S2108).

[0198] The video decoding process applied to N blocks is as follows: If N blocks correspond to a residual block, the video encoding device generates a prediction block for the current block and sums the N blocks and the prediction block to generate a reconstructed block. If N blocks correspond to a prediction block, the video encoding device generates a residual block for the current block and sums the residual block and the N blocks to generate a reconstructed block. If N blocks correspond to a reconstructed block, the video encoding device applies a loop filter to N blocks. Additionally, if N blocks correspond to a loop-filtered block, the video encoding device stores the frame containing N blocks in the DPB (Deep Packet Block).

[0199] The video encoding device determines the NeRV utilization flag based on the first and second blocks (S2110). Here, the NeRV utilization flag indicates whether the NeRV model and N blocks are utilized. The video encoding device can determine the NeRV utilization flag in terms of rate-distortion optimization. For example, if the first block is optimal, the NeRV utilization flag can be determined to be false. Conversely, if the second block is optimal, the NeRV utilization flag can be determined to be true.

[0200] The video encoding device encodes the NeRV utilization flag (S2112).

[0201] The video encoding device checks the NeRV utilization flag (S2114).

[0202] If the NeRV utilization flag is true (Yes in S2114), the video encoding device encodes the encoding information of the second block.

[0203] The types of N blocks and the parameters of the NeRV model are encoded (S2116).

[0204] As described above, N blocks correspond to the residual block, prediction block, reconstruction block, or loop filter block of the current block. The video coding device determines the type of N blocks for rate-distortion optimization. The video coding device encodes the type of N blocks.

[0205] In one example, when building a NeRV model through training, the video encoding device uses, for example, Figure 12 The coding pipeline shown encodes the parameters of the NeRV model.

[0206] As another example, when constructing a NeRV model using parameters of the NeRV model of the previously decoded N blocks adjacent to the current block, the video encoding device encodes the index indicating the previously decoded N blocks.

[0207] If the NeRV utilization flag is false (No in S2114), the video encoding device encodes the encoding information of the first block (S2120). Here, according to the conventional video encoding method, the encoding information of the first block includes residual block, prediction mode, quantization parameters, transform kernel, etc.

[0208] Figure 22 This is a flowchart of a method for reconstructing the current block by a video decoding device according to at least one embodiment of the present disclosure.

[0209] The video decoding device decodes the NeRV utilization flag (S2200) from the bitstream. Here, the NeRV utilization flag indicates whether the NeRV model and N-block are used.

[0210] The video decoding device checks the NeRV utilization flag (S2202).

[0211] If the NeRV utilization flag is true (Yes in S2202), the video decoding device performs the following steps.

[0212] The video decoding device obtains the NeRV model (S2204).

[0213] In one example, the video decoding device decodes the parameters of the NeRV model from the bitstream. Using the decoded parameters, the video decoding device then... Figure 13 The decoding pipeline shown is used to build the NeRV model.

[0214] In another example, the video decoding device decodes from the bitstream an index indicating N previously decoded blocks. The video decoding device constructs a NeRV model using parameters of the NeRV model of the N previously decoded blocks indicated by the decoded indexes.

[0215] The video decoding device constructs inputs for the NeRV model (S2206).

[0216] The video decoding device applies the input to the NeRV model to output N blocks (S2208). Here, N blocks correspond to the residual block, prediction block, reconstruction block, or loop filter block of the current block.

[0217] The video decoding device decodes the type of N blocks from the bitstream (S2210).

[0218] The video decoding device applies the video decoding process to N blocks based on the type of N blocks (S2212).

[0219] The video decoding process applied to N blocks is as follows: If N blocks correspond to a residual block, the video decoding device generates a prediction block for the current block and sums the N blocks and the prediction block to generate a reconstructed block. If N blocks correspond to a prediction block, the video decoding device generates a residual block for the current block and sums the residual block and the N blocks to generate a reconstructed block. If N blocks correspond to a reconstructed block, the video decoding device applies a loop filter to N blocks. Additionally, if N blocks correspond to a loop-filtered block, the video decoding device stores the frame containing N blocks in the DPB (Devices Block Buffer).

[0220] If the NeRV utilization flag is false (No in S2202), the video decoding device performs the following steps.

[0221] The video decoding device decodes the reconstruction information of the current block from the bitstream (S2220). Here, the reconstruction information of the current block includes residual blocks, prediction modes, quantization parameters, and transform kernels according to conventional video decoding methods.

[0222] The video decoding device generates a reconstructed block for the current block using a video decoding process based on the reconstruction information of the current block (S2222). The video decoding device can generate a reconstructed block for the current block using a conventional video decoding process.

[0223] Although the steps in the various flowcharts are described as being performed sequentially, these steps merely exemplify the technical concepts of some embodiments of this disclosure. Therefore, those skilled in the art to which this disclosure pertains can perform the steps by changing the order described in the various figures or by performing two or more steps in parallel. Thus, the steps in the various flowcharts are not limited to the temporal sequence shown.

[0224] It should be understood that the above description presents illustrative implementations that can be implemented in various other ways. The functionality described in some implementations can be implemented by hardware, software, firmware, and / or a combination of hardware, software, and firmware. It should also be understood that the functional components described in this disclosure are designated with “…unit” to strongly emphasize their potential for independent implementation.

[0225] Furthermore, the various methods or functions described in some embodiments can be implemented as instructions stored in a non-transitory recording medium capable of being read and executed by one or more processors. For example, the non-transitory recording medium can include various types of recording devices in which data is stored in a computer-readable form. For example, the non-transitory recording medium can include storage media such as erasable programmable read-only memory (EPROM), flash memory drives, optical disc drives, magnetic hard disk drives, and solid-state drives (SSDs).

[0226] Although embodiments of this disclosure have been described for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible without departing from the concept and scope of this disclosure. Therefore, embodiments of this disclosure have been described for the sake of brevity and clarity. The scope of the technical concept of the embodiments of this disclosure is not limited by the illustrations. Therefore, those skilled in the art should understand that the scope of this disclosure should not be limited by the embodiments explicitly described above, but rather by the claims and their equivalents.

[0227] (Reference number)

[0228] 120: Predictor

[0229] 155: Entropy Decoder

[0230] 1210: Video Overfitter

[0231] 1220: Model Trimmer

[0232] 1230: Model Quantizer

[0233] 1240: Weight Encoder

[0234] Cross-reference to related applications

[0235] This application claims priority and benefit to Korean Patent Application No. 10-2023-0035657, filed on March 20, 2023, and Korean Patent Application No. 10-2024-00004603, filed on January 11, 2024, the entire contents of each of which are incorporated herein by reference.

Claims

1. A method for reconstructing a current block using a video decoding device, the method comprising: Obtain the NeRV (Video Neural Representation) model; Construct the input for the NeRV model; The input is applied to the NeRV model to output N blocks, which correspond to the residual block of the current block, the prediction block of the current block, the reconstruction block of the current block, or the loop filter block of the current block; Decode the type of the N blocks from the bitstream; as well as Based on the type of the N blocks, the video decoding process is applied to the N blocks.

2. The method according to claim 1, further comprising: Decode the NeRV utilization flag from the bitstream, the NeRV utilization flag indicating whether the NeRV model and the N block are to be used; as well as Check the NeRV utilization flag. Specifically, when the NeRV utilization flag is true, decoding of the NeRV model is performed, and then the video decoding process is applied to the N blocks.

3. The method according to claim 1, wherein, The method for obtaining the NeRV model includes: Decode the parameters of the NeRV model from the bitstream; and The NeRV model is constructed using the parameters described above.

4. The method according to claim 1, wherein, The method for obtaining the NeRV model includes: Decode from the bitstream the index of the N blocks that have been previously decoded; and The NeRV model is constructed using the parameters of the previously decoded N blocks of the NeRV model.

5. The method according to claim 1, wherein, The inputs for constructing the NeRV model include: The input is constructed to include the spatial index of the current block and the temporal index of the frame containing the current block.

6. The method according to claim 5, wherein, The inputs for constructing the NeRV model include: Decode local features from the bitstream, the local features being features of the current block or features of neighboring blocks of the current block, and extracted by a deep learning-based extraction model in the video encoding device; and The input is constructed to include the local features in addition to the index of the current block and the time index.

7. The method according to claim 5, wherein, The inputs for constructing the NeRV model include: The index of the current block and the time index are embedded in a high-dimensional space, and the input is constructed to include the embedded index of the current block and the embedded time index.

8. The method according to claim 1, wherein, The method for obtaining the NeRV model includes: Different NeRV models are obtained based on the type or quantization parameters of the frame containing the current block.

9. The method according to claim 8, wherein, The inputs for constructing the NeRV model include: The input is constructed to include the type of the frame or the quantization parameter, the index of the current block in space, and the time index of the frame.

10. The method according to claim 1, wherein, The step of applying the video decoding process to the N blocks includes: when the N blocks correspond to the residual block of the current block: The predicted block of the current block is generated, and the N blocks and the predicted block are summed to generate the reconstructed block of the current block.

11. The method according to claim 1, wherein, The application of the video decoding process to the N blocks includes: when the N blocks correspond to the reconstructed block of the current block: Loop filtering is applied to the N blocks.

12. A method for encoding a current block by a video encoding device, the method comprising: The video encoding process is applied to the original block of the current block to generate a first block, which is the loop filter block of the current block; Construct the input for the NeRV (Video Neural Representation) model; The NeRV model is constructed based on the input; The input is applied to the NeRV model to output N blocks, which correspond to the residual block of the current block, the prediction block of the current block, the reconstruction block of the current block, or the loop filter block of the current block; as well as Based on the type of the N blocks, the video decoding process is applied to the N blocks to generate a second block corresponding to the loop filter block of the current block.

13. The method according to claim 12, wherein, The construction of the NeRV model includes: The NeRV model is trained using the input and the original block in a direction that reduces the loss function, thereby constructing the NeRV model, wherein the loss function is a combination of L1 loss and SSIM (structural similarity index) loss.

14. The method according to claim 12, wherein, The construction of the NeRV model includes: The NeRV model is constructed using parameters from the NeRV model of the N previously decoded blocks adjacent to the current block.

15. The method of claim 12, further comprising: Based on the first block and the second block, a NeRV utilization flag is determined, which indicates whether the NeRV model and the N block should be used; as well as The NeRV utilization flag is encoded.

16. The method of claim 15, further comprising: Check the NeRV utilization flag. Wherein, when the NeRV utilization flag is true, and when the NeRV model is constructed through training, the method further includes: The types of the N blocks and the parameters of the NeRV model are encoded.

17. The method of claim 15, further comprising: Check the NeRV utilization flag. Wherein, when the NeRV utilization flag is true, and when the NeRV model is constructed using parameters of the NeRV model of N previously decoded blocks adjacent to the current block, the method further includes: The type of the N blocks and the index indicating the previously decoded N blocks are encoded.

18. A computer-readable recording medium storing a bitstream generated by a video coding method, wherein, The video encoding method includes: The video encoding process is applied to the original block of the current block to generate the first block, which is the loop-filtered block of the current block; Construct the input for the NeRV (Video Neural Representation) model; The NeRV model is constructed based on the input; The input is applied to the NeRV model to output N blocks, the N blocks corresponding to the residual block of the current block, the prediction block of the current block, the reconstruction block of the current block, or the loop filter block of the current block; and Based on the type of the N blocks, the video decoding process is applied to the N blocks to generate a second block corresponding to the loop filter block of the current block.

Images