Method and device for video coding using implicit neural representation model based on temporal dependency
By integrating temporal correlation into the Implicit Neural Representation model for video coding, the method enhances video coding efficiency and quality by using information from both current and past frames, overcoming the limitations of existing INRVC technologies.
Patent Information
- Application Number
- PCT/KR2024/095738
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-04-11
- Filing Date
- 2024-04-15
- Publication Date
- 2024-11-21
AI Technical Summary
Existing video coding technologies face inefficiencies in compressing and transmitting video data due to increasing data sizes, and current Implicit Neural Representation for Video Coding (INRVC) methods only utilize information from the current frame, limiting coding efficiency and quality improvements.
The proposed method employs a video coding approach that utilizes temporal correlation by incorporating information from both current and past video frames through an Implicit Neural Representation model, including a multi-layer perceptron and upscaling module, to predict motion information and enhance coding efficiency and quality.
This approach improves video coding efficiency and quality by leveraging temporal correlation, enabling better prediction and compression of video frames, thereby addressing the limitations of existing INRVC technologies.
Smart Images

Figure KR2024095738_21112024_PF_FP_ABST
Abstract
Description
Method and device for video coding using an implicit neural network representation model based on temporal correlation
[0001] The present disclosure relates to a video coding method and device using an implicit neural network representation model based on temporal correlation.
[0002] The content described below merely provides background information related to the present invention and does not constitute prior art.
[0003] Since video data has a large amount of data compared to voice data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.
[0004] Therefore, when storing or transmitting video data, an encoder is used to compress the video data and store or transmit it, and a decoder receives the compressed video data, decompresses it, and plays it back. Examples of such video compression technologies include H.264 / AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves encoding efficiency by about 30% compared to HEVC.
[0005] However, as the size, resolution, and frame rate of images are gradually increasing, and the amount of data that needs to be encoded is also increasing, a new compression technology that has better encoding efficiency and better image quality improvement than existing compression technologies is required.
[0006] Recently, deep learning-based image processing technologies are being applied to existing encoding element technologies. By applying deep learning-based image processing to existing encoding techniques, such as inter-prediction, intra-prediction, in-loop filters, and transformation, encoding efficiency can be improved. Representative application examples include inter-prediction based on virtual reference frames generated using deep learning, and in-loop filters based on noise removal models.
[0007] Meanwhile, research on implicit neural network representation models, which represent various data, including images, using neural network structures, has been active recently. Conventional video representation methods use an explicit representation method, which uses RGB pixel values for each pixel location. To replace this explicit representation method, an implicit neural representation method is being introduced, which uses a neural network to represent a function that generates (r,g,b) values from (x,y) coordinates, which are pixel locations in an image. Video coding using implicit neural representation is called Implicit Neural Representation for Video Coding (INRVC). Existing INRVC technology only utilizes information from the current video frame. Therefore, to improve video encoding efficiency and video quality, methods that utilize information from past video frames in INRVC technology need to be considered.
[0008] The present disclosure aims to provide a video coding method and device that utilize temporal correlation based on information of a current video frame and past video frames in Implicit Neural Representation for Video Coding (INRVC).
[0009] According to an embodiment of the present disclosure, a method for restoring a current image, performed by an image decoding device, is provided, characterized in that the method comprises: configuring an input of an INRVC (Implicit Neural Representation for Video Coding) model including a current time index of the current image; obtaining the INRVC model, wherein the INRVC model includes a multi-layer perceptron and an upscaling module; applying the input to the multi-layer neural network to generate a multi-layer neural network output; predicting motion information between the current image and the previous output using a previous output of the INRVC model corresponding to a previous time index; and inputting a combination of the multi-layer neural network output and the motion information to the upscaling module to generate a current output corresponding to the current time index.
[0010] According to another embodiment of the present disclosure, a method for encoding a current image, performed by a video encoding device, is provided, comprising: configuring an input of an INRVC (Implicit Neural Representation for Video Coding) model including a current time index of the current image; configuring the INRVC model, wherein the INRVC model includes a multi-layer perceptron and an upscaling module; applying the input to the multi-layer neural network to generate a multi-layer neural network output; predicting motion information between the current image and the previous output using a previous output of the INRVC model corresponding to a previous time index; and inputting a combination of the multi-layer neural network output and the motion information to the upscaling module to generate a current output corresponding to the current time index.
[0011] According to another embodiment of the present disclosure, a method for providing video data to a video decoding device is provided, comprising: encoding the video data into a bitstream; and transmitting the bitstream to the video decoding device, wherein the encoding the video data comprises: configuring an input of the INRVC model including a current time index of a current image; configuring an INRVC (Implicit Neural Representation for Video Coding) model, the INRVC model including a multi-layer perceptron and an upscaling module; applying the input to the multi-layer neural network to generate a multi-layer neural network output; predicting motion information between the current image and the previous output using a previous output of the INRVC model corresponding to a previous time index; and inputting a combination of the multi-layer neural network output and the motion information to the upscaling module to generate a current output corresponding to the current time index.
[0012] As described above, according to the present embodiment, in video coding based on implicit neural network representation, a video coding method and device are provided that utilize temporal correlation based on information of a current video frame and past video frames, thereby making it possible to improve video encoding efficiency and video quality.
[0013] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure.
[0014] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.
[0015] FIGS. 3A and 3B are diagrams illustrating multiple intra prediction modes, including wide-angle intra prediction modes.
[0016] Figure 4 is an example diagram of the surrounding blocks of the current block.
[0017] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure.
[0018] Figure 6 is an example diagram showing the operation of a convolution layer.
[0019] Figure 7 is an example diagram showing a SISR (Single Image Super Resolution) network.
[0020] Figure 8 is an example diagram showing a residual block used in SISR.
[0021] Figure 9 is an example diagram showing a fixed coefficient in-loop filter based on a CNN (Convolutional Neural Network).
[0022] Figure 10 is an example diagram showing the NeRV (Neural Representations for Videos) model.
[0023] FIG. 11A and FIG. 11B are exemplary diagrams illustrating the concept of the NeRV model according to one embodiment of the present disclosure.
[0024] FIG. 12 is an exemplary diagram illustrating an encoding pipeline of a NeRV model according to one embodiment of the present disclosure.
[0025] FIG. 13 is an exemplary diagram illustrating a decryption pipeline of a NeRV model according to one embodiment of the present disclosure.
[0026] FIG. 14 is an exemplary diagram illustrating an INRVC (Implicit Neural Representation for Video Coding) model including an in-loop according to one embodiment of the present disclosure.
[0027] FIG. 15 is an exemplary diagram illustrating an INRVC model including optical flow according to one embodiment of the present disclosure.
[0028] Figure 16 is an example diagram showing deformable convolution.
[0029] FIG. 17 is an exemplary diagram illustrating a block-based INRVC model according to another embodiment of the present disclosure.
[0030] FIG. 18 is an exemplary diagram showing inter prediction performed by an INRVC model according to one embodiment of the present disclosure.
[0031] FIG. 19 is a flowchart illustrating a method by which an image encoding device encodes a current image according to one embodiment of the present disclosure.
[0032] FIG. 20 is a flowchart illustrating a method for an image decoding device to restore a current image according to one embodiment of the present disclosure.
[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.
[0034] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image encoding device and its subcomponents will be described with reference to the illustration in FIG. 1.
[0035] The video encoding device may be configured to include a picture segmentation unit (110), a prediction unit (120), a subtractor (130), a transformation unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse transformation unit (165), an adder (170), a loop filter unit (180), and a memory (190).
[0036] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.
[0037] A single image (video) is composed of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). Each CTU is then divided into one or more Coding Units (CUs) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks within a single slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). And, information commonly referenced by one or more SPS is encoded in a Video Parameter Set (VPS). In addition, information commonly applied to one tile or tile group may be encoded as syntax of a tile or tile group header. Syntaxes included in an SPS, PPS, slice header, tile or tile group header may be referred to as high level syntax.
[0038] The picture segmentation unit (110) determines the size of the CTU. Information about the size of the CTU (CTU size) is encoded as the syntax of SPS or PPS and transmitted to the image decoding device.
[0039] The picture segmentation unit (110) divides each picture constituting an image into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a CU, which is a basic unit of encoding.
[0040] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, or a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that mixes two or more of the QT structures, BT structures, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be combined and referred to as a MTT (Multiple-Type Tree).
[0041] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT structure.
[0042] As illustrated in FIG. 2, a CTU may first be split into a QT structure. The quadtree splitting may be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in the QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoding unit (155) and signaled to the image decoding device. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further split into one or more of the BT structure or the TT structure. There may be multiple splitting directions in the BT structure and / or the TT structure. For example, there may be two directions in which the block of the corresponding node is split horizontally and two directions in which the block is split vertically. As illustrated in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (Binary or Ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.
[0043] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the CU split flag (split_cu_flag) value indicates that the node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is a basic unit of encoding. If the CU split flag (split_cu_flag) value indicates that the node is split, the video encoding device starts encoding from the first flag in the above-described manner.
[0044] As another example of a tree structure, when QTBT is used, there may be two types: a type that horizontally splits the block of the corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that vertically splits it (i.e., symmetric vertical splitting). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating the type of split are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may additionally be a type that splits the block of the corresponding node into two blocks of an asymmetrical shape. The asymmetric shape may include a shape that splits the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that splits the block of the corresponding node in a diagonal direction.
[0045] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." Depending on the QTBTTT partitioning employed, the current block may be rectangular as well as square.
[0046] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).
[0047] In general, each current block within a picture can be predictively coded. Prediction of the current block can typically be performed using either intra-prediction (using data from the picture containing the current block) or inter-prediction (using data from a picture coded before the picture containing the current block). Inter-prediction encompasses both unidirectional and bidirectional prediction.
[0048] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in Fig. 3a, the multiple intra prediction modes may include two non-directional modes including the Planar mode and the DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently depending on each prediction mode.
[0049] For efficient directional prediction for a rectangular current block, directional modes (intra prediction modes 67 to 80 and -1 to -14) indicated by dotted arrows in Fig. 3b may be additionally used. These may be referred to as "wide-angle intra-prediction modes." In Fig. 3b, the arrows point to corresponding reference samples used for prediction, and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide-angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. At this time, among the wide-angle intra-prediction modes, some wide-angle intra-prediction modes available for the current block may be determined based on the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes (intra prediction modes 67 to 80) having an angle less than 45 degrees are available when the current block is a rectangular shape whose height is smaller than its width, and wide-angle intra prediction modes (intra prediction modes -1 to -14) having an angle greater than -135 degrees are available when the current block is a rectangular shape whose width is larger than its height.
[0050] The intra prediction unit (122) can determine an intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit (122) can encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra prediction unit (122) can calculate bit-rate distortion values using rate-distortion analysis for multiple tested intra prediction modes and select an intra prediction mode with the best bit-rate distortion characteristics among the tested modes.
[0051] The intra prediction unit (122) selects one intra prediction mode from among multiple intra prediction modes and predicts the current block using surrounding pixels (reference pixels) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.
[0052] The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for a block most similar to the current block within reference pictures that were encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture is generated. Generally, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma component and the chroma component. The motion information including information on the reference picture used to predict the current block and information on the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.
[0053] The inter prediction unit (124) may perform interpolation on a reference picture or a reference block to improve prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed on the interpolated reference picture, the motion vector can be expressed up to a precision in decimal units rather than a precision in integer sample units. The precision or resolution of the motion vector can be set differently for each target region to be encoded, such as a slice, tile, CTU, CU, etc. When such adaptive motion vector resolution (AMVR) is applied, information on the motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on the motion vector resolution applied to each CU is signaled. Information on the motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.
[0054] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors indicating the block position most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture to generate a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged or weighted averaged to generate a prediction block for the current block. Then, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is transmitted to the entropy encoding unit (155). Here, reference picture list 0 may be composed of pictures that are before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures that are after the current picture in display order among the restored pictures. However, this is not necessarily limited to this, and restored pictures that are after the current picture in display order may be additionally included in reference picture list 0, and conversely, restored pictures that are before the current picture may be additionally included in reference picture list 1.
[0055] Various methods can be used to minimize the number of bits required to encode motion information.
[0056] For example, if the reference picture and motion vector of the current block are identical to those of a neighboring block, the motion information of the current block can be transmitted to the image decoding device by encoding information that can identify the neighboring block. This method is called 'merge mode'.
[0057] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.
[0058] As the surrounding blocks for deriving merge candidates, all or part of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block within the current picture may be used, as illustrated in FIG. 4. In addition, a block located within a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block within the reference picture or blocks adjacent to the block at the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.
[0059] The inter prediction unit (124) uses these surrounding blocks to construct a merge list containing a predetermined number of merge candidates. Among the merge candidates included in the merge list, the merge candidate to be used as motion information of the current block is selected and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the video decoding device.
[0060] Merge Skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy encoding are close to zero, only neighboring block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high encoding efficiency can be achieved for low-motion images, still images, and screen content images.
[0061] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.
[0062] Another method for encoding motion information is Advanced Motion Vector Prediction (AMVP) mode.
[0063] In AMVP mode, the inter prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive predicted motion vector candidates, all or some of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block in the current picture as shown in FIG. 4 may be used. In addition, a block located in a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as the neighboring block used to derive predicted motion vector candidates. For example, a block co-located with the current block in the reference picture or blocks adjacent to the block in the co-located block may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.
[0064] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of these surrounding blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to produce a differential motion vector.
[0065] The predicted motion vector can be obtained by applying a predefined function (e.g., median, mean, etc.) to the predicted motion vector candidates. In this case, the image decoding device also knows the predefined function. In addition, since the surrounding blocks used to derive the predicted motion vector candidates are blocks that have already been encoded and decoded, the image decoding device also already knows the motion vectors of the surrounding blocks. Therefore, the image encoding device does not need to encode information to identify the predicted motion vector candidates. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.
[0066] Alternatively, the predicted motion vector can be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the differential motion vector and the reference picture used to predict the current block.
[0067] The subtractor (130) subtracts the prediction block generated by the intra prediction unit (122) or inter prediction unit (124) from the current block to generate a residual block.
[0068] The transformation unit (140) transforms residual signals within a residual block having pixel values in a spatial domain into transform coefficients in a frequency domain. The transformation unit (140) may transform the residual signals within the residual block using the entire size of the residual block as a transformation unit, or may divide the residual block into a plurality of sub-blocks and use the sub-blocks as transformation units to perform the transformation. Alternatively, the residual signals may be transformed using only the transformation domain sub-block as a transformation unit by dividing the sub-blocks into two sub-blocks, that is, a transformation domain and a non-transform domain. Here, the transformation domain sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only a sub-block has been converted, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. In addition, the size of the conversion area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.
[0069] Meanwhile, the transformation unit (140) can individually perform transformations on the residual block in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices can be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one transformation function pair with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) on the transformation function pair selected among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoding device.
[0070] The quantization unit (145) quantizes the transform coefficients output from the transform unit (140) using quantization parameters and outputs the quantized transform coefficients to the entropy encoding unit (155). The quantization unit (145) may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit (145) may also apply different quantization coefficients (scaling values) according to the positions of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to an image decoding device.
[0071] The rearrangement unit (150) can perform rearrangement of coefficient values for quantized residual values.
[0072] The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from the DC coefficient to the coefficients of the high-frequency region by using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, the scanning method to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan.
[0073] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transform coefficients output from the rearrangement unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.
[0074] In addition, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoding device can divide the block in the same manner as the image encoding device. In addition, the entropy encoding unit (155) encodes information about a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., information about an intra prediction mode) or inter prediction information (information about an encoding mode of motion information (merge mode or AMVP mode), a merge index in the case of a merge mode, and a reference picture index and a differential motion vector in the case of an AMVP mode) according to the prediction type. In addition, the entropy encoding unit (155) encodes information related to quantization, that is, information about a quantization parameter and information about a quantization matrix.
[0075] The inverse quantization unit (160) inversely quantizes the quantized transform coefficients output from the quantization unit (145) to generate transform coefficients. The inverse transform unit (165) transforms the transform coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.
[0076] An adder (170) adds the restored residual block and the predicted block generated by the prediction unit (120) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting the next block.
[0077] The loop filter unit (180) performs filtering on restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation / quantization. The loop filter unit (180) may include all or part of a deblocking filter (182), a sample adaptive offset (SAO) filter (184), and an adaptive loop filter (ALF, 186) as an in-loop filter.
[0078] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-based encoding / decoding, and the SAO filter (184) and the ALF (186) perform additional filtering on the deblocking-filtered image. The SAO filter (184) and the ALF (186) are filters used to compensate for the differences between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in units of CTUs. In contrast, the ALF (186) performs block-based filtering, and compensates for distortion by applying different filters by distinguishing the edges and degrees of variation of the corresponding block. Information on filter coefficients to be used in the ALF can be encoded and signaled to an image decoding device.
[0079] The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in the memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.
[0080] The video encoding device can store the bitstream of encoded video data on a non-transitory storage medium or transmit it to the video decoding device using a communication network.
[0081] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image decoding device and its subcomponents will be described with reference to FIG. 5.
[0082] The video decoding device may be configured to include an entropy decoding unit (510), a rearrangement unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).
[0083] Similar to the video encoding device of FIG. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.
[0084] The entropy decoding unit (510) decodes the bitstream generated by the image encoding device to extract information related to block division, thereby determining the current block to be decoded, and extracts prediction information, information on residual signals, etc. required to restore the current block.
[0085] The entropy decoding unit (510) extracts information about the CTU size from the Sequence Parameter Set (SPS) or the Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and the CTU is divided using the tree structure by extracting division information about the CTU.
[0086] For example, when splitting a CTU using the QTBTTT structure, first, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, the second flag (mtt_split_flag) related to the splitting of MTT and the split direction (vertical / horizontal) and / or split type (binary / ternary) information are extracted, and the corresponding leaf nodes are split into the MTT structure. Accordingly, each node below the leaf nodes of QT are split recursively into the BT or TT structure.
[0087] As another example, when splitting a CTU using the QTBTTT structure, the CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and if the block is split, the first flag (QT_split_flag) may be extracted. During the splitting process, each node may undergo zero or more repeated QT splits followed by zero or more repeated MTT splits. For example, a CTU may undergo an MTT split right away, or conversely, may undergo only multiple QT splits.
[0088] As another example, when splitting a CTU using the QTBT structure, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Furthermore, for nodes corresponding to leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and splitting direction information are extracted.
[0089] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded by using the division of the tree structure, it extracts information on the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-prediction, the entropy decoding unit (510) extracts syntax elements for intra-prediction information (intra-prediction mode) of the current block. If the prediction type information indicates inter-prediction, the entropy decoding unit (510) extracts syntax elements for inter-prediction information, i.e., information indicating a motion vector and a reference picture referenced by the motion vector.
[0090] Additionally, the entropy decoding unit (510) extracts information about the quantized transform coefficients of the current block as information related to quantization and information about residual signals.
[0091] The rearrangement unit (515) can change the sequence of one-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the image encoding device.
[0092] The inverse quantization unit (520) inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantization unit (520) may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantized coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.
[0093] The inverse transform unit (530) inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a residual block for the current block.
[0094] In addition, when the inverse transform unit (530) inversely transforms only a portion of a transform block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only a sub-block of the transform block has been transformed, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block, and inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to restore residual signals, and fills “0” values with residual signals for areas that have not been inversely transformed, thereby generating a final residual block for the current block.
[0095] In addition, when MTS is applied, the inverse transform unit (530) determines a transform function or a transform matrix to be applied in the horizontal and vertical directions using MTS information (mts_idx) signaled from the image encoding device, and performs inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.
[0096] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is inter prediction.
[0097] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.
[0098] The inter prediction unit (544) uses the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510) to determine the motion vector of the current block and the reference picture referenced by the motion vector, and predicts the current block using the motion vector and the reference picture.
[0099] An adder (550) adds the residual block output from the inverse transform unit (530) and the predicted block output from the inter prediction unit (544) or the intra prediction unit (542) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting a block to be decoded later.
[0100] The loop filter unit (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as in-loop filters. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter (564) and the ALF (566) perform additional filtering on restored blocks after deblocking filtering to compensate for differences between restored pixels and original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about filter coefficients decoded from the non-stream.
[0101] The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in the memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within a picture to be encoded later.
[0102]
[0103] The present embodiment relates to encoding and decoding of images (videos) as described above. More specifically, in Implicit Neural Representation for Video Coding (INRVC), a video coding method and device are provided that utilize temporal correlation based on information of a current video frame and past video frames.
[0104] The following embodiments may be performed by various components within a video encoding apparatus. Furthermore, the following embodiments may be performed by various components within a video decoding apparatus.
[0105] The video encoding device can generate signaling information related to the present embodiment in terms of rate distortion optimization in encoding the current block. The video encoding device can encode the signaling information using the entropy encoding unit (155) and then transmit it to the video decoding device. The video decoding device can decode the signaling information related to the decoding of the current block from the bitstream using the entropy decoding unit (510).
[0106] In the following description, the term "target block" may be used interchangeably with the current block or coding unit (CU). Alternatively, the term "target block" may also refer to a portion of a coding unit.
[0107] Also, a value of a flag being true indicates that the flag is set to 1. Also, a value of a flag being false indicates that the flag is set to 0.
[0108] I. CNN(Convolutional Neural Network)
[0109] CNN, a neural network comprised of multiple convolutional layers and pooling layers, is a deep learning technique known to be ideal for image processing. Convolutional layers extract feature maps (also known as "features" or "characteristics") using multiple kernels or filters. The kernel coefficients that make up the filters are parameters determined during the learning process.
[0110] Among the convolutional layers of CNN, the front layer closer to the input extracts feature maps that respond to simple, low-level image features such as lines, points, or surfaces, while the back layer closer to the output extracts feature maps that respond to higher-level features such as textures and object parts.
[0111] FIG. 6 is an exemplary diagram showing the operation of a convolutional layer according to one embodiment of the present disclosure.
[0112] A convolutional layer generates a feature map from an input image using a convolution operation. The example in Fig. 6 illustrates a kernel (or filter) with a kernel size of 3×3. The kernel size is also referred to as the kernel size or filter size. The kernel has kernel parameters, also called weights. The kernel illustrated in Fig. 6 has a total of nine kernel parameters. The kernel parameters are initially set to arbitrary values, and their values can be updated based on learning.
[0113] The convolution layer performs convolution operations using blocks of the same size as the kernel size in the input image. At this time, the blocks of the same size as the kernel size in the input image are referred to as windows.
[0114] When filtering an input image in raster scan order, the window movement size is called the stride. In the example of Fig. 6, the stride is 1. If the stride is set to 2, the convolution operation is performed by spacing the window by 2 samples, and as a result, the width and height of the feature map become half the width and height of the input image.
[0115] As mentioned above, a single convolutional layer can include multiple filters. The number of filters or kernels is called a channel. In other words, the number of channels is equal to the number of filters. Furthermore, the number of filters determines the dimensionality of the feature map.
[0116] Padding refers to a method of expanding input data by filling the area around it with a specific value before performing a convolution operation. Padding is primarily used to adjust the spatial size of the output data. The padding value can be determined by hyperparameters, but zero-padding is commonly used. Without padding, the spatial size of the output data decreases with each convolutional layer, potentially causing boundary information to disappear. Therefore, padding is used to prevent this problem. Specifically, padding can be used to match the spatial sizes of the output data from the convolutional layer to the input data.
[0117] The deconvolution layer performs the opposite operation to the convolution layer. It generates the desired data image from the input feature map as output.
[0118] The pooling layer performs pooling, a process of subsampling the feature map generated by the convolutional layer. The pooling layer uses a 2×2 window to select samples so that the output is half the width and height of the input. In other words, the pooling layer is used to reduce the size of the input image or input feature map by condensing a 2×2 region into a single sample.
[0119] The opposite concept of a pooling layer is defined as an unpooling layer. Unpooling layers, in contrast to pooling layers, function to expand the dimensionality and are primarily used after deconvolution layers.
[0120] The convolutional encoder-decoder architecture is a network structure composed of pairs of convolutional layers and deconvolutional layers. The convolutional encoder consists of a convolutional layer and a pooling layer, and outputs a feature map (or feature vector) from the input image. The final output vector of the convolutional encoder is also referred to as a latent vector. The convolutional decoder consists of a deconvolutional layer and an unpooling layer, and generates an output image from the feature map or latent vector.
[0121] The inputs and outputs of a convolutional encoder-decoder can be configured in various ways depending on the application and network purpose. For example, the inputs and outputs can be optical flow maps, saliency maps, image frames, etc.
[0122] Figure 7 is an example diagram showing an SISR network.
[0123] One example of CNN application is Single Image Super Resolution (SISR). The SISR network generates a high-resolution image from a low-resolution input image. As illustrated in Figure 7, the SISR network may include multiple convolutional layers. Each convolutional layer includes an activation function, such as the Rectified Linear Unit (ReLU). The parameters of the SISR network can be trained so that the resulting SR (Super Resolution) image approximates the Ground Truth (GT).
[0124] SR methods using CNN can improve SR performance by increasing the depth (e.g., increasing the number of convolutional layers). To overcome the overfitting problem that may occur in learning due to the increase in depth, a residual block that can perform skip connection and residual learning can be used in the SISR network. The residual block, as illustrated in Fig. 8, is a block that stores input features x l In addition to the path that applies the convolution operation, it includes a skip path. In addition, the residual block outputs x l+1 When generating, a path or skip path for applying convolution operations can also be selected based on learning efficiency. In the example of Fig. 8, the residual block includes a BN (Batch Normalization) layer.
[0125] For example, Enhanced Deep Residual Networks (EDSR) improves network performance by continuously connecting residual blocks to increase their depth. Another example is Accurate Image Super-Resolution Using Very Deep Convolutional Networks (VDSR), a CNN model based on the Visual Geometry Group (VGG) network. It uses residual learning, a method that adds residual frames to the final output. VDSR adds residual signals to the very end of the network, thereby adding them to the input signal.
[0126] As another example, CNN can be used as an in-loop filter in an image encoding device or an image decoding device. In this case, the deep learning-based in-loop filter can be applied to any position within the existing loop filter unit (180, 560) consisting of a deblocking filter, an SAO filter, and an ALF.
[0127] Among the deep learning-based in-loop filters are fixed-coefficient in-loop filters. Fixed-coefficient deep learning in-loop filters use the same CNN kernel parameters stored on the image encoding and decoding devices.
[0128] Figure 9 is an example diagram showing a CNN-based fixed coefficient in-loop filter.
[0129] The input block (or input frame) is passed through a normalized QP map and then passed to the subsequent stage. The normalized QP map is used to reduce inference errors when quantization noise of different intensities is mixed during the learning and inference processes. The kernel parameters that constitute the DRU (Dense Residual Unit) and the convolutional layer can be stored and used identically in the image encoding device and the image decoding device. In the example of Fig. 9, each DRU may include all or part of the convolutional layer, the ReLU layer, and the depth-wise separable convolutional (DSC) layer.
[0130] Fixed-coefficient deep learning in-loop filters have the disadvantage of providing general performance for various video frames, which increases the number of layers in the CNN and thus increases the computation time.
[0131] II. Implicit neural representation model
[0132] Recently, there has been active research on implicit neural network representation models that represent various data, including images, using neural network structures. Conventional video representation methods use an explicit representation method, which represents RGB pixel values for each pixel location. To replace this explicit representation method, implicit neural network representation models are being introduced, which represent functions that generate (r,g,b) values from (x,y) coordinates, which are pixel locations in an image, using neural networks. Compared to explicit representation methods, implicit neural network representation models can be used regardless of image resolution. In this case, a multi-layer perceptron (MLP) network containing multiple fully-connected layers (FC) can be used as an implicit neural network representation model.
[0133] As an example of video coding that utilizes an implicit neural network representation model, there is Implicit Neural Representation for Video Coding (INRVC). As mentioned above, the INRVC technique generates (r,g,b) values for the (x,y) coordinates, which are pixel locations in an image. As an example of the INRVC technique, there is the NeRV (Neural Representation for Video) model. As mentioned above, the explicit representation method expresses pixel values on the (x,y,t) grid using the pixel locations (x,y) in the image and the time index t of the frame. The implicit neural network representation model outputs RGB pixel values from the input representing the location (x,y,t). However, training the neural network at all locations (x,y,t) can significantly increase the computational complexity, so, as shown in the example of FIG. 10, the NeRV model outputs the RGB image of the entire frame at the time index t using a neural network structure that only takes the time index t as input.
[0134] FIG. 11A and FIG. 11B are exemplary diagrams illustrating the concept of the NeRV model according to one embodiment of the present disclosure.
[0135] To easily provide image output for a time index t input, it may be more effective to use a convolutional layer than to use an MLP network of an existing implicit representation model. The NeRV model can generate output using a structure in which NeRV blocks composed of multiple convolutional layers are stacked. As shown in the example of Fig. 11a, an MLP-based output and a NeRV block-based output can be compared. The NeRV block includes convolutional layers that perform convolution, a pixel shuffle layer, and an activation layer, as shown in the example of Fig. 11b. In the example of Fig. 11b, C represents the number of channels, W and H represent the width and height of the input of the NeRV block, and S represents an upscaling factor.
[0136] Additionally, the NeRV model embeds the time index t in a high-dimensional space as in Equation 1 and then uses the embedded index as input.
[0137]
[0138] As illustrated in Equation 1, the temporal index t can be mapped to a 2L-dimensional vector γ(t). By using the embedded temporal index t to train the NeRV model, the NeRV model can better predict video data containing high-frequency variations.
[0139] As the loss function of the NeRV model, a combination of L1 loss and SSIM (Structural Similarity Index) loss can be used. During training of the NeRV model, the aforementioned loss can be calculated at all pixel locations of the image estimated by the NeRV model based on the input and the ground truth (GT) image (i.e., the original image). The L1 loss is calculated using the absolute value of the difference between pixels in the estimated image and the GT image. The SSIM loss is calculated based on the mean, standard deviation, and correlation of pixels in the estimated image and the GT image. Thereafter, the weights (or parameters) that constitute the NeRV model during the training process can be updated based on the calculated loss.
[0140] FIG. 12 is an exemplary diagram illustrating an encoding pipeline of a NeRV model according to one embodiment of the present disclosure.
[0141] Since the original video can be approximately represented using the NeRV model, an end-to-end compression technology using a neural network can be implemented by compressing and transmitting the NeRV model. An encoding pipeline for encoding the NeRV model may include all or part of a video overfitter (1210), a model pruner (1220), a model quantizer (1230), and a weight encoder (1240), as shown in the example of Fig. 12.
[0142] The video overfitter (1210) expresses the input video frame as a NeRV model. At this time, the video encoding device trains the NeRV model using the aforementioned loss function, thereby allowing the NeRV model to express the video frame. The model pruner (1220) simplifies the NeRV model structure, which has an MLP or a mixed form of MLP and convolutional layers, using pruning. For example, pruning sets weights smaller than a preset threshold to zero. The model quantization unit (1230) quantizes the pruned weights (e.g., weights greater than the threshold). The weight encoding unit (1240) applies entropy encoding to the quantized weights to generate a bitstream of the weights.
[0143] Meanwhile, as a reverse order of the encoding pipeline illustrated in FIG. 12, a decoding pipeline for decoding a NeRV model can be illustrated as in FIG. 13. The decoding pipeline can include all or part of a weight decoder (1310), a model dequantizer (1320), a model reconstructor (1330), and a video generator (1340), as illustrated in FIG. 13.
[0144] The weight decoding unit (1310) applies entropy encoding to the bitstream of the weights of the NeRV model to generate quantized weights. The model dequantization unit (1320) dequantizes the quantized weights to generate pruned weights. The model restoration unit (1330) restores the NeRV model based on the pruned weights. The video generation unit (1340) uses the restored NeRV model to generate restored video frames according to the time index.
[0145] As mentioned above, the NeRV model uses the time index t of the frame as input. Therefore, when a video having pixels of size t×h×w is input, the NeRV model samples the input video only t times, not t×h×w times, so a large gain in both encoding speed and decoding speed can be expected.
[0146] As mentioned above, existing INRVC techniques utilize only information from the current frame, leaving limited intra- or inter-frame spatial information available for output calculation. To address this issue, we describe an INRVC model that utilizes encoding information from previously encoded frames to increase coding efficiency for the current frame.
[0147] The following embodiments are described with reference to an image decoding device including an INRVC model, but can be implemented identically or similarly in an image encoding device including an INRVC model.
[0148] III. Embodiments according to the present disclosure
[0149] FIG. 14 is an exemplary diagram illustrating an INRVC model including an in-loop according to one embodiment of the present disclosure.
[0150] The INRVC model according to the present embodiment includes an in-loop that feeds back the output corresponding to the previous time index (t-1) when predicting a frame for the current time index (t). In the example of Fig. 14, an MLP (1410), an upscaling module (1420), a feature extractor (1430), and an upscaling module (1440) constitute a conventional INRVC model. As described above, the conventional INRVC model uses the time index t as an input and outputs the current frame. The upscaling modules (1420, 1440) include the convolutional layer illustrated in Fig. 6 and the pixel shuffling layer included in Fig. 11b, and upscales the input using a combination of the convolutional layer and the pixel shuffling layer. The MLP (1410) and the upscaling modules (1420, 1440) constitute an implicit neural network representation for the current frame. The feature extractor (1430) generates spatiotemporal information for the input features. The feature extractor (1430) is trained to generate spatiotemporal information for the features of the current t using the feature information of the previous t-1 and the feature information of the current t.
[0151] The in-loop includes a Motion Transfer Module (MTM, 1450) and a Decoded Feature Buffer (DFB, 1460) as components. The DFB (1460) is included in the memory (570) and stores feature information of past frames. Since the INRVC model uses feature information to generate a frame, the feature information of the frame can be stored in the DFB (1460). Therefore, the INRVC model according to the present embodiment can utilize information about other time indices for coding the current frame without additional input. To efficiently utilize the feature information of the previous t-1, the MTM (1450) generates an embedding for the feature information of the previous t-1, thereby reducing the size of the feature information of the previous t-1. The MTM (1450) improves the training efficiency at the current t according to the implicit neural network representation by performing motion information prediction and motion compensation between the feature information of the previous t-1 and the feature information of the current t. For example, the motion information may include the motion of each pixel constituting the image. The embedding generated by MTM (1450) is generated to match the characteristics of the output of MLP (1410) and is concatenated with the output of MLP (1410).
[0152] The INRVC model generates outputs for the t-th and t-1-th time indices using in-loop components. Since the output for the t-1-th time index is already stored in the DFB (1460), the output for the t-1-th time index is a bypassed output and can be used to improve the output for the t-th time index thereafter.
[0153] As an example, MTM (1450) can be implemented as a deep learning model. In this case, an INRVC model including an in-loop can be trained in advance.
[0154] Below, we describe the methods by which the INRVC model uses MTM (1450) to predict and compensate for motion.
[0155] Since video is used as input for the INRVC model, temporal information, in addition to spatial information, also affects model performance. The existing NeRV extracts only one frame per temporal index, and each frame is generated separately, failing to account for temporal information.
[0156] As an example, to consider temporal information, after outputting both the current and previous frames during the feature extraction process, the INRVC model can use the temporal correlation based on the optical flow of the two frames. As shown in the example of Fig. 15, the INRVC model uses an OF estimator (optical flow estimator, 1520) to estimate the optical flow for the two frames and improves the output for the t-th and t-1-th temporal indices using the estimated optical flow. The optical flow can be a pixel-by-pixel motion vector that represents the movement of pixels that occurs between the t-th picture and the t-1-th picture.
[0157] At this time, if the quality of the frame output from the upscaling module is low, it is difficult to predict the accurate optical flow. The INRVC model uses a quality evaluator (1510) to measure the quality of the output of the previously generated upscaling module, i.e., the output for the t-1th time index. If the quality of the output for the t-1th time index is higher than a preset threshold, the INRVC device calculates the optical flow to improve the output for the t-th and t-1th time indices. If the quality of the output for the t-1th time index is lower than the preset threshold, the optical flow calculation process is omitted, and the output generated from the upscaling module is used as the final output.
[0158] As another example, the INRVC model includes a deformable convolution module, and can generate temporal information between features using the deformable convolution. The deformable convolution module can replace the quality assessor (1510) and the OF estimator (1520). In addition to extracting features from a fixed kernel (e.g., a 3×3 kernel as illustrated on the left side of FIG. 16), the deformable convolution can extract features from a more flexible region, as illustrated on the right side of FIG. 16. The deformable convolution module generates offsets based on feature information at different times and adaptively adjusts the size of the convolution kernel using the offsets. The deformable convolution module effectively generates relationships between features with different temporal information using the adjusted kernels. The INRVC model can be trained so that the deformable convolution module generates offsets and generates temporal information between features. The INRVC model predicts motion between frames by generating an offset based on the outputs of the upscaling module as illustrated in Figure 15, adjusting the size of the kernel using the offset, and utilizing the temporal correlation between features generated by the adjusted kernel.
[0159] As an example, the INVRC model can utilize the features of spatiotemporally adjacent blocks, as illustrated in the example of Fig. 17. By extending the NeRV model, which is an INRVC model that receives a conventional temporal index t as input, the block index (i, j) of a Coding Tree Unit (CTU) or a CU can be used for training. At this time, by extracting block signal information using the aforementioned block index, the INVRC model can be more usefully applied to existing coding devices. Here, the CTU and CU indicated by the block index represent the current CTU and the current block, respectively. Hereinafter, for convenience, the block represents the current block.
[0160] In the example of Fig. 17, local features including spatiotemporal information of video frames, including block index (i, j) and time index t, are additionally used as inputs of the INVRC model. Here, local features are extracted from video frames by a deep learning-based extraction model. As local features, latent features including features of the (i, j)-th block or features of spatiotemporally adjacent blocks are used. For example, spatiotemporally adjacent blocks are blocks that have already been decoded, and represent samples of blocks located to the left or above the current block in intra prediction. In inter prediction, neighboring blocks represent blocks in the preceding frame in the decoding order. In the example of Fig. 17, the INVRC model combines local features (δ) that are features of the (i, j)-th block and spatiotemporally adjacent blocks. t , δ t-1 ) is additionally used as input.
[0161] For example, a video encoding device trains the extraction model and the block-based INVRC model as illustrated in FIG. 17 end-to-end, thereby causing the extraction model to extract local features and causing the INVRC model to represent blocks. The video encoding device transmits the block-based INVRC model to a video decoding device using the encoding pipeline described above. Additionally, the video encoding device encodes local features associated with a block and transmits the encoded local features to the video decoding device. The video decoding device reconstructs the block-based INVRC model using the decoding pipeline described above. Additionally, the video decoding device decodes the local features. Thereafter, the video decoding device can input the block index, the time index, and the local features into the block-based INVRC model to reconstruct the block corresponding to the block index.
[0162] The video encoding device encodes the INVRC model according to an encoding pipeline as in the example of FIG. 12, similar to the method of NeRV, and then transmits the encoded model to the video decoding device. The INVRC model is encoded by applying model pruning, quantization, and entropy encoding to the INVRC model after training to reduce its size. At this time, the quantization process can select a quantization method according to post-training quantization or training considering quantization. The video decoding device restores the INVRC model according to a video decoding pipeline as in the example of FIG. 13. Considering universality for heterogeneous devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), the INVRC model can be implemented using a floating point operation method or a fixed point operation method using N (e.g., 8, 16, 32, etc.) bit model parameters.
[0163] Meanwhile, local features extracted from a video encoding device or decoded from a video decoding device are stored in the DFB (1460). Since the local features can be input to the MTM (1450), the (i, j)-th block generated by the INVRC model may not be stored in the DFB (1460). Meanwhile, the INVRC model can refer to the features of N spatially adjacent blocks and M temporally adjacent blocks. When restoring the (i, j)-th block, the features of N spatially adjacent blocks, the features of M temporally adjacent blocks, and the block temporal index (i, j, t) are used as input. At this time, N and M are selected according to one of the following methods.
[0164] As an example, adjacent blocks can be referenced in a cubic form, as in the example of Fig. 17. For example, blocks from [iN / 2, jN / 2] to [i+N / 2, j+N / 2] spatially and blocks from [iM / 2, jM / 2] to [i+M / 2, j+M / 2] temporally can be referenced.
[0165] As another example, adjacent blocks may be referenced in any predefined shape, such as a cross, a diamond, etc.
[0166] As another example, by measuring the similarity between spatially and / or temporally adjacent blocks defined as described above and the (i, j) block, K candidates extracted in order of similarity starting from the most similar block can be referenced.
[0167] In case of intra prediction, some / all samples of blocks located to the left, top, top left, bottom left, and top right of the current block may be referenced, and in case of inter prediction, samples within a frame preceding in the decoding order may be referenced.
[0168] Below, we describe the combination between the temporal relevance-based INRVC model and existing video coding techniques.
[0169] FIG. 18 is an exemplary diagram showing inter prediction performed by an INRVC model according to one embodiment of the present disclosure.
[0170] As an example, the temporal correlation-based INRVC model performs block-by-block inter prediction as follows. In the existing prediction / decoding, the video decoding device decodes the residual block and adds the residual block and the predicted block to generate a reconstructed block of the current block. Instead of the existing method, the INRVC model configures input signals by combining the temporal index and the block index of the block or frame to be decoded, and outputs a video block at the corresponding temporal and spatial position from the input signals to generate a reconstructed block of the current block. At this time, as in the example of FIG. 18, local features stored in the DFB (1460) can be used to predict motion information. Here, the motion information may be motion information of the current block. The video encoding device compares the existing mode and the INRVC mode in terms of rate distortion optimization, selects a mode that provides better encoding efficiency, and explicitly signals the selected mode at a predetermined block level (CTU, CU, picture, transform unit, etc.). Hereinafter, the flag indicating the use of the INRVC mode is referred to as the INRVC use flag. As in the example of Fig. 18, when a video decoding device decodes a bitstream and the parsed flag indicates the use of the INRVC mode, the video decoding device uses the INRVC model instead of the existing mode.
[0171] In the example of Fig. 18, the video decoding device selects local features of spatiotemporally adjacent blocks from the DFB (1460) to utilize the INVRC model. The video decoding device performs motion prediction using the MTM (1450) based on the selected local features for motion prediction and compensation at the current time t, thereby generating the current feature and block (in conclusion, frame). The video decoding device decode the video block by utilizing the video frame decoded according to the existing video coding technology and the frame generated using the INVRC model together.
[0172] In this disclosure, the INRVC model is described to compress and restore video frames arranged in a sequence of time indices t, but is not necessarily limited thereto. The INRVC model can be applied to process video frames arranged in any sequence. For example, the INRVC model can be applied to compress and restore multi-view video frames arranged in a sequence of time indices v.
[0173] Hereinafter, a method for generating a current image or current block using the INVRC model using the cities of FIGS. 19 and 20 is described.
[0174] FIG. 19 is a flowchart illustrating a method by which an image encoding device encodes a current image according to one embodiment of the present disclosure.
[0175] The image encoding device configures an input of an INRVC model including a current time index of a current image (S1900).
[0176] The image encoding device configures an INRVC model (S1902). Here, the INRVC model includes a multilayer neural network, upscaling modules, and a feature extractor.
[0177] As an example, an image encoding device can configure an INRVC model by training the INRVC model in a direction that reduces the loss function using input and original images (i.e., GT images). At this time, a combination of L1 loss and SSIM (Structural Similarity Index) loss can be used as the loss function, as in the example of the NeRV model.
[0178] The image encoding device applies the input to a multilayer neural network of the INRVC model to generate a multilayer neural network output (S1904).
[0179] The video encoding device predicts motion information between the current video and the previous output using the previous output of the INRVC model corresponding to the previous time index (S1906).
[0180] The image encoding device inputs a combination of a multilayer neural network output and motion information into the upscaling module of the INRVC model to generate a current output corresponding to the current time index (S1908).
[0181] The INRVC model generates spatiotemporal information from features passed from the upscaling module using a feature extractor.
[0182] As an example, the INRVC model additionally includes a quality assessor and an OF estimator.
[0183] The INRVC model generates the current output and the previous output corresponding to the previous time index, and then estimates the optical flow for the current and previous outputs using the OF estimator. The INRVC model improves the current and previous outputs using the estimated optical flow. At this time, the INRVC model uses a quality estimator to estimate the quality of the previous output. If the quality of the previous output is above a preset threshold, the INRVC model estimates the optical flow. On the other hand, if the quality of the previous output is below the preset threshold, the INRVC model skips the optical flow estimation.
[0184] As another example, the INRVC model additionally includes a deformable convolution module.
[0185] The INRVC model generates a previous output corresponding to the current output and the previous time index, and then uses a deformable convolution module to create a temporal association between the current and previous outputs. The deformable convolution generates an offset from the current and previous outputs and adaptively adjusts the size of the convolution kernel based on the offset. The deformable convolution uses the adjusted convolution kernel to create a temporal association between the current and previous outputs.
[0186] The video encoding device stores the current output in the DFB (S1910).
[0187] If the input of the INRVC model contains only the current time index, the current output corresponds to the restored image of the current image.
[0188] Alternatively, the input of the INRVC model can be configured as follows. The video encoding device generates local features. Here, the local features are features of the current block or blocks spatiotemporally adjacent to the current block, and are extracted by a deep learning-based extraction model. The video encoding device configures the input of the INRVC model to additionally include local features in addition to the index of the current block and the current time index. The video encoding device predicts motion information of the current block using the local features. At this time, as the current output, the video encoding device generates a first reconstructed block of the current block corresponding to the index of the current block and the current time index.
[0189] The image encoding device applies an existing image encoding process to the current block to generate a second restoration block.
[0190] The video encoding device determines the INRVC utilization flag based on the first and second reconstruction blocks. Here, the INRVC utilization flag indicates whether to use the INRVC model for each block. In terms of rate distortion optimization, the video encoding device determines the INRVC utilization flag. For example, if the first reconstruction block is optimal, the INRVC utilization flag may be determined to be true. On the other hand, if the second reconstruction block is optimal, the INRVC utilization flag may be determined to be false. The video encoding device encodes the INRVC utilization flag.
[0191] The video encoding device checks the INRVC utilization flag. If the INRVC utilization flag is true, the video encoding device encodes the parameters of the INRVC model according to the encoding pipeline illustrated in FIG. 12. On the other hand, if the INRVC utilization flag is false, the video encoding device can encode encoding information according to the existing video encoding process.
[0192] FIG. 20 is a flowchart illustrating a method for an image decoding device to restore a current image according to one embodiment of the present disclosure.
[0193] The video decoding device configures an input of an INRVC model including a current time index of the current video (S2000).
[0194] The image decoding device obtains an INRVC model (S2002). Here, the INRVC model includes a multilayer neural network, upscaling modules, and a feature extractor.
[0195] As an example, a video decoding device decodes parameters of an INRVC model from a bitstream. Using the decoded parameters, the video decoding device constructs an INRVC model according to a decoding pipeline as illustrated in FIG. 13.
[0196] The image decoding device applies the input to the multilayer neural network of the INRVC model to generate a multilayer neural network output (S2004).
[0197] The video decoding device predicts motion information between the current video and the previous output using the previous output of the INRVC model corresponding to the previous time index (S2006).
[0198] The image decoding device inputs a combination of a multilayer neural network output and motion information into the upscaling module of the INRVC model to generate a current output corresponding to the current time index (S2008).
[0199] Since the operation of the INRVC model has already been described in relation to the operation of the image encoding device, further detailed description is omitted.
[0200] The video decoding device stores the current output in the DFB (S2010).
[0201] If the input of the INRVC model contains only the current time index, the current output corresponds to the restored image of the current image.
[0202] Alternatively, the input of the INRVC model can be configured as follows. The video decoding device decodes the INRVC utilization flag from the bitstream. Here, the INRVC utilization flag indicates whether the INRVC model is used for each block. If the INRVC utilization flag is false, the video decoding device can generate a reconstruction block of the current block according to the existing video decoding method. On the other hand, if the INRVC utilization flag is true, the video decoding device configures the input as follows to generate a reconstruction block of the current block.
[0203] A video decoding device decodes local features from a bitstream. Local features are latent features of the current block or blocks spatiotemporally adjacent to the current block, and are extracted using a deep learning-based extraction model. The video decoding device configures the input of an INRVC model to include local features in addition to the current block index and current time index. The video decoding device uses the local features to predict motion information of the current block. At this time, as the current output, the video decoding device generates a reconstructed block of the current block corresponding to the current block index and current time index.
[0204] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.
[0205] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.
[0206] Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).
[0207] The above description is merely an example of the technical idea of the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.
[0208] (Explanation of symbols)
[0209] 155: Entropy encoding unit
[0210] 510: Entropy decryption unit
[0211] 1410: Multi-layer Perceptron (MLP)
[0212] 1420, 1440: Upscaling module
[0213] 1430: Feature Extractor
[0214] 1450: MTM (Motion Transfer Module)
[0215] 1460: Decoded Feature Buffer (DFB)
[0216] 1510: Quality Evaluator
[0217] 1520: Optical Flow Estimator
[0218]
[0219] CROSS-REFERENCE TO RELATED APPLICATION
[0220] This patent application claims priority to Korean patent application No. 10-2023-0063866, filed in Korea on May 17, 2023, and Korean patent application No. 10-2024-0048651, filed in Korea on April 11, 2024, the entire contents of which are incorporated herein by reference.
Claims
1. A method for restoring a current image, performed by an image decoding device, A step of configuring an input of an INRVC (Implicit Neural Representation for Video Coding) model including a current time index of the current image; A step of obtaining the INRVC model, wherein the INRVC model includes a multi-layer perceptron and an upscaling module; A step of applying the above input to the multilayer neural network to generate a multilayer neural network output; A step of predicting motion information between the current image and the previous output using the previous output of the INRVC model corresponding to the previous time index; and A step of inputting the combination of the above multilayer neural network output and the above motion information into the above upscaling module to generate a current output corresponding to the above current time index. A method, characterized by including:
2. In paragraph 1, The steps for generating the current output above are: A method, characterized by comprising a step of generating spatiotemporal information from features transmitted from the above upscaling module.
3. In paragraph 1, The steps for generating the current output above are: A step of generating the current output and the previous output of the INRVC model corresponding to the previous time index; A step of estimating optical flow for the current output and the previous output; and A step of improving the current output and the previous output using the optical flow. A method, characterized in that it further comprises:
4. In paragraph 3, The steps for generating the current output above are: a step of evaluating the quality of the above previous output; and Step to check the quality of the previous output Including more, A method characterized in that the step of estimating the optical flow is performed when the quality of the above-mentioned previous output is greater than or equal to a preset threshold.
5. In paragraph 1, The steps for generating the current output above are: generating a previous output of the INRVC model corresponding to the current output and the previous time index; and A step of generating a temporal correlation between the current output and the previous output. A method, characterized by including:
6. In paragraph 5, The step of creating the above temporal correlation is: A step of generating an offset from the current output and the previous output; a step of adaptively adjusting the size of the convolution kernel based on the above offset; and A step of generating a temporal correlation between the current output and the previous output using the above-mentioned adjusted convolution kernel. A method, characterized by including:
7. In paragraph 1, A step of decoding an INRVC utilization flag from a bitstream, wherein the INRVC utilization flag indicates whether the INRVC model is used for each block; and Step to check the above INRVC utilization flag Including more, A method characterized in that the step of configuring the input is performed when the INRVC utilization flag is true.
8. In paragraph 7, The steps for configuring the above input are: A step of decoding local features from a bitstream, wherein the local features are latent features of a current block or blocks spatiotemporally adjacent to the current block, and are extracted by a deep learning-based extraction model; a step of storing the above local features in a decoded feature buffer; and A method characterized in that the input is configured to additionally include local features in addition to the index of the current block and the current time index.
9. In paragraph 8, The step of predicting the above movement information is: A method characterized by predicting motion information of the current block using the above local features.
10. In paragraph 8, The steps for generating the current output above are: A method characterized by generating a restoration block of the current block corresponding to the index of the current block and the current time index.
11. A method for encoding a current image, performed by an image encoding device, A step of configuring an input of an INRVC (Implicit Neural Representation for Video Coding) model including a current time index of the current image; A step of configuring the INRVC model, wherein the INRVC model includes a multi-layer perceptron and an upscaling module; A step of applying the above input to the multilayer neural network to generate a multilayer neural network output; A step of predicting motion information between the current image and the previous output using the previous output of the INRVC model corresponding to the previous time index; and A step of inputting the combination of the above multilayer neural network output and the above motion information into the above upscaling module to generate a current output corresponding to the above current time index. A method, characterized by including:
12. In paragraph 11, The steps for configuring the above input are: A step of generating local features, wherein the local features are features of the current block or blocks spatiotemporally adjacent to the current block, and are extracted by a deep learning-based extraction model; and A method characterized in that the input is configured to additionally include local features in addition to the index of the current block and the current time index.
13. In paragraph 12, The steps for generating the current output above are: A method characterized by generating a first restoration block of the current block corresponding to the index of the current block and the current time index.
14. In paragraph 13, A step of generating a second restoration block by applying an image encoding process to the current block; A step of determining an INRVC utilization flag based on the first restoration block and the second restoration block, wherein the INRVC utilization flag indicates whether to use the INRVC model for each block; and Step of encoding the above INRVC utilization flag A method, characterized in that it further comprises:
15. In paragraph 14, Further comprising a step of checking the INRVC utilization flag, A method characterized in that it further includes a step of encoding parameters of the INRVC model when the INRVC utilization flag is true.
16. In paragraph 12, A method, characterized in that it further comprises a step of storing the local features in a decoded feature buffer.
17. In paragraph 11, The steps for generating the current output above are: A method, characterized in that it further comprises a step of generating spatiotemporal information from features transmitted from the above upscaling module.
18. A method for providing video data to a video decoding device, A step of encoding the above video data into a bitstream; and A step of transmitting the above bitstream to the image decoding device Including, The step of encoding the above video data is: A step of constructing an input of the INRVC model including the current time index of the current image; A step of configuring an INRVC (Implicit Neural Representation for Video Coding) model, wherein the INRVC model includes a multi-layer perceptron and an upscaling module; A step of applying the above input to the multilayer neural network to generate a multilayer neural network output; A step of predicting motion information between the current image and the previous output using the previous output of the INRVC model corresponding to the previous time index; and A step of inputting the combination of the above multilayer neural network output and the above motion information into the above upscaling module to generate a current output corresponding to the above current time index. A method, characterized by including:
Citation Information
Patent Citations
How to organize and create a color chart with uniform color distribution
KR1020240169818A
KR20230022093A
KR20230041601A
