Video encoding method and apparatus using implicit neural representation model based on temporal correlation

Abstract

The embodiment of the invention provides a video coding method and device using an implicit neural representation model based on temporal correlation. In an embodiment, an image decoding device constructs an input to a video coding implicit neural representation model (INRVC model), the input including a current temporal index of a current image, and obtains the INRVC model. Here, the INRVC model includes a multi-layer sensor and an up-sampling module. An image decoding apparatus applies an input to a multi-layer perceptron and generates a multi-layer perceptron output. The image decoding apparatus predicts motion information between the current image and a previous output using the previous output of the INRVC model corresponding to the previous time index, and inputs a combination of the multi-layer perceptron output and the motion information to an up-sampling module, thereby generating a current output corresponding to the current time index.

Description

TECHNICAL FIELD

[0001] The disclosure relates to a video encoding method and apparatus using an implicit neural representation model based on time correlation. BACKGROUND

[0002] The statements in this section merely provide background information related to the disclosure and can not constitute the prior art.

[0003] Because video data has a large amount of data compared to audio or still image data, a large amount of hardware resources (including memory) is required to store or transmit the video data without a process for compression.

[0004] Thus, an encoder is generally used to compress and store or transmit video data. A decoder receives compressed video data, decompresses the received compressed video data, and plays the decompressed video data. Video compression techniques include H.264 / Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), which has improved coding efficiency of about 30% or more compared to HEVC.

[0005] However, as the image size, resolution, and frame rate gradually increase, the amount of data to be encoded also increases. Thus, there is a need for a new compression technique that provides higher coding efficiency and improved image enhancement effects than existing compression techniques.

[0006] Recently, deep learning-based image processing techniques have been applied to existing coding element techniques. In the existing coding techniques, compression techniques such as inter prediction, intra prediction, in-loop filter, transform, etc. can employ deep learning-based image processing techniques to improve coding efficiency. Typical application examples include inter prediction based on a virtual reference frame generated by deep learning, in-loop filter based on a denoising model, etc.

[0007] Meanwhile, recently, implicit neural representation models that represent images and different data representations in a neural network structure have been actively researched. The conventional video representation scheme in use is an explicit representation scheme that assigns an RGB pixel value to each pixel position. To replace such an explicit representation scheme, a new implicit neural representation scheme is newly introduced, which provides a neural network representation of a function that generates (r, g, b) values at (x, y) coordinates where image pixels are located. Video encoding using the implicit neural representation scheme is referred to as implicit neural representation-based video coding (INRVC) or implicit neural representation-based video coding. The existing INRVC technique exclusively utilizes information on the current video frame. Thus, to improve video coding efficiency and enhance video quality, the INRVC technique needs a method of utilizing past video frame information. SUMMARY

[0008] [Technical Problem] The disclosure seeks to provide a video encoding method and apparatus that utilizes temporal correlation based on information about a current video frame and past video frames when performing video encoding based on implicit neural representation (implicit neural representation for video coding or INRVC).

[0009] [Technical Solution] At least one aspect of the disclosure provides a method for reconstructing a current image by a video decoding apparatus. The method includes constructing an input to a video coding implicit neural representation model (INRVC model), the input including a current time index of the current image. The method also includes obtaining the INRVC model including a multi-layer perceptron and an upsampling module. The method also includes generating a multi-layer perceptron output by applying the input to the multi-layer perceptron. The method also includes predicting motion information between the current image and a previous output of the INRVC model corresponding to a previous time index. The method also includes generating a current output corresponding to the current time index by inputting the combined multi-layer perceptron output and the motion information to the upsampling module.

[0010] Another aspect of the disclosure provides a method for encoding a current image by a video encoding apparatus. The method includes constructing an input to a video coding implicit neural representation model (INRVC model), the input including a current time index of the current image. The method also includes constructing the INRVC model including a multi-layer perceptron and an upsampling module. The method also includes generating a multi-layer perceptron output by applying the input to the multi-layer perceptron. The method also includes predicting motion information between the current image and a previous output of the INRVC model corresponding to a previous time index. The method also includes generating a current output corresponding to the current time index by inputting the combined multi-layer perceptron output and the motion information to the upsampling module.

[0011] Still another aspect of the disclosure provides a method for providing video data to a video decoding apparatus. The method includes encoding video data into a bitstream and transmitting the bitstream to the video decoding apparatus. Encoding the video data includes constructing an input to a video coding implicit neural representation model (INRVC model), the input including a current time index of the current image. Encoding the video data also includes constructing the INRVC model including a multi-layer perceptron and an upsampling module. Encoding the video data also includes generating a multi-layer perceptron output by applying the input to the multi-layer perceptron. Encoding the video data also includes predicting motion information between the current image and a previous output of the INRVC model corresponding to a previous time index. Encoding the video data also includes generating a current output corresponding to the current time index by inputting the combined multi-layer perceptron output and the motion information to the upsampling module.

[0012] [Advantageous Effects] As described above, the disclosure provides a video encoding method and apparatus that utilizes temporal correlation based on information about a current video frame and past video frames when performing video encoding based on implicit neural representation. Thereby, the video encoding method and apparatus improves video encoding efficiency and enhances video quality. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a block diagram of a video encoding apparatus to which the techniques of the disclosure can be implemented.

[0014] Figure 2 A method for partitioning a block using a quad-tree plus binary-tree ternary-tree (QTBTTT) structure is illustrated.

[0015] Figure 3a and Figure 3b A plurality of intra prediction modes including a wide-angle intra prediction mode is illustrated.

[0016] Figure 4 A neighboring block of a current block is illustrated.

[0017] Figure 5 is a block diagram of a video decoding apparatus to which the techniques of the disclosure can be implemented.

[0018] Figure 6 is a diagram illustrating an operation of a convolution layer.

[0019] Figure 7 is a diagram illustrating a single image super-resolution (SISR) network.

[0020] Figure 8 is a diagram illustrating a residual block utilized in SISR.

[0021] Figure 9 is a diagram illustrating a convolutional neural network (CNN) based fixed coefficient loop filter.

[0022] Figure 10 is a diagram illustrating a neural representation for a video model (NeRV).

[0023] Figure 11a and Figure 11b is a diagram illustrating a concept of a NeRV model according to at least one embodiment of the disclosure.

[0024] Figure 12 is a diagram illustrating an encoding pipeline of a NeRV model according to at least one embodiment of the disclosure.

[0025] Figure 13 is a diagram illustrating a decoding pipeline of a NeRV model according to at least one embodiment of the disclosure.

[0026] Figure 14 FIG. 1 is a diagram illustrating an implicit neural representation video coding model (INRVC model) including a loop according to at least one embodiment of the present disclosure.

[0027] Figure 15 FIG. 2 is a diagram illustrating an INRVC model including optical flow according to at least one embodiment of the present disclosure.

[0028] Figure 16 FIG. 3 is a diagram illustrating deformable convolution.

[0029] Figure 17 FIG. 4 is a diagram illustrating a block-based INRVC model according to another embodiment of the present disclosure.

[0030] Figure 18 FIG. 5 is a diagram illustrating inter prediction performed by an INRVC model according to at least one embodiment of the present disclosure.

[0031] Figure 19 FIG. 6 is a flowchart of a method of encoding a current block by a video coding apparatus according to at least one embodiment of the present disclosure.

[0032] Figure 20 FIG. 7 is a flowchart of a method of reconstructing a current block by a video decoding apparatus according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Hereinafter, some embodiments of the present disclosure are described in detail with reference to the accompanying drawings. In the following description, the same drawing reference numerals are used for the same elements even in different drawings. Further, in the following description of some embodiments, for the purpose of clarity and conciseness, detailed description of related known components and functions can be omitted when it is deemed that such an omission does not cause ambiguity to the subject matter of the present disclosure.

[0034] Figure 1 FIG. 1 is a block diagram of a video coding apparatus to which the techniques of the present disclosure can be implemented. Hereinafter, the video coding apparatus and components of the apparatus are described with reference to the diagram of FIG. 1. Figure 1

[0035] The coding apparatus can include an image partitioner 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a rearrangement unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filter unit 180, and a memory 190.

[0036] Each component of the coding apparatus can be implemented as hardware or software or as a combination of hardware and software. Further, the functions of each component can be implemented as software, and a microprocessor can also be implemented to perform the functions of the software corresponding to each component.​

[0037] A video is composed of one or more sequences including a plurality of pictures. Each picture is divided into a plurality of regions, and encoding is performed for each region. For example, a picture is divided into one or more tiles or / and slices. Here, one or more tiles can be defined as a tile group. Each tile or / and slice is divided into one or more coding tree units (CTUs). In addition, each CTU is divided into one or more coding units (CUs) by a tree structure. Information applied to each coding unit (CU) is coded as a syntax of the CU, and information commonly applied to CUs included in one CTU is coded as a syntax of the CTU. Furthermore, information commonly applied to all blocks in one slice is coded as a syntax of a slice header, and information applied to all blocks constituting one or more pictures is coded as a picture parameter set (PPS) or a picture header. Furthermore, information commonly referred to a plurality of pictures is coded to a sequence parameter set (SPS). In addition, information commonly referred to one or more SPSs is coded to a video parameter set (VPS). Furthermore, information commonly applied to one tile or tile group can also be coded as a syntax of a tile or tile group header. Syntaxes included in the SPS, PPS, slice header, tile or tile group header can be referred to as high-level syntax.

[0038] The picture partitioner 110 determines the size of a coding tree unit (CTU). Information about the size of the CTU (CTU size) is coded as a syntax of the SPS or PPS and is delivered to the video decoding apparatus.

[0039] The picture partitioner 110 divides each picture constituting a video into a plurality of coding tree units (CTUs) having a predetermined size, and then recursively divides the CTUs by using a tree structure. A leaf node in the tree structure becomes a coding unit (CU), which is a basic unit of encoding.

[0040] The tree structure can be a quad tree (QT) in which a higher node (or parent node) is divided into four lower nodes (or child nodes) having the same size. The tree structure can also be a binary tree (BT) in which a higher node is divided into two lower nodes. The tree structure can also be a ternary tree (TT) in which a higher node is divided into three lower nodes in a ratio of 1:2:1. The tree structure can also be a structure in which two or more of the QT structure, the BT structure, and the TT structure are mixed. For example, a quad tree plus binary tree (QTBT) structure can be used or a quad tree plus binary tree ternary tree (QTBTTT) structure can be used. Here, a binary tree ternary tree (BTTT) is added to the tree structure to be referred to as a multi-type tree (MTT).

[0041] Figure 2 is a diagram for describing a method of dividing a block by using a QTBTTT structure.

[0042] like Figure 2 As shown, the CTU can first be partitioned into a QT structure. Quadtree partitioning can be recursive until the size of the partitioned blocks reaches the minimum block size (MinQTSize) allowed for leaf nodes in the QT. A first flag (QT_split_flag) indicating whether each node in the QT structure is partitioned into the four lower-level nodes is encoded by the entropy encoder 155 and sent as a signal to the video decoding device. When the leaf node of the QT is not larger than the maximum block size (MaxBTSize) allowed for the root node in the BT, the leaf node can be further partitioned into at least one of a BT structure or a TT structure. Multiple partitioning directions can exist in the BT structure and / or TT structure. For example, two directions can exist: the direction in which the blocks of the corresponding node are horizontally partitioned and the direction in which the blocks of the corresponding node are vertically partitioned. Figure 2 As shown, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether a node has been split, a flag indicating the splitting direction (vertical or horizontal), and / or a flag indicating the splitting type (binary or ternary) if a node has been split are encoded by the entropy encoder 155 and sent to the video decoding device.

[0043] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node has been divided into the four lower-level nodes, the CU splitting flag (split_cu_flag) indicating whether a node has been split can also be encoded. When the value of the CU splitting flag (split_cu_flag) indicates that each node has not been split, the block of the corresponding node becomes a leaf node in the splitting tree structure and becomes a CU as the basic unit of encoding. When the value of the CU splitting flag (split_cu_flag) indicates that each node has been split, the video encoding device first begins encoding the first flag using the above scheme.

[0044] When QTBT is used as another example of a tree structure, two types can exist: one where the corresponding node's block is horizontally divided into two blocks of the same size (i.e., symmetrical horizontal division), and another where the corresponding node's block is vertically divided into two blocks of the same size (i.e., symmetrical vertical division). A split flag (split_flag) indicating whether each node in the BT structure is divided into blocks of the lower level, and split type information indicating the split type, are encoded by an entropy encoder 155 and delivered to the video decoding device. Additionally, a type that divides the corresponding node's block into two asymmetrical blocks can also be presented. Asymmetrical forms may include the form where the corresponding node's block is divided into two rectangular blocks with a 1:3 size ratio, or the form where the corresponding node's block is divided along a diagonal direction.

[0045] A CU can have different sizes according to QTBT or QTBTT partitioning from a CTU. In the following, a block corresponding to a CU to be encoded or decoded (i.e., a leaf node of QTBTT) is referred to as a "current block". Due to adoption of QTBTT partitioning, the shape of the current block can be rectangular shape in addition to square shape.

[0046] The predictor 120 predicts the current block to generate a prediction block. The predictor 120 includes an intra predictor 122 and an inter predictor 124.

[0047] In general, each of the current blocks in a picture can be predictively encoded. In general, the prediction of the current block can be performed by using an intra prediction technique (using data from the picture including the current block) or an inter prediction technique (using data from a picture that is encoded before the picture including the current block). The inter prediction includes both uni-prediction and bi-prediction.

[0048] The intra predictor 122 predicts pixels in the current block by using pixels (reference pixels) located adjacent to the current block in the current picture including the current block. According to a prediction direction, there are a plurality of intra prediction modes. For example, as shown in Figure 3a , the plurality of intra prediction modes can include 2 non-directional modes including a planar mode and a DC mode, and can include 65 directional modes. Pixels and an arithmetic equation to be used are defined differently according to each prediction mode.

[0049] To perform efficient directional prediction for the current block having a rectangular shape, directional modes (intra prediction modes #67 to #80, #-1 to #-14) as shown by dotted arrows in Figure 3b may be additionally used. The directional modes can be referred to as "wide-angle intra prediction modes". In Figure 3b , the arrows indicate corresponding reference samples for prediction and do not represent prediction directions. The prediction directions are opposite to the directions shown by the arrows. When the current block has a rectangular shape, the wide-angle intra prediction mode is a mode in which prediction is performed in a direction opposite to a specific directional mode without additional bit transmission. In this case, in the wide-angle intra prediction mode, some wide-angle intra prediction modes available for the current block can be determined by a ratio of the width and the height of the current block having a rectangular shape. For example, when the current block has a rectangular shape having a height smaller than a width, wide-angle intra prediction modes (intra prediction modes #67 to #80) having an angle smaller than 45 degrees are available. When the current block has a rectangular shape having a width greater than a height, wide-angle intra prediction modes having an angle greater than -135 degrees are available.

[0050] The intra predictor 122 can determine an intra prediction to be used for encoding the current block. In some examples, the intra predictor 122 can encode the current block by using a plurality of intra prediction modes, and can also select an appropriate intra prediction mode to be used from among the tested modes. For example, the intra predictor 122 can calculate rate-distortion values by using rate-distortion analysis for a plurality of tested intra prediction modes, and can also select an intra prediction mode having the best rate-distortion characteristics among the tested modes.

[0051] The intra predictor 122 selects one of a plurality of intra prediction modes, and predicts the current block by using neighboring pixels (reference pixels) and an arithmetic equation determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoder 155, and is delivered to a video decoding apparatus.

[0052] The inter predictor 124 generates a prediction block for the current block by using a motion compensation process. The inter predictor 124 searches for a block most similar to the current block among reference pictures that are encoded and decoded earlier than the current picture, and generates a prediction block for the current block by using the searched block. In addition, a motion vector (MV) corresponding to a displacement between the current block in the current picture and the prediction block in the reference picture is generated. Generally, motion estimation is performed for a luma component, and a motion vector calculated based on the luma component is used for both the luma component and the chroma component. Motion information including information about the reference picture and information about the motion vector used to predict the current block is encoded by the entropy encoder 155, and is delivered to a video decoding apparatus.

[0053] The inter predictor 124 can also perform interpolation on a reference picture or a reference block in order to increase the accuracy of prediction. In other words, a sub-sample between two consecutive integer samples is interpolated by applying a filter coefficient to a plurality of consecutive integer samples including the two integer samples. When performing a process of searching for a block most similar to the current block with respect to an interpolated reference picture, a motion vector can be expressed not in integer sample unit accuracy but in decimal unit accuracy. The accuracy or resolution of a motion vector can be set differently for each target region (e.g., a unit such as a slice, a tile, a CTU, a CU, etc.) to be encoded. When this adaptive motion example resolution (AMVR) is applied, information about a motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, information about a motion vector resolution applied to each CU is signaled. The information about the motion vector resolution can be information indicating the accuracy of a motion vector difference to be described below.

[0054] Meanwhile, the inter predictor 124 can perform inter prediction by using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors representing a block position most similar to the current block in each of the reference pictures are used. The inter predictor 124 selects a first reference picture and a second reference picture from a reference picture list 0 (RefPicList0) and a reference picture list 1 (RefPicListl), respectively. The inter predictor 124 also searches for blocks most similar to the current block in the respective reference pictures to generate a first reference block and a second reference block. In addition, a prediction block of the current block is generated by averaging or weighted-averaging the first reference block and the second reference block. Further, motion information including information on the two reference pictures used to predict the current block and including information on the two motion vectors is delivered to the entropy encoder 155. Here, the reference picture list 0 can consist of pictures preceding the current picture in display order among the pre-reconstructed pictures, and the reference picture list 1 can consist of pictures following the current picture in display order among the pre-reconstructed pictures. However, although not specifically limited thereto, a pre-reconstructed picture following the current picture in display order can additionally be included in the reference picture list 0. Conversely, a pre-reconstructed picture preceding the current picture can also be additionally included in the reference picture list 1.

[0055] In order to minimize the number of bits consumed for encoding the motion information, various methods can be used.

[0056] For example, when the reference picture and the motion vector of the current block are the same as those of a neighboring block, information of the neighboring block capable of being identified is encoded to deliver the motion information of the current block to the video decoding apparatus. This method is referred to as a merge mode.

[0057] In the merge mode, the inter predictor 124 selects a predetermined number of merge candidate blocks (hereinafter, referred to as "merge candidates") from neighboring blocks of the current block.

[0058] As shown in Figure 4 As shown in FIG. 2, all or some of a left block A0, a lower-left block Al, an upper block B0, an upper-right block Bl, and an upper-left block B2 neighboring the current block in the current picture can be used as the neighboring blocks for deriving the merge candidates. Further, in addition to the current picture in which the current block is located, blocks located within a reference picture (which can be the same as or different from the reference picture used to predict the current block) can also be used as the merge candidates. For example, a block located at the same location as the current block or a block neighboring the block located at the same location within the reference picture can additionally be used as the merge candidates. If the number of the merge candidates selected by the above-described method is less than a preset number, zero vectors are added to the merge candidates.

[0059] The inter-frame predictor 124 uses a neighboring block configuration to include a merge list containing a predetermined number of merge candidates. It selects a merge candidate from the merge candidates included in the merge list to be used as motion information for the current block, and generates merge index information to identify the selected candidate. The generated merge index information is encoded by the entropy encoder 155 and delivered to the video decoding device.

[0060] The merge-skip mode is a special case of the merge mode. After quantization, when all transform coefficients used for entropy coding are close to zero, only the neighbor block selection information is sent, and the residual signal is not sent. By using the merge-skip mode, relatively high coding efficiency can be achieved for images with slight motion, still images, and screen content images.

[0061] In the following text, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0062] Another method for encoding motion information is the Advanced Motion Vector Prediction (AMVP) pattern.

[0063] In AMVP mode, the inter-frame predictor 124 derives motion vector predictor candidates for the motion vector of the current block by using neighboring blocks of the current block. As neighboring blocks used for deriving motion vector predictor candidates, they can use... Figure 4 The image shows all or some of the following adjacent blocks in the current image: left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2. In addition to the current image where the current block is located, blocks located within a reference image (which may be the same as or different from the reference image used to predict the current block) can also be used as adjacent blocks for deriving motion vector predictor candidates. For example, blocks located in the same location as the current block within the reference image, or blocks adjacent to blocks located in the same location, can be used. If the number of motion vector candidates selected by the above method is less than a preset number, a zero vector is added to the motion vector candidates.

[0064] The inter-frame predictor 124 derives motion vector predictor candidates using motion vectors from adjacent blocks, and determines the motion vector predictor for the current block using the motion vector predictor candidates. Additionally, the motion vector difference is calculated by subtracting the motion vector predictor from the motion vector of the current block.

[0065] The motion vector predictor can be obtained by applying a predefined function (e.g., center value and average value calculation, etc.) to the motion vector predictor candidates. In this case, the video decoding apparatus also knows the predefined function. Furthermore, since the neighboring blocks used to derive the motion vector predictor candidates are blocks whose encoding and decoding have already been completed, the video decoding apparatus can also already know the motion vectors of the neighboring blocks. Therefore, the video encoding apparatus does not need to encode information for identifying the motion vector predictor candidates. Thus, in this case, the information about the motion vector difference and the information about the reference picture used to predict the current block are encoded.

[0066] Meanwhile, the motion vector predictor can also be determined by a scheme of selecting any one of the motion vector predictor candidates. In this case, information for identifying the selected motion vector predictor candidate is additionally encoded jointly with the information about the motion vector difference and the information about the reference picture used to predict the current block.

[0067] The subtractor 130 generates a residual block by subtracting the prediction block generated by the intra predictor 122 or the inter predictor 124 from the current block.

[0068] The transformer 140 transforms a residual signal in the residual block having pixel values in a spatial domain into a transform coefficient in a frequency domain. The transformer 140 can transform the residual signal in the residual block by using the total size of the residual block as a transform unit, or can also divide the residual block into a plurality of sub-blocks and can perform the transformation by using the sub-blocks as transform units. Alternatively, the residual block is divided into two sub-blocks, a transform region and a non-transform region, to transform the residual signal using only the transform region sub-block as a transform unit. Here, the transform region sub-block can be one of two rectangular blocks having a size ratio of 1:1 based on a horizontal axis (or a vertical axis). In this case, a flag (cu_sbt_flag) indicates that only the sub-block is transformed, and directional (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoder 155 and signaled to the video decoding apparatus. Furthermore, the size of the transform region sub-block can have a size ratio of 1:3 based on a horizontal axis (or a vertical axis). In this case, a flag (cu_sbt_quad_flag) indicating that the corresponding division is additionally encoded by the entropy encoder 155 and signaled to the video decoding apparatus.

[0069] Meanwhile, the transformer 140 can perform a transform on the residual block separately in a horizontal direction and a vertical direction. For the transform, different types of transform functions or transform matrices can be used. For example, a pair of transform functions used for horizontal transform and vertical transform can be defined as a multiple transform set (MTS). The transformer 140 can select one transform function pair having the highest transform efficiency among the MTS, and can transform the residual block in each of the horizontal and vertical directions. Information (mts_idx) of the transform function pair in the MTS is encoded by the entropy encoder 155 and signaled to the video decoding apparatus.

[0070] The quantizer 145 quantizes the transform coefficients output from the transformer 140 using a quantization parameter, and outputs the quantized transform coefficients to the entropy encoder 155. The quantizer 145 can also quantize the related residual block immediately without a transform for any block or frame. The quantizer 145 can also apply different quantization coefficients (scaling values) according to positions of the transform coefficients in the transform block. A quantization matrix applied to the quantized transform coefficients arranged in 2 dimensions can be encoded and signaled to the video decoding apparatus.

[0071] The rearrangement unit 150 can perform a rearrangement of coefficient values for the quantized residual values.

[0072] The rearrangement unit 150 can change a 2D coefficient array into a 1D coefficient sequence by using a coefficient scan. For example, the rearrangement unit 150 can output a 1D coefficient sequence by scanning a DC coefficient into a high frequency domain coefficient using a zigzag scan or a diagonal scan. According to a size of a transform unit and an intra prediction mode, a vertical scan scanning a 2D coefficient array in a column direction and a horizontal scan scanning a 2D block type coefficient in a row direction can also be used instead of the zigzag scan. In other words, according to the size of the transform unit and the intra prediction mode, a scan method to be used can be determined among the zigzag scan, the diagonal scan, the vertical scan, and the horizontal scan.

[0073] The entropy encoder 155 generates a bitstream by encoding a sequence of the 1D quantized transform coefficients output from the rearrangement unit 150 using various encoding schemes including context-based adaptive binary arithmetic coding (CABAC), exponential Golomb, etc.

[0074] Further, the entropy encoder 155 encodes information related to the block partitioning, such as a CTU size, a CTU partition flag, a QT partition flag, an MTT partition type, an MTT partition direction, etc., to allow a video decoding apparatus to equally partition the blocks to the video encoding apparatus. Further, the entropy encoder 155 encodes information on a prediction type indicating whether the current block is encoded by intra prediction or inter prediction. The entropy encoder 155 encodes intra prediction information (i.e., information on an intra prediction mode) or inter prediction information (i.e., information on a reference picture index and a motion vector difference in the case of a merge mode, a merge index, and in the case of an AMVP mode) according to the prediction type. Further, the entropy encoder 155 encodes information related to quantization (i.e., information on a quantization parameter and information on a quantization matrix).

[0075] The inverse quantizer 160 de-quantizes the quantized transform coefficients output from the quantizer 145 to generate transform coefficients. The inverse transformer 165 transforms the transform coefficients output from the inverse quantizer 160 from a frequency domain to a spatial domain to reconstruct a residual block.

[0076] The adder 170 adds the reconstructed residual block and the prediction block generated by the predictor 120 to reconstruct the current block. Pixels in the reconstructed current block can be used as reference pixels when performing intra prediction on the next sequential block.

[0077] The loop filter unit 180 performs filtering on reconstructed pixels in order to reduce block artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transform / quantization. The loop filter unit 180 as a loop filter can include all or some of a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.

[0078] The deblocking filter 182 filters boundaries between reconstructed blocks in order to remove block artifacts generated due to block unit encoding / decoding, and the SAO filter 184 and the ALF 186 perform additional filtering for the filtered video from deblocking. The SAO filter 184 and the ALF 186 are filters for compensating for a difference between reconstructed pixels and original pixels that occurs due to lossy encoding. The SAO filter 184 applies an offset as a CTU unit to enhance subjective image quality and encoding efficiency. On the other hand, the ALF 186 performs block unit filtering, and compensates for distortion by dividing a boundary of a corresponding block and a degree of change and applying different filters. Information on filter coefficients to be used for the ALF can be encoded and signaled to a video decoding apparatus.

[0079] The reconstructed blocks filtered by the deblocking filter 182, the SAO filter 184, and the ALF 186 are stored in the memory 190. When all blocks in one picture are reconstructed, the reconstructed picture can be used as a reference picture for inter prediction of blocks within a picture to be encoded later.

[0080] The video encoding device can store the bitstream of the encoded video data in a non-transitory storage medium, or transmit the bitstream to a video decoding device through a communication network.

[0081] Figure 5 is a functional block diagram of a video decoding device to which the techniques of the present disclosure can be implemented. Hereinafter, the video decoding device and components of the device are described with reference to Figure 5

[0082] The video decoding device can include an entropy decoder 510, a rearrangement unit 515, an inverse quantizer 520, an inverse transformer 530, a predictor 540, an adder 550, a loop filter unit 560, and a memory 570.

[0083] Similar to the video encoding device of Figure 1 Each component of the video decoding device can be implemented as hardware or software or as a combination of hardware and software. Furthermore, the functions of each component can be implemented as software, and a microprocessor can also be implemented to perform the functions of the software corresponding to each component.

[0084] The entropy decoder 510 extracts information related to block partitioning by decoding a bitstream generated by a video encoding device to determine a current block to be decoded, and extracts prediction information and information about a residual signal required to reconstruct the current block.

[0085] The entropy decoder 510 determines the size of a CTU by extracting information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), and partitions a picture into CTUs having the determined size. In addition, the CTU is determined to be the highest layer of a tree structure, i.e., a root node, and partition information of the CTU can be extracted to partition the CTU using the tree structure.

[0086] For example, when the CTU is partitioned using a QTBTTT structure, a first flag (QT_split_flag) related to partitioning of a QT is first extracted to partition each node into four nodes of a lower layer. In addition, a second flag (mtt_split_flag) related to partitioning of an MTT, a partition direction (vertical / horizontal), and / or a partition type (binary / ternary) are extracted for a node corresponding to a leaf node of the QT to partition the corresponding leaf node into an MTT structure. As a result, each node under the leaf node of the QT is recursively partitioned into a BT or TT structure. ​

[0087] As another example, when a CTU is partitioned by using a QTBT TT structure, a CU split flag indicating whether a CU is split is extracted. When the corresponding block is split, a first flag (QT_split_flag) can also be extracted. During the partitioning process, for each node, a recursive MTT partition of 0 or more times can occur after a recursive QT partition of 0 or more times. For example, for a CTU, an MTT partition can occur immediately, or conversely, only a QT partition of multiple times can occur.

[0088] As another example, when a CTU is partitioned using a QTBT structure, a first flag (QT_split_flag) related to the partition of a QT is extracted to partition each node into four nodes of a lower level. In addition, a split flag (split_flag) indicating whether a node corresponding to a leaf node of a QT is further partitioned into a BT and partition direction information are extracted.

[0089] Meanwhile, when the entropy decoder 510 determines a current block to be decoded by using the partition of the tree structure, the entropy decoder 510 extracts information about prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 510 extracts a syntax element for intra-prediction information (intra-prediction mode) of the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 extracts information representing a syntax element for inter-prediction information (i.e., a motion vector and a reference picture referred to by the motion vector).

[0090] In addition, the entropy decoder 510 extracts quantization-related information and extracts information about a quantized transform coefficient of the current block as information about a residual signal.

[0091] The rearrangement unit 515 can change the sequence of the 1D quantized transform coefficients entropy-decoded by the entropy decoder 510 into a 2D coefficient array (i.e., a block) again in an order opposite to the coefficient scan order performed by the video encoding apparatus.

[0092] The inverse quantizer 520 dequantizes the quantized transform coefficients and dequantizes the quantized transform coefficients by using a quantization parameter. The inverse quantizer 520 can also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in 2D. The inverse quantizer 520 can perform dequantization by applying a matrix of quantization coefficients (scaling values) from the video encoding apparatus to the 2D array of quantized transform coefficients.

[0093] The inverse transformer 530 generates a residual block for the current block by reconstructing a residual signal by inversely transforming the dequantized transform coefficients from the frequency domain into the spatial domain.

[0094] Further, when the inverse transformer 530 inverse transforms a partial region (sub-block) of the transform block, the inverse transformer 530 extracts a flag (cu_sbt_flag) that only the sub-block of the transform block is transformed, direction (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block. The inverse transformer 530 also inverse transforms transform coefficients of the corresponding sub-block from the frequency domain into the spatial domain to reconstruct a residual signal and fills a region not inverse transformed with a value "0" as a residual signal to generate a final residual block for the current block.

[0095] Further, when the MTS is applied, the inverse transformer 530 determines a transform function or a transform matrix applied in each of horizontal and vertical directions by using MTS information (mts_idx) signaled from the video encoding apparatus. The inverse transformer 530 also performs inverse transform on transform coefficients in the transform block in the horizontal and vertical directions by using the determined transform function.

[0096] The predictor 540 can include an intra predictor 542 and an inter predictor 544. The intra predictor 542 is activated when the prediction type of the current block is intra prediction, and the inter predictor 544 is activated when the prediction type of the current block is inter prediction.

[0097] The intra predictor 542 determines an intra prediction mode of the current block among a plurality of intra prediction modes according to a syntax element of the intra prediction mode extracted from the entropy decoder 510. The intra predictor 542 also predicts the current block by using neighboring reference pixels of the current block according to the intra prediction mode.

[0098] The inter predictor 544 determines a motion vector and a reference picture of a motion vector reference of the current block by using syntax elements for the inter prediction mode extracted from the entropy decoder 510, and predicts the current block by using the motion vector and the reference picture.

[0099] The adder 550 reconstructs the current block by adding the residual block output from the inverse transformer 530 and the prediction block output from the inter predictor 544 or the intra predictor 542. When the current block is intra-predicted, pixels within the reconstructed current block are used as reference pixels.

[0100] The loop filter unit 560 as a loop filter can include a deblocking filter 562, an SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on boundaries between reconstructed blocks in order to remove blocking artifacts that occur due to block unit decoding. The SAO filter 564 and the ALF 566 perform additional filtering on the reconstructed blocks after deblocking filtering in order to compensate for differences between reconstructed pixels and original pixels that occur due to lossy encoding. Filter coefficients of the ALF are determined by using information about the filter coefficients decoded from the bitstream.

[0101] The reconstructed blocks filtered by the deblocking filter 562, the SAO filter 564, and the ALF 566 are stored in the memory 570. When all blocks in one picture are reconstructed, the reconstructed picture can be used as a reference picture for inter prediction of blocks within pictures to be encoded later.

[0102] In some embodiments, the present disclosure relates to encoding and decoding video images as described above. More specifically, the present disclosure provides a video encoding method and apparatus that utilizes temporal correlation based on information about a current video frame and past video frames when performing implicit neural representation based video coding (implicit neural representation for video coding, INRVC).

[0103] The following embodiments can be performed by a plurality of components in a video encoding apparatus. The following embodiments can also be performed by a plurality of components in a video decoding apparatus.

[0104] The video encoding apparatus can generate signaling information associated with the present embodiments in terms of rate-distortion optimization when encoding a current block. The video encoding apparatus can encode the signaling information using the entropy encoder 155 and transmit the encoded signaling information to the video decoding apparatus. The video decoding apparatus can decode the signaling information associated with decoding of the current block from the bitstream using the entropy decoder 510.

[0105] In the following description, the term “target block” can be used interchangeably with a current block or a coding unit (CU), or can refer to a certain region of a coding unit.

[0106] Further, a value of true for one flag indicates a time when the flag is set to 1. In addition, a value of false for one flag indicates a time when the flag is set to 0.

[0107] I. Convolutional Neural Network (CNN) A CNN refers to a neural network composed of a plurality of convolutional layers and pooling layers, and is a deep learning technique known to be most suitable for image processing. A convolutional layer uses a plurality of kernels or filters to extract feature maps, also referred to as “features”. Kernel coefficients constituting the filters are parameters determined during a learning process.

[0108] In the convolutional layers of a CNN, the front-end layers close to the input extract feature maps that are responsive to simple low-level image features such as lines, edges, or faces. In the convolutional layers of a CNN, the back-end layers close to the output extract feature maps that are responsive to higher-level features such as textures and object parts.

[0109] Figure 6 is a diagram illustrating the operation of a convolutional layer according to at least one embodiment of the present disclosure.

[0110] A convolutional layer generates a feature map from an input image using a convolution operation. Figure 6 The example of illustrates a kernel (or filter) having a kernel size of 3x3. The kernel size is also referred to as a filter size. The kernel has kernel parameters or filter parameters, also referred to as weights. Figure 6 The kernel illustrated in the example of has a total of nine kernel parameters. The kernel parameters can be initially set to random values, and their values can be updated based on training.

[0111] A convolutional layer performs a convolution operation by using a block equal to the kernel size in the input image. In this case, the block equal to the kernel size in the input image is referred to as a window.

[0112] When filtering an input image in a raster-scan order, the amount of movement of the window is referred to as a stride. In the example of, the stride is 1. If the stride is set to 2, then a convolution operation that shifts the window by 2 samples is performed, resulting in the horizontal and vertical dimensions of the feature map being half of the horizontal and vertical dimensions of the input image. Figure 6

[0113] As described above, a convolutional layer can include a plurality of filters. The number of filters or the number of kernels is referred to as the number of channels. That is, the number of channels is equal to the number of filters. The number of filters also determines the dimensions of the feature map.

[0114] Padding refers to a method of expanding input data by padding a certain value around the input data before performing a convolution operation. Padding is generally used to control the spatial size of the output data. The value used for padding can be determined by a hyperparameter, but zero padding is generally used. If padding is not used, the spatial size of the output data decreases with each pass through the convolutional layer, which can result in the loss of boundary information. Therefore, in order to avoid this problem, padding is used. Padding can be used to equalize the spatial size of the output data and the input data of the convolutional layer.

[0115] A deconvolutional layer performs the inverse operation of a convolutional layer. A deconvolutional layer generates a desired data image as an output from a feature map as an input.

[0116] ​A pooling layer performs pooling, which is a process of subsampling the feature maps generated by a convolution layer. The pooling layer utilizes a 2x2 window to select samples such that the output is half the width and half the height of the input. In other words, a pooling layer is used to reduce the size of an input image or input feature map by aggregating a 2x2 region into a single sample.

[0117] The opposite face of a pooling layer is defined as an unpooling layer. In contrast to a pooling layer, an unpooling layer is used to increase the dimensionality and is typically used after a deconvolution layer.

[0118] A convolutional encoder-decoder structure is a network structure composed of pairs of convolution and deconvolution layers. A convolutional encoder is composed of convolution layers and pooling layers to output a feature map (or feature vector) from an input image. The final output vector of a convolutional encoder is also referred to as a latent vector. A convolutional decoder is composed of deconvolution layers and unpooling layers to generate an output image from a feature map or a latent vector.

[0119] The input and output of a convolutional encoder-decoder can vary depending on the application and the purpose of the network. For example, the input and output can be an optical flow map, a saliency map, an image frame, etc.

[0120] Figure 7 is a diagram illustrating an SISR network.

[0121] One example application of a CNN is single image super-resolution (SISR). An SISR network generates a high-resolution output image from a low-resolution input image. As shown in Figure 7 , an SISR network can include multiple convolution layers. Each convolution layer includes an activation function, such as a rectified linear unit (ReLU). The parameters of an SISR network can be trained such that the resulting super-resolution (SR) image closely approximates a ground truth (GT).

[0122] An SR method using a CNN can improve SR performance by increasing the depth (e.g., by increasing the number of convolution layers). To overcome the problem of overfitting that can occur in learning as the depth increases, an SISR network can utilize a residual block that can perform skip connections and residual learning. As shown in Figure 8 , in addition to a path for applying a convolution operation to an input feature xl, the residual block includes a skip path. Further, when generating an output xl+1, the residual block can apply a path on which a convolution operation is selected based on learning efficiency or a skip path. In Figure 8 , an example, the residual block includes a batch normalization (BN) layer.

[0123] In one example, an enhanced deep residual network for SISR (EDSR) enhances the performance of the network by increasing the depth by concatenating residual blocks in succession. Another example is accurate image super-resolution using a very deep convolutional network (VDSR), which is a CNN model based on a visual geometry group network (VGG network) using residual learning, which is adding a residual frame to the final output. VDSR adds the residual signal at the very end of the network and thereby adds the residual signal to the input signal.

[0124] As another example, a CNN can be used as a loop filter in a video encoding apparatus or a video decoding apparatus. In this case, the deep learning based loop filter can be employed at any location within the existing loop filter unit 180 or 560 consisting of a deblocking filter, a SAO filter, and an ALF.

[0125] The deep learning based loop filter can be a fixed coefficient loop filter. The fixed coefficient deep learning loop filter utilizes common CNN kernel parameters stored on both sides of the video encoding apparatus and the video decoding apparatus.

[0126] Figure 9 is a diagram showing a CNN based fixed coefficient loop filter.

[0127] The input block (or input frame) is mapped by a normalization QP and passed to the subsequent stages. The normalization QP mapping is used to reduce the inference error when mixing different intensities of quantization noise in the learning and inference processes. The kernel parameters constituting the dense residual unit (DRU) and the convolution layer can be identically stored and used in the video encoding apparatus and the video decoding apparatus. In Figure 9 In the example of, each DRU can include all or part of a convolution layer, a ReLU layer, and a depthwise separable convolution (DSC) layer.

[0128] The fixed coefficient deep learning loop filter needs to provide general performance on a wide range of video frames, and thus they have the disadvantage of increasing the number of layers in the CNN and accordingly lengthening the computation time.

[0129] II. Implicit Neural Representation Model Recently, there has been active research on implicit neural representation models that represent different data including images in neural network structures. The conventional video representation scheme used is an explicit representation scheme in which an RGB pixel value is represented for each pixel position. To replace this explicit representation scheme, a newly introduced implicit neural representation scheme represents a function for converting (x, y) coordinates of a video pixel position into (r, g, b) values through a neural network. The implicit neural representation scheme can be used regardless of the resolution of an image compared to the explicit representation scheme. Thus, for the implicit neural representation model, a multi-layer perceptron (MLP) network including a plurality of fully connected layers (FC) can be used.

[0130] As an example of video encoding using an implicit neural representation model, there is implicit neural representation based video coding (implicit neural representation for video coding, INRVC). As described above, the INRVC technique generates (r, g, b) values with respect to (x, y) coordinates representing a pixel position in an image. As an example, there is a neural representation for video model (NeRV model) for the INRVC technique. As described above, the explicit representation scheme represents a pixel value on an (x, y, t) grid by using a pixel position (x, y) in an image and a time index t of a frame. The implicit neural representation model outputs an RGB pixel value from an input representing a position at (x, y, t). However, training a neural network at each position at (x, y, t) can significantly increase computational complexity, so the NeRV model outputs an RGB image of the entire frame at a time index t by using a neural network structure having only a time index t as an input, as Figure 10 indicated.

[0131] Figure 11a and Figure 11b are diagrams illustrating a concept of a NeRV model according to at least one embodiment of the disclosure.

[0132] To facilitate providing an image output for an input for a time index t, a convolutional layer can be more efficient than an MLP (multi-layer perceptron) network of an existing implicit representation model. The NeRV model can generate an output by using a stacked structure of a NeRV block consisting of a plurality of convolutional layers. As Figure 11a indicated, an output based on an MLP can be compared with an output based on a NeRV block. As Figure 11b indicated, the NeRV block includes a convolutional layer performing convolution, a pixel rearrangement layer, and an activation layer. In Figure 11b an example, C denotes the number of channels, W and H denote the width and height of an input of the NeRV block, and S denotes a scaling factor.

[0133] In addition, the NeRV model embeds a time index t into a higher dimensional space and uses the embedded index as an input, as indicated in Equation 1.

[0134] [Equation 1] As shown in Equation 1, the time index t can be mapped to a 2L-dimensional vector γ(t). By utilizing the embedded time index t in the training of the NeRV model, the NeRV model can better predict video data containing high frequency changes.

[0135] As a loss function of the NeRV model, a combination of L1 loss and structural similarity index loss (SSIM loss) can be used. During the training of the NeRV model, these losses can be calculated at all pixel positions in an image estimated by the NeRV model based on an input and a ground truth (GT) image (i.e., an original image). The L1 loss is calculated by using the absolute value of the difference between a pixel in the estimated image and the GT image. The SSIM loss is calculated based on the mean, standard deviation, and correlation of pixels in the estimated image and the GT image. Based on the calculated losses and during the training process, the weights (or parameters) constituting the NeRV model can then be updated.

[0136] Figure 12 is a diagram illustrating an encoding procedure of a NeRV model according to at least one embodiment of the present disclosure.

[0137] Since the NeRV model can be used to approximate an original video, compressing and transmitting the NeRV model can use a neural network to implement an end-to-end compression technique. An encoding procedure for encoding the NeRV model can include all or part of a video over-fitter 1210, a model pruner 1220, a model quantizer 1230, and a weight encoder 1240, as shown in Figure 12 .

[0138] The video over-fitter 1210 represents an input video frame by the NeRV model. The video encoding device trains the NeRV model by using the loss function described above, so that the NeRV model represents the video frame. The model pruner 1220 uses pruning to simplify the NeRV model structure, which can be an MLP (multi-layer perceptron) or a mixture of MLP and convolutional layers. For example, set weights less than a preset threshold are pruned to zero. The model quantizer 1230 quantizes the pruned weights, for example, weights greater than or equal to a threshold. The weight encoder 1240 applies entropy coding to the quantized weights to generate a bitstream of the weights.

[0139] In the reverse order of the encoding procedure shown in Figure 12 , a decoding procedure for decoding the NeRV model can be shown in Figure 13 . The decoding procedure can include all or part of a weight decoder 1310, a model dequantizer 1320, a model reconstructor 1330, and a video generator 1340, as shown inFigure 13 is shown.

[0140] The weight decoder 1310 applies entropy coding to the bitstream of the weights of the NeRV model to generate quantized weights. The model dequantizer 1320 dequantizes the quantized weights to generate pruned weights. The model reconstructor 1330 reconstructs the NeRV model based on the pruned weights. The video generator 1340 generates reconstructed video frames using the reconstructed NeRV model according to the temporal index.

[0141] As described above, the NeRV model utilizes the temporal index t of a frame as an input. Thereby, for an input video with pixels of size t x h x w, the NeRV model only samples the input video t times instead of t x h x w times, which can result in a significant gain in both encoding and decoding speed.

[0142] As described above, the existing INRVC technique only utilizes the information of the current frame, resulting in limited amount of intra- or inter-frame spatial information available for use in the output computation. To address this issue, an INRVC model solution is described that improves the encoding efficiency of the current frame by utilizing the encoding information of past frames that have already been encoded.

[0143] The following implementations are described for a video decoding apparatus incorporating an INRVC model, however, these implementations can be implemented identically or similarly in a video encoding apparatus incorporating an INRVC model.

[0144] III. Implementations According to the Disclosure Figure 14 is a diagram showing an INRVC model including a loop according to at least one implementation of the disclosure.

[0145] In some implementations, the INRVC model according to the disclosure includes a loop (in-loop), where the loop feeds back the output corresponding to a previous temporal index t-1 when predicting a frame with respect to a current temporal index t. In Figure 14 In the example of FIG. 14, the MLP 1410, the up-sampling module 1420, the feature extractor 1430, and the other up-sampling module 1440 constitute a regular INRVC model. As described above, the regular INRVC model uses an input of the temporal index t to output a current frame. The up-sampling modules 1420 and 1440 include convolution layers and Figure 6 convolution layers shown in FIG. 14, and Figure 11bThe pixel rearrangement layer is included in the INRVC model, and the combined convolutional layer and pixel rearrangement layer are used to upsample the input. The MLP 1410 and the up-sampling modules 1420 and 1440 construct an implicit neural representation with respect to the current frame. The feature extractor 1430 generates spatio-temporal information with respect to the input features. The feature extractor 1430 is trained to generate spatio-temporal information with respect to the current t features using the previous t-1 feature information and the current t feature information.

[0146] The loop includes a motion transfer module (MTM) 1450 and a decoded feature buffer (DFB) 1460 as components. The DFB 1460 is included in the memory 570 and stores feature information of past frames. Since the INRVC model utilizes feature information to generate frames, the feature information of the frames can be stored in the DFB 1460. Thus, the INRVC model according to some embodiments can utilize information with respect to other time indices to encode the current frame without requiring additional input. To effectively utilize the previous t-1 feature information, the MTM 1450 generates an embedding with respect to the previous t-1 feature information, thereby reducing the size of the previous t-1 feature information. The MTM 1450 improves the training efficiency at the current time t based on the implicit neural representation by performing motion information prediction and motion compensation between the previous t-1 feature information and the current t feature information. For example, the motion information can include pixel-wise motion that constitutes an image. The embedding generated by the MTM 1450 is generated to conform to the characteristics of the output of the MLP 1410 and is concatenated with the output of the MLP 1410.

[0147] The INRVC model uses the loop components to generate outputs with respect to the t and t-1 time indices. Since the output with respect to the t-1 time index has already been stored in the DFB 1460, this output is a bypass-type output and can be used to improve the subsequent output with respect to the t time index.

[0148] For example, the MTM 1450 can be implemented as a deep learning model. Then, the INRVC model including the loop can be pre-trained.

[0149] A method in which the INRVC model uses the MTM 1450 to perform motion prediction and compensation is described below.

[0150] Since a video input is used for the INRVC model, temporal information, in addition to spatial information, affects the model performance. The conventional NeRV extracts only one frame per time index, generates each frame separately, and thus does not consider temporal information.

[0151] As one example, to incorporate temporal information, the feature extraction process outputs both the current and previous frames. Then, the INRVC model can utilize temporal correlation based on the optical flow between the two frames. As Figure 15As shown, the INRVC model uses an optical flow estimator or OF estimator 1520 to compute the optical flow over two frames, and utilizes the computed optical flow to improve the output with respect to time indices t and t-1. Optical flow can be a pixel-wise motion vector representing the motion of pixels occurring between the t-th and t-1-th images.

[0152] At this point, when the quality of the frame output from the upsampling module is low, it is difficult to predict accurate optical flow. The INRVC model uses a quality evaluator 1510 to measure the quality of the output previously generated by the upsampling module (specifically, the output with respect to time index t-1). When the output quality for time index t-1 is equal to or higher than a preset threshold, the INRVC model calculates optical flow to improve the output for time indices t and t-1. When the output quality for time index t-1 is lower than the set threshold, the optical flow calculation process is skipped, and the output generated by the upsampling module is used as the final output.

[0153] As another example, the INRVC model can include deformable convolutional modules, and these deformable convolutions can be used to generate temporal information between features. The deformable convolutional modules can replace the quality estimator 1510 and the OF estimator 1520. Figure 16 As shown on the right, deformable convolution can start from a fixed kernel (e.g., Figure 16 The deformable convolution module extracts features using a 3×3 kernel (shown on the left), and furthermore, features can be extracted from more flexible regions. The deformable convolution module generates offsets based on feature information at different time points and adaptively adjusts the size of the convolution kernel using these offsets. The deformable convolution module uses the adjusted kernel to efficiently generate relationships between features with different temporal information. An INRVC model can be trained on the deformable convolution model to generate offsets and temporal information between features. Based on... Figure 15 The output of the upsampling module shown is used to generate offsets, which are then used to adjust the kernel size and generate features from the adjusted kernel. The temporal correlation between the generated features allows the INRVC model to predict motion between frames.

[0154] For example, the INVRC model can utilize the features of spatiotemporally adjacent blocks, such as Figure 17 As shown in the example. The NeRV model is an INRVC model that receives input at time index t, and according to the extended version of the NeRV model of this disclosure, the block index (i, j) of the coding tree unit (CTU) or CU can be used for training. By using this block index to extract block signal information, the INRVC model can be applied more efficiently to existing coding devices. Here, the CTU and CU indicated by the block index represent the current CTU and the current block, respectively. For ease of description, the term "block" refers to the current block in the following text.

[0155] Figure 17 An example of the INVRC model uses a block index (i, j) and a time index t and additional local features containing spatio-temporal information of a video frame as inputs. Here, the local features are extracted from the video frame by a deep learning based extraction model. As the local features, latent features including features of the (i, j)th block or spatio-temporally neighboring blocks are used. For example, the spatio-temporally neighboring blocks are blocks that have already been decoded, which represent samples of blocks located to the left or above the current block in intra prediction. In inter prediction, the neighboring blocks refer to blocks in previous frames in the decoding sequence. In the example of the INVRC model, as inputs, the INVRC model additionally utilizes local features (δ Figure 17 t , δ t-1 ) that are combined features of the (i, j)th block and its spatio-temporally neighboring blocks.

[0156] For example, the video encoding apparatus trains the extraction model and the block-based INVRC model end-to-end, as shown in Figure 17 , enabling the extraction model to extract the local features and the INVRC model to represent the blocks. The video encoding apparatus transmits the block-based INVRC model to the video decoding apparatus using the above-described encoding procedure. In addition, the video encoding apparatus encodes the local features associated with the blocks and transmits the encoded local features to the video decoding apparatus. The video decoding apparatus reconstructs the block-based INVRC model using the above-described decoding procedure. In addition, the video decoding apparatus decodes the local features. Subsequently, the video decoding apparatus inputs the block index, the time index, and the local features into the block-based INVRC model to reconstruct the blocks corresponding to the block index.

[0157] The video encoding apparatus follows the NeRV method to encode the INVRC model according to an encoding procedure, for example, an example in Figure 12 , and then transmits the encoded model to the video decoding apparatus. The trained INVRC model, which is trained, is encoded by reducing its size through model pruning, quantization, and entropy coding applied thereto. The quantization process can select a quantization method based on post-training quantization or training considering quantization. The video decoding apparatus reconstructs the INVRC model according to a video decoding procedure, for example, an example in Figure 13 . Considering its versatility for heterogeneous devices such as central processing units (CPUs) and graphics processing units (GPUs), the INVRC model can be implemented by using a floating point operation method or a fixed point operation method using N bits (e.g., 8, 16, 32 bits, etc.) of model parameters.

[0158] ​Meanwhile, the local features extracted by the video encoding apparatus or decoded by the video decoding apparatus are stored by the DFB 1460. Because the local features can be input to the MTM 1450, the (i, j)th block generated by the INVRC model can not be stored in the DFB 1460. Meanwhile, the INVRC model can refer to the features of N number of spatial neighboring blocks and the features of M number of temporal neighboring blocks. When reconstructing the (i, j)th block, the features of the N number of spatial neighboring blocks, the features of the M number of temporal neighboring blocks, and the block temporal index (i, j, t) are used as inputs. Here, N and M are selected according to one of the following methods.

[0159] For example, as shown in FIG. 4A, the neighboring blocks can be referred to in a cubic form. For example, the spatial neighboring blocks [i-N / 2, j-N / 2] to [i+N / 2, j+N / 2] and the temporal neighboring blocks [i-M / 2, j-M / 2] to [i+M / 2, j+M / 2] can be referred to. Figure 17

[0160] As another example, the neighboring blocks in a cross shape or a diamond shape among other pre-defined arbitrary shapes can be referred to.

[0161] Yet another example measures the similarity between the (i, j)th block and the spatial and / or temporal neighboring blocks defined as described above, extracts K number of candidate blocks in order of the similarity starting from the most similar block, and thereby refers to the extracted K number of candidate blocks in the order.

[0162] In intra prediction, some or all samples of the blocks located to the left, top, top-left, bottom-left, or top-right of the current block can be referred to. In inter prediction, samples in the frames preceding the current frame in decoding order can be referred to.

[0163] A time-dependent-based INRVC model combined with the existing video encoding technology is described below.

[0164] Figure 18 FIG. 4B is a diagram illustrating inter prediction performed by an INRVC model according to at least one embodiment of the present disclosure.

[0165] For example, the time-dependent-based INRVC model performs a block-wise inter prediction as follows. In a conventional prediction / decoding process, the video decoding apparatus decodes a residual block and adds it to a prediction block to generate a reconstructed block of a current block. Instead of the conventional method, the INRVC model constructs input signals by combining the temporal index and the block index of the block or frame to be decoded. Then, it outputs a video block at a corresponding temporal and spatial position according to the input signals to generate the reconstructed block of the current block. At this time, as shown in FIG. 4B, the INRVC model can be implemented in a form of a neural network. Figure 18 ​As shown, local features stored in the DFB 1460 can be used to predict motion information. Here, the motion information can be the motion information of the current block. The video coding apparatus compares existing modes with the INRVC mode in terms of rate distortion optimization, selects the mode that provides better coding efficiency, and explicitly transmits the selected mode with a signal at a predetermined block level in CTU, CU, picture unit, transform unit, etc. Hereinafter, the flag indicating the use of the INRVC mode will be called the INRVC usage flag. Figure 18 As shown, the video decoding device responds to the parsing flag by decoding the bitstream that indicates the use of the INRVC mode, thereby using the INRVC model instead of the regular mode.

[0166] exist Figure 18 In the example, the video decoding apparatus selects local features from spatiotemporally adjacent blocks in the DFB (Decoding Feature Buffer) 1460 to utilize the INRRC model. Based on the selected local features for correct motion prediction and compensation at the current time t, the video decoding apparatus performs motion prediction using the MTM (Motion Transfer Module) 1450, thereby generating the current features and the block (and therefore, the frame). The video decoding apparatus performs decoding of the video block by utilizing both the video frame decoded according to conventional video coding techniques and the frame generated using the INRRC model.

[0167] This disclosure describes the INRVC model for compressing and reconstructing video frames arranged in time index t, but is not limited thereto. The INRVC model can be applied to process video frames arranged in any order. For example, the INRVC model can be applied to compress and reconstruct multi-view video frames arranged in viewpoint index V.

[0168] In the following text, by using Figure 19 and Figure 20 The illustration depicts a method for generating the current image or current block using the INVRC model.

[0169] Figure 19 This is a flowchart of a method for encoding a current block by a video encoding apparatus according to at least one embodiment of the present disclosure.

[0170] The video encoding device constructs the input to the INRVC model, including the current time index of the current image (S1900).

[0171] The video encoding device constructs the INRVC model (S1902). Here, the INRVC model includes a multilayer perceptron, an upsampling module, and a feature extractor.

[0172] As an example, the video encoding device can construct the INRVC model by training the INRVC model using the input and the original image (i.e., the GT image) to minimize a loss function. Here, the loss function used can be a combination of L1 loss and structural similarity index loss (SSIM loss), as exemplified by the NeRV model.

[0173] The video encoding device applies the input to a multi-layer perceptron of the INRVC model to generate a multi-layer perceptron output (S1904).

[0174] The video encoding device predicts motion information between the current image and a previous output of the INRVC model corresponding to a previous time index (S1906).

[0175] The video encoding device inputs the multi-layer perceptron output combined with the motion information into an up-sampling module of the INRVC model to generate a current output corresponding to the current time index (S1908).

[0176] The INRVC model generates spatio-temporal information from features transmitted by the up-sampling module using a feature extractor.

[0177] As an example, the INRVC model additionally includes a quality evaluator and an optical flow evaluator.

[0178] The INRVC model generates the current output and a previous output corresponding to a previous time index. The INRVC model then estimates an optical flow with respect to the current output and the previous output using the OF estimator. The INRVC model uses the estimated optical flow to improve the current output and the previous output. At this time, the INRVC model calculates a quality of the previous output using the quality evaluator. When the quality of the previous output is equal to or higher than a preset threshold, the INRVC model estimates the optical flow. In contrast, when the quality of the previous output is lower than the preset threshold, the INRVC model skips the optical flow estimation.

[0179] As another example, the INRVC model further includes a deformable convolution module.

[0180] The INRVC model generates the current output and a previous output corresponding to a previous time index. It then generates a temporal correlation between the current output and the previous output using the deformable convolution module. The deformable convolution generates an offset from the current output and the previous output and uses the offset as a basis for adaptively adjusting a size of a convolution kernel. The deformable convolution uses the adjusted convolution kernel to generate the temporal correlation between the current output and the previous output.

[0181] The video encoding device stores the current output in the DFB (S1910).

[0182] When the input to the INRVC model only includes the current time index, the current output corresponds to a reconstructed image of the current frame.

[0183] Alternatively, the input to the INRVC model can be constructed as follows. The video encoding apparatus generates local features. Here, the local features are features of the current block or of a spatio-temporally neighboring block of the current block extracted by a deep learning based extraction model. The video encoding apparatus constructs the input to the INRVC model to include the local features in addition to the current block index and the current time index. The video encoding apparatus predicts the motion information of the current block by using the local features. At this time, as the current output, the video encoding apparatus generates a first reconstructed block of the current block corresponding to the current block index and the current time index.

[0184] The video encoding apparatus applies a regular image encoding process to the current block to generate a second reconstructed block.

[0185] The video encoding apparatus determines an INRVC usage flag based on the first reconstructed block and the second reconstructed block. Here, the INRVC usage flag indicates whether to use the INRVC model on a block-by-block basis. From the perspective of rate-distortion optimization, the video encoding apparatus determines the INRVC usage flag. For example, when the first reconstructed block is optimal, the INRVC usage flag can be determined to be true. In contrast, when the second reconstructed block is optimal, the INRVC usage flag can be determined to be false. The video encoding apparatus encodes the INRVC usage flag.

[0186] The video encoding apparatus checks the INRVC usage flag. If the INRVC usage flag is true, the video encoding apparatus encodes the parameters of the INRVC model according to the encoding procedure shown in FIG. 2B. In contrast, if the INRVC usage flag is false, the video encoding apparatus can encode the encoding information according to the regular image encoding process. Figure 12

[0187] Figure 20 is a flowchart of a method of reconstructing a current block by a video decoding apparatus according to at least one embodiment of the present disclosure.

[0188] The video decoding apparatus constructs an input to the INRVC model including a current time index of a current image (S2000).

[0189] The video decoding apparatus obtains the INRVC model (S2002). Here, the INRVC model includes a multi-layer perceptron, an up-sampling module, and a feature extractor.

[0190] As an example, the video decoding apparatus decodes the parameters of the INRVC model from a bitstream. As shown in FIG. 2B, the video decoding apparatus constructs the INRVC model according to a decoding procedure using the decoded parameters. Figure 13 As an example, the video decoding apparatus decodes the parameters of the INRVC model from a bitstream. As shown in FIG. 2B, the video decoding apparatus constructs the INRVC model according to a decoding procedure using the decoded parameters.​

[0191] The video decoding apparatus applies the input to a multi-layer perceptron of the INRVC model to generate a multi-layer perceptron output (S2004).

[0192] The video decoding apparatus predicts motion information between the current picture and a previous output of the INRVC model corresponding to a previous time index (S2006).

[0193] The video decoding apparatus inputs the multi-layer perceptron output combined with the motion information to an up-sampling module of the INRVC model to generate a current output corresponding to the current time index (S2008).

[0194] The operation of the INRVC model has been described with respect to the video encoding apparatus, so a more detailed explanation is omitted.

[0195] The video decoding apparatus stores the current output in the DFB (S2010).

[0196] When the input to the INRVC model contains only the current time index, the current output corresponds to a reconstructed picture of the current frame.

[0197] Alternatively, the input to the INRVC model can be constructed as follows. The video decoding apparatus decodes an INRVC usage flag from the bitstream. Here, the INRVC usage flag indicates whether the INRVC model is used on a block-by-block basis. If the INRVC usage flag is false, the video decoding apparatus can generate a reconstructed block of the current block according to a regular picture decoding method. In contrast, if the INRVC usage flag is true, the video decoding apparatus constructs the input to generate a reconstructed block of the current block as follows.

[0198] The video decoding apparatus decodes a local feature from the bitstream. Here, the local feature is a latent feature of the current block or a latent feature of a spatio-temporally neighboring block of the current block extracted by an extraction model based on deep learning. The video decoding apparatus constructs the input to the INRVC model to further include the local feature in addition to the index of the current block and the current time index. The video decoding apparatus predicts motion information of the current block using the local feature. At this time, as the current output, the video decoding apparatus generates a reconstructed block of the current block corresponding to the index of the current block and the current time index.

[0199] Although the steps in each flowchart are described as being executed sequentially, the steps merely illustrate the technical idea of some embodiments of the present disclosure. Therefore, a person of ordinary skill in the art to which the present disclosure pertains can execute the steps by changing the order described in each drawing or by executing two or more steps in parallel. Therefore, the steps in each flowchart are not limited to the time sequence order shown.

[0200] It should be understood that the above description presents illustrative embodiments that can be implemented in different other ways. The functions described in some embodiments can be implemented by hardware, software, firmware, and / or a combination thereof. It should also be understood that the functional components described in the present disclosure are marked with "… unit" to strongly emphasize the possibility of their independent implementation.

[0201] Meanwhile, the various methods or functions described in some embodiments can be implemented as instructions stored in a non-transitory recording medium readable and executable by one or more processors. For example, the non-transitory recording medium can include various types of recording devices in which data is stored in a form readable by a computer system. For example, the non-transitory recording medium can include storage media such as erasable programmable read-only memory (EPROM), a flash drive, an optical disk drive, a magnetic hard disk drive, and a solid state drive (SSD), etc.

[0202] Although embodiments of the present disclosure have been described for illustrative purposes, it will be recognized by those of ordinary skill in the art to which the present disclosure pertains that various modifications, additions, and substitutions are possible without departing from the concepts and scope of the present disclosure. Therefore, for the sake of brevity and clarity, embodiments of the present disclosure have been described. The scope of the technical idea of the embodiments of the present disclosure is not limited by the illustration. Therefore, those of ordinary skill in the art to which the present disclosure pertains should understand that the scope of the present disclosure should not be limited to the above explicitly described embodiments, but should be limited by the claims and their equivalents.

[0203] (Reference Signs) 155: Entropy encoder 510: Entropy decoder 1410: Multi-layer perceptron (MLP) 1420, 1440: Up-sampling module 1430: Feature extractor 1450: Motion transfer module (MTM) 1460: Decoded feature buffer (DFB) 1510: Quality evaluator 1520: Optical flow estimator (OF estimator) Cross Reference of Related Applications This application claims priority to and the benefit of Korean Patent Application No. 10-2023-0063866, filed on May 17, 2023, and Korean Patent Application No. 10-2024-0048651, filed on April 11, 2024, the entire contents of each of which are incorporated herein by reference.

Claims

1. A method for reconstructing a current image using a video decoding device, the method comprising: The input to the Implicit Neural Representation Video Coding (INRVC) model is constructed, the input including the current time index of the current image; Obtain the INRVC model including a multilayer perceptron (MLP) and an upsampling module; The input is applied to the multilayer perceptron to generate the multilayer perceptron output; Predict the motion information between the current image and the previous output of the INRVC model corresponding to the previous time index; as well as The combination of the multilayer sensor output and the motion information is input into the upsampling module to generate the current output corresponding to the current time index.

2. The method according to claim 1, wherein, Generating the current output includes: Spatiotemporal information is generated from the features passed by the upsampling module.

3. The method according to claim 1, wherein, Generating the current output also includes: Generate the current output and the previous output of the INRVC model corresponding to the previous time index; Estimate the optical flow of the current output and the previous output; and The optical flow is used to improve the current output and the previous output.

4. The method according to claim 3, wherein, Generating the current output also includes: Calculate the quality of the previously output; and Confirm the quality of the previously output. Specifically, when the quality of the previously output is equal to or higher than a preset threshold, the method performs the step of estimating the optical flow.

5. The method according to claim 1, wherein, Generating the current output includes: Generate the current output and the previous output of the INRVC model corresponding to the previous time index; and Generate the temporal correlation between the current output and the previous output.

6. The method according to claim 5, wherein, Generating the time correlation includes: An offset is generated from the current output and the previous output; The size of the convolution kernel is adaptively adjusted based on the offset; and The adjusted convolution kernel is used to generate the temporal correlation between the current output and the previous output.

7. The method according to claim 1, further comprising: The bitstream decoding indicates whether the INRVC usage flag of the INRVC model is used block by block; as well as Confirm the use of the INRVC flag. Specifically, when the INRVC flag is true, the method performs the step of constructing the input.

8. The method according to claim 7, wherein, Constructing the input includes: Decode local features from the bitstream, wherein the local features are latent features of the current block extracted by a deep learning-based extraction model or latent features of blocks that are spatiotemporally adjacent to the current block; The local features are stored in a decoding feature buffer (decoding feature buffer); and The input is constructed to include the local features in addition to the index of the current block and the index of the current time.

9. The method according to claim 8, wherein, The predicted motion information includes: Using the local features, the motion information of the current block is predicted.

10. The method according to claim 8, wherein, Generating the current output includes: Generate a reconstruction block corresponding to the index of the current block and the current time index of the current block.

11. A method for encoding a current image by a video encoding device, the method comprising: The input to the Implicit Neural Representation Video Coding (INRVC) model is constructed, the input including the current time index of the current image; Construct an INRVC model that includes a multilayer perceptron (MLP) and an upsampling module; The input is applied to the multilayer perceptron to generate the multilayer perceptron output; Predict the motion information between the current image and the previous output of the INRVC model corresponding to the previous time index; as well as The current output corresponding to the current time index is generated by inputting the combination of the multilayer sensor output and the motion information into the upsampling module.

12. The method according to claim 11, wherein, Constructing the input includes: Generate local features, wherein the local features are features of the current block extracted by a deep learning-based extraction model or features of blocks that are spatiotemporally adjacent to the current block; and The input is constructed to include the local features in addition to the index of the current block and the index of the current time.

13. The method according to claim 12, wherein, Generating the current output includes: Generate a first reconstructed block corresponding to the index of the current block and the current time index.

14. The method of claim 13, further comprising: Image encoding processing is applied to the current block to generate a second reconstructed block; Based on the first and second reconstruction blocks, an INRVC usage flag is determined, indicating whether the INRVC model is used block-by-block; and The INRVC is encoded using flags.

15. The method of claim 14, further comprising: Confirm the use of the INRVC flag; as well as When the INRVC usage flag is true, the parameters of the INRVC model are encoded.

16. The method of claim 12, further comprising: The local features are stored in the decoding feature buffer (decoding feature buffer).

17. The method according to claim 11, wherein, Generating the current output also includes: Spatiotemporal information is generated from the features passed by the upsampling module.

18. A method for providing video data to a video decoding device, the method comprising: The video data is encoded into a bitstream; as well as Send the bitstream to the video decoding device. The video data being encoded includes: The input to the INRVC model includes the current time index of the current image; Construct an implicit neural representation video coding (INRVC) model that includes a multilayer perceptron (MLP) and an upsampling module; The input is applied to the multilayer perceptron to generate the multilayer perceptron output; Predict the motion information between the current image and the previous output of the INRVC model corresponding to the previous time index; and The combination of the multilayer sensor output and the motion information is input into the upsampling module to generate the current output corresponding to the current time index.

Images