Video encoding method and device based on implicit neural video representation

Abstract

The present embodiment discloses a video encoding method and device based on implicit neural video representation (INVR). In the present embodiment, a video decoding device decodes the decoder parameters and the embedding of a 2D INVR model. Here, the embedding is a compressed representation of the low-frequency components of the current 2D frame and is generated and provided by an encoder of the 2D INVR model. The video decoding device reconstructs the decoder of the 2D INVR model by using the decoder parameters. The video decoding device reconstructs the current 2D frame by inputting the embedding to the decoder of the 2D INVR model.

Description

Technical Field

[0001] Video Coding Method and Device Based on Implicit Neural Video Representation Background Technology

[0002] The following description provides only background information in relation to this disclosure and does not constitute prior art.

[0003] Recently, research on implicit neural representation models using neural network structures to represent various types of data, including images, has been active. Existing video representation methods use explicit representations with RGB pixel values ​​for each pixel location. To replace this explicit representation, implicit neural representations are being introduced, which represent a function used to generate (r, g, b) values ​​at (x, y) coordinates using a neural network, where (x, y) coordinates are the pixel locations in the image. Compared to explicit representations, implicit neural representations offer practical advantages over existing decoders because they can be used regardless of image resolution and immediately reconstruct (r, g, b) values ​​when given the spatial and temporal coordinates of the input image pixels.

[0004] Meanwhile, video compression techniques based on implicit neural video representation (INVR) – or neural representation for video based on video representation models (NeRV) – generate implicit representations of the video in the pixel domain. According to the neural tangent kernel (NTK) theory associated with neural network learning, the neural network first learns low-frequency components and then high-frequency components. In existing INVR, because learning is performed according to NTK theory, the model size and learning time can be increased to learn high-frequency components. Therefore, to improve video coding efficiency and enhance video quality, it is necessary to construct a video representation model based on implicit neural video representation that considers frequency components. Summary of the Invention

[0005] Technical issues

[0006] The purpose of this disclosure is to provide a video coding method and apparatus that generate compressed embeddings by applying an implicit neural video representation (INVR) model to a video in the frequency domain, sending model parameters and embeddings, and reconstructing the video based on the sent model parameters and embeddings.

[0007] The purpose of this disclosure is to provide a video encoding method and apparatus that compresses low-frequency components of video based on the INVR model and generates high-frequency components of video on the decoder side as needed.

[0008] Technical solution

[0009] According to embodiments of this disclosure, a method for reconstructing a current 2D frame performed by a video decoding device includes: decoding decoder parameters and embeddings of a 2D (dimensional) INVR (implicit neural video representation) model from a bitstream, the embeddings being compressed representations of low-frequency components of the current 2D frame and generated and provided by an encoder of the 2D INVR model; reconstructing a decoder of the 2D INVR model using the decoder parameters; and reconstructing the current 2D frame by inputting the embeddings into the decoder of the 2D INVR model. Reconstructing the current 2D frame may include reconstructing low-frequency components of multiple resolutions by applying convolution-based upsampling to the embeddings, and reconstructing the low-frequency components of the current 2D frame by fusing the low-frequency components of the multiple resolutions.

[0010] According to another embodiment of this disclosure, a method for encoding a current 2D frame performed by a video encoding device may include: generating a compressed embedding by inputting the current 2D frame into an encoder of a 2D (dimensional) INVR (implicit neural video representation) model; reconstructing the current 2D frame by inputting the embedding into a decoder of a 2D INVR model; and encoding the embedding and decoder parameters of the 2D INVR model. Generating the embedding may include generating a low-frequency component of the current 2D frame by transforming the current 2D frame based on a preset transformation method and generating the embedding by applying convolution-based downsampling to the low-frequency component.

[0011] According to another embodiment of this disclosure, a method for providing video data to a video decoding device may include encoding the video data into a bitstream and sending the bitstream to the video decoding device. Encoding the video data may include: generating a compressed embedding by inputting a current 2D frame into an encoder of a 2D (dimensional) INVR (implicit neural video representation) model; reconstructing the current 2D frame by inputting the embedding into a decoder of the 2D INVR model; and encoding the embedding and decoder parameters of the 2D INVR model. Generating the embedding may include generating a low-frequency component of the current 2D frame by transforming the current 2D frame based on a preset transformation method and generating the embedding by applying convolution-based downsampling to the low-frequency component.

[0012] Beneficial effects

[0013] As described above, according to this embodiment, by providing a video coding method and apparatus that generates compressed embeddings by applying an INVR model to video in the frequency domain, transmits model parameters and embeddings, and reconstructs video based on the transmitted model parameters and embeddings, video coding efficiency and video quality can be improved.

[0014] According to this embodiment, by providing a video coding method and apparatus that compresses low-frequency components of video based on an INVR model and generates high-frequency components of video on the decoder side as needed, the overhead required for embedding high-frequency components in the INVR model can be reduced. Attached Figure Description

[0015] Figure 1 This is a block diagram illustrating an INVR (Implicit Neural Video Representation) based video encoding device according to one embodiment of the present disclosure.

[0016] Figure 2 This is a block diagram illustrating an INVR-based video decoding device according to one embodiment of the present disclosure.

[0017] Figure 3 This is an exemplary diagram illustrating the operation of a convolutional layer according to one embodiment of the present disclosure.

[0018] Figure 4 This is an exemplary diagram illustrating a single-image super-resolution (SISR) network.

[0019] Figure 5 This is an exemplary diagram illustrating the residual block used in SISR.

[0020] Figure 6 This is an exemplary diagram illustrating a model for neural representations used in video (NeRV).

[0021] Figure 7a and Figure 7b This is an exemplary diagram illustrating the concept of a NeRV model according to one embodiment of the present disclosure.

[0022] Figure 8 This is an exemplary diagram illustrating the coding pipeline of a NeRV model according to one embodiment of the present disclosure.

[0023] Figure 9 This is an exemplary diagram illustrating a decoding pipeline for a NeRV model according to one embodiment of the present disclosure.

[0024] Figure 10a and Figure 10b This is an exemplary diagram illustrating an INVR model based on spatial information according to one embodiment of the present disclosure.

[0025] Figure 11 This is an exemplary diagram illustrating an INVR model based on spatial information according to one embodiment of the present disclosure.

[0026] Figure 12 This is an exemplary diagram illustrating an encoding pipeline of an INVR model according to one embodiment of the present disclosure.

[0027] Figure 13 This is an exemplary illustration of a decoding pipeline for an INVR model according to one embodiment of the present disclosure.

[0028] Figure 14 This is a flowchart illustrating a method for encoding 2D video using a video encoding device according to one embodiment of the present disclosure.

[0029] Figure 15 This is a flowchart illustrating a method for reconstructing 2D video using a video decoding device according to one embodiment of the present disclosure. Detailed Implementation

[0030] In the following, embodiments of the present disclosure will be described in detail with reference to the exemplary accompanying drawings. It should be noted that when assigning reference numerals to components in each drawing, the same components should have the same reference numerals, even if the same components are shown in different drawings, provided it is practicable. Furthermore, in describing these embodiments, detailed descriptions of related known components or functions will be omitted if it is determined that such detailed descriptions might obscure the subject matter of these embodiments.

[0031] This embodiment relates to the encoding and decoding of images (videos), and more specifically, provides a video encoding method and apparatus that generates compressed embeddings by applying an implicit neural video representation (INVR) model to a 2D video converted to the frequency domain, sending model parameters and embeddings, and reconstructing the video based on the sent model parameters and embeddings.

[0032] Figure 1 This is a block diagram illustrating a 2D INVR-based video encoding device according to one embodiment of the present disclosure.

[0033] exist Figure 1 In the example, a 2D INVR-based video encoding device (hereinafter referred to as the "video encoding device") generates (i.e., trains) a 2D INVR model to learn embeddings for generating 2D video in the frequency domain, compresses the decoder and embeddings of the learned 2D INVR model to generate a bitstream, and transmits the generated bitstream. The video encoding device includes an encoder 110 and a decoder 120, respectively corresponding to the encoder and decoder of the 2D INVR model, and a neural network compressor 130. Here, the components included in the video encoding device according to this disclosure are not necessarily limited thereto. The video encoding device may additionally be provided with a training unit (not shown) for training the 2D INVR model, or may be implemented in the form of being linked to an external training unit.

[0034] Each component of a video encoding device can be implemented in hardware or software, or a combination of both. Furthermore, the functionality of each component can be implemented in software, and a microprocessor can be implemented to execute the software functions corresponding to each component.

[0035] Video encoding devices can store the bitstream of encoded video data in a non-transitory recording medium, or use a communication network to send the bitstream of encoded video data to a 2D INVR-based video decoding device.

[0036] Figure 2 This is a block diagram illustrating a 2D INVR-based video decoding device according to one embodiment of the present disclosure.

[0037] exist Figure 2 In the example, a 2D INVR-based video decoding device (hereinafter referred to as the "video decoding device") decompresses the bitstream to reconstruct a 2D INVR model and embedding, and uses the reconstructed 2D INVR model to generate a 2D video from the embedding. The video decoding device includes a neural network decompressor 210 and a decoder 220, the decoder 220 including a decoder for the reconstructed 2D INVR model.

[0038] and Figure 1 Similar to the video encoding device shown, each component of the video decoding device can be implemented in hardware or software, or a combination of hardware and software. Furthermore, the functionality of each component can be implemented in software, and a microprocessor can be implemented to execute the software functions corresponding to each component.

[0039] In existing INVR systems, an implicit representation of the video is generated in the pixel domain. However, the video encoding apparatus according to this disclosure converts the video to the frequency domain and inputs the video in the frequency domain into a 2D INVR model to perform video compression. When generating an implicit neural network for the video in the frequency domain, the video encoding apparatus separates the low-frequency and high-frequency signals of the video and then trains the implicit neural network. Since high frequencies are more difficult to learn than low frequencies, the video encoding apparatus skips the encoding processing of high-frequency signals based on a predetermined video threshold and prioritizes learning low-frequency signals and embedding them into the implicit neural network. To embed high-frequency signals, the size of the neural network can be increased and the transmission bit rate can be increased. Therefore, the video encoding apparatus can reduce the overhead required to embed high-frequency components by generating high-frequency components in the decoder as needed. Furthermore, the video decoding apparatus can improve the quality of the output video based on a fixed bitstream size.

[0040] In describing Figure 1 and Figure 2 Before showing the components, the deep learning-related component technologies used in this disclosure will be described.

[0041] I-1. MLP (Multilayer Perceptron)

[0042] A multilayer perceptron (MLP) consists of edges connecting multiple neurons. Multiple layers contain multiple neurons. In addition to the input and output layers, an MLP may include one or more hidden layers. Hidden layers may include one or more hidden nodes. Different weights can be assigned to each edge. Activation functions can be used during the process of propagating the output from one layer to the next. Activation functions can include the sigmoid function, the tangent hyperbolic function, or the ReLU (Modified Linear Unit) function.

[0043] The process of updating the weights that make up the MLP to learn the target behavior is called training. Training typically utilizes the stochastic gradient descent (SGD) method based on the backpropagation algorithm. The process of using the weights generated during training to compute the feedforward output of the MLP is called inference or testing.

[0044] In the following text, multilayer neural networks, MLPs, or feedforward networks may be used interchangeably.

[0045] I-2. CNN (Convolutional Neural Network)

[0046] CNN refers to a neural network consisting of multiple convolutional and pooling layers, and is known as the most suitable deep learning technique for image processing. Convolutional layers use multiple kernels or filters to extract feature maps (or something compatible with "features"). The kernel coefficients that make up the filters are parameters determined during the learning process.

[0047] In the convolutional layers of a CNN, the front layers near the input extract feature maps in response to simple and low-level image features (such as lines, points, or surfaces), while the back layers near the output extract feature maps in response to higher-level features (such as textures and object parts).

[0048] Figure 3 This is an exemplary diagram illustrating the operation of a convolutional layer according to one embodiment of the present disclosure.

[0049] Convolutional layers use convolution operations to generate feature maps from the input image. Figure 3 The example shows a kernel (or filter) with a kernel size of 3×3. The kernel size is also called the kernel size or filter size. The kernel has kernel parameters (or filter parameters), also known as weights. Figure 3 The kernel shown has a total of 9 kernel parameters. The kernel parameters are initially set to random values, and their values ​​can be updated based on learning.

[0050] Convolutional layers perform convolution operations using kernel-sized blocks of data from the input image. In this case, the kernel-sized block of data from the input image is called a window.

[0051] When filtering an input image in raster scan order, the size of the window movement is called the stride. Figure 3 In the example, the stride is 1. If the stride is set to 2, the window is spaced 2 samples apart to perform the convolution operation, and therefore, the width and height of the feature map are half the width and height of the input image.

[0052] As mentioned above, a single convolutional layer can include multiple filters. The number of filters, or kernels, is called the number of channels. That is, the number of channels equals the number of filters. Moreover, the number of filters determines the size of the feature map dimension.

[0053] Padding refers to the method of expanding the input data by filling the perimeter of the input data with specific values ​​before performing a convolution operation. Padding is mainly used to control the spatial size of the output data. The value used for padding can be determined by hyperparameters, but zero padding is primarily used. Without padding, the spatial size of the output data decreases with each pass through the convolutional layer, which can lead to the loss of boundary information. Therefore, padding is used to prevent this problem. In other words, padding can be used to match the spatial size of the output data of a convolutional layer with the spatial size of the input data.

[0054] Deconvolutional layers perform the opposite operation to convolutional layers. They generate the desired data image as the output from the feature maps that were input.

[0055] Pooling layers perform pooling, which is the process of subsampling the feature maps generated by convolutional layers. Pooling layers use a 2×2 window to select samples, such that the output is half the width and height of the input. In other words, pooling layers are used to reduce the size of the input image or input feature map by integrating 2×2 regions into a single sample.

[0056] The opposite of pooling layers is defined as non-pooling layers. Unlike pooling layers, non-pooling layers are used to increase the size and are primarily used after deconvolutional layers.

[0057] A convolutional encoder-decoder architecture is a network structure consisting of pairs of convolutional and deconvolutional layers. The convolutional encoder consists of convolutional and pooling layers and outputs a feature map (or feature vector) from the input image. The final output vector of the convolutional encoder is also called the latent vector. The convolutional decoder consists of deconvolutional and non-pooling layers and generates an output image from the feature map or latent vector.

[0058] The inputs and outputs of a convolutional encoder-decoder can be configured differently depending on the application and the purpose of the network. For example, the inputs and outputs can be optical flow maps, saliency maps, image frames, etc.

[0059] Figure 4This is an exemplary diagram illustrating a SISR network.

[0060] As an example of applying CNNs, there is Single Image Super-Resolution (SISR). A SISR network generates a high-resolution image as output from a low-resolution input image. A SISR network can include multiple convolutional layers, such as... Figure 4 As shown. Each convolutional layer includes activation functions such as ReLU (Revised Linear Unit). The parameters of the SISR network can be learned so that the generated SR (super-resolution) image is close to the ground truth (GT).

[0061] SR methods using CNNs can improve SR performance by increasing depth (e.g., increasing the number of convolutional layers). To overcome the overfitting problem that may occur with increasing depth during learning, residual blocks capable of performing skip connections and residual learning can be used in SISR networks. Figure 5 As shown, the residual block includes skip paths besides the path that applies the convolution operation to the input feature xl. Furthermore, when generating the output xl+1, the residual block can select either the path applying the convolution operation or the skip path based on learning efficiency. Figure 5 In the example, the residual block includes a batch normalization (BN) layer.

[0062] For example, EDSR (Enhanced Deep Residual Network for SISR) improves network performance by continuously connecting residual blocks to increase depth. As another example, VDSR (Accurate Image Super-Resolution Using Very Deep Convolutional Networks) is a CNN model based on a VGG (Visual Geometric Group) network that uses residual learning to add residual frames to the final output. VDSR adds residual signals to the input signal by appending the residual signals to the very end of the network.

[0063] I-3. Implicit Neural Representation Model

[0064] Recently, research on implicit neural representation models using neural network structures to represent various types of data, including images, has been active. Existing video representation methods use explicit representations with RGB pixel values ​​for each pixel location. To replace this explicit representation, implicit neural representation models are being introduced, which represent a function that uses a neural network to generate (r, g, b) values ​​at (x, y) coordinates, which are the pixel locations in the image. Compared to explicit representations, implicit neural representation models can be used regardless of image resolution.

[0065] As an example of 2D video coding using an implicit neural representation model, there exists Implicit Neural Video Representation (INVRC) for encoding. The INVRC technique generates (r, g, b) values ​​for (x, y) coordinates, which represent the pixel positions in the image. As another example of the INVRC technique, there exists the NeRV (Neural Representation for Video) model. As mentioned above, explicit representation methods use the pixel position (x, y) within the image and the frame's temporal index t to represent the pixel value on the (x, y, t) grid. Implicit neural representation models take input representing the position (x, y, t) and output RGB pixel values. However, training the neural network at each position (x, y, t) can significantly increase computational complexity, as in... Figure 6 As shown in the example, the NeRV model uses a neural network structure that takes only time index t as input to output the RGB image of the entire frame at time index t.

[0066] Figure 7a and Figure 7b This is an exemplary diagram illustrating the concept of a NeRV model according to one embodiment of the present disclosure.

[0067] To improve the output of an image with a time-indexed input of t, using convolutional layers instead of existing implicit representation models in MLP networks may be more efficient. NeRV models can generate the output using a stacked structure of NeRV blocks consisting of multiple convolutional layers. For example... Figure 7a As shown in the example, the output based on the MLP and the output based on the NeRV block can be compared. Figure 7b As shown in the example, a NeRV block includes convolutional layers, pixel rearrangement layers, and activation layers that perform convolutions. Figure 7b In the example, C represents the number of channels, W and H represent the width and height of the NeRV block input, and S represents the scaling factor.

[0068] Furthermore, the NeRV model, as shown in Equation 1, embeds the time index t into a high-dimensional space and then uses the embedded index as input.

[0069] [Equation 1]

[0070] As shown in the example of Equation 3, the time index t can be mapped to a 2L-dimensional vector γ(t). By using the embedded time index t for training the NeRV model, the NeRV model can better predict video data that includes high-frequency variations.

[0071] As the loss function for the NeRV model, a combination of L1 loss and SSIM (Structural Similarity Index) loss can be used. During the training of the NeRV model, the loss as described above can be computed at all pixel locations of the image estimated by the NeRV model based on the input and the ground truth (GT) image (i.e., the original image). The L1 loss is computed using the absolute value of the difference between pixels in the estimated image and the GT image. The SSIM loss is computed based on the mean, standard deviation, and correlation of pixels in the estimated image and the GT image. Subsequently, the weights (or parameters) constituting the NeRV model can be updated during the training process based on the computed losses.

[0072] Figure 8 This is an exemplary diagram illustrating the coding pipeline of a NeRV model according to one embodiment of the present disclosure.

[0073] Since the original video can be approximated using a NeRV model, end-to-end compression using neural networks can be achieved by compressing and transmitting the NeRV model. The encoding pipeline for encoding the NeRV model may include all or some of the following: a video overfitter 910, a model trimmer 920, a model quantizer 930, and a weight encoder 940, such as... Figure 8 As shown in the example.

[0074] The video overfitter 910 represents the input video frame using the NeRV model. At this point, the video encoding device uses the aforementioned loss function to train the NeRV model, enabling it to represent the video frame. The model pruner 920 uses pruning to simplify the NeRV model structure, which has an MLP or a combination of MLP and convolutional layers. For example, weights less than a preset threshold are pruned to zero. The model quantizer 930 quantizes the pruned weights (e.g., weights greater than or equal to the threshold). The weight encoder 940 applies entropy encoding to the quantized weights to generate a bitstream of weights.

[0075] At the same time, as Figure 8 The reverse of the encoding pipeline shown can be used to decode the NeRV model's decoding pipeline. Figure 9 As shown in the image. Figure 9 As shown, the decoding pipeline may include all or some of the weight decoder 1010, model dequantizer 1020, model reconstructor 1030, and video generator 1040.

[0076] The weight decoder 1010 generates quantized weights by applying entropy encoding to the bitstream of the weights in the NeRV model. The model dequantizer 1020 generates pruned weights by dequantizing the quantized weights. The model reconstructor 1030 reconstructs the NeRV model based on the pruned weights. The video generator 1040 uses the reconstructed NeRV model to generate reconstructed video frames according to time indices.

[0077] As mentioned above, the NeRV model uses the time index t of the frame as input. Therefore, when the input is a video with pixels of size t×h×w, since the NeRV model samples the input video only t times instead of t×h×w times, a large gain can be expected in both encoding and decoding speeds.

[0078] Meanwhile, the encoding pipeline and decoding pipeline described above can also be used for the compression and reconstruction of INVR models.

[0079] In the following text, the implicit neural video representation model, INVR model, and 2D INVR model are used interchangeably.

[0080] II. Implementation of this Disclosure

[0081] Figure 1 The video encoding apparatus shown can generate 2D video by training a 2D INVR model comprising encoder 110 and decoder 120 using 2D video. In this case, a video frame at time t is used as the input and ground truth (GT) of the 2D INVR model. That is, encoder 110 and decoder 120 constitute an autoencoder and can be trained end-to-end. Encoder 110 can be trained to generate embeddings, wherein low-frequency components of the video frame are compressed, and decoder 120 can be trained to generate video frames based on the embeddings. The training unit updates the parameters (weights) of the 2D INVR model using the difference between the output of the 2D INVR model and the GT, enabling the INVR model to learn the generation of 2D video. The trained 2D INVR model can incorporate 2D planar information included in the 2D video based on the parameters within the model.

[0082] The neural network compressor 130 compresses the parameters and embeddings of a trained 2D INVR model to generate a bitstream. For example, it can be used... Figure 8 The illustrated encoding pipeline is used to compress the 2D INVR model. The video encoding device sends the generated bitstream to the video decoding device.

[0083] For example, video encoding devices can encode quantization parameters related to the compression of the INVR model as additional information. Video encoding devices can also encode information about the structure of the INVR model as additional information.

[0084] Figure 2 The video decoding device shown uses a neural network decompressor 210 to decode the parameters and embeddings of a 2D INVR model from a bitstream. For example, the decoder for the 2D INVR model can use... Figure 9 The decoding pipeline shown is used for reconstruction.

[0085] For example, a video decoding device can decode quantization parameters related to the compression of the INVR model into additional information. The video decoding device can also decode information about the structure of the INVR model into additional information.

[0086] Decoder 220 uses the decoder of the reconstructed 2D INVR model to generate 2D video from the embedding. The video decoding device decodes the embedding and then inputs the decoded embedding into the 2D INVR model to generate the reconstructed 2D video.

[0087] Decoder 120 in the video encoding device and decoder 220 in the video decoding device have the same structure. However, decoder 120 in the video encoding device is trained end-to-end using encoder 110 via a training unit, and decoder 220 in the video decoding device operates based on the trained parameters. In the following text, focus will be placed on encoder 110 and decoder 120 in the video encoding device. The structure and operation of decoder 220 in the video decoding device can be replaced by the description of decoder 120 in the video encoding device.

[0088] For example, a video encoding device using the INVR model based on spatial information will be described.

[0089] Figure 10a and Figure 10b This is an exemplary diagram illustrating an INVR model based on spatial information according to one embodiment of the present disclosure.

[0090] The INVR model divides the input video into low-frequency and high-frequency components using 2D Discrete Wavelet Transform (DWT) to improve the learning of the implicit representation of the video based on the mitigation of spectral bias. The encoder 110 of the INVR model uses the low-frequency components, which exhibit minimal variation across frames, to form a compact embedding. The embedding includes spatial information about the input video frames. The decoder 120 of the INVR model reconstructs the low-frequency components based on the embeddings sent by the encoder 110 and uses the reconstructed low-frequency components to reconstruct the high-frequency components.

[0091] Video V = [V1, V2, ..., V] T Any video frame V at time t in ] t It is set as the input of encoder 110. Here, 1 ≤ t ≤ T and V t ∈V. Encoder 110 can transform video V t To generate a signal w in the frequency domain t And perform INVR-based compression. For example, in Figure 10a In the example, encoder 110 may include a 2D DWT module and a CNN-based downsampling block (DB).

[0092] To utilize spatial information even after transformation, encoder 110 can apply a 2D DWT to a video frame to convert it into multiple low-frequency wavelet components. For example, by applying a Haar wavelet transform, encoder 110 can transform video frame V... t Transform into w t =[w t (LL), w t (LH), w t (HL), w t (HH)]. Here, w t (LL), w t (LH), w t (HL) and w t (HH) is a frequency domain signal generated by using low-frequency and high-frequency basis functions in the vertical and horizontal directions using Haar wavelet transform. For example, w t (LL) is a low-frequency signal generated by applying low-frequency basis functions along the vertical and horizontal directions.

[0093] To improve the image quality information of the INVR model, encoder 110 can embed only w t The low-frequency components. Encoder 110 can only transmit w t w t (LL) is input to DB and spatially adaptive embedding is generated based on downsampling. Encoder 110 sends the generated embedding to decoder 120.

[0094] Decoder 120 reconstructs the frequency domain signal w based on embedding. t And for the reconstructed w t_hat Perform an inverse transform to generate the reconstructed frame V. t_hat .like Figure 10a As shown in the example, the decoder 120 may include all or some of the following modules: CNN-based upsampling block (UB), multi-resolution fusion unit (MFU), high-frequency restorer (HFR), and 2D inverse discrete wavelet transform (inverse DWT, IDWT).

[0095] Decoder 120 uses multiple unit blocks (UBs) to reconstruct low-frequency components from the embedding. By sequentially upsampling the spatial size of the embedding to the original size based on 2D convolution and pixel rearrangement, multiple UBs reconstruct the low-frequency components to the original dimensions.

[0096] Because there are limitations to improving reconstruction quality solely through upsampling, decoder 120 improves the quality of the reconstructed low-frequency components by using MFUs with coarse to fine structures. This is achieved by utilizing low-frequency components of various resolutions generated by the UB (e.g., Figure 10a MFU (Methods for reconstructing low-frequency components) can improve the quality of the reconstructed low-frequency components (S1, S2, S3). The reconstructed low-frequency components with improved quality are denoted as w.t_hat (LL).

[0097] like Figure 10b As shown in the example, the MFU includes at least one multi-resolution fusion block (MFB), and each MFB includes a transposed convolution (CT) block and a residual block (RB). The MFB applies transposed convolution to the low-frequency components of the first resolution (e.g., Figure 10b (S1 in the original text) to generate a first low-frequency component with a second resolution. The second resolution of the first low-frequency component may be the same as the resolution of the second low-frequency component (e.g., ...). Figure 10b (S2 in the example). MFB concatenates the first and second low-frequency components that have been applied with transposed convolutions, and sends the concatenated low-frequency components to the residual block. RB consists of two 2D convolution (Conv2D) blocks and a leak-corrected linear unit (LReLU) activation block, such as... Figure 10b As shown. Additionally, RB includes a jump path between the input and output. RB generates a residual of the cascaded low-frequency components and adds the generated residual to the cascaded low-frequency components. By repeatedly applying MFB, MFU can improve the quality of the reconstructed low-frequency components based on the fusion of low-frequency components at various resolutions.

[0098] HFR can be achieved by using the reconstructed low-frequency component w t_hat (LL) Reconstructing the remaining frequency band components w t (LH), w t (HL) and w t (HH) to construct w t_hat HFR includes two Conv2D blocks and an LReLU activation block, such as Figure 10b As shown, HFR can reconstruct the remaining frequency components by applying 2D convolution to the low-frequency components.

[0099] Decoder 120 outputs the reconstructed video frame V by applying the reconstructed frequency components to the 2D IDWT module and performing an inverse discrete wavelet transform. t_hat Instead of learning high-frequency components to reconstruct them, the number of parameters used during training can be reduced by synthesizing high-frequency components based on HFR using decoder 120.

[0100] As described above, the training unit can train the deep learning-based components (DB, UB, MFU, HFR, etc.) included in the encoder 110 and decoder 120 end-to-end. For example, a loss function can be defined based on the difference between the reconstructed frame of the decoder 120 and the ground truth. The training unit can update the parameters of the encoder 110 and decoder 120 included in the INVR model in the direction of reducing the loss function.

[0101] As described above, the decoder 220 in the video decoding device has the same structure as the decoder 120 in the video encoding device, and operates in the same manner as the decoder 120 in the video encoding device based on trained parameters. Therefore, a description of the structure and operation of the decoder 220 in the video decoding device will be omitted.

[0102] The total bits generated by INVR-based compression are the size of the embedding vector and network information of decoder 120. The size of the embedding vector depends on the determination of w t =[w t (LL), w t (LH), w t (HL), w t (HH)] low-frequency band for transmission. Furthermore, the size of the network information of the decoder 120 includes network parameters and network structure information that can be proportional to the network's elasticity. Therefore, it is possible to base the original video V... t With the reconstructed video V t_hat The distortion between D t Combined with the total bits Rt, the low-frequency band to be transmitted is selected according to rate-distortion optimization. For example, the minimum J shown in Equation 2 can be selected. t The low-frequency band.

[0103] [Equation 2]

[0104] In Equation 2, λ is a Lagrange multiplier.

[0105] As previously mentioned, the spatially based INVR model learns each video frame individually. As another example incorporating temporal information, a video coding device using a spatiotemporally based INVR model will be described.

[0106] Figure 11 This is an exemplary diagram illustrating an INVR model based on spatiotemporal information according to one embodiment of the present disclosure.

[0107] The spatiotemporal information-based INVR model not only utilizes embedded information about the current frame, but also additionally uses information about the differences between adjacent frames as input. To utilize this information about the differences between adjacent frames, in addition to the same components as the spatial information-based INVR model, the spatiotemporal information-based INVR model also includes a 2D DWT module and a 1D DWT module.

[0108] Encoder 110 performs a 3D discrete wavelet transform using frame t and the adjacent frames t-1 and t+1 as follows. Encoder 110 uses a 2D discrete wavelet transform to decompose low-frequency and high-frequency components along the spatial axis, and further uses an additional 1D discrete wavelet transform on the 2D transform result to decompose low-frequency and high-frequency components along the time axis. This is in addition to the embedding for frame t (…). Figure 11 In addition to the embedding (t) in the original code, encoder 110 can generate two additional embeddings (t) by downsampling the low-frequency components of the space-time generated using DB. Figure 11 The encoder 110 sends the embeddings to the decoder 120. The decoder 120 reconstructs the frequency domain signal of the embeddings, and as described above, processes the reconstructed w... t_hat Perform an inverse transform to generate the reconstructed frame V at time t. t_hat .

[0109] Since the 2D INVR model can represent the image to be reconstructed, sending the model has the same effect as sending the image. That is, by compressing and sending the 2D INVR model, end-to-end compression techniques using neural networks can be achieved. The following describes a method for compressing the image using encoding parameters and the output embedding of the decoder 120 that generates the image.

[0110] Figure 12 This is an exemplary diagram illustrating an encoding pipeline for a 2D INVR model according to one embodiment of the present disclosure.

[0111] The encoding pipeline for a 2D INVR model may include all or some of the following: a video overfitter 910, a model trimmer 920, a model quantizer 930, an embedding quantizer 1210, and a parametric encoder 1220, such as... Figure 12 As shown.

[0112] The video overfitter 910 represents the reconstructed image based on the parameters of the 2D INVR model. The video coding device uses the aforementioned loss function to train the INVR model, thereby allowing the INVR model to represent video frames. As the loss function, a combination of L1 loss and SSIM loss based on the difference between the reconstructed frame and the ground truth (GT) can be used. In addition to the loss based on the difference between the reconstructed frame and the GT, an additional loss related to the reconstructed 2D wavelet coefficients can be calculated to better represent high-frequency components.

[0113] The model pruner 920 lightens the INVR model by removing parameters of low importance through pruning. For example, a global unstructured pruning technique that removes the least connections on the global scale of the model can be used. As another example, pruning can set parameters below a preset threshold to zero.

[0114] Model quantizer 930 quantizes the pruned parameters (e.g., parameters equal to or greater than a threshold). Embedding quantizer 1210 quantizes the embeddings generated by encoder 110. The final parameters and embeddings of the lightweight INVR model can be quantized into m bits and n bits, respectively (m and n are arbitrary positive integers).

[0115] The parameter encoder 1220 can reduce the size of the bitstream by using lossless compression, such as Huffman coding or entropy coding, for quantization parameters and embeddings. Because there are many pruned parameters, parameter encoding can efficiently compress the model's parameters using fewer bits.

[0116] As mentioned above, video encoding devices can reduce the size of parameters required by video decoding devices by quantizing embedding and reducing the input space size based on DWT.

[0117] For example, video coding devices can encode quantization parameters related to the compression of the INVR model as additional information. Video coding devices can also encode information about the structure of the INVR model as additional information. For example, the IPS (INVR model parameter set) can be used to convey additional information. Video coding devices can include the IPS in the bitstream.

[0118] At the same time, as in Figure 12 The reverse of the encoding pipeline shown in the diagram, the decoding pipeline for decoding the INVR model, can be shown as follows: Figure 13 As shown in Figure 10, the decoding pipeline may include all or some of the following: parameter decoder 1310, embedding dequantizer 1320, model dequantizer 1020, model reconstructor 1030, and video generator 1040.

[0119] The parameter decoder 1310 generates quantized weights and quantized embeddings from the bitstream using, for example, Huffman coding or entropy coding. The embedding dequantizer 1320 dequantizes the quantized embeddings to generate reconstructed embeddings. The model dequantizer 1020 dequantizes the quantized weights to generate pruned parameters. The model reconstructor 1030 reconstructs the INVR model based on the pruned parameters. The video generator 1040 uses the reconstructed INVR model and the reconstructed embeddings to generate reconstructed video frames.

[0120] For example, a video decoding device can decode quantization parameters related to the compression of the INVR model into additional information. The video decoding device can also decode information about the structure of the INVR model into additional information.

[0121] For example, in addition to using 2D DWT, other 1D / 2D transforms can be utilized (e.g., DCT (Discrete Cosine Transform), DST (Discrete Sine Transform, Inseparable Transform, etc.)). For instance, a video encoding device can generate transform coefficients based on arbitrary transforms and then use specific coordinate values ​​(x, y, y) within the transform block / image. th y th The transform coefficients are divided into low-frequency and high-frequency components. If the coordinate values ​​of the transform coefficients are 0 ≤ x ≤ x... th And 0≤y≤y th Then the video encoding device can distinguish the corresponding transform coefficients into low-frequency components and the remaining transform coefficients into high-frequency components. Here, a specific coordinate value x th and y th It is a positive integer greater than or equal to 0 and can be preset according to the protocol between the video encoding device and the video decoding device. Alternatively, the specific coordinate value is determined by the video encoding device for rate-distortion optimization and can be signaled to the video decoding device.

[0122] As another example, when scanning the generated transform coefficients, the video encoding device can distinguish between low-frequency and high-frequency components based on a preset threshold. When scanning starts with high-frequency components, if the scan index is greater than or equal to the preset threshold, the video encoding device can classify the corresponding transform factor as a low-frequency component and the remaining transform coefficients as high-frequency components. When scanning starts with low-frequency components, if the scan index is less than or equal to the preset threshold, the video encoding device can classify the corresponding transform factor as a low-frequency component and the remaining transform coefficients as high-frequency components. Here, the preset threshold is a positive integer greater than or equal to 0 and can be preset according to the protocol between the video encoding device and the video decoding device. Alternatively, the preset threshold can be determined by the video encoding device for rate-distortion optimization and sent as a signal to the video decoding device.

[0123] In the following text, use Figure 14 and Figure 15 This will describe a method for encoding or reconstructing 2D video based on a 2D INVR model.

[0124] Figure 14 This is a flowchart illustrating a method for encoding 2D video using a video encoding device according to one embodiment of the present disclosure.

[0125] The video encoding device generates a compressed embedding by inputting the current 2D frame into the encoder of the 2D INVR model (S1400).

[0126] Video encoding devices can generate embeds as follows.

[0127] The video encoding device generates the low-frequency component and residual frequency component of the current 2D frame by transforming the current 2D frame based on a preset transformation method (S1410). Here, the preset transformation method may include 2D discrete wavelet transform, DCT, DST, or non-separable transform.

[0128] The video encoding device generates an embedding by applying convolution-based downsampling to the low-frequency components (S1412).

[0129] The video encoding device reconstructs the current 2D frame by embedding a decoder into the 2D INVR model (S1402).

[0130] The video encoding device can reconstruct the current 2D frame as follows.

[0131] The video encoding device reconstructs low-frequency components at multiple resolutions by upsampling the embedding (S1420).

[0132] The video encoding device reconstructs the low-frequency components of the current 2D frame by fusing low-frequency components of multiple resolutions (S1422).

[0133] The video encoding device reconstructs the remaining frequency components of the current 2D frame by applying 2D convolution to the low-frequency components of the current 2D frame (S1424).

[0134] The video encoding device performs an inverse transformation on the low-frequency components and remaining frequency components of the current 2D frame based on a preset transformation method (S1426).

[0135] Video encoding devices can train the encoder and decoder of a 2D INVR model end-to-end based on a loss function. Here, the loss function can be defined based on the difference between the input frames of the encoder and the output frames of the decoder.

[0136] The video encoding device encodes the embedding and decoder parameters of the 2D INVR model (S1404).

[0137] In addition, video encoding devices can quantize decoder parameters and embeddings based on quantization parameters. The video encoding device can then encode the quantization parameters into additional information.

[0138] Figure 15 This is a flowchart illustrating a method for reconstructing 2D video using a video decoding device according to one embodiment of the present disclosure.

[0139] The video decoding device decodes the embedding and decoder parameters of the 2D INVR model from the bitstream (S1500). Here, the embedding is a compressed representation of the low-frequency components of the current 2D frame and is generated and provided by the encoder of the 2D INVR model.

[0140] In addition, the video decoding device decodes quantization parameters from the bitstream as additional information. The video decoding device decodes the decoder parameters and embeds them using the quantization parameters.

[0141] The video decoding device uses the decoder parameters to reconstruct the 2D INVR model decoder (S1502).

[0142] The video decoding device reconstructs the current 2D frame by embedding a decoder into the 2D INVR model (S1504).

[0143] The video decoding device can reconstruct the current 2D frame as follows.

[0144] The video decoding device reconstructs low-frequency components at multiple resolutions by applying convolution-based upsampling to the embedding (S1514).

[0145] The video decoding device reconstructs the low-frequency components of the current 2D frame by fusing low-frequency components of multiple resolutions (S1512).

[0146] The video decoding device reconstructs the remaining frequency components of the current 2D frame by applying 2D convolution to the low-frequency components of the current 2D frame (S1516).

[0147] The video decoding device performs an inverse transformation on the low-frequency components and remaining frequency components of the current 2D frame based on a preset transformation method (S1518). Here, the preset transformation method may include 2D discrete wavelet transform, DCT, DST, or non-separable transform.

[0148] The encoder and decoder of a 2D INVR model can be pre-trained end-to-end based on a loss function. The loss function can be defined based on the difference between the input frames of the encoder and the output frames of the decoder.

[0149] Although the flowcharts / timing diagrams in this specification describe steps executed sequentially, this is merely an illustrative interpretation of the technical concept of one embodiment of this disclosure. In other words, those skilled in the art to which one embodiment of this disclosure pertains can modify and adapt the flowcharts / timing diagrams in various ways without departing from the essential characteristics of one embodiment of this disclosure, such as changing the order described in the flowcharts / timing diagrams or executing one or more processes in parallel. Therefore, the flowcharts / timing diagrams are not limited to a temporal order.

[0150] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described in this specification are labeled "...unit" to particularly emphasize their implementation independence.

[0151] Furthermore, the various functions or methods described in this embodiment can be implemented as instructions stored in a non-transitory recording medium that can be read and executed by one or more processors. For example, a non-transitory recording medium includes any type of recording device in which data is stored in a form readable by a computer system. For example, non-transitory recording media include storage media such as EPROM (erasable programmable read-only memory), flash memory drives, optical disk drives, magnetic hard disk drives, and solid-state drives (SSDs).

[0152] The above description is merely an illustrative explanation of the technical concept of this embodiment, and those skilled in the art can make various modifications and changes within the scope of the essential features of this embodiment. Therefore, this embodiment is intended to illustrate, not limit, the technical concept of this embodiment, and the scope of the technical concept of this embodiment is not limited by these embodiments. The protection scope of this embodiment should be determined by the protection scope of the claims, and all technical concepts within the equivalent scope should be determined by the protection scope of the claims.

[0153] Cross-references to related applications

[0154] This patent application claims priority to Korean Patent Application No. 10-2023-0176421, filed on December 7, 2023, and Korean Patent Application No. 10-2024-0154232, filed on November 4, 2024, all of which are incorporated herein by reference.

Claims

1. A method for reconstructing a current 2D frame, performed by a video decoding device, the method comprising: Decoder parameters and embeddings of a 2D implicit neural video representation INVR model are decoded from a bitstream. The embeddings are compressed representations of the low-frequency components of the current 2D frame and are generated and provided by the encoder of the 2D INVR model. The decoder is used to reconstruct the decoder of the 2D INVR model using the decoder parameters; as well as The embedding is input into the decoder of the 2D INVR model to reconstruct the current 2D frame. The reconstruction of the current 2D frame includes: Convolution-based upsampling is applied to the embedding to reconstruct low-frequency components at multiple resolutions; and The low-frequency components of the current 2D frame are reconstructed by fusing the low-frequency components from the multiple resolutions.

2. The method according to claim 1, wherein, Reconstructing the current 2D frame further includes: applying 2D convolution to the low-frequency components of the current 2D frame to reconstruct the remaining frequency components of the current 2D frame.

3. The method according to claim 2, wherein, Reconstructing the current 2D frame includes: performing an inverse transformation on the low-frequency components and the remaining frequency components of the current 2D frame based on a preset transformation method.

4. The method according to claim 1, wherein, The encoder and decoder of the 2D INVR model are pre-trained end-to-end based on a loss function, which is defined based on the difference between the input frame of the encoder and the output frame of the decoder.

5. The method according to claim 1, further comprising: Decode the quantization parameters from the bitstream as additional information; The decoder parameters and the embedding are dequantized using the quantization parameters.

6. A method for encoding a current 2D frame performed by a video encoding device, the method comprising: The current 2D frame is input into the encoder of the 2D implicit neural video representation (INVR) model to generate a compressed embedding; The embedded input is fed into the decoder of the 2D INVR model to reconstruct the current 2D frame; as well as The embedding and decoder parameters of the 2D INVR model are encoded. Generating the embedding includes: Based on a preset transformation method, the current 2D frame is transformed to generate the low-frequency component of the current 2D frame; and The embedding is generated by applying convolution-based downsampling to the low-frequency components.

7. The method according to claim 6, wherein, Reconstructing the current 2D frame includes: Convolution-based upsampling is applied to the embedding to reconstruct low-frequency components at multiple resolutions; and The low-frequency components of the current 2D frame are reconstructed by fusing low-frequency components from multiple resolutions.

8. The method according to claim 6, wherein, Reconstructing the current 2D frame further includes: applying 2D convolution to the low-frequency components of the current 2D frame to reconstruct the remaining frequency components of the current 2D frame.

9. The method according to claim 8, wherein, Reconstructing the current 2D frame further includes: performing an inverse transformation on the low-frequency component and the remaining frequency component of the current 2D frame based on a preset transformation method.

10. The method of claim 6, further comprising: The encoder and decoder of the 2D INVR model are trained end-to-end based on the loss function. The loss function is defined based on the difference between the input frame of the encoder of the 2D INVR model and the output frame of the decoder of the 2D INVR model.

11. The method of claim 6, further comprising: The decoder parameters and the embedding are quantized based on the quantization parameters; as well as The quantization parameters are encoded as additional information.

12. A method for providing video data to a video decoding device, the method comprising: The video data is encoded into a bitstream; as well as Send the bitstream to the video decoding device; Encoding the video data includes: The current 2D frame is input into the encoder of the 2D implicit neural video representation (INVR) model to generate a compressed embedding. The embedding is input into the decoder of the 2D INVR model to reconstruct the current 2D frame; and The embedding and decoder parameters of the 2D INVR model are encoded. Generating the embedding includes: Based on a preset transformation method, the current 2D frame is transformed to generate the low-frequency component of the current 2D frame; and The embedding is generated by applying convolution-based downsampling to the low-frequency components.

Images